Registration and tracking by 3D scanning the topology of anatomic structures

Abstract

A system and method for registering and tracking anatomical structures during surgical procedures by leveraging continuously updated 3D topological data of objects within the surgical field. Registration allows different datasets such as preoperative CT scans and intraoperative 3D scans to be aligned, combined, and transformed into a common coordinate space, enabling more precise surgical planning and execution. Tracking involves continuously registering these datasets over time, monitoring changes in position or orientation, and keeping objects aligned within the common coordinate space, updating the spatial alignment of registered objects multiple times per second, which allows for dynamic monitoring of anatomical structures and surgical tools as they move throughout the procedure. Unlike traditional systems that rely on physical optical markers (OMs) and optical tracker constellations (OTCs), the system and method disclosed herein utilizes the data captured by a 3D scanner to perform Continuous Anatomic Auto Tracking (CAAT).

Description

VIS-002-W01REGISTRATION AND TRACKING BY 3D SCANNING THE TOPOLOGY OF ANATOMIC STRUCTURESCROSS-REFERENCES TO RELATED APPLICATIONS

[0001] The present application claims the benefit of priority from U.S. Provisional Application Serial No. 63 / 705,487, filed October 9, 2024 and entitled “Registration and Tracking in Surgery Using 3D Scanning,” U.S. Provisional Application Serial No. 63 / 716,932, filed November 6, 2024 and entitled “Registration & Tracking by 3D Scanning the Topology of Anatomic Structures”, and U.S. Provisional Application Serial No. 63 / 717,095, filed November 6, 2024 and entitled “Cartilage and Bone Misalignment Correction in Computed Tomography Image-Based Surgical Navigation”, and U.S. Provisional Application Serial No. 63 / 844,018, filed July 14, 2025 and entitled “3D Scanner”, the entire contents of which are hereby incorporated by reference into this disclosure as if set forth fully herein.FIELD

[0002] The present disclosure relates generally to object scanning, and more specifically to using three-dimensional (3D) scanning to register and track 3D objects in a real environment relative to objects in a virtual environment.BACKGROUND

[0003] In modern surgical practice, particularly in orthopedic procedures like Total Knee Arthroplasty (TKA) and Hip Arthroplasty, precise alignment and tracking of anatomical structures are crucial for successful outcomes. Over the past two decades, surgical navigation systems have become standard tools for enhancing accuracy during these procedures. These systems typically rely on optical trackers and stereoscopic cameras to guide surgeons through intricate surgical steps, ensuring correct positioning of instruments and implants.

[0004] By way of example, an optical tracker (OT) is a system used to track the position of anatomical structures, instruments, or equipment in the surgical field. An optical tracker is composed of a collection of optical markers (OMs) arranged in a specific geometric pattern, or a "constellation." These constellations allow the system to precisely identify and differentiate between tracked objects based on the unique arrangement of their optical markers.VIS-002-W01

[0005] By way of example, an optical marker (OM) is a marker used in traditional surgical navigation systems to track the position of anatomical structures and tools in real-time via stereoscopic cameras. One common example is a reflective sphere, though multiple types of optical markers can be used depending on the system and application.

[0006] However, despite their effectiveness, these systems present several challenges. The setup of optical trackers is often time-consuming and technically complex, requiring precise calibration and positioning of cameras and tracking equipment. Additionally, the physical nature of optical trackers introduces mechanical vulnerabilities — trackers can become dislodged, damaged, or obstructed during surgery (e.g., affecting line of sight between the camera and the OTs), leading to potential inaccuracies in tracking and necessitating recalibration. These issues add to overall procedural time, which may increase patient exposure to anesthesia and complicate workflow.

[0007] Moreover, the reliance on optical trackers and complex equipment adds significant costs and requires highly trained personnel to operate the systems. These challenges have limited the widespread adoption of such systems, particularly in smaller or resource-constrained healthcare facilities. As a result, there is a pressing need for more robust and efficient solutions that maintain or improve the accuracy of surgical navigation while reducing complexity and cost.

[0008] The following disclosure generally describes methods currently used in the field of surgical navigation, for example a navigation system using optical markers and stereoscopic cameras. By way of example only, the methods discussed and / or disclosed herein are done so in the context of a Total Knee Arthroplasty (TKA), however it should be understood that the systems and methods disclosed herein may be used in any scenario (surgical, medical, or otherwise) to generate a reconstructed 3D model of an object.

[0009] Optical markers (OMs) are small reflective markers that are easily segmented from other elements in an image. These markers retro-reflect infrared light emitted by stereoscopic cameras, making them highly detectable to cameras that are located close to the near infrared (NIR) source. OMs are essential for tracking anatomical landmarks, surgical instruments, and equipment during surgery, ensuring reliable visibility even in complex surgical environments.VIS-002-W01

[0010] Optical markers (OMs) are often mechanically grouped into specific rigid geometric patterns, known as optical trackers (OTs). These unique arrangements allow the system to differentiate between various anatomical structures, instruments, or equipment being tracked. By way of example, OTs can be attached to patient anatomy (e.g., bones) near the surgical site, mounted on surgical instruments, (e.g., pointers or measurement tools), and / or fixed to active equipment (e.g., robotic arms, saws, or drills). The distinct geometries of OTs enable the system to identify what each OM represents within the surgical field.

[0011] Stereoscopic cameras are foundational to spatial tracking in surgical navigation systems. These cameras capture images from slightly different angles, similar to human binocular vision. By detecting the reflective OMs in the surgical field, the cameras calculate the 3D positions of all visible optical trackers (OTs).

[0012] Typically, the navigation system compares the relative positions of the detected OMs to the predefined geometric pattern of each OT. Once an OT is identified, it is assigned to its corresponding surgical tool, anatomical structure, or piece of equipment. The identified OTs are then registered in a common coordinate system, or space, providing a comprehensive 3D representation of the surgical field. The system can now track the position and orientation of all OTs in real time.

[0013] Once the positions of the OTs are known in the common coordinate system, they are transformed into a Surgical Coordinate System (SCS), which provides a fixed reference frame for the procedure. The SCS could be equivalent to camera space, CT space, tracker space, or based on another tracked location, such as the base of a robot. By applying a 4x4 rigid transformation matrix 2 (see Figure 1), any OT can be positioned within the SCS, ensuring consistency in tracking across the system. To accurately transform the object’s origin into the SCS, a sequence of transformations may be applied. The process involves transformations between, for example, an object’s local space, OT space, camera space, and a final SCS, which is a chosen common space for all relevant items for navigation.

[0014] By applying these transformations, the system ensures precise positioning of objects within the SCS, providing consistent and accurate tracking across the surgical environment. It should be noted that each transformation matrix introduces error into the equation. Because ofVIS-002-W01 this, the more transformations that are required to move an object into the SCS, the greater the potential for error. For example, in the equation presented above, three transformations are required.

[0015] For surgical navigation systems to be effective, they must track rigidly attached OTs in real-time, updating positions multiple times per second. This continuous updating enables precise monitoring of tool positions, implant alignment, and anatomical movement, ensuring accurate and timely adjustments throughout the surgery.

[0016] Locating anatomy poses unique challenges because the position of the Optical Tracker (OT) relative to the anatomy is not always well defined. When an OT is attached to an instrument, its location relative to the rest of the instrument is precisely engineered and known. However, with anatomical structures, the initial placement of an OT follows general guidelines but is somewhat random within those parameters.

[0017] Another reason this anatomical registration can be difficult is the attachment method. For instruments, the OT can be attached rigidly with screws, ensuring stability and precision. However, for anatomical structures, the attachment must be created during surgery. For TKA, this typically involves drilling two pins bi-cortically (through both sides of the bone) and then attaching the OT to the pins. These pins are subject to flex due to the bone's characteristics, and the bone quality may not be ideal, further complicating the attachment. The OT must always be “visible” to the stereo camera and not be obscured by surgeons, OR staff, instruments, drapes or other equipment to function properly. Additionally, the OT may be bumped or deflected during the procedure by the surgeon or other equipment, leading to inaccuracies in tracking.

[0018] OTs are securely attached to bony landmarks using these pins or clamps. For instance, in procedures such as Total Knee Arthroplasty (TKA), OTs might be placed on the femur and tibia. These OTs serve as proxies for the position of the underlying anatomy, allowing for continuous tracking throughout the procedure.

[0019] Before the OTs attached to anatomy can be used to locate the anatomical structure within the SCS, the anatomy must first be registered. This registration process typically involves identifying key anatomical landmarks, such as specific points on the bone, and matching themVIS-002-W01 with pre-operative imaging (CT or MRI scans). The system then uses the positions of the attached OTs to align the anatomy with the pre-operative 3D model.

[0020] Surgeons often use a pointer or probe (equipped with its own OT) to touch specific anatomical points, the tip-points of which are then recorded by the system. These points help to map (register) the patient’s anatomy with preoperative imaging or a model, allowing for accurate tracking during surgery. However, because the pointer introduces another OT and another transformation matrix into the sequence of transformations required to locate anatomy in the SCS, potential for error is increased.

[0021] Before surgery, detailed imaging of the target anatomy is often obtained using CT (Computed Tomography) or MRI (Magnetic Resonance Imaging). These scans create a 3D model of the anatomy, which serves as a reference during the procedure. This model is crucial for planning surgical interventions and aligning the patient’s real-time anatomy with the preoperative plan.

[0022] However, some systems and robotic surgical systems do not rely on pre-operative imaging. In these cases, a standard 3D model of bones or other anatomy is used during the planning process. These standard models have landmarks identified within them, which can be used to properly locate the standard model. To make the model more visually realistic in surgical planning applications, the standard models can be scaled, deformed, and / or sheared using an “affine” transformation matrix if needed. By way of example, comparing the landmark data in the CT and the landmark data obtained using a pointer, this affine transformation matrix can be computed. For instance, if an OT attached to a pointer is used to locate the medial and lateral epicondyles of the femur, the real distance between these two points could be calculated and used to scale the standard model. In this way, an appropriately sized model can be used to visualize the placement of various implants in proper scale, though it may not be anatomically accurate for that patient. Other transformation matrices noted herein that are for registration or tracking are rigid transformations and do not include scaling, deformation or shearing effects.

[0023] In both cases, whether pre-operative images are available or not, the process is essentially similar and will be treated as such for this disclosure. If pre-operative data, CT data,VIS-002-W01 or similar data is referenced, it is possible that a scaled, deformed, and / or sheared standard model could be used in its place.

[0024] Example Workflow of Optical Tracking Systems for TKA (Prior Art)

[0025] All Total Knee Arthroplasty (TKA) surgeries are different. The patients are different, and the surgeons who perform the procedures have varying training, experience, and methodologies that shape their approach to surgery and their definition of an ideal TKA. There are also many different surgical navigation systems, robotic assistants, and implant systems to choose from. Because of these differences, a standard TKA workflow cannot be universally defined. The procedure that follows is an example of a typical prior art system. By way of example this workflow is represented in Figure 2.

[0026] Prior to surgery, detailed imaging of the target anatomy is often acquired using CT (Computed Tomography) or MRI (Magnetic Resonance Imaging). These scans are segmented to create a 3D model of the relevant structures, such as the femur and tibia in TKA. These 3D models are essential for planning the procedure and aligning the patient's real-time anatomy with the surgical plan. Important anatomical planes and landmarks are also labeled within the model to assist the surgeon during the operation.

[0027] In some systems, pre-operative imaging is not required. This is referred to as "imageless" mode. In this approach, a standard 3D model of the bones is used instead of patientspecific scans. Anatomical landmarks, such as the medial and lateral epicondyles of the femur, are identified during surgery using OTs and a pointer. The distance between these landmarks is used to scale the standard model to match the patient’s unique anatomy, allowing the surgeon to visualize implant placement with appropriate proportions, even in the absence of patient-specific imaging. Simple scaling or uniform scaling may be used but it is likely that more advanced scaling is used where the scaling factor in various directions is different. There may also be some twisting, deformation and / or shearing required to make the landmarks line up best.

[0028] For this workflow, we assume a patient-specific CT scan has been segmented to provide a detailed 3D model of the femur and tibia, though the imageless mode can also be utilized if necessary.VIS-002-W01

[0029] Before surgery begins, the system requires calibration. Stereoscopic cameras are positioned to ensure full visibility of the surgical field, and OTs may be attached to the surgical instruments, the robotic arm, and the robotic base. Calibration ensures that the system aligns correctly with the surgical setup, accounting for distances, camera angles, and the positioning of all equipment.

[0030] Robotic equipment requires additional steps for calibration to ensure that the robot functions correctly. This process includes aligning the robot's physical movements with the virtual model of the surgery to maintain accuracy and ensure that the robot performs as expected during the procedure.

[0031] After setup and calibration, the patient is positioned on the operating table, typically in a supine position with the knee exposed for TKA procedures. The surgical site is prepped and draped in a sterile fashion. Surgical exposure follows, with an incision made to access the knee joint and prepare it for the robotic system to perform the procedure.

[0032] Patient registration starts when OTs are attached to the patient’s anatomy, typically using two pins driven bi-cortically. For femoral registration, the camera system tracks both the OT attached to the femur and the OT attached to a pointer. The system’s user interface guides the surgeon through the process of identifying specific anatomical points on the femur. During this process, the surgeon collects approximately 50 or more points on the surface of the femur, though the number may vary depending on the system.

[0033] These points generate a sparse point cloud representing the surface of the femur. The point cloud is registered to pre-operative imaging (CT, MRI scans, or scaled standard models) using algorithms like RANSAC (Random Sample Consensus) and ICP (Iterative Closest Point). Once registration is complete, the system knows how the pre-operative imaging data must be transformed to align properly in camera space (and vice versa). With trackers also captured in camera space, and their registration to their respective OT virtual spaces, the real-time anatomical model can then be transformed and maintained within the chosen Surgical Coordinate System (SCS), enabling accurate tracking of the patient’s anatomy throughout the procedure.VIS-002-W01

[0034] Following patient registration, the surgeon identifies key anatomical landmarks necessary for establishing the mechanical axis of the leg, for example including the hip center (e.g. identified by rotating the leg while the optical tracking system calculates the center of rotation of the femoral head), knee center (e.g., identified either from pre-operative imaging or by locating the medial and lateral epicondyles of the femur), and / or ankle center (e.g., identified using the optical tracking system on the medial and lateral malleoli (ankle bones) to calculate the center of the ankle joint). These landmarks are crucial for ensuring proper implant alignment. The mechanical axis is one common methodology, but other alignment methods, like kinematic alignment, may use different anatomical landmarks depending on the procedure.

[0035] At this stage, a virtual surgical plan is developed (e.g., by the surgeon based on their experience and surgical philosophy) to guide the robotic system and surgeon throughout the procedure. If pre-operative planning was done, it serves as an initial basis but must be verified and perhaps adjusted in real time as the surgery progresses.

[0036] The virtual plan typically involves the following steps, which may be applied in a different order or iteratively, depending on the system:

[0037] 1. Implant Size Selection. The system assists the surgeon in selecting the optimal implant size based on the patient’s anatomy. Proper implant sizing ensures both the long-term success of the surgery and post-operative joint function.

[0038] 2. Implant Positioning. The system helps the surgeon determine the ideal implant position to achieve accurate joint alignment. The goal is to ensure proper leg alignment and balance between the medial and lateral sides of the knee. The alignment is measured in terms of varus (inward tilt) and valgus (outward tilt), with the system assisting the surgeon in making appropriate adjustments.

[0039] 3. Gap Balancing. By way of example, some surgeons may assess joint laxity by measuring gaps on the medial and lateral sides of the knee at various flexion angles, using this information to guide implant positioning in a process called “gap balancing.” Gap balancing strives for equal tension in the soft tissues (primarily ligaments) around the knee joint throughout its range of motion. The system dynamically tracks the relative positions of the bones andVIS-002-W01 calculates the space between them to guide the surgeon in balancing the medial and lateral gaps during the procedure. However, it should be noted that surgical techniques and philosophies can vary significantly, and some surgeons omit this step entirely if they do not subscribe to a gap balancing philosophy.

[0040] 4. Osteotomy Planning. Based on the selected implant size and position, the system defines the necessary osteotomies (bone cuts). These cuts must be precise to ensure proper implant placement. By way of example, the necessary bone cuts are determined not just by general planning but specifically by the selected implant size and its final planned position, which are derived from the geometry and design specifics of the chosen implant system.

[0041] 5. Robotic-Assisted Bone Resection. During bone resection, the robotic system assists the surgeon in making precise cuts according to the surgical plan. By way of example, the robotic system assists the surgeon by maintaining the saw blade within the correct cutting plane, while still allowing the surgeon to control when the saw is activated and to move the saw freely within that defined plane as needed during the procedure. The system continuously tracks the patient's anatomy and provides feedback, ensuring the cuts are executed accurately.

[0042] 6. Intraoperative Verification and Adjustment. After each bone cut, the accuracy of the cut is verified. By way of example, the surgeon typically performs this cut verification by using an instrument specifically designed for this purpose, which is equipped with its own OT, to manually check the accuracy of the cut. If deviations are detected between the actual cuts and the surgical plan, the surgeon may need to adjust the plan to correct for any differences.

[0043] 7. Trialing and Final Implantation. After bone cuts are verified, trial implants are placed to assess the fit, alignment, and joint movement. If necessary, adjustments are made before the final implant is inserted.

[0044] 8. Post-Operative Evaluation and Closure. The surgeon performs a final assessment to ensure proper alignment and stability of the new joint. Once confirmed, the surgical site is closed, and the patient is prepared for recovery.

[0045] Despite their widespread use, stereoscopic camera and optical tracking systems present several challenges that affect their efficiency and accuracy in surgical applications.VIS-002-W01

[0046] For example, one challenge of stereoscopic camera and optical tracking systems is the complex setup and calibration of the optical tracking system. The setup of optical tracking systems is time-intensive and requires precise positioning of cameras and optical trackers. Even slight misalignment during calibration can lead to errors in tracking the patient's anatomy or surgical instruments, reducing surgical precision. Additionally, setup time adds to the length of the procedure, potentially increasing patient exposure to anesthesia and overall operative time.

[0047] Another challenge of stereoscopic camera and optical tracking systems is their reliance on optical trackers. The physical nature of optical trackers introduces several potential points of failure. For example, tracker placement can affect surgical exposure, requiring larger incisions or altered approaches that may increase invasiveness. Additionally, trackers are prone to physical damage, shifting, or detachment during surgery. Further, if any optical marker on the tracker becomes dirty during surgery, it may affect its registration to the virtual OT space, reducing accuracy. Certainly, if a tracker becomes dislodged or its line of sight is sufficiently obstructed, the system cannot accurately track the anatomy or instruments, leading to error. The specialized optical trackers, cameras, and calibration equipment are expensive, limiting their use to well-funded hospitals and surgical centers. The additional equipment also increases the logistical complexity of the operating room. These systems include a lot of equipment and specific procedures must be followed to ensure their proper use and functioning. Thus, the training and learning curve associated with these systems is substantial. While these systems can save time in some areas, they take time to set up and use properly. Depending on the surgeon this could be a net savings but can also result in a net increase in surgical time.

[0048] Additionally, every time an optical tracker is used, an additional transformation matrix is required to localize it and the additional transformations may introduce additional errors. Such transformations lead to errors depending on the distance the trackers have with the anatomy in question. In many cases trackers are more than 6 inches away from relevant anatomy, so any angular error is magnified by that distance. It is an advantage to navigate based on anatomy to remove this distance error as much as possible.

[0049] As mentioned, with current optical tracking navigation systems the line of sight between the cameras and optical trackers must remain unobstructed for the system to functionVIS-002-W01 effectively. However, during surgery, this line of sight can be blocked by surgical staff or equipment, causing the system to lose track of the optical markers. Limited field of view (FOV) of stereoscopic cameras can cause tracking loss if anatomy or instruments move outside the tracked area, necessitating frequent repositioning. These interruptions necessitate recalibration or repositioning, disrupting the surgical workflow and potentially adding stress and time to the procedure. For example, intraoperative re-registration is required after a tracker becomes dislodged or bumped such that it shifts relative to anatomy. This re-registration can significantly prolong surgery, impacting workflow and anesthesia time.

[0050] Additionally, there are limits to the speed of the system. If OTs are moving too fast, the system can lose registrations. When this happens, the system must notify the surgeon, and the work must slow down.

[0051] High cost and limited adoption is another challenge of current optical tracking navigation systems. The substantial resources required to acquire, maintain, and operate these systems have limited their widespread adoption. Beyond the financial investment, the time and effort needed for training surgical teams, the complexity of setup, and the extended procedural time also present significant barriers. These systems demand highly trained personnel to ensure proper use, and the intricate setup process can increase the overall time required for surgeries. Additionally, the ongoing need for system and / or tool calibration and troubleshooting during procedures adds to the operational complexity, making it challenging for smaller hospitals and clinics to adopt these systems. As a result, accessibility remains limited, particularly in resource- constrained environments.

[0052] While stereoscopic cameras and optical trackers have become valuable tools for improving surgical precision, they come with several limitations. The reliance on physical optical trackers, complex setup, time-intensive calibration, and susceptibility to mechanical failures all point to the need for a more robust and flexible solution. Additionally, the combination of high financial costs, lengthy training requirements, and the complexity of operating these systems has made broader adoption across surgical specialties challenging.

[0053] In image-based surgical navigation, pre-operative CT imaging is widely used due to its capability to generate detailed 3D data of hard tissues, especially bone. CT imaging is highlyVIS-002-W01 regarded for surgical planning in procedures such as TKA because of its high spatial resolution, allowing for precise segmentation of bone structures. Bone, with its high density and X-ray attenuation properties, appears distinctly in CT scans, making it straightforward to segment and align in navigation workflows. However, the same properties that make bone highly visible on CT scans create limitations for visualizing cartilage and other soft tissues.

[0054] Cartilage presents several technical challenges for CT imaging, including lower X-ray attenuation, lack of distinct radiodensity, and partial volume effect. Regarding lower X-ray attenuation, cartilage is a soft tissue with significantly lower density and X-ray attenuation compared to bone. This means it absorbs fewer X-rays, resulting in poor contrast on CT images, where it often appears as an indistinct or faint structure. This lower contrast is especially problematic when cartilage is surrounded by other soft tissues, as the boundaries between these tissues are less distinct, making it difficult to segment cartilage accurately. Regarding lack of distinct radiodensity, unlike bone which has a high and consistent radiodensity, cartilage does not provide a sharp enough contrast to allow clear boundary detection. This lack of contrast makes it challenging to differentiate cartilage from surrounding tissues such as ligaments or synovial fluid, further complicating segmentation. Regarding partial volume effect, the thickness of cartilage can vary across articular surfaces, and in areas where cartilage is thin, CT imaging may fail to capture its structure adequately. This results in partial volume effects, where the cartilage appears blended with adjacent soft tissue or bone, leading to inaccurate or incomplete representation in the resulting 3D model.

[0055] Due to these limitations, CT imaging data used for surgical planning typically focuses on bone structures, where high contrast and spatial resolution allow accurate segmentation. However, cartilage, being poorly visible on CT, is generally omitted or only partially represented. As a result, the CT data lacks the comprehensive anatomical detail needed to accurately model the joint surface, particularly where cartilage plays a critical role in joint alignment.

[0056] For example, with the use of optical topological scanners, a critical technical issue emerges: the pre-operative CT data and the intraoperative optical scan represent different anatomical surfaces. CT data accurately reflects the bone structure but provides little to no dataVIS-002-W01 on the articular cartilage surface. In contrast, the topological scanner captures the visible surface, which in a typical TKA exposure consists largely of cartilage rather than bone. Registering these two data sets without correction leads to a systemic registration error directly related to cartilage thickness. Cartilage thickness varies across different areas of the joint surface, so this mismatch results in an inaccurate registration of the CT-based bone data with the surgical coordinate system.

[0057] If left uncorrected, this cartilage-bone mismatch compromises the accuracy of the navigation system. Surgical planning relies on precise alignment between pre-operative imaging and intraoperative anatomical data, and any errors in registration can propagate throughout the procedure. These inaccuracies may lead to suboptimal implant positioning, joint misalignment, or unintended joint mechanics, ultimately affecting patient outcomes.

[0058] To bridge the gap between the surface that the patient presents, which is cartilage, and bone surface segmented from CT scans, navigation systems during surgery often rely on the manual placement of optical trackers mounted on a pointed instrument, typically a sharp-tipped pointer to physically mark specific anatomical landmarks. The sharp pointer is pushed through the cartilage to the bone surface, enabling the navigation system to approximate bone surface positions for accurate registration with CT data. However, this process introduces multiple sources of variability and potential error, such as tool selection, penetration depth, and variability across surgeons and techniques. Regarding penetration depth, surgeons may vary in how deeply they penetrate the cartilage. Some may avoid full penetration, while others may exceed the necessary depth. These inconsistencies impact the accuracy of the registration data, leading to potential misalignment between the CT data and the intraoperative coordinate system.Regarding variability across surgeons and techniques, the manual nature of the process introduces inconsistencies based on individual surgical technique, skill, and interpretation. Such variability can accumulate across multiple anatomical landmarks, further compromising registration accuracy and potentially impacting surgical outcomes.

[0059] The use of sufficiently fast and accurate optical topological scanners coupled with sufficiently advanced algorithms, can eliminate the need for optical trackers and manual data acquisition steps, offering high-resolution, real-time surface capture without invasive tools.VIS-002-W01Introducing optical trackers and pointed instruments solely to obtain bone alignment undermines these advantages, adding time, complexity, and potential error back into the workflow. The reliance on manual penetration of cartilage becomes counterproductive to the streamlined precision these scanners are intended to provide.SUMMARY

[0060] This disclosure builds upon the innovations presented in U.S. Patent Application No. 18 / 505,973 (U.S. Pub. 2024 / 0102795) (“the ‘973 app”), filed Nov. 9, 2023 and entitled “GENERATION OF ONE OR MORE EDGES OF LUMINOSITY TO FORM THREE- DIMENSIONAL MODELS OF SCENES”. More specifically, the present disclosure introduces advanced clinical applications for the previously described 3D scanning system. The ‘973 app unveiled a groundbreaking 3D scanner whose capabilities were not previously available in surgical settings. The present disclosure expands on the utility of that technology, particularly emphasizing its application in orthopedic surgeries such as Total Knee Arthroplasty (TKA) and Robotic Assisted TKA (RA-TKA).

[0061] The enhancements detailed herein expand upon the exemplary clinical uses disclosed in the ‘973 app, with the goal of illustrating how the technology can be highly beneficial and tailored in certain arthroplasty procedures. For example, by leveraging advanced scanning methodologies, the system reduces the need for optical trackers, streamlining setup and improving surgical precision. While this disclosure can be applied broadly across multiple surgical domains, particular emphasis is given to its use in orthopedic procedures such as Total Knee Arthroplasty (TKA). The subsequent sections will outline the significant advancements this technology brings to the clinical environment, setting a new standard for precision and efficiency in surgical navigation and instrumentation. Moreover, although described herein in this context as a specific example, it should be understood that the system and methods disclosed herein may be used in any scenario (surgical, medical, or otherwise) to generate a reconstructed 3D model of an object, register it to a reference object in a different coordinate space, and track movement of the object in real space with respect to the reconstructed 3D model of the object within the coordinate space.VIS-002-W01

[0062] The present disclosure describes a 3D scanner specifically designed for surgical applications, capable of collecting visible topological data from the surgical field in real time. This represents a new class of data that was previously unavailable during surgery. By capturing continuous, high-rate 3D topological data during the procedure, this scanner can replace traditional workflows that rely on optical trackers — without the associated costs, setup time, and limitations. Although described within the context of surgical applications as a specific example, the system and methods disclosed herein may be used in a wide variety of non-surgical applications to register and track an object within a coordinate space.

[0063] This disclosure expands on how this topological data can be applied in a surgical setting to perform various procedures. Although the current disclosure emphasizes applications in Total Knee Arthroplasty (TKA), the underlying principles can be adapted for other surgeries with minimal modifications by those skilled in the art. While the 3D scanner described herein is the preferred method for collecting this data, the use of the scanner is not a requirement for this innovation, as other technology capable of capturing 3D surface data during surgery may be utilized to achieve similar results.

[0064] When a continuously updated 3D topological data of sufficient accuracy and rate of the surgical site is available, the data can be used for Continuous Anatomical Auto Tracking (CAAT). This innovation eliminates the need for introduced physical optical markers and reduces the complexity of setup, offering a more streamlined, cost-effective, and accurate approach to surgical navigation. CAAT allows for real-time registration and tracking of anatomical structures, instruments and equipment without using OMs or OTs or other fiducials, marking a significant advancement in surgical technology. CAAT promises greater precision, flexibility, and efficiency in the operating room, benefiting a wide range of surgical procedures.

[0065] Additionally, this disclosure addresses a critical challenge in CT image-based surgical navigation: the misalignment between pre-operative CT data, which accurately captures bone structures, and intraoperative topological scans, which predominantly capture the cartilage- covered bone surfaces exposed during surgery. Cartilage, with its variable thickness and low visibility in CT scans, introduces alignment errors when CT data is registered directly to topological scans, especially in procedures like Total Knee Arthroplasty (TKA).VIS-002-W01

[0066] By way of example, this disclosure introduces a novel approach to overcome cartilage- to-bone surface mismatch, providing an immediate solution that aligns the cartilage and bone surfaces without reintroducing optical trackers. Specifically, this disclosure addresses the challenges of cartilage and bone misalignment in surgical navigation, providing a method to accurately register CT data within the surgical coordinate system, enhancing alignment accuracy and improved outcomes in image-guided surgery. By accurately accounting for the cartilage thickness in the registration process, this method enables the navigation system to achieve reliable alignment with the bone structure, maintaining the high-precision, non-invasive advantages of topological scanning.

[0067] This disclosure provides a method to correct for cartilage-bone misalignment by segmenting and registering bone-only data. Initially, optical intraoperative scans are segmented to isolate captured surface bone separate from the cartilage, creating datasets that align with the bone-only data in pre-operative CT scans. One may also use regions of known hard tissue (particularly thin cartilage), such as the edges of the condyles, and the sulcus between the condyles. Using a combination of coarse alignment (e.g., RANSAC) and fine alignment (e.g., ICP), a transformation matrix is generated to accurately co-locate the CT dataset with the optical data set in a shared coordinate space. Once established, this matrix allows the entire set of related intraoperative data — both bone and cartilage — to be precisely aligned in the common space without further manual adjustments.

[0068] By way of example, the method disclosed herein sets the foundation for achieving accurate surgical navigation in the presence of cartilage-bone mismatch, while maintaining the streamlined, automated benefits of modern topological scanners. This single method provides a practical and effective solution, improving the quality and reliability of image-based surgical navigation.

[0069] As additional description to the embodiments described below, the present disclosure describes the following embodiments.

[0070] Embodiment 1 is a system for continuous registration and tracking of objects in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source; a liquid crystal matrix; and an image capture device; a computer systemVIS-002-W01 including a memory and at least one processor; and computer readable media embodied in a non- transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation of a topological surface of a physical object using a first dataset, the first dataset derived from one or more images captured by the image capture device using infrared light or near infrared light without optical trackers or applied fiducials; register the three-dimensional data representation to a second dataset such that the three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; and track the physical object relative to the common coordinate system without said optical trackers or applied fiducials.

[0071] Embodiment 2 is the system of embodiment 1, wherein the light source is a linear light source.

[0072] Embodiment 3 is the system of embodiments 1 or 2, wherein the three-dimensional data representation is a point cloud.

[0073] Embodiment 4 is the system of any of embodiments 1 through 3, wherein the captured images are structured light patterned images that encode the surface of the physical object.

[0074] Embodiment 5 is the system of any of embodiments 1 through 4, wherein the three- dimensional structured light scanner further includes a heat reduction assembly.

[0075] Embodiment 6 is the system of any of embodiments 1 through 5, wherein the physical object is an anatomical structure in a surgical procedure.

[0076] Embodiment 7 is the system of any of embodiments 1 through 6, wherein the three- dimensional data representation of the topological surface of the physical object is constructed by: decoding light patterns contained in the one or more captured images by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images based on the decoded light patterns; and combining the triangulated three-dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the physical object in physical space.VIS-002-W01

[0077] Embodiment 8 is the system of any of embodiments 1 through 7, wherein the constructed three-dimensional data representation of the surface topology is comprised of at least 200,000 coordinates.

[0078] Embodiment 9 is the system of any of embodiments 1 through 8, wherein the one or more images are captured at a rate of at least 20 Hertz.

[0079] Embodiment 10 is the system of any of embodiments 1 through 9, wherein the digital three-dimensional data representation of the physical object is constructed within a processor located on the structured light three-dimensional scanner.

[0080] Embodiment 11 is the system of any of embodiments 1 through 10, wherein the three- dimensional data representation of the physical object comprises a first three-dimensional data representation of a first physical object, and the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: construct a second digital three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of surface topology of the second physical object captured by the image capture device using infrared light and without the use of optical trackers or applied fiducials; register the second three-dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned with one another and the with the first three-dimensional data representation and second dataset within the common coordinate system without the use of optical trackers or applied fiducials; continuously track the second physical object relative to the common coordinate system without the use of optical trackers or applied fiducials.

[0081] Embodiment 12 is the system of any of embodiments 1 through 11, wherein the first physical object is a first anatomical structure and the second physical object is a second anatomical structure.

[0082] Embodiment 13 is the system of any of embodiments 1 through 12, wherein the first physical object is an anatomical structure and the second physical object is a surgical instrument.VIS-002-W01

[0083] Embodiment 14 is the system of any of embodiments 1 through 13, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

[0084] Embodiment 15 is the system of any of embodiments 1 through 14, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked three-dimensional data representation of the physical object.

[0085] Embodiment 16 is the system of any of embodiments 1 through 15, wherein tracking the physical object relative to the common coordinate system comprises continuously updating the physical object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

[0086] Embodiment 17 is a system for continuous registration and tracking of objects in in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source; a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non- transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a first three-dimensional data representation of a topological surface of a first physical object using a first dataset, the first dataset derived from one or more images of the topological surface of the first physical object captured by the image capture device using infrared light or near infrared light without optical trackers or applied fiducials; register the first three-dimensional data representation to a second dataset such that the first three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; track the first physical object relative to the common coordinate system without said optical trackers or applied fiducials; construct a second three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of the topological surface of the second physicalVIS-002-W01 object captured by the image capture device using infrared light without optical trackers or applied fiducials; register the second three-dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned within the common coordinate system, the registration occurring without said optical trackers or applied fiducials; and track the second physical object relative to the common coordinate system without said optical trackers or applied fiducials.

[0087] Embodiment 18 is the system of embodiment 17, wherein the light source is a linear light source.

[0088] Embodiment 19 is the system of embodiments 17 or 18, wherein the first three- dimensional data representation is a point cloud.

[0089] Embodiment 20 is the system of any of embodiments 1 through 19, wherein the captured images of the topological surface of the first physical object are structured light patterned images that encode the surface of the first physical object.

[0090] Embodiment 21is the system of any of embodiments 17 through 20, wherein the three- dimensional structured light scanner further includes a heat reduction assembly.

[0091] Embodiment 22 is the system of any of embodiments 17 through 21, wherein the first physical object is an anatomical structure in a surgical procedure.

[0092] Embodiment 23 is the system of any of embodiments 17 through 22, wherein the second physical object is a surgical instrument.

[0093] Embodiment 24 is the system of any of embodiments 17 through 23, wherein the first three-dimensional data representation of the topological surface of the first physical object is constructed by: decoding light patterns contained in the one or more captured images of the topological surface of the first physical object by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three- dimensional coordinate for each pixel in the one or more captured images of the topological surface of the first physical object based on the decoded light patterns; and combining theVIS-002-W01 triangulated three-dimensional coordinates of each pixel into the first three-dimensional point cloud representation of the surface of the first physical object in physical space.

[0094] Embodiment 25 is the system of any of embodiments 17 through 24, wherein the first three-dimensional data representation of the surface topology is comprised of at least 200,000 coordinates.

[0095] Embodiment 26 is the system of any of embodiments 17 through 25, wherein the one or more images of the topological surface of the first physical object and the one or more images of the topological surface of the second physical object are captured at a rate of at least 20 Hertz.

[0096] Embodiment 27 is the system of any of embodiments 17 through 26, wherein the first three-dimensional data representation of the first physical object is constructed within a processor located on the structured light three-dimensional scanner.

[0097] Embodiment 28 is the system of any of embodiments 17 through 27, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

[0098] Embodiment 29 is the system of any of embodiments 17 through 28, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked first three-dimensional data representation of the first physical object and the tracked second three-dimensional data representation of the second physical object.

[0099] Embodiment 30 is the system of any of embodiments 17 through 29, wherein tracking the first physical object relative to the common coordinate system comprises continuously updating the first physical object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

[0100] Embodiment 31 is a system for continuous registration and tracking of objects in in an operating room environment, comprising: a structured light three-dimensional scanner,VIS-002-W01 including: a light source; a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non- transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation of a topological surface of a physical object using a first dataset, the first dataset derived from one or more images captured by the image capture device without optical trackers or applied fiducials; register the three-dimensional data representation to a second dataset such that the three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; and track the physical object relative to the common coordinate system without said optical trackers or applied fiducials; wherein the digital three-dimensional data representation of the physical object is constructed within a processor located on the structured light three- dimensional scanner.

[0101] Embodiment 32 is system of embodiment 31, wherein the light source is a linear light source.

[0102] Embodiment 33 is the system of embodiments 31 or 32, wherein the three-dimensional data representation is a point cloud.

[0103] Embodiment 34 is the system of any of embodiments 31 through 33, wherein the captured images are structured light patterned images that encode the surface of the physical object.

[0104] Embodiment 35 is the system of any of embodiments 31 through 34, wherein the three-dimensional structured light scanner further includes a heat reduction assembly.

[0105] Embodiment 36 is the system of any of embodiments 31 through 35, wherein the physical object is an anatomical structure in a surgical procedure.

[0106] Embodiment 37 is the system of any of embodiments 31 through 36, wherein the three-dimensional data representation of the topological surface of the physical object is constructed by: decoding light patterns contained in the one or more captured images by analyzing brightness history to create camera pixel correspondence to light angle projection;VIS-002-W01 using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images based on the decoded light patterns; and combining the triangulated three- dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the physical object in physical space.

[0107] Embodiment 38 is the system of any of embodiments 31 through 37, wherein constructed three-dimensional data representation of the surface topology is comprised of at least 200,000 coordinates.

[0108] Embodiment 39 is the system of any of embodiments 31 through 38, wherein the one or more images are captured at a rate of at least 20 Hertz.

[0109] Embodiment 40 is the system of any of embodiments 31 through 39, wherein the one or more images are captured by the image capture device using infrared light or near infrared light.

[0110] Embodiment 41 is the system of any of embodiments 31 through 40, wherein the three-dimensional data representation of the physical object comprises a first three-dimensional data representation of a first physical object, and the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: construct a second digital three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of surface topology of the second physical object captured by the image capture device without the use of optical trackers or applied fiducials; register the second three-dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned with one another and the with the first three-dimensional data representation and second dataset within the common coordinate system without the use of optical trackers or applied fiducials; and continuously track the second physical object relative to the common coordinate system without the use of optical trackers or applied fiducials; wherein the second digital three-dimensional data representation of the second physical object is constructed within a processor located on the structured light three-dimensional scanner.VIS-002-W01

[0111] Embodiment 42 is the system of any of embodiments 31 through 41 , wherein the first physical object is a first anatomical structure and the second physical object is a second anatomical structure.

[0112] Embodiment 43 is the system of any of embodiments 31 through 42, wherein the first physical object is an anatomical structure and the second physical object is a surgical instrument.

[0113] Embodiment 44 is the system of any of embodiments 31 through 43, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

[0114] Embodiment 45 is the system of any of embodiments 31 through 44, wherein the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked three-dimensional data representation of the physical object.

[0115] Embodiment 46 is the system of any of embodiments 31 through 45, wherein tracking the physical object relative to the common coordinate system comprises continuously updating the physical object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

[0116] Embodiment 47 is a system for continuous registration and tracking of objects in in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source; a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non- transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a first three-dimensional data representation of a topological surface of a first physical object using a first dataset, the first dataset derived from one or more images of the topological surface of the first physical object captured by the image capture device using infrared light or near infrared light; register the first three-dimensional data representation to a second dataset such that the first three-dimensionalVIS-002-W01 data representation and the second dataset are aligned within a common coordinate system; track the first physical object relative to the common coordinate system; construct a second three- dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of the topological surface of the second physical object captured by the image capture device using infrared light; register the second three-dimensional data representation to a fourth dataset such that the second three- dimensional data representation and the fourth dataset are aligned within the common coordinate system; and track the second physical object relative to the common coordinate system; wherein the first digital three-dimensional point cloud representation of the first physical object and the second digital three-dimensional point cloud representation of the second physical object are constructed within a processor located on the 3D scanner.

[0117] Embodiment 48 is the system of embodiment 47, wherein the light source is a linear light source.

[0118] Embodiment 49 is the system of embodiments 47 or 48, wherein the first three- dimensional data representation is a point cloud.

[0119] Embodiment 50 is the system of any of embodiments 47 through 49, wherein the captured images of the topological surface of the first physical object are structured light patterned images that encode the surface of the first physical object.

[0120] Embodiment 51 is the system of any of embodiments 47 through 50, wherein the three-dimensional structured light scanner further includes a heat reduction assembly.

[0121] Embodiment 52 is the system of any of embodiments 47 through 51, wherein the first physical object is an anatomical structure in a surgical procedure.

[0122] Embodiment 53 is the system of any of embodiments 47 through 52, wherein the second physical object is a surgical instrument.

[0123] Embodiment 54 is the system of any of embodiments 47 through 53, wherein the first three-dimensional data representation of the topological surface of the first physical object is constructed by: decoding light patterns contained in the one or more captured images of theVIS-002-W01 topological surface of the first physical object by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three- dimensional coordinate for each pixel in the one or more captured images of the topological surface of the first physical object based on the decoded light patterns; and combining the triangulated three-dimensional coordinates of each pixel into the first three-dimensional point cloud representation of the surface of the first physical object in physical space.

[0124] Embodiment 55 is the system of any of embodiments 47 through 54, wherein the first three-dimensional data representation of the surface topology is comprised of at least 200,000 coordinates.

[0125] Embodiment 56 is the system of any of embodiments 47 through 55, wherein the one or more images of the topological surface of the first physical object and the one or more images of the topological surface of the second physical object are captured at a rate of at least 20 Hertz.

[0126] Embodiment 57 is the system of any of embodiments 47 through 56, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

[0127] Embodiment 58 is the system of any of embodiments 47 through 57, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked first three-dimensional data representation of the first physical object and the tracked second three-dimensional data representation of the second physical object.

[0128] Embodiment 59 is the system of any of embodiments 47 through 58, wherein tracking the first physical object relative to the common coordinate system comprises continuously updating the first physical object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.VIS-002-W01

[0129] Embodiment 60 is the system of any of embodiments 47 through 59, wherein the one or more images of the topological surface of the first physical object are captured by the image capture device without the use of optical trackers or fiducials.

[0130] Embodiment 61 is the system of any of embodiments 47 through 60, wherein one or more images of the topological surface of the second physical object are captured by the image capture device without the use of optical trackers or fiducials.

[0131] Embodiment 62 is the system of any of embodiments 47 through 61, wherein the first generated three-dimensional point cloud representation is registered to the second dataset within the common coordinate system without the use of optical trackers or fiducials.

[0132] Embodiment 63 is the system of any of embodiments 47 through 62, wherein the first physical object is tracked within the common coordinate system without the use of optical trackers or fiducials.

[0133] Embodiment 64 is the system of any of embodiments 47 through 63, wherein the second generated three-dimensional point cloud representation is registered to the fourth dataset within the common coordinate system without the use of optical trackers or fiducials.

[0134] Embodiment 65 is the system of any of embodiments 47 through 64, wherein the second physical object is tracked within the common coordinate system without the use of optical trackers or fiducials.

[0135] Embodiment 66 is a system for continuous registration and tracking of objects in in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source; a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non- transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a first three-dimensional data representation of a topological surface of a first physical object using a first dataset, the first dataset derived from one or more images of the topological surface of the first physical object captured by the image capture device using infrared light or near infrared light without optical trackers or applied fiducials; register the first three-dimensional data representation to a secondVIS-002-W01 dataset such that the first three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; track the first physical object relative to the common coordinate system without said optical trackers or applied fiducials; construct a second three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of the topological surface of the second physical object captured by the image capture device using infrared light; register the second three- dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned within the common coordinate system; and track the second physical object relative to the common coordinate system; wherein the first physical object is an anatomical structure; and wherein the second physical object is a surgical instrument.

[0136] Embodiment 67 is the system of embodiment 66, wherein the light source is a linear light source.

[0137] Embodiment 68 is the system of embodiments 66 or 67, wherein the first three- dimensional data representation is a point cloud.

[0138] Embodiment 69 is the system of any of embodiments 66 through 68, wherein the captured images of the topological surface of the first physical object are structured light patterned images that encode the surface of the first physical object.

[0139] Embodiment 70 is the system of any of embodiments 66 through 69, wherein the three-dimensional structured light scanner further includes a heat reduction assembly.

[0140] Embodiment 71 is the system of any of embodiments 66 through 70, wherein the first three-dimensional data representation of the topological surface of the first physical object is constructed by: decoding light patterns contained in the one or more captured images of the topological surface of the first physical object by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three- dimensional coordinate for each pixel in the one or more captured images of the topological surface of the first physical object based on the decoded light patterns; and combining theVIS-002-W01 triangulated three-dimensional coordinates of each pixel into the first three-dimensional point cloud representation of the surface of the first physical object in physical space.

[0141] Embodiment 72 is the system of any of embodiments 66 through 71, wherein the first three-dimensional data representation of the surface topology is comprised of at least 200,000 coordinates.

[0142] Embodiment 73 is the system of any of embodiments 66 through 72, wherein the one or more images of the topological surface of the first physical object and the one or more images of the topological surface of the second physical object are captured at a rate of at least 20 Hertz.

[0143] Embodiment 74 is the system of any of embodiments 66 through 73, wherein the first three-dimensional data representation of the first physical object is constructed within a processor located on the structured light three-dimensional scanner.

[0144] Embodiment 75 is the system of any of embodiments 66 through 74, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

[0145] Embodiment 76 is the system of any of embodiments 66 through 75, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked first three-dimensional data representation of the first physical object and the tracked second three-dimensional data representation of the second physical object.

[0146] Embodiment 77 is the system of any of embodiments 66 through 76, wherein tracking the first physical object relative to the common coordinate system comprises continuously updating the first physical object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

[0147] Embodiment 78 is a method for continuous registration and tracking of objects in in an operating room environment, comprising: constructing a three-dimensional data representation ofVIS-002-W01 a topological surface of a physical object using a first dataset, the first dataset derived from one or more images captured by the image capture device using infrared light or near infrared light without optical trackers or applied fiducials; registering the three-dimensional data representation to a second dataset such that the three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; and tracking the physical object relative to the common coordinate system without said optical trackers or applied fiducials.

[0148] Embodiment 79 is the method of embodiment 78, wherein the light source is a linear light source.

[0149] Embodiment 80 is the method of embodiments 78 or 79, wherein the three- dimensional data representation is a point cloud.

[0150] Embodiment 81 is the method of any of embodiments 78 through 80, wherein the captured images are structured light patterned images that encode the surface of the physical object.

[0151] Embodiment 82 is the method of any of embodiments 78 through 81, wherein the three-dimensional structured light scanner further includes a heat reduction assembly.

[0152] Embodiment 83 is the method of any of embodiments 78 through 82, wherein the physical object is an anatomical structure in a surgical procedure.

[0153] Embodiment 84 is the method of any of embodiments 78 through 83, wherein constructing the three-dimensional data representation of the topological surface of the physical object comprises: decoding light patterns contained in the one or more captured images by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images based on the decoded light patterns; and combining the triangulated three- dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the physical object in physical space.VIS-002-W01

[0154] Embodiment 85 is the method of any of embodiments 78 through 84, wherein the constructed three-dimensional data representation of the surface topology is comprised of at least 200,000 coordinates.

[0155] Embodiment 86 is the method of any of embodiments 78 through 85, wherein the one or more images are captured at a rate of at least 20 Hertz.

[0156] Embodiment 87 is the method of any of embodiments 78 through 86, wherein the digital three-dimensional data representation of the physical object is constructed within a processor located on the structured light three-dimensional scanner.

[0157] Embodiment 88 is the method of any of embodiments 78 through 87, wherein the three-dimensional data representation of the physical object comprises a first three-dimensional data representation of a first physical object, the method further comprising: constructing a second digital three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of surface topology of the second physical object captured by the image capture device using infrared light and without the use of optical trackers or applied fiducials; registering the second three- dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned with one another and the with the first three- dimensional data representation and second dataset within the common coordinate system without the use of optical trackers or applied fiducials; and tracking the second physical object relative to the common coordinate system without the use of optical trackers or applied fiducials.

[0158] Embodiment 89 is the method of any of embodiments 78 through 88, wherein the first physical object is a first anatomical structure and the second physical object is a second anatomical structure.

[0159] Embodiment 90 is the method of any of embodiments 78 through 89, wherein the first physical object is an anatomical structure and the second physical object is a surgical instrument.

[0160] Embodiment 91 is the method of any of embodiments 78 through 90, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.VIS-002-W01

[0161] Embodiment 92 is the method of any of embodiments 78 through 91, further comprising: displaying, on a display device, the tracked three-dimensional data representation of the physical object.

[0162] Embodiment 93 is the method of any of embodiments 78 through 92, wherein tracking the physical object relative to the common coordinate system comprises continuously updating the physical object’s position relative to the common coordinate system by: repeatedly acquiring surface data; registering the repeatedly acquired surface data in the common coordinate system; and updating transformation matrices.

[0163] Embodiment 94 is a system for continuous registration and tracking of objects in in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source: a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non- transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a unique three-dimensional data representation of a topological surface of each of a plurality of physical objects using a first dataset, the first dataset derived from one or more images captured by the image capture device using infrared light or near infrared light without optical trackers or applied fiducials; register each unique three-dimensional data representation to a corresponding unique reference dataset for each one of said plurality of physical objects such that each unique three-dimensional data representation is aligned with the corresponding unique reference dataset within a common coordinate system, each registration occurring without said optical trackers or applied fiducials; and track each of said plurality of physical objects relative to the common coordinate system without said optical trackers or applied fiducials.BRIEF DESCRIPTION OF THE DRAWINGS

[0164] Many advantages of the present disclosure will be apparent to those skilled in the art with a reading of this specification in conjunction with the attached drawings, wherein like reference numerals are applied to like elements and wherein:VIS-002-W01

[0165] Fig. 1 illustrates an example of a prior art rigid transformation matrix for transferring a 3D data set from one coordinate system to another;

[0166] Fig. 2 is a flowchart depicting an example of a typical prior art workflow for a robotically assisted total knee arthroplasty;

[0167] Fig. 3 is a block diagram illustrating essential features of a continuous anatomical automatic tracking (CAAT) system of the present disclosure, according to some embodiments;

[0168] Fig. 4 is a perspective view of an example of a surgical environment utilizing the CAAT system of Fig. 3, according to some embodiments;

[0169] Fig. 5 is a perspective view of an example of a 3D scanner forming part of the CAAT system of the Fig. 3, according to some embodiments;

[0170] Fig. 6 is a perspective view of the 3D scanner of Fig. 5 with the cover removed, according to some e embodiments;

[0171] Fig. 7 is a plan view of the 3D scanner of Fig. 6, according to some embodiments;

[0172] Fig. 8 is another perspective view of the 3D scanner of Fig. 6, according to some embodiments;

[0173] Fig. 9 is a perspective view of an example of a rigid scaffolding forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0174] Figs. 10-12 are front perspective, rear perspective, and exploded views, respectively, of an example of an optical system forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0175] Figs. 13-15 are perspective views of an example of a light source assembly forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0176] Fig. 16 is an exploded view of the light source assembly of Fig. 13, according to some embodiments;VIS-002-W01

[0177] Figs. 17-19 are perspective, bottom plan, and side plan views, respectively, of an example of a linear light source forming part of the light source assembly of Fig. 13, according to some embodiments;

[0178] Figs. 20-21 are perspective and plan views, respectively, of an example of a light shield forming part of the light source assembly of Fig. 13, according to some embodiments;

[0179] Figs. 22-23 are bottom perspective and top perspective views, respectively, of an example of a light box forming part of the light source assembly of Fig. 13, according to some embodiments;

[0180] Figs. 24-25 are perspective and top plan views, respectively of an example of a top portion of the light box of Fig. 22, according to some embodiments;

[0181] Figs. 26-28 are exploded, side plan, and bottom plan views, respectively, of the top portion of the light box of Fig. 24 assembled with the linear light source of Fig. 18 and light shield of Fig. 20, according to some embodiments;

[0182] Figs. 29-30 are top perspective and bottom perspective views, respectively, of an example of a bottom portion of the light box of Fig. 22, according to some embodiments;

[0183] Figs. 31-33 are perspective, bottom plan, and side plan views, respectively, of an example of a pattern generator forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0184] Figs. 34-35 are top perspective and bottom perspective views, respectively, of an example of a pattern generator cover forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0185] Fig. 36 is a perspective view of an example of a thermal management system forming part of the 3D scanner of Fig. 5, according to some embodiments;

[0186] Fig. 37 is a perspective view of the thermal management system of Fig. 36 shown with the thermally insulating spacers removed, according to some embodiments;VIS-002-W01

[0187] Fig. 38 is a partial exploded view of the thermal management system of Fig. 36, according to some embodiments;

[0188] Fig. 39 is a perspective view of an example of a heat transfer assembly forming part of the TMS of Fig. 36, according to some embodiments;

[0189] Fig. 40 is a partially exploded view of the heat transfer assembly of Fig. 39, according to some embodiments;

[0190] Figs. 41-42 are perspective and exploded views, respectively, of an example of a first heat transfer portion forming part of the heat transfer assembly of Fig. 39, according to some embodiments;

[0191] Figs. 43-44 are perspective and exploded views, respectively, of an example of a second heat transfer portion forming part of the heat transfer assembly of Fig. 39, according to some embodiments;

[0192] Fig. 45-46 are perspective and side plan views, respectively, of an example of a heat sink forming part of the thermal management system of Fig. 36, according to some embodiments;

[0193] Fig. 47 is a perspective view of an example of an air box forming part of the thermal management system of Fig. 36, according to some embodiments;

[0194] Fig. 48 is a perspective view of an example of a vent box forming part of the thermal management system of Fig. 36, according to some embodiments;

[0195] Fig. 49 is an example of an optical system of Fig. 10 with a field of view and linear light source of Fig. 17 with a light field extending directly through a pattern generator of Fig. 31, according to some embodiments;

[0196] Fig. 50 is an example of an optical system of Fig. 10 with a field of view and linear light source of Fig. 17 with a light field extending indirectly through a pattern generator of Fig. 31, according to some embodiments;VIS-002-W01

[0197] Figs. 51 -52 is an example of a linear light source of Fig. 17 reflecting light from side mirrors, according to some embodiments;

[0198] Fig. 53 is an example of a linear light source of Fig. 17 reflecting light from multiple side mirrors, according to some embodiments;

[0199] Fig. 54 is an example of the linear light source of Fig. 51 reflecting light from mirrors, illustrating the angular advantage of using mirrors, according to some embodiments; and

[0200] Fig. 55 is an example of a linear light source of Fig. 17 reflecting light from multiple side mirrors multiple times, according to some embodiments.

[0201] Fig. 56 is a block diagram depicting an example of a data acquisition process using the CAAT system of Fig. 3, according to some embodiments;

[0202] Fig. 57 is an example of reconstructed point cloud of an exposed knee, simulated with sawbones and using a visible light source generated using the CAAT system of Fig. 3, according to some embodiments;

[0203] Fig. 58 depicts a typical prior art stereoscopic camera field of view (on the left) and what the cameras actually capture (on the right), according to some embodiments;

[0204] Fig. 59 depicts an image and 3D point cloud from a 3D scanner forming part of the CAAT system of Fig. 3, using infrared light instead of visible light, according to some embodiments;

[0205] Fig. 60 depicts a 3D point cloud from a 3D scanner forming part of the CAAT system of Fig. 3, using infrared light instead of visible light and shown at an oblique angle, which illustrates in particular the 3D nature of the regeneration, according to some embodiments;

[0206] Fig. 61 depicts the 3D point cloud of Fig. 60, with some anatomical structures segmented for easy identification, according to some embodiments;VIS-002-W01

[0207] Fig. 62 is a block diagram depicting an example of a method of registration of anatomical structures and / or surgical instruments using the CAAT system of Fig. 3, according to some embodiments;

[0208] Fig. 63 illustrates three data sets registered into the same coordinate space using the CAAT system of Fig. 3, particularly CT data from the femur and tibia registered to a 3D topological scan, according to some embodiments;

[0209] Fig. 64 is a block diagram depicting an example of a method of calculating a hip center of rotation using the CAAT system of Fig. 3, according to some embodiments;

[0210] Fig. 65 is a block diagram depicting an example of a method of assessing gaps between femoral condyles (e.g., medial and lateral) and a tibial plateau using the CAAT system of Fig. 3, according to some embodiments;

[0211] Fig. 66 is a block diagram depicting an example of a method of tracking movable anatomy and a movable surgical instrument configured to affect the movable anatomy using the CAAT system of Fig. 3, according to some embodiments;

[0212] Fig. 67A-B is a block diagram depicting an example of a method of complex anatomy tracking with robot-controlled instrument in the context of a Total Knee Arthroplasty (TKA) using the CAAT system of Fig. 3, according to some embodiments;

[0213] Fig. 68 is a block diagram depicting an example robotically assisted TKA workflow with pre-operative imaging using the CAAT system of Fig. 3, according to some embodiments;

[0214] Fig. 69 is a block diagram depicting an example robotically assisted TKA workflow without pre-operative imaging using the CAAT system of Fig. 3, according to some embodiments;

[0215] Figs. 70-72 are block diagrams depicting various steps of an example of a method of addressing the registration mismatch between pre-operative CT data and intraoperative topological scans in image-based surgical navigation, particularly in procedures like TKA (including robot-assisted TKA), according to some embodiments.VIS-002-W01

[0216] Figs. 73 and 74 are block diagrams illustrating example computer systems with which any of the devices or systems described herein may be implemented, according to some embodimentsDETAILED DESCRIPTION OF A PREFERRED EMBODIMENT

[0217] Illustrative embodiments of the disclosure are described below. In the interest of clarity, not all features of an actual implementation are described in this specification. It will of course be appreciated that in the development of any such actual embodiment, numerous implementation-specific decisions must be made to achieve the developers’ specific goals, such as compliance with system-related and business-related constraints, which will vary from one implementation to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming but would nevertheless be a routine undertaking for those of ordinary skill in the art having the benefit of this disclosure. The system and method for registration and tracking by 3D scanning the topology of anatomic structures disclosed herein boasts a variety of inventive features and components that warrant patent protection, both individually and in combination.

[0218] By way of example, this disclosure introduces a novel system and method for registering and tracking anatomical structures during surgical procedures by leveraging continuously updated 3D topological data of the surgical field. Unlike traditional systems that rely on physical optical markers (OMs) and Optical Tracker Constellations (OTCs), the system and method disclosed herein utilizes the data captured by a 3D scanner to perform Continuous Anatomic Auto Tracking (CAAT). Hereinafter, the system disclosed herein may be referred to as “CAAT system 10”. While shown and described herein within the specific context of a surgical application (specifically a TKA procedure), the CAAT system 10 including the 3D scanner 12 may be used in any scenario (surgical, medical, or otherwise) to generate a reconstructed 3D model of an object.

[0219] Referring to Fig. 3, and by way of example, the CAAT system 10 disclosed herein may feature a 3D scanner 12, hub 14, user interface 16, and distributed computation 18. By way of example, the CAAT system 10 also includes a software program 20 embodied in a non-VIS-002-W01 transitory computer readable storage medium (e.g., the hub 14) and comprising a set of instructions configured to enable the computer to perform the various tasks described herein. In some embodiments, one or more peripheral computing devices 21 may be associated with the CAAT system 10, for example to integrate use and control of certain medical and surgical equipment that may be used in robotics-driven procedures. Such peripheral computing device(s) 21 may be configured to be controlled by the hub 14 and / or user interface 16.

[0220] In some embodiments, the 3D scanner 12 continuously captures topological data of one or more objects within a field of view at a high repetition rate, providing detailed surface information of anything within the field of view (e.g., anatomical structures, surgical equipment, surgical tools, surgical staff, etc.,) and ensuring uninterrupted tracking during dynamic surgical procedures. While the 3D scanner 12 disclosed herein is optimized for this application, the CAAT system 10 is compatible with any 3D scanner capable of optically generating topological information from the surgical site. In some embodiments, the 3D scanner 12 may be enclosed in a sterile drape to maintain sterility, allowing the 3D scanner 12 to function within the sterile surgical environment.

[0221] In some embodiments, the hub 14 is a dedicated computer or computer system responsible for receiving raw image data from the 3D scanner 12. Equipped with specialized hardware and software, the hub 14 processes this data in real-time, converting the data into a usable format for tracking anatomical structures and surgical instruments. In some embodiments, the CAAT system 10 performs automatic registration of the data to a Surgical Coordinate System (SCS), eliminating the need for manual optical trackers or registration steps. In some embodiments, the hub 14 may comprise multiple computer systems that communicate with one another so that data can be processed serially or in parallel on separate hardware.

[0222] In some embodiments, the hub 14 transmits transformation matrices for each tracked object to a separate or second computer system, where the data is further processed based on the surgical workflow. This information is presented to the surgeon in an intuitive, user-friendly interface 16, providing real-time visualization of anatomy and surgical tools. In some embodiments, the interface guides the surgeon through key actions such as implant positioning, bone cuts, and other critical tasks.VIS-002-W01

[0223] In some embodiments, while the hub 14 plays a central role in data processing, the CAAT system 10 architecture allows for flexibility in where computations occur. For example, in some embodiments, some processing tasks, such as initial data filtering or compression and / or reconstruction of a 3D point cloud, may be performed by a processor within the 3D scanner 12 itself to reduce data transmission loads. Additionally, certain computations may be handled by other computers within the system, either in series or parallel with the hub 14, depending on the complexity of the procedure and the system's overall configuration. In some cases, specific computational tasks, such as visualization rendering or more advanced data analysis, may be managed directly within the user interface (UI) 16 system. This distributed computation 20 approach enables scalable performance and ensures that real-time requirements are met across a range of surgical applications.

[0224] 3D Scanner

[0225] Fig. 4 illustrates an example of a surgical environment 22 which may be a sterile field within an operating room at a hospital or surgical center, in which a patient 24 is positioned on an operating table 26 for a surgical procedure such as a Total Knee Arthroplasty (TKA). By way of example, a surgical instrument 28 (e.g., drill, oscillating saw, etc.) may be provided at the end of a first robotic arm 30, which itself is attached to a first cart 32. A 3D scanner 12 forming part of the CAAT system 10 is provided at the distal end (or “end effector”) of a second robotic arm 34 which is provided on a second cart 36. It should be noted that this surgical environment 22 is provided by way of example only to illustrate one possible setup and positioning of the 3D camera in relation to a patient during a surgical procedure. In some embodiments, the second robotic arm 34 to which the 3D scanner 12 is attached may be attached to a wall, ceiling, immobile table, or other structure.

[0226] Figs. 5-8 illustrate an example of a 3D scanner 12 forming part of the CAAT system 10 of the present disclosure, according to some embodiments. In some embodiments, the 3D scanner 12 comprises an optical system 42, a light source assembly 44, a thermal management system (TMS) 46, a cover 48, and a rigid scaffolding 50. In some embodiments, the optical system 42 includes a camera, lens, and filter, and is configured for continuously capturing data including high repetition images of patterns projected onto the surgical field (from whichVIS-002-W01 topological data is derived and used in reconstructing a 3D point cloud) and communicating the captured data to the hub 14. In some embodiments, the light source assembly 44 is configured to project a structured light pattern onto the target surface, which the CAAT system 10 may then analyze as described below to generate a 3D model of the surface area with the optical system’s field of view, as described herein. In some embodiments, the TMS 46 is configured to remove heat from the 3D scanner 12. More specifically the TMS 46 removes heat generated by the light source and the optical system 42 by using airflow to transfer the heat through a heat sink and ultimately out of the 3D scanner 12 through ventilation tubes 178, as described in detail below. The TMS 46 shown and described herein is representative of one example of removing heat from the 3D scanner 12. However, the 3D scanner 12 may be modified to incorporate other methods of removing heat in addition to or instead of the TMS 46 described herein.

[0227] In some embodiments, the optical system 42 has a field of view 68 with a central axis Li, and the light source assembly 44 has an illumination field 80 with a central axis L2. In some embodiments, the optical system 42 and light source assembly 44 may be positioned at an angle 0i relative to one another defined as the offset angle between the central axis Li of the optical system 42 and the central axis L2 of the light source assembly 44, as shown by way of example in Fig. 7. This arrangement produces an overlap area (or “working volume”) 69 in which the field of view 68 overlaps with the illumination field 80. Objects within the overlap area 69 may be scanned by the 3D scanner 12. Objects that are not in the overlap area 69 will not be scanned by the 3D scanner 12. In some embodiments, the angle 0i may also be defined as the apex angle in a triangulation between the optical system 42, the light source assembly 44, and the target site. By way of example, the apex location in the triangulation represents the optimal point within the overlap area 69 in which the optical system 42 produces the best imaging results. In some embodiments, the optical system 42 and light source assembly 44 may be positioned such that the angle 9i may be within a range of 10° to 30°. In some embodiments, the optical system 42 and light source assembly 44 may be positioned such that the angle 0i may be within a range of 10° to 20°. In some embodiments, the optical system 42 and light source assembly 44 may be positioned such that the angle 0i = 16.5°. In some embodiments, the 3D scanner 12 may be configured such that the angle 0i may be adjustable during use, for example to adjust the working volume 69 of the 3D scanner 12 or if the positioning of the 3D scanner 12 relative to theVIS-002-W01 target site needs to be moved. In some embodiments, the 3D scanner 12 may be configured such that the focus is operable by a motor. In some embodiments, the 3D scanner 12 may be configured such that the focus is automatically adjustable. In some embodiments, the 3D scanner 12 may be configured such that the working volume 69 may be adjusted by changing the lateral distance between the optical system 42 and the light source assembly 44. In some embodiments, the 3D scanner 12 may be configured such that the focus may be adjusted by changing angle 0i between the optical system 42 and the light source assembly 44. In some embodiments, the 3D scanner 12 may be configured to include an adjustable and / or motorized iris to enable a user to fine tune the level of light entering the camera.

[0228] Fig. 9 illustrates an example of a rigid scaffolding 50 forming part of the CAAT system 10 of the present disclosure, according to some embodiments. By way of example, the scaffolding 10 is configured to securely hold the various components of the CAAT system 10 in place. In some embodiments, the scaffolding 50 comprises a main frame body 52, one or more torsion plates 54 configured to secure the TMS 46 in place, a camera frame 56 configured to securely hold the optical system 42, and one or more mounting brackets 58 configured for attachment to the light box assembly 44. The scaffolding 50 is shown by way of example only, as other configurations are possible to accomplish the goal of rigidly securing the various components of the 3D scanner 12 in place.

[0229] Figs. 10-12 illustrate an example of an optical system 42 forming part of the CAAT system 10 of the present disclosure, according to some embodiments. In some embodiments, the optical system 42 may comprise a camera 60, lens 62, filter 64, and a data connection 66. In some embodiments, the camera 60 may be any compatible camera available in the market currently or in the future. In some embodiments, the camera 60 may have a silicon sensor configured to collect the light. In some embodiments, the sensor may be selected to have a high sensitivity to a specific range of wavelengths of light, including but not limited to (and by way of example only) a range of 750-850 nm, or 750-800 nm, or 800-850 nm. In some embodiments, the camera 60 has a field of view 68 defined as the area in front of the camera lens 60 and / or filter 64 that is captured by the camera. Points of interest within the field of view 68 may be captured by the camera 60 and transmitted as data to the hub 14. In some embodiments, the data may be sent directly to the hub 14 by way of the data connection 66. In some embodiments, theVIS-002-W01 data may be sent indirectly to the hub 14 by way of an intermediate component, which may process the data before sending the data to the hub 14. Points of interest that are outside of the field of view 68 will not be captured by the camera 60. In some embodiments, the lens 62 extends from the camera 60 and may be configured to have an adjustable and / or motorized field of view range. Alternatively, the lens 62 may have a fixed field of view and changing the lens 62 would then change the field of view.

[0230] In some embodiments, the filter 64 may be a high pass wavelength filter or spectral filter. In some embodiments, the filter 64 may be selected and / or configured to block light wavelengths below a predetermined threshold value from passing through the filter 64. In some embodiments, the predetermined threshold value may be 800 nm such that wavelength values below 800 nm are blocked and wavelength values above 800 nm (e.g., including infrared light) are able to pass through the filter 64. In some embodiments, the predetermined threshold value may be 750 nm. In some embodiments, the filter 64 may comprise a band pass filter configured to block light wavelengths below a minumum predetermined threshold value and above a maximum predetermined value from passing through the filter 64 such that light having wavelength values within a specified range may pass through the filter 64. In some embodiments, the specified range may be 800-850 nm. In some embodiments, the specified range may be 810-840 nm. By way of example, such specified ranges may block visible light (e g., light wavelengths below 800 nm) and higher light wavelengths (e g., light wavelengths above 850 nm) from passing through the filter 64. This helps to maximize the dynamic range of the sensor to capture the pattern projected. For example, in a surgical environment, a NIR wavelength range would not be visible to the surgeon or other staff and the ambient lighting within the operating room would be blocked by the filter, allowing only light reflecting off the target area to be allowed into the camera.

[0231] In some embodiments, the data connection 66 may be any high-speed connection that is capable of transferring data captured by the camera 60 to a computer such as the hub 14. In some embodiments, the data connection 66 may comprise a FireFly™ connector and the like.

[0232] Figs. 13-16 illustrate an example of a light source assembly 44 forming part of the CAAT system 10 of the present disclosure, according to some embodiments. In someVIS-002-W01 embodiments the light source assembly 44 includes a linear light source 70, a light shield 72, a light box 74, a transmissive LCD (or pattern generator) 76, and a pattern generator cover 78. In some embodiments, light generated by the light source 70 passes through the light shield 72, light box 74, and pattern generator 76 on its way to the surgical target site. In some embodiments, the light source assembly 44 has an illumination field 80 defined as the maximum possible area in front of the light source assembly 44 that may be illuminated by light emanating from the linear light source 70.

[0233] Figs. 17-19 illustrate an example of a linear light source 70 forming part of the light source assembly 44, according to some embodiments. In some embodiments, the linear light source 70 comprises an elongated emission component configured to emit light through the light shield 72 and light box 74. In some embodiments, the emitted light may have a wavelength value within a specified range. In some embodiments, the specified range may be 800-850 nm. In some embodiments, the specified range may be 810-840 nm. In some embodiments, the linear light source 70 comprises a plurality of LED emitters mounted to a chip-on-board (COB) printed circuit board (hereinafter “COB 82”) in a linear orientation. In some embodiments, the linear light source 70 may comprise one or more groups of LED emitters, with each of the one or more groups of LED emitters independently powered with voltage and current. In some embodiments, the linear light source 70 may comprise 10 groups of 21 LED emitters. In some embodiments, the COB 82 further includes a connector 84 configured to connect the COB 82 to a COB PCB 83 (Fig. 15) which facilitates the electrical connections for the COB 82 and the linear light source 70. In some embodiments, the COB 82 further includes a plurality of apertures 86 configured to receive fasteners that secure the COB 82 to the light box 74, as described below.

[0234] In some embodiments, the linear light source 70 may be configured to approach a theoretical one-dimensional light source as much as possible, in that the elongated linear light source 70 must be as thin as possible to project the sharpest shadows. In some embodiments, the 10 groups of 21 LED emitters (e.g., 210 LED emitters total) may have a width dimension w where rr = 350pm and a height dimension h where h = 350pm such that the thickness of linear light source 70 is 350pm plus any tolerance in the position of the LEDs on the COB 82. This dimension is directly related to the sharpness of the shadows projected onto objects and sharper edges improve signal to noise ratio. In some embodiments, the linear light source 70 may have aVIS-002-W01 length dimension / where I = 120 mm, however the length dimension may be longer or shorter depending on the overall size of the light source and / or 3D scanner 12. In some embodiments, the COB 82 enables the LED emitters to be as densely packed as possible. In contrast, a 0402 electronic package (for example) would only allow a pitch between emitters of about 1-1.2 mm whereas a COB style assembly enables a pitch closer to 600 pm. This represents a significant advantage for the 3D scanner 12 disclosed herein regarding improved light output per unit length without increasing the thickness of the LED emitters. Additionally, heat transfer away from the LED emitters is important for prolonged life of the 3D scanner 12 and efficient conversion of the electrical input to light rather than heat. By way of example, different PCB board materials and dialectrics have different thermal resistances. In some embodiments, the 3D scanner 12 of the present invention may have metal PCB boards and thin ceramic dialectrics, each of which have low thermal resistance compared to aluminum and copper.

[0235] Figs. 20-21 illustrate an example of a light shield 72 forming part of the light source assembly 44, according to some embodiments. By way of example, the light shield 72 is positioned between the linear light source 70 and the top of the light box 74, as shown in Figs. 26-28, and is configured to block a substantial amount of light emanating from the linear light source 70 to reduce or eliminate light reflection in the light box 74 and ensure the maximum amount of light is directed to the transmissive LCD 76 to improve pattern edge sharpness. Additionally, the light shield 72 may reduce or eliminate light scatter from reflective electronic components located near the bright linear light source 70, which improves contrast. In some embodiments, the light shield may be coated in a light-absorptive paint. In some embodiments, the light shield 72 is generally planar and has an elongated aperture 88 sized and configured to enable light passage from the linear light source 70 to the pattern generator while blocking most of the unwanted light emanating from the light source 70 (e.g., reflected light or light not shining directly from the linear light source 70 through the elongated aperture 88) from passing through the pattern generator. In some embodiments, the light shield 72 includes a plurality of passthrough holes 90 having a diameter value configured to allow passage of linear light source fasteners from the linear light source 70 to the light box 74 without securing the linear light source fasteners to the light shield 72 and while providing clearance for movement of the linear light source fasteners due to heated expansion of the linear light source 70 during use. In some embodiments, the light shield 72 includes a plurality of alignment apertures 92 configured toVIS-002-W01 enable alignment of the light shield 72 with the light box 74 (e.g., through visual alignment of the alignment apertures 92 with alignment apertures 120 of the light box 74 and / or mechanical alignment of the respective alignment apertures 92, 120) and a plurality of fastener apertures 94 each configured to receive fastener to secure the light shield 72 to the light box 74. In some embodiments, the light shield 72 further includes a cutout 96 configured to create space for one or more of a power cable, control cable, or thermal measurement component (for example) to the linear light source 70.

[0236] Figs. 22-23 illustrate an example of a light box 74 forming part of the light box assembly 44, according to some embodiments. In some embodiments, the light box 74 may comprise a first or top portion 98 and a second or bottom portion 100 that combine to define a light chamber 102. In some embodiments, the light chamber 102 helps to minimize reflection of the light emitted from the linear light source 70. In some embodiments, the inner chamber 102 is bounded by a top wall 104, a bottom wall 106, and sidewalls 108 with inner surfaces 110. In some embodiments, the top wall 104 comprises part of the top portion 98. In some embodiments, the bottom wall 106 comprises part of the bottom portion 100. In some embodiments, the bottom wall 106 further comprises an egress opening 112 configured to allow the light to pass out of the light box 74 (e.g., through the transmissive LCD 76 and LCD cover 78). In some embodiments, the inner surfaces 110 of the light chamber 102 may be painted or otherwise treated with a light-absorbing material to minimize or eliminate light reflection. In some embodiments, the light-absorbing material may be capable of absorbing infrared light. In some embodiments, the light box 74 is not airtight but is configured in such a way that no direct path exists for light to exit the light box 74 and reenter the light box 74. In some embodiments, the light box 74 may be provided as a single unit rather than in separable first and second portions 98, 100 described herein.

[0237] Figs. 24-25 illustrate an example of a separable top portion 98 of the light box 74 forming part of the light source assembly 44, according to some embodiments. In some embodiments, the top portion 98 comprises the top wall 104 of the light box 74 and provides several of the side walls 108 of the light box 74. In some embodiments, the top portion 98 further includes a plurality of mounting flanges 114 extending from the top wall 104 and including a plurality of fastener apertures 116 configured to receive a fastener to secure the topVIS-002-W01 portion 98 to the bottom portion 100. In some embodiments, side walls 108 may also include a plurality of fastener apertures 116. In some embodiments, the top wall 104 may include an elongated aperture 118 that is aligned with the linear light source 70 upon assembly of the 3D scanner 12 such that light emitted from the linear light source 70 passes through the elongated aperture 118 and into the light box 74. In some embodiments, the perimeter edges of the elongated aperture 18 may be painted or coated in a nonreflective or absorptive substance, such as light-absorptive paint, to minimize or eliminate reflection that might diminish contrast of the projected pattern. In some embodiments, the top wall 104 includes a plurality of alignment apertures 120 configured to enable alignment of the light shield 72 with the light box 74 (e.g., through visual alignment of the alignment apertures 120 with alignment apertures 92 of the light shield 72 and / or mechanical alignment of the respective alignment apertures 92, 120) and a plurality of fastener apertures 122 each configured to receive a fastener to secure the light shield 72 to the light box 74.

[0238] In some embodiments, the top portion 98 of the light box 74 further includes a plurality of COB fastener apertures 124 formed within displaceable flanges 126 formed within the top portion 98 by way of a plurality of cuts 128 formed in the top wall 104 and side walls 108, as shown by way of example in Fig. 24. As noted previously, the COB 82 is fastened directly to the top portion 98 by inserting fasteners through fastener apertures 86 on the COB 82 into fastener apertures 124 of the top portion 98. By way of example, as the linear light source 70 is being used, the considerable amount of heat generated by the linear light source 70 may cause the linear light source 70 and COB 82 to expand longitudinally, causing the associated fasteners to also move in an outward direction parallel to a longitudinal axis of the linear light source 70. Since the COB fasteners are fastened to the fastener apertures 124 positioned within displaceable flanges 126, this longitudinal expansion of the linear light source 70 may cause outward displacement of the displaceable flanges 126 in order to allow this expansion and / or bending of the COB 82 to occur without affecting performance of the 3D scanner 12.

[0239] Figs. 26-28 illustrate the alignment and positioning of the linear light source 70 and light shield 72 relative to the top portion 98 of the light box 74, according to some embodiments. As previously mentioned, the light shield 72 is positioned between the COB 82 with the attached linear light source 70 and the top portion 98 of the light box 74. The light shield 72 is positionedVIS-002-W01 relative to COB 82 such that the elongated aperture 88 is aligned with the linear light source 70. The light shield 72 is positioned relative to the top portion 98 of the light box 74 such that the elongated aperture 88 of the light shield 72 and linear light source 70 are aligned with the elongated aperture 118 of the top portion 98 so that light emanating from the linear light source 70 is directed into the light box 74 through the elongated aperture 118. As shown by way of example in Fig. 27, in some embodiments, the COB 82 is not pressed against the light shield 72 but rather is separated from the light shield 72 by way of one or more heat resistant washers 130, creating a gap 132 between the COB 82 and light shield 72. In some embodiments, the light shield 72 may be mounted flush to the COB 82. In some embodiments, the heat-resistant washers 130 may be made from any material suitable to withstand the heat generated by the linear light source 70, including but not limited to high temperature PEEK plastic. In some embodiments, any of the fastener apertures described herein may be provided with any combination of nuts 134 (e.g., press-fit nuts) and washers 130, to facilitate fastening of the various components together.

[0240] Figs. 29-30 illustrate an example of a bottom portion 100 of the light box, according to some embodiments. In some embodiments, bottom portion 100 provides the main framework for the light chamber 102. As previously mentioned, the light chamber 102 helps to minimize reflection of the light emitted from the linear light source 70. In some embodiments, the bottom portion 100 further comprises an egress opening 112 configured to allow the light to pass out of the light box 74 (e g., through the transmissive LCD 76 and LCD cover 78). In some embodiments, the bottom portion 100 may include a plurality of mounting flanges 136 and fastener apertures 138 for securely connecting the top portion 98 of the light box 74. As well as additional fastener apertures 138 for attaching other components and / or accessories to the bottom portion 100.

[0241] Figs. 31-33 illustrate an example of a pattern generator 76 forming part of the 3D scanner 12, according to some embodiments. In some embodiments, the pattern generator 76 comprises a transmissive LCD screen with individually controllable opacity comprising a liquid crystal layer 140 positioned between first and second polarizing layers 142, 144. In some embodiments, a connector 146 may be provided to connect the pattern generator 76 to a computing device (e.g., the hub 14 or other computing device forming part of the CAAT systemVIS-002-W0110) by way of one or more PCBs positioned on the outside of the light box 74, including but not limited to an LCD PCB 148 (e.g., Fig. 15) and / or a main scanner PCB that connects to component PCB (e.g., LCD PCB 148, COB PCB 83, etc.) and the hub 14. By way of example, the first polarizing layer 142 is positioned facing the inner chamber 102 of the light box 74 between the linear light source 70 and the liquid crystal later 140, and the second polarizing layer 144 is positioned facing away from the light box 74.

[0242] In some embodiments, the pattern generator 76 comprising a transmissive LCD (liquid crystal display) with two polarizers works by controlling how much light passes through the liquid crystal layer 140 using the properties of polarization. In some embodiments, the light source behind the LCD (e.g., linear light source 70) emits unpolarized light through the light box 74 to the pattern generator 76. By way of example, the first polarizing layer 142 is positioned between the light source 70 and the liquid crystal layer 140 and is configured to only allow light waves oscillating in one direction (e.g., vertical) to pass through. This converts unpolarized light into linearly polarized light, ensuring only polarized light enters the liquid crystal layer 140.[00243J By way of example, the liquid crystal layer 140 comprises a matrix of liquid crystal (LC) molecules contained between two glass substrates with transparent electrodes. In their natural “twisted nematic” state, LC molecules rotate the plane of polarization of incoming light by 90 degrees. Thus, when no voltage is applied to the electrodes in the glass substrates, this twisting causes the polarized light to rotate as it passes through the LC layer 140. In some embodiments, a plurality of parallel energized lines 150 extend across the LCD screen. In some embodiments, the plurality of parallel energized lines 150 may also be oriented parallel to the linear light source 70. In some embodiments, the energized lines 150 may be 100 pm wide, separated by 15 pm of space between each line. By way of example, such an arrangement allows for good contrast when the 100 pm wide cells turn dark.

[0244] In some embodiments, the second polarizing layer 144 is polarized at 90 degrees relative to the first polarizer 142 such that the first and second polarizing layer 142, 144 comprise crossed polarizers. In some embodiments, if the LC layer 140 rotates the polarization by 90 degrees (e.g., no voltage applied), the rotated light matches the polarization of the second polarizer 144, so light passes through and the resulting pixels appear bright. In someVIS-002-W01 embodiments, when a voltage is applied to the electrodes in the glass substrates, an electric field is created which causes the LC molecules to align along the generated electric field and lose their ability to twist the light. The polarization of light remains unchanged, and since the second polarizing layer 144 is crossed relative to the first, light transmission is blocked, and the resulting pixels appear dark. In some embodiments, by applying varying voltages to each energized line 150 in the LCD, the LC molecules can partially rotate the light’s polarization, controlling how much light passes through and thus adjusting the brightness of the lines.

[0245] Thus, the first polarizing layer 142 ensures light entering the LC layer 140 is polarized. The LC layer’s twist (controlled by voltage) determines whether light’s polarization matches the second polarizer, and how much light is allowed through the LC layer 140. The second polarizing layer 144 acts as a gate that only passes light correctly rotated by the LC layer 140. This arrangement enables each energized line 150 within the LCD layer 140 to precisely modulate light intensity, forming a distinct and sharp pattern of edges.

[0246] In some embodiments, the first and second polarizing layers 142, 144 may be near infrared (NIR) polarizers (e.g., which allow near infrared light through but block most visible light) instead of more common optically transmissive polarizers. In some embodiments polarizers may also allow for transmission of some visible light (though not as well as the usual polarizers), allowing for dual functionality in the event that the 3D scanner 12 is configured to have both NIR and visible lights present on a single unit.

[0247] Figs. 34-35 illustrate an example of an LCD cover 78 forming part of the 3D scanner 12, according to some embodiments. In some embodiments, the LCD cover 78 includes a frame 152 with a central aperture 154 configured to allow passage of light emanating from the light box 74 therethrough. In some embodiments, the LCD cover 78 includes one or more clips 156 configured to hold the pattern generator 76 in place. In some embodiments, the LCD cover 78 further includes a plurality of attachment flanges 158 that include fastener apertures 160 extending therethrough. In some embodiments, the fastener apertures 160 may include angled, conical, or concave surfaces 162 therein to help position the LCD cover 78 during tightening. In some embodiments, the LCD cover 78 the LCD cover 78 is adjustable to enable alignment of LCD lines 150, which must be straight and parallel as they extend across the LCD screen. InVIS-002-W01 some embodiments, the cover fasteners may be offset slightly so tightening the cover fasteners compresses the cover to the light box 74 and securing the LCD screen 140 in place in proper alignment.

[0248] Figs. 36-38 illustrate an example of a thermal management system (TMS) 46 forming part of the 3D scanner 12, according to some embodiments. By way of example, the TMS 46 is provided to remove heat from the major heat sources in the 3D scanner 12, namely the linear light source 70 and the camera 60 to prevent overheating of the light source 70 and the camera 60. In some embodiments, the TMS 46 may include a heat transfer assembly 170, a heat sink 172, an air box 174, a vent box 176, and one or more ventilation tubes 178. In some embodiments, the TMS system 46 may further include a plurality of thermally insulating spacers 180 configured to ensure that the various components of the TMS system 10 are prevented from contacting the cover 48 and / or scaffolding 50 and to thermally insulate the cover 48 and / or scaffolding 50 from the TMS system 46 and the heat sources (e.g., linear light source 70 and / or optical system 42) to prevent thermal expansion and / or deformation of the cover 48 and / or scaffolding 50. In some embodiments, the spacers 180 may be made of super resilient high temperature silicone foam. In some embodiments, the TMS 46 may further include heat resistant silicone tape 182 wrapped at least partially around several components of the TMS system 46 to help hold them together. In the instant embodiment, the TMS system 10 has a pair of ventilation tubes 178, however more or less are possible. In some embodiments, heat is transferred from the linear light source 70 and the camera 60 to the heat sink 172 by the heat transfer assembly 170. Meanwhile, airflow is forcibly directed into one of the ventilation tubes 178 and / or forcibly pulled out of the other of the ventilation tubes 178 by fans that are at distal ends of the ventilation tubes which may be located outside of the sterile field (e.g., external of the sterile drape). In some embodiments, the forcible pushing and forcible pulling operate in concert so that the airflow is in one direction through the TMS 46. In some embodiments, only pulling fans may be used. In some embodiments, only pushing fans may be used. Heat that is pulled from the linear light source 70 and the camera 60 is directed into the heat sink 172, wherein it is transferred to the air that is directed through the heat sinkl72. This heated air is then directed out of the TMS 456 through a ventilation tube 178.VIS-002-W01

[0249] Figs. 39-40 illustrate an example of a heat transfer assembly 170 forming part of the TMS 46, according to some embodiments. In some embodiments, the heat transfer assembly 170 has a first heat transfer portion 184 configured to transfer heat from the camera 60 to the heat sink 172, a second heat transfer portion 186 configured to transfer heat from the light box assembly 44 to the heat sink 172, a pair of central heat plates 188 configured to absorb heat from the heat transfer pipes 196, 208 and transfer the absorbed heat to the heat sink 172, and a thermally conductive layer 190 (e.g., silicone rubber or similar) positioned between the bottom central heat transfer plate 188 and the air box 174. In some embodiments, the central heat plates have a plurality of elongated grooves sized and configured to receive the heat transfer pipes 196, 208 of the respective first and second heat transfer portions 184, 186 therein.

[0250] Figs. 41-42 illustrate an example of a first heat transfer portion 184 configured to transfer heat from the optical system 42 to the heat sink 172, according to some embodiments. In some embodiments, the first heat transfer portion 184 comprises a heat collection assembly 194 and one more heat transfer pipes 196. In some embodiments, the heat collection assembly 194 includes a pair of heat collection plates 198, a thermally conductive layer 200 (e.g., silicone rubber or similar) positioned between the heat collection plates 198 and the camera 60, and a flame-retardant layer 202 positioned between the heat collection plates 198 and the vent box 176. By way of example, upon assembly of the 3D scanner 12, the first heat transfer portion 184 is positioned such that the thermally conductive layer 200 is in contact with the camera 60 so that the heat transfer may occur (see, e.g., Fig. 8). In some embodiments, the heat collection plates 198 may have a pair of complementary grooves 204 facing one another and each configured to snugly receive a portion of the heat transfer pipe 196 therein so that the heat collection plates 198 may transfer collected heat to the heat transfer pipe 196. The first heat transfer portion is shown herein has having a single heat transfer pipe 196, however multiple heat transfer pipes 196 may be used to increase heat transfer capacity if necessary.

[0251] Figs. 43-44 illustrate an example of a second heat transfer portion 186 configured to transfer heat from the light source assembly 44 to the heat sink 172, according to some embodiments. In some embodiments, the second heat transfer portion 186 comprises a heat collection assembly 206 and a plurality of heat transfer pipes 208. In some embodiments, the heat collection assembly 206 includes a pair of heat collection plates 210, a thermally conductiveVIS-002-W01 layer 212 (e g., silicone rubber or similar) positioned between the heat collection plates 210 and the COB 82. By way of example, upon assembly of the 3D scanner 12, the second heat transfer portion 186 is positioned such that the thermally conductive layer 212 is in contact with the COB 82 so that the heat transfer may occur (see, e.g., Fig. 7). In some embodiments, the heat collection plates 210 may have a plurality of complementary grooves 214 facing each other and each configured to snugly receive at least a portion of one of the plurality of heat transfer pipes 208 therein so that the heat collection plates 210 may transfer collected heat to the heat transfer pipes 208. The second heat transfer portion 186 is shown herein has having six heat transfer pipes 208, however more or less heat transfer pipes 208 may be used to increase or decrease heat transfer capacity if necessary.

[0252] Figs. 45-46 illustrate an example of a heat sink 172 forming part of the TMS 46, according to some embodiments. By way of example, the heat sink 172 of the present embodiment is sized and configured for containment within the air box 174. In some embodiments, the heat sink 172 may comprise a base 216 and a plurality of vertical fins 218 extending from the base 216 and having elongated passages 220 therebetween. Upon assembly of the 3D scanner 12, the heat sink 172 is positioned such that the base 216 is in contact with the thermally conductive layer 190 of the heat transfer assembly 170 so that heat transferred by the heat pipes 196, 208 is then transferred into the heat sink 172. This transferred heat then heats the air being forced through the heat sink 172.

[0253] Fig. 47 is an example of an air box 174 forming part of the TMS 46, according to some embodiments. In some embodiments, the air box 174 may have a rectangular shape and comprises a base 222 and four perimeter walls 224 defining a compartment 226 sized and configured to contain the heat sink 172 therein. In some embodiments, the base 222 comprises a plurality of through-holes 228 configured to enable airflow therethrough.

[0254] Fig. 48 is an example of a vent box 176 forming part of the TMS 46, according to some embodiments. In some embodiments, the vent box 176 may have a rectangular shape and comprises a base 230, four perimeter walls 232, and a divider 234 defining first and second air compartments 236, 238. In some embodiments, the vent box 176 includes a pair of ingress / egress apertures 240, with one ingress / egress aperture 240 extending into each of the firstVIS-002-W01 and second air compartments 236, 238 through a perimeter wall 232. In some embodiments, the ingress / egress apertures 240 are configured to couple with the ventilation tubes 178 to enable passage of the forcible airflow therethrough. In some embodiments, each of the first and second air compartments 236, 238 include a plurality of side recesses 242 aligned with the through-holes 228 of the air box 174 to enable the flow of air into and / or out of the air box 174 and thus the heat sink 172. It should be noted that the direction of air flow through the TMS 46 does not matter so long as the air flows in one direction.

[0255] The specific example of the TMS 46 described herein is one example of a cooling system configured for use with the 3D scanner. In some embodiments, other cooling systems are possible, including but not limited to one or more internal fans that run over one or more heat sinks that allow heat exchange at the scanner surface area in contact with a drape or cooled room (for example). By way of example, such a cooling system may be used with a smaller 3D scanner with a smaller linear light source and / or optical system.

[0256] In some embodiments, a variety of timing mechanisms may be employed to configure the 3D scanner 12 such that the heat-generating components do not overheat the scanner. For example, in some embodiments, the 3D scanner 12 may be configured such that the linear light source 70 is powered on only during image capture by the camera 26, and powered off at all other times, including for example when the pattern generator 76 is updating the image. In some embodiments, this powering off may reduce the time the light source 70 is powered on by 50% or more. Furthermore, the 3D scanner 12 may be configured such that the exposure of the camera 26 may be timed to correspond with moments of high contrast of the LC layer 140 of the pattern generator 76, such that the camera 26 is not exposing while the LC layer 140 is “relaxing” from an energized state. By way of example, this amount of time can be significant when scanning at a high frame rate. In some embodiments, LCDs with shorter relaxation time may also be beneficial. Other LCDs that are actively driven to their relaxed state may also be beneficial, or those that operate in bend vs. twist modes, as their dynamics can be faster.

[0257] In some embodiments, it may be advantageous to change the form of the light box 74, for example such that the path of light through the light box 74 from the linear light source 70 to the pattern generator 76 is not a straight line. In such a case, one or more mirrors may beVIS-002-W01 selectively placed between the linear light source 70 and the pattern generator 76 to redirect the path of the light emanating from the light source 70. This may enable a 3D scanner of reduced physical size, for example, by shortening the required length of the light box 74 while maintaining a necessary optical distance. By of example, Fig. 49 illustrates a linear light source 70 with a light field 80 extending directly through a pattern generator 76 and an optical system 42 with a field of view 68 (e.g., similar to the depiction shown in Fig. 7 with much of the light box 74 and TMS 46 removed). By of example, in comparison, Fig. 50 illustrates a linear light source 70 with a light field 80 extending indirectly through a pattern generator 76 by way of mirror 244 and an optical system 42 with a field of view 68. By way of example, the working volume 69 in each figure is the same or substantially similar, and the images captured by the optical system 42 would be the same in each configuration. The difference lies in that the configuration of Fig. 50 with the mirror 244 allows for a substantially smaller light box. In some embodiments, the positioning of the linear light source 70 may be adjusted or changed, needing only a change in angulation of the mirror to ensure the light field 80 passes directly through the pattern generator 76. In some embodiments, a total internal reflection environment (e.g., a glass prism) may be used to decrease the size of the light box 74 by virtue of its extended optical path in the glass.

[0258] Referring now to Figs. 51-55, as previously noted, much of the light emitted by the linear light source 70 is not heading directly toward the pattern generator 76. While the 3D scanner 12 includes absorptive or blocking components to mitigate the problem of uncontrolled reflection from internal surfaces to ensure that the maximum possible light is directed at the pattern generator 76, additional features may be included to re-capture this light for scanning purposes. For example, in some embodiments, one or more front-surface mirrors 245 may be provided inside the light box 74 to affect light emanating from the linear light source 70 that is not heading directly toward the pattern generator 76 by reflecting said indirect light through the pattern generator 76 toward the object being scanned. In some embodiments, if properly sized and aligned, such mirrors 245 would make the light source 70 appear longer (e.g., almost infinite) along the linear axis of the light source 70. The mirrors 245 would allow more light to be delivered to the target surface without increasing the physical length of the light source. This effectively increases the amount of useful light without introducing extra scatter, while preserving the clean, directional character of the output.VIS-002-W01

[0259] By way of example, Figs. 51-52 illustrate a concept called “virtual LED”. As described above, the linear light source 70 may comprise a row of individual LED emitters 71 arranged in a straight line. Each individual LED emitter 71 shines light in a wide, diffuse pattern, but only a portion of that light travels directly through the window at the bottom of the light box where the pattern generator is located (not shown). Because of the box geometry, the amount of light that escapes through the window from a given emitter depends on where that emitter is located along the width. Emitters near the center have more direct access to the window, while edge emitters are more restricted. A “virtual LED” 247 is created when rays 246 leaving one of the LEDs 71 strikes a side mirror 245. The illuminated point 249 “sees” rays 246 from that LED 71 directly and receives reflected light 248a from a bounce of the mirror, which, by extension backwards is a virtual ray 248b which intersects with the virtual LED 247 on the other side of the mirror. All rays 248a emitted by the real LED 71 that go to the side mirror 245 cause a virtual object of the LED 247 on the other side of the side mirror 245, as shown by example as a ray adjacent to 248a.

[0260] Referring to Fig. 53, by way of example, an edge ray of reflected light 248a is shown resulting from a double bounce event (e.g., light 248a emanating from an emitter from LED 71 and reflecting or bouncing off each side mirror 245). Virtual ray 248b extends from the reflection point behind the mirror 245 to the virtual LED 247. No light reaches the illuminated point 249 from outside this angle a. The number of bounces depends on the distance of the illuminated point 249 from the LEDs 71, and the length of the mirror(s) 245. This example illustrates how virtual LEDs 247 can appear to extend distantly along the axis of the linear light source 70. By way of example, any illuminated point 249 between the mirrors 245 (along a normal of both mirrors) may be illuminated by an “infinite” number of points, except for selfobscuration from the LED assembly itself.

[0261] By way of example, Fig. 54 illustrates the angular advantage of the side mirrors 245. As shown, Oi represents the angular extent of the illumination onto the illuminated point 249 by direct light 246 emitted from the linear light source 70 without reflecting light off of the side mirrors 245. 02 represents the angular extent of the illumination onto the illuminated point 249 by light 248a reflected off the side mirrors 245. From this illustration, it is clear that 02 > 0i, and that light 248a that would otherwise have been lost is recovered.VIS-002-W01

[0262] By way of example, Fig. 55 illustrates an edge ray of reflected light 248a is shown resulting from a multiple bounce event (e.g., light 248a emanating from LED 71 and reflecting or bouncing off each side mirror 245 at least once). Although not shown a virtual ray 248b extends from each reflection point behind the mirror 245 to virtual LEDs 247 (e.g., similar to what is shown in Fig. 53). No light reaches the illuminated point 249 from outside this angle. The number of bounces depends on the distance of the illuminated point 249 from the LEDs 71, and the length of the mirror(s) 245. Any illuminated point within the mirrors (along a normal of both mirrors) would be illuminated from an “infinite” number of points via an infinite number of reflections, except for some limitations including but not limited to: (1) the object itself will itself limit the angles over which the light can bounce to the point via self-obscuration; (2) the light from a single LED will diminish in intensity as the path grows; (3) the light from a planar LED will diminish as the emission angle away from the normal increases, due to the smaller crosssection visible of the LED in that direction; and (4) no mirror is 100% reflective and as the number of bounces increases, reducing brightness. Alignment sensitivity is very high at long path lengths and high number of mirror bounces, constraining the practicality of paths that contain multiple bounces. This alignment sensitivity may be mitigated by the intensity reduction of such long paths relative to more direct paths, as described above.

[0263] Scanning and Data Acquisition Process

[0264] As shown in Fig. 4, by way of example, the 3D scanner 12 may be positioned to continuously capture the surgical field. In some embodiments, as the surgeon manipulates instruments and interacts with the patient’s anatomy, the 3D scanner 12 generates a point cloud of the visible surfaces within the overlap area 69 (e g., the surfaces that are within both the illumination field 80 of the light source assembly 44 and the field of view 68 of the optical system 42). In some embodiments, assuming data is collected at a rate exceeding 20 Hz, the data stream may be continuously updated in real-time. While faster rates are preferred to reduce system latency, rates above 20 Hz are generally sufficient for functional real-time performance. In some embodiments, the point cloud data is then used to create transforms that can be used to update the position and orientation of 3D reference geometry of the anatomy and any tracked instruments within the Surgical Coordinate System (SCS).VIS-002-W01

[0265] Fig. 56 is a block diagram depicting a data acquisition process 250 using a 3D scanner 12 described herein. While the 3D scanner 12 disclosed herein introduces several novel features, the general principles of structured light scanning are well established and are described here for clarity. Furthermore, this data acquisition process 250 may be used with other scanners, such as a shadow caster of the type disclosed in in the ‘973 app.

[0266] In some embodiments, a first step 252 in the data acquisition process 250 is to generate and project a structured light pattern onto a surface to be scanned. For example, in some embodiments, the 3D scanner 12 may illuminate the target surface with a predefined light pattern (e.g., commonly grids, stripes, or dots) projected onto the surfaces being scanned. By way of example, this may be accomplished by configuring the pattern generator 76 described above to produce the desired structured light pattern. In some embodiments, the 3D scanner 12 described herein may be configured to use infrared light, ensuring that the operating room lighting does not interfere with the scanning process. In some embodiments, a series of light patterns is projected to encode all the data necessary for reconstructing the 3D point cloud. Multiple pattern series can be used, depending on the specific requirements of data encoding. In some embodiments, variations in intensity, color, or phase may be applied to the pattern to enhance data accuracy or accelerate acquisition.

[0267] In some embodiments, a second step 254 in the data acquisition process 250 is image capture. For example, in some embodiments, cameras or sensors within the 3D scanner 12 (e.g., optical system 42 described above) capture the reflected structured light as it interacts with the target surface. In some embodiments, an image may be taken for each pattern projected in the series. By way of example, the captured images reflect deformations in the light pattern caused by the surface's contours and depth. In some embodiments, these images may be collected by one or multiple cameras, with the relative positions and orientations of the cameras being known to the CAAT system 10. In some embodiments, the first step 252 of generating and projecting structured light pattem(s) onto a surface to be scanned and the second step 254 of image capture may be repeated in an iterative loop until all the required images have been acquired.

[0268] In some embodiments, once all required images have been obtained by way of the previous steps, a third step 256 in the data acquisition process 250 is reconstruction of the 3DVIS-002-W01 surface, for example within the hub 14. In some embodiments, reconstruction of the 3D surface may occur on board the 3D scanner 12 in a main control board rather than in a separate hub 14, in which case the following sub-steps also occur within the main control board of the 3D scanner 12. In some embodiments, the reconstruction step 256 includes sub-steps of pattern decoding, triangulation and 3D point cloud generation. In some embodiments, after image capture, the images are sent to the hub 14 for processing. In some embodiments, a first sub-step 258 of the reconstruction step 256 is pattern decoding. By way of example, data can be transferred from the 3D scanner 12 to the hub 14 continuously as images are captured, or in bulk once an entire pattern series is complete. In some embodiments, the reconstruction step 254 cannot be completed until all captured images are transferred to the hub 14, but some computation can be completed on each image as it arrives in the hub 14. In some embodiments, the CAAT system 10 is configured to decode the captured light patterns by analyzing their brightness history to create camera pixel correspondences to light angle projection. This, together with known pixel observation ray, enable via triangulation in-depth information, allowing the CAAT system 10 to reconstruct the 3D surface as a 3D point cloud.

[0269] In some embodiments, a second sub-step 260 of the reconstruction step 256 is triangulation. By way of example, the CAAT system 10 may use triangulation to calculate the 3D coordinates of each point where the light interacts with the surface. With the known positions of the linear light source 70, camera 60, and pattern generator 76, the CAAT system 10 determines a 3D point for each pixel in the captured images.

[0270] In some embodiments, a third sub-step 262 of the reconstruction step 256 is generation of a 3D point cloud 268 representing the scanned target surface. By way of example, the triangulated points from each frame may then be combined into a 3D point cloud 268, representing the surface in 3D space, with each point assigned x, y, and z coordinates. An example of a 3D point 268 cloud is shown in Fig. 57. In some embodiments, depending on system capabilities, additional data such as color or intensity may be included to enhance detail and visual fidelity.

[0271] In some embodiments, a fourth step 264 in the data acquisition process 250 is data processing and filtering. For example, in some embodiments, the raw point cloud data mayVIS-002-W01 undergo post-processing to reduce noise, fdl gaps, and improve accuracy. In some embodiments, if multiple scans are taken from different angles, the data may be aligned, merged, or "stitched" together to create a cohesive 3D model.

[0272] In some embodiments, a fifth step 266 in the data acquisition process 250 is to output the data such that the final point cloud 268 is made available for further processes, such as segmentation and registration, which can be used to guide the surgical workflow.

[0273] Differences between optical tracker generated data and 3D scanned topology data

[0274] It is important to distinguish between the data generated by Optical Markers (OMs) and Optical Trackers (OTs) (e.g., as used in prior art system) versus the data produced through 3D scanning of a surgical site, as disclosed herein.

[0275] By way of example, in Figs. 58A-B, the image on the left (Fig. 58A) depicts a typical TKA surgery, showing externally applied optical trackers 270 attached to the robot 272, femur 274, and tibia 276. Each externally applied optical tracker 270 comprises four optical markers 278 in a predetermined arrangement. This image represents the typical field of view for stereoscopic systems. On the right (Fig. 58B), the image shows what one of the stereoscopic cameras captures after segmenting the image to isolate the OMs 278 (which by way of example appear as black markers on a white background). Clearly visible are three OTs 270, each consisting of four OMs 278. When two such images, taken from slightly different angles, are combined via triangulation, the OTs 270 can be located in space, inferring the positions of the bones and robotic instruments registered to their respective trackers.

[0276] In contrast, the data collected by a 3D topological scanner such as 3D scanner 12 presents a very different picture. By way of example, Figs. 59A-B illustrates images from the 3D scanner 12. By way of example, the image on the left (Fig. 59A) is a scanner image 280 captured by the 3D scanner 12 using infrared light, and the image on the right (Fig. 59B is a 3D point cloud 268 generated from the whole pattern series of collected image data.

[0277] For example, one key difference is the field of view. For a stereoscopic system to capture all the necessary OTs 270 during surgery, the field of view must be quite large, often requiring the cameras to be positioned several meters away, typically at an angle from the sideVIS-002-W01 and slightly above the surgical area. From this vantage point, the bones are often not visible to the naked eye, as they may be obstructed by surgeons, instruments, other tissues, etc. By contrast, the 3D scanner 12 provides a much narrower field of view, captured from a closer, top- down perspective, for example as shown in Fig. 4.

[0278] Another key difference is that the data density is markedly different between the two systems. By collecting points one by one, the stereoscopic system provides sparse data, providing around 50 possible data points on the surface of a femur, for example. This must be collected manually by using a pointer to touch a point and then a foot pedal may be activated to capture the point. This is time consuming and can be error prone if the pointer is moving. Also, on the articular surface of the femur and tibia, if the surgeon is trying to register to preoperative CT bone, then the pointer must be sharp and penetrate the cartilage fully without penetrating the bone. Conversely, the 3D scanner 12 of the present disclosure captures a much more robust data set without penetrating patient anatomy. For example, the image shown in Fig. 59 has approximately 4 megapixels, with about 300,000 individual points on the femur and 200,000 on the tibia. Additional points correspond to surrounding soft tissues. In some embodiments, a depth filter may be applied to the data to remove much of the background (such as the blue surgical drape).

[0279] By way of example, to further illustrate the 3D nature of the data, Fig. 60 presents the scan data (e.g., point cloud 268) from an oblique angle, giving a clearer sense of its depth. By way of example, for users familiar with knee anatomy, identifying key anatomical structures is relatively straightforward. However, for clarity, Fig. 61 highlights several major anatomical landmarks, by way of example, including the distal end of the femur 274, the anterior cruciate ligament (ACL) 282, the proximal end of the tibia 276, the patellar tendon 284, and a side view of the patella 286. Other points correspond to skin, muscle, fascia, and fat, though they are not specifically labeled. Note that these images use simulated tissue models commonly used for research, not real tissue.

[0280] By way of example, in real tissue, the visual characteristics will vary due to differences in how materials interact with light. These images are representative, but real tissue tend to introduce more noise, particularly in tissues that are partially transparent to infrared light.VIS-002-W01In some embodiments, this increased noise can be mitigated by applying more aggressive filtering algorithms, for example. In some embodiments, to help define a more reliable shape at one point in a procedure, a wavelength may be used that has a shorter optical penetration length. That wavelength may overlap with other sources of light in the operating room, but may accommodated for the sake of an initial accurate view for reference with subsequent scans taken at wavelengths that introduce more noise, but that do not compete with other sources of light in the operating room. In addition, it may also be a relatively narrow-band source (e.g. 5 to 10 nm bandwidth) that has a relative brightness at that wavelength much greater than that of the room light, such that spectrally filtering for that wavelength results in high-contrast scans with minimal competition with the ambient light.

[0281] Registration Using Topological Data

[0282] Once topological data is available in the form of a 3D point cloud 268, a wide range of capabilities become possible. In some embodiments, multiple 3D point clouds may be aligned and / or combined within a single coordinate space for a more accurate and efficient surgery. By way of example, this aligning and / or combining may occur by way of a process called “registration”, which aligns two or more 3D point cloud datasets, combining them into a common coordinate space. This registration process allows different datasets — such as preoperative CT scans and intraoperative 3D scans — to be aligned and even combined, enabling more precise surgical planning and execution. By way of example, Fig. 62 illustrates an example registration process 290 that may be performed by the CAAT system 10. In some embodiments, the registration process 290 comprises several steps including but not necessarily limited to preprocessing the data for accuracy 292, segmentation 294, registration of anatomical structures 296, registration of instruments and equipment 298, and making precise measurements 300. Although presented as a flowchart, many of these “steps” may be performed in any order, in series or parallel, and / or independently of one another. For example, anatomic registration and instrument registration may occur in any order, independent of one another, or one may occur without the other occurring.

[0283] In some embodiments, a first step 292 in the registration process 290 using topological point cloud data is pre-processing the raw point data cloud to ensure accuracy and reliability. InVIS-002-W01 some embodiments, this pre-processing step 292 may include several sub-steps including noise reduction 301, smoothing 302, upsampling 303 the 3D point cloud, and / or downsampling 304 the 3D point cloud. By way of example, noise in the data can be introduced from various sources, such as light interference or reflections from surgical instruments. In some embodiments, noise reduction techniques 302 like statistical outlier removal and radius-based filtering (for example) are used to clean the dataset, ensuring that only relevant points remain. In some embodiments, smoothing algorithms 303, such as Gaussian filtering or bilateral filtering, may be applied to create a more continuous and realistic surface representation. For example, this step may help to reduce local variations in point locations that are not significant for registration, ensuring a more refined dataset without losing critical anatomical details. In some embodiments, the 3D point cloud may optionally be upsampled 304 by increasing the data points in the point cloud. In some embodiments, the 3D point cloud may optionally be downsampled 304 by decreasing the data points in the point cloud. In some embodiments, further data processing techniques such as outlier removal 305, normal estimation 306, and / or feature extraction 307 may be used.

[0284] In some embodiments, a second step 294 in a registration process 290 using topological point cloud data is segmentation. Segmentation enables the CAAT system 10 to handle multiple objects simultaneously during registration while reducing the computational load. By way of example only, once the point cloud data has been pre-processed, the segmentation step 294 can be applied to isolate different anatomical structures or surgical tools. Segmentation is an optional step but provides significant benefits for both data handling and computational efficiency. One such benefit is that segmentation allows for the identification of different objects within the dataset. For example, all points belonging to the femur can be isolated from the main dataset, while points belonging to the tibia can be similarly separated. Any remaining data can be segmented or discarded if unnecessary. Another benefit of segmentation is related to speed and latency. For example, registering large point clouds can be computationally intensive. By segmenting and reducing the dataset to only the objects of interest, computational requirements are lowered, improving processing speed and reducing overall system latency. In some embodiments, the segmentation step 294 may occur before the pre-processing step 292 to perform segmentation using raw, unfiltered data.VIS-002-W01

[0285] In some embodiments, various methods of segmentation 294 may be used, including but not limited to manual segmentation 308, traditional algorithms 309, and artificial intelligence (Al) 310 powered segmentation. By way of example, manual segmentation 308 may be useful in complex cases, as operators can manually label objects in the point cloud to isolate key structures. In some embodiments, traditional algorithmic techniques 309 such as edge detection or region-growing may be used to identify structures based on their shape or intensity. In some embodiments, Al-powered segmentation 310 may be used so that pre-trained deep learning models can automatically segment point cloud data in real-time, offering high accuracy and minimal manual intervention. In some embodiments, the Al-powered segmentation 310 may operate within a 2D image space (e.g., a 2D image or 2D depth-map representation) or a 3D space (e.g., point cloud space).

[0286] In some embodiments, a third step 296 in a registration process 290 using topological data is anatomic registration. In most cases, registration involves aligning two datasets, such as a preoperative CT dataset and a 3D topological scan from the surgical site. By way of example only, the alignment may be performed using algorithms such as RANSAC (Random Sample Consensus) 312 for initial, coarse registration and ICP (Iterative Closest Point) 314 for finetuning. These algorithms compare corresponding points between datasets and compute a transformation matrix that minimizes alignment errors. For example, RANSAC 312 selects key features in the point cloud, filtering out outliers, and computes an initial transformation matrix by focusing on valid, inlier points, and ICP 314 iteratively refines the resulting transformation matrix by minimizing the distance between corresponding points in the two datasets, leading to precise alignment. The resulting transformation is a 4x4 rigid matrix that translates, rotates, and scales one dataset to align with the other, for example similar to the transformation matrix shown in Fig. 1. In the context of a TKA procedure, for instance, this could align a CT scan with a realtime topological scan of the femur 274 and tibia 276

[0287] In some embodiments, the anatomic registration step 296 can be repeated for multiple objects, such as the femur 274 and tibia 276, allowing simultaneous registration of more than one object assuming the computational horsepower is available. Fig. 63 illustrates the registration of CT data from the femur 274 and tibia 276, along with a 3D topological scan or point cloud 268, into a common coordinate space 316. By using registered datasets, the CAAT system 10VIS-002-W01 provides surgeons with accurate measurements that aid in making critical decisions during surgery. By way of example only, Fig. 63 shows the registration of three datasets — CT data from the femur and tibia, and a 3D topological scan — into a common coordinate space for this purpose.

[0288] The registration step 296 is a critical step in aligning two 3D point cloud datasets, such as preoperative CT data with a real-time topological scan in a surgical setting. This method achieves the same results as registration done with Optical Markers (OMs) 278 and Optical Trackers (OTs) 270, but without the need for external tools anchored to bones or an additional transformation to account for the indirect determination of a transformation matrix via the trackers. Additionally, the topological dataset is far denser and more detailed than what a surgeon could collect using a pointer with OT.

[0289] In some embodiments, a fourth step 298 in a registration process 290 using topological data is instrument and equipment registration. By way of example, this step is optional and may occur before, during, or independently of anatomic registration in the previous step 296. In addition to anatomical structures, instruments and surgical equipment can be registered using topographical data. By way of example, each instrument has a unique shape that can be used for registration. For instance, a bone saw has a distinct geometry compared to a scalpel, making identification possible in the topological data. In some embodiments, the same registration methods used for anatomy can be applied to instruments, but some challenges exist. For example, instruments with reflective surfaces may not be well-captured in the 3D scan. In some embodiments, treating surfaces to scatter light can reduce specular reflections and improve scan quality. Another challenge is that some materials absorb light, causing poor visibility in the scan. As with specular reflections, in some embodiments treating surfaces to scatter light can mitigate this issue. By way of example, another challenge with instrument and equipment registration is occlusions, in that instruments and / or equipment may be partially obscured by the surgeon's hand. In some embodiments, by extending the handle or otherwise introducing distinct geometry, the CAAT system 10 can still register the instrument even when partially blocked. In some embodiments, distinct geometry may include (but is not limited to) portions of the tracker that are not optical markers, patterns printed on the instruments (e.g., particularly on their mostVIS-002-W01 distal end) and / or attachment of a fiducial to the instrument enabling identification and understanding of the instrument position and / or orientation at the surgical site itself.

[0290] In cases where an anatomical point is not directly visible (e.g., the center of the ankle), a pointer instrument can be used. In some embodiments, for example, registering an end of the pointer to the scanned data, the anatomical point can be inferred accurately. In some embodiments, this could be done by registering a proximal end (e.g., that would extend into camera view) of the pointer in the common coordinate system and, based at least in part on the known attributes or the pointer, the relative positioning of the distal tip (e.g., that would be positioned near the target out-of-view anatomical point) would be known or inferred. In this way, moving the scanner in order to change its field of view to see anatomy at the ankle (for example) may be avoided.

[0291] In some embodiments, a fifth step 300 in a registration process 290 using topological data is measurements. For example, once the objects are registered within the same coordinate space, precise measurements can be made, providing crucial information during surgery. These measurements may include, but are not limited to distance measurements, angle measurements, and / or gap measurements. By way of example, measurements made after registrations are relative measurements between two registered objects within the coordinate space or a single registered object and the scanner reference frame in the coordinate space. In some embodiments, the CAAT system 10 can measure the distance between two points on anatomical structures or instruments. For example, measuring the width of the knee’s sulcus is as simple as selecting two points (either manually, or with the points identified by Al or by registration with a model, or with CT), with the system computing the distance in real-time. In some embodiments, more complex measurements, like assessing the varus or valgus angle of the knee, can be performed by identifying the sagittal planes of the femur and tibia in the registered datasets. Once identified, the system calculates the angle between the planes, providing vital information for implant alignment. In some embodiments, if two objects are registered, measurements between the two objects can be computed. For example, in TKA surgeries, the medial and lateral gaps between the femoral condyles and tibial plateau can be measured to assess ligament balance. Surgeons often aim to balance these gaps to ensure even tension in the knee, which is critical for the success of the implant.VIS-002-W01

[0292] Also, by measuring known anatomic features of the knee, careful fitting of a parametric model of the knee may be possible directly for implant selection or even real-time manufacture.

[0293]

[0294] Tracking with Topological Data

[0295] By way of example, tracking and registration are closely related, but they serve distinct purposes in a surgical setting. As previously described, registration is the process of aligning two or more 3D point cloud datasets, bringing them into a single coordinate system. This is typically a one-time event or a relatively infrequent operation. In contrast, tracking involves continuously registering these datasets over time, monitoring changes in position or orientation, and keeping objects aligned within the same coordinate system. Tracking updates the spatial alignment of registered objects multiple times per second, which allows for dynamic monitoring of anatomical structures and surgical tools as they move throughout the procedure.

[0296] Real-time tracking is essentially registration performed in a continuous loop, where the system rapidly processes new data to reflect movements or changes as they occur. The faster this loop is executed, the more accurately the system can represent the current state of the surgical environment. In some embodiments, and by way of example only, registration uses a dense point cloud, while tracking may use “sparse” scans, which are not the highest pixel density scans, but are much faster to take. The goal of real-time tracking is to create a seamless experience where movements of anatomical structures, instruments, or other objects are captured and reflected without noticeable delay.

[0297] Defining what constitutes "real-time" is challenging because it varies depending on the context and application. In the world of cinema, for example, a standard film is displayed at 24 frames per second (fps). At this rate, the human brain perceives the sequence of still images as continuous motion, which most viewers consider to be real-time. Computer monitors typically operate at higher refresh rates, such as 60Hz or 120Hz, offering smoother visuals. Gaming monitors push this further, with refresh rates of up to 240Hz, though this is generally only advantageous in specific, fast-paced applications.VIS-002-W01

[0298] From a physiological standpoint, studies suggest that the human brain begins to interpolate between frames at speeds around 24 fps, making discrete images appear as a fluid, continuous motion. For the purposes of this disclosure, we define real-time tracking as performing at least 20 registrations per second (rps). This threshold provides a sufficiently smooth experience, making the data appear continuous to the surgeon. Faster registration speeds are always preferable, as they can further reduce latency, but they come with significant tradeoffs in terms of the system’s requirements. Higher speeds increase the demands on light intensity, the speed of image sensors, and computational power to process the data quickly enough to maintain real-time performance.

[0299] In some embodiments, speed may be increased by bypassing the coarse or global registration step (e.g., RANSAC as described above) during tracking, and instead loop through the fine registration (e g., ICP) step. This is because typically the tracked object does not move very much or very far in the short time between scan registrations, it is already close enough for successful ICP operation. In some embodiments, the exclusion of a coarse registration step may provide a significant increase in tracking speed. In some embodiments, tracking may be lost during ICP-only operation (e.g., the tracked object moves faster or farther than ICP can successfully register). In such cases, by way of example only, the system 10 may revert to a global registration step which may be RANSAC as described above, or in some embodiments an Al-driven analysis of captured images (e.g., from the 3D scanner 12) may be used to quickly determine the location of the tracked object.

[0300] In some embodiments, the CAAT system 10 may increase tracking speed by using a previously registered object position to constrain the considered data points of where the object may move to next for fine registration. In some embodiments, this would reduce the points for consideration during fine registration (e.g., ICP), and fewer points to analyze generally leads to increased processing speed. By way of example, in some embodiments, this data may be constrained based on velocity and / or acceleration. In some embodiments, an object can’t go faster than a maximum determined velocity in any direction, which limits or constrains the potential data points needed for fine registration. In some embodiments, the data points could be constrained base on likely acceleration of the object. In some embodiments, object velocity may be used to extrapolate an initial guess location for the fine registration.VIS-002-W01

[0301] In some embodiments, the CAAT system 10 may constrain the considered data points of where the object may move to next for fine registration by limiting points of consideration in the acquired topological scan(s) that are a certain distance away from the previously registered object points. In some embodiments, this may include points located within an axis-aligned bounding box in every direction by an amount that is limited by practical physics, for example a set amount (e.g., a guess without first measuring velocity). In some embodiments, the CAAT system 10 may be configured to select a direction for the bounding box growth based on its previous velocity.

[0302] In some embodiments, the CAAT system 10 may constrain the considered data points of where the object may move to next for fine registration by considering only points within a radius from each of the relevant points (and optionally in a particular direction).

[0303] Single object tracking (Hip Center Example)

[0304] The simplest case of tracking is the real-time monitoring of a single object, where its position is continuously updated in the Surgical Coordinate System (SCS). This type of tracking is critical in various surgical procedures where accurate positioning of key anatomical landmarks or surgical tools is required. In this example, the scanner is kept stationary, or its motion is itself tracked relative to the operating room via some other navigation device.

[0305] A practical example of single object tracking in a Total Knee Arthroplasty (TKA) procedure is the calculation of the hip center of rotation. In some TKA methodologies, such as the Mechanical Alignment approach, determining the precise location of the hip center is essential for establishing the mechanical axis of the leg, which guides the alignment of implants. By way of example, Fig. 64 presents a block diagram depicting steps of a method 320 of determining the hip center of rotation using the CAAT system 10 described herein. In some embodiments, the first eight steps of the method may be repeated in a (rapidly occurring) loop until sufficient data has been accumulated to accurately determine and report a hip center of rotation and femur length. By way of example, a first step 322 in the method 320 of determining the hip center of rotation is using the 3D scanner 12 described herein in tracking mode to acquire topological scan data of the femur 274. In some embodiments, a second step 324 of the method 320 of determining the hip center of rotation is registering the acquired scan data to a 3D modelVIS-002-W01 of the femur 274, which for example may be derived from a pre-operative CT scan. In some embodiments, a third step 326 in the method 320 of determining the hip center of rotation is producing a transformation matrix for each time point. This transformation matrix 2 maps the position of the femur into camera space, allowing the system to track the femur's movements in real-time as the surgeon manipulates the leg.

[0306] In some embodiments, a fourth step 328 of the method 320 of determining the hip center of rotation is to use the generated transformation matrix to compute the location of achosen anatomic point for determining the length of the femur in camera space. A fifth step 330 of the method 320 of determining the hip center of rotation is to send coordinates of the tracked chosen anatomic point to the hub 14 (for example) for sphere fitting.

[0307] As previously mentioned, the first eight steps of the method may be repeated in a (rapidly occurring) loop until sufficient data has been accumulated to accurately determine and report a hip center of rotation and femur length. The first five steps identified above describe the CAAT system 10 producing a data point comprising calculated (x,y,z) coordinates for the femur origin location with each scan, which may be happening at least 20 times per second. Thus, a sixth step 332 of the method 320 of determining the hip center of rotation is to add a newly determined coordinates data point to the dataset containing the previously calculated coordinates. In some embodiments, a seventh step 334 of the method 320 of determining the hip center of rotation is to calculate the size and location of a sphere to which the accumulated data points were best fit, for example using least squares fitting to find the best-fitting sphere, which minimizes the difference between the points and the surface of the sphere. The center of this sphere represents the hip center of rotation. In some embodiments, an eighth step 336 of the method 320 of determining the hip center of rotation is to compare residual errors of the fit of the sphere to the predetermined acceptance criteria to assess whether the generated best fit sphere is a good fit (e.g., decision box 336). If the sphere is not a good fit, then the workflow reverts to the first step 322 of acquiring topological scan data of the femur 274. In some embodiments, if the generated best fit sphere is determined to be a good fit, then a final step 338 in the method 320 of determining the hip center of rotation is to report the hip center and femur origin length (e g., based upon the position of the sphere origin and the radius of femur).VIS-002-W01

[0308] While some noise is inevitably introduced due to factors such as movement artifacts in the pattern images or reconstructed object, or slight variations in scan accuracy, the CAAT system 10 can compensate for these by averaging or applying noise-reduction techniques. Once the hip center is identified in camera space, it can easily be transformed into any other coordinate system (e.g., SCS), ensuring that this critical anatomical point is available for accurate surgical planning and alignment, including but not limited to intraoperative real-time modification of the surgical plan.

[0309] This example highlights the value of single object tracking in real-time during surgery, where critical anatomical landmarks must be continuously monitored and accurately calculated to guide the procedure.

[0310] Two Object Tracking and Measuring (Gap Balancing Example)

[0311] A more complex use case for tracking occurs when two or more objects are tracked simultaneously. While a previous section described how distances can be calculated between two objects through registration, this concept can be extended into a continuous tracking mode, where multiple registrations are performed over time to monitor the relative positions of multiple objects. This is especially valuable in real-time scenarios, where dynamic forces are applied, and accurate measurements must be taken as the objects move in relation to each other.

[0312] An example of this in a Total Knee Arthroplasty (TKA) procedure is when the surgeon uses a gap balancing methodology to assess the gaps between the medial and lateral condyles of the femur and the tibial plateau. By way of example, Fig. 65 is a flowchart depicting an example method 350 of assessing gaps between two objects, for example femoral condyles (e.g., medial and lateral) and a tibial plateau. In this method 350, the goal is to ensure that the gaps between the femur and tibia are balanced when the knee is in both extension and flexion, which helps maintain proper knee alignment and functionality after the procedure.

[0313] Measuring these gaps presents a challenge because the surgeon does not perform this measurement passively. Instead, they manually apply forces to the knee in both a varus (inward) and valgus (outward) direction. This creates a dynamic scenario where the knee joint, the entire leg, and the surgeon’s hands are moving as the force is applied. The critical challenge is that theVIS-002-W01 surgeon is not looking for a single static gap value but rather the peak gap under maximum load. Because significant force is applied during this process, the positions of both the femur and tibia are constantly shifting, and these movements happen quickly. The gap measurement needs to be recorded at high speed, tracking the movement of both the femur and tibia in real time to capture the moments where the force is greatest, and the gap is widest.

[0314] In some embodiments, a first step 352 in the method 350 of assessing gaps between femoral condyles and a tibial plateau is to acquire topological scan data of the femur and tibia and generate reconstructed 3D point clouds of each, for example using techniques described above. In some embodiments, a second step 354 of the method 350 of assessing gaps between femoral condyles and a tibial plateau is to independently register the femur and tibia in the CAAT system 10 as described above, for example using the acquired and reconstructed scan data in addition to reference data 356 of the femur and tibia, for example from preoperative imaging (e.g., CT imaging) or data collected intraoperatively. In some embodiments, a third step 358 in the method of 350 assessing gaps between femoral condyles and a tibial plateau is to use transformation matrices to convert reference data to the surgical coordinate system or to another virtual space. In some embodiments, a fourth step in the method 350 of assessing gaps between femoral condyles and a tibial plateau is to calculate the distances between key points on the femur and tibia in real time. As the surgeon applies varus and valgus forces, the system tracks the resulting movements and dynamically measures the gaps between the medial and lateral condyles and the tibial plateau. In some embodiments, this includes performing a first sub step 360 of computing the medial gap (e.g., the minimum distance between the femur and tibia on the medial side) and a second sub step 362 of computing the lateral gap (e.g., the minimum distance between the femur and tibia on the lateral side). In some embodiments, a fifth step 364 in the method 350 of assessing gaps between femoral condyles and a tibial plateau is to consolidate the data and present it to the surgeon in a usable format. In some embodiments, the next step 366 is to determine whether there is enough data to proceed with the procedure. If the answer is no, then the workflow reverts to the first step 352 of acquiring and reconstructing topological scans of the femur and tibia, and the workflow continues to cycle through the steps until enough data has been obtained to assess the gaps between femoral condyles and a tibial plateau, at which point the tracking may end (box 368).VIS-002-W01

[0315] In some embodiments, by performing multiple registrations rapidly (at a rate defined as real-time, e.g., 20+ registrations per second), the CAAT system 10 can record the gap values continuously as the knee moves under load. In some embodiments, recorded measurements may be analyzed to determine the peak gap value at the moment when the maximum force is applied in each direction. Because these forces are only applied for a brief period, rapid and continuous tracking is essential to ensure the correct measurements are captured at the precise moment.

[0316] There are several benefits to real-time gap measurement. For example, the ability to track the femur and tibia in real time allows for dynamic gap measurements under varying loads, providing more accurate and clinically relevant data than static measurements. Additionally, real-time tracking ensures that the CAAT system 10 captures the peak gap values, which are critical in determining the balance between the medial and lateral sides of the knee.Furthermore, since the leg and knee joint are moving throughout the measurement process, continuous tracking allows the system to compensate for these movements and still produce accurate results.

[0317] Tracking Anatomy with a Robot-Assisted Surgical Instrument

[0318] A more complex use case for tracking arises when a surgical instrument is attached as the end effector of a robot. In this scenario, the position and orientation of the instrument must be continuously updated to accommodate any anatomy movement. An example of this in TKA is when an oscillating saw is attached to a robotic system, and the robot assists the surgeon by positioning the saw in the correct plane for a planned osteotomy. By way of example, Fig. 4 illustrates an operating environment 22 in which a surgical instrument 28 is positioned on the end of a robotic arm 30.

[0319] By way of example, the robot’s function is to prevent the saw blade from drifting outside of the predefined cut plane, maintaining specific translational and angular tolerances. This process is straightforward if the knee is rigidly fixed. However, in practice, the knee may move slightly due to various factors such as patient movement or surgeon manipulation. As the knee moves, the robot must continuously adjust to follow the anatomy and ensure the osteotomy remains accurate. Traditionally, this can be achieved using Optical Markers (OMs) and Optical Trackers (OTs), but it can also be accomplished using real-time topological data collected fromVIS-002-W01 the 3D scanner 12 without the use of OMs and / or OTs. For example, Fig. 66 is a block diagram illustrating an example method 370 of tracking movable anatomy and a movable surgical instrument configured to affect the movable anatomy, according to some embodiments of the disclosure. In some embodiments, this method 370 is essentially an extension of the multi-object tracking described previously, but with the additional complexity of having to communicate the anatomy’s movement to the robot so that the position of the end effector can be updated in real time.

[0320] In this example scenario, multiple objects are tracked simultaneously. Thus, in some embodiments, a first step 372 in the method 370 is to acquire topological scan data of the target anatomy (e.g., a femur) and the surgical instrument 28 and generate reconstructed 3D point clouds of each, for example using techniques described above. In some embodiments, a second step 374 of the method 370 of tracking movable anatomy and a movable surgical instrument configured to affect the movable anatomy is to independently register the target anatomy and the surgical instrument 28 on the end of the robot arm 30 in the CAAT system 10 as described above, for example using the acquired and reconstructed scan data in addition to reference data 366 of the target anatomy and surgical instrument. In some embodiments, the reference data 366 of the target anatomy may be acquired from preoperative imaging (e.g., CT imaging) or data collected intraoperatively. In some embodiments, the reference data 366 of the surgical instrument 28 may be acquired from preoperative CAD data, preoperative scans, intraoperative scans, and / or a fiducial marker if the instrument does not scan well. In some embodiments, the origin point (or other point of interest) of the instrument 28 is identified in the data set or calibrated at the time of surgery.

[0321] In some embodiments, a third step 378 in the method 370 is to use transformation matrices to convert the target anatomy position and orientation into robot space. By way of example, the robot's base remains fixed, and a point on the base is defined as the origin of the surgical coordinate system (or “SCS”), which in this example may also be referred to as "robot space." In some embodiments, all other objects (3D scanner 12, instrument (e.g., saw) 28, femur 274, etc.) may be transformed into the robot space to ensure alignment. By way of example, by tracking the femur, its position can be continuously transformed into the robot space. In some embodiments, an optional step 380 of the method 370 may be performed to smooth the inputVIS-002-W01 data to reduce the effects of noise before the data is sent to the robot, for example using a moving average and / or Kalman fdter.

[0322] In some embodiments, a fourth step 382 in the method 370 is to send the target anatomy position and orientation to the robot. By way of example, the instrument 28 (e.g., saw blade) can be tracked either directly using the 3D scanner 12 as disclosed herein or by relying on data from the robot's internal control system, which already knows the position of its end effector (and hence the saw blade) in robot space.

[0323] In some embodiments, a practical result of the method 370 is that the position of the target anatomy is continuously tracked as it moves during surgery, for example using the CAAT system 10 and the 3D scanner 12. In some embodiments, the updated position of the target anatomy is continuously communicated to the robot 30, which as a fifth step 384 in the method 370, may intraoperatively plan the optimal kinematic motion path to maintain relative position to the target. Stated differently, the robot may adjust the instrument 28 (e.g., saw blade) movement in real time to maintain alignment with the target anatomy and the predefined cut plane. By way of example, the robot effectively compensates for any slight anatomical shifts, ensuring the accuracy of the osteotomy.

[0324] This real-time adjustment can be performed within predefined translational and angular tolerances, providing the surgeon with a more precise and reliable way of cutting within a moving surgical field. Depending on the design of the robot and its positional control system, it may be necessary to update the registration data at a rate faster than 20 registrations per second (rps) to achieve a finely tuned control system.

[0325] In some embodiments, a sixth step 386 in the method 370 is to execute the motion path of the instrument with minimal latency between the time the 3D topological data is captured to when an updated transformation matrix is sent to the robot. In such a use case, latency becomes critical. Any delay in updating the robot’s position relative to the anatomy can lead to the saw blade falling out of alignment with the femur, reducing the accuracy of the cut. Minimizing latency between the time the 3D topological data is captured and when the updated transformation matrix 2 is sent to the robot is essential for maintaining real-time performance. For instance, if the system operates at 30 registrations per second (rps) and the latency is 50VIS-002-W01 milliseconds (ms), then the robot will get data on its new position 50 ms after the bone being tracked was there. Additionally, there may be additional latency introduced by the robot’s control system (e.g., the robot needs to calculate an optimal path, engage motors, accelerate, and finally reach the intended point). If, for example, it takes the robot 20 ms to perform these tasks, the total lag would be 70 ms. Thus, it is essential to reduce both sources of lag in a real-time tracking system to ensure that the robot can respond in real time to anatomical movements.

[0326] In some embodiments, a seventh step 388 in the method 370 is to determine whether tracking of the target anatomy and surgical instrument needs to continue. If the determination is affirmative, then the workflow cycles back to step 372 of acquiring and reconstructing topological scan data of the target anatomy. If the answer is no (e.g., the osteotomy has been completed), then the system may end the tracking (e.g., box 390).

[0327] By increasing the registration rate and reducing computational latency, the system can deliver accurate, real-time adjustments to the robot, ensuring that the saw blade remains in the correct plane even as the target anatomy moves. This approach maintains the precision of the osteotomy and helps prevent unintended deviations, improving surgical outcomes.

[0328] Anatomy Tracking and Robot Following with a Mounted Scanner

[0329] In previous examples, the 3D scanner 12 was positioned at a fixed location. However, mounting the scanner on the end effector of a robotic arm 34 (e.g., as shown by way of example in Fig. 4) allows the 3D scanner’s 12 position and field of view 68 to change dynamically as the robot 34 moves. This configuration offers significant advantages during surgical procedures where the anatomy or surgical site frequently shifts.

[0330] For instance, in a Total Knee Arthroplasty (TKA) procedure, the knee is often moved by the surgeon, transitioning from full extension to 90 degrees of flexion, and sometimes into full flexion. While a fixed 3D scanner 12 can capture the surgical site in one position, maintaining a clear view becomes challenging as the anatomy moves. If the knee or other anatomical structures shift out of the 3D scanner 12’ s field of view 68, manual repositioning would be required, interrupting the workflow.VIS-002-W01

[0331] By mounting the 3D scanner 12 on the robot’s end effector, the system can continuously track bones like the femur or tibia and automatically adjust the scanner’s position to maintain a fixed perspective on the surgical site. As the knee moves through different angles, the robot dynamically updates the scanner's position, keeping the anatomy within the field of view. This automation reduces the need for manual intervention, saving time and effort. In some embodiments, the robot may be provided with one or more degrees of freedom to enable movement of the 3D scanner 12. By way of example, the robot shown in Fig. 4 includes seven (7) degrees of freedom, however robots with fewer degrees of freedom (e.g., 1, 2, 3, etc.) or more degrees of freedom (e.g., 8, 9, 10, etc.) may be used as may be needed or available.

[0332] In addition to maintaining a fixed view, the robot-mounted scanner synchronizes its movements with the anatomy in real-time. This ensures optimal scanner positioning throughout the procedure, allowing the surgeon to focus on the operation without worrying about viewpoint adjustments. Multiple viewpoints can be pre-programmed or customized for different stages of the surgery, with the surgeon easily switching between them via the user interface (UI), further enhancing procedural efficiency.

[0333] Complex Anatomy Tracking with Robot-Controlled Instrument (Oscillating Saw Example)

[0334] In this use case, both a 3D scanner 12 and an oscillating saw 28 are mounted to a robot arm. By way of example, the 3D scanner 12 tracks the anatomy while the robot positions the saw 28 to make precise cuts along a predefined cut plane, which may be modified intraoperatively based on the surgeon’s real-time observations. In some embodiments, if the 3D scanner 12 position changes as the robot moves, it may be necessary to apply one or more transformations to ensure that the anatomical data, 3D scanner 12, robot base, and saw 28 remain aligned in the same coordinate space. In some embodiments, the common coordinate system may be referenced to the patient, the 3D scanner 12, the robot base, or to another object.

[0335] By way of example only, Fig. 67A-B depicts a block diagram illustrating a method 390 of complex anatomy tracking with robot-controlled instrument in the context of a TKA, where the robot assists with cutting the femur.VIS-002-W01

[0336] In some embodiments, a first step 392 of the example method 390 of complex anatomy tracking with robot-controlled instrument is to mount the scanner 12 and the saw or saw blade 28 to a robot end effector 28, which by way of example may be an articulating arm having multiple degrees of freedom. In some embodiments, a next step 393 in the method 390 is to move (or initiate the robot to move) the robot arm to a known position in which the scanner 12 faces the robot base. In some embodiments, a next step 394 in the method 390 is to use the scanner 12 to acquire a dataset comprising structured light image data of the robot base (“robot base image dataset”). In some embodiments, a next step 395 in the method 390 is to upload a dataset comprising the robot base CAD geometry to the CAAT system 10 (“robot base CAD dataset”). In some embodiments, a next step 396 in the method 390 is to register the robot base image dataset to the uploaded robot base CAD dataset. In some embodiments, a next step 397 in the method 390 is to compute a transformation matrix T(B / S) (transformation matrix of the robot base dataset to the common coordinate system) at this fixed position to relate the robot base to the common coordinate system. In some embodiments, a next step 398 in the method 390 may be to track or continuously monitor robot position sensors and update the transformation matrix T(B / S) in real time.

[0337] By way of example, the above steps establish a robot base position in the common coordinate system. As previously mentioned, all other objects (3D scanner 12, instrument (e.g., saw) 28, femur 274, etc.) are transformed into the common coordinate system to ensure alignment. By way of example, the robot's base remains fixed, and in some embodiments a point the common coordinate system may be referenced to the base.

[0338] In some embodiments, the following steps of the method 390 of complex anatomy tracking with robot-controlled instrument may be followed to relate the position of the saw 28 or saw blade on the robot end effector to the common coordinate system. In some embodiments, this may be accomplished by expressing the position of the robot end effector in real time as a transformation matrix T(SB / S) (transformation matrix of the saw blade dataset to the common coordinate system). By way of example, the position of the robot's end effector may be known in real-time through the robot control system. This transformation matrix T(SB / S) is used to relate the end effector’s (and therefore the sawblade’s) position to the common coordinate system.VIS-002-W01

[0339] In some embodiments, a next step 399 in the method 390 is to use the 3D scanner 12 to acquire a structured light image dataset of the saw blade (“saw data set”) if the saw blade is positioned within the camera’s field of view or a saw blade calibration tool if the saw blade is not within the camera’s field of view. In some embodiments, a next step 400 in the method 390 is to upload a saw blade or saw blade calibration tool CAD geometry dataset to CAAT system 10. In some embodiments, a next step 401 in the method 390 is to register the saw dataset to the uploaded CAD geometry dataset. In some embodiments, a next step 402 in the method 390 is to compute a transformation matrix T(SB / S) (transformation matrix of the saw dataset to the scanner space) at the fixed position. In some embodiments, a next step 404 in the method 390 is to track or continuously monitor robot position sensors and update the transformation matrix T(SB / S) in real time.

[0340] In some embodiments, the following steps of the method 390 of complex anatomy tracking with robot-controlled instrument may be followed to register and track the femur position in real-time within the common coordinate system. In some embodiments, the CAAT system 10 may accomplish this by producing the transformation matrix T(F / S) (transformation matrix of the femur to the common coordinate system).

[0341] In some embodiments, a next step 406 in the method 390 is to manually move the robot to position where the scanner faces the femur. In some embodiments, a next step 408 in the method 390 is to use the 3D scanner 12 to acquire a structured light image dataset of the femur (“femur dataset”), which will include a subset data representative of femoral cartilage. In some embodiments, a next step 410 in the method 390 is to determine whether a reference dataset (e.g., preoperative CT scan) of the femur is available. If yes, then in some embodiments, a next step 412 in the method 390 is to upload a segmented reference dataset of the femur to the CAAT system 10. In some embodiments, a next step 414 in the method 390 is to register the femur / femoral cartilage scan dataset to the reference (e.g., CT) dataset, for example using one or more methods described herein. If there is no available reference dataset, then in some embodiments, a next step 416 in the method 390 is to create a femur / femur cartilage model dataset. In some embodiments, a next step 418 in the method 390 is to register the created model dataset to the femur / femoral cartilage scan dataset, for example using one or more methods described herein. In some embodiments, a next step 420 in the method 390 is to compute aVIS-002-W01 transformation matrix T(F / S) to transform this registration into the common coordinate system. In some embodiments, a next step 422 in the method 390 is to continuously repeat the previous steps (e.g., scan femur / femoral cartilage, register the scan dataset to the reference dataset, and recompute the transformation matrix T(F / S) in real time to track or monitor the position of the femur in the common coordinate system.

[0342] In some embodiments, the registrations described above may be performed in any order, since the goal of registering each of the robot, target anatomy (e.g., femur), surgical instrument (e.g., saw blade) and / or scanner into a common coordinate system is the same regardless of which occurs first. For example, in some embodiments, it may be advantageous to register the patient anatomy first, using the patient anatomy as the origin of the common coordinate system to minimize the need to re-register the patient anatomy in the event of movement.

[0343] In some embodiments, the following steps of the method 390 of complex anatomy tracking with robot-controlled instrument may be followed to register the surgical plan (e.g., cut plane) into the common coordinate space and perform the surgical cut. By way of example, the cut plane is initially determined during preoperative planning and adjusted during surgery based on patient-specific anatomy and knee kinematics. In some embodiments, the cut plane may be defined relative to the femur.

[0344] In some embodiments, a next step 424 in the method 390 is to load a preoperative CT dataset and / or femur image data into a surgical planning tool and determine proper cut planes in the common coordinate system. In some embodiments, a next step 426 in the method 390 is to continuously convert robot base, scanner, and saw blade into the common coordinate system by applying the correct respective transformation matrices and / or their inverses as needed. In some embodiments, a next step 428 in the method 390 is to continuously update the robot’s position to validate the position of the robot with respect to the anatomy, for example to ensure the saw blade remains in the cut plane at all times even with movement of the femur. In some embodiments, a next step 429 in the method 390 is to manually power and move the saw blade within the cut plane as needed to execute the cut as planned.VIS-002-W01

[0345] By way of example, all of these transformation matrices are computed in real-time, and the data is continuously fed into the robot's control system, to maintain the saw’s alignment within the predefined cut plane (for example) while adjusting its position based on real-time anatomical movement. This allows the surgeon to follow the operative plan, for example to manipulate the saw within the plane while the robot ensures that the blade remains within the precise boundaries of the cut plane.

[0346] The linear algebra described previously and in this section is crucial for handling these complex tracking scenarios. It is important to note that the same principles of transforming anatomy and objects into a common coordinate space apply to all the previous use cases described but the equations were not explicitly given. Whether it’s tracking a single object like the femur, maintaining a stable field of view, or synchronizing a scanner and robot, the underlying process of sequentially applying transformation matrices remains a key part of the system. These transformations ensure that all components of the system — whether anatomy, surgical instruments, or the robot — are correctly aligned within the SCS, enabling precise, realtime interactions.

[0347] By way of example, Fig. 68 is a block diagram depicting an example novel TKA workflow 430 with pre-operative imaging using the CAAT system 10 of the present disclosure. By way of example, the workflow 430 may be divided into several portions, including a preoperative portion 432, OR setup portion 434, initial registration portion 436, surgical planning portion 438, Bone resection portion 440, and implantation portion 442.

[0348] Regarding the pre-operative portion 432, prior to surgery, detailed imaging 444 of the target anatomy is often acquired using CT (Computed Tomography) or MRI (Magnetic Resonance Imaging). In some embodiments, these scans may be segmented to create a 3D model of the relevant structures, such as the femur and tibia in TKA. These 3D models are essential for pre-operative planning 446 of the procedure and aligning the patient's real-time anatomy with the surgical plan. Important anatomical planes and landmarks are also labeled within the model to assist the surgeon during the operation.VIS-002-W01

[0349] For this workflow, we assume a patient-specific CT scan has been segmented to provide a detailed 3D model of the femur and tibia, though the imageless mode can also be utilized if necessary.

[0350] By way of example, the OR setup portion 434 may include patient positioning 448, 3D scanner setup 450, robot setup 452, and instrument setup 454. Before surgery begins, the CAAT system 10 requires calibration. In some embodiments, scanner setup 450 comprises one or more 3D scanners 12 being positioned to ensure full visibility of the surgical field. Calibration ensures that the CAAT system 10 aligns correctly with the surgical setup, accounting for distances, camera angles, and the positioning of all equipment.

[0351] Robotic equipment requires additional steps for calibration to ensure that the robot functions correctly. The robot setup 452 process includes aligning the robot's physical movements with the virtual model of the surgery to maintain accuracy and ensure that the robot performs as expected during the procedure.

[0352] In some embodiments, patient positioning 448 occurs after setup and calibration. By way of example, in a TKA procedure the patient is positioned on the operating table, typically in a supine position with the knee exposed for TKA procedures, as shown by way of example in Fig. 4. The surgical site is prepped and draped in a sterile fashion.

[0353] By way of example, the initial registration portion 436 may include surgical exposure 456, scanning 458, bone to cartilage registration 460, hip center calculation 462 via tracking, and ankle center calculation 464 using a pointer. In some embodiments, upon completion of the OR setup 434 portion, surgical exposure 456 may occur, with an incision made to access the knee joint and prepare it for the robotic system to perform the procedure.

[0354] In some embodiments, femoral registration may occur as described above, by using the 3D scanner to generate a 3D point cloud model registered to the SCS. This real-time anatomical model is maintained within the Surgical Coordinate System (SCS), enabling accurate tracking of the patient’s anatomy throughout the procedure. In some embodiments, the CAAT system 10 may perform bone to cartilage registration 460, including misalignment correction as described below.VIS-002-W01

[0355] Following patient registration, the surgeon may identify key anatomical landmarks necessary for establishing the mechanical axis of the leg, for example including the hip center 462 (e.g. identified by rotating the leg while the CAAT system 10 calculates the center of rotation of the femoral head as described above), and / or ankle center 464 (e.g., identified using the optical tracking system on the medial and lateral malleoli (ankle bones) to calculate the center of the ankle joint). These landmarks, or others, or measured curvatures, for example, are crucial for ensuring proper implant alignment. The mechanical axis is one common methodology, but other alignment methods, like kinematic alignment, may use different anatomical landmarks depending on the procedure.

[0356] At this stage, a virtual surgical plan 438 is developed to guide the robotic system and surgeon throughout the procedure. If pre-operative planning was done, it serves as an initial basis but must be verified and adjusted in real time as the surgery progresses.

[0357] By way of example, the virtual surgical plan 438 may involve the implant selection 466, initial implant positioning 468, gap balancing via tracking 470, positioning adjustments 472, and cut plane definitions 474. Regarding implant selection 466, in some embodiments the CAAT system 10 may assist the surgeon in selecting the optimal implant size based on the patient’s anatomy. Proper implant sizing ensures both the long-term success of the surgery and post-operative joint function. Regarding implant positioning, in some embodiments the CAAT system 10 may help the surgeon determine the ideal implant position to achieve accurate joint alignment. The goal is to ensure proper leg alignment and balance between the medial and lateral sides of the knee. The alignment is measured in terms of varus (inward tilt) and valgus (outward tilt), with the system assisting the surgeon in making appropriate adjustments. In some embodiments, gap balancing 470 ensures equal tension in the soft tissues (primarily ligaments) around the knee joint throughout its range of motion. By way of example the CAAT system 10 dynamically tracks the relative positions of the bones and calculates the space between them to guide the surgeon in balancing the medial and lateral gaps during the procedure, as described above. At this time, any positioning adjustments 472 should be made. Regarding defining the cut plane 474, in some embodiments based on the selected implant size and position, the system defines the necessary osteotomies (bone cuts). These cuts must be precise to ensure proper implant placement.VIS-002-W01

[0358] In some embodiments, during the bone resection 440 portion, the CAAT system 10 assists the surgeon in making precise cuts according to the surgical plan. The system continuously tracks the patient's anatomy and provides feedback, ensuring the cuts are executed accurately. After each bone cut, the system verifies the accuracy of the cuts, implant alignment, and gap balancing. If deviations are detected, the system alerts the surgeon, allowing for realtime adjustments to be made. By way of example, this cut and verify process occurs for each osteotomy cut, including anterior 476, posterior 478, distal 480, posterior chamfer 482, anterior chamfer 484, and / or tibia 486.

[0359] In some embodiments, after bone cuts are verified, the workflow 430 proceeds with the implantation 442 portion. In some embodiments, trial implants 488 are placed to assess the fit, alignment, and joint movement. If necessary, adjustments are made before the final implant 490 is inserted. In some embodiments, the surgeon then performs a final assessment (e.g., including range of motion tests 492) to ensure proper alignment and stability of the new joint. Once confirmed, the surgical site is closed 494, and the patient is prepared for recovery.

[0360] By way of example, Fig. 69 is a flowchart depicting an example novel TKA workflow 630 using the CAAT system 10 of the present disclosure without pre-operative imaging. By way of example, the workflow 630 may be divided into several portions, including a pre-operative portion 632, OR setup portion 634, initial registration portion 636, surgical planning portion 638, Bone resection portion 640, and implantation portion 642.

[0361] Regarding the pre-operative portion 632, prior to surgery, detailed imaging 444 of the target anatomy is often acquired using CT (Computed Tomography) or MRI (Magnetic Resonance Imaging). In some embodiments, these scans may be segmented to create a 3D model of the relevant structures, such as the femur and tibia in TKA. These 3D models are essential for pre-operative planning 446 of the procedure and aligning the patient's real-time anatomy with the surgical plan. Important anatomical planes and landmarks are also labeled within the model to assist the surgeon during the operation.

[0362] Regarding the pre-operative portion 632, prior to surgery, a standard 3D model 646 of the bones may be used instead of patient-specific scans described in the previous workflow 430. Anatomical landmarks, such as the medial and lateral epicondyles of the femur, are identifiedVIS-002-W01 during surgery using the 3D scanner 12 as described above. In some embodiments, the distance between these landmarks is used to scale the standard model to match the patient’s unique anatomy, allowing the surgeon to visualize implant placement with appropriate proportions, even in the absence of patient-specific imaging. These 3D models 646 may be used for pre-operative planning 644 of the procedure and aligning the patient's real-time anatomy with the surgical plan. Important anatomical planes and landmarks are also labeled within the model to assist the surgeon during the operation.

[0363] By way of example, the OR setup portion 634 may include patient positioning 648, 3D scanner setup 650, robot setup 652, and instrument setup 654. In some embodiments, scanner setup 650 comprises one or more 3D scanners 12 being positioned to ensure full visibility of the surgical field and registration of the one or more 3D scanners 12 in a common coordinate system. In some embodiments, robot setup 652 may include registration of the robot base in the common coordinate system. In some embodiments, instrument setup may include registration of a surgical instrument within the common coordinate system, including but not limited to a saw blade (for example) attached as a robot end effector. By way of example, registration ensures that the various components of the CAAT system 10 align correctly to one another within the surgical setup, accounting for distances, camera angles, and the positioning of all equipment.

[0364] In some embodiments, patient positioning 648 occurs after setup and calibration. By way of example, in a TKA procedure the patient is positioned on the operating table, typically in a supine position with the knee exposed for TKA procedures, as shown by way of example in Fig. 4. The surgical site is prepped and draped in a sterile fashion.

[0365] By way of example, the initial registration portion 636 may include surgical exposure 656, scanning and segmentation 658, articular surface matching (ASM) registration 660, and initial size and position setup 662. In some embodiments, upon completion of the OR setup 634 portion, surgical exposure 656 may occur, with an incision made to access the knee joint and prepare it for the robotic system to perform the procedure.

[0366] In some embodiments, femoral registration may occur as described above, by using the 3D scanner 12 to generate a 3D point cloud model registered to the common coordinate system. This real-time anatomical model is maintained within the common coordinate system,VIS-002-W01 enabling accurate tracking of the patient’s anatomy throughout the procedure. By way of example, during ASM registration 660, the CAAT system 10 analyzes intraoperatively acquired topological data of the patient’s articular surfaces, and automatically compared them to digital models of available implants, assessing which implant design and size will best match the patient’s unique anatomy (e.g., initial size and position set 662).

[0367] At this stage, a virtual surgical plan 638 is developed to guide the robotic system and surgeon throughout the procedure. If pre-operative planning was done, it serves as an initial basis but must be verified and adjusted in real time as the surgery progresses.

[0368] By way of example, the virtual surgical plan 638 may involve gap balancing via tracking 670, positioning adjustments 672, and cut plane definitions 674. In some embodiments, gap balancing 670 ensures equal tension in the soft tissues (primarily ligaments) around the knee joint throughout its range of motion. By way of example the CAAT system 10 dynamically tracks the relative positions of the bones and calculates the space between them to guide the surgeon in balancing the medial and lateral gaps during the procedure, as described above. At this time, any positioning adjustments 672 should be made. Regarding defining the cut plane 674, in some embodiments based on the selected implant size and position, the system defines the necessary osteotomies (bone cuts). These cuts must be precise to ensure proper implant placement.

[0369] In some embodiments, during the bone resection 640 portion, the CAAT system 10 assists the surgeon in making precise cuts according to the surgical plan. The system continuously tracks the patient's anatomy and provides feedback, ensuring the cuts are executed accurately. After each bone cut, the system verifies the accuracy of the cuts, implant alignment, and gap balancing. If deviations are detected, the system alerts the surgeon, allowing for realtime adjustments to be made. By way of example, this cut and verify process occurs for each osteotomy cut, including anterior 676, posterior 678, distal 680, posterior chamfer 682, anterior chamfer 684, and / or tibia 686.

[0370] In some embodiments, after bone cuts are verified, the workflow 630 proceeds with the implantation 642 portion. In some embodiments, trial implants 688 are placed to assess the fit, alignment, and joint movement. If necessary, adjustments are made before the final implantVIS-002-W01690 is inserted. In some embodiments, the surgeon then performs a final assessment (e.g., including range of motion tests 692) to ensure proper alignment and stability of the new joint. Once confirmed, the surgical site is closed 694, and the patient is prepared for recovery.

[0371] Cartilage-to-Bone Misalignment

[0372] By way of example, Fig. 70 is a block diagram depicting a method 700 of addressing the registration mismatch between pre-operative CT data and intraoperative topological scans in image-based surgical navigation, particularly in procedures like TKA (including robot-assisted TKA). By focusing on acquiring intraoperative scan data that maximally exposes bone surfaces, this method ensures that registration accuracy is maintained without the need for optical trackers or invasive pointers, thus preserving the streamlined benefits of topological scanning. By way of example, the method 700 includes a first step 702 of performing a series of scans from multiple angles, capturing as much of the exposed bone surface as possible at each viewpoint and a second step 704 of stitching multiple point cloud datasets together to form a comprehensive 3D model in which each dataset is correctly oriented and positioned relative to the surgical field. The advantage of data taken with a 3D scanner with sufficient accuracy and high density of points is that it captures fine details that the registration algorithms can use to register accurately multiple scans. Each of these steps comprises several sub steps as described below.

[0373] In some embodiments, the first step 702 of performing a series of scans from multiple angles, capturing as much of the exposed bone surface as possible at each viewpoint may be comprised of multiple sub-steps, which are presented by way of example only in Fig. 71. In some embodiments, at the beginning of the surgical procedure, the surgeon initiates a series of scans from multiple angles (e.g., at least one or more of a medial view 714, superior view 16, lateral view 718, and / or inferior view 720), capturing as much of the exposed bone surface as possible at each viewpoint. Each scan aims to maximize the visibility of hard tissues at the edges of the articular cartilage, such as the femur and tibia, without necessitating additional incision expansion. By way of example, to achieve optimal exposure for a medial view 714, the surgeon may use a retractor or manual technique to move the surgical exposure toward the medial side, obtaining a scan focused on the femur from this angle. This position provides critical anatomical detail of the femur's medial surfaces. By way of example, to achieve optimal exposure for aVIS-002-W01 superior view 716, the surgeon may adjust the angle of exposure to capture a top-down perspective of the femur, scanning areas that may include both femoral and tibial surfaces if visible from this orientation. By way of example, to achieve optimal exposure for a lateral view 718, the surgeon may shift to the lateral side, gathering data focused on the lateral aspects of the femur using a shifted-exposure approach similar to that used for the medial side. This view further supports a comprehensive topological mapping of the femur's surface. By way of example, to achieve optimal exposure for an inferior view 720, the surgeon may adjust the retractor or exposure to scan areas that complement the data obtained from other angles.

[0374] In some embodiments, this process may be repeated for the tibia, ensuring detailed scans are acquired from different angles to form a well-rounded model of both bone surfaces. In some views, scans may incidentally capture both femoral and tibial surfaces, providing overlapping data that aids in reinforcing the accuracy of the final registration.

[0375] In some embodiments it may be possible that a single anterior scan captures sufficient bone surface from the tibia and sufficient hard tissue (very thin cartilage) at the femoral condyle edges of the articular cartilage to affect a registration of both bones from a single view.

[0376] In some embodiments, for optimal data collection, an optional sub step 706 of preparing the tissue for scanning to ensure a clear, unobstructed view of the bone surfaces to be scanned is critical. In situations where the bone is covered by biological tissues — such as periosteum, meniscus, ACL, or other structures — the surgeon may consider excising or repositioning these tissues to achieve the best possible scan. This preparation is particularly relevant in applications beyond Total Knee Arthroplasty (TKA), where tissue interference may vary widely across anatomical sites.

[0377] By way of example, to improve the fidelity of the scan, bone surfaces may be cleaned and dried 708 to remove any remaining biological matter that may obscure anatomical features. In some embodiments, if tissues like periosteum or other adherent biological matter cover the area of interest, excising these tissues 710 can expose a clearer bone surface, facilitating more accurate imaging. Furthermore, in cases where ligamentous structures, such as the meniscus or ACL, obscure the surface of interest, removal of the ligamentous structures 712 may furtherVIS-002-W01 enhance scan quality. These adjustments are at the surgeon’s discretion, allowing for tailored preparation based on the procedure’s specific requirements and the structures being imaged.

[0378] This tissue preparation step 706 is optional and intended to support image quality as needed. By preparing the area for scanning, the surgeon can ensure that the registered images align as accurately as possible, particularly when biological variations may affect the clarity of bone surfaces in scanner data.

[0379] In some embodiments, if it is necessary to obtain multiple scans, once they have been acquired from various angles each dataset is represented as a 3D point cloud capturing different views of the bone surfaces. In some embodiments, the next step 704 of the method 700 is stitching multiple point cloud datasets together to form a comprehensive 3D model in which each dataset is correctly oriented and positioned relative to the surgical field. To form a comprehensive model, these point clouds are stitched together through a registration process using for example RANSAC and ICP. Other registration methods are available, but RANSAC and ICP are given as a valid method. The basic process for stitching together different scanned datasets 704 includes one or more of the following sub steps, which are presented by way of example only in Fig. 72: (a) pre-processing 722, (b) segmentation 724, (c) coarse registration using RANSAC 726, (d) fine registration using ICP 728, (e) generation of the composite 3D model 730, (f) segmentation of the composite 3D model 732, (g) registration of bone datasets to generate transformation matrix 734, and (h) application of the transformation matrix for complete data integration 736. Each step in the process is discussed in further detail below.

[0380] Preprocessing 722

[0381] In some embodiments, each point cloud from the topological scans may undergo preprocessing 722 to reduce noise and standardize resolution across datasets, with filters applied to remove outliers and smooth the data.

[0382] Segmentation 724

[0383] In some embodiments, each scan may undergo segmentation to separate it into multiple targeted datasets, isolating specific anatomical structures relevant for analysis and registration. By way of example, for each bone, a new dataset is created that eliminatesVIS-002-W01 extraneous data, preserving only the hard bone and any overlying cartilage. This segmentation process refines the data, ensuring only the most pertinent anatomical structures are retained for accurate registration. Segmentation can be performed using several methods, each suited to different data conditions and surgical requirements.

[0384] By way of example, one method of segmentation 724 is manual segmentation 738. In this approach, an operator visually inspects each scan and manually selects the areas corresponding to the bone and cartilage, excluding all surrounding soft tissue. This method provides high accuracy and control, particularly valuable in complex anatomical regions. However, manual segmentation 738 can be time-intensive and may require expert oversight, making it best suited for cases where automated methods are less effective.

[0385] By way of example, another method of segmentation 724 involves the registration of a key unique sub-structure (example sulcus region of the femur) pre-defined as a region from the overall preoperative or stitched reference scan. This sub-structure is unique enough to typically only fit the entire optical scan in a single way. Notably, this registration could be used also as the coarse registration 726. Once this registration is successful the resulting transformation is applied to the larger overall structure of interest (example the femur with both condyles), also pre-selected. Once the larger structure is registered in the scan, the optical scan points within a certain neighborhood of the registered larger overall structure of interest are selected. This selects optical scan points that correspond to the anatomy of interest. This technique may also be used as a means of subtracting known objects form the scene, reducing its complexity for registration of the remaining components (example tibia). Further, such subtraction of objects from consideration for subsequent registrations in the scene, may occur in order of objects more easily registered (due to unique geometries) to those that are more difficult to register.

[0386] By way of example, another method of segmentation 724 involves the use of computer vision algorithms 740. Standard computer vision algorithms 740 can automate segmentation by detecting features and patterns in the scan data. Techniques such as edge detection can identify the boundaries of bone structures, while region growing can cluster similar regions based on density or texture, effectively isolating bone and cartilage. These algorithms are efficient forVIS-002-W01 well-defined structures but may require parameter tuning to handle noise or areas with less distinct boundaries.

[0387] Another example of a method of segmentation 724 is Al-based segmentation 742 using trained models. By way of example, an Al model, trained on a large dataset of similar anatomical structures, can automatically segment bone and cartilage surfaces in the scans. In some embodiments, the model applies learned patterns to distinguish between different tissue types, accurately isolating bone and cartilage with minimal manual intervention. This method is highly efficient and scalable, especially in high-volume or real-time settings, though it requires a robust training dataset and may need refinement to adapt to specific surgical environments. In some embodiments, for example in the case of a 3D scanner 12 using near-infrared (NIR) light, Al training becomes easier because the NIR light is the only effective illumination on the scene and may be viewed from a standard point of view, constraining the need for a large range of scenarios for Al train, such as would be required in the case of other illuminations coming from other directions (e.g., ambient operating room lighting). In some embodiments, standard views of the anatomy would enable faster training as less data is required.

[0388] By way of example, another method of segmentation 724 is to utilize tissue differentiation 744 capabilities of the 3D scanner 12, if applicable. In some embodiments, the 3D scanner 12 has inherent tissue differentiation capabilities that can distinguish between bone, cartilage, and surrounding tissues based on optical properties, for example as described in commonly-owned PCT / US25 / 24625, filed April 14, 2025 and entitled “System and Related Methods for the Characterization of Objects through Subsurface Scattering Across One or More Edges of Luminosity” (incorporated by reference). When available, these capabilities can streamline the segmentation process, allowing the 3D scanner 12 to pre-process the data and isolate relevant structures directly, minimizing the need for additional post-processing.

[0389] Coarse Registration Using RANSAC 726

[0390] In some embodiments, after preprocessing 722 and segmentation 724, coarse registration 726 may be performed on the segmented datasets to bring them into approximate alignment. By way of example, Random Sample Consensus (RANSAC) is one algorithm that can be used for this initial registration step. While other coarse registration methods areVIS-002-W01 available, RANSAC provides a robust approach for handling noise and outliers, establishing a reliable foundation for the finer registration that follows.

[0391] In some embodiments, a first sub step 746 in the RANSAC process for coarse registration 726 is random sampling, in which the RANSAC algorithm selects random subsets of points from each point cloud. These points are used to estimate potential transformations that align the point clouds. The random nature is such that subsequent executions may find different solutions. It is the goal that the different solutions all are sufficiently accurate to enable consistent ICP-found position solutions.

[0392] In some embodiments, a second sub step 748 in the RANSAC process for coarse registration 726 is model estimation, in which for each subset of points, the RANSAC algorithm calculates a transformation matrix that attempts to align the points between corresponding point clouds. Each transformation is evaluated based on its ability to bring the datasets into approximate alignment.

[0393] In some embodiments, a third sub step 750 in the RANSAC process for coarse registration 726 is inlier determination, in which for each estimated transformation, the RANSAC algorithm assesses the number of "inlier" points — points that fall within a specified tolerance when the transformation is applied. A higher inlier count indicates a better alignment between point clouds.

[0394] In some embodiments, a fourth step 752 in the RANSAC process for coarse registration 726 is best fit selection, in which after iterating over multiple random samples, the RANSAC algorithm selects the transformation with the highest inlier count as the best fit for coarse alignment. This transformation is then applied to the point clouds to achieve an approximate alignment.

[0395] By way of example, this coarse alignment 726 positions the segmented datasets in a similar orientation, ensuring that they are close enough for precise refinement in the next stage. By using a RANSAC algorithm, the process is resistant to noise and outliers, which is crucial when working with real-world anatomical data.

[0396] Fine Registration Using ICP 728VIS-002-W01

[0397] Following initial alignment using the RANSAC process 726 described above, an Iterative Closest Point (ICP) algorithm 728 may be applied to refine the registration by further minimizing the distance between corresponding points in overlapping regions of the point clouds. ICP performs fine adjustments through rigid transformations, specifically translations and rotations, to ensure a precise fit between the datasets.

[0398] In some embodiments, a first sub step 754 in the ICP process for fine registration 728 is correspondence matching. For example, in each iteration, ICP identifies corresponding points between the overlapping regions of the aligned point clouds. These correspondences are usually based on proximity, with each point in one dataset matched to the nearest point in the other dataset. Unlike RANSAC, ICP does not have a random component, gaining the same solution given a coarse registration from RANSAC. The goal is that the same ICP solution is attained given the entire range of randomly-seeded RANSAC solutions, achieved for the given point clouds.

[0399] In some embodiments, a second sub step 756 in the ICP process for fine registration 728 is transformation calculation. For example, based on the correspondences, ICP calculates a transformation matrix that includes optimal translations and rotations to minimize the distance between the matched points. This transformation aligns the datasets more closely in 3D space.

[0400] In some embodiments, a third sub step 758 in the ICP process for fine registration 728 is fitness evaluation. For example, after applying the transformation, the algorithm evaluates the alignment using a fitness function that measures the mean squared error (or another distance metric) between corresponding points. This fitness value quantifies the alignment quality, with lower values indicating closer alignment.

[0401] In some embodiments, a fourth sub step 760 in the ICP process for fine registration 728 is convergence check. For example, ICP iterates through steps of correspondence matching, transformation calculation, and fitness evaluation until the fitness function achieves a specified tolerance or improvement between iterations falls below a defined threshold. This iterative process ensures that the alignment error is minimized and that the datasets are finely aligned.VIS-002-W01

[0402] In some embodiments, a fifth sub step 762 in the TCP process for fine registration 728 is resulting fine alignment. For example, once convergence is reached, the final transformation matrix is applied to the datasets, achieving a highly accurate alignment. This refined registration aligns the point clouds within tight tolerances, ensuring precision suitable for detailed anatomical analysis and surgical navigation.

[0403] By iteratively minimizing alignment error, ICP produces accurate downstream processing and analysis based on the fully registered datasets.

[0404] Generation of the Composite 3D Model 730

[0405] In some embodiments, once the point clouds are fully aligned through RANSAC and ICP, they are merged 764 to form a single, cohesive composite 3D model. This model provides a unified, high-resolution view of the bone surfaces, capturing data from all scanned angles to ensure comprehensive anatomical coverage. For clarity in further analysis and registration, the following naming conventions will be used herein for these composite 3D model datasets (e.g., from topological scans) and CT datasets (e.g., from pre-operative imaging).

[0406] As used herein, “ScompositeFemur” refers to the composite 3D model dataset representing the entire femur, including bone and cartilage, created by merging all femoral point clouds.

[0407] As used herein, “ScompositeTibia” refers to the composite 3D model dataset representing the entire tibia, including bone and cartilage, created by merging all tibial point clouds.

[0408] As used herein, “CTFemur” refers to the CT dataset representing the femoral bone from the pre-operative CT scan.

[0409] As used herein “CT-Tibia” refers to the CT dataset representing the tibial bone from the pre-operative CT scan.

[0410] Thus, after alignment, the individual point clouds are combined to create single 3D datasets ScompositeFemur and ScompositeTibia. This merging process 764 consolidates overlapping regions, maximizing spatial resolution by incorporating details from each scan angle. To ensure anatomical continuity, surface reconstruction techniques may optionally be applied across the composite point clouds. Techniques such as Poisson reconstruction or Delaunay triangulationVIS-002-W01 interpolate across any sparse or incomplete regions within the composite model. This step fills gaps and smooths transitions, yielding a continuous representation that reflects the contours and textures of the femur and tibia.

[0411] While ScompositeFemur and ScompositeTibia provide comprehensive datasets that incorporate both bone and cartilage surfaces, they cannot be used directly for registration with the CT datasets. Since the CT data (e.g., CTpemur and CTubia) represents only the bone structures, additional processing is required to separate the bone from the cartilage in the composite datasets. To do this, an additional segmentation step 732 is required for each composite dataset.

[0412] Segmentation of Composite 3D Model 732

[0413] To prepare the composite 3D models for accurate registration with pre-operative CT datasets, an additional segmentation step 732 is required to isolate the bone surfaces from the cartilage in each composite dataset. Since ScompositeFemur and ScompositeTibia contain both bone and cartilage, this segmentation process ensures that only the bone surfaces are retained, matching the structure in the CTpemur and CTribia datasets.

[0414] By way of example, the segmentation process 732 involves the following sub steps. For clarity, only the femoral dataset, CTpemur will be discussed, but the method applies equally to the tibia or any other bone in an orthopedic procedure.

[0415] Identification of Bone Regions 768

[0416] In some embodiments, a first sub step 768 in the segmentation of the composite 3D model 732 is identifying the bone regions within ScompositeFemur, for example using manual and / or automated techniques. In some embodiments, this step may involve visual inspection, computer vision algorithms, and / or tissue differentiation depending upon the scanner’s capabilities. In some embodiments, algorithms such as edge detection and region growing (and / or sub-structure registration described above) may be used to assist in distinguishing denser bone structures from the surrounding cartilage. These techniques are especially valuable for semi-automated segmentation in high-resolution point clouds, where bone and cartilage boundaries may vary. In some embodiments, for added precision, Al-based models trained on labeled anatomical data can be applied to segment bone and cartilage surfaces. These models are trained to distinguish boneVIS-002-W01 from cartilage based on specific patterns and density variations, providing high accuracy with minimal manual intervention. Al-based segmentation helps ensure consistency across datasets and reduces the need for manual adjustments.

[0417] Isolation of Bone Data 770

[0418] In some embodiments, once the bone regions within ScompositeFemur are identified, a second sub step 770 in the segmentation of the composite 3D model 732 is isolating the data to create a dataset specific to the femoral bone surface. This process removes cartilage and other non-bone data from ScompositeFemur, resulting in a dataset focused solely on the exposed bone. The isolated dataset is labeled SFemurBone, representing only the femoral bone surface. This dataset is now prepared for alignment with the corresponding CT dataset, CTpemur .

[0419] Isolation of Cartilage Data 772

[0420] In some embodiments, based on the data excluded from ScompositeFemur when creating SFemurBone, a third sub step 772 in the segmentation of the composite 3D model 732 is the isolation of cartilage data for example by creating an additional data set which only includes the cartilage on the surface of the femur. This additional dataset is labeled SFemurCartiiage. This dataset may be useful for the surgeon or system to identify the disease state if rheumatoid arthritis is present, or the patient’s likely gait, as well as possible treatment options.

[0421] Registration of Bone Datasets to Generate Transformation Matrix 734

[0422] In some embodiments, following segmentation 732, the isolated bone dataset SFemurBone is registered 734 to the pre-operative CTpemur dataset to generate a transformation matrix. This matrix enables the accurate alignment of CT data within the scanner’s coordinate space, ensuring consistency between pre-operative imaging and intraoperative scans, as outlined in previous disclosures. Although only the femur is provided as an example here, this process is applied equally to the tibia or any other relevant bone in orthopedic procedures.

[0423] In some embodiments, the registration process 734 includes initial coarse alignment 774, fine alignment using ICP 776, and transformation matrix generation 778. By way of example, an initial alignment 774 is performed using RANSAC (as described above) to bringVIS-002-W01SpemurBone and CTpemur into approximate correspondence. Following RANSAC, TCP 776 is applied to refine the alignment (as described previously), adjusting until an acceptable tolerance is achieved. Upon completion of fine registration, a transformation matrix 778 is generated, accurately co-locating CTpemur within a common coordinate system. By way of example, this transformation matrix provides a stable basis for aligning pre-operative CT data within the intraoperative coordinate system, for precise navigation and accurate image-based guidance in the surgical workflow.

[0424] Application of the Transformation Matrix for Complete Data Integration 736

[0425] By using the generated transformation matrix, all related datasets — both pre-operative CT and intraoperative composite datasets — can be aligned within a common coordinate system. This matrix ensures that each dataset is correctly oriented and positioned relative to the surgical field.

[0426] In some embodiments, if cartilage were not present on the bone, the registration process could be simplified by aligning the entire composite dataset (e.g., SremurBone) directly to the CT dataset (e.g., CTremur ). However, because cartilage adds a variable thickness over the bone surface, directly registering the composite dataset to the CT dataset would introduce an alignment error. This error arises from the inherent surface mismatch between the exposed cartilage layer and the underlying bone structure captured in the CT scan.

[0427] The approach disclosed herein circumvents this problem by first segmenting and registering the bone-only datasets, excluding cartilage. With SremurBone accurately registered to CTpemur the transformation matrix allows all subsequent registrations of the femur dataset, including cartilage, to proceed. This matrix enables the ScompositeFemur dataset, which includes cartilage, to align directly with future intraoperative scans of the femur that also capture cartilage surfaces.

[0428] By applying the same transformation matrix, both the CT dataset and the femur datasets containing cartilage can be moved consistently into a common coordinate system. Once the composite femur is in place, subsequent scans with cartilage can be registered to this model, using all available visible stable tissues that match the composite femur scan. This is an exampleVIS-002-W01 where any unchanged portions in common with both optical scans are used to bridge stages of modification of anatomy. This can even include some soft tissues, which may be selected as inliers mathematically in the registration process itself, especially if such features are statistically significant in the point clouds being registered. This approach ensures an anatomically accurate alignment between the CT and intraoperative scans, effectively addressing the surface mismatch introduced by cartilage thickness and providing a cohesive solution for image-based surgical navigation.

[0429] Figs 73-74 are example block diagrams of computer-implemented electronic devices 500, 550 that may be used to implement the systems and methods described in this document, as either a client or as a server or plurality of servers. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 550 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, and other similar computing devices. In this example, computing device 500 may represent stationary computer or hub 12 and / or other computing systems referenced in this disclosure. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations described and / or claimed in this document.

[0430] Referring to Fig. 73, computing device 500 includes a processor 502, memory 504, a storage device 506, a high-speed interface 508 connecting to memory 504 and high-speed expansion ports 510, and a low-speed interface 512 connecting to low-speed bus 514 and storage device 506. Each of the components 502, 504, 506, 508, 510, and 512, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 502 can process instructions for execution within the computing device 500, including instructions stored in the memory 504 or on the storage device 506 to display graphical information for a graphic user interface (GUI) on an external input / output device, such as display 516 coupled to high-speed interface 508. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices 500 may be connected, withVIS-002-W01 each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

[0431] The memory 504 stores information within the computing device 500. By way of example only, the memory 504 may be a volatile memory unit, non-volatile memory unit, or another form of computer-readable medium, such as a magnetic or optical disk (for example).

[0432] The storage device 506 is capable of providing mass storage for the computing device 500. In one implementation, the storage device 506 may be or contain a non-transitory computer-readable medium (e.g., any and all computer-readable media except transitory, propagating signals), such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in an information carrier. The computer program product may also contain instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 504, the storage device 506, or memory on processor 502.

[0433] The high-speed interface 508 manages bandwidth-intensive operations for the computing device 500, while the low-speed interface 512 manages lower bandwidth-intensive operations. Such allocation of functions is by way of example only. In one implementation, the high-speed interface 508 is coupled to memory 504, display 516 (e.g., through a graphics processor or accelerator), and to high-speed expansion ports 510, which may accept various expansion cards (not shown). In the implementation, low-speed interface 512 is coupled to storage device 506 and low-speed expansion port 514. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be coupled to one or more input / output devices, such as a keyboard 518, a printer 520, a scanner 522, or a networking device such as a switch or router 524, e.g., through a network adapter.

[0434] The computing device 500 may be implemented in a number of different forms. For example, it may be implemented as a standard server, or multiple times in a group of such servers. It may also be implemented as part of a rack server system. In addition, it may beVIS-002-W01 implemented in a personal computer such as a laptop computer. Alternatively, components from computing device 500 may be combined with other components in a mobile device, such as device 550 (Fig. 74). Each of such devices may contain one or more of computing device 500, 550, and an entire system may be made up of multiple computing devices 500, 550 communicating with each other.

[0435] Referring to Fig. 74, computing device 550 includes a processor 552, memory 554, an input / output device such as a display 556, a communication interface 558, and a transceiver 560, among other components. The device 550 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 550, 552, 554, 556, 558, and 560, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

[0436] The processor 552 can execute instructions within the computing device 550, including instructions stored in the memory 554. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. Additionally, the processor may be implemented using any of a number of architectures. For example, the processor 552 may be a CISC (Complex Instruction Set Computers) processor, a RISC (Reduced Instruction Set Computer) processor, or a MISC (Minimal Instruction Set Computer) processor. The processor may provide, for example, for coordination of the other components of the device 550, such as control of user interfaces, applications run by device 550, and wireless communication by device 550.

[0437] The processor 552 may communicate with a user through control interface 562 and display interface 564 coupled to a display 556. The display 556 may be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface 564 may comprise appropriate circuitry for driving the display 556 to present graphical and other information to a user. The control interface 562 may receive commands from a user and convert them for submission to the processor 552. In addition, an external interface 566 may be provided in communication with processor 552, so as to enable near area communication of device 550 with other devices. External interface 566 may provide, for example, for wired communicationin some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

[0438] The memory 554 stores information within the computing device 550. The memory 554 can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Expansion memory 568 may also be provided and connected to device 550 through expansion interface 570, which may include, for example, a SIMM (Single In Line Memory Module) card interface. Such expansion memory 568 may provide extra storage space for device 550 or may also store applications or other information for device 550. Specifically, expansion memory 568 may include instructions to carry out or supplement the processes described above and may include secure information also. Thus, for example, expansion memory 568 may be provided as a security module for device 550 and may be programmed with instructions that permit secure use of device 550. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner. In some embodiments, and by way of example only, the memory 554 may also include one or more of a log of scanner use, calibration information, hardware and software versions, date of assembly, hardware configuration, date last calibrated, service history, and the like.

[0439] The memory may include, for example, flash memory and / or NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, cause performance of one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 554, expansion memory 568, or memory on processor 552 that may be received, for example, over transceiver 560 or external interface 566.

[0440] Device 550 may communicate wirelessly through communication interface 558, which may include digital signal processing circuitry where necessary. Communication interface 558 may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, through radio-frequencyVIS-002-W01 transceiver 560. In addition, short-range communication may occur, such as using a Bluetooth, WiFi, or other such transceiver (not shown). In addition, GPS (Global Positioning System) receiver module 572 may provide additional navigation- and location-related wireless data to device 550, which may be used as appropriate by applications running on device 550.

[0441] Device 550 may also communicate audibly using for example audio codec 574, which may receive spoken information from a user and convert it to usable digital information. Audio codec 574 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device 550. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on device 550.

[0442] The computing device 550 may be implemented in a number of different forms, some of which are shown in the figure. For example, it may be implemented as a cellular telephone. It may also be implemented as part of a smart-phone, personal digital assistant, or other similar mobile device.

[0443] Additionally computing device 500 or 550 can include Universal Serial Bus (USB) flash drives. The USB flash drives may store operating systems and other applications. The USB flash drives can include input / output components, such as a wireless transmitter or USB connector that may be inserted into a USB port of another computing device.

[0444] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0445] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in aVIS-002-W01 high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor.

[0446] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0447] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), peer-to-peer networks (having ad-hoc or static members), grid computing infrastructures, and the Internet.

[0448] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.VIS-002-W01

[0449] Additionally, by way of example, the computer system may include one or more specialized components such as GPU, field programmable gate array (FPGA), digital signal processor (DSP), neural processing unit (NPU), high speed camera interfaces, PCIe, etc.

[0450] Definitions

[0451] As used herein, a “3D Scanner” is a device that captures three-dimensional (3D) topological data of the surgical field, including anatomical structures and surgical tools, in real time. While there are multiple types of 3D scanners, this disclosure focuses on a proprietary 3D structured light scanner. However, any 3D topological scanner capable of collecting similar data may be used, as this disclosure emphasizes the use of the scanner and its data rather than the emphasizing the scanner itself.

[0452] As used herein, “continuous anatomical automatic tracking (CAAT)” is a system feature that continuously updates the position and orientation of anatomical structures and instruments by collecting data with a 3D scanner. In some embodiments, this process eliminates the need for physical optical markers, offering real-time tracking throughout the surgical procedure.

[0453] As used herein, “gap balancing” is a technique used to ensure equal spacing (gaps) between bones in both flexion and extension during surgeries like Total Knee Arthroplasty (TKA). Accurate gap balancing helps maintain proper joint stability and alignment.

[0454] As used herein, a “hub” is a computer responsible for processing raw data from the 3D scanner and automatically registering it to the Surgical Coordinate System.

[0455] As used herein, an “iterative closest point (ICP)” is an algorithm that refines the registration process by minimizing the distance between corresponding points in two datasets for precise alignment.

[0456] As used herein, an “osteotomy” is a surgical procedure that involves cutting and reshaping bones, commonly used in orthopedic surgeries to correct deformities or prepare the bone for implants.VIS-002-W01

[0457] As used herein, a “point cloud” is a set of points in 3D space, each representing a specific location on a scanned surface, used to generate a topological map of the surgical field. By way of example, the 3D scanner described herein provides a particularly dense point cloud, offering high-resolution data that enhances precision in surgical procedures, and it does it at >25 rps, with high accuracy, optionally in the near-infrared, and with an advantageous illumination coverage and depth of field in the pattern projection.

[0458] As used herein, a “random sample consensus (RANSAC)” is an algorithm used for initial coarse registration by selecting key points in the dataset and filtering out outliers.

[0459] As used herein, “real-time tracking” is defined as continuous registration of anatomical structures or tools, updating multiple times per second to reflect movements and changes in position within the surgical field. This ensures that data is captured and updated in real time without noticeable delays.

[0460] As used herein, a “robot-mounted scanner” is a 3D scanner attached to the robotic arm, or to the end effector of a robotic arm that allows the scanner to dynamically track anatomical structures as the robot moves during surgery.

[0461] As used herein, “sphere fitting” is a method used to estimate the center of rotation of a joint, such as the hip, by fitting a mathematical sphere to points collected during motion tracking of the femur.

[0462] As used herein, a “structured light pattern” is a method of projecting a predefined light pattern (such as grids, stripes, or dots) onto surfaces in order to capture 3D topological data.

[0463] As used herein, a “user interface (UI)” is a visual interface that provides real-time visualizations and guides the surgeon through key steps in the procedure.

[0464] As used herein, “varus angles” and “valgus angles” are the angles that represent the inward (varus) or outward (valgus) tilting of bones, often measured during surgeries to assess and correct misalignment in joints like the knee.

[0465] As used herein, the term "coupled" is defined as connected, although not necessarily directly, and not necessarily mechanically. The use of the word "a" or "an" when used inVIS-002-W01 conjunction with the term "comprising" in the claims and / or the specification may mean "one," but it is also consistent with the meaning of "one or more" or "at least one." The term "about" means, in general, the stated value plus or minus 5%. The use of the term "or" in the claims is used to mean "and / or" unless explicitly indicated to refer to alternatives only or the alternative are mutually exclusive, although the disclosure supports a definition that refers to only alternatives and "and / or."

[0466] As used herein, the terms "comprise" (and any form of comprise, such as "comprises" and "comprising"), "have" (and any form of have, such as "has" and "having"), "include" (and any form of include, such as "includes" and "including") and "contain" (and any form of contain, such as "contains" and "containing") are open-ended linking verbs. As a result, a method or device that "comprises," "has," "includes" or "contains" one or more steps or elements, possesses those one or more steps or elements, but is not limited to possessing only those one or more elements. Likewise, a step of a method or an element of a device that "comprises," "has," "includes" or "contains" one or more features, possesses those one or more features, but is not limited to possessing only those one or more features. Furthermore, a device or structure that is configured in a certain way is configured in at least that way but may also be configured in ways that are not listed.

[0467] All patents and publications mentioned in this specification are indicative of the levels of those skilled in the art to which the disclosure pertains. It is to be understood that while a certain form of the disclosure is illustrated, it is not to be limited to the specific form or arrangement herein described and shown. It will be apparent to those skilled in the art that various changes may be made without departing from the scope of the disclosure and the disclosure is not to be considered limited to what is shown and described in the specification and any drawings / figures included herein.

[0468] One skilled in the art will readily appreciate that the present disclosure is well adapted to carry out the objectives and obtain the ends and advantages mentioned, as well as those inherent therein. The embodiments, methods, procedures and techniques described herein are presently representative of the preferred embodiments, are intended to be exemplary and are not intended as limitations on the scope. Changes therein and other uses will occur to those skilledVIS-002-W01 in the art which are encompassed within the spirit of the disclosure and are defined by the scope of the appended claims. Although the disclosure has been described in connection with specific preferred embodiments, it should be understood that the disclosure as claimed should not be unduly limited to such specific embodiments. Indeed, various modifications of the described modes for carrying out the disclosure which are obvious to those skilled in the art are intended to be within the scope of the following claims.

Claims

1. VIS-002-W01CLAIMSWhat is claimed is:

1. A system for continuous registration and tracking of objects in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source; a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation of a topological surface of a physical object using a first dataset, the first dataset derived from one or more images captured by the image capture device using infrared light or near infrared light without optical trackers or applied fiducials; register the three-dimensional data representation to a second dataset such that the three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; and track the physical object relative to the common coordinate system without said optical trackers or applied fiducials.

2. The system of claim 1, wherein the light source is a linear light source.

3. The system of claim 1, wherein the three-dimensional data representation is a point cloud.

4. The system of claim 1, wherein the captured images are structured light patterned images that encode the surface of the physical object.

5. The system of claim 1, wherein the three-dimensional structured light scanner further includes a heat reduction assembly.VIS-002-W016. The system of claim 1, wherein the physical object is an anatomical structure in a surgical procedure.

7. The system of claim 1, wherein the three-dimensional data representation of the topological surface of the physical object is constructed by: decoding light patterns contained in the one or more captured images by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images based on the decoded light patterns; and combining the triangulated three-dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the physical object in physical space.

8. The system of claim 7, wherein the constructed three-dimensional data representation of the surface topology is comprised of at least 200,000 coordinates.

9. The system of claim 1, wherein the one or more images are captured at a rate of at least 20 Hertz.

10. The system of claim 1, wherein the digital three-dimensional data representation of the physical object is constructed within a processor located on the structured light three- dimensional scanner.

11. The system of claim 1, wherein the three-dimensional data representation of the physical object comprises a first three-dimensional data representation of a first physical object, and the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: construct a second digital three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of surface topology of the second physical object captured by the image capture device using infrared light and without the use of optical trackers or applied fiducials;VIS-002-W01 register the second three-dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned with one another and the with the first three-dimensional data representation and second dataset within the common coordinate system without the use of optical trackers or applied fiducials; continuously track the second physical object relative to the common coordinate system without the use of optical trackers or applied fiducials.

12. The system of claim 11, wherein the first physical object is a first anatomical structure and the second physical object is a second anatomical structure.

13. The system of claim 11, wherein the first physical object is an anatomical structure and the second physical object is a surgical instrument.

14. The system of claim 1, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

15. The system of claim 1, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked three-dimensional data representation of the physical object.

16. The system of claim 1, wherein tracking the physical object relative to the common coordinate system comprises continuously updating the physical object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

17. A system for continuous registration and tracking of objects in in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source; a liquid crystal matrix; andItOVIS-002-W01 an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a first three-dimensional data representation of a topological surface of a first physical object using a first dataset, the first dataset derived from one or more images of the topological surface of the first physical object captured by the image capture device using infrared light or near infrared light without optical trackers or applied fiducials; register the first three-dimensional data representation to a second dataset such that the first three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; track the first physical object relative to the common coordinate system without said optical trackers or applied fiducials; construct a second three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of the topological surface of the second physical object captured by the image capture device using infrared light without optical trackers or applied fiducials; register the second three-dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned within the common coordinate system, the registration occurring without said optical trackers or applied fiducials; and track the second physical object relative to the common coordinate system without said optical trackers or applied fiducials.

18. The system of claim 17, wherein the light source is a linear light source.

19. The system of claim 17, wherein the first three-dimensional data representation is a point cloud.VIS-002-W0120. The system of claim 17, wherein the captured images of the topological surface of the first physical object are structured light patterned images that encode the surface of the first physical object.

21. The system of claim 17, wherein the three-dimensional structured light scanner further includes a heat reduction assembly.

22. The system of claim 17, wherein the first physical object is an anatomical structure in a surgical procedure.

23. The system of claim 17, wherein the second physical object is a surgical instrument.

24. The system of claim 17, wherein the first three-dimensional data representation of the topological surface of the first physical object is constructed by: decoding light patterns contained in the one or more captured images of the topological surface of the first physical object by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images of the topological surface of the first physical object based on the decoded light patterns; and combining the triangulated three-dimensional coordinates of each pixel into the first three-dimensional point cloud representation of the surface of the first physical object in physical space.

25. The system of claim 24, wherein the first three-dimensional data representation of the surface topology is comprised of at least 200,000 coordinates.

26. The system of claim 17, wherein the one or more images of the topological surface of the first physical object and the one or more images of the topological surface of the second physical object are captured at a rate of at least 20 Hertz.

27. The system of claim 17, wherein the first three-dimensional data representation of the first physical object is constructed within a processor located on the structured light three-dimensional scanner.VIS-002-W0128. The system of claim 17, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

29. The system of claim 17, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked first three-dimensional data representation of the first physical object and the tracked second three-dimensional data representation of the second physical object.

30. The system of claim 17, wherein tracking the first physical object relative to the common coordinate system comprises continuously updating the first physical object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

31. A system for continuous registration and tracking of objects in in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source; a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a three-dimensional data representation of a topological surface of a physical object using a first dataset, the first dataset derived from one or more images captured by the image capture device without optical trackers or applied fiducials; register the three-dimensional data representation to a second dataset such that the three-dimensional data representation and the second dataset are alignedVIS-002-W01 within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; and track the physical object relative to the common coordinate system without said optical trackers or applied fiducials; wherein the digital three-dimensional data representation of the physical object is constructed within a processor located on the structured light three- dimensional scanner.

32. The system of claim 31, wherein the light source is a linear light source.

33. The system of claim 31, wherein the three-dimensional data representation is a point cloud.

34. The system of claim 31, wherein the captured images are structured light patterned images that encode the surface of the physical object.

35. The system of claim 31, wherein the three-dimensional structured light scanner further includes a heat reduction assembly.

36. The system of claim 31 , wherein the physical object is an anatomical structure in a surgical procedure.

37. The system of claim 31, wherein the three-dimensional data representation of the topological surface of the physical object is constructed by: decoding light patterns contained in the one or more captured images by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images based on the decoded light patterns; and combining the triangulated three-dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the physical object in physical space.

38. The system of claim 37, wherein constructed three-dimensional data representation of theVIS-002-W01 surface topology is comprised of at least 200,000 coordinates.

39. The system of claim 31, wherein the one or more images are captured at a rate of at least 20 Hertz.

40. The system of claim 31, wherein the one or more images are captured by the image capture device using infrared light or near infrared light.

41. The system of claim 31, wherein the three-dimensional data representation of the physical object comprises a first three-dimensional data representation of a first physical object, and the computer readable media includes a further set of instructions, that, when executed by a processor, cause the computer system to: construct a second digital three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of surface topology of the second physical object captured by the image capture device without the use of optical trackers or applied fiducials; register the second three-dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned with one another and the with the first three-dimensional data representation and second dataset within the common coordinate system without the use of optical trackers or applied fiducials; and continuously track the second physical object relative to the common coordinate system without the use of optical trackers or applied fiducials; wherein the second digital three-dimensional data representation of the second physical object is constructed within a processor located on the structured light three- dimensional scanner.

42. The system of claim 41, wherein the first physical object is a first anatomical structure and the second physical object is a second anatomical structure.

43. The system of claim 41, wherein the first physical object is an anatomical structure and the second physical object is a surgical instrument.

44. The system of claim 31, wherein the second dataset is derived from a preoperativeVIS-002-W01 computed tomography scan or magnetic resonance imaging scan.

45. The system of claim 31, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked three-dimensional data representation of the physical object.

46. The system of claim 31, wherein tracking the physical object relative to the common coordinate system comprises continuously updating the physical object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

47. A system for continuous registration and tracking of objects in in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source; a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a first three-dimensional data representation of a topological surface of a first physical object using a first dataset, the first dataset derived from one or more images of the topological surface of the first physical object captured by the image capture device using infrared light or near infrared light; register the first three-dimensional data representation to a second dataset such that the first three-dimensional data representation and the second dataset are aligned within a common coordinate system; track the first physical object relative to the common coordinate system;VIS-002-W01 construct a second three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of the topological surface of the second physical object captured by the image capture device using infrared light; register the second three-dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned within the common coordinate system; and track the second physical object relative to the common coordinate system; wherein the first digital three-dimensional point cloud representation of the first physical object and the second digital three-dimensional point cloud representation of the second physical object are constructed within a processor located on the 3D scanner.

48. The system of claim 47, wherein the light source is a linear light source.

49. The system of claim 47, wherein the first three-dimensional data representation is a point cloud.

50. The system of claim 47, wherein the captured images of the topological surface of the first physical object are structured light patterned images that encode the surface of the first physical object.

51. The system of claim 47, wherein the three-dimensional structured light scanner further includes a heat reduction assembly.

52. The system of claim 47, wherein the first physical object is an anatomical structure in a surgical procedure.

53. The system of claim 47, wherein the second physical object is a surgical instrument.

54. The system of claim 47, wherein the first three-dimensional data representation of the topological surface of the first physical object is constructed by: decoding light patterns contained in the one or more captured images of theVIS-002-W01 topological surface of the first physical object by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images of the topological surface of the first physical object based on the decoded light patterns; and combining the triangulated three-dimensional coordinates of each pixel into the first three-dimensional point cloud representation of the surface of the first physical object in physical space.

55. The system of claim 54, wherein the first three-dimensional data representation of the surface topology is comprised of at least 200,000 coordinates.

56. The system of claim 47, wherein the one or more images of the topological surface of the first physical object and the one or more images of the topological surface of the second physical object are captured at a rate of at least 20 Hertz.

57. The system of claim 47, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

58. The system of claim 47, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked first three-dimensional data representation of the first physical object and the tracked second three-dimensional data representation of the second physical object.

59. The system of claim 47, wherein tracking the first physical object relative to the common coordinate system comprises continuously updating the first physical object’s position relative to the common coordinate system by repeatedly acquiring surface data, registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

60. The system of claim 47, wherein the one or more images of the topological surface of the first physical object are captured by the image capture device without the use of opticalVIS-002-W01 trackers or fiducials.

61. The system of claim 47, wherein one or more images of the topological surface of the second physical object are captured by the image capture device without the use of optical trackers or fiducials.

62. The system of claim 47, wherein the first generated three-dimensional point cloud representation is registered to the second dataset within the common coordinate system without the use of optical trackers or fiducials.

63. The system of claim 47, wherein the first physical object is tracked within the common coordinate system without the use of optical trackers or fiducials.

64. The system of claim 47, wherein the second generated three-dimensional point cloud representation is registered to the fourth dataset within the common coordinate system without the use of optical trackers or fiducials.

65. The system of claim 47, wherein the second physical object is tracked within the common coordinate system without the use of optical trackers or fiducials.

66. A system for continuous registration and tracking of objects in in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source; a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a first three-dimensional data representation of a topological surface of a first physical object using a first dataset, the first dataset derived from one or more images of the topological surface of the first physical object captured by the image capture device using infrared light or near infrared light withoutVIS-002-W01 optical trackers or applied fiducials; register the first three-dimensional data representation to a second dataset such that the first three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; track the first physical object relative to the common coordinate system without said optical trackers or applied fiducials; construct a second three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of the topological surface of the second physical object captured by the image capture device using infrared light; register the second three-dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned within the common coordinate system; and track the second physical object relative to the common coordinate system; wherein the first physical object is an anatomical structure; and wherein the second physical object is a surgical instrument.

67. The system of claim 66, wherein the light source is a linear light source.

68. The system of claim 66, wherein the first three-dimensional data representation is a point cloud.

69. The system of claim 66, wherein the captured images of the topological surface of the first physical object are structured light patterned images that encode the surface of the first physical object.

70. The system of claim 66, wherein the three-dimensional structured light scanner further includes a heat reduction assembly.

71. The system of claim 66, wherein the first three-dimensional data representation of the topological surface of the first physical object is constructed by:VIS-002-W01 decoding light patterns contained in the one or more captured images of the topological surface of the first physical object by analyzing brightness history to create camera pixel correspondence to light angle projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images of the topological surface of the first physical object based on the decoded light patterns; and combining the triangulated three-dimensional coordinates of each pixel into the first three-dimensional point cloud representation of the surface of the first physical object in physical space.

72. The system of claim 71, wherein the first three-dimensional data representation of the surface topology is comprised of at least 200,000 coordinates.

73. The system of claim 66, wherein the one or more images of the topological surface of the first physical object and the one or more images of the topological surface of the second physical object are captured at a rate of at least 20 Hertz.

74. The system of claim 66, wherein the first three-dimensional data representation of the first physical object is constructed within a processor located on the structured light three-dimensional scanner.

75. The system of claim 66, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

76. The system of claim 66, wherein: the computer system further comprises a user interface and a display device, and the computer readable media includes a further set of instructions that, when executed by the processor, cause the computer system to display, on a display device, the tracked first three-dimensional data representation of the first physical object and the tracked second three-dimensional data representation of the second physical object.

77. The system of claim 66, wherein tracking the first physical object relative to the common coordinate system comprises continuously updating the first physical object’s position relative to the common coordinate system by repeatedly acquiring surface data,VIS-002-W01 registering the repeatedly acquired surface data in the common coordinate system, and updating transformation matrices.

78. A method for continuous registration and tracking of objects in in an operating room environment, comprising: constructing a three-dimensional data representation of a topological surface of a physical object using a first dataset, the first dataset derived from one or more images captured by the image capture device using infrared light or near infrared light without optical trackers or applied fiducials; registering the three-dimensional data representation to a second dataset such that the three-dimensional data representation and the second dataset are aligned within a common coordinate system, the registration occurring without said optical trackers or applied fiducials; and tracking the physical object relative to the common coordinate system without said optical trackers or applied fiducials.

79. The method of claim 78, wherein the light source is a linear light source.

80. The method of claim 78, wherein the three-dimensional data representation is a point cloud.

81. The method of claim 78, wherein the captured images are structured light patterned images that encode the surface of the physical object.

82. The method of claim 78, wherein the three-dimensional structured light scanner further includes a heat reduction assembly.

83. The method of claim 78, wherein the physical object is an anatomical structure in a surgical procedure.

84. The method of claim 78, wherein constructing the three-dimensional data representation of the topological surface of the physical object comprises: decoding light patterns contained in the one or more captured images by analyzing brightness history to create camera pixel correspondence to light angleVIS-002-W01 projection; using triangulation to calculate a three-dimensional coordinate for each pixel in the one or more captured images based on the decoded light patterns; and combining the triangulated three-dimensional coordinates of each pixel into the three-dimensional point cloud representation of the surface of the physical object in physical space.

85. The method of claim 84, wherein the constructed three-dimensional data representation of the surface topology is comprised of at least 200,000 coordinates.

86. The method of claim 78, wherein the one or more images are captured at a rate of at least 20 Hertz.

87. The method of claim 78, wherein the digital three-dimensional data representation of the physical object is constructed within a processor located on the structured light three- dimensional scanner.

88. The method of claim 78, wherein the three-dimensional data representation of the physical object comprises a first three-dimensional data representation of a first physical object, the method further comprising: constructing a second digital three-dimensional data representation of a topological surface of a second physical object using a third dataset, the third dataset derived from one or more images of surface topology of the second physical object captured by the image capture device using infrared light and without the use of optical trackers or applied fiducials; registering the second three-dimensional data representation to a fourth dataset such that the second three-dimensional data representation and the fourth dataset are aligned with one another and the with the first three-dimensional data representation and second dataset within the common coordinate system without the use of optical trackers or applied fiducials; and tracking the second physical object relative to the common coordinate system without the use of optical trackers or applied fiducials.VIS-002-W0189. The method of claim 88, wherein the first physical object is a first anatomical structure and the second physical object is a second anatomical structure.

90. The method of claim 88, wherein the first physical object is an anatomical structure and the second physical object is a surgical instrument.

91. The method of claim 78, wherein the second dataset is derived from a preoperative computed tomography scan or magnetic resonance imaging scan.

92. The method of claim 78, further comprising: displaying, on a display device, the tracked three-dimensional data representation of the physical object.

93. The method of claim 78, wherein tracking the physical object relative to the common coordinate system comprises continuously updating the physical object’s position relative to the common coordinate system by: repeatedly acquiring surface data; registering the repeatedly acquired surface data in the common coordinate system; and updating transformation matrices.

94. A system for continuous registration and tracking of objects in in an operating room environment, comprising: a structured light three-dimensional scanner, including: a light source; a liquid crystal matrix; and an image capture device; a computer system including a memory and at least one processor; and computer readable media embodied in a non-transitory storage medium comprising a set of instructions that, when executed by the one or more processors, cause the computer system to: construct a unique three-dimensional data representation of a topological surface of each of a plurality of physical objects using a first dataset, the firstVIS-002-W01 dataset derived from one or more images captured by the image capture device using infrared light or near infrared light without optical trackers or applied fiducials; register each unique three-dimensional data representation to a corresponding unique reference dataset for each one of said plurality of physical objects such that each unique three-dimensional data representation is aligned with the corresponding unique reference dataset within a common coordinate system, each registration occurring without said optical trackers or applied fiducials; and track each of said plurality of physical objects relative to the common coordinate system without said optical trackers or applied fiducials.

Images