Markerless Tracking with Spectral Imaging Cameras

Abstract

Markerless tracking using a spectral imaging camera comprises imaging an area containing one or more objects using at least one spectral imaging camera, the imaging comprising acquiring intensity signals at one or more selected wavelengths or wavelength ranges that correlate with a selected material of at least one of the one or more objects, using the acquired signals to determine a position of each of the at least one object in space, and tracking the position of one object in space over time.

Description

[Technical Field]

[0001] In various clinical applications, it may be beneficial to relay information about the current and / or past position and orientation (spatial location or pose) of an object to a processor, i.e., to track the object. [Background technology]

[0002] Two-dimensional (2D) and three-dimensional (3D) representations of the tracked object are often displayed. As a non-limiting example, a tracking system may display preoperatively captured data, such as a reconstructed computed tomography (CT) view of a patient's preoperative anatomy. Tracking systems that display views of anatomy and surgical instruments are sometimes referred to as navigation systems. As a non-limiting example, in orthopedic joint reconstruction, an arthritic joint is replaced with a prosthetic joint. As a non-limiting example, in knee replacement surgery, a series of bone resections are performed to place the implant. Summary of the Invention

[0003] The shortcomings of the prior art are overcome and additional advantages are provided by providing a computer-implemented method. The method uses at least one spectral imaging camera to image an area containing one or more objects. The imaging includes acquiring intensity signals at one or more selected wavelengths or wavelength ranges that correlate with a selected material of at least one of the one or more objects. The method also uses the acquired signals to determine the position of each of the at least one object in space and to track the position of one object in space over time.

[0004] Further aspects of the present disclosure relate to systems and computer program products configured to perform the methods described above and herein. This summary is not intended to describe every aspect, every implementation, and / or every embodiment of the present disclosure. Additional functionality and advantages are realized through the concepts described herein.

[0005] The aspects described herein are particularly pointed out and distinctly claimed as examples in the claims at the conclusion of the specification. These and other objects, features, and advantages of the present disclosure will become apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0006] [Figure 1] Figure 1 shows an example of tracking an anatomical structure via a tracking array. [Figure 2] Figure 2 shows an example of an array articulation with a series of joints. [Figure 3] Figure 3 shows an example of a point registration approach. [Figure 4] Figure 4 shows an example of a camera with depth sensing capabilities. [Figure 5] Figure 5 shows an example where cartilage obscures the view of the underlying bone surface. [Figure 6] Figure 6 shows an example of hyperspectral imaging, which assigns three-dimensional spectral information to each pixel of a two-dimensional image. [Figure 7] Figure 7 shows examples of the unique reflectance characteristics for each of six different materials / objects. [Figure 8] FIG. 8 shows an example of the resections performed during a total knee replacement. [Figure 9] Figure 9 shows the anatomy of the knee joint. [Figure 10] FIG. 10 shows an example of an exposed knee joint and its anatomical features. [Figure 11] FIG. 11 shows an example of an exposed knee joint and its anatomical features. [Figure 12]FIG. 12 illustrates an exemplary process of markerless tracking with a spectral imaging camera according to embodiments described herein. [Figure 13] FIG. 13 illustrates an exemplary computer system for carrying out aspects described herein. DETAILED DESCRIPTION OF THE INVENTION

[0007] Aspects described herein present novel approaches, processes, and methods for tracking objects (e.g., anatomical structures and, optionally, other objects such as surgical instruments) that do not rely on the placement of known objects in a scene. Furthermore, these aspects reduce surgical preparation time and the number of surgical steps required to perform a navigated surgery, which is advantageous because in current systems, surgical preparation and registration time account for a large portion of the overall surgical time. Current / traditional approaches also suffer from additional drawbacks, including:

[0008] <Rigid array placement for navigation> For navigational procedures, tracking arrays are often rigidly attached to target anatomical structures, such as bone. In many current approaches, the array is rigidly secured with 3-millimeter (mm) bicortical bone pins, reinforced by a sleeve between the pins, typically 50 mm in length. These approaches require the array to be rigid relative to the target anatomical structures. This configuration may not provide array stability in all directions.

[0009] In Figure 1, tracking an object of interest requires that the localization camera have a line of sight to the tracking arrays 100, 110. The array's fiducials (102, 104, 106, 108, etc., for array 100) must also be oriented so that they are all visible to the camera. This requires a full range of articulation of the array relative to its mount.

[0010] Traditionally, this is accomplished through a series of joints that are tightened using a tool. Referring to FIG. 2, articulation / motion of the array 200 is provided by three articulation points 202, 204, and 206, each of which is adjusted by loosening a bolt, making the adjustment, and then tightening the bolt. Points 202 and 204 provide a rotatable joint, while point 206 allows a sliding member 208 to slide along 210 to move the array assembly toward or away from the anatomy. This setup results in a lack of rigidity in the mechanism until all joints are tightened. Therefore, this method requires some dexterity and two hands to orient and tighten, increasing time, stress, and cost.

[0011] The above examples describe the placement of arrays to optically track the position of an object. Other tracking methods may require the placement of fixed markers on the object of interest. As a non-limiting example, approaches exist that propose the fixed placement of beacons to use radar to track bones. In contrast, embodiments described herein provide a markerless tracking approach, i.e., a tracking approach that does not use or rely on markers, thereby eliminating the need to place tracking markers. As a non-limiting example, the placement of fixed markers increases surgical time, complexity, and cost. Bone pins are also invasive and are often inserted outside the incision.

[0012] <The position of the markers relative to the anatomical structures must be registered> Once the array is fixed, it is impossible to know with high precision where the markers were placed on the anatomy, so a transformation between the position of the array relative to the anatomy must be calculated. This is typically achieved through a process called point registration. Several methods can be used for this; ultrasound is one example. Referring to Figure 3, the most common method is to use a sharp instrument 302 that probes the cartilage at multiple locations, capturing a point cloud of points on the bone surface, as shown in Figure 3. Various mathematical calculations are then used to correlate the point cloud with a pre-operative model (e.g., a CT scan) or a generalized anatomical model, returning the pose of the object.

[0013] Since the aspects described herein propose tracking actual objects rather than markers, the user does not need to register the position of the tracking markers relative to the anatomical structures of interest; in other words, the approach described herein does not require a registration step.

[0014] <Track objects using depth-sensing cameras instead of markers> As a non-limiting example, a camera may have an integrated red, green, and blue wavelength (RGB) camera / sensor and an infrared sensor. Other variations include a structured light projector and camera receiver to project a known pattern onto a scene and calculate depth and surface information for objects within the scene. The example camera in Figure 4 shows an RGBD camera incorporating an RGB camera 402 and a depth sensor 404.

[0015] A limitation of such technology is that the fast feature detection algorithms that depth cameras rely on have difficulty correlating the collected data with preoperative data sets, especially when the surgical exposure is minimal (minimally invasive), where the observable surface area is limited and potentially occluded by other objects (cartilage, blood, surgical instruments, and other soft tissues). However, a perhaps even more significant limitation is that most preoperative imaging is based on x-rays (especially CT scans, which are a series of x-rays). Cartilage does not show up on x-rays. X-rays are most useful for imaging bone. When cameras such as depth cameras are tasked with correlating a scene with preoperative data, the scene includes cartilage. Very little bone is exposed. These cameras cannot "see through" the cartilage and other objects that make up the preoperative data sets derived from x-rays to see the bone surface. An example of this is shown in Figure 5, which shows a surgical incision where bone is partially exposed, but in region 302, at least part of the bone is occluded by cartilage.

[0016] Accordingly, aspects described herein address these and other shortcomings, including, for example:

[0017] Markerless tracking before altering anatomy According to some embodiments, spectral imaging (using a spectral imaging camera) is used to track objects of interest. Examples of spectral imaging that can be used include hyperspectral imaging (using one or more hyperspectral cameras) and multispectral imaging (using one or more multispectral imaging cameras). Hyperspectral imaging, like other spectral imaging, collects and processes information from the entire electromagnetic spectrum. The goal of such imaging is to obtain the spectrum of each pixel in an image for the purpose of finding objects, identifying materials, or detecting processes. While the human eye only perceives colors in three bands of the visible light spectrum, primarily long-wavelength (red), mid-wavelength (green), and short-wavelength (blue), hyperspectral imaging perceives a broader range of wavelengths beyond the visible light range. Certain objects leave unique "fingerprints" in the electromagnetic spectrum. These "fingerprints," known as spectral signatures, allow the identification of the materials comprising the scanned object. As a non-limiting example, the parameter may be the relative absorbance of light at wavelength t.

[0018] In other words, a hyperspectral camera can spatially scan, or detect, various materials regardless of relative obstructions. Simply put, every object emits or absorbs radiation at wavelengths specific to that material (e.g., physical material properties). A hyperspectral camera can detect this radiation within a certain distance from the object. For example, cartilage reflects and absorbs radiation at wavelengths around 500–600 nanometers (nm), sufficient for tissue identification. While the naked eye can only see cartilage, a hyperspectral imaging camera can detect bone surfaces and other structures that are obscured by cartilage. Therefore, a hyperspectral camera can "see" bone surfaces for the purpose of determining their location, despite their obscuration by cartilage. Bone surfaces are invisible to conventional cameras and depth-sensing cameras. Given that bone surfaces are of preoperative and clinical interest, direct imaging of bone surfaces facilitates markerless tracking.

[0019] Because each organ has a unique spectral fingerprint, machine learning can be used to help identify tissues. Specifically, neural networks and other artificial intelligence models can be trained using training data sets that provide the reflection and absorption of various wavelengths in specific materials. Using AI models (such as neural networks) can allow for more generalized tissue identification.

[0020] According to Figure 6, hyperspectral imaging (HSI) works by assigning three-dimensional spectral information to each pixel of a traditional two-dimensional digital image. The spectral information includes the pixel's wavelength-specific reflectance intensity. The result is a three-dimensional data cube with two spatial dimensions (x, y) and a third (spectral) dimension (λ).

[0021] As mentioned above, different materials have different electromagnetic properties or characteristics, meaning they emit, reflect, and absorb wavelengths from a light source differently. These differences can be used to distinguish tissues based on their spectral characteristics. Figure 7 shows examples of such unique reflectance characteristics for each of six different materials / objects.

[0022] To track objects of interest, the process searches for / detects objects of interest (e.g., bones) in a scene based on their electromagnetic signature and then uses a mathematical algorithm to correlate the observed objects in the scene with the pre-operation dataset and return a pose (i.e., track the bone). Hyperspectral cameras are not required to track surgical incision sites. For example, it may prove beneficial to track additional incision areas, such as the proximal tibia and the proximal tibia. This may be necessary when detecting radiation emitted or absorbed by an object is difficult due to material that shields it, such as detecting bone through cartilage.

[0023] Markerless tracking after altering anatomy (especially performing robotic or navigational surgery) In general, but not exclusively, the goal of many surgical procedures is to alter the preoperative anatomy. Navigation surgery helps surgeons plan and execute such alterations to the anatomy. For example, in a robotic total knee arthroplasty, the surgeon may perform a series of resections (six example resections are shown in Figure 8) to facilitate implant placement. As the anatomy is surgically altered, the navigation system will track such alterations. The navigation system may need to account for such alterations and update the preoperative dataset accordingly so that the surgically altered anatomy observed from the hyperspectral imaging camera can be correlated with the corresponding preoperative dataset.

[0024] A hyperspectral camera may be able to see through skin and other anatomical structures well enough to see intact areas of bone. Therefore, a hyperspectral camera may not be necessary to view the surgical site or the surrounding area. For example, in the case of the tibia, one might track the shank rather than the proximal tibia as described above.

[0025] Importantly, when correlating with a preoperative model, the navigation system takes into account changes in the anatomy and updates the correlated model based on those changes.

[0026] Markerless tracking to visualize anatomical structures that are not occluded by cartilage but may be occluded by other materials such as soft tissue, ligaments, tendons, blood, and fat Some hyperspectral cameras may be limited in their ability to view bone surfaces obscured by cartilage. In certain embodiments, the camera can be positioned so that anatomical structures unobstructed by cartilage are within the camera's field of view. As a non-limiting example, consider the knee joint of FIG. 9 , which has anatomical structures including a femur, medial femoral epicondyle 904, patella 906, medial femoral condyle 908, medial tibial condyle 910, tibial tubercle 912, tibia 914, fibula 916, fibular neck 918, fibular head 920, proximal tibiofibular joint 922, lateral tibial condyle 924, lateral femoral condyle 926, and lateral femoral epicondyle 928. The hyperspectral camera for the femur (902) can be positioned to view one or more of the following anatomical structures: the lateral femoral condyle 926, the lateral tibial condyle 924, the medial femoral epicondyle 904, the medial femoral condyle 908, or a portion of the femur 902 other than the distal portion. Multiple cameras can be used to image different anatomical regions of the same bone. By way of non-limiting example, the hyperspectral camera for the tibia 914 can be positioned to view one or more of the following anatomical structures: the medial tibial condyle 910, the tibial tubercle 912, the shin (anterior surface of the tibia), the lateral tibial condyle 924, or a portion of the tibia 914 other than the proximal portion.

[0027] In cases where a hyperspectral camera does not have the properties to visualize the bone surface through cartilage, or where such properties are present but for some reason seeing through the cartilage is less desirable, embodiments can visualize the bone surface in specific areas that are not obstructed by cartilage but may be obstructed by other anatomical structures.

[0028] Referring to FIG. 10, a hyperspectral camera may not be able to see the bone surface through the cartilage 1002, or it may be less desirable to visualize through the cartilage 1002, but by way of non-limiting example, may be able to see the bone surface through the soft tissue of the lateral condyle / epicondyle 1004.

[0029] In the case of Figure 11, the area indicated by 1102 represents the exposed bone surface that is not covered by cartilage and can be tracked by the hyperspectral camera.

[0030] The aspects described herein may be useful in any navigation surgery and other applications.

[0031] In certain examples, hyperspectral / multispectral imaging techniques have been integrated into orthopedic surgical robots, specifically as localizing cameras. The hyperspectral camera can be mounted, for example, on a cart, a tripod, a fixture attached to the operating table, or the robot. In certain embodiments, multiple such cameras may be installed in various configurations within the operating room. The camera's function may include tracking objects of clinical interest. In one embodiment of the system, the location of the tracked object can be used to plan surgical procedures / motions, specifically the trajectory of a tool mounted on the robot.

[0032] In general, all objects emit or absorb radiation at wavelengths specific to the properties of that material; in other words, all materials have an electromagnetic signature. Some objects may not emit enough radiation to be detected by a spectral imaging camera. To detect such objects, an external light source (of different wavelengths of light) can be used to detect the diffraction / absorption of light in different materials. In some configurations, there is a light source or energy source that is placed near the object of interest and illuminates it.

[0033] Accordingly, FIG. 12 illustrates an exemplary process for markerless tracking with a spectral imaging camera, according to embodiments described herein. The process includes imaging 1202 an area containing one or more objects using at least one spectral imaging camera. The imaging may include, for example, acquiring intensity signals at one or more selected wavelengths or wavelength ranges that correlate with a selected material of at least one of the one or more objects. The process continues by using the resulting signals 1204 to determine the respective positions of the at least one object in space. This imaging 1202 and use 1204 may be repeated at different times, e.g., periodically or aperiodically, to track the position 1206 of an object in space over time. Examples of spectral imaging cameras include one or more hyperspectral imaging cameras for hyperspectral imaging of an area and / or one or more multispectral imaging cameras for multispectral imaging of an area.

[0034] In an example, the region at least partially includes a surgical scene, and the at least one object includes a patient anatomical structure. The patient anatomical structure may include, for example, bones or other selected anatomical structures. The process further includes correlating each determined position of the anatomical structure to a pre-acquired model of the anatomical structure or a modified version of the pre-acquired model. The pre-acquired model may include a pre-operative two-dimensional or three-dimensional anatomical model. In an embodiment, the process tracks changes in the anatomical structure during the surgical procedure, updates the pre-acquired model according to the tracked changes to provide a modified version of the pre-acquired model, and correlates the changed anatomical structure observed from the image to the corresponding modified version of the pre-acquired model.

[0035] Using 1204 the signals includes using at least one algorithm to correlate the anatomical structures to the pre-operative data set or a modified version of the pre-operative data set and return the position / pose of the anatomical structures. In an embodiment, this process tracks changes in the anatomical structures during the surgical procedure, updates the pre-operative data set according to the tracked changes to provide a modified version of the pre-operative data set, and correlates changed anatomical structures observed from the images to the corresponding modified version of the pre-operative data set.

[0036] Localization can be accomplished with or without the use of tracking fiducials or other markers on the object or within the area, by deploying arrays to optically track the object's location, and by using beacons and radar-based tracking.

[0037] In an embodiment, using (1204) includes applying an artificial intelligence (AI) model to identify the at least one object. The AI ​​model can be configured to identify the selected material using machine learning and at least one data set providing reflectance / absorbance of various wavelengths for various specific materials to train the AI ​​model.

[0038] One or more embodiments described herein may be incorporated into, executed by, and / or used by one or more computer systems, such as, for example, one or more systems that are or communicate with a camera system, a tracking system, and / or an orthopedic surgical robot. The processes described herein may be performed by one or more computer systems, either singly or collectively. As used herein, a computer system may also be referred to as a data processing device / system, a computing device / system / node, or simply a computer. A computer system may be based on one or more of a variety of system architectures and / or instruction set architectures.

[0039] FIG. 13 illustrates a computer system 1300 in communication with an external device 1312. The computer system 1300 includes one or more processors 1302, e.g., central processing units (CPUs). A processor may include functional components used to execute instructions, such as a functional component that fetches program instructions from a location such as a cache or main memory, a functional component that decodes program instructions, a functional component that executes program instructions, a functional component that accesses memory for instruction execution, and a functional component that writes the results of executed instructions. The processor 1302 may also include registers used by one or more functional components. The computer system 1300 also includes memory 1304, input / output (I / O) devices 1308, and I / O interfaces 1310, which may be coupled to the processor 1302 and each other via one or more buses and / or other connections. The bus connections may represent one or more of several types of bus structures, such as a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor bus or local bus using various bus architectures, etc. Examples of such architectures include, but are not limited to, Industry Standard Architecture (ISA), Micro Channel Architecture (MCA), Enhanced ISA (EISA), Video Electronics Standards Association (VESA) Local Bus, and Peripheral Component Interconnect (PCI).

[0040] The memory 1304 may be or include a main memory or system memory (e.g., random access memory), a storage device such as a hard drive, flash media, or optical media, and / or a cache memory used to execute program instructions. The memory 1304 may include a cache, such as a shared cache, which may be coupled to a local cache (e.g., an L1 cache, an L2 cache, etc.) of the processor(s) 1302. Additionally, the memory 1304 may be or include at least one computer program product having a set (e.g., at least one) of program modules, instructions, code, etc. configured, when executed by one or more processors, to perform the functions of the embodiments described herein.

[0041] The memory 1304 can store an operating system 1305 and other computer programs 1306, such as one or more computer programs / applications that execute to perform aspects described herein. In particular, the programs / applications can include computer-readable program instructions that can be configured to perform the functions of embodiments of the aspects described herein.

[0042] Examples of I / O devices 1308 include, but are not limited to, microphones, speakers, global positioning system (GPS) devices, RGB, IR, and / or spectral cameras, lights, accelerometers, gyroscopes, magnetometers, sensor devices configured to sense light, proximity, heart rate, body and / or ambient temperature, blood pressure, and / or skin resistance, enrollment probes, activity monitors, etc. As shown, the I / O devices can be incorporated into the computer system, although in some embodiments, the I / O devices may also be considered external devices (1312) coupled to the computer system via one or more I / O interfaces 1310.

[0043] The computer system 1300 can communicate with one or more external devices 1312 through one or more I / O interfaces 1310. Examples of external devices include a keyboard, pointing device, display, and / or other devices that allow a user to interact with the computer system 1300. Other examples of external devices include any device that allows the computer system 1300 to communicate with one or more other computing systems or peripheral devices, such as a printer. A network interface / adapter is an example of an I / O interface that allows the computer system 1300 to communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet), enabling communication with other computing devices or systems, storage devices, etc. Ethernet-based interfaces (e.g., Wi-Fi) and Bluetooth® adapters are just a few examples of currently available types of network adapters used in computer systems (BLUETOOTH® is a registered trademark of Bluetooth SIG, Inc., Kirkland, Washington, USA).

[0044] Communication between I / O interface 1310 and external device 1312 may occur via wired and / or wireless communication link 1311, such as an Ethernet-based wired or wireless connection. Examples of wireless connections include cellular, Wi-Fi, Bluetooth, proximity-based, short-range, and other types of wireless connections. More generally, communication link 1311 may be any suitable wireless and / or wired communication link for communicating data.

[0045] Specific external device 1312 may include one or more data storage devices capable of storing one or more programs, one or more computer-readable program instructions, and / or data. Computer system 1300 may include and / or be coupled to and in communication with removable / non-removable, volatile / non-volatile computer system storage media (e.g., as a device external to the computer system). For example, computer system 1300 may include and / or be coupled to non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"), magnetic disk drives that read from and write to removable, non-volatile magnetic disks (e.g., "floppy disks"), and / or optical disk drives that read from and write to removable, non-volatile optical disks, such as CD-ROMs, DVD-ROMs, and other optical media.

[0046] The computer system 1300 is operational with numerous other general-purpose or special-purpose computing system environments or configurations, and may take many different forms, well-known examples of which include, but are not limited to, personal computer (PC) systems, server computer systems such as messaging servers, thin clients, thick clients, workstations, laptops, handheld devices, mobile devices / computers such as smartphones, tablets, and wearable devices, multi-processor systems, microprocessor-based systems, telephony devices, network appliances (such as edge appliances), virtualization devices, storage controllers, set-top boxes, programmable consumer electronics devices, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0047] Aspects of the present invention are systems, methods, and / or computer program products, any of which may be configured to perform or facilitate the aspects described herein.

[0048] In some embodiments, aspects of the present invention may take the form of a computer program product, which may be embodied as a computer-readable medium. The computer-readable medium may be a tangible storage device / medium on which computer-readable program code / instructions are stored. Examples of computer-readable media include, but are not limited to, electronic, magnetic, optical, or semiconductor storage devices or systems, or combinations thereof. Exemplary embodiments of computer-readable media include hard drives or other mass storage devices, electrical connections with wires, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory such as EPROM or flash memory, fiber optics, portable computer disks / diskettes such as compact disc read-only memory (CD-ROM) or digital versatile disks (DVD), optical storage devices, magnetic storage devices, or any combination thereof. The computer-readable medium is readable by a processor, processing unit, etc., which can retrieve and execute data (e.g., instructions) from the medium. In certain examples, a computer program product is or includes one or more computer-readable media that contain / store computer-readable program code for providing and facilitating one or more aspects described herein.

[0049] As discussed above, program instructions contained in or stored on a computer-readable medium can be retrieved and executed by any of various suitable components, such as a processor, of a computer system to cause the computer system to operate and function in a particular manner. Such program instructions for performing operations to implement, accomplish, or facilitate aspects described herein can be written in any programming language or compiled from code written in any programming language. In some embodiments, such programming languages ​​include object-oriented and / or procedural programming languages, such as C, C++, C#, Java, etc.

[0050] The program code may include one or more program instructions acquired for execution by one or more processors. The computer program instructions may be provided, for example, to one or more processors of one or more computer systems to generate a machine that, when executed by the one or more processors, performs, accomplishes, or facilitates aspects of the invention, such as the operations or functions described in the flowcharts and / or block diagrams set forth herein. Thus, each block, or combination of blocks, of the flowchart diagrams and / or block diagrams shown and described herein may, in some embodiments, be implemented by computer program instructions.

[0051] Although various embodiments have been described above, these are merely examples.

[0052] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly dictates otherwise. Furthermore, it will be understood that the terms "comprises" and / or "comprises," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0053] The corresponding structure, material, acts, and equivalents (if any) of all means or step-plus-function elements in the following claims are intended to include any structure, material, or acts for performing the function in combination with other claimed elements as specifically claimed. The description of one or more embodiments has been presented for purposes of illustration and description and is not intended to be exhaustive or limited to the precise form disclosed. Many modifications and variations will be apparent to those skilled in the art. The present embodiments have been chosen and described to best explain various aspects and practical applications and to enable those skilled in the art to recognize various embodiments with various modifications suited to the particular use envisioned.

Claims

1. 1. A computer-implemented method comprising: imaging an area containing one or more objects using at least one spectral imaging camera, said imaging comprising obtaining intensity signals for selected one or more wavelengths or wavelength ranges that correlate with selected materials of at least one of the one or more objects; determining a position of each of said at least one object in space using the obtained signals; tracking the position of the at least one object in space over time; A method for providing the above.

2. said tracking comprising repeating said imaging and using one or more times at different time points; The method of claim 1.

3. The at least one spectral imaging camera comprises: (i) one or more hyperspectral imaging cameras for hyperspectral imaging of an area; and (ii) one or more multispectral imaging cameras for multispectral imaging of the area; At least one selected from the group consisting of: The method of claim 1.

4. the region comprises a surgical scene, and the at least one object comprises a patient anatomy, the patient anatomy comprising a bone or other selected anatomical structure; The method of claim 1.

5. correlating each determined position of the anatomical structure to a pre-obtained model or a modified version of a pre-obtained model of the anatomical structure. The method of claim 4.

6. the pre-obtained model comprises a pre-operative two-dimensional or three-dimensional model of the anatomical structure; The method of claim 5.

7. The method comprises: tracking changes in the anatomical structure during a surgical procedure and updating the pre-obtained model according to the tracked changes to provide a modified version of the pre-obtained model; correlating the altered anatomical structures observed from the imaging with a corresponding modified version of the previously obtained model; Further provided with The method of claim 6.

8. said using comprising using at least one algorithm to correlate said anatomical structure with the pre-operative dataset or a modified version of the pre-operative dataset and return a position / pose of the anatomical structure. The method of claim 4.

9. The method comprises: tracking changes in the anatomical structure during a surgical procedure and updating the pre-operative dataset according to the tracked changes to provide a modified version of the pre-operative dataset; correlating the altered anatomical structures observed from the imaging with a corresponding modified version of the pre-operative data set; Further provided with The method of claim 8.

10. the determination of location is performed without the use of or reliance on tracking of fiducials or other markers on the object or within the region, the placement of arrays to optically track the object's location, and beacon and radar-based tracking; The method of claim 1.

11. said using comprising applying an artificial intelligence (AI) model to identify at least one object; The AI ​​model is configured to identify the selected material based on training the AI ​​model using machine learning and at least one dataset providing reflection / absorption of various wavelengths for various specific materials. The method of claim 10.

12. 1. A computer system comprising: Memory and a processor in communication with the memory; Equipped with The computer system is configured to perform the following method: The method comprises: imaging an area containing one or more objects using at least one spectral imaging camera, said imaging comprising obtaining intensity signals for selected one or more wavelengths or wavelength ranges that correlate with selected materials of at least one of the one or more objects; determining a respective position of at least one object in space using the obtained signals; tracking the position of the at least one object in space over time; Equipped with Computer systems.

13. said tracking comprising one or more repetitions of said imaging and using at different time points; 13. The computer system of claim 12.

14. The at least one spectral imaging camera comprises: (i) one or more hyperspectral imaging cameras for hyperspectral imaging of an area; and (ii) one or more multispectral imaging cameras for multispectral imaging of an area; At least one selected from the group consisting of:

13. The computer system of claim 12.

15. the region comprises a surgical scene, and the at least one object comprises a patient anatomy, the patient anatomy comprising a bone or other selected anatomical structure; 13. The computer system of claim 12.

16. The method further comprises correlating each determined position of the anatomical structure to a pre-acquired model of the anatomical structure or a modified version of the pre-acquired model.

16. The computer system of claim 15.

17. the pre-acquired model comprises a pre-operative two-dimensional or three-dimensional model of the anatomical structure; 17. The computer system of claim 16.

18. The method comprises: tracking changes in anatomical structures during a surgical procedure and updating said pre-obtained model according to the tracked changes to provide a modified version of said pre-obtained model; correlating the altered anatomical structures observed from the imaging with a corresponding modified version of the previously obtained model; Further provided with 18. The computer system of claim 17.

19. said using comprising applying an artificial intelligence (AI) model to identify at least one object; The AI ​​model is configured to identify the selected material based on training the AI ​​model using machine learning and at least one dataset providing reflection / absorption of various wavelengths for various specific materials.

13. The computer system of claim 12.

20. 1. A computer program product comprising: a computer-readable storage medium readable by a processing circuit and storing instructions for performing a method that is executed by the processing circuit, said method comprising: imaging an area including one or more objects using at least one spectral imaging camera, said imaging comprising obtaining intensity signals for selected one or more wavelengths or wavelength ranges that correlate with selected materials of at least one of the one or more objects; determining a respective position of at least one object in space using the obtained signals; tracking the position of the at least one object in space over time; A computer program product comprising:

Images