Registration of time-intervalled x-ray images

Abstract

A method according to one embodiment of the present disclosure includes: receiving a first image of a patient's anatomical structure, the first image being generated and depicting a plurality of rigid units at a first time; receiving a second image of the patient's anatomical structure, the second image being generated and depicting the plurality of rigid units at a second time after the first time; determining a transformation from the first image to the second image for each of the plurality of rigid units to generate a set of transformations; calculating the homography of each transformation in the set of transformations to generate a set of homography; and using the set of homography to identify common portions of each transformation attributable to camera pose changes and individual portions of each transformation attributable to rigid unit pose changes.

Description

Technical Field

[0001] This technology involves surgical imaging and navigation as a whole, and more specifically, tracking anatomical elements before, during, and after surgery. Background Technology

[0002] Imaging can be used by medical providers for diagnostic and / or therapeutic purposes. Patient anatomy can change over time, especially after medical implants are placed within the patient's anatomy. Registration of one image with another allows for the identification and quantification of changes in anatomical location. Summary of the Invention

[0003] Exemplary aspects of this disclosure include:

[0004] A method comprising: receiving a first image of a patient's anatomy, the first image being generated at a first time and depicting a plurality of rigid units, each of the plurality of rigid units being movable relative to at least one other rigid unit; receiving a second image of the patient's anatomy, the second image being generated at a second time after the first time and depicting the plurality of rigid units; determining a transformation from the first image to the second image for each of the plurality of rigid units to generate a set of transformations; and using the set of transformations to identify common portions of each transformation attributable to camera pose changes and individual portions of each transformation attributable to rigid unit pose changes.

[0005] Any aspect of this article, which also includes: registering the second image with the first image based on the identified common parts of each transformation.

[0006] Any aspect of this paper, which also includes updating the preoperative model based on individual parts of each transformation.

[0007] Any aspect of this paper, which also includes updating the registration of one of the robot space or navigation space with the image space based on the common part of each transformation or the individual part of each transformation.

[0008] Any aspect of this paper, wherein each transformation is homography, and the set of transformations is a set of homography.

[0009] In any aspect of this paper, the identification step utilizes clustering to separate the transformations caused by camera pose changes in the set of transformations.

[0010] In any aspect of this paper, the registration step includes associating both the first and second images with a common vector space.

[0011] Any aspect of this article, wherein the first image is a preoperative image.

[0012] In any aspect of this article, at least one of the first and second images is an intraoperative image.

[0013] In any aspect of this paper, the computational transformation includes identifying at least four points on each of the plurality of rigid elements as depicted in the first image, and corresponding at least four points on each of the plurality of rigid elements as depicted in the second image.

[0014] Any aspect of this article, wherein the first and second images are two-dimensional.

[0015] In any aspect of this article, the first and second images are three-dimensional.

[0016] Any aspect of this article, wherein the plurality of rigid units include multiple vertebrae of the patient’s spine.

[0017] In any aspect of this article, the plurality of rigid units include at least one implant.

[0018] Any aspect of this paper, which also includes: quantifying the attitude change of at least one of the plurality of rigid elements from a first time to a second time.

[0019] A method for correlated images captured at different times, the method comprising: segmenting each of the plurality of rigid units in a first image captured at a first time and in a second image of the plurality of rigid units captured at a second time after the first time; calculating the homography of each of the plurality of rigid units to generate a set of homography, each homography relating the rigid unit as depicted in the first image to the rigid unit as depicted in the second image; arranging the set of homography into a homography cluster based on at least one characteristic; selecting a homography cluster based on at least one parameter; and projecting each of the plurality of rigid units as depicted in the second image onto the first image using the average value of the selected homography cluster to generate a projected image.

[0020] In any aspect of this article, the second time is at least one month after the first time.

[0021] In any aspect of this article, the second time is at least one year after the first time.

[0022] In any aspect of this article, at least one parameter is a profile.

[0023] In any aspect of this article, at least one of the plurality of rigid units is an implant.

[0024] Any aspect of this article, wherein the plurality of rigid units include multiple vertebrae of the patient’s spine.

[0025] Any aspect of this document, which also includes: measuring at least one of an angle or a distance corresponding to the attitude change of one of the plurality of rigid elements as reflected in the projected image.

[0026] Any aspect of this article, including: removing from the set of homography any homography affected by one or more of the compression fractures or osteophytes depicted in the second image rather than the first image.

[0027] In any aspect of this paper, calculating the homography of each of the plurality of rigid elements includes identifying the edge points of the vertebral endplate.

[0028] A system for comparing images, the system comprising: at least one processor; and a memory. The memory stores instructions for execution by the processor, which, when executed, cause the processor to: identify a plurality of units in a first image generated at a first time; identify the plurality of units in a second image generated at a second time after the first time; calculate the homography of each of the plurality of units using the first image and the second image to generate a set of homography; and determine, based on the set of homography,: a first pose change of one or more units of the plurality of units in the second image relative to the first image, the first pose change being attributable to an imaging device position change from the first image to the second image relative to the plurality of units; and a second pose change of at least one unit of the plurality of units in the second image relative to the first image, the second pose change being not attributable to the imaging device position change.

[0029] In any aspect of this document, the memory stores additional instructions for execution by the processor, which, when executed, further enable the processor to: register the second image with the first image based on the first pose change.

[0030] In any aspect of this paper, the memory stores additional instructions for execution by the processor, which, when executed, further enable the processor to update the preoperative model based on the second pose change.

[0031] In any aspect of this document, the memory stores additional instructions for execution by the processor, which, when executed, further cause the processor to: update the registration of either the robot space or the navigation space with the image space based on either a first pose change or a second pose change.

[0032] Details of one or more aspects of this disclosure are set forth in the following drawings and description. Other features, objects, and advantages of the technology described in this disclosure will be apparent from the specification, drawings, and claims.

[0033] The phrases “at least one,” “one or more,” and “and / or” are open-ended expressions that possess both connective and disjoint qualities in operation. For example, the expressions “at least one of A, B, and C,” “at least one of A, B, or C,” “one or more of A, B, and C,” “one or more of A, B, or C,” and “A, B, and / or C” all mean only A, only B, only C, A and B together, A and C together, B and C together, or A, B, and C together. When each of A, B, and C in the above expressions refers to an element such as X, Y, and Z, or such as X1-X… n Y1-Y m and Z1-Z o When referring to a class of elements, the phrase is intended to mean a single element selected from X, Y, and Z; a combination of elements selected from the same class (e.g., X1 and X2); and elements selected from two or more classes (e.g., Y1 and Z). o () combination.

[0034] The term "a / an" refers to one or more of the same entity. Therefore, the terms "a / an," "one or more," and "at least one" are used interchangeably herein. It should also be noted that the terms "comprising / including" and "having" are used interchangeably.

[0035] The foregoing is a simplified overview of this disclosure to provide an understanding of some aspects thereof. This summary is neither a broad nor an exhaustive overview of this disclosure and its various aspects, embodiments, and configurations. It is not intended to identify key or essential elements of this disclosure, nor to define its scope, but rather to present selected concepts in a simplified form as an introduction to the more detailed description presented below. It should be understood that other aspects, embodiments, and configurations of this disclosure may utilize one or more of the features set forth above or described in detail below, individually or in combination.

[0036] Many additional features and advantages of the invention will become apparent to those skilled in the art upon consideration of the embodiments described below. Attached Figure Description

[0037] The accompanying drawings are incorporated in and form part of this specification to illustrate several examples of this disclosure. These drawings, together with the description, explain the principles of this disclosure. The drawings illustrate only preferred and alternative examples of how to implement and use this disclosure, and these examples should not be construed as limiting this disclosure solely to the examples shown and described. Further features and advantages will become apparent from the following more detailed description of various aspects, embodiments, and configurations of this disclosure, as illustrated by the accompanying drawings referenced below.

[0038] Figure 1 It is a block diagram of a system according to at least one embodiment of the present disclosure;

[0039] Figure 2 It is a series of X-ray images of the patient's anatomical structures taken at different times;

[0040] Figure 3 This is a flowchart of a method according to at least one embodiment of the present disclosure;

[0041] Figure 4 This is a flowchart of another method according to at least one embodiment of this disclosure; and

[0042] Figure 5 This is a flowchart of another method according to at least one embodiment of the present disclosure. Detailed Implementation

[0043] It should be understood that the various aspects disclosed herein can be combined in combinations different from those specifically presented in the specification and drawings. It should also be understood that, depending on the example or embodiment, certain actions or events of any process or method described herein may be performed in a different order, and / or may be added, combined, or omitted entirely (e.g., implementing the disclosed technology may not require all described actions or events depending on the different embodiments of this disclosure). Furthermore, although some aspects of this disclosure are described for clarity as being performed by a single module or unit, it should be understood that the technology of this disclosure can be performed by a combination of units or modules associated with, for example, computing devices and / or medical devices.

[0044] In one or more examples, the described methods, processes, and techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a non-transitory computer-readable medium, which corresponds to a tangible medium, such as a data storage medium (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer).

[0045] The instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors (e.g., Intel Core i3, i5, i7, or i9 processors; Intel Celeron processors; Intel Xeon processors; Intel Pentium processors; AMD Ryzen processors; AMD Athlon processors; AMD Phenom processors; Apple A10 or 10X Fusion processors; Apple A11, A12, A12X, A12Z, or A13 Bionic processors; or any other general-purpose microprocessor), application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other physical structures suitable for implementing the described techniques. Furthermore, this technology can be fully implemented in one or more circuit or logic elements.

[0046] Before explaining any embodiment of this disclosure in detail, it should be understood that this disclosure is not limited in its application to the construction details and component arrangements set forth in the following description or shown in the accompanying drawings. This disclosure can have other embodiments and can be practiced or implemented in various ways. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered limiting. The use of “including / comprising” or “having” and variations thereof herein is intended to cover the items listed thereafter and their equivalents, as well as additional items. In addition, this disclosure may use examples to illustrate one or more aspects thereof. Unless otherwise expressly stated, the use or listing of one or more examples (which may be indicated by “for example,” “by means of an example,” “e.g.,” “such as,” or similar language) is not intended to, and does not limit, the scope of this disclosure.

[0047] Images of a portion of a patient's anatomy taken at different time points can reflect considerable structural variability. This is especially true when images are taken preoperatively and postoperatively, and / or when there are long intervals between images (including months or years). For example, the spinal structure of a patient after the insertion of a spinal rod may differ significantly from that before the rod's insertion. Furthermore, a patient's spine may undergo significant deformation weeks, months, and years after rod insertion. It is necessary to identify and quantify changes in posture of one or more anatomical units from the first time the first image was taken to the second time the second image was taken.

[0048] Taking structural changes in a patient's spine as an example, several factors can prevent the comparison of periodic measurements of such changes. These factors may include: changes in the orientation of the camera or other imaging device that generates the first and second images; changes in the orientation of the patient's anatomy when the first and second images are captured (e.g., the first image may be taken when the patient is in a prone or supine position, and the second image may be taken when the patient is in a standing position); noise in the source label due to noisy images and / or segmentation errors; and non-rigid transformations of the spine over a period of time or before and after surgery.

[0049] The embodiments of this disclosure utilize corresponding points along the periphery of each vertebra depicted in the first and second images taken at times t1 and t2, respectively. For example, edge points of the vertebral endplates in AP or LT projections obtained at any two times t1 and t2 can be used. These points can be identified manually or automatically.

[0050] Because the spine undergoes non-rigid transformations over time, or before and after surgery, it is impossible to directly calculate the transformation between times t1 and t2. In other words, because the vertebrae of the spine can move and rotate in different ways, simply comparing the changes in the entire spinal structure from time t1 to time t2 does not provide accurate results. Instead, this problem can be addressed within the vertebral region by utilizing the segmental rigidity of the spine. Since the motion of each vertebra itself can be assumed to be rigid, the transformation can be calculated for each vertebra. Furthermore, since the periphery of the vertebra can be represented as a plane (e.g., endplate, lateral, lateral, or anterior projection), the homography transformation H can be a sufficiently useful representation.

[0051] Then, according to the embodiments of this disclosure, for each vertebra, at least four corresponding points in each image are used to calculate the homography H parameter. To reduce noise in the calculation, these points can be used if more points are available; typical computer vision methods can be used to automatically fix the marked corner points; and interpolation points along the marked lines can be used.

[0052] If spinal motion is rigid, all calculated homography {H} will be more or less the same. However, due to some movement in individual vertebrae, the calculated homography is expected to differ. Additionally, noise in the marker points will increase noise homography.

[0053] Based on the foregoing, a set of homography {H} in the transformation space (whether 9-dimensional or reduced) can be clustered according to predetermined characteristics. The most coherent clusters can be selected, and / or clusters / homography can be filtered according to other criteria. The average of the resulting clusters can then be considered as the homography H' between times t1 and t2. H' can then be used to project all vertebrae from t2 onto t1, and the measurement results / features can be calculated in a more comparable manner.

[0054] The embodiments of this disclosure are based on the assumption that changes in bone structure over time are less pronounced than changes in soft tissue. Even so, compression fractures and osteophyte (bone spur) changes can interfere with the successful utilization of the embodiments of this disclosure. In cases where one or more monographys are affected by compression fractures and / or osteophytes (and / or other changes in the shape of rigid anatomical units), it may be necessary to filter out such monographys before other monographys are averaged or otherwise utilized.

[0055] For registration of two-dimensional images, at least four corresponding points are required, while for registration of three-dimensional images, at least eight corresponding points are required. In some embodiments, the implant itself can replace or supplement the vertebral endplate or other anatomical features as a source of corresponding points. For example, a rod can provide two corresponding points (e.g., one point at each end of the rod), making it possible to obtain four corresponding points between two images using two rods, one rod and one screw, or even two screws. Of course, when one or more implants are used to define one or more corresponding points, only the image pairs depicting the one or more implants can be registered with each other. Therefore, images taken before the insertion of such implants cannot be used in these embodiments. Even so, using implants to define corresponding points advantageously takes advantage of the fact that, unlike some anatomical units, implant structures generally do not change over time.

[0056] The embodiments of this disclosure advantageously achieve long-term registration, i.e., registration of two images generated at intervals of weeks, months, or even years. The embodiments of this disclosure also advantageously utilize the piecewise rigidity of the spine (and / or other anatomical units composed of multiple individual rigid units) to overcome the computational challenges of directly determining transformations of the spine or other anatomical units. By combining data science methods with typical computer vision methods, a set of potential transformations can be generated, which are then analyzed using clustering methods.

[0057] The embodiments of this disclosure provide technical solutions to one or more of the following problems: (1) generating accurate geometric measurements of X-ray images of the same patient taken at different time points, with intervals of weeks, months or even years; (2) during the registration of two images taken at two different times and possibly by two different imaging devices, (i) taking into account the effect of changes in camera pose relative to the patient's anatomy from one image to another, (ii) taking into account the effect of changes in body pose and position from one image to another, and (iii) taking into account noise in the source markers due to noisy images or segmentation errors; (3) registering two spinal images generated at different times to each other, regardless of non-rigid transformations of the spine during the time interval between the generation of the first image and the generation of the second image; and (4) distinguishing the changes in the pose of one or more rigid units depicted in the two images due to changes in camera pose from changes due to changes in the pose of the one or more rigid units themselves.

[0058] First turn Figure 1 This diagram illustrates a block diagram of a system 100 according to at least one embodiment of the present disclosure. System 100 can be used to register two time-spaced images to each other and / or perform one or more other aspects of one or more methods disclosed herein. System 100 includes a computing device 102, one or more imaging devices 112, a navigation system 114, a robot 130, a database 136, and a cloud 138. Systems according to other embodiments of the present disclosure may include more or fewer components than system 100. For example, system 100 may not include the navigation system 114, the robot 130, one or more components of the computing device 102, the database 136, and / or the cloud 138.

[0059] The computing device 102 includes a processor 104, a memory 106, a communication interface 108, and a user interface 110. Other embodiments of the computing device according to this disclosure may include more or fewer components than the computing device 102.

[0060] The processor 104 of the computing device 102 may be any processor described herein or any similar processor. The processor 104 may be configured to execute instructions stored in the memory 106, which may cause the processor 104 to perform one or more computational steps using or based on data received from the imaging device 112, the robot 130, the navigation system 114, the database 136 and / or the cloud 138.

[0061] Memory 106 may be or include RAM, DRAM, SDRAM, other solid-state memory, any memory described herein, or any other tangible, non-transitory memory used to store computer-readable data and / or instructions. Memory 106 may store information or data used to perform any steps of methods 300, 400, and / or 500 or any other method described herein. Memory 106 may store, for example, one or more image processing algorithms 120, one or more segmentation algorithms 122, one or more transformation algorithms 124, one or more homography algorithms 126, and / or one or more registration algorithms 128. In some embodiments, such instructions or algorithms may be organized into one or more applications, modules, packages, layers, or engines. Algorithms and / or instructions may enable processor 104 to manipulate data stored in memory 106 and / or received from or via imaging device 112, robot 130, database 136, and / or cloud 138.

[0062] The computing device 102 may also include a communication interface 108. The communication interface 108 can be used to receive data or other information from external sources (such as imaging device 112, navigation system 114, robot 130, database 136, cloud 138, and / or any other system or component not part of system 100), and / or to transmit instructions, images, or other information to external systems or devices (e.g., another computing device 102, navigation system 114, imaging device 112, robot 130, database 136, cloud 138, and / or any other system or component not part of system 100). The communication interface 108 may include one or more wired interfaces (e.g., USB port, Ethernet port, FireWire port) and / or one or more wireless transceivers or interfaces (configured to transmit and / or receive information via, for example, one or more wireless communication protocols such as 802.11a / b / g / n, Bluetooth, NFC, Zifeng, etc.). In some implementations, the communication interface 108 can be used to enable the device 102 to communicate with one or more other processors 104 or computing devices 102, whether to reduce the time required to complete computationally intensive tasks or for any other reason.

[0063] The computing device 102 may also include one or more user interfaces 110. The user interface 110 may be or include a keyboard, mouse, trackball, monitor, television, screen, touchscreen, and / or any other means for receiving information from and / or providing information to the user. The user interface 110 may be used, for example, to receive user selections or other user input regarding any step of any method described herein. Nevertheless, any required input for any step of any method described herein may be automatically generated by system 100 (e.g., by processor 104 or another component of system 100) or received by system 100 from a source external to system 100. In some embodiments, the user interface 110 may be used to allow a surgeon or other user to modify instructions to be executed by processor 104, and / or modify or adjust settings of other information displayed on or corresponding to the user interface 110, according to one or more embodiments of this disclosure.

[0064] Although user interface 110 is shown as part of computing device 102, in some embodiments, computing device 102 may utilize user interface 110, which may be housed separately from one or more other components of computing device 102. In some embodiments, user interface 110 may be located near one or more other components of computing device 102, while in other embodiments, user interface 110 may be located away from one or more other components of computing device 102.

[0065] Imaging device 112 can be used to image anatomical features (e.g., bones, veins, tissues, etc.) and / or other aspects of a patient's anatomy to generate image data (e.g., image data depicting or corresponding to bones, veins, tissues, etc.). Image data may be or include preoperative images, postoperative images, or images taken independently of any surgical procedure. In some embodiments, a first imaging device 112 can be used to acquire first image data (e.g., a first image) at a first time, and a second imaging device 112 can be used to acquire second image data (e.g., a second image) at a second time after the first time. The first and second times may be spaced between the time of a surgical procedure (e.g., one may be preoperative and the other postoperative) or a period of time (e.g., days, weeks, months, or years). Imaging device 112 may be able to capture 2D or 3D images to generate image data. As used herein, "image data" refers to data generated or captured by imaging device 112, including data in machine-readable form, graphical / visual form, and any other form. In various examples, image data may include data corresponding to a patient's anatomical features or a portion thereof. The imaging device 112 may be or include, for example, an ultrasound scanner (which may include, for example, physically separate transducers and receivers, or a single ultrasound transceiver), a radar system (which may include, for example, a transmitter, receiver, processor, and one or more antennas), an O-arm, a C-arm, a G-arm, or any other device that utilizes X-ray-based imaging (e.g., a fluorescence microscope, a CT scanner, or other X-ray machine), a magnetic resonance imaging (MRI) scanner, an optical coherence tomography scanner, an endoscope, a telescope, a thermal imaging camera (e.g., an infrared camera), or any other imaging device 112 adapted to obtain images of the patient's anatomical features.

[0066] In some embodiments, imaging device 112 may include more than one imaging device 112. For example, a first imaging device may provide first image data and / or a first image, and a second imaging device may provide second image data and / or a second image. In other embodiments, the same imaging device may be used to provide both first image data and second image data and / or any other image data described herein. Imaging device 112 may be used to generate an image data stream. For example, imaging device 112 may be configured to use an open shutter operation or to use shutter operations that alternate continuously between open and closed to capture a series of images. For the purposes of this disclosure, unless otherwise specified, if the image data represents two or more frames per second, the image data may be considered continuous and / or provided as an image data stream.

[0067] During operation, navigation system 114 can provide navigation for the surgeon and / or surgical robot. Navigation system 114 can be any navigation system currently known or developed in the future, including, for example, Medtronic StealthStation. TM The S8 surgical navigation system or any successor thereof. Navigation system 114 may include one or more cameras or other sensors for tracking one or more reference markers, navigation trackers, or other objects within the operating room or some or all other rooms located therein in system 100. In various embodiments, navigation system 114 may be used to track the position and orientation (i.e., attitude) of imaging device 112, robot 130 and / or robotic arm 132 and / or one or more surgical instruments (or more specifically, for tracking the attitude of navigation trackers directly or indirectly attached in a fixed relationship to one or more of the foregoing). Navigation system 114 may include a display for displaying one or more images from an external source (e.g., computing device 102, imaging device 112, or other sources) or for displaying images and / or video streams from cameras or other sensors of navigation system 114. In some embodiments, system 100 may operate without using navigation system 114. The navigation system 114 can be configured to provide guidance to the surgeon or other users of the system 100 or its components, to the robot 130 or any other element of the system 100, regarding, for example, the posture of one or more anatomical units and / or whether the tools are in the appropriate trajectory (and / or how to move the tools into the appropriate trajectory) to perform surgical tasks according to the preoperative plan.

[0068] Robot 130 can be any surgical robot or surgical robot system. Robot 130 can be, or include, for example, Mazor X. TM The Stealth Edition robotic guidance system. Robot 130 can be configured to position the imaging device 112 in one or more precise locations and orientations, and / or return the imaging device 112 to the same location and orientation at a later point in time. Robot 130 may additionally or alternatively be configured to manipulate surgical instruments (whether or not based on guidance from navigation system 114) to perform or assist surgical tasks. Robot 130 may include one or more robotic arms 132. In some embodiments, robotic arms 132 may include a first robotic arm and a second robotic arm, but robot 130 may include more than two robotic arms. In some embodiments, one or more of the robotic arms 132 may be used to hold and / or manipulate the imaging device 112. In embodiments where the imaging device 112 includes two or more physically separate components (e.g., transmitters and receivers), one robotic arm 132 may hold one such component, and another robotic arm 132 may hold another such component. Each robotic arm 132 may be positioned independently of the other robotic arms.

[0069] The robot 130, together with the robotic arm 132, may have, for example, at least five degrees of freedom. In some embodiments, the robotic arm 132 has at least six degrees of freedom. In other embodiments, the robotic arm 132 may have fewer than five degrees of freedom. Furthermore, the robotic arm 132 can be in any pose, planar, and / or focally positioned or locatable. The pose includes position and orientation. Therefore, the imaging device 112, surgical instrument, or other object held by the robot 130 (or more specifically, held by the robotic arm 132) can be precisely positioned in one or more desired and specific locations and orientations.

[0070] In some embodiments, reference markers (i.e., navigation markers) may be placed on the robot 130 (including, for example, on the robotic arm 132), the imaging device 112, or any other object in the surgical space. The reference markers may be tracked by the navigation system 114, and the results of the tracking may be used by the operator of the robot 130 and / or by the operator of the system 100 or any component thereof. In some embodiments, the navigation system 114 may be used to track other components of the system (e.g., the imaging device 112), and the system may be operated without the use of the robot 130 (e.g., a surgeon manually manipulating the imaging device 112 and / or one or more surgical instruments, for example, based on information and / or instructions generated by the navigation system 114).

[0071] System 100 or similar systems can be used, for example, to perform one or more aspects of any of the methods 300, 400, and / or 500 described herein. System 100 or similar systems can also be used for other purposes. In some embodiments, system 100 can be used to generate and / or display a 3D model of a patient's anatomical features or anatomical volume. For example, a robotic arm 132 (controlled by a processor of robot 130, processor 104 of computing device 102, or some other processor, with or without any manual input) can be used to position imaging device 112 in a plurality of predetermined known poses, such that imaging device 112 can acquire one or more images at each of the predetermined known poses. Because the pose at which each image is captured is known, the resulting images can be assembled to form or reconstruct a 3D model. As described elsewhere herein, system 100 can update the model based on information received from imaging device 112 (e.g., fragment tracking information).

[0072] Now go to Figure 2Embodiments of this disclosure can be used, for example, to register two images 200 with a time interval. For example, embodiments of this disclosure can be used to: register a preoperative image 200A with a postoperative image 200B, an image 200C taken six months postoperatively, and / or an image 200D taken one year postoperatively; register a postoperative image 200B with an image 200C taken six months postoperatively and / or an image 200D taken one year postoperatively; and / or register an image 200C taken six months postoperatively with an image 200D taken one year postoperatively. Additionally, embodiments of this disclosure can be used to obtain precise geometric measurements of changes in the pose of one or more anatomical units or medical implants depicted in the registered images, even though the images cannot be directly registered to each other (e.g., by simply overlaying one image onto another and aligning corresponding points) due to: changes in the pose of the camera used to capture images relative to the imaged patient anatomy; changes in the patient's pose when the images are generated; noise in the images; and / or non-rigid transformations of the anatomical structures depicted in the images.

[0073] Although Figure 2 Preoperative image 200A, postoperative image 200B taken immediately after surgery for implantation of the rod and screw depicted in image 200B, image 200C taken six months after the same surgery, and image 200D taken one year after the same surgery are shown. However, embodiments of this disclosure can be used to register two images with intervals longer or shorter than the time from preoperative to postoperative, six months, and / or one year. In some embodiments, this disclosure can be used to register two images taken two, five, ten, or more years apart. In other embodiments, this disclosure can be used to register two images taken one, two, three, four, five, seven, eight, nine, ten, or eleven months apart. In other embodiments, this disclosure can be used to register two images taken over periods of several weeks or days apart. While the benefits of embodiments of this disclosure may be most significant when there is a significant transformation of the imaged anatomical unit from one image to another, those same embodiments can be used regardless of the degree of transformation of the anatomical unit between the two images taken.

[0074] Figure 3 Method 300 is described that can be used for registration between any two or more of the following: long-term registration, short-term registration, preoperative models updating patient anatomy, and / or updating robot space, navigation space, and / or patient space. The term "long-term registration" is intended to mean that method 300 can be used to register images with time intervals, including images taken days, weeks, months, or even years apart. Even so, method 300 can also be used to register images taken at relatively close times (e.g., preoperative and intraoperative).

[0075] Method 300 (and / or one or more steps thereof) may be performed, for example, by at least one processor or otherwise. At least one processor may be the same as or similar to processor 104 of the computing device 102 described above. At least one processor may be part of a robot (such as robot 130) or a navigation system (such as navigation system 114). Processors other than any of the processors described herein may also be used to perform method 300. At least one processor may perform method 300 by executing instructions stored in a memory such as memory 106. These instructions may correspond to one or more steps of method 300 described below. These instructions may cause the processor to execute one or more algorithms, such as image processing algorithm 120, segmentation algorithm 122, transformation algorithm 124, homography algorithm 126, and / or registration algorithm 128.

[0076] Method 300 includes receiving a first image of the patient's anatomical structure (step 304). The first image is generated by an imaging device (such as imaging device 112) and is generated at a first time. The first time may be one day or more days, weeks or months before a surgery affecting the imaged anatomy, or the first time may be immediately before the surgery (e.g., while the patient is on the operating table and / or in the operating room), or the first time may be after the surgery. In some embodiments, the first image is taken independently of any surgical procedure.

[0077] The imaged anatomical structure can be, for example, the spine of a patient or a portion thereof comprising multiple vertebrae. In other embodiments, the imaged anatomical structure can be any other anatomical object composed of multiple rigid or substantially rigid subunits, or any other anatomical object undergoing non-rigid deformation and capable of being analyzed at the subunit level.

[0078] The first image can be received directly or indirectly from an imaging device such as imaging device 112. The first image can be a two-dimensional image or a three-dimensional image. In some embodiments, the first image is an X-ray image or an image generated using X-rays, such as a CT image or a fluorescein image. However, the image can be generated using any other imaging modality, such as ultrasound, magnetic resonance imaging, optical coherence tomography, or another imaging modality. Therefore, the imaging device can be a CT scanner, a magnetic resonance imaging (MRI) scanner, an optical coherence tomography (OCT) scanner, an O-arm (including, for example, an O-arm 2D long film scanner), a C-arm, a G-arm, another device utilizing X-ray-based imaging (e.g., a fluorescein or other X-ray machine), or any other imaging device.

[0079] Method 300 also includes receiving a second image of the patient's anatomy (step 308). The second image is also generated by an imaging device (such as imaging device 112), but the imaging device used to generate the second image may be different from the imaging device used to generate the first image. Furthermore, the second image is generated at a second time after the first time. The second time may be the time interval between the first and second time intervals, such as the time of a surgical procedure (e.g., the first image may be a preoperative image and the second image may be a postoperative image). The second time may be one day or more, several weeks, months, or years after the first time interval.

[0080] The second image typically corresponds to an anatomical region or portion of the same patient anatomy as the first image or a part thereof. Therefore, for example, if the first image depicts the patient's spine or a segment thereof, the second image data also depicts the spine or a segment thereof. As another example, if the first image depicts the patient's knee or a portion thereof, the second image data also depicts the knee or a portion thereof.

[0081] A second image can be received directly or indirectly from an imaging device such as imaging device 112. The second image may have the same dimension as the first image (e.g., two-dimensional or three-dimensional). The second image may be an image generated using the same imaging device as the first image or a different imaging device. The imaging device that generates the second image may have the same imaging modality as the imaging device that generated the first image, or a related imaging modality. In some embodiments, different imaging modalities may be used to generate the first image and the second image.

[0082] Method 300 further includes determining a transformation from the first image to the second image for each of a plurality of rigid units in the first and second images to produce a set of transformations (step 312). In some embodiments, step 312 may include preprocessing the first and second images using one or more image processing algorithms 120 to: remove noise and / or artifacts from them, ensure that the two images have the same scale, and otherwise prepare the images for other aspects of step 312. One or more image processing algorithms 120 may also be used to identify a plurality of rigid units in each image, regardless of whether feature recognition, edge detection, or other object detection methods are used.

[0083] In some implementations, step 312 includes segmenting the first and second images to identify and / or label individual rigid units within each image. Such segmentation can be accomplished using one or more segmentation algorithms 122 and / or any other segmentation algorithm or process. Step 312 may also include using anatomical atlases, biomechanical models, or other references to identify anatomical objects within the first and second images, determine which of those anatomical objects are rigid units, and / or determine the relationship (if any) between two or more identified rigid units. Thus, for example, an anatomical atlas may be referenced to determine that two adjacent vertebrae are connected by an intervertebral disc, or a patient-specific biomechanical model may be referenced to determine that two adjacent vertebrae have fused and should move as a whole within the patient's anatomy.

[0084] The plurality of rigid units may include individual bone or other hard tissue anatomical objects. The plurality of rigid units may also include one or more medical implants, such as pedicle screws, vertebral rods, surgical pins, and / or intervertebral bodies. A particular rigid unit may be excluded from the plurality of rigid units if it appears in a first image but not in a second image, or vice versa. Similarly, in some embodiments, the plurality of rigid units may not include each rigid unit depicted in one or two images. For the purposes of this disclosure, a unit of bone anatomy or other hard tissue may be considered rigid, even if the unit has a degree of flexibility. At least one of the plurality of rigid units is movable relative to at least another rigid unit in the plurality of rigid units.

[0085] To determine the transformation from the first image to the second image for each of the plurality of rigid units, one or more transformation algorithms 124 may be used. This determination may include superimposing the second image onto the first image or defining any other relationship between the first and second images. In some embodiments, a “best guess” alignment between the first and second images may be performed automatically or manually, such as by aligning prominent edges or surfaces in the two images (e.g., visible edges of the patient, such as the patient’s back or side; one or more surfaces of the patient’s hip or pelvis; or one or more surfaces of another hard tissue unit that is less likely to move over time than the rigid unit in question). To determine the transformation, a fixed relationship must be established between the two images; however, the fixed relationship does not need to be accurate, as the remaining steps of method 300 will distinguish between aspects of each transformation attributable to camera pose, patient position, or other parameters that affect the depiction of each rigid unit in the same way, and aspects of each transformation attributable to movement of the rigid unit.

[0086] The determined transformation of each of the plurality of rigid units can be homography. Homography relates a given rigid unit as depicted in the first image to the same rigid unit as depicted in the second image. For the purpose of calculating homography, a plurality of points on the rigid unit (visible in both the first and second images) can be selected. For example, these points can be points along the periphery of the rigid unit in anteroposterior (AP) or lateral (LT) projections. In the case of a vertebra, these points can be the edge points of the vertebral endplate. In the case of a screw, these points can be at both ends of the screw (e.g., at the top of the screw head and at the screw tip). In the case of a rod, these points can be at opposite ends of the rod. For the purposes of this disclosure, a plurality of screws in a single anatomical unit can be considered as a single rigid unit. These points can be specified manually (e.g., via a user interface such as user interface 110) or automatically (e.g., using image processing algorithm 120, segmentation algorithm 122, or any other algorithm). Homography can be calculated using homography algorithms such as homography algorithm 126. Any known method for calculating homography can be used.

[0087] In some implementations, the homography of adjacent rigid elements can be calculated. Therefore, for example, the homography of each pair of adjacent vertebrae can be calculated using a determined transformation corresponding to each pair of adjacent vertebrae.

[0088] When the first and second images are two-dimensional images, the plurality of points includes at least four points. When the first and second images are three-dimensional images, the plurality of points includes at least eight points. Whether using 2D or 3D images, more points than the minimum number can be utilized. Additionally, the required points may include points defined by reference to patient anatomy, points defined by reference to one or more implants (e.g., screws, rods), or any combination thereof. The selected points can be connected by marker lines, and one or more points can be interpolated along the marker lines. It is worth noting that noise in the points used to calculate homography (e.g., any difference between the position of a point in each image and its corresponding rigid element in that image) will result in the calculation of noisy homography.

[0089] Given that implants are less likely to change over time than anatomical units, it is advantageous to use screws, rods and / or other implants as rigid units for the purposes of this disclosure.

[0090] The determination of a transformation (whether homography or otherwise) for each of the plurality of rigid elements results in a set of transformations. Each transformation may include one or more distances, angles, and / or other measurements sufficient to describe the attitude change of the rigid element for which its transformation is determined. In some embodiments, each determined transformation may simply include segmented images of the rigid element at a first position (e.g., as depicted in the first image) and at a second position (e.g., as depicted in the second image). In other embodiments, the determined transformation may include equations or a set of equations describing the movement of the rigid element from the attitude depicted in the first image to the attitude depicted in the second image.

[0091] Method 300 also includes calculating the homography of each transformation.

[0092] Method 300 also includes identifying common components of each transformation attributable to pose changes (step 316). For example, this identification can be based on calculated transformations. Based on the assumption that most transformations will be determined solely by camera pose changes (e.g., because the corresponding vertebra or other rigid unit has not yet moved), or alternatively, that the same or nearly identical transformations are caused by camera pose changes (which more or less equally affect each rigid unit, and the motion of individual rigid units does not necessarily have any correlation with the motion of other rigid units), clustering data science methods can be used to separate transformations that originate solely from camera pose from those that are caused by a combination of camera pose changes and the motion of rigid units.

[0093] When using clustering, clustering can be performed in a transformed space (e.g., a 9-dimensional space) or a reduced space. The resulting clusters can be analyzed using contour measurements, variance, magnitude, or another parameter that can be used to separate those transformations attributable to camera pose changes from those attributable to both camera pose changes and the motion of rigid elements. The most coherent clusters (or clusters selected by applying other parameters) can be averaged, where the cluster average is considered as a transformation corresponding to the camera pose change. Using clustering is beneficial in explaining noise in the transformations caused by noise in the markers used to calculate the transformations.

[0094] The transformation corresponding to the camera pose change can explain the motion of most individual rigid units in the first and second images. Regardless of the number of rigid units whose transformation is explained solely by the camera pose change, the portion of each transformation explained solely by the camera pose change constitutes a common part of each transformation (e.g., because this portion equally affects each transformation).

[0095] Regardless of how step 316 is performed, the result is to determine how the pose changes of the camera used to capture the first and second images lead to the determined transformation of each rigid unit.

[0096] Method 300 further includes identifying a separate portion of each transformation attributable to a change in the rigid unit's pose (step 320). Identifying a separate portion of each transformation attributable to a change in the rigid unit's pose may include, for example, projecting each rigid unit from the second image onto the first image using the common portion of each transformation determined in step 316. The result of this projection will be to align a rigid unit depicted in the second image that has not moved between the first and second times with its corresponding rigid unit depicted in the first image. However, for rigid units that have moved between the first and second times, the result of this projection will be to remove the effect of the camera pose change from the depiction of the rigid unit. Therefore, any misalignment between a rigid unit projected from the second image onto the first image and its corresponding rigid unit as depicted in the first image can be attributed to the movement of the rigid unit itself. In other words, any difference between the pose of a rigid unit projected from the second image onto the first image and its corresponding rigid unit as depicted in the first image constitutes a separate portion of each transformation attributable to a change in the rigid unit's pose.

[0097] In some embodiments, the identification may not include using the common portion of each transformation determined in step 316 to project the rigid element from the second image onto the first image. Instead, the identification may include calculating the difference between the common portion of each transformation determined in step 316 and the transformation calculated for each individual rigid element. In some embodiments, any such calculated difference below a predetermined threshold may be discarded due to noise or otherwise constituting non-substantial motion. According to embodiments of this disclosure, other methods may also be utilized to identify individual portions of each transformation attributable to changes in the rigid element's attitude.

[0098] Method 300 further includes registering the second image to the first image based on the identified common portions (step 324). Step 324 may occur prior to step 320 (and other steps) and may include projecting rigid units from the second image onto the first image using the common portions of each transformation identified in step 316. The registration may also include otherwise aligning the second image to the first image based on units known to have not moved from the first time to the second time (e.g., units that appear to have moved only due to camera pose changes but have not actually moved (or have not moved within a certain tolerance)). The registration may utilize one or more registration algorithms, such as registration algorithm 128.

[0099] In some embodiments, step 324 may alternatively include updating the preoperative model based on individual portions of each transformation. The preoperative model may have been generated, for example, based on preoperative images, and the update may include updating each rigid cell depicted in the preoperative model to reflect any change in the position of that rigid cell from the time the preoperative image was taken to the time the second image was taken. In this embodiment, the second image may be an intraoperative image or a postoperative image.

[0100] Similarly, in some embodiments, step 324 may alternatively include updating the registration of one of the robot space or navigation space with the image space based on either a common portion of each transformation or a separate portion of each transformation. This update can help maintain accurate registration, which in turn can increase the accuracy of surgical procedures.

[0101] Method 300 further includes quantifying the attitude change of at least one rigid unit (step 328). This quantization utilizes only portions of each transformation attributable to the attitude change of the rigid unit. In other words, the quantization includes quantifying one or more aspects of the attitude change of the rigid unit caused by the movement of the particular rigid unit from a first time to a second time (rather than the apparent attitude change of the rigid unit attributable to the attitude change of the camera used to image the particular rigid unit at the first and second times).

[0102] The quantization may include, for example, determining the rotation angle of the rigid element from a first time point to a second time point, and / or determining the translation distance of the rigid element. The quantization may include comparing the attitude of the rigid element at the second time point with a desired attitude change, and may be expressed as a percentage of the desired attitude change (e.g., based on a comparison with a physical or virtual model of the ideal attitude of the rigid element, whether in surgical planning, treatment planning, or otherwise). The quantization may also include quantifying the attitude change of each rigid element within the rigid element.

[0103] This disclosure covers embodiments of method 300 that include more or fewer steps than those described above, and / or one or more steps that differ from those described above.

[0104] Figure 4A method 400 for associating images captured at different times is described. Method 400 (and / or one or more steps thereof) may be performed, for example, by at least one processor or otherwise. The at least one processor may be the same as or similar to processor 104 of the computing device 102 described above. The at least one processor may be part of a robot (such as robot 130) or a navigation system (such as navigation system 114). Processors other than any of the processors described herein may also be used to perform method 400. The at least one processor may perform method 400 by executing instructions stored in a memory such as memory 106. The instructions may correspond to one or more steps of method 400 described below. These instructions may cause the processor to execute one or more algorithms, such as image processing algorithm 120, segmentation algorithm 122, transformation algorithm 124, homography algorithm 126, and / or registration algorithm 128.

[0105] Method 400 includes segmenting each of a plurality of rigid units in a first image and a second image (step 404). The first image is taken at a first time, and the second image is taken at a second time after the first time. The first image may be identical or similar to any other first image described herein, and the second image may be identical or similar to any other second image described herein. The first and second images each depict common portions of the patient's anatomy, but the first and second images may not be perfectly aligned (e.g., the first image may depict one or more portions of the patient's anatomy not depicted in the second image, in addition to common portions depicted in both the first and second images, and vice versa). The plurality of rigid units may be, or include, for example, one or more vertebrae and / or other bone anatomy structures or hard tissue units, and / or one or more implants (e.g., pedicle screws, cortical screws, rods, pins, and / or other implants).

[0106] This segmentation can be accomplished using one or more segmentation algorithms 122 and / or any other segmentation algorithm or process. Step 312 may also include using anatomical atlases, biomechanical models, or other references to identify anatomical objects within the first and second images, determine which of those anatomical objects are rigid units, and / or determine the relationships (if any) between two or more identified rigid units. Thus, for example, an anatomical atlas may be referenced to determine that two adjacent vertebrae are connected by an intervertebral disc, or a patient-specific biomechanical model may be referenced to determine that two adjacent vertebrae have fused and should move as a whole within the patient's anatomy. This segmentation enables the determination of the perimeter of each rigid unit in the first and second images, allowing each rigid unit to be analyzed individually.

[0107] Method 400 further includes calculating a set of homography that associates the depiction of each rigid cell in the first image with a corresponding rigid cell in the second image (step 408). The homography can be calculated using any known method. This calculation may utilize one or more homography algorithms 126. Each calculated homography describes the relationship between a rigid cell in the first image and a corresponding rigid cell in the second image. In other words, each homography associates a rigid cell in the first image with a corresponding rigid cell in the second image. In other words, using the calculated homography and the depiction of rigid cells in either the first or second image, a depiction of rigid cells can be generated in the other of the first or second image.

[0108] For the purpose of calculating homography, multiple points (visible in both the first and second images) on each rigid unit can be selected. For example, these points can be points along the periphery of the rigid unit in anteroposterior (AP) or lateral (LT) projections. In the case of a vertebra, these points can be the edge points of the vertebral endplate. In the case of a screw, these points can be at both ends of the screw (e.g., at the top of the screw head and at the screw tip). In the case of a rod, these points can be at opposite ends of the rod. For the purposes of this disclosure, multiple screws in a single anatomical unit can be considered as a single rigid unit. These points can be specified manually (e.g., via a user interface such as user interface 110) or automatically (e.g., using image processing algorithm 120, segmentation algorithm 122, or any other algorithm). Homography can be calculated using homography algorithms such as homography algorithm 126. Any known methods for calculating homography can be used.

[0109] When the first and second images are two-dimensional images, the plurality of points includes at least four points. When the first and second images are three-dimensional images, the plurality of points includes at least eight points. Whether using 2D or 3D images, more points than the minimum number can be utilized. Additionally, the required points may include points defined by reference to patient anatomy, points defined by reference to one or more implants (e.g., screws, rods), or any combination thereof. The selected points can be connected by marker lines, and one or more points can be interpolated along the marker lines. It is worth noting that noise in the points used to calculate homography (e.g., any difference between the position of a point in each image and its corresponding rigid element in that image) will result in the calculation of noisy homography.

[0110] Given that implants are less likely to change over time than anatomical units, it is advantageous to use screws, rods and / or other implants as rigid units for the purposes of this disclosure.

[0111] Method 400 also includes removing any homography from the set of homography affected by physical changes in the shape of the rigid unit (step 412). Method 400 is based on the assumption that changes in bone structure (and more generally, changes in the shape of rigid units) are less noticeable than changes in soft tissue, and that any rigid unit whose shape has changed will add unwanted noise. However, the shape of rigid units does sometimes change. Such shape changes can be caused by, for example, compression fractures, osteophytes (bone spurs), and / or other reasons.

[0112] The removal of homography affected by physical changes in shape can be done manually or automatically. In some embodiments, shape changes can be identified by a therapist or other user before any homography is calculated (e.g., from the first and second images). In other embodiments, the therapist or other user can view the first and second images after the homography has been calculated and can identify one or more rigid units whose shape has changed, based on which the corresponding homography can be discarded or ignored. In other embodiments, the processor can use one or more image processing algorithms 120 or other algorithms to identify shape changes, whether before or after segmentation in step 404. In such embodiments, shape changes can be identified based on a coarse comparison of the edges of each rigid unit in the first and second images (e.g., as detected using an edge detection algorithm, segmentation algorithm, or other algorithm).

[0113] Method 400 further includes arranging the group of homography into homography clusters (step 416). Homography can be clustered using any data science clustering method. The purpose of clustering is to identify the most similar homography, which can be assumed to correspond to rigid units that have not moved from the first time to the second time, but whose pose change in the second image relative to the first image can be attributed entirely or almost entirely to the camera pose change. Therefore, any clustering method that results in similar homography being grouped together can be used. Clustering can be done in a transformed space (e.g., 9-dimensional space) or a reduced space.

[0114] Method 400 also includes selecting homography clusters based on parameters (step 420). Parameters can be profiles, variances, sizes, or another parameter that can be used to separate homography attributable to camera attitude changes from homography attributable to both camera attitude changes and the motion of rigid elements. Clusters can include most or a few of the homography types utilized in cluster analysis. Because (e.g., from a first time to a second time) camera attitude changes will affect each rigid element equally (and the motion of each rigid element will not necessarily be related to the motion of any other rigid element), the most coherent clusters are most likely to include homography that reflects only the perceived motion caused by that camera attitude change. However, even the most coherent clusters cannot have perfectly matching homography due to noise in the markers used to calculate homography, the segmentation of each rigid element, and any other aspect of Method 400 that may lack 100% accuracy.

[0115] Method 400 further includes using the average of selected homography clusters to project each rigid unit as depicted in the second image onto the first image to produce a projected image (step 424). The average of the selected homography is used to reduce the effect of any noise affecting the homography in the most coherent (or other selected) clusters. The selected homography average is then used to project the rigid units from the second image onto the first image. Because the selected homography corresponds to the effect of the camera's pose change from a first time (when the first image is captured) to a second time (when the second image is captured), the projection results in any projected rigid unit that has not moved from the first time to the second time being aligned and overlapped with the corresponding rigid unit from the first image. For any rigid unit that has moved from the first time to the second time, the projection of such a rigid unit removes any effect from the camera pose change on the pose of such projected rigid unit, such that the projected image only depicts the actual pose change of such rigid unit from the first time to the second time.

[0116] Method 400 further includes measuring the variation between a first and a second pose of the rigid unit, as depicted in the projected image (step 428). As described above, the projected image comprises an image of each rigid unit from a second image that has been projected onto the first image using an average of a selected homography. Therefore, any pose variation between two corresponding anatomical units in the projected image can be assumed to reflect an actual pose change of the rigid unit. This pose variation can be measured to produce one or more rotation angles, translational distances, and / or other parameters describing the movement of the rigid unit from a first time to a second time. In some embodiments, the measured quantity can be compared to a desired quantity (such as that reflected in a treatment plan) to produce a percentage of achievement or a similar parameter. In other embodiments, the measured quantity can be compared to a time quantity of the first time versus the second time interval to produce a rate of change that can be used to predict future pose changes of the one or more rigid units, to predict whether and when additional surgery or other treatment will be required, or for any other useful purpose.

[0117] Additionally, the measured quantities (and / or the results of any calculations performed using the measured quantities) may be displayed to the therapist or other users on a user interface such as user interface 110. The measured quantities may be displayed as numbers or converted into indicators (e.g., a red indicator if the quantity is within a predetermined range of unacceptable values; a yellow indicator if the quantity is within a predetermined range of unsatisfactory values; and a green indicator if the quantity is within a predetermined range of acceptable values).

[0118] This disclosure covers embodiments of method 400 that include more or fewer steps than those described above, and / or one or more steps that differ from those described above.

[0119] Figure 5A method 500 for comparing images is described. Method 500 (and / or one or more steps thereof) may be performed, for example, by at least one processor or otherwise executed, which may be part of a system. The at least one processor may be the same as or similar to processor 104 of the computing device 102 described above. The at least one processor may be part of a robot (such as robot 130) or a navigation system (such as navigation system 114). Processors other than any of the processors described herein may also be used to execute method 500. At least one processor may execute method 500 by executing instructions stored in a memory such as memory 106. The instructions may correspond to one or more steps of method 500 described below. These instructions may cause the processor to execute one or more algorithms, such as image processing algorithm 120, segmentation algorithm 122, transformation algorithm 124, homography algorithm 126, and / or registration algorithm 128.

[0120] Method 500 includes identifying a plurality of units in a first image (step 504). The first image can be captured using any imaging device (e.g., imaging device 112) and is captured at a first moment. The first image depicts a portion of a patient's anatomical structure. Identification can be performed using one or more image processing algorithms, such as image processing algorithm 120. Each of these units is a rigid unit and can be an anatomically rigid unit (e.g., a bone anatomy or hard tissue unit) or a rigid implant (e.g., a screw, rod, pin). The plurality of units may include both one or more anatomically rigid units and one or more rigid implants. In some embodiments, identifying the plurality of units in the first image also includes segmenting the plurality of units in the first image, which can be done in any manner described herein or in any other known manner of segmenting units in an image.

[0121] Method 500 further includes identifying a plurality of units in a second image taken after the first image (step 508). Similar to the first image, the second image may be taken using any imaging device (e.g., imaging device 112) and depicts the same (or at least substantially overlapping) portions of patient anatomy as the first image. The second image is taken at a second time after the first time. As with other embodiments of this disclosure, the second time can be days, weeks, months, or even years after the first time. Identification may be performed using one or more image processing algorithms, such as image processing algorithm 120. The plurality of units identified in the second image are the same as the plurality of units identified in the first image. In some embodiments, identifying the plurality of units in the second image further includes segmenting the plurality of units in the second image, which may be done in any manner described herein or in any other known manner of segmenting units in an image.

[0122] Method 500 further includes calculating the homography of each of the plurality of cells (step 512). Step 512 is the same as or similar to step 316 of method 300 and / or step 408 of method 400.

[0123] Method 500 further includes determining a first attitude change attributable to a change in the position of the imaging device and a second attitude change not attributable to a change in the position of the imaging device based on homography (step 516). Step 516 is the same as or similar to a combination of steps 316 and 320 of method 300 and / or a combination of steps 416, 420 and 424 of method 400.

[0124] Method 500 further includes registering the second image with the first image based on the first pose change (step 520). Step 520 is the same as or similar to step 324 of method 300.

[0125] In some embodiments, step 520 may alternatively include updating the preoperative model based on individual portions of each transformation. The preoperative model may have been generated, for example, based on preoperative images, and the update may include updating each rigid cell depicted in the preoperative model to reflect any change in the position of that rigid cell from the time the preoperative image was taken to the time the second image was taken. In this embodiment, the second image may be an intraoperative image or a postoperative image.

[0126] Similarly, in some embodiments, step 520 may alternatively include updating the registration of one of the robot space or navigation space with the image space based on either a common portion of each transformation or a separate portion of each transformation. This update can help maintain accurate registration, which in turn can increase the accuracy of surgical procedures.

[0127] This disclosure covers embodiments of method 500 that include more or fewer steps than those described above, and / or one or more steps that differ from those described above.

[0128] As stated above, this disclosure covers those having more than Figure 3 , Figure 4 and Figure 5 The methods that identify all steps with fewer steps (and the corresponding descriptions of methods 300, 400, and 500), as well as those exceeding... Figure 3 , Figure 4 and Figure 5 The method includes additional steps identified in the steps described herein (and corresponding descriptions of methods 300, 400, and 500). This disclosure also covers methods that include one or more steps from one method described herein and one or more steps from another method described herein. Any correlation described herein may be or includes registration or any other correlation.

[0129] The foregoing is not intended to limit this disclosure to the one or more forms disclosed herein. In the foregoing specific embodiments, for example, for the purpose of simplifying this disclosure, various features of this disclosure are grouped together in one or more aspects, embodiments, and / or configurations. Features of aspects, embodiments, and / or configurations of this disclosure may be combined in alternative aspects, embodiments, and / or configurations other than those discussed above. The approach of this disclosure should not be construed as reflecting an intention that the claims require more features than expressly recited in each claim. Rather, as reflected in the following claims, aspects of the invention lie in fewer than all the features of a single foregoing aspect, embodiment, and / or configuration. Therefore, the following claims are hereby incorporated into this specific embodiment, wherein each claim exists independently as a separate preferred embodiment of this disclosure.

[0130] Furthermore, while the foregoing has already included descriptions of one or more aspects, embodiments, and / or configurations, as well as certain variations and modifications, other variations, combinations, and modifications may be made within the scope of this disclosure, for example, within the skill and knowledge of those skilled in the art, upon understanding of this disclosure. It is intended to obtain, to the permissible extent, rights including alternative aspects, embodiments, and / or configurations, including alternative, interchangeable, and / or equivalent structures, functions, scopes, or steps of those claimed, regardless of whether such alternative, interchangeable, and / or equivalent structures, functions, scopes, or steps are disclosed herein, and not to disclose for use in any patentable subject matter.

Claims

1. A method for correlated images captured at different times, comprising: Receive a first image of the patient's anatomical structure, the first image being generated and depicting a plurality of rigid units in a first instant, each of the plurality of rigid units being movable relative to at least one other rigid unit of the plurality of rigid units; Receive a second image of the patient's anatomical structure, the second image being generated at a second time after the first time and depicting the plurality of rigid units; For each of the plurality of rigid elements, a transformation from the first image to the second image is determined to generate a set of transformations; Using the set of transformations, common components attributable to camera pose changes are identified by separating transformations in the set of transformations that originate solely from camera pose changes from transformations in the set of transformations that originate from a combination of camera pose changes and rigid pose changes; and The set of transformations is used to identify individual portions of each transformation attributable to the attitude changes of the rigid unit by leveraging the clustering.

2. The method according to claim 1, further comprising: The second image is registered with the first image based on the identified common parts of each transformation.

3. The method according to claim 1, further comprising: The preoperative model is updated based on the individual parts of each transformation.

4. The method according to claim 1, further comprising: The registration of either the robot space or the navigation space with the image space is updated based on either the common portion of each transformation or the individual portion of each transformation.

5. The method of claim 1, wherein each transformation is homography, and the set of transformations is a set of homography.

6. The method of claim 1, wherein the method further comprises determining the most coherent clusters and, in the step of projecting the second image onto the first image, using the average value of the most coherent clusters as a transformation corresponding to the camera pose change.

7. The method of claim 2, wherein the registration step comprises associating both the first image and the second image with a common vector space.

8. The method according to claim 1, wherein the first image is a preoperative image.

9. The method of claim 1, wherein at least one of the first image and the second image is an intraoperative image.

10. The method of claim 1, wherein determining the transformation comprises identifying at least four points on each of the plurality of rigid elements as depicted in the first image, and corresponding at least four points on each of the plurality of rigid elements as depicted in the second image.

11. The method according to claim 1, wherein the first image and the second image are two-dimensional.

12. The method according to claim 1, wherein the first image and the second image are three-dimensional.

13. The method of claim 1, wherein the plurality of rigid units comprises a plurality of vertebrae of the patient's spine.

14. The method of claim 1, wherein the plurality of rigid units comprises at least one implant.

15. The method of claim 1, further comprising quantifying the attitude change of at least one of the plurality of rigid elements from the first time to the second time.

16. A method for correlated images captured at different times, comprising: Each rigid unit is segmented in a first image of a plurality of rigid units captured at a first time and in a second image of the plurality of rigid units captured at a second time after the first time. The homography of each of the plurality of rigid elements is calculated to generate a set of homography, each homography relating the rigid element as depicted in the first image to the rigid element as depicted in the second image; The set of homography is arranged into a homography cluster to identify the most similar homography, which corresponds to a rigid unit that has not moved from the first time to the second time but whose pose change in the second image relative to the first image can be fully attributed to the camera pose change. Select the most coherent homography cluster; and The average value of the selected homography clusters is used to project each of the plurality of rigid elements depicted in the second image onto the first image to produce a projected image.

17. The method of claim 16, wherein the second time is at least one month after the first time.

18. The method of claim 16, wherein the second time is at least one year after the first time.

19. The method of claim 16, wherein at least one of the plurality of rigid units is an implant.

20. The method of claim 16, wherein the plurality of rigid units comprise a plurality of vertebrae of the patient's spine.

21. The method of claim 16, further comprising measuring at least one of an angle or a distance corresponding to an attitude change of one of the plurality of rigid elements as reflected in the projected image.

22. The method of claim 16, further comprising removing from the set of homography any homography affected by one or more of the compression fractures or osteophytes depicted in the second image but not the first image.

23. The method of claim 16, wherein calculating the homography of each of the plurality of rigid elements includes identifying edge points of the vertebral endplate.

24. A system for correlating images captured at different times, comprising: At least one processor; and A memory that stores instructions for execution by the processor, which, when executed, cause the processor to: Identify multiple units in the first image generated in the first instant; Identify the plurality of units in a second image generated at a second time after the first time; The first image and the second image are used to calculate the homography of each of the plurality of cells to generate a set of homography; as well as Based on the set of homography, it is determined that: one or more units of the plurality of units in the second image have a first pose change relative to the first image, and the first pose change can be attributed to the change in the position of the imaging device from the first image to the second image relative to the plurality of units; And a second pose change of at least one of the plurality of units in the second image relative to the first image, the second pose change not attributable to a change in the position of the imaging device.

25. The system of claim 24, wherein the memory stores additional instructions for execution by the processor, the additional instructions further causing the processor, when executed, to: The second image is registered with the first image based on the first pose change.

26. The system of claim 24, wherein the memory stores additional instructions for execution by the processor, the additional instructions further causing the processor, when executed, to: The preoperative model is updated based on the second pose change.

27. The system of claim 24, wherein the memory stores additional instructions for execution by the processor, the additional instructions further causing the processor, when executed, to: The registration between the robot space or the navigation space and the image space is updated based on either the first or the second pose change.

Images