Systems and methods for patient registration using 2d image planes

Abstract

A system or method according to the present disclosure includes a processor; and a memory coupled to the processor and storing data thereon that, when processed by the processor, enables the processor to: receive three-dimensional (3D) scan data depicting a portion of a patient generated with an imaging device; project the 3D scan data into a plurality of two-dimensional (2D) image planes including a first 2D image plane and a second 2D image plane; match, based on a geometry of the portion of the patient, an area of the first 2D image plane to an area of the second 2D image plane that corresponds to a common location in 3D space; and register, based on the matching, the 3D scan data and the imaging device.

Description

SYSTEMS AND METHODS FOR PATIENT REGISTRATION USING 2D IMAGE PLANESCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 536,253 filed 1 September 2023. The entire disclosure of which is incorporated by reference herein.BACKGROUND

[0002] The present disclosure is generally directed to image registration, and relates more particularly to registration using stereo image data.

[0003] Imaging may be used by a medical provider for diagnostic and / or therapeutic purposes during a surgery or surgical procedure. Patient anatomy can change over time, particularly following placement of a medical implant in the patient anatomy.BRIEF SUMMARY

[0004] Example aspects of the present disclosure include:

[0005] A system according to at least one embodiment of the present disclosure comprises: a processor; and a memory coupled to the processor and storing data thereon that, when processed by the processor, enables the processor to: receive three-dimensional (3D) scan data depicting a portion of a patient generated with an imaging device; project the 3D scan data into a plurality of two- dimensional (2D) image planes including a first 2D image plane and a second 2D image plane; match, based on a geometry of the portion of the patient, an area of the first 2D image plane to an area of the second 2D image plane that corresponds to a common location in 3D space; and register, based on the matching, the 3D scan data and the imaging device.

[0006] Any of the aspects herein, wherein the geometry of the portion of the patient is determined from a 3D image of the portion of the patient.

[0007] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, enables the processor to: generate a similarity metric indicating an amount of similarity between the area of the first 2D image plane and the area of the second 2D image plane.

[0008] Any of the aspects herein, wherein the amount of similarity is based on an entropy of the area of the first 2D image plane and an entropy of the area of the second 2D image plane.

[0009] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, enables the processor to: iteratively optimize an objective function associated with the similarity metric.

[0010] Any of the aspects herein, wherein as part of the iterative optimization, the processor further: generates an initial guess for a transformation matrix that transforms coordinates from an imaging device coordinate system associated with the imaging device to an exam coordinate system associated with the 3D scan data; generates a second similarity metric that indicates an amount of similarity between a second area of the first 2D image plane and a second area of the second 2D image plane; determines, based on a combination of the first similarity metric and the second similarity metric, the objective function; and adjusts the transformation matrix using the objective function.

[0011] Any of the aspects herein, wherein the geometry of the portion of the patient comprises a contour of an anatomical element of the patient.

[0012] A system according to at least one embodiment of the present disclosure comprises: an imaging device; a processor; and a memory coupled to the processor and storing data thereon that, when processed by the processor, enables the processor to: receive three-dimensional (3D) scan data depicting a portion of a patient generated with the imaging device; project the 3D scan data into a plurality of two-dimensional (2D) image planes including a first 2D image plane and a second 2D image plane; match, based on a geometry of the portion of the patient, an area of the first 2D image plane to an area of the second 2D image plane that corresponds to a common location in 3D space; and register, based on the matching, the 3D scan data and the imaging device.

[0013] Any of the aspects herein, wherein the geometry of the portion of the patient is determined from a 3D image of the portion of the patient.

[0014] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, enables the processor to: generate a similarity metric indicating an amount of similarity between the area of the first 2D image plane and the area of the second 2D image plane.

[0015] Any of the aspects herein, wherein the amount of similarity is based on an entropy of the area of the first 2D image plane and an entropy of the area of the second 2D image plane.

[0016] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, enables the processor to: iteratively optimize an objective function associated with the similarity metric.

[0017] Any of the aspects herein, wherein as part of the iterative optimization, the processor further: generates an initial guess for a transformation matrix that transforms coordinates from an imaging device coordinate system associated with the imaging device to an exam coordinate system associated with the 3D scan data; generates a second similarity metric that indicates an amount of similarity between a second area of the first 2D image plane and a second area of the second 2D image plane; determines, based on a combination of the first similarity metric and the second similarity metric, the objective function; and adjusts the transformation matrix using the objective function.

[0018] Any of the aspects herein, wherein the geometry of the portion of the patient comprises a contour of an anatomical element of the patient.

[0019] A method according to at least one embodiment of the present disclosure comprises: receiving three-dimensional (3D) scan data depicting a portion of a patient generated with an imaging device; projecting the 3D scan data into a plurality of two-dimensional (2D) image planes including a first 2D image plane and a second 2D image plane; matching, based on a geometry of the portion of the patient, an area of the first 2D image plane to an area of the second 2D image plane that corresponds to a common location in 3D space; and registering, based on the matching, the 3D scan data and the imaging device.

[0020] Any of the aspects herein, wherein the geometry of the portion of the patient is determined from a 3D image of the portion of the patient.

[0021] Any of the aspects herein, further comprising: generating a similarity metric indicating an amount of similarity between the area of the first 2D image plane and the area of the second 2D image plane.

[0022] Any of the aspects herein, further comprising: iteratively optimizing an objective function associated with the similarity metric.

[0023] Any of the aspects herein, wherein iteratively optimizing the objective function further comprises: generating an initial guess for a transformation matrix that transforms coordinates from an imaging device coordinate system associated with the imaging device to an exam coordinate system associated with the 3D scan data; generating a second similarity metric that indicates an amount of similarity between a second area of the first 2D image plane and a second area of the second 2D image plane; determining, based on a combination of the first similarity metric and the second similarity metric, the objective function; and adjusting the transformation matrix using the objective function.

[0024] Any of the aspects herein, wherein the geometry of the portion of the patient comprises information about a contour of an anatomical element of the patient.

[0025] Any aspect in combination with any one or more other aspects.

[0026] Any one or more of the features disclosed herein.

[0027] Any one or more of the features as substantially disclosed herein.

[0028] Any one or more of the features as substantially disclosed herein in combination with any one or more other features as substantially disclosed herein.

[0029] Any one of the aspects / features / embodiments in combination with any one or more other aspects / features / embodiments.

[0030] Use of any one or more of the aspects or features as disclosed herein.

[0031] It is to be appreciated that any feature described herein can be claimed in combination with any other feature(s) as described herein, regardless of whether the features come from the same described embodiment.

[0032] The details of one or more aspects of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques described in this disclosure will be apparent from the description and drawings, and from the claims.

[0033] The phrases “at least one”, “one or more”, and “and / or” are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions “at least one of A, B and C”, “at least one of A, B, or C”, “one or more of A, B, and C”, “one or more of A, B, or C” and “A, B, and / or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B and C together. When each one of A, B, and C in the above expressions refers to an element, such as X, Y, and Z, or class of elements, such as Xl-Xn, Y 1-Ym, and Zl-Zo, the phrase is intended to refer to a single element selected from X, Y, and Z, a combination of elements selected from the same class (e.g., XI and X2) as well as a combination of elements selected from two or more classes (e.g., Y 1 and Zo).

[0034] The term “a” or “an” entity refers to one or more of that entity. As such, the terms “a” (or “an”), “one or more” and “at least one” can be used interchangeably herein. It is also to be noted that the terms “comprising”, “including”, and “having” can be used interchangeably.

[0035] The preceding is a simplified summary of the disclosure to provide an understanding of some aspects of the disclosure. This summary is neither an extensive nor exhaustive overview of the disclosure and its various aspects, embodiments, and configurations. It is intended neither to identify key or critical elements of the disclosure nor to delineate the scope of the disclosure but to present selected concepts of the disclosure in a simplified form as an introduction to the more detaileddescription presented below. As will be appreciated, other aspects, embodiments, and configurations of the disclosure are possible utilizing, alone or in combination, one or more of the features set forth above or described in detail below.

[0036] Numerous additional features and advantages of the present disclosure will become apparent to those skilled in the art upon consideration of the embodiment descriptions provided hereinbelow.BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0037] The accompanying drawings are incorporated into and form a part of the specification to illustrate several examples of the present disclosure. These drawings, together with the description, explain the principles of the disclosure. The drawings simply illustrate preferred and alternative examples of how the disclosure can be made and used and are not to be construed as limiting the disclosure to only the illustrated and described examples. Further features and advantages will become apparent from the following, more detailed, description of the various aspects, embodiments, and configurations of the disclosure, as illustrated by the drawings referenced below.

[0038] Fig. 1 is a block diagram of a system according to at least one embodiment of the present disclosure;

[0039] Fig. 2A shows additional aspects of the system according to at least one embodiment of the present disclosure;

[0040] Fig. 2B shows a projection of three-dimensional (3D) scan data into a plurality of two- dimensional (2D) image planes according to at least one embodiment of the present disclosure;

[0041] Fig. 2C shows aspects the plurality of 2D image planes according to at least one embodiment of the present disclosure;

[0042] Fig. 3 is a flowchart according to at least one embodiment of the present disclosure; and

[0043] Fig. 4 is a flowchart according to at least one embodiment of the present disclosure.DETAILED DESCRIPTION

[0044] It should be understood that various aspects disclosed herein may be combined in different combinations than the combinations specifically presented in the description and accompanying drawings. It should also be understood that, depending on the example or embodiment, certain acts or events of any of the processes or methods described herein may be performed in a different sequence, and / or may be added, merged, or left out altogether (e.g., all described acts or events may not be necessary to carry out the disclosed techniques according to different embodiments of the present disclosure). In addition, while certain aspects of this disclosure are described as beingperformed by a single module or unit for purposes of clarity, it should be understood that the techniques of this disclosure may be performed by a combination of units or modules associated with, for example, a computing device and / or a medical device.

[0045] In one or more examples, the described methods, processes, and techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware -based processing unit. Alternatively or additionally, functions may be implemented using machine learning models, neural networks, artificial neural networks, or combinations thereof (alone or in combination with instructions). Computer-readable media may include non-transitory computer-readable media, which corresponds to a tangible medium such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer).

[0046] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors (e.g., Intel Core i3, i5, i7, or i9 processors; Intel Celeron processors; Intel Xeon processors; Intel Pentium processors; AMD Ryzen processors; AMD Athlon processors; AMD Phenom processors; Apple A 10 or 10X Fusion processors; Apple Al l, A 12, A12X, A12Z, or Al 3 Bionic processors; or any other general purpose microprocessors), graphics processing units (e.g., Nvidia GeForce RTX 2000-series processors, Nvidia GeForce RTX 3000-series processors, AMD Radeon RX 5000-series processors, AMD Radeon RX 6000-series processors, or any other graphics processing units), application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor” as used herein may refer to any of the foregoing structure or any other physical structure suitable for implementation of the described techniques. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0047] Before any embodiments of the disclosure are explained in detail, it is to be understood that the disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The disclosure is capable of other embodiments and of being practiced or of being carried out in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” or “having” and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Further, the present disclosure may use examples toillustrate one or more aspects thereof. Unless explicitly stated otherwise, the use or listing of one or more examples (which may be denoted by “for example,” “by way of example,” “e.g.,” “such as,” or similar language) is not intended to and does not limit the scope of the present disclosure.

[0048] During surgeries or surgical procedures, a patient is registered to a coordinate system of a surgical navigation system using stereo image data. The process of registration involves registering a 3D image (e.g., patient exam data) to a 3D scene (e.g., a patient on an operating table) by first projecting the 3D scene onto two or more 2D image planes. One approach for completing the registration is to de -project the 2D images to 3D space by finding corresponding pixels in multiple 2D image planes and using triangulation to determine the position of the area represented by the pixel relative to the cameras in 3D space. The stereo imaging system may search pixel-by-pixel looking for correspondences that can be de -projected to construct a 3D point cloud. The 3D image of the scene and the 3D exam can then be registered using point cloud registration techniques. For a stereo imaging system without a priori information such as known fiducials or a known light pattern, the process of de -projecting from the 2D space to the 3D space requires identifying corresponding pixels using similarity metrics between the images. In this workflow, the matching of the corresponding pixels may limit the accuracy of the registration. According to at least one embodiment of the present disclosure, systems and methods for registering the exam data in the 2D image planes provide a constraint on this correspondence problem to improve the accuracy of the registration.

[0049] According to at least one embodiment of the present disclosure, a known geometry is used to constrain the correspondence problem. Instead of searching for individual corresponding pixels, the systems and methods of embodiments disclosed herein may search for a set of correspondences that best describe a known geometry. In some cases such as patient registration, the known geometry may be a 3D medical image (e.g., a preoperative 3D scan of the patient). The registration process may be performed entirely in 2D image planes. An algorithm according to at least one example embodiment may begin with an initial guess for a pose (e.g., a position and an orientation) of the patient relative to a camera. The 3D exam data of the patient may then be projected onto the 2D image planes. For some or all points in the exam point cloud, a similarity metric is calculated for the region surrounding the projected pixel locations in the 2D images. The similarity metrics of the point(s) are then combined into an overall objective function describing how well the initial guess fits the raw image data. The objective function can then be used to adjust the pose of the patient and / or adjust the transformation matrix, and the process may be repeated until a best fit is achieved (e.g., until the objective function is optimized). In some examples, other image -based registrationmetrics, such as mutual information, contour matching, combinations thereof, and / or the like can be included in the objective function to further improve registration accuracy.

[0050] Embodiments of the present disclosure beneficially improve the accuracy of photo-based patient registration by constraining the stereo matching problem using known geometric data, which may enable single-snapshot patient registration. Embodiments of the present disclosure may also beneficially enable a multi-camera and / or stereo camera registration system. The resolution of the stereo camera may be determined by the resolution of the image sensors used in the stereo camera. The resolution of the stereo camera can be limited by the need to check for correspondences between all captured pixels. As the resolution increases, the processing time also increases. Using systems and methods described herein, only projected points from the exam point cloud are checked for correspondence. Accordingly, increased resolution of the camera sensors does not increase processing time.

[0051] Embodiments of the present disclosure provide technical solutions to one or more of the problems of (1) high resolution stereo camera systems creating increased processing times, (2) inaccurate registration, and (3) inaccurate correspondence matching.

[0052] Turning first to Fig. 1, a block diagram of a system 100 according to at least one embodiment of the present disclosure is shown. The system 100 may be used to register a patient to a surgical navigation system coordinate system; to register patient exam data to a camera or other imaging device; to control, pose, and / or otherwise manipulate a surgical mount system and / or surgical tools attached thereto; and / or to carry out one or more other aspects of one or more of the methods disclosed herein. The system 100 comprises a computing device 102, one or more imaging devices 112, a robot 114, a navigation system 118, a database 130, and / or a cloud or other network 134. Systems according to other embodiments of the present disclosure may comprise more or fewer components than the system 100. For example, the system 100 may not include the robot 114, one or more components of the computing device 102, the database 130, and / or the cloud 134.

[0053] The computing device 102 is illustrated to include a processor 104, a memory 106, a communication interface 108, and a user interface 110. Computing devices according to other embodiments of the present disclosure may comprise more or fewer components than the computing device 102.

[0054] The processor 104 of the computing device 102 may be any processor described herein or any similar processor. The processor 104 may be configured to execute instructions stored in the memory 106, which instructions may cause the processor 104 to carry out one or more computing steps utilizing or based on data received from the imaging device 112, the robot 114, the navigationsystem 118, the database 130, and / or the cloud 134. The processor 104 may be or comprise one or more digital signal processors (DSPs), general purpose microprocessors (e.g., Intel Core i3, i5, i7, or i9 processors; Intel Celeron processors; Intel Xeon processors; Intel Pentium processors; AMD Ryzen processors; AMD Athlon processors; AMD Phenom processors; Apple A10 or 10X Fusion processors; Apple Al l, A12, A12X, A12Z, or A13 Bionic processors; or any other general purpose microprocessors), graphics processing units (e.g., Nvidia GeForce RTX 2000-series processors, Nvidia GeForce RTX 3000-series processors, AMD Radeon RX 5000-series processors, AMD Radeon RX 6000-series processors, or any other graphics processing units), application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry.

[0055] The memory 106 may be or comprise RAM, DRAM, SDRAM, other solid-state memory, any memory described herein, or any other tangible, non-transitory memory for storing computer- readable data and / or instructions. The memory 106 may store information or data useful for completing, for example, any step of the methods 300 and / or 400 described herein, or of any other methods. The memory 106 may store, for example, instructions and / or machine learning models that support one or more functions of the computing device 102, the imaging devices 112, the navigation system 118, and / or the like. For instance, the memory 106 may store content (e.g., instructions and / or machine learning models) that, when executed by the processor 104, enable image processing 120, segmentation 122, transformation 124, and / or registration 128. Such content, if provided as in instruction, may, in some embodiments, be organized into one or more applications, modules, packages, layers, or engines. Alternatively or additionally, the memory 106 may store other types of content or data (e.g., machine learning models, artificial neural networks, deep neural networks, etc.) that can be processed by the processor 104 to carry out the various method and features described herein. Thus, although various contents of memory 106 may be described as instructions, it should be appreciated that functionality described herein can be achieved through use of instructions, algorithms, and / or machine learning models. The data, algorithms, and / or instructions may cause the processor 104 to manipulate data stored in the memory 106 and / or received from or via the imaging device 112, the robot 114, the database 130, and / or the cloud 134.

[0056] The communication interface 108 may be used for receiving image data or other information from an external source (such as the imaging device 112, the robot 114, the navigation system 118, the database 130, the cloud 134, and / or any other system or component not part of the system 100), and / or for transmitting instructions, images, or other information to an external system or device (e.g., another computing device 102, the imaging device 112, the robot 114, the navigationsystem 118, the database 130, the cloud 134, and / or any other system or component not part of the system 100). The communication interface 108 may comprise one or more wired interfaces (e.g., a USB port, an Ethernet port, a Firewire port) and / or one or more wireless transceivers or interfaces (configured, for example, to transmit and / or receive information via one or more wireless communication protocols such as 802.1 la / b / g / n, Bluetooth, NFC, ZigBee, and so forth). In some embodiments, the communication interface 108 may be useful for enabling the device 102 to communicate with one or more other processors 104 or computing devices 102, whether to reduce the time needed to accomplish a computing-intensive task or for any other reason.

[0057] The computing device 102 may also comprise one or multiple user interfaces 110. The user interface(s) 110 may be or comprise a keyboard, mouse, trackball, monitor, television, screen, touchscreen, and / or any other device for receiving information from a user and / or for providing information to a user. The user interface(s) 110 may be used, for example, to receive a user selection or other user input regarding any step of any method described herein. Notwithstanding the foregoing, any required input for any step of any method described herein may be generated automatically by the system 100 (e.g., by the processor 104 or another component of the system 100) or received by the system 100 from a source external to the system 100. In some embodiments, the user interface 110 may be useful to allow a surgeon or other user to modify instructions to be executed by the processor 104 according to one or more embodiments of the present disclosure, and / or to modify or adjust a setting of other information displayed on the user interface 110 or corresponding thereto.

[0058] Although the user interface 110 is shown as part of the computing device 102, in some embodiments, the computing device 102 may utilize a user interface 110 that is housed separately from one or more remaining components of the computing device 102. In some embodiments, the user interface 110 may be located proximate one or more other components of the computing device 102, while in other embodiments, the user interface 110 may be located remotely from one or more other components of the computer device 102.

[0059] The imaging device 112 may be operable to image anatomical feature(s) (e.g., a bone, veins, tissue, etc.) and / or other aspects of patient anatomy to yield image data (e.g., image data depicting or corresponding to a bone, veins, tissue, etc.). “Image data” as used herein refers to the data generated or captured by an imaging device 112, including in a machine -readable form, a graphical / visual form, and in any other form. In some cases, the image data may be or comprise 3D image data 132 generated by one or more 3D imaging device(s) 140 (e.g., an 0-arm, a C-arm, a G-arm, a CT scanner, etc.) and / or 2D image data 136 generated by one or more 2D imaging device(s) 144 (e.g., an emitter / detector pair).

[0060] In various examples, the image data may comprise data corresponding to an anatomical feature of a patient, or to a portion thereof. The image data may be or comprise a preoperative image, an intraoperative image, a postoperative image, or an image taken independently of any surgical procedure. The imaging device 112, the 3D imaging device(s) 140, and / or the 2D imaging device(s) 144 may be capable of taking a 2D image or a 3D image to yield the image data. The imaging device 112 may be or comprise, for example, a stereo camera, an ultrasound scanner (which may comprise, for example, a physically separate transducer and receiver, or a single ultrasound transceiver), an 0-arm, a C-arm, a G-arm, or any other device utilizing X-ray-based imaging (e.g., a fluoroscope, a CT scanner, or other X-ray machine), a magnetic resonance imaging (MRI) scanner, an optical coherence tomography (OCT) scanner, an endoscope, a microscope, an optical camera, a thermographic camera (e.g., an infrared camera), a radar system (which may comprise, for example, a transmitter, a receiver, a processor, and one or more antennae), or any other imaging device 112 suitable for obtaining images of an anatomical feature of a patient. The imaging device 112 may be contained entirely within a single housing, or may comprise a transmitter / emitter and a receiver / detector that are in separate housings or are otherwise physically separated. In some embodiments, a first imaging device 112 may be used to obtain first image data (e.g., a first image) at a first time, and a second imaging device 112 may be used to obtain second image data (e.g., a second image) at a second time after the first time. In some examples (and as discussed in further detail below) the first imaging device can be an imaging device that is used for navigation (e.g., in conjunction with the navigation system 118). The first imaging device may be or comprise, for example, any of the example imaging devices described above (an ultrasound scanner, an 0-arm, a C-arm, a G-arm, or any other device utilizing X-ray-based imaging, a magnetic resonance imaging scanner, an OCT scanner, an endoscope, a microscope, an optical camera, a thermographic camera, a radar system, a stereo camera, etc.). The second imaging device may be an imaging device used for registration (e.g., to facilitate alignment of patient scan data with the patient in the surgical environment). The second imaging device may be or comprise a stereo camera, as discussed in further detail below.

[0061] In some embodiments, the imaging device 112 may comprise more than one imaging device 112. For example, a first imaging device may provide first image data and / or a first image, and a second imaging device may provide second image data and / or a second image. In still other embodiments, the same imaging device may be used to provide both the first image data and thesecond image data, and / or any other image data described herein. The imaging device 112 may be operable to generate a stream of image data. For example, the imaging device 112 may be configured to operate with an open shutter, or with a shutter that continuously alternates between open and shut so as to capture successive images. For purposes of the present disclosure, unless specified otherwise, image data may be considered to be continuous and / or provided as an image data stream if the image data represents two or more frames per second. In some embodiments, reference markers (e.g., navigation markers) may be placed on the imaging device 112 and / or any other object in the surgical space. The reference markers may be tracked by the navigation system 118, and the results of the tracking may be used by an operator of the system 100 or any component thereof.

[0062] In some embodiments, the imaging device 112 may be or comprise a stereo camera or other similar imaging device with two or more lenses and separate image sensors for each lens. Each image sensor may generate image information to the computing device 102, which may use image processing 120 to generate an image from the image information generated by the image sensor. The image sensors may be physically spaced apart or otherwise separated from one another in the imaging device 112, such that each image sensor captures a different view of an object imaged by the imaging device 112. In some cases, the stereo camera may be used to facilitate registration of a patient to a coordinate system associated with the navigation system, as discussed in further detail below.

[0063] The robot 114 may be any surgical robot or surgical robotic system. The robot 114 may be or comprise, for example, the Mazor X™ Stealth Edition robotic guidance system. The robot 114 may be configured to position the imaging device 112 at one or more precise position(s) and orientation / s), and / or to return the imaging device 112 to the same position(s) and orientation(s) at a later point in time. The robot 114 may additionally or alternatively be configured to manipulate a surgical tool (whether based on guidance from the navigation system 118 or not) to accomplish or to assist with a surgical task. In some embodiments, the robot 114 may be configured to hold and / or manipulate an anatomical element during or in connection with a surgical procedure. The robot 114 may comprise one or more robotic arms 116. In some embodiments, the robotic arm 116 may comprise a first robotic arm and a second robotic arm, though the robot 114 may comprise more than two robotic arms. In some embodiments, one or more of the robotic arms 116 may be used to hold and / or maneuver the imaging device 112. In embodiments where the imaging device 112 comprises two or more physically separate components (e.g., a transmitter and receiver), one robotic arm 116 may hold one such component, and another robotic arm 116 may hold another such component.Each robotic arm 116 may be positionable independently of the other robotic arm. The robotic arms 116 may be controlled in a single, shared coordinate space, or in separate coordinate spaces.

[0064] The robot 114, together with the robotic arm 116, may have, for example, one, two, three, four, five, six, seven, or more degrees of freedom. Further, the robotic arm 116 may be positioned or positionable in any pose, plane, and / or focal point. The pose includes a position and an orientation. As a result, an imaging device 112, surgical tool, or other object held by the robot 114 (or, more specifically, by the robotic arm 116) may be precisely positionable in one or more needed and specific positions and orientations.

[0065] The robotic arm(s) 116 may comprise one or more sensors that enable the processor 104 (or a processor of the robot 114) to determine a precise pose in space of the robotic arm (as well as any object or element held by or secured to the robotic arm).

[0066] In some embodiments, reference markers (e.g., navigation markers) may be placed on the robot 114 (including, e.g., on the robotic arm 116), the imaging device 112, or any other object in the surgical space. The reference markers may be tracked by the navigation system 118, and the results of the tracking may be used by the robot 114 and / or by an operator of the system 100 or any component thereof. In some embodiments, the navigation system 118 can be used to track other components of the system (e.g., imaging device 112) and the system can operate without the use of the robot 114 (e.g., with the surgeon manually manipulating the imaging device 112 and / or one or more surgical tools, based on information and / or instructions generated by the navigation system 118, for example).

[0067] The navigation system 118 may provide navigation during an operation. The navigation system 118 may be any now-known or future-developed navigation system, including, for example, the Medtronic StealthStation™ S8 surgical navigation system or any successor thereof. The navigation system 118 may include one or more cameras or other sensor(s) for tracking one or more reference markers, navigated trackers, or other objects within the operating room or other room in which some or all of the system 100 is located. The one or more cameras may be optical cameras, infrared cameras, or other cameras. In some embodiments, the navigation system 118 may comprise one or more electromagnetic sensors. In various embodiments, the navigation system 118 may be used to track a position and orientation (e.g., a pose) of the imaging device 112, the robot 114 and / or the robotic arm 116, and / or one or more surgical tools (or, more particularly, to track a pose of a navigated tracker attached, directly or indirectly, in fixed relation to the one or more of the foregoing). The navigation system 118 may include a display for displaying one or more images from an external source (e.g., the computing device 102, imaging device 112, or other source) or fordisplaying an image and / or video stream from the one or more cameras or other sensors of the navigation system 118. In some embodiments, the system 100 can operate without the use of the navigation system 118. The navigation system 118 may be configured to provide guidance to a surgeon or other user of the system 100 or a component thereof or to any other element of the system 100 (e.g., the robot 114) regarding, for example, a pose of one or more anatomical elements, whether or not a tool is in the proper trajectory, and / or how to move a tool into the proper trajectory to carry out a surgical task according to a preoperative or other surgical plan.

[0068] The database 130 may store information that correlates one coordinate system to another (e.g., a patient coordinate system to a navigation coordinate system or vice versa). The database 130 may additionally or alternatively store, for example, one or more surgical plans (including, for example, pose information about a target and / or image information about a patient’s anatomy at and / or proximate the surgical site, for use by the robot 114, the navigation system 118, and / or a user of the computing device 102 or of the system 100); one or more images useful in connection with a surgery to be completed by or with the assistance of one or more other components of the system 100; and / or any other useful information. The database 130 may be configured to provide any such information to the computing device 102 or to any other device of the system 100 or external to the system 100, whether directly or via the cloud 134. In some embodiments, the database 130 may be or comprise part of a hospital image storage system, such as a picture archiving and communication system (PACS), a health information system (HIS), and / or another system for collecting, storing, managing, and / or transmitting electronic medical records including image data.

[0069] The cloud 134 may be or represent the Internet or any other wide area network. The computing device 102 may be connected to the cloud 134 via the communication interface 108, using a wired connection, a wireless connection, or both. In some embodiments, the computing device 102 may communicate with the database 130 and / or an external device (e.g., a computing device) via the cloud 134.

[0070] The system 100 or similar systems may be used, for example, to carry out one or more aspects of any of the methods 300 and / or 400 described herein. The system 100 or similar systems may also be used for other purposes.

[0071] Turn next to Figs. 2A-2C, additional aspects of the system 100 are shown in accordance with embodiments of the present disclosure. Fig. 2A depicts an imaging device 202 that operates as a stereo camera with a first image sensor 204 and a second image sensor 208 that can be used to capture stereo images of the patient 212. The imaging device 202 may be similar to or the same as the imaging device 112 or any other imaging device depicted and / or described herein.

[0072] The first image sensor 204 and the second image sensor 208 are each capable of generating information or image data that can be processed to produce a pair of 2D images of the patient 212. The first image sensor 204 and the second image sensor 208 may be separated by a predetermined or otherwise known distance, such that the field of view (FOV) of the first image sensor 204 is different than the FOV of the second image sensor 208. For example, the first image sensor 204 may have a first FOV 216 while the second image sensor 208 may have a different second FOV 220. As a result, data generated by the first image sensor 204 is different from the data generated by the second image sensor 208 when the imaging device 202 images the patient 212, such that each image of the pair of 2D images depicts features of the patient 212 from different angles, directions, and / or orientations. In other words, a first 2D image of the pair of 2D images depicts features of the patient 212 in a first orientation, while a second 2D image of the pair of 2D images depicts the features of the patient 212 from a different angle, direction, and / or orientation.

[0073] As depicted in Fig. 2 A, the first image sensor 204 is positioned left of a centerline 214 of the patient 212 while the second image sensor 208 is positioned to the right of the centerline 214 of the patient 212. The resulting data from the first image sensor 204, when processed by image processing 120, may result in an image depicting a left-side view of the patient 212 (also referred to herein as a left eye view of the patient 212). Similarly, the resulting data from the second image sensor 208, when processed by the image processing 120, may result in an image depicting a rightside view of the patient 212 (also referred to herein as a right eye view of the patient 212). In some cases, one or both of the 2D images (e.g., the right-side view and / or the left-side view) may be rendered to a display to enable a user of the system 100 to view the 2D image(s). It should be understood that, while embodiments of the present disclosure discuss left-side and right-side views of the patient 212, additional or alternative imaging of the patient may occur, and that any portion of the patient 212 may be imaged by the stereo camera to produce a pair of 2D images depicting the portion of the patient 212.

[0074] The pose of the first image sensor 204 relative to the second image sensor 208 (and vice versa) may be known or determined by the system 100. In some cases, the pose information may be stored in the database 130. Additionally or alternatively, the pose information may be determined by the navigation system 118 based on the pose of tracked fiducials in a known pose relative to the first image sensor 204 and / or the second image sensor 208.

[0075] In some cases, the imaging device 202 may project one or more light fields that can be used during the course of stereo image capture. For example, the imaging device 202 may project light fields in a predetermined pattern that are captured by the first image sensor 204 and the secondimage sensor 208. As a result, the pair of 2D images may include depictions of the light field in different orientations. The light fields may be used as additional information when performing correspondence matching between the pair of 2D images to improve accuracy of the correspondence matching.

[0076] With reference to Fig. 2B, the stereo camera with the first image sensor 204 and the second image sensor 208 may be used during the course of registering the patient 212 to the imaging device 202. The registration may begin with the processor 104 receiving 3D scan data (which may be similar to or the same as 3D image data 132) associated with the patient (e.g., preoperative images, intraoperative images taken before the registration process, etc.). The processor 104 may access the database 130 to retrieve the 3D scan data, may receive the 3D scan data from another component of the system 100, or may receive the 3D scan data from a component outside of the system 100. The 3D scan data may be or comprise a CT scan, an MRI scan, fluoroscopic images, combinations thereof, and / or the like comprising information about one or more portions of the patient 212. For example, the 3D scan data may be a CT scan of a portion or the entirety of the patient 212 depicting the pose of one or more anatomical elements that was taken before the registration process.

[0077] Once the 3D scan data is received, the processor 104 may perform an initial guess of the pose of the patient 212 relative to the imaging device 202, and generate a transformation 124 that maps coordinates associated with the imaging device 202 to a coordinate system associated the patient 212. The initial guess of the patient pose may be based on information retrieved from the database 130 (such as previously-used initial guesses from previous similar surgeries or surgical procedures or a default initial guess), information received via the user interface 110 (e.g., a user such as a physician enters an initial guess), information generated by the processor 104 (e.g., a random number generator), a default initial guess based on a recommended setup of the operating room (e.g., based on how the stereo camera is positioned relative to the operating table), combinations thereof, and / or the like. The transformation 124 may be or comprise one or more algorithms, machine learning models, artificial intelligence models, combinations thereof, and / or the like capable of transforming coordinates associated with one coordinate system into coordinates associated with another, different coordinate system. For example, the transformation 124 may comprise one or more transformation matrices that transform the 3D coordinates associated with the imaging device 202 into coordinates associated with the patient 212.

[0078] Once the initial guess and the transformation 124 have been generated, the processor 104 may project the 3D scan data onto a pair of 2D image planes. The pair of 2D image planes may represent or correspond to the image planes seen by the first image sensor 204 and the second imagesensor 208 of the imaging device 202. The pair of 2D image planes may be similar to any stereo image planes discussed herein. In one embodiment, the pair of 2D image planes comprises a left image plane 224 that is associated with the first image sensor 204 and a right image plane 228 that is associated with the second image sensor 208. The projection onto the pair of 2D image planes is based on the expected views or depictions of the 3D scan data generated by the first image sensor 204 and the second image sensor 208 when the patient 212 is imaged by the imaging device 202. In other words, the processor 104 projects the 3D scan data onto the left image plane 224 to depict a left eye image that would have been generated using information from the first image sensor 204 had the patient 212 been imaged by the imaging device 202 while the patient 212 was in the initial guess pose. Similarly, the processor 104 projects the 3D scan data into the right image plane 228 to depict an image that would have been generated using information from the second image sensor 208 had the patient 212 been imaged by the imaging device 202 while the patient 212 was in the initial guess pose.

[0079] The processor 104 may use image processing 120 when projecting the 3D scan data onto the left image plane 224 and the right image plane 228. The image processing 120 may be or comprise one or more algorithms, machine learning models, artificial intelligence models, combinations thereof, and / or the like that map the 3D scan data to 2D image planes. In one embodiment, the image processing 120 may map each 3D data point in the 3D scan data to corresponding locations in each of the left image plane 224 and the right image plane 228. For example, a point 240 of the 3D scan data may be projected along a left trace line 232 into the left image plane 224, and along a different right trace line 236 to the right image plane 228, resulting in the point 240 being depicted in a different location in the left image plane 224 than in the right image plane 228.

[0080] With reference to Fig. 2C, once the 3D scan data is projected into the left image plane 224 and the right image plane 228, the processor 104 may perform correspondence matching between the two 2D image planes. The correspondence matching includes the processor 104 identifying points in both the left image plane 224 and the right image plane 228 that correspondence to the same location in 3D space. For example, the point 240 corresponds to a point in 3D space, but is depicted differently in the left image plane 224 than in the right image plane 228. During correspondence matching, the processor 104 may identify the point 240 in both a portion of the left image plane 224 and a portion of the right image plane 228 and determine that these two portions represent different views of the point 240. In some instances, the processor 104 may perform correspondence matching for each data point from the 3D scan that was projected into the pair of 2D image planes.

[0081] In some embodiments, the correspondence matching may be limited by one or more parameters. In some cases where the processor 104 uses a pixel-by-pixel evaluation of left image plane 224 and the right image plane 228, the individual differences in pixel value between the depiction of any data point in the 3D scan data may result in the processor 104 failing to match pixels that correspond to the same data point. This failure of correspondence matching for some data points may negatively impact the accuracy of the registration, since fewer matching data points result in sparser 3D point clouds, which may result in larger uncertainty of the position of the patient 212 in 3D space. To address this issue, the one or more parameters may be used to confine the evaluation of the left image plane 224 and the right image plane 228, such that a greater number of correspondences are matched.

[0082] In some embodiments, the one or more parameters comprises a predetermined or known geometry of the patient 212. In other words, since the 2D image planes are constructed from 3D scan data of the patient, the processor 104 may know and use the geometry of one or more portions of the patient 212 when performing correspondence matching. For example, a left area 244 of the left image plane 224 may include to a first feature 252 associated with a known geometry or portion of the patient 212 (e.g., a structure just under the eye, a spinous process of a vertebra, etc.) that the processor 104 may identify. Then, the processor 104 may identify a similar or the same feature in the right image plane 228, such as a second feature 256 in a right area 248. By constraining the correspondence to require that the same geometry be present in both areas before the areas are matched, the correspondence may ensure that the matched areas correspond to the same location in 3D space.

[0083] In some embodiments, the one or more parameters comprises contour matching when comparing portions of the left image plane 224 and the right image plane 228. For example, the first feature 252 may be a curved portion of the spine of the patient 212. The processor 104 may identify the first feature 252 and then search for a corresponding curve in the right image plane 228. When the processor 104 identifies a matching curved portion (e.g., the second feature 256 located in the right area 248), the processor 104 may determine that the first feature 252 and the second feature 256 correspond to the location in 3D space. In some cases, the processor 104 may use image processing 120 (e.g., edge detection algorithms) to identify the edges or contours of various portions of the 2D images while performing the correspondence matching.

[0084] In some embodiments, the one or more parameters comprises mutual information matching when comparing portions of the left image plane 224 and the right image plane 228. Mutual information may be based on a pattern of pixel values over a portion of the left image plane 224and / or the right image plane 228, regardless of the actual values of the pixel. For instance, the projection of the 3D scan data into the pair of 2D image planes may result in an instance where the point 240 includes the first feature 252 that changes from a high pixel value to a low pixel value in the left image plane 224, while the second feature 256 of the right image plane 228 changes from a low pixel value to a high pixel value, with both features 252, 256 displaying the same amount of change. In such examples, the processor 104 may identify that the features, though having different pixel values that change in different directions, represent the same location in 3D space, and may correspond the two features together.

[0085] The one or more parameters, such as predetermined or known geometry, contour matching, mutual information, and / or the like may be based on the region surrounding the pixels in each of the pair of 2D image planes. In other words, the geometry, contour, or other information may be captured in a region around each pixel (e.g., a 3x2 pixel region, 3x3 pixel region, a 3x4 pixel region, 4x4 pixel region, a 5x5 pixel region, etc.). It is to be understood that, while examples of parameters used to constrain the correspondence matching have been provided, such examples are in no way limiting, and additional or alternative parameters may be used to constrain the correspondence matching. In some embodiments, the processor 104 may compare a measurement of the similarity between the two features or regions to a threshold value (e.g., a value stored in the database 130) and correspond the two features to the same location in 3D space when the measurement of similarity is greater than or equal to a threshold value.

[0086] During the correspondence matching and for each data point from the 3D scan, the processor 104 may generate a similarity metric. The similarity metric may be or comprise a measurement or indicator of a degree of match or amount of similarity between the region surrounding the projected pixel locations. The amount of similarity may be based on the overall pixel value in the region (e.g., average pixel intensity of the area, average pixel entropy of the area, etc.).

[0087] Once the processor 104 has determined a similarity metric for each point in the 3D scan data that has been projected, the processor 104 may construct an objective function based on the similarity metrics that describes how well the initial guess of the patient pose fits the 3D projected data. In other words, the objective function may indicate how accurate the initial guess was at generating a transformation that maps stereo camera coordinates into coordinates associated with the patient. The objective function may then be iteratively optimized to adjust the transformation to better align projected data with the stereo camera, and may used the optimized transformation to register the stereo camera with the patient 212, as discussed in further detail below.

[0088] In one embodiment, the 3D scan data of the patient is projected into 2D images, while any data captured by the imaging device 202 is not projected into 2D images. For example, the 3D scan data may be or comprise a 3D model constructed based on a preoperative image of soft tissues and / or hard tissues of the patient 212 captured before a surgery or surgical procedure. The 3D model may be a vector-based CAD model or the like that comprises a point cloud of data that is projected into the 2D images. In some embodiments, the nature of the 3D model may change based on the imaging device used to generate the 3D scan data (e.g., a 3D model generated from a CT scan may comprise more data points than a 3D model based on an x-ray emitter / detector configuration). In some cases, the 3D model may also change based on the type of anatomical tissue of the patient 212 that is imaged (e.g., the 3D model may be different for hard tissues than for soft tissues).

[0089] The imaging device 202 may then capture another 3D surgical scene image of the patient 212 (e.g., when the patient 212 is laying on an operating table). The processor 104 may then data points from the preoperative image in performing the initial guess, transformation, and iterative optimization of the objective function for registration. Accordingly, additional processing time associated with higher resolution imaging devices 202 can be minimized, effectively conserving resources.

[0090] Fig. 3 depicts a method 300 that may be used, for example, to register 3D scan data to an imaging device.

[0091] The method 300 (and / or one or more steps thereof) may be carried out or otherwise performed, for example, by at least one processor. The at least one processor may be the same as or similar to the processor(s) 104 of the computing device 102 described above. The at least one processor may be part of a robot (such as a robot 114) or part of a navigation system (such as a navigation system 118). A processor other than any processor described herein may also be used to execute the method 300. The at least one processor may perform the method 300 by executing elements stored in a memory such as the memory 106. The elements stored in memory and executed by the processor may cause the processor to execute one or more steps of a function as shown in method 300. One or more portions of a method 300 may be performed by the processor executing any of the contents of memory, such as an image processing 120, a segmentation 122, a transformation 124, and / or a registration 128.

[0092] The method 300 comprises receiving 3D scan data depicting a portion of a patient generated with an imaging device (step 304). The 3D scan data may be based on one or more preoperative or intraoperative images of the patient 212 captured using the imaging device 112 or another imaging device. The one or more images may be or comprise a CT scan, an MRI scan, twoor more fluoroscopic images, combinations thereof, and / or the like. In some examples, the 3D scan data may be or comprise a point cloud representing one or more portions of the patient 212. The portions may comprise features of the patient such as anatomical elements or features (e.g., eyes, nose, ears, mouth, arms, legs, torso, vertebrae, etc.). In some embodiments, the 3D scan data may be retrieved from the database 130 and rendered to the user interface 110.

[0093] The method 300 also comprises projecting the 3D scan data into a plurality of 2D image planes including a first 2D image plane and a second 2D image plane (step 308). As previously discussed, the projecting of the 3D scan data may be based on a transformation matrix that, when applied to the imaging device 112 or other imaging device, transforms coordinates associated with the imaging device 112 or other imaging device into coordinates associated with the patient 212. The projection onto the pair of 2D image planes is based on the expected views or depictions of the 3D scan data generated by the first image sensor 204 and the second image sensor 208 when the patient 212 is imaged by the imaging device 112. In other words, the processor 104 projects the 3D scan data onto the left image plane 224 (which may correspond to the first 2D image plane) to depict a left eye image that would have been generated using information from the first image sensor 204 had the patient 212 been imaged by the imaging device 112 while the patient 212 was in the initial guess pose. Similarly, the processor 104 projects the 3D scan data into the right image plane 228 (which may correspond to the second 2D image plane) to depict an image that would have been generated using information from the second image sensor 208 had the patient 212 been imaged by the imaging device 112 while the patient 212 was in the initial guess pose.

[0094] The method 300 also comprises matching, based on a geometry of the portion of the patient, an area of the first 2D image plane to an area of the second 2D image plane that corresponds to a common location in 3D space (step 312). The matching includes the processor 104 identifying points in both the left image plane 224 and the right image plane 228 that correspondence to the same location in 3D space. For example, the point 240 corresponds to a point in 3D space, but is depicted differently in the left image plane 224 than in the right image plane 228. During correspondence matching, the processor 104 may identify the point 240 in both a portion of the left image plane 224 and a portion of the right image plane 228 and determine that these two portions represent different views of the point 240. In some instances, the processor 104 may perform correspondence matching for each data point from the 3D scan that was projected into the pair of 2D image planes.

[0095] The geometry of the patient 212 may be a predetermined or known shape of the patient 212 that may be used to constrain the matching of the areas of the 2D image planes. Since the 2D imageplanes are constructed from 3D scan data of the patient, the processor 104 may know and use the geometry of one or more portions of the patient 212 when performing correspondence matching. For example, a left area 244 of the left image plane 224 may include to a first feature 252 associated with a known geometry or portion of the patient 212 (e.g., a structure just under the eye, a spinous process of a vertebra, etc.) that the processor 104 may identify. Then, the processor 104 may identify a similar or the same feature in the right image plane 228, such as a second feature 256 in a right area 248. By constraining the correspondence to require that the same geometry be present in both areas before the areas are matched, the correspondence may ensure that the matched areas correspond to the same location in 3D space.

[0096] Additionally or alternatively, the step 312 may include additional or alternative constraints on the correspondence matching. For example, contour matching may be implemented when comparing portions of the left image plane 224 and the right image plane 228. The contour matching may be based on identified contours within the left image plane 224 and the right image plane 228. For example, the first feature 252 may be a curved portion of the spine of the patient 212. The processor 104 may identify the first feature 252 and then search for a corresponding curve in the right image plane 228. When the processor 104 identifies a matching curved portion (e.g., the second feature 256 located in the right area 248), the processor 104 may determine that the first feature 252 and the second feature 256 correspond to the location in 3D space. In some cases, the processor 104 may use image processing 120 (e.g., edge detection algorithms) to identify the edges or contours of various portions of the 2D images while performing the correspondence matching.

[0097] As another constraint example, mutual information matching may be used when comparing portions of the left image plane 224 and the right image plane 228. Mutual information may be based on a pattern of pixel values over a portion of the left image plane 224 and / or the right image plane 228, regardless of the actual values of the pixel. For instance, the projection of the 3D scan data into the pair of 2D image planes may result in an instance where the point 240 includes the first feature 252 that changes from a high pixel value to a low pixel value in the left image plane 224, while the second feature 256 of the right image plane 228 changes from a low pixel value to a high pixel value, with both features 252, 256 displaying the same amount of change. In such examples, the processor 104 may identify that the features, though having different pixel values that change in different directions, represent the same location in 3D space, and may correspond the two features together.

[0098] The method 300 also comprises registering, based on the matching, the 3D scan data and the imaging device (step 316). Once the two 2D image planes are matched, the transformationmatrix used to project the 3D data may be used as the registration between the 3D scan data and the imaging device. In some embodiments, and as discussed in further detail below, the transformation matrix may be the result of an iterative optimization of an objective function, such that the transformation matrix may represent the best mapping from the stereo camera to the patient 212. In some embodiments, the transformation matrix may be further used, along with the known position of the stereo camera in 3D space, to register the stereo camera and the patient 212 in the surgical scene. Based on these registrations, the patient 212 and the navigation system 118 may then be registered, enabling the navigation system 118 to navigate one or more components (e.g., surgical tools) relative to the patient 212.

[0099] The present disclosure encompasses embodiments of the method 300 that comprise more or fewer steps than those described above, and / or one or more steps that are different than the steps described above.

[0100] Fig. 4 depicts a method 400 that may be used, for example, to iteratively optimize an objective function for registration.

[0101] The method 400 (and / or one or more steps thereof) may be carried out or otherwise performed, for example, by at least one processor. The at least one processor may be the same as or similar to the processor(s) 104 of the computing device 102 described above. The at least one processor may be part of a robot (such as a robot 114) or part of a navigation system (such as a navigation system 118). A processor other than any processor described herein may also be used to execute the method 400. The at least one processor may perform the method 400 by executing elements stored in a memory such as the memory 106. The elements stored in memory and executed by the processor may cause the processor to execute one or more steps of a function as shown in method 400. One or more portions of a method 400 may be performed by the processor executing any of the contents of memory, such as an image processing 120, a segmentation 122, a transformation 124, and / or a registration 128.

[0102] The method 400 comprises receiving 3D scan data depicting a portion of a patient generated with an imaging device (step 404). The step 404 may be similar to or the same as the step 304 of the method 300. The 3D scan data may be or comprise a point cloud representing one or more portions of the patient 212. The portions may comprise features of the patient such as anatomical elements or features (e.g., eyes, nose, ears, mouth, arms, legs, torso, vertebrae, etc.).

[0103] The method 400 also comprises generating an initial guess about a pose of the patient in a surgical environment (step 408). The initial guess of the patient pose may be based on information retrieved from the database 130 (such as previously -used initial guesses from previous similarsurgeries or surgical procedures or a default initial guess), information received via the user interface 110 (e.g., a user such as a physician enters an initial guess), information generated by the processor 104 (e.g., a random number generator), a default initial guess based on a recommended setup of the operating room (e.g., based on how the stereo camera is positioned relative to the operating table), combinations thereof, and / or the like. The initial guess may additionally or alternatively comprise information about the pose of the patient relative to the stereo camera within the surgical environment.

[0104] The method 400 also comprises generating a transformation matrix associated with the pose of the patient (step 412). The transformation matrix may be generated by transformation 124, which may be or comprise one or more algorithms, machine learning models, artificial intelligence models, combinations thereof, and / or the like capable of transforming coordinates associated with one coordinate system into coordinates associated with another, different coordinate system. For example, the transformation 124 may comprise one or more transformation matrices that transform the 3D coordinates associated with the imaging device 112 into coordinates associated with the patient 212. In some embodiments, the transformation 124 may generate the transformation matrix based off historical data of other similar surgeries or surgical procedures.

[0105] The method 400 also comprises projecting, using the transformation matrix, the 3D scan data into a left 2D image plane and a right 2D image plane (step 416). The left 2D image plane is associated with the first image sensor 204, while the right 2D image plane is associated with the second image sensor 208. The projection onto the left and right 2D image planes is based on the expected views or depictions of the 3D scan data generated by the first image sensor 204 and the second image sensor 208 when the patient 212 is imaged by the imaging device 112. In other words, the processor 104 projects the 3D scan data onto the left image plane 224 to depict a left eye image that would have been generated using information from the first image sensor 204 had the patient 212 been imaged by the imaging device 112 while the patient 212 was in the initial guess pose. Similarly, the processor 104 projects the 3D scan data into the right image plane 228 to depict an image that would have been generated using information from the second image sensor 208 had the patient 212 been imaged by the imaging device 112 while the patient 212 was in the initial guess pose.

[0106] The processor 104 may use image processing 120 when projecting the 3D scan data onto the left image plane 224 and the right image plane 228. The image processing 120 may be or comprise one or more algorithms, machine learning models, artificial intelligence models, combinations thereof, and / or the like that map the 3D scan data to 2D image planes. In oneembodiment, the image processing 120 may map each 3D data point in the 3D scan data to corresponding locations in each of the left image plane 224 and the right image plane 228.

[0107] The method 400 also comprises determining, for each point in the 3D scan data, a similarity metric of a region surrounding projected pixel locations for the 2D image planes (step 420). The region surrounding the projected pixel locations may comprise one or more proximate pixels (e.g., a 3x2 pixel region, 3x3 pixel region, a 3x4 pixel region, 4x4 pixel region, a 5x5 pixel region, etc.). The similarity metric may be or comprise a measurement or indicator of a degree of match or amount of similarity between the region surrounding the projected pixel locations. The amount of similarity may be based on the overall pixel value in the region (e.g., average pixel intensity of the area, average pixel entropy of the area, etc.). In some embodiments, the similarity metric may be based on the Hamming distance between the pixels in the first 2D image plane region and the pixels in the second 2D image plane region after a census transform has been performed in each of the two regions.

[0108] The method 400 also comprises combining similarity metrics into an overall objective function describing how well the initial guess fits the 3D scan data (step 424). The objective function may be or comprise a mathematical combination of each similarity metric calculated between the regions of the first 2D image plane and the corresponding region of the second 2D image plane. In one embodiment, the objective function is the sum of absolute differences between the regions of the 2D image planes. The value of the objective function, when large, may indicate that the first 2D image plane regions do not match well with the regions of the second 2D image plane, suggesting that the transform used to project the 3D scan data could be improved. In some embodiments, additional or alternative image -based registration metrics discussed herein, such as contour matching and mutual information matching, may be integrated into the objective function. For instance, when contour matching is implemented, a measure of the percentage of regions that found complementary or corresponding contours (which may be indicative of correct correspondence matching) may be provided by the objective function. In such examples, a low value of the objective function may suggest that few contours were found and matched, suggesting the transformation matrix could be improved.

[0109] The method 400 also comprises determining if the objective function is optimized (step 428). The objective function may be considered optimized when the overall value of the objective function reaches a minimum or maximum value. The minimum or maximum value may be determined based on a predetermined threshold value (e.g., a value stored in the database 130). Additionally or alternatively, the objective function may be considered optimized after apredetermined or selected number of iterations (e.g., after 12 iterations, the transformation used in the 12th iteration is selected as being considered optimized), the best fit obtained over a range of iterations (e.g., the objective function is optimized 15 times, and the 13th iteration produces the maximum value, so the transformation matrix associated with the 13th iteration is used), and / or the like. Any one or more criteria may be used to determine if the objective function has been optimized.

[0110] The method 400 also comprises adjusting, when the objective function is not optimized, the transformation matrix (step 432). When the objective function can be further optimized (e.g., there is a change to the transformation matrix that could lead to a lower objective function value), the processor 104 may determine an adjustment to the transformation matrix to lower the overall objective function. The optimization may use one or more numerical optimization methods (e.g., gradient descent, stochastic gradient descent, Newton’s method, etc.) to adjust the objective function toward a maximum or minimum value. The adjusted transformation matrix may then be used in steps 416, 420, and 424 to produce an updated objective function.

[0111] The method 400 also comprises registering, using the transformation matrix, the 3D scan data to the imaging device (step 436). The step 436 may be similar to or the same as the step 316 of the method 300. Once the objective function has been optimized, the transformation matrix may represent the best mapping from the stereo camera to the patient 212. The transformation matrix may be further used, along with the known position of the stereo camera in 3D space, to register the stereo camera and the patient 212 in the surgical scene. Based on the registration of the 3D scan data to the stereo camera, and the registration of the stereo camera to the patient, the patient 212 and 3D scan data can be registered. Similarly, the navigation system 118 and the patient 212 may be registered, enabling the navigation system 118 to navigate one or more components (e.g., surgical tools) relative to the patient 212.

[0112] The present disclosure encompasses embodiments of the method 400 that comprise more or fewer steps than those described above, and / or one or more steps that are different than the steps described above.

[0113] As noted above, the present disclosure encompasses methods with fewer than all of the steps identified in Figs. 3 and 4 (and the corresponding description of the methods 300 and 400), as well as methods that include additional steps beyond those identified in Figs. 3 and 4 (and the corresponding description of the methods 300 and 400). The present disclosure also encompasses methods that comprise one or more steps from one method described herein, and one or more steps from another method described herein. Any correlation described herein may be or comprise a registration or any other correlation.

[0114] The foregoing is not intended to limit the disclosure to the form or forms disclosed herein. In the foregoing Detailed Description, for example, various features of the disclosure are grouped together in one or more aspects, embodiments, and / or configurations for the purpose of streamlining the disclosure. The features of the aspects, embodiments, and / or configurations of the disclosure may be combined in alternate aspects, embodiments, and / or configurations other than those discussed above. This method of disclosure is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed aspect, embodiment, and / or configuration. Thus, the following claims are hereby incorporated into this Detailed Description, with each claim standing on its own as a separate preferred embodiment of the disclosure.

[0115] Moreover, though the foregoing has included description of one or more aspects, embodiments, and / or configurations and certain variations and modifications, other variations, combinations, and modifications are within the scope of the disclosure, e.g., as may be within the skill and knowledge of those in the art, after understanding the present disclosure. It is intended to obtain rights which include alternative aspects, embodiments, and / or configurations to the extent permitted, including alternate, interchangeable and / or equivalent structures, functions, ranges or steps to those claimed, whether or not such alternate, interchangeable and / or equivalent structures, functions, ranges or steps are disclosed herein, and without intending to publicly dedicate any patentable subject matter.

[0116] Example statement of the claims:

[0117] Example 1. A system, comprising: a processor; and a memory coupled to the processor and storing data thereon that, when processed by the processor, enables the processor to: receive three-dimensional (3D) scan data depicting a portion of a patient generated with an imaging device; project the 3D scan data into a plurality of two-dimensional (2D) image planes including a first 2D image plane and a second 2D image plane; match, based on a geometry of the portion of the patient, an area of the first 2D image plane to an area of the second 2D image plane that corresponds to a common location in 3D space; and register, based on the matching, the 3D scan data and the imaging device.

[0118] Example 2. The system of example 1, wherein the geometry of the portion of the patient is determined from a 3D image of the portion of the patient.

[0119] Example 3. The system of any of examples 1 to 2, wherein the memory stores further data for processing by the processor that, when processed, enables the processor to: generate a similarity metric indicating an amount of similarity between the area of the first 2D image plane and the area of the second 2D image plane.

[0120] Example 4. The system of example 3, wherein the amount of similarity is based on an entropy of the area of the first 2D image plane and an entropy of the area of the second 2D image plane.

[0121] Example 5. The system of any of examples 3 to 4, wherein the memory stores further data for processing by the processor that, when processed, enables the processor to: iteratively optimize an objective function associated with the similarity metric.

[0122] Example 6. The system of example 5, wherein as part of the iterative optimization, the processor further: generates an initial guess for a transformation matrix that transforms coordinates from an imaging device coordinate system associated with the imaging device to an exam coordinate system associated with the 3D scan data; generates a second similarity metric that indicates an amount of similarity between a second area of the first 2D image plane and a second area of the second 2D image plane; determines, based on a combination of the first similarity metric and the second similarity metric, the objective function; and adjusts the transformation matrix using the objective function.

[0123] Example 7. The system of any of examples 1 to 6, wherein the geometry of the portion of the patient comprises a contour of an anatomical element of the patient.

[0124] Example 8. A system, comprising: an imaging device;a processor; and a memory coupled to the processor and storing data thereon that, when processed by the processor, enables the processor to: receive three-dimensional (3D) scan data depicting a portion of a patient generated with the imaging device; project the 3D scan data into a plurality of two-dimensional (2D) image planes including a first 2D image plane and a second 2D image plane; match, based on a geometry of the portion of the patient, an area of the first 2D image plane to an area of the second 2D image plane that corresponds to a common location in 3D space; and register, based on the matching, the 3D scan data and the imaging device.

[0125] Example 9. The system of example 8, wherein the geometry of the portion of the patient is determined from a 3D image of the portion of the patient.

[0126] Example 10. The system of any of examples 8 to 9, wherein the memory stores further data for processing by the processor that, when processed, enables the processor to: generate a similarity metric indicating an amount of similarity between the area of the first 2D image plane and the area of the second 2D image plane.

[0127] Example 11. The system of example 10, wherein the amount of similarity is based on an entropy of the area of the first 2D image plane and an entropy of the area of the second 2D image plane.

[0128] Example 12. The system of any of examples 10 to 11, wherein the memory stores further data for processing by the processor that, when processed, enables the processor to: iteratively optimize an objective function associated with the similarity metric.

[0129] Example 13. The system of example 12, wherein as part of the iterative optimization, the processor further: generates an initial guess for a transformation matrix that transforms coordinates from an imaging device coordinate system associated with the imaging device to an exam coordinate system associated with the 3D scan data;generates a second similarity metric that indicates an amount of similarity between a second area of the first 2D image plane and a second area of the second 2D image plane; determines, based on a combination of the first similarity metric and the second similarity metric, the objective function; and adjusts the transformation matrix using the objective function.

[0130] Example 14. The system of any of examples 8 to 13, wherein the geometry of the portion of the patient comprises a contour of an anatomical element of the patient.

[0131] Example 15. A method, comprising: receiving three-dimensional (3D) scan data depicting a portion of a patient generated with an imaging device; projecting the 3D scan data into a plurality of two-dimensional (2D) image planes including a first 2D image plane and a second 2D image plane; matching, based on a geometry of the portion of the patient, an area of the first 2D image plane to an area of the second 2D image plane that corresponds to a common location in 3D space; and registering, based on the matching, the 3D scan data and the imaging device.

[0132] Example 16. The method of example 15, wherein the geometry of the portion of the patient is determined from a 3D image of the portion of the patient.

[0133] Example 17. The method of any of examples 15 to 16, further comprising: generating a similarity metric indicating an amount of similarity between the area of the first 2D image plane and the area of the second 2D image plane.

[0134] Example 18. The method of example 17, further comprising: iteratively optimizing an objective function associated with the similarity metric.

[0135] Example 19. The method of example 18, wherein iteratively optimizing the objective function further comprises:generating an initial guess for a transformation matrix that transforms coordinates from an imaging device coordinate system associated with the imaging device to an exam coordinate system associated with the 3D scan data; generating a second similarity metric that indicates an amount of similarity between a second area of the first 2D image plane and a second area of the second 2D image plane; determining, based on a combination of the first similarity metric and the second similarity metric, the objective function; and adjusting the transformation matrix using the objective function.

[0136] Example 20. The method of any of examples 15 to 19, wherein the geometry of the portion of the patient comprises information about a contour of an anatomical element of the patient.

Claims

CLAIMSWhat is claimed is:

1. A system, comprising: a processor; and a memory coupled to the processor and storing data thereon that, when processed by the processor, enables the processor to: receive three-dimensional (3D) scan data depicting a portion of a patient generated with an imaging device; project the 3D scan data into a plurality of two-dimensional (2D) image planes including a first 2D image plane and a second 2D image plane; match, based on a geometry of the portion of the patient, an area of the first 2D image plane to an area of the second 2D image plane that corresponds to a common location in 3D space; and register, based on the matching, the 3D scan data and the imaging device.

2. The system of claim 1 , wherein the geometry of the portion of the patient is determined from a 3D image of the portion of the patient.

3. The system of any of claims 1 to 2, wherein the memory stores further data for processing by the processor that, when processed, enables the processor to: generate a similarity metric indicating an amount of similarity between the area of the first 2D image plane and the area of the second 2D image plane.

4. The system of claim 3, wherein the amount of similarity is based on an entropy of the area of the first 2D image plane and an entropy of the area of the second 2D image plane.

5. The system of any of claims 3 to 4, wherein the memory stores further data for processing by the processor that, when processed, enables the processor to: iteratively optimize an objective function associated with the similarity metric.

6. The system of claim 5, wherein as part of the iterative optimization, the processor further:generates an initial guess for a transformation matrix that transforms coordinates from an imaging device coordinate system associated with the imaging device to an exam coordinate system associated with the 3D scan data; generates a second similarity metric that indicates an amount of similarity between a second area of the first 2D image plane and a second area of the second 2D image plane; determines, based on a combination of the first similarity metric and the second similarity metric, the objective function; and adjusts the transformation matrix using the objective function.

7. The system of any of claims 1 to 6, wherein the geometry of the portion of the patient comprises a contour of an anatomical element of the patient.

8. A system, comprising: an imaging device; a processor; and a memory coupled to the processor and storing data thereon that, when processed by the processor, enables the processor to: receive three-dimensional (3D) scan data depicting a portion of a patient generated with the imaging device; project the 3D scan data into a plurality of two-dimensional (2D) image planes including a first 2D image plane and a second 2D image plane; match, based on a geometry of the portion of the patient, an area of the first 2D image plane to an area of the second 2D image plane that corresponds to a common location in 3D space; and register, based on the matching, the 3D scan data and the imaging device.

9. The system of claim 8, wherein the geometry of the portion of the patient is determined from a 3D image of the portion of the patient.

10. The system of any of claims 8 to 9, wherein the memory stores further data for processing by the processor that, when processed, enables the processor to: generate a similarity metric indicating an amount of similarity between the area of the first 2D image plane and the area of the second 2D image plane.

11. The system of claim 10, wherein the amount of similarity is based on an entropy of the area of the first 2D image plane and an entropy of the area of the second 2D image plane.

12. The system of any of claims 10 to 11, wherein the memory stores further data for processing by the processor that, when processed, enables the processor to: iteratively optimize an objective function associated with the similarity metric.

13. The system of claim 12, wherein as part of the iterative optimization, the processor further: generates an initial guess for a transformation matrix that transforms coordinates from an imaging device coordinate system associated with the imaging device to an exam coordinate system associated with the 3D scan data; generates a second similarity metric that indicates an amount of similarity between a second area of the first 2D image plane and a second area of the second 2D image plane; determines, based on a combination of the first similarity metric and the second similarity metric, the objective function; and adjusts the transformation matrix using the objective function.

14. The system of any of claims 8 to 13, wherein the geometry of the portion of the patient comprises a contour of an anatomical element of the patient.

15. A method, comprising: receiving three-dimensional (3D) scan data depicting a portion of a patient generated with an imaging device; projecting the 3D scan data into a plurality of two-dimensional (2D) image planes including a first 2D image plane and a second 2D image plane; matching, based on a geometry of the portion of the patient, an area of the first 2D image plane to an area of the second 2D image plane that corresponds to a common location in 3D space; and registering, based on the matching, the 3D scan data and the imaging device.

Images