Systems and methods for patient registration using light patterns

Abstract

A system according to an embodiment of the present disclosure includes: a processor; and a memory storing data thereon that, when processed by the processor, enable the processor to: receive an input representing an initial guess of a pose of an object in three-dimensional (3D) space; determine, based on the input, an illumination that, when projected from a projector, results in a first light pattern appearing on the object; cause the projector to project the illumination onto the object; receive, from an imaging device, an image depicting a second light pattern appearing on the object; determine a difference between the first light pattern and the second light pattern; determine, when the difference satisfies a predetermined condition, that the initial guess substantially corresponds to an actual pose of the object; and register, in response to determining that the initial guess corresponds to the actual pose of the object, the imaging device with the object.

Description

SYSTEMS AND METHODS FOR PATIENT REGISTRATION USING LIGHT PATTERNSBACKGROUND

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 612,886, filed 20 December 2023, the entire content of which is incorporated herein by reference.

[0002] The present disclosure is generally directed to image registration, and relates more particularly to registration using light patterns.

[0003] Imaging may be used by a medical provider for diagnostic and / or therapeutic purposes during a surgery or surgical procedure. Patient anatomy can change over time, particularly following placement of a medical implant in the patient anatomy.BRIEF SUMMARY

[0004] Example aspects of the present disclosure include:

[0005] A system according to at least one embodiment of the present disclosure comprises: a processor; and a memory storing data thereon that, when processed by the processor, enable the processor to: receive an input representing an initial guess of a pose of an object in three- dimensional (3D) space; determine, based on the input, an illumination that, when projected from a projector, results in a first light pattern appearing on the object; cause the projector to project the illumination onto the object; receive, from an imaging device, an image depicting a second light pattern appearing on the object; determine a difference between the first light pattern and the second light pattern; determine, when the difference satisfies a predetermined condition, that the initial guess substantially corresponds to an actual pose of the object; and register, in response to determining that the initial guess corresponds to the actual pose of the object, the imaging device with the object.

[0006] Any of the aspects herein, wherein the memory comprises further data that, when processed by the processor, enables the processor to: determine, using a path tracing simulation, a view seen by a virtual imaging device when a virtual projector projects the first light pattern onto the object.

[0007] Any of the aspects herein, wherein, in the path tracing simulation, the virtual imaging device is positioned in a same pose as the projector and the virtual projector is positioned in a same pose as the imaging device.

[0008] Any of the aspects herein, wherein the memory comprises further data that, when processed by the processor, enables the processor to: iteratively optimize an objective function associated with the difference between the first light pattern and the second light pattern.

[0009] Any of the aspects herein, wherein as part of iteratively optimizing the objective function, the processor further: determines, based on the difference, a change to the initial guess to reduce the difference; and generates, based on the change, a second guess of the pose of the object.

[0010] Any of the aspects herein, wherein as part of iteratively optimizing the objective function, the processor further: generates, based on the second guess of the pose, a second illumination that, when projected from the projector, results in the first light pattern appearing on the object; causes the projector to project the second illumination onto the object; receives, from the imaging device, a second image of the object illuminated by a third light pattern; and determines a difference between the first light pattern and the third light pattern.

[0011] Any of the aspects herein, wherein the first light pattern comprises a set of two-dimensional (2D) grid lines.

[0012] Any of the aspects herein, wherein the object comprises a portion of a patient.

[0013] A system according to at least one embodiment of the present disclosure comprises: an imaging device; a projector; a processor; and a memory storing data thereon that, when processed by the processor, enable the processor to: determine, based on an input representing initial guess of a pose of an object in three-dimensional (3D) space, an illumination that, when emitted from the projector, results in a first light pattern appearing on the object; cause the projector to emit the illumination to illuminate the object; cause the imaging device to capture an image the object, wherein the image depicts a second light pattern appearing on the object; determine a difference between the first light pattern and the second light pattern; and determine, when the difference satisfies a predetermined condition, that the initial guess substantially corresponds to an actual pose of the object.

[0014] Any of the aspects herein, wherein the memory comprises further data that, when processed by the processor, enables the processor to: determine, using a path tracing simulation, a view seen by a virtual imaging device when a virtual projector projects the first light pattern onto the object.

[0015] Any of the aspects herein, wherein, in the path tracing simulation, the virtual imaging device is positioned in a same pose as the projector and the virtual projector is positioned in a same pose as the imaging device.

[0016] Any of the aspects herein, wherein the memory comprises further data that, when processed by the processor, enables the processor to: iteratively optimize an objective function associated with the difference between the first light pattern and the second light pattern.

[0017] Any of the aspects herein, wherein as part of iteratively optimizing the objective function, the processor further: determines, based on the difference, a change to the initial guess to reduce the difference; and generates, based on the change, a second guess of the pose of the object.

[0018] Any of the aspects herein, wherein as part of iteratively optimizing the objective function, the processor further: generates, based on the second guess of the pose, a second illumination that, when projected from the projector, results in the first light pattern appearing on the object; causes the projector to project the second illumination onto the object; receives, from the imaging device, a second image of the object illuminated by a third light pattern; and determines a difference between the first light pattern and the third light pattern.

[0019] Any of the aspects herein, wherein the first light pattern comprises a set of two-dimensional (2D) grid lines.

[0020] Any of the aspects herein, wherein the object comprises a portion of a patient.

[0021] Any of the aspects herein, wherein the memory stores further data that, when processed by the processor, enable the processor to: register, in response to determining that the initial guess corresponds to the actual pose of the object, the imaging device with the object.

[0022] A method according to at least one embodiment of the present disclosure comprises: receiving an input representing an initial guess of a pose of an object in three-dimensional (3D) space; generating, based on the input, an illumination that, when projected from a projector, results in a first light pattern appearing on the object; causing the projector to project the illumination onto the object; receiving, from an imaging device, an image that depicts the object with a second light pattern; determining a difference between the first light pattern and the second light pattern; determining, when the difference satisfies a predetermined condition, that the initial guess substantially corresponds to an actual pose of the object; and registering, in response to determining that the initial guess corresponds to the actual pose of the object, the imaging device with the object.

[0023] Any of the aspects herein, further comprising: iteratively optimizing an objective function associated with the difference between the first light pattern and the second light pattern.

[0024] Any of the aspects herein, wherein the iteratively optimizing comprises: generating, based on a second guess of the pose, a second illumination that, when projected from the projector, results in the first light pattern appearing on the object; causing the projector to project the second illumination onto the object; receiving, from the imaging device, a second image that depicts the object with a third light pattern; and determining a difference between the first light pattern and the third light pattern.

[0025] Any aspect in combination with any one or more other aspects.

[0026] Any one or more of the features disclosed herein.

[0027] Any one or more of the features as substantially disclosed herein.

[0028] Any one or more of the features as substantially disclosed herein in combination with any one or more other features as substantially disclosed herein.

[0029] Any one of the aspects / features / embodiments in combination with any one or more other aspects / features / embodiments .

[0030] Use of any one or more of the aspects or features as disclosed herein.

[0031] It is to be appreciated that any feature described herein can be claimed in combination with any other feature(s) as described herein, regardless of whether the features come from the same described embodiment.

[0032] The details of one or more aspects of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques described in this disclosure will be apparent from the description and drawings, and from the claims.

[0033] The phrases “at least one”, “one or more”, and “and / or” are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions “at least one of A, B and C”, “at least one of A, B, or C”, “one or more of A, B, and C”, “one or more of A, B, or C” and “A, B, and / or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B and C together. When each one of A, B, and C in the above expressions refers to an element, such as X, Y, and Z, or class of elements, such as XI -Xn, Yl-Ym, and Zl-Zo, the phrase is intended to refer to a single element selected from X, Y, and Z, a combination of elements selected from the same class (e.g., XI and X2) as well as a combination of elements selected from two or more classes (e.g., Y1 and Zo).

[0034] The term “a” or “an” entity refers to one or more of that entity. As such, the terms “a” (or “an”), “one or more” and “at least one” can be used interchangeably herein. It is also to be noted that the terms “comprising”, “including”, and “having” can be used interchangeably.

[0035] The preceding is a simplified summary of the disclosure to provide an understanding of some aspects of the disclosure. This summary is neither an extensive nor exhaustive overview of the disclosure and its various aspects, embodiments, and configurations. It is intended neither to identify key or critical elements of the disclosure nor to delineate the scope of the disclosure but to present selected concepts of the disclosure in a simplified form as an introduction to the more detailed description presented below. As will be appreciated, other aspects, embodiments, and configurations of the disclosure are possible utilizing, alone or in combination, one or more of the features set forth above or described in detail below.

[0036] Numerous additional features and advantages of the present disclosure will become apparent to those skilled in the art upon consideration of the embodiment descriptions provided hereinbelow.BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0037] The accompanying drawings are incorporated into and form a part of the specification to illustrate several examples of the present disclosure. These drawings, together with the description, explain the principles of the disclosure. The drawings simply illustrate preferred and alternative examples of how the disclosure can be made and used and are not to be construed as limiting the disclosure to only the illustrated and described examples. Further features and advantages will become apparent from the following, more detailed, description of the various aspects, embodiments, and configurations of the disclosure, as illustrated by the drawings referenced below.

[0038] Fig. 1 is a block diagram of a system according to at least one embodiment of the present disclosure;

[0039] Fig. 2A depicts additional aspects of the system according to at least one embodiment of the present disclosure;

[0040] Fig. 2B depicts a light pattern according to at least one embodiment of the present disclosure;

[0041] Fig. 2C is an image of an object illuminated by the light pattern according to at least one embodiment of the present disclosure;

[0042] Fig. 3A depicts aspects of a path tracing simulation according to at least one embodiment of the present disclosure;

[0043] Fig. 3B depicts a virtual image of the path tracing simulation according to at least one embodiment of the present disclosure;

[0044] Fig. 3C depicts a first image of an object according to at least one embodiment of the present disclosure;

[0045] Fig. 3D depicts a second image of the object according to at least one embodiment of the present disclosure;

[0046] Fig. 3E depicts a third image of the object according to at least one embodiment of the present disclosure;

[0047] Fig. 4 is a flowchart according to at least one embodiment of the present disclosure; and

[0048] Fig. 5 is a flowchart according to at least one embodiment of the present disclosure.DETAILED DESCRIPTION

[0049] It should be understood that various aspects disclosed herein may be combined in different combinations than the combinations specifically presented in the description and accompanying drawings. It should also be understood that, depending on the example or embodiment, certain acts or events of any of the processes or methods described herein may be performed in a different sequence, and / or may be added, merged, or left out altogether (e.g., all described acts or events may not be necessary to carry out the disclosed techniques according to different embodiments of the present disclosure). In addition, while certain aspects of this disclosure are described as being performed by a single module or unit for purposes of clarity, it should be understood that the techniques of this disclosure may be performed by a combination of units or modules associated with, for example, a computing device and / or a medical device.

[0050] In one or more examples, the described methods, processes, and techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Alternatively or additionally, functions may be implemented using machine learning models, neural networks, artificial neural networks, or combinations thereof (alone or in combination with instructions). Computer-readable media may include non-transitory computer-readable media, which corresponds to a tangible medium such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer).

[0051] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors (e.g., Intel Core i3, i5, i7, or i9 processors; Intel Celeron processors; Intel Xeon processors; Intel Pentium processors; AMD Ryzen processors; AMD Athlon processors; AMD Phenom processors; Apple A10 or 10X Fusion processors; Apple Al l, A12, A12X, A12Z, or A13 Bionic processors; or any other general purpose microprocessors), graphics processing units (e.g., Nvidia GeForce RTX 2000-series processors, Nvidia GeForce RTX 3000-series processors, AMD Radeon RX 5000-series processors, AMD Radeon RX 6000-series processors, or any other graphics processing units), application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor” as used herein may refer to any of the foregoing structure or any other physical structure suitable for implementation of the described techniques. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0052] Before any embodiments of the disclosure are explained in detail, it is to be understood that the disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The disclosure is capable of other embodiments and of being practiced or of being carried out in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” or “having” and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Further, the present disclosure may use examples to illustrate one or more aspects thereof. Unless explicitly stated otherwise, the use or listing of one or more examples (which may be denoted by “for example,” “by way of example,” “e.g.,” “such as,” or similar language) is not intended to and does not limit the scope of the present disclosure.

[0053] One approach to surgical registration uses a structured light system, which includes a projector and a camera separated by a known distance. The projector projects a known light pattern onto a three-dimensional (3D) scene, and the camera records how the light pattern is distorted by the geometry in the scene. For example, the projector may project a horizontal pattern of lines onto a 3D object (e.g., a portion of a patient such as the patient’s head), and the camera positioned below the projector may record an image of the patient. By measuring how far the projected horizontal lines shift from the known reference pattern, the distance between the object’s surface and the camera can be determined at each point along the lines. This process can then be repeated with several different light patterns to generate a point cloud of the 3D object. Surgical registration may then be performed by mapping the 3D object to a patient’s surgical images (e.g., preoperative surgical images) using 3D point cloud registration techniques.

[0054] According to at least one embodiment of the present disclosure, a method for registering known geometries in a scene using structured light patterns is provided. The method includes modifying the structured light pattern for the geometry of interest to identify its position and orientation (e.g., pose) directly. The method may begin with an initial guess of the pose of the geometry of interest. A light pattern is then generated such that, when the light pattern is projected onto the scene, the pattern from the camera’s point of view will match the reference pattern if and only if the initial guess of the pose of the geometry of interest matches the actual pose of the geometry of interest. The light pattern may be generated through a path tracing simulation where the projector and camera’s roles and positions are reversed from that of the structured light system. In other words, the reference pattern is projected from a “virtual projector” at the location of the camera, while the virtual camera is positioned at the location of the projector in the structured lightsystem. The light reflects off the surface of the geometry of interest and an image is generated using the virtual camera at the location of the projector. As an example, a grid of two-dimensional (2D) lines may be projected by the virtual projector from the position of the camera in the structured light system, and a virtual camera may generate an image representing the view from the projector.

[0055] If the simulated image is then projected as a light pattern from the actual projector, the physical camera will see the reference pattern if the object of interest is in the pose associated with the initial guess. An objective function can be constructed representing the error between the pattern as seen by the camera and the reference pattern. The guess for the object’s pose can be iterated in order to minimize the error. Once the objective function converges (e.g., the error falls below a threshold value), the true pose of the object will be identified. The true pose may then be used to register the object with the camera.

[0056] Embodiments of the present disclosure provide technical solutions to one or more of the problems of (1) inaccurate correspondence matching in 3D point cloud registration techniques and (2) inaccurate registration.

[0057] Turning first to Fig. 1, a block diagram of a system 100 according to at least one embodiment of the present disclosure is shown. The system 100 may be used to register a patient to a surgical navigation system coordinate system and / or to an imaging device coordinate system; to register patient exam data to a camera or other imaging device; to control, pose, and / or otherwise manipulate a surgical mount system and / or surgical tools attached thereto; and / or to carry out one or more other aspects of one or more of the methods disclosed herein. The system 100 comprises a computing device 102, one or more imaging devices 112, a robot 114, a navigation system 118, a database 130, a cloud or other network 134, and / or a projector 136. Systems according to other embodiments of the present disclosure may comprise more or fewer components than the system 100. For example, the system 100 may not include the robot 114, the navigation system 118, one or more components of the computing device 102, the database 130, and / or the cloud 134.

[0058] The computing device 102 is illustrated to include a processor 104, a memory 106, a communication interface 108, and a user interface 110. Computing devices according to other embodiments of the present disclosure may comprise more or fewer components than the computing device 102.

[0059] The processor 104 of the computing device 102 may be any processor described herein or any similar processor. The processor 104 may be configured to execute instructions stored in the memory 106, which instructions may cause the processor 104 to carry out one or more computing steps utilizing or based on data received from the imaging device 112, the robot 114, the navigationsystem 118, the database 130, the cloud 134, and / or the projector 136. The processor 104 may be or comprise one or more digital signal processors (DSPs), general purpose microprocessors (e.g., Intel Core i3, i5, i7, or i9 processors; Intel Celeron processors; Intel Xeon processors; Intel Pentium processors; AMD Ryzen processors; AMD Athlon processors; AMD Phenom processors; Apple A10 or 10X Fusion processors; Apple Al l, Al 2, A12X, A12Z, or Al 3 Bionic processors; or any other general purpose microprocessors), graphics processing units (e.g., Nvidia GeForce RTX 2000- series processors, Nvidia GeForce RTX 3000-series processors, AMD Radeon RX 5000-series processors, AMD Radeon RX 6000-series processors, or any other graphics processing units), application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry.

[0060] The memory 106 may be or comprise RAM, DRAM, SDRAM, other solid-state memory, any memory described herein, or any other tangible, non-transitory memory for storing computer- readable data and / or instructions. The memory 106 may store information or data useful for completing, for example, any step of the methods 400 and / or 500 described herein, or of any other methods. The memory 106 may store, for example, instructions and / or machine learning models that support one or more functions of the computing device 102, the imaging devices 112, the navigation system 118, the projector 136, and / or the like. For instance, the memory 106 may store content (e.g., instructions and / or machine learning models) that, when executed by the processor 104, enable image processing 120, segmentation 122, transformation 124, and / or registration 128.

[0061] The image processing 120 enables the processor 104 to process image data of an image (received from, for example, the imaging device 112, an imaging device of the navigation system 118, or any imaging device) for the purpose of, for example, identifying information about a patient and / or an object depicted in the image. The information may comprise, for example, a pose of the patient, a boundary of the patient, etc. The information may enable registration of the patient to a common coordinate frame of the imaging device 112, as discussed in further detail below. The image processing 120 may use segmentation 122 to identify the patient and / or the one or more objects, as described below.

[0062] The segmentation 122 enables the processor 104 to segment the image data so as to identify the patient and / or one or more objects in the image data. The segmentation 122 may enable the processor 104 to identify a boundary of an object or the patient by using, for example, feature recognition. For example, the segmentation 122 may enable the processor 104 to identify the patient’s head in the image data. In other instances, the segmentation 122 may enable the processor104 to identify a boundary of an object (e.g., the patient’s head) by determining a difference in or contrast between colors or grayscales of image pixels.

[0063] The transformation 124 enables the processor 104 to transform one coordinate system into another coordinate system. In other words, the transformation 124 enables the processor 104 to transform the first coordinate system (e.g., the patient coordinate system) into the second coordinate system (e.g., the reference frame coordinate system) based on, for example, the registration of the first coordinate system and the third coordinate system and the registration of the second coordinate system and the third coordinate system.

[0064] The registration 128 enables the processor 104 to correlate one coordinate system with another coordinate system. For example, the registration 128 may enable the processor 104 to correlate or map a first coordinate system (e.g., a patient coordinate system) with a third coordinate system (e.g., an imaging device coordinate system) and a second coordinate system (e.g., a reference frame coordinate system) with the third coordinate system (e.g., the imaging device coordinate system).

[0065] The path tracing simulation 140 enables the processor 104 to generate a plurality of different light patterns and predict how the light pattern may appear on and reflect off of an object. The path tracing simulation 140 may enable the processor 104 to determine what a virtual camera would see when a predetermined pattern is projected from a virtual projector onto a virtual object or person (e.g., a virtual representation of the head of a patient). For example, the path tracing simulation 140 may be or comprise a program or instructions that cause the processor 104 to determine, based on an input pattern and a known geometry of the virtual object, what an image captured by a virtual camera would look like when the input pattern is projected onto the object. The path tracing simulation 140 may enable the processor 104 to use a known pose of a virtual projector relative to the virtual camera (and vice versa) to perform ray tracing of the light pattern from the virtual projector onto the surface of the virtual object, and then from the surface of the virtual object to the virtual camera. Based on the known light pattern, the surface geometry of the object, and the position of the virtual camera relative to the virtual projector, the path tracing simulation 140 can determine the shape of the light pattern on the virtual object as seen by the virtual camera. The shape of the light pattern may then be used by the processor 104 to generate an actual light pattern that is projected by the projector 136. The imaging device 112 may then capture an image of the object illuminated by the light pattern, which may be used by the processor 104 in determining the pose of the object relative to the camera, as discussed in further detail below. While the path tracing simulation 140 is depicted as being stored in the memory 106, in some cases the path tracingsimulation 140 may be stored in the database 130 or in another storage device outside the system 100.

[0066] The objective function 144 may be or comprise information related to how well an initial guess of a pose of the object fits the actual pose of the object. In other words, the objective function 144 may describe how accurate the initial guess was based on a difference between a light pattern that appears on an object and an expected light pattern. Continuing from the above example, the initial guess of the pose of the object may be used in conjunction with the path tracing simulation 140 to enable the processor 104 to determine a light pattern to project from the projector 136. Once the light pattern is projected and illuminates the object, the imaging device 112 may capture an image of the object. The objective function 144 may describe a difference or error (e.g., an average of the difference in distance squared) between the light pattern as it appears on the object and the reference light pattern that is expected to appear on the object.

[0067] The objective function 144 may be iteratively optimized to adjust the guess of the pose of the object to converge to the actual pose of the object. For example, the objective function 144 may yield a first difference value with a first initial guess, and the processor 104 may use one or more numerical optimization methods (e.g., gradient descent, stochastic gradient descent, Newton’s method, etc.) to determine a change in the first initial guess that results in a decrease in the first difference value output by the objective function 144. The process of adjusting the initial guess of the pose, the generation of the light patterns, and the comparison between the expected light pattern and the light pattern depicted in the image captured by the imaging device 112 may be repeated until the objective function 144 converges to a maximum or minimum value.

[0068] The minimum or maximum value of the objective function 144 may correspond to the difference between the expected light pattern and the light pattern depicted in the image captured by the imaging device 112, and may be determined based on a predetermined threshold value (e.g., a value stored in the database 130). Additionally or alternatively, the objective function may be considered optimized after a predetermined or selected number of iterations (e.g., after 12 iterations, the transformation used in the 12th iteration is selected as being considered optimized), the best fit obtained over a range of iterations (e.g., the objective function is optimized 15 times, and the 13th iteration produces the maximum value, so the transformation matrix associated with the 13th iteration is used), and / or the like. Any one or more criteria may be used to determine if the objective function has been optimized. In some examples, once the measured difference satisfies a predetermined condition (e.g., the difference between the light pattern appearing on the object and the expected light pattern falls below a threshold value), the processor 104 may use registration 128to register the camera to the object based on the predicted pose that resulted in the optimized objective function 144.

[0069] Such content, if provided as in instruction, may, in some embodiments, be organized into one or more applications, modules, packages, layers, or engines. Alternatively or additionally, the memory 106 may store other types of content or data (e.g., machine learning models, artificial neural networks, deep neural networks, etc.) that can be processed by the processor 104 to carry out the various method and features described herein. Thus, although various contents of memory 106 may be described as instructions, it should be appreciated that functionality described herein can be achieved through use of instructions, algorithms, and / or machine learning models. The data, algorithms, and / or instructions may cause the processor 104 to manipulate data stored in the memory 106 and / or received from or via the imaging device 112, the robot 114, the database 130, the cloud 134, and / or the projector 136.

[0070] The communication interface 108 may be used for receiving image data or other information from an external source (such as the imaging device 112, the robot 114, the navigation system 118, the database 130, the cloud 134, the projector 136, and / or any other system or component not part of the system 100), and / or for transmitting instructions, images, or other information to an external system or device (e.g., another computing device 102, the imaging device 112, the robot 114, the navigation system 118, the database 130, the cloud 134, the projector 136, and / or any other system or component not part of the system 100). The communication interface 108 may comprise one or more wired interfaces (e.g., a USB port, an Ethernet port, a Firewire port) and / or one or more wireless transceivers or interfaces (configured, for example, to transmit and / or receive information via one or more wireless communication protocols such as 802.1 la / b / g / n, Bluetooth, NFC, ZigBee, and so forth). In some embodiments, the communication interface 108 may be useful for enabling the device 102 to communicate with one or more other processors 104 or computing devices 102, whether to reduce the time needed to accomplish a computing-intensive task or for any other reason.

[0071] The computing device 102 may also comprise one or multiple user interfaces 110. The user interface(s) 110 may be or comprise a keyboard, mouse, trackball, monitor, television, screen, touchscreen, and / or any other device for receiving information from a user and / or for providing information to a user. The user interface(s) 110 may be used, for example, to receive a user selection or other user input regarding any step of any method described herein. Notwithstanding the foregoing, any required input for any step of any method described herein may be generated automatically by the system 100 (e.g., by the processor 104 or another component of the system 100)or received by the system 100 from a source external to the system 100. In some embodiments, the user interface 110 may be useful to allow a surgeon or other user to modify instructions to be executed by the processor 104 according to one or more embodiments of the present disclosure, and / or to modify or adjust a setting of other information displayed on the user interface 110 or corresponding thereto.

[0072] Although the user interface 110 is shown as part of the computing device 102, in some embodiments, the computing device 102 may utilize a user interface 110 that is housed separately from one or more remaining components of the computing device 102. In some embodiments, the user interface 110 may be located proximate one or more other components of the computing device 102, while in other embodiments, the user interface 110 may be located remotely from one or more other components of the computing device 102.

[0073] The imaging device 112 may be operable to image anatomical feature(s) (e.g., a bone, veins, tissue, etc.) and / or other aspects of patient anatomy to yield image data (e.g., image data depicting or corresponding to a bone, veins, tissue, etc.). “Image data” as used herein refers to the data generated or captured by an imaging device 112, including in a machine-readable form, a graphical / visual form, and in any other form. In some cases, the image data may be or comprise 3D image data generated by one or more 3D imaging device(s) (e.g., an O-arm, a C-arm, a G-arm, a CT scanner, etc.) and / or 2D image data generated by one or more 2D imaging device(s) (e.g., an emitter / detector pair). In various examples, the image data may comprise data corresponding to an anatomical feature of a patient, or to a portion thereof. The image data may be or comprise a preoperative image, an intraoperative image, a postoperative image, or an image taken independently of any surgical procedure. The imaging device 112, the 3D imaging device(s), and / or the 2D imaging device(s) may be capable of taking a 2D image or a 3D image to yield the image data. The imaging device 112 may be or comprise, for example, a stereo camera, an ultrasound scanner (which may comprise, for example, a physically separate transducer and receiver, or a single ultrasound transceiver), an O-arm, a C-arm, a G-arm, or any other device utilizing X-ray-based imaging (e.g., a fluoroscope, a CT scanner, or other X-ray machine), a magnetic resonance imaging (MRI) scanner, an optical coherence tomography (OCT) scanner, an endoscope, a microscope, an optical camera, a thermographic camera (e.g., an infrared camera), a radar system (which may comprise, for example, a transmitter, a receiver, a processor, and one or more antennae), or any other imaging device 112 suitable for obtaining images of an anatomical feature of a patient.

[0074] The imaging device 112 may be contained entirely within a single housing, or may comprise a transmitter / emitter and a receiver / detector that are in separate housings or are otherwisephysically separated. In some embodiments, a first imaging device 112 may be used to obtain first image data (e.g., a first image) at a first time, and a second imaging device 112 may be used to obtain second image data (e.g., a second image) at a second time after the first time. In some examples (and as discussed in further detail below) the first imaging device can be an imaging device that is used for navigation (e.g., in conjunction with the navigation system 118). The first imaging device may be or comprise, for example, any of the example imaging devices described above (an ultrasound scanner, an O-arm, a C-arm, a G-arm, or any other device utilizing X-raybased imaging, a magnetic resonance imaging scanner, an OCT scanner, an endoscope, a microscope, an optical camera, a thermographic camera, a radar system, a stereo camera, etc.). The second imaging device may be an imaging device used for registration (e.g., to facilitate alignment of patient scan data with the patient in the surgical environment). The second imaging device may be or comprise a stereo camera, as discussed in further detail below.

[0075] In some embodiments, the imaging device 112 may comprise more than one imaging device 112. For example, a first imaging device may provide first image data and / or a first image, and a second imaging device may provide second image data and / or a second image. In still other embodiments, the same imaging device may be used to provide both the first image data and the second image data, and / or any other image data described herein. The imaging device 112 may be operable to generate a stream of image data. For example, the imaging device 112 may be configured to operate with an open shutter, or with a shutter that continuously alternates between open and shut so as to capture successive images. For purposes of the present disclosure, unless specified otherwise, image data may be considered to be continuous and / or provided as an image data stream if the image data represents two or more frames per second. In some embodiments, reference markers (e.g., navigation markers) may be placed on the imaging device 112 and / or any other object in the surgical space. The reference markers may be tracked by the navigation system 118, and the results of the tracking may be used by an operator of the system 100 or any component thereof.

[0076] The projector 136 may be or comprise a vertical-cavity surface-emitting laser (VCSEL) or a simple laser with a diffraction grating to project a pattern. In some cases, the projector 136 may comprise any projector 136 configured to project any pattern such as, for example, a digital light processing (DLP) projector, a light emitting diode (LED) projector, an LCD projector, or a liquid crystal on silicon (LCOS) projector). The projector 136 may be supported manually by, for example, a user, or may be statically supported by, for example, a stand. In other embodiments, the projector136 may be supported and positioned by, for example, the robot 114 (and more specifically, by a robotic arm 116 of the robot 114).

[0077] The pattern may be projected onto, for example, a patient, an object, or a combination thereof. The projected pattern may aid in identifying at least a portion of a patient onto which the pattern is projected onto (wherein the patient may be identified in the image data via, for example, by the processor 104 using the image processing 120 described above). The pattern may be any pattern such as, for example, a plurality of dots, a plurality of lines (e.g., a set of 2D grid lines), a noise pattern, a plurality of any shape, or a plurality of any combination of shapes.

[0078] The robot 114 may be any surgical robot or surgical robotic system. The robot 114 may be or comprise, for example, the Mazor X™ Stealth Edition robotic guidance system. The robot 114 may be configured to position the imaging device 112 at one or more precise position(s) and orientation(s), and / or to return the imaging device 112 to the same position(s) and orientation(s) at a later point in time. The robot 114 may additionally or alternatively be configured to manipulate a surgical tool (whether based on guidance from the navigation system 118 or not) to accomplish or to assist with a surgical task. In some embodiments, the robot 114 may be configured to hold and / or manipulate an anatomical element during or in connection with a surgical procedure. The robot 114 may comprise one or more robotic arms 116. In some embodiments, the robotic arm 116 may comprise a first robotic arm and a second robotic arm, though the robot 114 may comprise more than two robotic arms. In some embodiments, one or more of the robotic arms 116 may be used to hold and / or maneuver the imaging device 112. In embodiments where the imaging device 112 comprises two or more physically separate components (e.g., a transmitter and receiver), one robotic arm 116 may hold one such component, and another robotic arm 116 may hold another such component.Each robotic arm 116 may be positionable independently of the other robotic arm. The robotic arms 116 may be controlled in a single, shared coordinate space, or in separate coordinate spaces.

[0079] The robot 114, together with the robotic arm 116, may have, for example, one, two, three, four, five, six, seven, or more degrees of freedom. Further, the robotic arm 116 may be positioned or positionable in any pose, plane, and / or focal point. The pose includes a position and an orientation. As a result, an imaging device 112, surgical tool, or other object held by the robot 114 (or, more specifically, by the robotic arm 116) may be precisely positionable in one or more needed and specific positions and orientations.

[0080] The robotic arm(s) 116 may comprise one or more sensors that enable the processor 104 (or a processor of the robot 114) to determine a precise pose in space of the robotic arm (as well as any object or element held by or secured to the robotic arm).

[0081] In some embodiments, reference markers (e.g., navigation markers) may be placed on the robot 114 (including, e.g., on the robotic arm 116), the imaging device 112, or any other object in the surgical space. The reference markers may be tracked by the navigation system 118, and the results of the tracking may be used by the robot 114 and / or by an operator of the system 100 or any component thereof. In some embodiments, the navigation system 118 can be used to track other components of the system (e.g., imaging device 112) and the system can operate without the use of the robot 114 (e.g., with the surgeon manually manipulating the imaging device 112 and / or one or more surgical tools, based on information and / or instructions generated by the navigation system 118, for example).

[0082] The navigation system 118 may provide navigation during an operation. The navigation system 118 may be any now-known or future-developed navigation system, including, for example, the Medtronic StealthStation™ S8 surgical navigation system or any successor thereof. The navigation system 118 may include one or more cameras or other sensor(s) for tracking one or more reference markers, navigated trackers, or other objects within the operating room or other room in which some or all of the system 100 is located. The one or more cameras may be optical cameras, infrared cameras, or other cameras. In some embodiments, the navigation system 118 may comprise one or more electromagnetic sensors. In various embodiments, the navigation system 118 may be used to track a position and orientation (e.g., a pose) of the imaging device 112, the robot 114 and / or the robotic arm 116, and / or one or more surgical tools (or, more particularly, to track a pose of a navigated tracker attached, directly or indirectly, in fixed relation to the one or more of the foregoing). The navigation system 118 may include a display for displaying one or more images from an external source (e.g., the computing device 102, imaging device 112, the projector 136, or other source) or for displaying an image and / or video stream from the one or more cameras or other sensors of the navigation system 118. In some embodiments, the system 100 can operate without the use of the navigation system 118. The navigation system 118 may be configured to provide guidance to a surgeon or other user of the system 100 or a component thereof or to any other element of the system 100 (e.g., the robot 114) regarding, for example, a pose of one or more anatomical elements, whether or not a tool is in the proper trajectory, and / or how to move a tool into the proper trajectory to carry out a surgical task according to a preoperative or other surgical plan.

[0083] The database 130 may store information that correlates one coordinate system to another (e.g., a patient coordinate system to a navigation coordinate system or vice versa). The database 130 may additionally or alternatively store, for example, one or more surgical plans (including, for example, pose information about a target and / or image information about a patient’s anatomy atand / or proximate the surgical site, for use by the robot 114, the navigation system 118, and / or a user of the computing device 102 or of the system 100); one or more images useful in connection with a surgery to be completed by or with the assistance of one or more other components of the system 100; one or more optimization values or information (e.g., threshold values for use in determining when the objective function 144 is optimized); information about one or more light patterns (e.g., the shape of light patterns that can be projected by the projector 136); and / or any other useful information. The database 130 may be configured to provide any such information to the computing device 102 or to any other device of the system 100 or external to the system 100, whether directly or via the cloud 134. In some embodiments, the database 130 may be or comprise part of a hospital image storage system, such as a picture archiving and communication system (PACS), a health information system (HIS), and / or another system for collecting, storing, managing, and / or transmitting electronic medical records including image data.

[0084] The cloud 134 may be or represent the Internet or any other wide area network. The computing device 102 may be connected to the cloud 134 via the communication interface 108, using a wired connection, a wireless connection, or both. In some embodiments, the computing device 102 may communicate with the database 130 and / or an external device (e.g., a computing device) via the cloud 134.

[0085] The system 100 or similar systems may be used, for example, to carry out one or more aspects of any of the methods 400 and / or 500 described herein. The system 100 or similar systems may also be used for other purposes.

[0086] Figs. 2A-2C illustrate additional aspects of the system 100 in accordance with embodiments of the present disclosure. Fig. 2A depicts a projector 204 and an imaging device 212 positioned relative to an object 208, which in some cases may be a patient or a portion thereof (e.g., the head of the patient). In some examples, the projector 204 may be similar to or the same as the projector 136 or any other projector depicted and / or described herein, and the imaging device 212 may be similar to or the same as the imaging device 112 or any other imaging device depicted and / or described herein.

[0087] In some cases, the projector 204 may be programmed or otherwise configured to project a light field onto the object 208. The light field may be based on one or more predetermined patterns. In one embodiment, a light pattern 216 may be projected onto the object 208. The light pattern 216 may be a predetermined or known pattern that may aid in identifying at least a portion of a patient onto which the pattern is projected. The pattern may be any pattern such as, for example, a plurality of dots, a plurality of lines (e.g., a set of 2D grid lines), a noise pattern, a plurality of any shape, or aplurality of any combination of shapes. As one example and as depicted in Fig. 2B, the light pattern 216 may comprise a plurality of horizontal light lines 220A-220N that, when projected onto the object 208, illuminate the object 208. The light lines 220A-220N may comprise a first light line 220 A, a second light line 220B, etc. up to an nth light line 220N.

[0088] The imaging device 212 may capture an image 224 depicting the object 208 illuminated by the light pattern 216 after the light pattern 216 has been projected onto the object 208. Due to the surface geometry of the object 208, one or more portions of the light lines 220A-220N may be distorted, as illustrated by the lines 228 on the surface of the object 208. The processor 104 may use image processing 120 to determine the shift of the lines 228 relative to the known light pattern 216. Then, based on the determined shift and the known pose of the imaging device 212, the processor 104 may determine the distance between the surface of the object 208 and the imaging device 212 at each point along the lines 228. Given the known pose of the projector 204 relative to the imaging device 212 (and vice versa), the processor 104 can determine the pose of the object 208 relative to the imaging device 212 after several iterations of projecting a known light pattern (e.g., light pattern 216) onto the object 208 and using image processing 120 to determine the distance between the surface of the object 208 and the imaging device 212.

[0089] The pose of the projector 204 relative to the imaging device 212 (and vice versa) may be known or determined by the system 100. In some cases, the pose information may be stored in the database 130. Additionally or alternatively, the pose information may be determined by the navigation system 118 based on the pose of tracked fiducials in a known pose relative to the projector 204 and / or the imaging device 212.

[0090] With reference to Figs. 3A-3B, aspects of the path tracing simulation 140 are shown in accordance with embodiments of the present disclosure. Fig. 3 A depicts a virtual projector 304 that projects a light pattern onto a virtual object 308, as well as a virtual imaging device 312 that captures a virtual image of the virtual object 308 illuminated by the light pattern. In some cases, the path tracing simulation 140 operates to simulate the functionality of the projector 204 and the imaging device 212, except the path tracing simulation 140 flips the positions of the virtual projector 304 and the virtual imaging device 312 relative to the virtual object 308. In other words, the virtual projector 304 is positioned in the same pose as the imaging device 212 would be, and the virtual imaging device 312 is positioned in the same pose as the projector 204 would be.

[0091] The path tracing simulation 140 may be used to determine an appropriate light pattern to be projected from the projector 204. The path tracing simulation 140 may emit, from the virtual projector 304, a predetermined light pattern (such as the light pattern 216) onto the virtual object308. The geometry of the virtual object 308 may be known based on, for example, one or more models of the object 208. Such models may be generated, for example, from one or more preoperative and / or intraoperative scans of the patient. The pose of the virtual object 308 relative to the virtual projector 304 and / or the virtual imaging device 312 may be based on an initial guess of the pose of the object 208. The path tracing simulation 140 may generate trace lines 316 representative of the light pattern 216 as the light pattern 216 is shone on the virtual object 308. The path tracing simulation 140 may be used by the processor 104 to determine, based on the geometry of the virtual object 308 and the trace lines 316, resulting trace lines 320 that would reach a virtual imaging device 312. Based on the angles of the trace lines 320 representative of light reflecting off the virtual object 308, the processor 104 may generate a virtual image 324 representative of a view that would be seen by the virtual imaging device 312. In some embodiments, the light pattern 216, the virtual imaging device 312, the virtual object 308, and / or the like may be rendered to a display (e.g., user interface 110) to permit a user to view and / or interact with aspects of the path tracing simulation 140.

[0092] By flipping the relative pose of projector and imaging device, the path tracing simulation 140 can determine, based on the view seen by the virtual camera (e.g., the virtual image 324), a light pattern that can be projected by the projector 204 to confirm that the pose of the virtual object 308 (which corresponds to an initial guess of the pose of the object 208) matches the actual pose of the object 208. In other words, the trace lines 320 depicted in the virtual image 324 may be used as the light pattern that is emitted from the projector 204 and shone on the object 208. When the virtual object 308 and the object 208 have the same or substantially the same pose, then the difference (e.g., error) between the reference light pattern initially emitted by the virtual projector 304 in the path tracing simulation 140 and the light pattern appearing on the object 208 in images captured by the imaging device 212 will be small. When the virtual object 308 and the object 208 do not have the same or substantially the same pose, then the difference in pose will result in distortions to the light pattern depicted on the object 208 in images captured by the imaging device 212, and the distance (e.g., error) between the light patterns will be large. As a result, when the differences between the reference light pattern projected from the virtual projector 304 and the light pattern depicted in the image of the object 208 captured by the imaging device 212 are small (e.g., meet or fall below a threshold value), the processor 104 can confirm that the actual pose of the object 208 matches the initial guess of the pose of the virtual object 308, and can use such information to register the object 208 with the imaging device 212.

[0093] As used herein and unless otherwise specified, “substantially corresponds to” or “substantially matches” means that the difference between the actual pose of the object and the initial guess or predicted pose of the object meets a predetermined condition (e.g., meets or falls below a threshold value). In other words, the actual pose of the object and the initial guess may not be exact, but rather are sufficiently close enough to perform registration and carry out a surgery or surgical procedure. In some embodiments, the predetermined condition may be defined by the user (e.g., a physician or surgeon), based on a value or information retrieved from the database 130, based on various governmental regulatory requirements, combinations thereof, and / or the like.

[0094] With reference to Figs. 3C-3E, various images captured by the imaging device 212 depicting the object 208 illuminated by a light pattern are shown in accordance with embodiments of the present disclosure. The images may correspond to the views seen by the imaging device 212 when different initial guesses of the pose of the object 208 are input into the path tracing simulation 140. The path tracing simulation 140 may begin by receiving a reference pattern 348 (which may correspond to a set of 2D grid lines) and an input indicative of an initial guess of the pose of the object 208. Based on the initial guess, the path tracing simulation 140 may project the reference pattern 348 from the virtual projector 304 onto the virtual object 308 (which has been oriented in accordance with the initial guess) and, using trace lines 316 and trace lines 320, determine a view (e.g., virtual image 324) of the virtual imaging device 312. The view may then be used as a light pattern projected by the projector 136. In other words, the trace lines 320 depicted in the virtual image 324 may be the light pattern emitted by the projector 204. Once the light pattern is shone on the object 208, the imaging device 212 may be used to capture the images depicted in Figs. 3C-3E.

[0095] Fig. 3C depicts a first image 328 of the object 208. The first image 328 may correspond to an image depicting the object 208 when the initial guess of the object 208 is several millimeters (mm) too close to the imaging device 212. In other words, the initial guess of the pose of the object 208 does not match the actual pose of the object 208. As a result, the path tracing simulation 140 may generate a light pattern for use that, when shone by the projector 204, results in a distorted light pattern 332 (rather than the desired reference pattern 348) appearing on the object 208. Similarly in Fig. 3D, a second image 336 may correspond to a scenario where the initial guess of the object 208 is several millimeters too far away from the imaging device 212. As a result, the path tracing simulation 140 may generate a light pattern for use that, when shone by the projector 204, results in a distorted light pattern 340 (rather than the desired reference pattern 348) appearing on the object 208.

[0096] In Fig. 3E, the initial guess of the pose of the object 208 may correspond to or substantially correspond to the actual pose of the object 208. In this case, the virtual object 308 matches or substantially matches the pose of the object 208, such that the path tracing simulation 140 generates a light pattern for use that, when shone by the projector 204, results in the reference pattern 348 appearing on the object 208. In this example, since the reference pattern 348 appears in both the third image 344 captured by the imaging device 212 and in the pattern generated by the virtual projector 304, the processor 104 may determine that the initial guess of the pose of the object 208 and the actual pose of the object 208 match. The processor 104 may then use registration 128 to register the imaging device 212 with the object 208. Such registration may beneficially enable and / or facilitate the one or more components of the system 100 to carry out out one or more surgical tasks relative to the object 208 (e.g., the head of the patient).

[0097] Fig. 4 depicts a method 400 that may be used, for example, to register an imaging device with an object.

[0098] The method 400 (and / or one or more steps thereof) may be carried out or otherwise performed, for example, by at least one processor. The at least one processor may be the same as or similar to the processor(s) 104 of the computing device 102 described above. The at least one processor may be part of a robot (such as a robot 114) or part of a navigation system (such as a navigation system 118). A processor other than any processor described herein may also be used to execute the method 400. The at least one processor may perform the method 400 by executing elements stored in a memory such as the memory 106. The elements stored in memory and executed by the processor may cause the processor to execute one or more steps of a function as shown in method 400. One or more portions of a method 400 may be performed by the processor executing any of the contents of memory, such as image processing 120, segmentation 122, transformation 124, registration 128, the path tracing simulation 140, and / or the objective function 144.

[0099] The method 400 comprises receiving an input representing an initial guess of a pose of an object in 3D space (step 404). The input may be received from a user (e.g., a physician) via the user interface 110, or may be automatically retrieved from the database 130. The initial guess may specify the pose (e.g., position and orientation) of the object (e.g., object 208) in 3D space. The object may be or comprise a portion of a patient, such as the patient’s head.

[0100] The method 400 also comprises determining, based on the input, an illumination that, when projected from a projector, results in a first light pattern appearing on the object (step 408). The projector may be similar to or the same as the projector 204 and / or the projector 136. The illumination may be similar to or the same as the trace lines 320 depicted in the virtual image 324,while the first light pattern that appears on the object may be similar to or the same as the reference pattern 348. In other words, the step 408 may include using the path tracing simulation 140 to generate the illumination by using the virtual projector 304, the virtual object 308, and the virtual imaging device 312 based on the reference pattern 348.

[0101] The method 400 also comprises causing the projector to project the illumination onto the object (step 412). Once the illumination has been determined, the projector 204 (which may be positioned in the same pose as the virtual imaging device 312 of the path tracing simulation 140) may project the illumination on the object 208.

[0102] The method 400 also comprises receiving, from an imaging device, an image depicting a second light pattern appearing on the object (step 416). The imaging device (which may be positioned in the same pose as the virtual projector 304 of the path tracing simulation 140) may capture an image of the object. The imaging device may be similar to or the same as the imaging device 212 and / or the imaging device 112. The object may be illuminated such that a second light pattern appears on the object (e.g., distorted light pattern 332, distorted light pattern 340, reference pattern 348, etc.), depending on how close the initial guess of the pose of the object was to the actual pose of the object. In some embodiments, image processing 120 may be used on the captured image to determine the second light pattern.

[0103] The method 400 also comprises determining a difference between the first light pattern and the second light pattern (step 420). The difference between the first light pattern and the second light pattern may be based on the overall difference in distance between the shapes that make up the first light pattern and the shapes that make up the second light pattern. When the light patterns comprise sets of lines, the difference may be based on the sum of the squared error between each line in the first light pattern with a corresponding line in the second light pattern. It is to be understood that, while sum of the squared error is provided as an example, such an example is not limited, and other mathematical differences may be used in determining the difference between the first light pattern and the second light pattern.

[0104] The method 400 also comprises determining, when the difference satisfies a predetermined condition, that the initial guess substantially corresponds to an actual pose of the object (step 424). The predetermined condition may be or comprise a threshold value (e.g., a value provided by the user, a value retrieved from the database 130, etc.), such that when the difference meets or falls below the threshold value, the initial guess substantially corresponds to the actual pose of the object. In other examples, other types of predetermined conditions may be used to determine that the initial guess substantially corresponds to the actual pose of the object.

[0105] The method 400 also comprises registering, in response to determining that the initial guess corresponds to the actual pose of the object, the imaging device with the object (step 428). The processor 104 may use registration 128 to register the imaging device 212 with the object 208.

[0106] The present disclosure encompasses embodiments of the method 400 that comprise more or fewer steps than those described above, and / or one or more steps that are different than the steps described above.

[0107] Fig. 5 depicts a method 500 that may be used, for example, to iteratively optimize an objective function for registration.

[0108] The method 500 (and / or one or more steps thereof) may be carried out or otherwise performed, for example, by at least one processor. The at least one processor may be the same as or similar to the processor(s) 104 of the computing device 102 described above. The at least one processor may be part of a robot (such as a robot 114) or part of a navigation system (such as a navigation system 118). A processor other than any processor described herein may also be used to execute the method 500. The at least one processor may perform the method 500 by executing elements stored in a memory such as the memory 106. The elements stored in memory and executed by the processor may cause the processor to execute one or more steps of a function as shown in method 500. One or more portions of a method 500 may be performed by the processor executing any of the contents of memory, such as image processing 120, segmentation 122, transformation 124, registration 128, the path tracing simulation 140, and / or the objective function 144.

[0109] The method 500 comprises generating, using a virtual projector, a first light pattern that is projected onto a virtual object (step 504). In some cases, the steps 504, 508, and 512 may be part of the step 408 of the method 400 described above. The virtual projector may be similar to or the same as the virtual projector 304. The first light pattern may be a predetermined, received, and / or known light pattern (e.g., light pattern 216, reference pattern 348, etc.) that can be projected onto the virtual object, which may be similar to or the same as the virtual object 308.

[0110] The method 500 also comprises determining a view seen by a virtual imaging device when the first light pattern is projected onto the virtual object (step 508). The step 508 may be performed wholly or in part using the path tracing simulation 140, which may determine the view seen by the virtual imaging device (e.g., virtual imaging device 312) based on trace lines 316 and trace lines 320 as the first light pattern is shone on and reflects off the virtual object 308. In one example, the view may be similar to or the same as the virtual image 324.

[0111] The method 500 also comprises determining, based on the view, an illumination to project from the projector onto the object (step 512). The step 512 may be performed wholly or in part usingthe path tracing simulation 140, which may use the trace lines 320 to determine the illumination to be emitted from the projector 204. For example, the path tracing simulation 140 may determine that the trace lines 320 as seen from the view from the virtual imaging device 312 should be used as the illumination. In this example, the projector 204 may be adjusted such that the illumination emitted from the projector 204 onto the object 208 corresponds to the same shape and orientation as the trace lines 320 of the virtual image 324. In some embodiments, the step 512 may continue onto the steps 412, 416, and 420 of the method 400, where the projector projects the illumination, the imaging device captures an image of the object under the illumination with a second light pattern, and the difference between the first light pattern and the second light pattern is determined, respectively.

[0112] The method 500 also comprises determining if an objective function is optimized (step 516). The objective function may be considered optimized when the overall value of the objective function — which may correspond to the difference between the first light pattern and the second light pattern — reaches a minimum or maximum value. The minimum or maximum value may be determined based on a predetermined threshold value (e.g., a value stored in the database 130). Additionally or alternatively, the objective function may be considered optimized after a predetermined or selected number of iterations (e.g., after 12 iterations, the transformation used in the 12th iteration is selected as being considered optimized), the best fit obtained over a range of iterations (e.g., the objective function is optimized 15 times, and the 13th iteration produces the maximum value, so the transformation matrix associated with the 13th iteration is used), and / or the like. Any one or more criteria may be used to determine if the objective function has been optimized. If the objective function is optimized, the step 516 may proceed to the step 424 of the method 400, where the initial guess of the pose of the object is determined to substantially correspond to the actual pose of the object.

[0113] The method 500 also comprises generating, when the objective function is not optimized, another guess of the pose of the object (step 520). When the objective function can be further optimized (e.g., there is a change to the guess of the pose of the object 208 could lead to a lower objective function value or otherwise reduce the difference between the first light pattern and the second light pattern), the processor 104 may determine an adjustment to the initial guess of the pose of the object 208 to lower the overall objective function value. The optimization may use one or more numerical optimization methods (e.g., gradient descent, stochastic gradient descent, Newton’s method, etc.) to adjust the objective function toward a maximum or minimum value. The new guess of the pose of the object 208 may then be passed back to the step 504, and the steps 504, 508, and 512 may be repeated to produce an updated illumination that is shone on the object 208.

[0114] The present disclosure encompasses embodiments of the method 500 that comprise more or fewer steps than those described above, and / or one or more steps that are different than the steps described above.

[0115] As noted above, the present disclosure encompasses methods with fewer than all of the steps identified in Figs. 4 and 5 (and the corresponding description of the methods 400 and 500), as well as methods that include additional steps beyond those identified in Figs. 4 and 5 (and the corresponding description of the methods 400 and 500). The present disclosure also encompasses methods that comprise one or more steps from one method described herein, and one or more steps from another method described herein. Any correlation described herein may be or comprise a registration or any other correlation.

[0116] The foregoing is not intended to limit the disclosure to the form or forms disclosed herein. In the foregoing Detailed Description, for example, various features of the disclosure are grouped together in one or more aspects, embodiments, and / or configurations for the purpose of streamlining the disclosure. The features of the aspects, embodiments, and / or configurations of the disclosure may be combined in alternate aspects, embodiments, and / or configurations other than those discussed above. This method of disclosure is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects he in less than all features of a single foregoing disclosed aspect, embodiment, and / or configuration. Thus, the following claims are hereby incorporated into this Detailed Description, with each claim standing on its own as a separate preferred embodiment of the disclosure.

[0117] Moreover, though the foregoing has included description of one or more aspects, embodiments, and / or configurations and certain variations and modifications, other variations, combinations, and modifications are within the scope of the disclosure, e.g., as may be within the skill and knowledge of those in the art, after understanding the present disclosure. It is intended to obtain rights which include alternative aspects, embodiments, and / or configurations to the extent permitted, including alternate, interchangeable and / or equivalent structures, functions, ranges or steps to those claimed, whether or not such alternate, interchangeable and / or equivalent structures, functions, ranges or steps are disclosed herein, and without intending to publicly dedicate any patentable subject matter.

[0118] A set of example statements are provided below:

[0119] Example 1: A system, comprising: a processor (104); and a memory (106) storing data thereon that, when processed by the processor (104), enable the processor (104) to: receive an inputrepresenting an initial guess of a pose of an object (208) in three-dimensional (3D) space; determine, based on the input, an illumination that, when projected from a projector (136, 204), results in a first light pattern appearing on the object (208); cause the projector (136, 204) to project the illumination onto the object (208); receive, from an imaging device (112, 212), an image depicting a second light pattern appearing on the object (208); determine a difference between the first light pattern and the second light pattern; determine, when the difference satisfies a predetermined condition, that the initial guess substantially corresponds to an actual pose of the object (208); and register, in response to determining that the initial guess corresponds to the actual pose of the object (208), the imaging device (112, 212) with the object (208).

[0120] Example 2. The system of Example 1, wherein the memory (106) comprises further data that, when processed by the processor (104), enables the processor (104) to: determine, using a path tracing simulation (140), a view seen by a virtual imaging device (312) when a virtual projector (304) projects the first light pattern onto the object (208).

[0121] Example 3: The system of Example 2, wherein, in the path tracing simulation (140), the virtual imaging device (312) is positioned in a same pose as the projector (136, 204) and the virtual projector (304) is positioned in a same pose as the imaging device (112, 212).

[0122] Example 4: The system of any of Examples 1-3, wherein the memory (106) comprises further data that, when processed by the processor (104), enables the processor (104) to: iteratively optimize an objective function (144) associated with the difference between the first light pattern and the second light pattern.

[0123] Example 5: The system of Example 4, wherein as part of iteratively optimizing the objective function (144), the processor (104) further: determines, based on the difference, a change to the initial guess to reduce the difference; and generates, based on the change, a second guess of the pose of the object (208).

[0124] Example 6: The system of Example 5, wherein as part of iteratively optimizing the objective function (144), the processor (104) further: generates, based on the second guess of the pose, a second illumination that, when projected from the projector (136, 204), results in the first light pattern appearing on the object (208); causes the projector (136, 204) to project the second illumination onto the object (208); receives, from the imaging device (112, 212), a second image of the object (208) illuminated by a third light pattern; and determines a difference between the first light pattern and the third light pattern.

[0125] Example 7: The system of any of Examples 1-6, wherein the first light pattern comprises a set of two-dimensional (2D) grid lines.

[0126] Example 8: The system of any of Examples 1-7, wherein the object (208) comprises a portion of a patient.

[0127] Example 9: A system, comprising: an imaging device (112, 212); a projector (136, 204); a processor (104); and a memory (106) storing data thereon that, when processed by the processor (104), enable the processor (104) to: determine, based on an input representing initial guess of a pose of an object (208) in three-dimensional (3D) space, an illumination that, when emitted from the projector (136, 204), results in a first light pattern appearing on the object (208); cause the projector (136, 204) to emit the illumination to illuminate the object (208); cause the imaging device (112, 212) to capture an image the object (208), wherein the image depicts a second light pattern appearing on the object (208); determine a difference between the first light pattern and the second light pattern; and determine, when the difference satisfies a predetermined condition, that the initial guess substantially corresponds to an actual pose of the object (208).

[0128] Example 10: The system of Example 9, wherein the memory (106) comprises further data that, when processed by the processor (104), enables the processor (104) to:

[0129] determine, using a path tracing simulation (140), a view seen by a virtual imaging device (312) when a virtual projector (304) projects the first light pattern onto the object (208).

[0130] Example 11 : The system of Example 10, wherein, in the path tracing simulation (140), the virtual imaging device (312) is positioned in a same pose as the projector (136, 204) and the virtual projector (304) is positioned in a same pose as the imaging device (112, 212).

[0131] Example 12: The system of any of Examples 9-11, wherein the memory (106) comprises further data that, when processed by the processor (104), enables the processor (104) to:

[0132] iteratively optimize an objective function (144) associated with the difference between the first light pattern and the second light pattern.

[0133] Example 13: The system of Example 12, wherein as part of iteratively optimizing the objective function (144), the processor (104) further: determines, based on the difference, a change to the initial guess to reduce the difference; and generates, based on the change, a second guess of the pose of the object (208).

[0134] Example 14: The system of Example 13, wherein as part of iteratively optimizing the objective function (144), the processor (104) further: generates, based on the second guess of the pose, a second illumination that, when projected from the projector (136, 204), results in the first light pattern appearing on the object (208); causes the projector (136, 204) to project the second illumination onto the object (208); receives, from the imaging device (112, 212), a second image ofthe object (208) illuminated by a third light pattern; and determines a difference between the first light pattern and the third light pattern.

[0135] Example 15: The system of any of Examples 9-14, wherein the first light pattern comprises a set of two-dimensional (2D) grid lines.

[0136] Example 16: The system of any of Examples 9-15, wherein the object (208) comprises a portion of a patient.

[0137] Example 17: The system of any of Examples 9-16, wherein the memory (106) stores further data that, when processed by the processor (104), enable the processor (104) to: register, in response to determining that the initial guess corresponds to the actual pose of the object (208), the imaging device (112, 212) with the object (208).

[0138] Example 18: A method, comprising: receiving an input representing an initial guess of a pose of an object (208) in three-dimensional (3D) space; generating, based on the input, an illumination that, when projected from a projector (136, 204), results in a first light pattern appearing on the object (208); causing the projector (136, 204) to project the illumination onto the object (208); receiving, from an imaging device (112, 212), an image that depicts the object (208) with a second light pattern; determining a difference between the first light pattern and the second light pattern; determining, when the difference satisfies a predetermined condition, that the initial guess substantially corresponds to an actual pose of the object (208); and registering, in response to determining that the initial guess corresponds to the actual pose of the object (208), the imaging device (112, 212) with the object (208).

[0139] Example 19: The method of Example 18, further comprising: iteratively optimizing an objective function (144) associated with the difference between the first light pattern and the second light pattern.

[0140] Example 20: The method of Example 19, wherein the iteratively optimizing comprises: generating, based on a second guess of the pose, a second illumination that, when projected from the projector (136, 204), results in the first light pattern appearing on the object (208); causing the projector (136, 204) to project the second illumination onto the object (208); receiving, from the imaging device (112, 212), a second image that depicts the object (208) with a third light pattern; and determining a difference between the first light pattern and the third light pattern.

Claims

CLAIMSWhat is claimed is:

1. A system, comprising: a processor (104); and a memory (106) storing data thereon that, when processed by the processor (104), enable the processor (104) to: receive an input representing an initial guess of a pose of an object (208) in three- dimensional (3D) space; determine, based on the input, an illumination that, when projected from a projector (136, 204), results in a first light pattern appearing on the object (208); cause the projector (136, 204) to project the illumination onto the object (208); receive, from an imaging device (112, 212), an image depicting a second light pattern appearing on the object (208); determine a difference between the first light pattern and the second light pattern; determine, when the difference satisfies a predetermined condition, that the initial guess substantially corresponds to an actual pose of the object (208); and register, in response to determining that the initial guess corresponds to the actual pose of the object (208), the imaging device (112, 212) with the object (208).

2. The system of claim 1, wherein the memory (106) comprises further data that, when processed by the processor (104), enables the processor (104) to: determine, using a path tracing simulation (140), a view seen by a virtual imaging device (312) when a virtual projector (304) projects the first light pattern onto the object (208).

3. The system of claim 2, wherein, in the path tracing simulation (140), the virtual imaging device (312) is positioned in a same pose as the projector (136, 204) and the virtual projector (304) is positioned in a same pose as the imaging device (112, 212).

4. The system of any of claims 1-3, wherein the memory (106) comprises further data that, when processed by the processor (104), enables the processor (104) to: iteratively optimize an objective function (144) associated with the difference between the first light pattern and the second light pattern.

5. The system of claim 4, wherein as part of iteratively optimizing the objective function (144), the processor (104) further: determines, based on the difference, a change to the initial guess to reduce the difference; and generates, based on the change, a second guess of the pose of the object (208).

6. The system of claim 5, wherein as part of iteratively optimizing the objective function (144), the processor (104) further: generates, based on the second guess of the pose, a second illumination that, when projected from the projector (136, 204), results in the first light pattern appearing on the object (208); causes the projector (136, 204) to project the second illumination onto the object (208); receives, from the imaging device (112, 212), a second image of the object (208) illuminated by a third light pattern; and determines a difference between the first light pattern and the third light pattern.

7. The system of any of claims 1 -6, wherein the first light pattern comprises a set of two-dimensional (2D) grid lines.

8. The system of any of claims 1-7, wherein the object (208) comprises a portion of a patient.

9. A system, comprising: an imaging device (112, 212); a projector (136, 204); a processor (104); and a memory (106) storing data thereon that, when processed by the processor (104), enable the processor (104) to: determine, based on an input representing initial guess of a pose of an object (208) in three-dimensional (3D) space, an illumination that, when emitted from the projector (136, 204), results in a first light pattern appearing on the object (208); cause the projector (136, 204) to emit the illumination to illuminate the object (208); cause the imaging device (112, 212) to capture an image the object (208), wherein the image depicts a second light pattern appearing on the object (208); determine a difference between the first light pattern and the second light pattern; anddetermine, when the difference satisfies a predetermined condition, that the initial guess substantially corresponds to an actual pose of the object (208).

10. The system of claim 9, wherein the memory (106) comprises further data that, when processed by the processor (104), enables the processor (104) to: determine, using a path tracing simulation (140), a view seen by a virtual imaging device (312) when a virtual projector (304) projects the first light pattern onto the object (208).

11. The system of claim 10, wherein, in the path tracing simulation (140), the virtual imaging device (312) is positioned in a same pose as the projector (136, 204) and the virtual projector (304) is positioned in a same pose as the imaging device (112, 212).

12. The system of any of claims 9-11, wherein the memory (106) comprises further data that, when processed by the processor (104), enables the processor (104) to: iteratively optimize an objective function (144) associated with the difference between the first light pattern and the second light pattern.

13. The system of claim 12, wherein as part of iteratively optimizing the objective function (144), the processor (104) further: determines, based on the difference, a change to the initial guess to reduce the difference; generates, based on the change, a second guess of the pose of the object (208); generates, based on the second guess of the pose, a second illumination that, when projected from the projector (136, 204), results in the first light pattern appearing on the object (208); causes the projector (136, 204) to project the second illumination onto the object (208); receives, from the imaging device (112, 212), a second image of the object (208) illuminated by a third light pattern; and determines a difference between the first light pattern and the third light pattern.

14. A method, comprising: receiving an input representing an initial guess of a pose of an object (208) in three- dimensional (3D) space; generating, based on the input, an illumination that, when projected from a projector (136, 204), results in a first light pattern appearing on the object (208);causing the projector (136, 204) to project the illumination onto the object (208); receiving, from an imaging device (112, 212), an image that depicts the object (208) with a second light pattern; determining a difference between the first light pattern and the second light pattern; determining, when the difference satisfies a predetermined condition, that the initial guess substantially corresponds to an actual pose of the object (208); and registering, in response to determining that the initial guess corresponds to the actual pose of the object (208), the imaging device (112, 212) with the object (208).

15. The method of claim 14, further comprising: generating, based on a second guess of the pose, a second illumination that, when projected from the projector (136, 204), results in the first light pattern appearing on the object (208); causing the projector (136, 204) to project the second illumination onto the object (208); receiving, from the imaging device (112, 212), a second image that depicts the object (208) with a third light pattern; and determining a difference between the first light pattern and the third light pattern.

Images