Patient registration using stereoscopic vision

Abstract

Systems and methods for performing patient registration using active stereoscopic vision are provided. One or more first images may be received, and at least one object in each first image may be identified. A pose of each object and a boundary on a mask positioned on a projector may be determined. A portion of the mask within the boundary may obscure a corresponding portion of the projector. One or more second images may be received, and at least one object and patient in each second image may be identified. The first coordinate system and the second coordinate system may each be registered to a third coordinate system. A transformation for associating the first coordinate system with the second coordinate system may be determined based on the registration of the first coordinate system and the second coordinate system.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 536,173, filed September 1, 2023. The entire disclosure of this U.S. Provisional Patent Application is incorporated herein by reference. BACKGROUND

[0003] The present disclosure relates generally to image registration, and more particularly to registration using stereoscopic image data.

[0004] Surgical robots can assist surgeons or other medical providers in performing surgical procedures, or can autonomously complete one or more surgical procedures. Imaging can be used by medical providers for diagnostic and / or therapeutic purposes. Patient anatomy can change over time, particularly after a medical implant is placed in the patient anatomy. SUMMARY

[0005] Example aspects of the present disclosure include: A system according to at least one embodiment of the present disclosure includes a processor; and a memory coupled to the processor and having data stored thereon that, when processed by the processor, enables the processor to: receive one or more first images; identify at least one object in each of the first images; determine a pose of each object; determine a boundary on a mask positioned on a projector, the boundary corresponding to an outline of the at least one object; cause a portion of the mask within the boundary to occlude a corresponding portion of the projector; receive one or more second images; identify a patient in the at least one image and each of the second images; register a first coordinate system to a third coordinate system; register a second coordinate system to the third coordinate system; and determine a transformation for associating the first coordinate system with the second coordinate system based on the registration of the first coordinate system and the registration of the second coordinate system.

[0006] Any of the aspects herein, wherein the memory stores further data for the processor to process that, when processed, enables the processor to: turn on the projector.

[0007] Any of the aspects herein, wherein the one or more first images are received when the projector is turned off and the one or more second images are received when the projector is turned on.

[0008] Any of the aspects herein, wherein the first coordinate system comprises a coordinate system of the patient, the second coordinate system comprises a coordinate system of the object, and the third coordinate system comprises a coordinate system of an imaging device.

[0009] In any of the aspects herein, further comprising: an imaging device configured to obtain the one or more first images and the one or more second images.

[0010] In any of the aspects herein, wherein the imaging device comprises a stereo camera.

[0011] In any of the aspects herein, wherein the one or more first images comprise images without a pattern and the one or more second images comprise images with a pattern and a portion of the pattern is occluded by the portion of the mask within the boundary.

[0012] In any of the aspects herein, wherein the pattern comprises a plurality of dots.

[0013] In any of the aspects herein, wherein the imaging device is supported by at least one of a stand, a user, or a robotic arm.

[0014] In any of the aspects herein, wherein the mask comprises a liquid crystal display (LCD), each pixel on the LCD is controllable, and wherein pixels within the boundary on the LCD are blacked out.

[0015] A system according to at least one embodiment of the disclosure comprises: an imaging device; a projector configured to project a pattern; a mask configured to occlude at least a portion of the projector; a processor; and a memory coupled to the processor and having data stored thereon, which when processed by the processor enables the processor to: receive one or more first images from the imaging device; identify at least one object in each of the first images; determine a pose of each object; determine a boundary on a mask positioned on a projector, the boundary corresponding to the at least one object; cause a portion of the mask within the boundary to occlude a corresponding portion of the projector; receive one or more second images from the imaging device; register a first coordinate frame with a third coordinate frame; register a second coordinate frame with the third coordinate frame; and determine a transformation for associating the first coordinate frame with the second coordinate frame based on the registration of the first coordinate frame and the registration of the second coordinate frame.

[0016] In any of the aspects herein, wherein the memory stores further data for processing by the processor, which when processed enables the processor to: turn on the projector.

[0017] Any of the aspects herein, wherein the one or more first images are received when the projector is off and the one or more second images are received when the projector is on.

[0018] Any of the aspects herein, wherein the first coordinate system comprises a coordinate system of the patient, the second coordinate system comprises a coordinate system of the object, and the third coordinate system comprises a coordinate system of an imaging device.

[0019] Any of the aspects herein, wherein the imaging device comprises a stereo camera.

[0020] Any of the aspects herein, wherein the one or more first images comprise images without a pattern and the one or more second images comprise images with a pattern and a portion of the pattern is occluded by the portion of the mask within the boundary.

[0021] Any of the aspects herein, wherein the pattern comprises a plurality of dots.

[0022] Any of the aspects herein, wherein the imaging device is supported by at least one of a stand, a user, or a robotic arm.

[0023] Any of the aspects herein, wherein the mask comprises a liquid crystal display (LCD), each pixel on the liquid crystal display is controllable, and wherein pixels within the boundary on the LCD are blacked out.

[0024] A system according to at least one embodiment of the disclosure comprises: a stereo camera; a projector configured to project a pattern onto a patient; an LCD positioned on the projector and configured to occlude at least a portion of the projector; a processor; and a memory coupled to the processor and having data stored thereon that, when processed by the processor, enables the processor to: receive one or more first images from the stereo camera; identify a reference marker disposed on the patient in each of the first images; determine a pose of the reference marker; determine a boundary on the LCD corresponding to an outline of the reference marker; cause the LCD to occlude a corresponding portion of the projector at a portion of the boundary; receive one or more second images from the stereo camera; identify the patient and the reference marker in each of the second images; register a patient coordinate system to an imaging device coordinate system; register a reference frame coordinate system to the imaging device coordinate system; and determine a transformation associating the patient coordinate system with the reference frame coordinate system based on the registration of the patient coordinate system and the registration of the reference frame coordinate system.

[0025] Any of the aspects in combination with any one or more of the other aspects.

[0026] Any one or more of the features disclosed herein.

[0027] Any one or more of the features substantially as herein disclosed.

[0028] Any one or more of the features substantially as herein disclosed in combination with any one or more of the other features substantially as herein disclosed.

[0029] Any of these aspects / features / embodiments in combination with any one or more of the other aspects / features / embodiments.

[0030] Use of any one or more of the aspects or features disclosed herein.

[0031] It should be understood that any of the features described herein can be claimed in combination with any other feature(s) as described herein, regardless of whether the features come from the same embodiment described.

[0032] The details of one or more aspects of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the technologies described in this disclosure will be apparent from the description and drawings, and from the claims.

[0033] The phrases “at least one of,” “one or more of,” and “and / or,” as used herein, are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions “at least one of A, B, and C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, or C,” and “A, B, and / or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together. When each of the above expressions is recited with respect to elements such as X, Y, and Z, or classes of elements such as X1-Xn, Y1-Ym, and Z1-Zo, it is intended that the phrase mean a single element selected from X, Y, and Z, a combination of elements selected from the same class (for example, X1and X2), and a combination of elements selected from two or more classes (for example, Y1and Zo).

[0034] The term “a” or “an” entity refers to one or more of that entity. As such, the terms “a” (or “an”), “one or more,” and “at least one” can be used interchangeably herein. It is also to be noted that the terms “comprising,” “including,” and “having” can be used interchangeably.

[0035] The foregoing is a summary of the disclosure and is not intended to be exhaustive or to necessarily present the scope of the disclosure. It is intended to be illustrative only and is not intended to be limiting in any way. The summary serves to present selected concepts of the disclosure in a simplified form as an introduction to the more detailed description provided below. As will be understood, other aspects, embodiments, and configurations of the disclosure can utilize one or more features of the above-recited or below-described aspects.

[0036] Many additional features and advantages of the disclosure will be made apparent from the following description of the embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings are incorporated in and constitute a part of this specification and, together with the description, explain the principles of the disclosure. The drawings, together with the description, explain the principles of the disclosure. The drawings simply show preferred and alternative examples of how the disclosure can be made and used and should not be construed as limiting the disclosure to the examples presented in the drawings. Additional features and advantages will become apparent from the following more detailed description of various aspects, embodiments, and configurations of the disclosure as illustrated by the drawings referenced below.

[0038] Figure 1 is a block diagram of a system in accordance with at least one embodiment of the disclosure; Figure 2 Additional aspects of a system in accordance with at least one embodiment of the disclosure are illustrated; Figure 3 is a schematic diagram of a system in accordance with at least one embodiment of the disclosure; Figure 4A is a patient registered image in accordance with at least one embodiment of the disclosure; Figure 4B is a patient registered image in accordance with at least one embodiment of the disclosure; Figure 4C is a patient registered image in accordance with at least one embodiment of the disclosure; and Figure 5 is a flowchart in accordance with at least one embodiment of the disclosure. DETAILED DESCRIPTION

[0039] It should be understood that various aspects disclosed herein can be combined in different combinations than the combinations expressly presented in the description and accompanying drawings. It should also be understood that, depending on the example or embodiment, certain acts or events of any of the processes described herein can be performed in a different sequence, and / or certain acts or events can be omitted, and / or certain acts or events can be added, corresponding to the application of the technology of the present disclosure to different examples or embodiments. Furthermore, although certain aspects of the present disclosure are described as being performed by a single module or unit for purposes of clarity, it should be understood that the technology of the present disclosure can be performed by a combination of units or modules associated with, for example, computing devices and / or medical devices.

[0040] In one or more examples, the described methods, processes, and techniques can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on a computer-readable medium and executed by a hardware-based processing unit. Alternatively, or in addition, the functions can be implemented using a machine learning model, neural network, artificial neural network, or combination thereof, alone or in combination with instructions. The computer-readable medium can include a non-transitory computer-readable medium that corresponds to a tangible medium like data storage media (e.g., RAM, ROM, EEPROM, flash memory or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer).

[0041] The instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors (e.g., Intel Core i3, i5, i7, or i9 processors; Intel Celeron processors; Intel Xeon processors; Intel Pentium processors; AMD Ryzen processors; AMD Athlon processors; AMD Phenom processors; Apple A10 or 10X Fusion processors; Apple A11, A12, A12X, A12Z, or A13 Bionic processors; or any other general purpose microprocessors), graphics processing units (e.g., Nvidia GeForce RTX 2000 series processors, Nvidia GeForce RTX 3000 series processors, AMD Radeon RX 5000 series processors, AMD Radeon RX 6000 series processors, or any other graphics processing unit), application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor” as used herein can refer to any of the foregoing structure or any other physical structure suitable for implementation of the described techniques. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0042] Before any embodiments of the disclosure are explained in detail, it is to be understood that the application of the disclosure is not limited to the detailed description thereof or the constructions described therein but is applicable to other constructions and arrangements as well. The disclosure is capable of other embodiments and of being practiced or being carried out in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and not of limitation. The use of the terms “including,” “comprising,” or “having” and variations thereof herein is intended to be broad and encompass the presence of one or more items or components. Further, the disclosure can be illustrated using examples. The use of one or more examples, or the use of a phrase such as “for example,” “by way of example,” “e.g.,” “such as,” or similar language, to describe a particular aspect of the disclosure, is not intended to limit the scope of the disclosure to only those examples. It is to be understood that other aspects of the disclosure will become apparent from the following detailed description.

[0043] The terms "proximal" and "distal" are used in this disclosure in their conventional medical meaning, with "proximal" being closer to an operator or user of the system and further from a surgical area of interest in or on a patient, and "distal" being closer to the surgical area of interest in or on the patient and further from the operator or user of the system.

[0044] Patient registration can be performed prior to or during a surgical procedure. One way to perform patient registration using a stereo imaging system is to register both the patient anatomy and the navigation reference frame in the common coordinate system of the imaging system. To register the patient anatomy, the geometric features determined by the stereo imaging system are aligned with the geometric features present in the medical images of the anatomy, and the same can be done for the reference frame. However, optical targets such as checkerboards, aruco markers, QR codes, etc. can be used on the reference frame, improving the accuracy of the reference frame registration. The corners of these targets can be identified in the stereo images at sub-pixel resolution, enabling very accurate and reproducible 3D registration. A challenge arises when trying to register the patient anatomy and the optical targets at the same time. Stereo imaging systems rely on texture to identify corresponding pixels in the left and right stereo image pairs. The patient's anatomy often does not contain enough texture to accurately perform stereo matching. Then, a laser dot pattern is often projected onto the patient to add artificial texture to the images and significantly improve the accuracy of the scan. This is often referred to as "active stereo". The problem is that the laser dot pattern tends to disrupt or obscure the images of the optical targets.

[0045] There are several ways to address this problem. For example, two snapshots can be taken to perform the registration - one with the dot pattern and one without. However, for handheld applications, this is not ideal because the stereo camera can move between the two acquisitions, and the images will no longer be in the same coordinate system. Another approach is to add another pair of cameras that cannot see the dot pattern. One example is to use an infrared (IR) dot projector and add two cameras with infrared filters. The cameras with infrared filters will register the optical targets, and the cameras without the IR filters will be responsible for registering the patient anatomy. This addresses the simultaneity problem, but is quite expensive and requires more complex calibration of the four camera system.

[0046] At least one embodiment according to the present disclosure includes using a mask such as a two-dimensional (2D) liquid crystal display (LCD) array in front of a point projector to occlude the point pattern at the location of the optical target or reference frame. In such an embodiment, the registration process would start with the projector off and determine the location of the optical target. The outline of the target can then be projected onto the LCD panel to determine which pixels to darken or otherwise occlude the point pattern within the bounds. Some margin can be added to account for motion of the camera. Finally, the point projector is turned on with the LCD panel and a pair of images are acquired in which the patient anatomy contains texture from the point pattern while the optical target is in shadow and fiducials are clearly visible. This registration, which advantageously is non-contact, can improve the surgical workflow and navigation accuracy. Combining the optical target on the navigation reference frame with an active stereo imaging system can significantly improve accuracy and / or reduce patient registration time.

[0047] Embodiments of the present disclosure provide technical solutions to one or more of the following problems: (1) registering both the patient and reference markers using optical targets, (2) improving accuracy of registration using handheld imaging devices, and (3) improving non-contact registration.

[0048] Turning first to Figure 1 , a block diagram of a system 100 according to at least one embodiment of the present disclosure is shown. The system 100 can be used to register or correlate one or more coordinate frames with each other and / or to perform one or more other aspects of one or more methods disclosed herein. The system 100 includes a computing device 102, one or more imaging devices 112, a projector 136, a robot 114, a navigation system 118, a database 130, and / or a cloud or other network 134. Systems according to other embodiments of the present disclosure can include more or fewer components than the system 100. For example, the system 100 can not include the robot 114, one or more components of the computing device 102, the database 130, and / or the cloud 134.

[0049] The computing device 102 includes a processor 104, a memory 106, a communication interface 108, and a user interface 110. Computing devices according to other embodiments of the present disclosure can include more or fewer components than the computing device 102.

[0050] The processor 104 of the computing device 102 can be any processor described herein or any similar processor. The processor 104 can be configured to execute instructions stored in the memory 106 that can cause the processor 104 to perform one or more computational steps with or based on data received from the imaging devices 112, the projector 136, the robot 114, the navigation system 118, the database 130, and / or the cloud 134.

[0051] Memory 106 may be or include RAM, DRAM, SDRAM, other solid-state memory, any memory described herein, or any other tangible non-transitory memory used to store computer-readable data and / or instructions. Memory 106 may store information or data that can be used to perform any step of, for example, the method 500 described herein or any other method. Memory 106 may store instructions and / or machine learning models, for example, supporting one or more functions of projector 136 and / or imaging device 112. For example, memory 106 may store contents (e.g., instructions and / or machine learning models) that implement image processing 120, segmentation 122, registration 124, and / or transformation 128 when executed by processor 104.

[0052] Image processing 120 enables processor 104 to process image data (received from, for example, the imaging device of imaging apparatus 112, the imaging device of navigation system 118, or any imaging device) to, for example, identify information about a patient and / or an object (such as reference marker 140 depicted in the image). This information may include, for example, the patient's pose, the pose of reference marker 140, the boundaries of reference marker 140, etc. Information obtained from image processing 120 may enable, for example, determining boundaries on a mask such as mask 138 (described in more detail below) corresponding to reference marker 140. This information may also enable registration of the patient to the common coordinate system of imaging apparatus 112, and / or registration of reference marker 140 to the common coordinate system of imaging apparatus 112. Image processing 120 may use segmentation 122 to identify the patient and / or one or more objects, as described below.

[0053] Segmentation 122 enables processor 104 to segment image data to identify a patient and / or one or more objects (e.g., reference marker 140) within the image data. Segmentation 122 enables processor 104 to identify the boundaries of objects or a patient by using, for example, feature recognition. For instance, segmentation 122 enables processor 104 to identify a patient's head in the image data. In other instances, segmentation 122 enables processor 104 to identify the boundaries of an object (e.g., reference marker 140) by determining the differences or contrast between the colors or grayscale values ​​of image pixels.

[0054] Registration 124 enables processor 104 to associate one coordinate system with another. For example, registration 124 enables processor 104 to associate or map a first coordinate system (e.g., patient coordinate system) with a third coordinate system (e.g., imaging device coordinate system), and to associate or map a second coordinate system (e.g., reference coordinate system) with a third coordinate system (e.g., imaging device coordinate system).

[0055] Transform 128 enables processor 104 to transform one coordinate system to another coordinate system. In other words, transform 128 enables processor 104 to transform a first coordinate system (e.g., a patient coordinate system) to a second coordinate system (e.g., a reference frame coordinate system) based on, for example, a registration of the first coordinate system to the third coordinate system and a registration of the second coordinate system to the third coordinate system.

[0056] In some embodiments, such content, if provided in the form of instructions, can be organized into one or more applications, modules, packages, layers, or engines. Alternatively or additionally, memory 106 can store other types of content or data (e.g., machine learning models, artificial neural networks, deep neural networks, etc.) that can be processed by processor 104 to carry out the various methods and features described herein. Thus, although the various content of memory 106 can be described as instructions, it should be understood that the functionality described herein can be implemented using instructions, algorithms, and / or machine learning models. The data, algorithms, and / or instructions can cause processor 104 to manipulate data stored in memory 106, and / or data received from or via imaging device 112, projector 136, robot 114, database 130, and / or cloud 134.

[0057] Computing device 102 can also include a communication interface 108. Communication interface 108 can be used to receive image data or other information from external sources (such as imaging device 112, projector 136, robot 114, navigation system 118, database 130, cloud 134, and / or any other system or component that is not part of system 100), and / or to transmit instructions, images, or other information to external systems or devices (e.g., another computing device 102, imaging device 112, projector 136, robot 114, navigation system 118, database 130, cloud 134, and / or any other system or component that is not part of system 100). Communication interface 108 can include one or more wired interfaces (e.g., USB ports, Ethernet ports, Firewire ports) and / or one or more wireless transceivers or interfaces (e.g., configured to send and / or receive information via one or more wireless communication protocols such as 802.11a / b / g / n, Bluetooth, NFC, ZigBee, etc.). In some embodiments, communication interface 108 can be usable to enable device 102 to communicate with one or more other processors 104 or computing devices 102, whether to reduce the time needed to complete computationally intensive tasks or for any other reason.

[0058] The computing device 102 can also include one or more user interfaces 110. The user interface 110 can be or include a keyboard, a mouse, a trackball, a monitor, a television, a screen, a touchscreen, and / or any other device for receiving information from a user and / or for providing information to a user. The user interface 110 can be used, for example, to receive user selections or other user input regarding any step of any method described herein. Although as previously described, any required input for any step of any method described herein can be automatically generated by the system 100 (e.g., by the processor 104 or another component of the system 100) or received by the system 100 from a source external to the system 100. In some embodiments, the user interface 110 can be usable to allow a surgeon or other user to modify instructions to be executed by the processor 104 in accordance with one or more embodiments of the present disclosure, and / or to modify or adjust settings of other information displayed on or corresponding to the user interface 110.

[0059] Although the user interface 110 is shown as part of the computing device 102, in some embodiments, the computing device 102 can utilize a user interface 110 housed separately from one or more remaining components of the computing device 102. In some embodiments, the user interface 110 can be positioned proximate to one or more other components of the computing device 102, while in other embodiments, the user interface 110 can be positioned remote from one or more other components of the computing device 102.

[0060] The imaging device 112 can be operable to image anatomical feature(s) (e.g., bones, veins, tissue, etc.) and / or other aspects of a patient’s anatomy to produce image data (e.g., image data depicting or corresponding to bones, veins, tissue, etc.). As used herein, “image data” refers to data generated or captured by the imaging device 112, including data in machine-readable form, graphical / visual form, and any other form. In various examples, the image data can include data corresponding to anatomical features of a patient or a portion thereof or any object, such as the reference marker 140. The image data can be or include pre-operative images, intra-operative images, post-operative images, or images taken independent of any surgical procedure.

[0061] The imaging device 112 can be capable of taking 2D images or 3D images to produce image data. The imaging device 112 can be or include, for example, an ultrasound scanner (which can include, for example, physically separate transducers and receivers, or a single ultrasound transceiver), an O-arm, a C-arm, a G-arm, or any other device that utilizes X-ray based imaging (e.g., a fluoroscope, a CT scanner, or other X-ray machine), a magnetic resonance imaging (MRI) scanner, an optical coherence tomography (OCT) scanner, an endoscope, a microscope, an optical camera, a thermal imaging camera (e.g., an infrared camera), a radar system (which can include, for example, a transmitter, a receiver, a processor, and one or more antennas), a stereoscopic camera, or any other imaging device 112 suitable for obtaining images of anatomical features of a patient. The imaging device 112 can be entirely contained within a single housing, or can include a transmitter / emitter and a receiver / detector located in separate housings or otherwise physically separated. In some embodiments, a first imaging device 112 can be used to obtain first image data (e.g., a first image) at a first time, and a second imaging device 112 can be used to obtain second image data (e.g., a second image) at a second time after the first time. In some embodiments (and as will be described below), the first imaging device can be a navigation imaging device used with the navigation system 118. In other words, the first imaging device can be a navigation imaging device used with the navigation system 118. The first imaging device can be, for example, any of the example imaging devices described above (ultrasound scanner, O-arm, C-arm, G-arm, or any other device that utilizes X-ray based imaging, magnetic resonance imaging scanner, OCT scanner, endoscope, microscope, optical camera, thermal imaging camera, radar system, stereoscopic camera, etc.). The second imaging device can be a registration imaging device. In some embodiments, the second imaging device is a stereoscopic camera, as will be discussed in detail below with respect to FIG. 3. Figure 2

[0062] In some embodiments, the imaging device 112 can include more than one imaging device 112. For example, a first imaging device can provide first image data and / or a first image, and a second imaging device can provide second image data and / or a second image. In yet other embodiments, the same imaging device can be used to provide both first image data and second image data and / or any other image data described herein. The imaging device 112 can be operable to generate a stream of image data. For example, the imaging device 112 can be configured to operate with an open shutter or with a shutter that constantly alternates between open and closed in order to capture successive images. For purposes of the present disclosure, unless otherwise noted, image data can be considered to be successive and / or provided as a stream of image data if the image data represents two or more frames per second.

[0063] ​The projector 136 can include a vertical-cavity surface-emitting laser (VCSEL) or a simple laser with a diffraction grating to project a pattern. In other embodiments, the projector 136 can include any projector 136 configured to project any pattern, such as a digital light processing (DLP) projector, a light-emitting diode (LED) projector, an LCD projector, or a liquid crystal on silicon (LCOS) projector. The projector 136 can be manually supported by, for example, a user, or can be statically supported by, for example, a stand. In other embodiments, the projector 136 can be supported and positioned by, for example, the robot 114 (and more specifically, by the robotic arm 116 of the robot 114).

[0064] The pattern can be projected onto, for example, a patient, an object, or a combination thereof. The projected pattern can assist in identifying at least a portion of the patient that is projected upon (wherein the patient can be identified in the image data via, for example, image processing 120 using the processor 104 described above). For example, the pattern can be used to identify smooth portions on the patient’s face that can otherwise be difficult to identify. The pattern can be any pattern, such as a plurality of dots, a plurality of lines, a noise pattern, a plurality of any shape, or a plurality of any shape combination.

[0065] The mask 138 can be configured to occlude at least a portion of the projected pattern from the projector 136. As will be described in more detail in Figures 3 to 5 A reference mark, such as the reference mark 140, can be positioned on a portion of the patient. When the projected pattern is projected onto the portion of the patient having the reference mark 140, the projected pattern can distort the reference mark 140. Thus, it is desirable to occlude or mask the portion of the projected pattern that would cover and distort the reference mark 140.

[0066] The mask 138 can be positioned within a trajectory of the pattern projected by the projector 136 and can be positioned, for example, in front of the projector 136. In other instances, the projector 136 and the mask 138 are integrated. For example, the projector 136 and the mask 138 can include an LCD projector with an LCD panel included thereon. In other embodiments, the projector 136 and the mask 138 (e.g., the LCD panel) are separate components. In embodiments where the mask 138 is an LCD panel, the LCD panel includes pixels that can be individually controlled by, for example, the processor 104 of the computing device 102, the processor of the navigation system 118, or any other processor. Thus, the pixels within the determined boundary can be controlled by, for example, turning each pixel within the boundary black to occlude the pattern within the boundary. In yet other instances, the mask 138 can include a hand-made physical mask (e.g., cardboard, tape, cardstock, plywood, etc.) that can be shaped into the desired boundary shape and positioned over the projector to physically occlude a portion of the projected pattern.

[0067] The robot 114 can be any surgical robot or surgical robot system. The robot 114 can be or include, for example, the Mazor X® Stealth Edition robotic guidance system. The robot 114 can be or include, for example, the Mazor X® Stealth Edition robotic guidance system. TM The robot 114 can be configured to position the imaging device 112 at one or more precise locations and orientations, and / or to return the imaging device 112 to the same location(s) and orientation(s) at a later point in time. The robot 114 can additionally or alternatively be configured to manipulate a surgical tool (whether or not based on guidance from the navigation system 118) to complete or assist with a surgical task. In some embodiments, the robot 114 can be configured to hold and / or manipulate anatomical elements during or in conjunction with a surgical procedure. The robot 114 can include one or more robotic arms 116. In some embodiments, the robotic arms 116 can include a first robotic arm and a second robotic arm, although the robot 114 can include more than two robotic arms. In some embodiments, one or more of the robotic arms 116 can be used to hold and / or manipulate the imaging device 112. In embodiments in which the imaging device 112 includes two or more physically separate components (e.g., a transmitter and a receiver), one robotic arm 116 can hold one such component and another robotic arm 116 can hold the other such component. Each robotic arm 116 can be positioned independently of the other robotic arm. The robotic arms 116 can be controlled in a single shared coordinate space or in separate coordinate spaces.

[0068] The robot 114, along with the robotic arms 116, can have, for example, one, two, three, four, five, six, seven, or more degrees of freedom. Further, the robotic arms 116 can be positioned or positionable at any pose, plan, and / or focal point. A pose includes a position and an orientation. Thus, the imaging device 112, surgical tool, or other object held by the robot 114 (or, more specifically, held by the robotic arms 116) can be precisely positioned at one or more desired and specific locations and orientations.

[0069] The robotic arm(s) 116 can include one or more sensors that enable the processor 104 (or a processor of the robot 114) to determine the precise pose of the robotic arm (and any object or element held by or fixed to the robotic arm) in space.

[0070] In some embodiments, one or more reference markers 140 (e.g., navigation markers) may be placed on the robot 114 (including, for example, on the robotic arm 116), the imaging device 112, the patient, or any other object in the surgical space. The reference markers 140 may be tracked by the navigation system 118, and the results of the tracking may be used by the operator of the robot 114 and / or by the operator of the system 100 or any component thereof. In some embodiments, the navigation system 118 may be used to track other components of the system (e.g., the imaging device 112), and the system may be operated without the use of the robot 114 (e.g., a surgeon manually manipulating the imaging device 112 and / or one or more surgical instruments, for example, based on information and / or instructions generated by the navigation system 118). The reference markers 140 may include one or more active markers, one or more passive markers, or a combination of active and passive markers. The reference markers 140 may be, for example, light-emitting diodes (LEDs), infrared LEDs, reflective markers, etc.

[0071] During operation, navigation system 118 can provide navigation for the surgeon and / or surgical robot. Navigation system 118 can be any navigation system currently known or developed in the future, including, for example, Medtronic StealthStation. TM The S8 surgical navigation system or any subsequent system thereof. Navigation system 118 may include one or more cameras or (multiple) other sensors for tracking one or more reference markers, navigation trackers, or other objects within the operating room or other rooms where part or all of system 100 is located. The one or more cameras may be optical cameras, infrared cameras, or other cameras. In some embodiments, navigation system 118 may include one or more electromagnetic sensors. In various embodiments, navigation system 118 may be used to track the position and orientation (e.g., pose) of imaging device 112, the patient (via reference marker 140), robot 114 and / or robotic arm 116, and / or one or more surgical instruments (or more specifically, to track the pose of navigation trackers directly or indirectly attached in a fixed relationship to one or more of the foregoing). Navigation system 118 may include a display for displaying one or more images from an external source (e.g., computing device 102, imaging device 112, projector 136, or other sources), or for displaying images and / or video streams from one or more cameras or other sensors of navigation system 118. In some embodiments, system 100 may operate without using navigation system 118. The navigation system 118 can be configured to provide guidance to the surgeon or other users of the system 100 or its components, to the robot 114 or any other element of the system 100, regarding, for example, the pose of one or more anatomical elements, whether the tool is in the appropriate trajectory, and / or how to move the tool into the appropriate trajectory to perform surgical tasks according to the preoperative or other surgical plan.

[0072] The database 130 can store information associating one coordinate system with another coordinate system (e.g., one or more robot coordinate systems with a patient coordinate system and / or a navigation coordinate system). The database 130 can additionally or alternatively store, for example: one or more surgical plans (including, for example, pose information about a target, and / or image information about patient anatomy at and / or near a surgical site, for use by the robot 114, the navigation system 118, and / or a user of the computing device 102 or system 100); one or more images that can be used in connection with a procedure to be performed by or with the assistance of one or more other components of the system 100; and / or any other useful information. The database 130 can be configured to provide any such information to the computing device 102, or to any other device of or external to the system 100, directly or via the cloud 134. In some embodiments, the database 130 can be or include part of a hospital image storage system, such as a picture archiving and communication system (PACS), a health information system (HIS), and / or another system for collecting, storing, managing, and / or transmitting electronic medical records including image data.

[0073] The cloud 134 can be or represent the Internet or any other wide area network. The computing device 102 can connect to the cloud 134 using a wired connection, a wireless connection, or both, via the communication interface 108. In some embodiments, the computing device 102 can communicate with the database 130 and / or external devices (e.g., computing devices) via the cloud 134.

[0074] The system 100 or a similar system can be used, for example, to perform one or more aspects of any of the methods 500 described herein. The system 100 or a similar system can also be used for other purposes.

[0075] Figure 2 Additional aspects of the system 100 according to embodiments of the present disclosure are demonstrated. Figure 2 The imaging device 112 is depicted as operating as a stereo camera, with a first image sensor 204 and a second image sensor 208 that can be used to capture stereo images of the patient 212.

[0076] The first image sensor 204 and the second image sensor 208 are each capable of generating information or image data that can be processed to produce a pair of 2D images of the patient 212. The first image sensor 204 and the second image sensor 208 can be separated by a predetermined or otherwise known distance such that a field of view (FOV) of the first image sensor 204 is different than a FOV of the second image sensor 208. For example, the first image sensor 204 can have a first FOV 216, while the second image sensor 208 can have a second FOV 220 that is different than the first FOV 216. Thus, when the imaging device 112 images the patient 212, the data generated by the first image sensor 204 is different than the data generated by the second image sensor 208 such that each image of the pair of 2D images depicts features of the patient 212 from a different angle, direction, and / or orientation. In other words, a first 2D image of the pair of 2D images depicts features of the patient 212 in a first orientation or pose, while a second 2D image of the pair of 2D images depicts features of the patient 212 from a different angle, direction, and / or orientation or pose.

[0077] As Figure 2 As depicted in FIG. 2, the first image sensor 204 is positioned to the left of the centerline 214 of the patient 212, while the second image sensor 208 is positioned to the right of the centerline 214 of the patient 212. When processed by the image processing 120, the resulting data from the first image sensor 204 can produce an image that depicts a left side view of the patient 212 (also referred to herein as a left eye view of the patient 212). Similarly, when processed by the image processing 120, the resulting data from the second image sensor 208 can produce an image that depicts a right side view of the patient 212 (also referred to herein as a right eye view of the patient 212). In some cases, one or both of these 2D images (e.g., the right side view and / or the left side view) can be presented to a display to enable a user of the system 100 to view the 2D image(s). It should be understood that while embodiments of the present disclosure discuss a left side view and a right side view of the patient 212, additional or alternative imaging of the patient can also be performed, and any portion of the patient 212 can be imaged by the stereoscopic camera to produce a pair of 2D images that depict the portion of the patient 212.

[0078] The pose of the first image sensor 204 relative to the second image sensor 208 (and vice versa) can be known or determined by the system 100. In some cases, the pose information can be stored in the database 130. Additionally or alternatively, the pose information can be determined by the navigation system 118 based on a pose of a tracked fiducial that is in a known pose relative to the first image sensor 204 and / or the second image sensor 208.

[0079] Figure 3Additional aspects of the system 100 according to embodiments of the present disclosure are demonstrated. Figure 3 The imaging device 112 is depicted having a first image sensor 204 and a second image sensor 208 operating as a stereo camera, as described above Figure 2 The projector 136 is configured to project a pattern 302, and a mask 138 is positioned within a trajectory of the pattern 302 and configured to mask or occlude at least a portion of the pattern 302. As shown, a portion of the mask 304 defined by a border 304 can occlude a portion of the pattern 302. In instances where the mask 304 is an LCD screen, pixels within the border 304 can be controlled to turn black or dark in order to occlude the pattern 302. The border 304 is determined by a shape of the reference marker 140 (as will be described below). By occluding the portion of the pattern 302 in which the reference marker 140 is positioned, the patient 212 can be identified in the image data (therefore, improving the accuracy of identifying and obtaining the pose of the patient 212) and the reference marker 140 can be identified in the image data without the pattern 302 obscuring and distorting the reference marker 140, as will also be described below.

[0080] Figures 4A to 4C Example reference markers that can be identical or similar to the reference marker 140 positioned on a patient (such as the patient 212) are demonstrated with the projector 136 off, the projector 136 on, the projector 136 on and the mask 138 enabled within the border 304.

[0081] As shown and previously described in Figure 4A The reference marker (such as the reference marker 140) positioned on the patient 212 for patient tracking by, for example, the navigation system 118. In the demonstrated example, the reference marker 140 includes a combination of an electromagnetic (EM) tracker and an optical tracker. As shown, the optical tracker is in the form of a cube having a checkerboard pattern that is easily identifiable within image data captured or obtained by the imaging device 112. It should be understood that in other embodiments, the reference marker 140 can include any type of marker or tracker, any pattern, and any combination of markers or trackers or patterns. For example, the optical tracker can be an acruco marker, a QR code, or the like.

[0082] In addition to identifying and tracking the reference markers 140 for registration, the patient 212 is identified in the image data for registration. Because portions of the patient 212 (e.g., the forehead) can be difficult to identify in the image data, a pattern, such as the pattern 302, can be projected onto the patient 212 by the projector 136. Such a pattern 302 provides texture to the patient 212, improving the accuracy of identifying the patient 212 in the image data. However, as shown, the pattern 302, when projected onto the patient 212 and the reference markers 140, can obscure the reference markers 140, making it difficult to identify the reference markers 140 in the image data. Thus, it is desirable to project the pattern 302 onto the patient 212, but not onto the reference markers 140. As described previously, to achieve projecting the pattern 302 onto the patient 212, but not onto the reference markers 140, the mask 138 is configured to occlude a portion of the pattern 302, such that the pattern 302 is not projected onto the reference markers 140. Figure 4B

[0083] To determine the boundary 304 to occlude using the mask 138, the reference markers 140 are identified in image data obtained from, for example, the imaging device 112 (e.g., a stereo camera) when the projector 136 is off. The outline or edge of the reference markers 140 can be identified in the image data, and the corresponding boundary 304 can be determined based on the outline or edge of the reference markers 140. In other embodiments, the dimension(s) of the reference markers 140 are known and provided to, for example, the system 100 or the navigation system 118. When the imaging device 112 is a stereo camera, the pose of the reference markers 140 can also be determined from the image data using the image processing 120 or, for example, triangulation in a pair of images obtained from the imaging device 112. The outline of the reference markers 140 can be projected onto the mask 138 based on the determined pose. The boundary 304 can then be determined from the projected outline. Alternatively or additionally, the identified outline or known dimensions of the reference markers 140 can be used, along with the determined pose, to determine the boundary 304. In embodiments where the mask 138 includes an LCD screen, the pixels within the boundary 304 can be blacked out or otherwise controlled to occlude the pattern 302 within the boundary 304. When the mask 138 with the boundary 304 is enabled and the projector 136 is turned on, the pattern 302 is projected onto the patient 212, but not onto the reference markers 140, as shown. Figure 4C

[0084] Figure 5 A method 500 that can be used for non-contact registration of one or more coordinate systems, for example, using a projector, such as the projector 136, and a mask, such as the mask 138, is depicted.

[0085] ​​The method 500 (and / or one or more steps thereof) can be carried out or otherwise performed, for example, by at least one processor. The at least one processor can be the same as or similar to the processor(s) 104 of the computing device 102 described above. The at least one processor can be part of a robot, such as the robot 114, or part of a navigation system, such as the navigation system 118. A processor other than any processor described herein can also be used to perform the method 500. The at least one processor can perform the method 500 by executing elements stored in a memory, such as the memory 106. The elements stored in the memory and executed by the processor can cause the processor to perform one or more steps of the functionality shown in the method 500. One or more portions of the method 500 can be performed by the processor executing any content of the memory, such as the image processing 120, the segmentation 122, the registration 124, and / or the transformation 128.

[0086] The method 500 includes receiving one or more first images (step 504). The first image(s) can be received from an imaging device, such as the imaging device 112. In embodiments where the imaging device is a stereo camera, the one or more first images can include a pair of first images. In other embodiments, the one or more first images can be received via a user interface, such as the user interface 110, and / or a communication interface, such as the communication interface 108, of a computing device, such as the computing device 102, and can be stored in a memory, such as the memory 106, of the computing device. The image(s) can also be received from an external database or image repository (e.g., a hospital image storage system, such as a Picture Archiving and Communication System (PACS), a Health Information System (HIS), and / or another system for collecting, storing, managing, and / or transmitting electronic medical records including image data) and / or via the Internet or another network. The image(s) can also be generated by and / or uploaded to any other component of the system, such as the system 100. In some embodiments, the images can be received indirectly via a node of a network to which any other component of the system is connected.

[0087] The images can be 2D images or 3D images, or a set of 2D and / or 3D images. For example, when the imaging device is a stereo camera, the image(s) are a pair of 2D images taken from a first image sensor, such as the first image sensor 204, at a first pose and from a second image sensor, such as the second image sensor 208, at a second pose different from the first pose. The images can depict a patient, such as the patient 212, and a reference marker, such as the reference marker 140, positioned on the patient. The images can be obtained with the projector, such as the projector 136, turned off.

[0088] In some embodiments, the images can be captured preoperatively (e.g., before surgery) and can be stored in a system (e.g., system 100) and / or one or more components thereof (e.g., database 130). The stored images can then be received (e.g., by processor 104) preoperatively (e.g., before surgery) and / or intraoperatively (e.g., during surgery), as described above.

[0089] Method 500 further includes identifying at least one object in each first image (step 508). The object can be a reference marker. As previously described, the reference marker can include a combination of an EM tracker and an optical tracker, although it should be appreciated that the reference marker can include any type of reference marker or combination thereof. The optical tracker can include a pattern, such as a checkerboard pattern, that is easily identifiable in the first image(s).

[0090] Each image can be processed by a processor, such as processor 104, using image processing, such as image processing 120, to identify the patient and / or the object(s) (e.g., reference marker) in each first image. In some embodiments, feature recognition can be used to identify features of the reference marker. For example, a checkerboard pattern (or any pattern depicted on the reference marker) is identified in the first image. In other embodiments, the image processing can use segmentation, such as segmentation 122, to identify the object.

[0091] Method 500 further includes determining a pose of the object (step 510). The pose of the object can also be determined by the processor using image processing. More specifically, in embodiments where the imaging device is a stereo camera, a pair of first images obtained from the stereo camera can be used to triangulate the pose of the reference marker.

[0092] The method 500 further includes determining a boundary on the mask based on the object (step 512). As previously described, to determine a boundary (e.g., boundary 304) for obscuring a portion of a pattern (e.g., pattern 302) projected by the projector using a mask (e.g., mask 138), the reference marker is identified in the first image data when the projector is off in step 508 described above. A contour or edge of the reference marker can be identified in the image data (also in, e.g., step 508), and a corresponding boundary can be determined based on the contour or edge of the reference marker. In other embodiments, the dimension(s) of the reference marker are known and provided to, e.g., the system or navigation system. The pose of the reference marker can also be determined from the image data as described in step 510. The contour of the reference marker can be projected onto the mask based on the determined pose. The boundary can then be determined from the projected contour and in embodiments. Alternatively or additionally, the boundary can be determined from the pose of the reference marker and the known dimensions of the reference marker. In embodiments where the mask comprises an LCD screen or LCD projector, the pixels within the boundary can be darkened or otherwise controlled to obscure the pattern within the boundary. Thus, when the mask with the boundary is enabled and the projector is turned on, the pattern is projected onto the patient but not onto the reference marker.

[0093] The method 500 further includes causing a portion of the mask within the boundary to obscure a corresponding portion of the projector (step 516). The portion of the mask within the boundary can be configured to obscure the corresponding portion of the pattern projected by the projector. In embodiments where the mask is an LCD screen or panel, the pixels within the boundary can be darkened or otherwise controlled to obscure the pattern within the boundary. In other embodiments, the mask can comprise a physical mask that can be shaped in the shape of the boundary and positioned to physically obscure a portion of the pattern within the boundary.

[0094] The method 500 further includes turning on the projector (step 518). The projector can be turned on automatically by, e.g., the system or navigation system. In other embodiments, the projector can be turned on based on user input.

[0095] The method 500 further includes receiving one or more second images (step 520). Step 520 is the same as or similar to step 508 described above with respect to receiving image(s). The one or more second images are obtained when the projector is turned on and projects the pattern, and the mask is enabled to obscure a portion of the pattern within the boundary. Thus, the image data depicts the patient (e.g., the reference marker) and the object with the pattern projected onto the patient and no pattern projected onto the object.

[0096] The method 500 also includes identifying the patient and the object in each second image (step 522). Step 522 can be the same as or similar to step 510, except that the patient in each second image is also identified (along with the object (e.g., the reference marker)).

[0097] The method 500 also includes registering the first coordinate system (step 524). The first coordinate system can be, for example, a patient coordinate system of the patient identified in the second image(s) in step 522. The processor can use a registration (such as the registration 124) to register the patient coordinate system to associate, map to, or transform to a third coordinate system, which can be, for example, an imaging device coordinate system. The registration can enable conversion of coordinates defining a particular location with respect to one coordinate space (e.g., the patient coordinate space) to coordinates defining the particular location with respect to another coordinate space (e.g., the imaging device coordinate space).

[0098] The method 500 also includes registering the second coordinate system (step 528). The second coordinate system can be, for example, a reference marker coordinate system of the reference marker identified in the second image(s) in step 522. The reference marker coordinate system can be registered by the processor using the registration 124 to associate, map to, or transform to the third coordinate system (e.g., the imaging device coordinate system). The registration can enable conversion of coordinates defining a particular location with respect to one coordinate space (e.g., the reference marker coordinate space) to coordinates defining the particular location with respect to another coordinate space (e.g., the imaging device coordinate space).

[0099] The method 500 also includes determining a transformation for associating the first coordinate system with the second coordinate system (step 532). The first coordinate system (e.g., the patient coordinate system) and the second coordinate system (e.g., the reference marker coordinate system) can be associated based on their common registration with the third coordinate system (e.g., the imaging device coordinate system). The processor can use a transformation (such as the transformation 128) to associate the first coordinate system with the second coordinate system, which enables the processor to transform one coordinate system to another coordinate system. In other words, the transformation enables the processor to transform the first coordinate system (e.g., the patient coordinate system) to the second coordinate system (e.g., the reference system coordinate system) based on the registration of the first coordinate system and the third coordinate system and the registration of the second coordinate system and the third coordinate system. Thus, the first coordinate system or the patient coordinate system can be associated with the second coordinate system or the reference coordinate system.

[0100] The present disclosure encompasses embodiments of the method 500 that include more or fewer steps than the steps described above and / or one or more steps that are different than the steps described above.

[0101] The embodiments described herein beneficially provide a non-contact registration process of a patient and objects, such as reference markers, that can be performed using an imaging device, such as a stereo camera. Moreover, the registration process is performed in a short amount of time (about 30 ms) such that the stereo camera can be handheld without reducing the accuracy of the stereo camera (and subsequent registration). Thus, the registration process is simplified and easy to perform relative to conventional registration processes.

[0102] As described above, the present disclosure encompasses methods having fewer than Figure 5 all of the steps identified in the foregoing (and corresponding descriptions of method 500), as well as methods including additional steps beyond those identified in the foregoing (and corresponding descriptions of method 500). The present disclosure also encompasses methods including one or more steps from one method described herein and one or more steps from another method described herein. Any dependency described herein can be or include a registration or any other dependency. Figure 5

[0103] The foregoing is not intended to limit the present disclosure to the form or forms in which it happens to be described herein. For example, in the foregoing Detailed

[0104] Furthermore, although the foregoing has been described in some detail for purposes of clarity and the known scope thereof, it should be apparent that various

[0105] Example recitations of claims: ​Statement 1 : A system comprising: a processor (104); and a memory (106) coupled to the processor and having data stored thereon that, when processed by the processor, enables the processor to: receive one or more first images; identify at least one object in each of the first images; determine a pose of each object; determine a boundary on a mask positioned on a projector, the boundary corresponding to an outline of the at least one object; occlude a portion of the mask within the boundary from a corresponding portion of the projector; receive one or more second images; identify a patient in the at least one image and each of the second images; register a first coordinate system to a third coordinate system; register a second coordinate system to the third coordinate system; and determine a transformation for associating the first coordinate system with the second coordinate system based on the registration of the first coordinate system and the registration of the second coordinate system.

[0106] Statement 2: The system of example 1, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: turn on the projector.

[0107] Statement 3: The system of any one of statements 1-2, wherein the one or more first images are received when the projector is off and the one or more second images are received when the projector is on.

[0108] Statement 4: The system of any one of statements 1-3, wherein the first coordinate system comprises a coordinate system of the patient, the second coordinate system comprises a coordinate system of the object, and the third coordinate system comprises a coordinate system of an imaging device.

[0109] Statement 5: The system of any one of statements 1-4, further comprising: an imaging device (112) configured to obtain the one or more first images and the one or more second images.

[0110] Statement 6: The system of any one of statements 1-5, wherein the imaging device comprises a stereo camera.

[0111] Statement 7: The system of any one of statements 1-6, wherein the one or more first images comprise images without a pattern and the one or more second images comprise images with a pattern and a portion of the pattern is occluded by the portion of the mask within the boundary.

[0112] Statement 8: The system of any one of statements 1-7, wherein the pattern comprises a plurality of dots.

[0113] Statement 9: The system of any of statements 1-8, wherein the imaging device is supported by at least one of a stand, a user, or a robotic arm.

[0114] Statement 10: The system of any of statements 1-9, wherein the mask comprises a liquid crystal display (LCD), each pixel on the liquid crystal display is controllable, and wherein the pixels within the boundary on the LCD are blacked out.

[0115] Statement 11 : A system comprising: an imaging device (112); a projector (136) configured to project a pattern (302); a mask (138) configured to occlude at least a portion of the projector; a processor (104); and a memory (106) coupled to the processor and having data stored thereon, which when processed by the processor, enable the processor to: receive one or more first images from the imaging device; identify at least one object in each of the first images; determine a pose of each object; determine a boundary on a mask positioned over a projector, the boundary corresponding to the at least one object; cause a portion of the mask within the boundary to occlude a corresponding portion of the projector; receive one or more second images from the imaging device; register a first coordinate system with a third coordinate system; register a second coordinate system with the third coordinate system; and determine a transformation for associating the first coordinate system with the second coordinate system based on the registration of the first coordinate system and the registration of the second coordinate system.

[0116] Statement 12: The system of example 11, wherein the memory stores further data for the processor to process, which when processed, cause the processor to: turn on the projector.

[0117] Statement 13: The system of any of statements 11-12, wherein the one or more first images are received when the projector is off and the one or more second images are received when the projector is on.

[0118] Statement 14: The system of any of statements 11-13, wherein the first coordinate system comprises a coordinate system of the patient, the second coordinate system comprises a coordinate system of the object, and the third coordinate system comprises a coordinate system of an imaging device.

[0119] Statement 15: The system of any of statements 11-14, wherein the imaging device comprises a stereo camera.

[0120] Statement 16: The system of any of statements 11 to 15, wherein the one or more first images comprise images without a pattern, and the one or more second images comprise images with a pattern and a portion of the pattern is occluded by the portion of the mask within the boundary.

[0121] Statement 17: The system of any of statements 11 to 16, wherein the pattern comprises a plurality of dots.

[0122] Statement 18: The system of any of statements 11 to 17, wherein the imaging device is supported by at least one of a stand, a user, or a robotic arm.

[0123] Statement 19: The system of any of statements 11 to 18, wherein the mask comprises a liquid crystal display (LCD), each pixel on the liquid crystal display is controllable, and wherein pixels within the boundary on the LCD are blacked out.

[0124] Statement 20: A system comprising: a stereo camera (112); a projector (136) configured to project a pattern (302) onto a patient; an LCD (138) positioned on the projector and configured to occlude at least a portion of the projector; a processor (104); and a memory (106) coupled to the processor and having data stored thereon, which when processed by the processor, causes the processor to: receive one or more first images from the stereo camera; identify a reference marker disposed on the patient in each of the first images; determine a pose of the reference marker; determine a boundary on the LCD corresponding to an outline of the reference marker; cause the LCD to occlude a corresponding portion of the projector at a portion within the boundary; receive one or more second images from the stereo camera; identify the patient and the reference marker in each of the second images; register a patient coordinate frame to an imaging device coordinate frame; register a reference frame coordinate frame to the imaging device coordinate frame; and determine a transformation associating the patient coordinate frame with the reference frame coordinate frame based on the registration of the patient coordinate frame and the registration of the reference frame coordinate frame.

Claims

1. A system comprising: Processor (104); as well as Memory (106), coupled to the processor and storing data thereon, which, when processed by the processor, enables the processor to: Receive one or more first images; Identify at least one object in each of the first images; Determine the pose of each object; Determine the boundaries on a mask positioned on a projector, the boundaries corresponding to the outline of the at least one object; The mask is used to partially obscure a corresponding portion of the projector within the boundary. Receive one or more second images; Identify the patient in the at least one image and in each of the second images; Register the first coordinate system to the third coordinate system; Register the second coordinate system to the third coordinate system; as well as The transformation used to associate the first coordinate system with the second coordinate system is determined based on the registration of the first coordinate system and the registration of the second coordinate system.

2. The system as claimed in claim 1, wherein, The memory stores additional data for the processor to process, and when processed, this data enables the processor to: Turn on the projector.

3. The system as described in claim 2, wherein, The one or more first images are received when the projector is off, and the one or more second images are received when the projector is on.

4. The system as described in any one of claims 1 to 3, wherein, The first coordinate system includes the patient's coordinate system, the second coordinate system includes the object's coordinate system, and the third coordinate system includes the imaging device's coordinate system.

5. The system as described in any one of claims 1 to 4, further comprising: An imaging device (112) is configured to acquire the one or more first images and the one or more second images.

6. The system of claim 5, wherein, The imaging device includes a stereo camera.

7. The system as described in any one of claims 1 to 6, wherein, The one or more first images include images without patterns, and the one or more second images include images with patterns, and a portion of the pattern is obscured by the mask within the boundary.

8. The system of claim 7, wherein, The pattern comprises multiple dots.

9. The system of claim 5, wherein, The imaging device is supported by at least one of a support frame, a user, or a robotic arm.

10. The system as described in any one of claims 1 to 9, wherein, The mask includes a liquid crystal display (LCD), each pixel on which can be controlled, and wherein pixels within the boundaries on the LCD are blacked out.

11. A system comprising: Imaging equipment (112); Projector (136), the projector being configured to project a pattern (302); A mask (138) configured to block at least a portion of the projector; Processor (104); and Memory (106), coupled to the processor and storing data thereon, which, when processed by the processor, enables the processor to: Receive one or more first images from the imaging device; Identify at least one object in each of the first images; Determine the pose of each object; Determine the boundary on the mask positioned on the projector, the boundary corresponding to the at least one object; The mask is used to partially obscure a corresponding portion of the projector within the boundary. Receive one or more second images from the imaging device; Register the first coordinate system with the third coordinate system; Register the second coordinate system with the third coordinate system; as well as The transformation used to associate the first coordinate system with the second coordinate system is determined based on the registration of the first coordinate system and the registration of the second coordinate system.

12. The system of claim 11, wherein, The memory stores additional data for the processor to process, and when processed, this data enables the processor to: Turn on the projector.

13. The system of claim 12, wherein, The one or more first images are received when the projector is off, and the one or more second images are received when the projector is on.

14. The system as described in any one of claims 11 to 13, wherein, The first coordinate system includes the patient's coordinate system, the second coordinate system includes the object's coordinate system, and the third coordinate system includes the imaging device's coordinate system.

15. A system comprising: Stereo camera (112); A projector (136) configured to project a pattern (302) onto the patient; LCD (138), the LCD being positioned on the projector and configured to obscure at least a portion of the projector; Processor (104); and Memory (106), coupled to the processor and storing data thereon, which, when processed by the processor, enables the processor to: Receive one or more first images from the stereo camera; Identify reference markers set on the patient in each of the first images in the first image; Determine the pose of the reference marker; Determine the boundary on the LCD corresponding to the outline of the reference mark; The LCD within the boundary partially obscures a corresponding portion of the projector; Receive one or more second images from the stereo camera; Identify the patient and the reference marker in each of the second images; Register the patient coordinate system to the imaging device coordinate system; Register the reference coordinate system to the imaging device coordinate system; as well as The transformation that associates the patient coordinate system with the reference coordinate system is determined based on the registration of the patient coordinate system and the registration of the reference coordinate system.

Images