Registration and navigation in cranial procedures

Abstract

The disclosed systems, methods, and techniques perform imaging registration and cranial surgical procedure navigation. In one aspect, the systems, methods, and techniques create a 3D image source using a modal scan of a patient's skull obtained preoperatively. The systems, methods, and techniques then obtain surface imaging data of a portion of the patient's face or skull when the patient lies atop the operating table in a supine position, using a machine vision system embedded in or on a movable elliptical mask attached to the robotic arm. The surface imaging data is used to create a 3D image target. The 3D image target is then registered to the 3D image source to complete the imaging registration. Once the imaging registration is completed, a surgeon may perform an image-guided surgery on the skull of the patient.

Description

[0001] Background Information

[0002] In image-guided surgery for glioma removal, a neurosurgeon typically plans the resection on images acquired before the surgery and uses the acquired images for guidance during subsequent interventions. However, after the surgical procedure has started, the pre-planned images may become unreliable due to, for example, brain shift phenomena that may be caused by anatomical changes and inaccuracies in the neuro-navigation system.

[0003] To obtain an updated view of the resection cavity, a solution can be to collect intraoperative data, which can be acquired at different stages of the procedure to provide a better understanding of the resection. Such a solution may require re-registration of the intraoperative acquired images. Re-registration of intraoperative images may include manually placing fiducials on points of interest of anatomical parts. Some algorithms can be used for fiducial initialization, surface merging initialization, image-to-image registration, various calculations, etc. Summary of the Invention

[0004] The disclosed systems, methods, and techniques perform imaging registration and cranial surgical procedure navigation. These systems, methods, and techniques use preoperative acquired modal scans of a patient's skull to create a 3D image source. The modal scans can be computed tomography (CT) scans, magnetic resonance imaging (MRI) scans, positron emission tomography (PET) scans, single photon emission computed tomography (SPECT) scans, or a combination thereof. Then, using a machine vision system embedded in or located on a movable elliptical mask attached to a robotic arm, these systems, methods, and techniques acquire surface imaging data of a portion of the patient's face or skull while the patient lies supine on top of the operating table. To capture the surface imaging data, the machine vision system utilizes at least one imaging sensor, at least one infrared sensor, or a combination thereof. The surface imaging data is used to create a 3D image target. Then the 3D image target is registered to the 3D image source to complete the imaging registration. Once the imaging registration is completed, the surgeon can perform image-guided surgery on the patient's skull.

[0005] In one aspect, a method for performing imaging registration and cranial surgical procedure navigation includes obtaining a modal scan of a patient's skull. The method further includes creating a first 3D model of the patient's skull using the modal scan. The method further includes obtaining surface imaging data of a portion of the patient's face or skull using a movable elliptical mask attached to a robotic arm while the patient lies supine on top of an operating table. The robotic arm moves the elliptical mask in a circular motion around a portion of the patient's face or skull and gradually moves approximately 180 degrees from: i) a first position to a second position, where the first position includes the elliptical mask generally facing the patient's nose and the second position includes the elliptical mask generally facing the patient's ear; ii) the second position to a third position, where the third position includes the elliptical mask generally facing the patient's other ear; and iii) the third position to the first position. The method further includes creating a second 3D model of a portion of the patient's face or skull using the surface imaging data. The method further includes registering the second 3D model to the first 3D model. The method further includes guiding a physician during a cranial surgical procedure by displaying on a display screen the first 3D model, the second 3D model, or a combination thereof.

[0006] In another aspect, a computing device has at least one processor and a computer-readable medium storing instructions that, when executed by the processor, configure the device to perform the above method.

[0007] In some embodiments, the computer-readable medium can be and / or include any suitable data storage medium, such as volatile memory and / or non-volatile memory. Examples of volatile memory can include random access memory (RAM), such as static RAM (SRAM), dynamic RAM (DRAM), or a combination thereof. Examples of non-volatile memory can include read-only memory (ROM), flash memory (e.g., NAND flash, NOR flash), magnetic storage media, optical media, ferroelectric RAM (FeRAM), resistive RAM (RRAM), etc. Additionally, the computer-readable medium does not include transient propagated signals or carrier waves.

[0008] In some embodiments, a processor can be substantially any electronic device that can be capable of processing, receiving, and / or transmitting instructions that can be included in, permanently or temporarily stored on, and / or accessed by a computer-readable medium. The processor can be implemented using one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or other circuitry), where the other circuitry can include at least one or more of an application-specific integrated circuit (ASIC), a field-programmable gate array (ASIC), a microprocessor, a microcomputer, etc. Additionally, the processor can be configured to execute instructions in a parallel, local, and / or cross-network manner by, for example, using cloud and / or server computing resources.

[0009] In one aspect, a system for image registration for a cranial surgical procedure includes a movable machine vision system that can be mounted to a robotic arm and is configured to obtain surface imaging data of a portion of a patient's face or skull. The system also includes a fixation device for holding the skull, where the fixation device includes a first arm, a second arm, and a third arm to jointly define the size and position of the skull by using respective reflective spheres that are visible or trackable by the movable machine vision system. The system also includes a console that is configured to create a first 3D model of a portion of the face or skull using the surface imaging data, and the console is configured to register the first 3D model to a second 3D model. The second 3D model is created using preoperative medical imaging data (such as a CT scan, an MRI scan, a PET scan, a SPECT scan, or a combination thereof). The system includes a display that is configured to present the first 3D model, the second 3D model, or a combination thereof to a physician during the cranial surgical procedure.

[0010] The present disclosure describes systems, methods, and techniques for reducing the need for or eliminating manual registration and enhancing the accuracy of registration. Thus, the embodiments described in the present disclosure can be incorporated into the above processes for cranial surgical procedures.

[0011] Based on the following drawings, detailed description, and claims, other technical features may be apparent to those skilled in the art.

[0012] Additional aspects and advantages will be apparent from the following detailed description of embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 An environment of a system for performing imaging registration and cranial surgical procedure navigation according to an example described herein is shown.

[0014] Figure 2Shows an exemplary illustration of a side view of an adjustable three - arm fixation device and an elliptical mask according to an example described herein, where the adjustable three - arm fixation device and the elliptical mask are used during imaging registration and / or cranial surgical procedure navigation.

[0015] Figure 3 Shows an exemplary illustration of a patient - facing view of an elliptical mask according to an example described herein, where the elliptical mask includes various sensors used during imaging registration and / or cranial surgical procedure navigation.

[0016] Figure 4 Shows an exemplary illustration of facial landmarks including a patient's face according to an example described herein, where the facial landmarks are used to determine various features of the patient's face.

[0017] Figure 5 Shows an environment with a display screen that displays an image of a patient's skull acquired using one of various possible modalities of scanning according to an example described herein.

[0018] Figure 6 Illustrates a block diagram of an exemplary method for performing imaging registration and cranial surgical procedure navigation according to an example described herein.

[0019] Figure 7 Illustrates an exemplary method according to an example described herein having Figure 6 additional details of.

[0020] Figure 8 Illustrates a block diagram of a method for matching features of a 3D image target with the same features of a 3D image source according to an example described herein. Detailed Description

[0021] Figure 1 Illustrates an environment of a system 100 for performing imaging registration and cranial surgical procedure navigation according to an example described herein. In Figure 1 the example, patient 102 lies on top of an operating table 104 (“table 104”) equipped with an adjustable three - arm fixation device 106 in a supine position. Figure 1 The side view of shows the first lateral arm 108 and the central arm 110 of the adjustable three - arm fixation device 106. The second lateral arm of the adjustable three - arm fixation device 106 is not explicitly illustrated as it is blocked by the skull 112 of patient 102. The skull 112 of patient 102 is placed in the opening of the adjustable three - arm fixation device 106.

[0022] This disclosure focuses on system 100 that utilizes adjustable three - arm fixture 106, because adjustable three - arm fixture 106 can effectively determine, firmly hold, and / or continuously monitor the position of the skull 112 of patient 102. System 100 also utilizes elliptical mask 114 mounted on robotic arm 116, and is further described in part with reference to Figure 2 and Figure 3 elliptical mask 114. As Figure 1 shown, robotic arm 116 is mounted on cart 118, and cart 118 is movable and can be positioned anywhere in the operating room. In some embodiments, robotic arm 116 can be controlled from a remote location using console 120, and console 120 can be partially located outside the operating room to reduce the risk of contaminating the operating room and / or save some space in the operating room.

[0023] In some embodiments, imaging registration enables a medical professional (not shown) to perform image - guided surgery (IGS) by displaying a portion or the entire skull 112 on display screen 122. In some embodiments, display screen 122 can be mounted on a second cart 124, which can be located inside the operating room. Additionally or alternatively, system 100 can utilize a communication network to display the images shown on display screen 122 on another display screen (e.g., the display screen of console 120) that can be located outside the operating room.

[0024] System 100 is designed for a specific anatomical part of the patient, namely the skull (e.g., skull 112 of patient 102). However, system 100 can be modified to use other adjustable fixtures and / or masks of different shapes. For example, the system can be modified to use an adjustable fixture with more than three arms, such as an adjustable five - arm fixture. As another example, an elliptical mask may be advantageous for skull - related procedures, but another shape may be more advantageous for another anatomical part of the patient. Thus, the systems, methods, and techniques described herein can be modified to perform image registration and surgical procedure navigation for surgical procedures other than cranial surgical procedures.

[0025] Figure 2 An example illustration of side view 200 of adjustable three - arm fixture 106 and elliptical mask 114 according to an example described herein is shown, where adjustable three - arm fixture 106 and elliptical mask 114 are used for imaging registration and / or cranial surgical procedure navigation. Note that Figure 2 is illustrated and described in the context of Figure 1 . Additionally, Figure 2 may include Figure 1 one or more reference numerals of

[0026] As shown, the adjustable three - arm fixation device 106 includes a first transverse arm 108, a central arm 110, and a second transverse arm (obscured by the skull 112 of patient 102). The adjustable three - arm fixation device 106 can be adjusted mechanically and / or electromechanically according to the size of the skull 112. Each arm of the adjustable three - arm fixation device 106 includes a reflective ball or a reflective sphere. Specifically, the first transverse arm 108 includes a first transverse reflective ball 202; the central arm 110 includes a central reflective ball 204; and the second transverse arm (not shown) includes a second transverse reflective ball (not shown). Each reflective ball includes a passive infrared (PIR) sensor that can be embedded in (or located on) the reflective ball. For clarity, as illustrated by the dashed lines, the first transverse PIR sensor 206 can be embedded inside the first transverse reflective ball 202; the central PIR sensor 208 can be embedded inside the central reflective ball 204; and the second transverse PIR sensor (not shown) can be embedded inside the second transverse reflective ball (not shown).

[0027] In some embodiments, each PIR sensor includes a transmitter and a receiver for, e.g., infrared (IR) light. The transmitter of the PIR sensor can emit IR light (e.g., omnidirectionally), and the receiver receives the IR light reflected from the skull 112 of patient 102. Thus, each PIR sensor determines the distance of the corresponding arm of the adjustable three - arm fixation device 106 and / or the corresponding reflective ball from the skull 112 of patient 102. Based on the distances determined by the PIR sensors, in some embodiments, the system 100 (or its components) can automatically (e.g., electromechanically) adjust each arm of the adjustable three - arm fixation device 106 around the skull 112. In various aspects, the predetermined configuration can be based on the size and / or dimensions of the skull 112 of patient 102. In other aspects, the predetermined configuration can be established pre - operatively based on modality scans and / or 3D image sources. Each PIR sensor helps the system 100 to adjust each arm of the adjustable three - arm fixation device 106 to a range of distances from the skull 112 (head), where the range of distances can include a low threshold distance (e.g., 0.2 mm) and a high threshold distance (e.g., 1.0 mm) from the skull 112. The adjustable three - arm fixation device 106 can be adjusted to accommodate the heads of infant patients, adult patients, and / or any other head sizes.

[0028] Additionally or alternatively, the PIR sensors can assist in guiding a medical professional to manually (e.g., mechanically) adjust each arm of the adjustable three-arm fixture 106 around the skull 112 of the patient 102. For example, a medical professional can manually adjust each arm of the adjustable three-arm fixture 106, and the system can include a speaker and / or a display screen to guide the medical professional using acoustic beeps, alarms, and / or phrases (such as "Move the arm closer to the patient's skull", "You are outside the range distance", and / or other phrases of such effects). Additionally or alternatively, the system 100 and / or the adjustable three-arm fixture 106 can utilize proximity sensors using other sensor technologies (e.g., radar technology, etc.) to determine the distance of each arm of the adjustable three-arm fixture 106 and / or each reflective ball from the skull 112 of the patient 102.

[0029] In some aspects, the three reflective balls can communicate continuously with the camera (not shown in Figure 2 ) of the elliptical mask 114, and the camera of the elliptical mask 114 can track its position relative to the three reflective balls. In other aspects, the three reflective balls (by means of their respective PIR sensors) can create a reference frame for the adjustable three-arm fixture 106 and / or the skull 112 of the patient 102 by, for example, determining and / or continuously monitoring the position of the skull 112 during imaging registration and cranial surgical procedure navigation. Note that each PIR sensor can establish x-y-z coordinates in space. Based on three different x-y-z coordinates, the system 100 can define the positions of the reflective balls, each arm of the adjustable three-arm fixture 106, and / or the skull 112 of the patient 102. In other aspects, the reflective balls help the system 100 map the live anatomical structure (e.g., the skull 112) of the patient 102 to a three-dimensional (3D) model (e.g., a 3D image source) of the anatomical structure.

[0030] Figure 3 An example diagram of a patient-facing view 300 of the elliptical mask 114 according to an example described herein is shown, where the elliptical mask 114 includes various sensors used during imaging registration and / or cranial surgical procedure navigation. Figure 3 is illustrated and described with reference to Figure 1 and Figure 2 and may include Figure 3 one or more reference numerals of Figure 1 and Figure 2 .

[0031] The elliptical mask 114 is constructed using materials suitable for and / or graded for the operating room environment. For example, the elliptical mask 114 can be constructed using a medical-grade polycarbonate material. The elliptical mask 114 includes at least one light source. In some embodiments, the light source can be a light-emitting diode (LED) 302, which can be encapsulated in a medical-grade silicone frame. During imaging registration and / or cranial surgical procedure navigation, the LED 302 can continuously illuminate the reflective spheres of the adjustable three-arm fixture 106 and the Figure 4 face 402 of the patient 102.

[0032] In some embodiments, the various sensors of the elliptical mask 114 used during imaging registration and / or cranial surgical procedure navigation can be incorporated into a machine vision system (or multi-camera system). The machine vision system can utilize one or more optical (e.g., visible) light digital cameras 304, such as one or more red, green, blue (RGB) cameras.

[0033] In some embodiments, the camera 304 of the elliptical mask 114 continuously monitors the reflective spheres of the adjustable three-arm fixture 106. By doing so, the camera 304 of the elliptical mask 114 can track its position relative to the three reflective spheres, the adjustable three-arm fixture 106, and / or the anatomical structure (e.g., the skull 112) during imaging registration and / or during a cranial surgical procedure.

[0034] In some embodiments, the camera 304 can use various techniques capable of capturing high-resolution images and converting visible light into an electrical signal. Such camera techniques can include complementary metal oxide semiconductor (CMOS) cameras, charge-coupled device (CCD) cameras, or another camera (or sensor) technology. The resolution of the camera 304 can depend on the working distance, the field of view (FOV), the count of physical pixels in the sensor of the camera 304, and / or other factors and / or parameters. Regardless of the camera technology, the image sensor of the camera 304 can capture a large pixel array of images with a significant amount of physical detail (such as size, edge position, movement, relative position, color information, etc.). The machine vision system can utilize a digitizing device (e.g., a frame grabber), which converts (e.g., transforms) the images captured by the camera 304 into a digital output.

[0035] Although in Figure 3Although not explicitly shown in the figure, in some embodiments, the camera 304 may include and utilize two image sensors (e.g., two CMOS image sensors, two CCD image sensors) that can be separated by a minimum distance (e.g., 1 cm, 2 cm, 5 cm, or another distance). In such a case, the camera 304 may be able to capture stereoscopic vision, and the machine vision system may perform 3D measurements with depth perception (e.g., similar to two eyes). The two image sensors of the camera 304 may capture a target, where the target may be a 3D image target of the face of the patient 102 and / or a part of the skull 112 of the patient 102 (such as the part of the skull 112 facing the oval mask 114).

[0036] As Figure 3 shown, the oval mask 114 also includes one or more IR sensors 306 ("IR sensors 306"), and the one or more IR sensors may be embedded in the oval mask 114 or located at various key positions on the oval mask. In some embodiments, each of the IR sensors 306 includes an IR light emitter (e.g., an IR LED) and an IR receiver. The IR sensors 306 are used by a navigation tracker, where the navigation tracker is an IR tracker. The navigation tracker may utilize the IR sensors 306 to continuously track the positions of various medical devices (not shown) used by a medical professional (e.g., a surgeon) during a cranial surgical procedure. To this end, each of the medical devices may include one or more reflective balls embedded at one or more positions of the medical device. In addition, the IR sensors 306 may assist the medical professional in tracking the positions of the medical devices relative to the anatomical part (e.g., the skull 112), the 3D image target, and / or the preoperative 3D image source. It should be understood that the oval mask 114 eliminates the need for another housing that is only associated with the navigation tracker.

[0037] In addition, since the IR sensors 306 can be effectively and / or accurately used to determine the depth information of an image (or video), in addition to the camera 304, the machine vision system may also utilize one, some, or all of the IR sensors 306. Therefore, in some embodiments, the IR sensors 306 of the oval mask 114 may be utilized by the navigation tracker and the machine vision system. In the case where the machine vision system utilizes the camera 304 and one, some, or all of the IR sensors 306 to capture images, the captured images may be referred to as RGB and IR depth (RGBIRD) images.

[0038] Since in some embodiments, a machine vision system may utilize more than one image sensor technology, the machine vision system may utilize an automatic exposure (AE) algorithm to synchronize in real time the AE output, frame length time, frames per second frequency, and / or frame length lines of an RGB camera (e.g., camera 304) and an IR sensor (e.g., IR sensor 306) to enhance RGB-IR images. Typically, the duty cycle of an RGB sensor is greater than (longer than) the duty cycle of an IR sensor. However, the machine vision system may synchronize the timing of each IR sensor to match the timing of the RGB sensor. Additionally, the machine vision system may align the active portion of the duty cycle of the IR sensor to fall within the active portion of the duty cycle of the RGB sensor. By doing so, the machine vision system may capture RGB information (e.g., color, width, length) and IR depth information of the same image frame. Accordingly, the machine vision system and / or the elliptical mask 114 may accurately capture a 3D image (e.g., 3D image target) of a portion of the face of the patient 102 and / or the skull 112 of the patient 102 during imaging registration and / or a cranial surgical procedure.

[0039] Figure 4 An example diagram 400 is shown that includes facial landmarks 404 (e.g., psychometric points) of a patient face 402 in accordance with an example described herein, where the facial landmarks 404 are used to determine various characteristics of the patient face 402. Figure 4 is described in the context of Figure 1 , Figure 2 and Figure 3 of

[0040] The x-y-z coordinates of each of the facial landmarks 404 on the patient's face 402 are unique with respect to the patient 102. The x-y coordinates of the facial landmarks 404 can define the distance between a first specific facial landmark and a second specific facial landmark, and the distance between two specific facial landmarks is also unique with respect to the patient 102. For example, the facial landmarks 404 can define the distance between the innermost parts of two eyebrows; the distance between the outermost parts of two eyebrows; the distance between two irises (if the patient is awake); the distance between the centers of two eyes (if the patient's eyes are closed); the distance between two ears; the distance between two jawlines (bones); the distance between two cheekbones; the distance between two temporal bones; the width of the lips; the width of the nose; the width of the head and / or other features that may not be explicitly illustrated and / or described herein. In addition, the facial landmarks 404 (e.g., psychometric points) include depth information of the patient 102's face 402 and / or skull 112. The various facial landmarks 404 can have different depths (e.g., z coordinates) and can assist the system 100 and / or its components (e.g., machine vision system) in determining the topographical information of a part of the patient 102's face 402 and / or skull 112 to perform imaging registration and cranial surgical procedure navigation.

[0041] Figure 5 An environment 500 is shown with a display screen 502 having a first image 504 and a second image 506 of a patient's skull 112 according to an example described herein, where the first image 504 and the second image 506 are acquired using one of various modality scans. Figure 5 The display screen 502 of Figure 1 is the same as or equivalent to

[0042] The first image 504 and the second image 506 are CT scans of the patient 102's skull 112. However, medical professionals can use various modality scans, such as CT scans, MRI scans, PET scans, SPECT scans, or combinations thereof (e.g., CT-PET scans).

[0043] The modality scans are preoperatively acquired using a corresponding scanner (not shown) such as a CT scanner, an MRI scanner, a PET scanner, a SPECT scanner, a CT-PET scanner or using more than one type of scanner. Based on the modality scans, using application software, medical professionals (e.g., technicians, engineers, surgeons) can create a 3D model of the skull 112 preoperatively, where the 3D model of the skull 112 is the contour (e.g., outer shape) of the skull 112. The 3D model of the skull 112 may be referred to herein as the 3D image source.

[0044] In some embodiments, the 3D image source is the indexing of a 3D image target during imaging registration and / or craniofacial surgical procedure navigation, as further described herein. Additionally, the 3D image source is the indexing of various instruments used by medical professionals during, for example, a craniofacial surgical procedure.

[0045] Figure 6 FIG. 600 is a block diagram illustrating an example method for performing imaging registration and craniofacial surgical procedure navigation according to an example described herein. Figure 6 is described in part in the context of Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 ; and Figure 6 may include Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 one or more reference numerals of. Unless otherwise specified, the steps of the method need not necessarily be performed in any particular order, or even sequentially, nor need the steps be performed only once. Additionally, the method may be utilized by using Figure 6 one, more than one, and / or all of the steps illustrated in. Thus, the method does not necessarily include the fewest, optimal, or greatest number of steps required to implement the systems, methods, and techniques described herein.

[0046] At step 602, the method may include obtaining one or more modality scans of the skull 112 of patient 102. As discussed, the modality scans may include CT scans, MRI scans, PET scans, SPECT scans, or combinations thereof (e.g., CT-PET scans). For clarity, step 602 of the method is performed and / or completed preoperatively.

[0047] At step 604, the method can include uploading the modality scan onto system 100 (or its components) to create a 3D model and / or 3D image source of the skull 112. To create the 3D model and / or 3D image source, the method includes converting two-dimensional (2D) images (e.g., modality scans) into 3D images (e.g., 3D models, 3D image sources) using one or more techniques such as volume rendering techniques. For example, the 2D images can be represented using a first file format such as the Digital Imaging and Communications in Medicine (DICOM) file format, and the 3D images (e.g., 3D models) can be represented using a second file format such as point cloud or 3D model. In various aspects, the volume pixels (e.g., voxels) in 3D space can be a function of the size of the 2D pixels, where the size of the 2D pixels can be the width along a first axis (e.g., x-axis) and the height along a second axis (e.g., y-axis). For clarity, step 604 of the method is performed and / or completed preoperatively.

[0048] At step 606, the method includes a healthcare professional preparing patient 102. In some embodiments, the preparation of patient 102 can include having patient 102 lie on the operating table 104; placing patient 102's skull 112 within the adjustable three-arm fixation device 106; placing the oval mask 114 in a position facing patient 102's face 402; using the adjustable three-arm fixation device 106 to hold patient 102's skull 112 in a secure position; and manually (e.g., mechanically) or automatically (e.g., electromechanically) adjusting each arm of the adjustable three-arm fixation device 106 around the skull 112 as described in reference Figure 1 and Figure 2 section.

[0049] At step 608, a medical professional may initiate the registration process by, for example, entering a command, pressing a label on the display screen, clicking a mouse, or remotely (e.g., using console 120) or inside the operating room (e.g., using cart 118) using any other user interface. The camera 304 of the elliptical mask 114 may focus on the reflective sphere of the adjustable three-arm fixture 106, and the position of the reflective sphere may be determined. After the camera 304 of the elliptical mask 114 determines the position of the reflective sphere of the adjustable three-arm fixture 106, the robotic arm 116 positions the elliptical mask 114 directly above the patient's skull 112 and facing the patient's face 402. The robotic arm 116 moves (or rotates) the elliptical mask 114 approximately 180 degrees (e.g., +90 degrees and -90 degrees) to capture features of at least a portion of the patient's face 402 and / or skull 112 using the camera 304 and / or IR sensor 306 of the elliptical mask 114. In addition to the camera 304, in some embodiments, the elliptical mask 114 may also utilize one, some, or all of the IR sensors 306 to capture features of at least a portion of the patient's face 402 and / or skull 112.

[0050] In some embodiments, the robotic arm 116 may index three positions: a first position, a second position, and a third position. The first position may be a neutral position or zero-degree position, where the robotic arm 116 may position the elliptical mask 114 directly above (and facing) the patient's face 402. In the neutral position, the camera 304 of the elliptical mask 114 may be equidistant from Figure 2 the first lateral reflective sphere 202 and a second lateral reflective sphere (not shown).

[0051] In some embodiments, the second position may be a first lateral position or +90-degree position, and the second position may be closest to the first lateral reflective sphere 202. In various aspects, the robotic arm 116 gradually moves (or rotates) the elliptical mask 114 from the neutral position (e.g., zero degrees) to the second position while continuously capturing images of the patient's face 402 using the camera 304 and / or IR sensor 306 embedded in or located on the elliptical mask 114. In some embodiments, in the second position, the camera 304 of the elliptical mask 114 generally faces or directly faces and captures an image of the patient's first ear (e.g., the right ear). Then, the robotic arm 116 gradually moves (or rotates) the elliptical mask 114 back from the second position to the neutral position.

[0052] In some embodiments, the third position may be the second lateral position or the -90 degree position, and the third position may be closest to a second reflective sphere (not shown). In various aspects, the robotic arm 116 gradually moves (or rotates) the elliptical mask 114 from a neutral position (e.g., zero degrees) to the third position while continuously capturing images of the face 402 of the patient 102 using the camera 304 and / or the IR sensor 306 embedded in or located on the elliptical mask 114. In some embodiments, in the third position, the camera 304 of the elliptical mask 114 is generally facing or directly facing the second ear (e.g., the left ear) of the patient 102 and captures an image of the second ear. The robotic arm 116 then gradually moves (or rotates) the elliptical mask 114 from the third position back to the neutral position.

[0053] Accordingly, the robotic arm 116 rotates the elliptical mask 114 from i) the neutral position to the first lateral position; ii) from the first lateral position to the second lateral position; and iii) from the second lateral position back to the neutral position. In some embodiments, the camera 304 and / or the IR sensor 306 of the elliptical mask 114 captures images of the face 402 and / or a portion of the skull 112 only during the movement of the elliptical mask 114 from the first lateral position to the second lateral position (which may be approximately 180 degrees). Additionally or alternatively, during the registration process, the camera 304 and / or the IR sensor 306 of the elliptical mask 114 may capture images of the face 402 and / or a portion of the skull 112 during all movements of the elliptical mask 114 and / or the robotic arm 116. In such a case, the elliptical mask 114 may capture images of the face 402 and / or a portion of the skull 112 twice. By doing so, the elliptical mask 114 may capture additional details of the face 402 and / or the skull 112.

[0054] Continuing with step 608, after the elliptical mask 114 captures an image of the face 402 and / or a portion of the skull 112, the system 100 creates a 3D image target of the portion of the skull 112. Similar to step 604, to create the 3D image target, the method includes converting the 2D images captured by the elliptical mask 114 into 3D images (e.g., 3D image targets) using one or more techniques such as the volume rendering techniques described previously.

[0055] At step 610, the method may include comparing a 3D image target created when a medical professional initiates a registration process at step 608 with a 3D image source created preoperatively at step 604. In some embodiments, different features of the 3D image target represent corresponding different features of a portion of the face 402 and / or the skull 112, and the different features of the 3D image target are included in a first data set. Similarly, different features of the 3D image source represent corresponding different features of the skull 112, and the different features of the 3D image source are included in a second data set. The different features in the two data sets may include dimensions, positions, and / or relative positions of specific portions of the face 402 and / or the skull 112, as illustrated and described in the context of Figure 4 and / or Figure 5 as shown and described.

[0056] In some embodiments, the method may include qualitatively and / or quantitatively comparing the 3D image target with the 3D image source. Specifically, the method may compare the first data set with the second data set by, for example, comparing the dimensions, positions, and / or relative positions of each of the different features of the 3D image target with the dimensions, positions, and / or relative positions of the same different features of the 3D image source. Examples of qualitative comparisons between the two data sets may include comparing the dimensions, positions, and / or relative positions of specific portions of the face 402 (such as the eyes, jawline, cheekbones, and / or other portions of the face 402). Another example of a qualitative comparison between the two data sets may include comparing the dimensions, positions, and / or relative positions of the facial landmarks 404 (e.g., psychometric points) of the 3D image target with the corresponding facial landmarks of the 3D image source. Once the method successfully completes the qualitative comparison, the method may map each of the facial landmarks 404 of the 3D image target to each corresponding different feature of the 3D image source.

[0057] Additionally, the method may include performing a quantitative comparison of different features, different parts, and / or facial landmarks 404 of the 3D image target with corresponding different features, different parts, and / or facial landmarks 404 of the 3D image source. In some embodiments, the method may determine the distance between a first psychometric point and a second psychometric point of the 3D image source. For example, based on the 3D image source, the distance may be 11.5 mm. The method may then determine the distance between the same first psychometric point and the same second psychometric point on the 3D image target. For example, based on the 3D image target, the distance may be 11.6 mm. In some embodiments, although the 3D image target may qualitatively match the 3D image source, if the distances (e.g., measurements, dimensions) are not within a predetermined accuracy threshold (e.g., 95%, 98% accuracy), the method may determine that the 3D image target fails to meet the predetermined criteria for registration. However, if the distances are within the predetermined accuracy threshold, the method may determine that the 3D image target meets the predetermined criteria for registration. In such a case, the 3D image target qualitatively and quantitatively matches the 3D image source.

[0058] Thus, at step 612, the method may qualitatively and quantitatively determine whether the 3D image target matches the 3D image source. If the 3D image target matches the 3D image source, then at step 614, system 100 or its components (e.g., speaker, display screen 502) may use the following measures to notify or indicate to the medical professional: an acoustic beep; an alarm; and / or phrases such as "registration complete;", "registration successful;", "registration has 99% accuracy;", "registration meets or exceeds the predetermined accuracy threshold;", "registration meets or exceeds a certain medical standard;", and / or other phrases of such effect. After the method and / or system 100 notifies the medical professional that the registration is successful, the medical professional may begin the surgery. If the registration is not successful, does not meet the predetermined accuracy threshold, and / or does not meet a certain medical standard, the method may repeat steps 608 to 612. Thus, some steps of the method may be an iterative process.

[0059] At step 616, the method determines whether the skull 112 of patient 102, the adjustable three-arm fixation device 106, and / or the operating table 104 have moved after imaging registration. In some aspects, during the course of the surgical procedure, the elliptical mask 114 may remain stationary in the neutral position, and the camera 304 of the elliptical mask 114 may continuously sample and / or monitor the reflective spheres of the adjustable three-arm fixation device 106. By doing so, the camera 304 of the elliptical mask 114 can detect and quantify the movement of the table 104. Similarly, one, some, or all of the camera 304 and / or the IR sensor 306 of the elliptical mask 114 may sample, monitor, and / or quantify any changes in the features of the face 402 of patient 102 and / or the facial landmarks 404. For example, even if the skull 112, the reflective spheres, the adjustable three-arm fixation device 106, and the worktable 104 remain stationary, the chin of patient 102 may drop slightly during the surgical procedure.

[0060] At step 618, the method includes performing an automatic correction of the imaging registration. To this end, the method includes using an automatic correction algorithm to account for the change in position by readjusting the registration. The automatic correction algorithm can readjust the 3D image target to match the 3D image source. For example, when the elliptical mask 114 remains stationary in the neutral position, the automatic correction algorithm creates a bounding box around each of the facial landmarks 404 of the face 402 of patient 102. The dimensions of the bounding box can be predefined as, for example, 5mm × 5mm. In addition, the automatic correction algorithm determines and tracks the x-y-z coordinates and the centroid of each bounding box. To quantify any movement detected at step 616, at step 618, the automatic correction algorithm calculates the Euclidean distance between the x-y-z coordinates before the detected movement and the centroid of each bounding box and the Euclidean distance between the x-y-z coordinates after the detected movement and the centroid of each bounding box. After the automatic correction algorithm quantifies the movement, the automatic correction algorithm updates the x-y-z coordinates of each bounding box to map the movement of any facial landmark 404 of the face 402 and / or the overall movement of the skull 112. Finally, the automatic correction algorithm and / or the method includes re-registering the x-y-z coordinates to reflect the movement of the facial landmarks 404 of the face 402 and / or the overall movement of the skull 112. By doing so, the medical professional can avoid reinitiating the registration process, and the medical professional can continue the surgery without interruption.

[0061] At step 620, the method includes tracking the position of a surgical instrument during a cranial surgical procedure to support and / or provide navigation for the cranial surgical procedure. In some embodiments, the robotic arm 116 positions the elliptical mask 114 in a neutral position. The camera 304 of the elliptical mask 114 continuously monitors the reflective spheres of the adjustable three-arm fixture 106 to create and / or establish a reference frame for the skull 112 and the surgical instrument. The IR sensor 306 of the elliptical mask 114 continuously captures the position and / or movement of the surgical instrument.

[0062] Figure 7 A block diagram 700 is illustrated with additional details of an example method having Figure 6 for performing imaging registration according to the examples described herein. Figure 7 is described in the context of Figures 1 to 6 Unless otherwise stated, the steps of the method need not necessarily be performed in any particular order, or even sequentially, nor need the steps be performed only once. Additionally, the method can be utilized by using Figure 7 one, more than one, and / or all of the steps illustrated in

[0063] Step 702 of block diagram 700 can be the same as, similar to, and / or equivalent to step 602 of block diagram 600. For clarity, at step 702, the method can include obtaining one or more modality scans of the patient 102's skull 112, where the modality scans can include CT scans, MRI scans, PET scans, SPECT scans, or a combination thereof (e.g., CT-PET scans).

[0064] Step 704 of block diagram 700 can be the same as, similar to, and / or equivalent to step 604 of block diagram 600. For clarity, at step 704, the method can include uploading the modality scans to the system 100 (or its components) to create a 3D model and / or 3D image source of the skull 112. To create the 3D model and / or 3D image source, the method includes converting 2D images (e.g., modality scans) to 3D images (e.g., 3D model, 3D image source) using one or more techniques (e.g., volume rendering techniques). Similar to Figure 6 , steps 702 and 704 of block diagram 700 are performed preoperatively.

[0065] At step 706, the machine vision system utilizes the camera 304 of the elliptical mask 114 to capture optical (e.g., visible) light reflected from the patient 102's face 402 and / or a portion of the skull 112. As for example referenced in Figure 3As described, camera 304 can convert reflected light into an electrical signal. The resolution of camera 304 can depend on the working distance, FOV, the count of physical pixels in the sensor of camera 304, and / or other factors and / or parameters. Additionally, camera 304 may be capable of capturing stereoscopic vision, and the machine vision system can perform 3D measurements with length, width, and depth perception.

[0066] In some embodiments, in addition to camera 304, the machine vision system can also utilize one, some, or all of the IR sensors 306 to capture RGB - IRD images. By doing so, the machine vision system can capture RGB (e.g., color, width, length) information and IRD (e.g., depth) information of the same image frame. Thus, at step 708, system 100 can convert the 2D image captured by the machine vision system into a 3D image (e.g., 3D image target, intraoperative image) of a part of the face 402 of patient 102 and / or the skull 112 of patient 102 during imaging registration and / or cranio - surgical procedures. To create the 3D image target, Figure 6 and / or Figure 7 System 100 and / or method can utilize one or more volume rendering techniques.

[0067] At step 710, system 100 can utilize one or more algorithms and / or machine - learning models to detect features (e.g., facial landmarks 404) of the face 402 in the 3D image target and the 3D image source. The algorithm can be a convolutional neural network (CNN) algorithm, a region - based convolutional network (RCNN) algorithm, a region - based fully convolutional network (R - FCN) algorithm, a feed - forward neural network (FNN) algorithm, a Harris corner detection algorithm, a Shi - Tomasi corner detection algorithm, a scale - invariant feature transform (SIFT) algorithm, a speeded - up robust features (SURF) algorithm, a binary large object (BLOB) detection algorithm, one or more feature descriptor algorithms, a histogram of oriented gradients (HOG) algorithm, a binary robust independent elementary features (BRIEF) algorithm, and / or their combination and / or another algorithm that can detect features of the face 402.

[0068] At step 712, system 100 can map the detected features on the 3D image target to the same detected features on the 3D image source. Thus, at step 712, system 100 establishes a correspondence between the same features on the 3D image target and the 3D image source. In some embodiments, the intensity distribution in the adjacent regions of each pixel in the 3D image target and the 3D image source can be used to establish the correspondence. Features can be matched based on similar, identical, or substantially identical measurements and / or dimensions between the corresponding anatomical and / or pathological information in the 3D image target and the 3D image source.

[0069] At step 714, system 100 transforms different data sets (e.g., 3D image target, 3D image source) into the same coordinate system. After the transformation, system 100 performs image registration. Image registration enables medical professionals to perform IGS. In some embodiments, image registration includes calculating a transformation that maps corresponding points (e.g., landmark points) between the 3D image target and the 3D image source.

[0070] At step 716, the machine vision system resamples facial landmarks 404 (e.g., landmark points) of the face 402 using the machine vision system to create a new 3D image target, and system 100 again transforms the coordinates of the new 3D image target into the same coordinate system of the 3D image source. In some embodiments, resampling and transformation (or re - transformation) of the 3D image target is performed periodically during a cranial surgical procedure. Additionally or alternatively, a medical professional may initiate resampling and transformation. After each resampling and transformation, at step 718, system 100 performs registration (or re - registration), and the accuracy of the registration may be calculated. Based on the accuracy of the registration, system 100 may utilize an automatic correction algorithm to readjust the 3D image target to match the 3D image source.

[0071] Figure 8 A block diagram 800 of an example method for matching features of a 3D image target with the same features of a 3D image source according to examples described herein is illustrated. Figure 8 is described in the context of Figures 1 to 7 In addition, Figure 8 the method of Figure 7 includes additional details of step 712 of Figure 8 For the sake of brevity, since the example method of Figure 8 for matching features of a 3D image target with the same features of a 3D image source includes steps that are in the prior art, the description of the example method of

[0072] However, for clarity, the first input to system 100 is the 3D image source 802, and the second input to system 100 is the 3D image target 804. The 3D image source 802 includes a first feature 806, and the 3D image target 804 includes a second feature 808. The first feature 806 and the second feature 808 represent the same feature. In Figure 8 the example illustration of

[0073] In some embodiments, Figure 8The method utilizes a CNN-based encoder-decoder model to extract features of the face 402 of patient 102. The encoder module of the CNN-based encoder-decoder model is illustrated in blocks 810 and 812. Specifically, the encoder module utilizes a stack of at least one convolutional network having a layer with an activation function (e.g., PReLU), at least one dropout layer, at least one max pooling / unpooling layer, and / or at least one convolutional network layer. The encoder generates maps of features, such as maps of a first feature 806 and a second feature 808.

[0074] The decoder module of the CNN-based encoder-decoder model is illustrated in Figure 8 blocks 814, 816, 818, and 820. In some embodiments, the decoder module also includes at least one convolutional network having a layer with an activation function (e.g., PReLU), at least one dropout layer, at least one max pooling / unpooling layer, and / or at least one convolutional network layer. The decoder module serves as a classifier of the CNN-based encoder-decoder model for using the features extracted by the encoder module.

[0075] After encoding and decoding, the CNN-based encoder-decoder model outputs a reconstructed 3D source image 822 and a reconstructed 3D target image 824. The reconstructed 3D source image 822 includes a first reconstructed feature 826, and the reconstructed 3D target image 824 includes a second reconstructed feature 828. The first reconstructed feature 826 and the second reconstructed feature 828 are the same features. Additionally, the reconstructed features 826 and 828 are the same as the features 806 and 808.

[0076] Those skilled in the art will understand that changes may be made to the details of the above-described embodiments without departing from the basic principles of the invention. Thus, the scope of the invention should be determined only by the appended claims and their equivalents.

Claims

1. A method for performing imaging registration and cranial surgical procedure navigation, the method comprising: Obtain a modal scan of the patient's skull; Use the modal scan to create a first 3D model of the patient's skull; Obtain surface imaging data of the patient's face or a portion of the skull while the patient lies supine on top of an operating table using a movable elliptical mask attached to a robotic arm, wherein the robotic arm moves the elliptical mask in a circular motion around the patient's face or the portion of the skull from the following positions by approximately 180 degrees: i) From a first position to a second position, wherein the first position includes the elliptical mask generally facing the patient's nose, and the second position includes the elliptical mask generally facing the patient's ear; ii) From the second position to a third position, wherein the third position includes the elliptical mask generally facing the patient's other ear; And iii) From the third position to the first position; Use the surface imaging data to create a second 3D model of the patient's face or the portion of the skull; Register the second 3D model to the first 3D model; and Guide a physician during a cranial surgical procedure by displaying on a display screen the first 3D model, the second 3D model, or a combination thereof.

2. A non-transitory computer-readable medium, the non-transitory computer-readable medium comprising instructions that, when executed by a processor, configure the processor to perform the method according to claim 1.

3. The method according to claim 1, wherein the modality scan comprises a computed tomography (CT) scan, a magnetic resonance imaging (MRI) scan, a positron emission tomography (PET) scan, a single photon emission computed tomography (SPECT) scan, or a combination thereof.

4. The method according to claim 1, wherein: The modal scan includes a first plurality of 2D images obtained preoperatively; and The surface imaging data includes a second plurality of 2D images obtained during the imaging registration, the cranial surgical procedure navigation, or a combination thereof.

5. The method according to claim 1, wherein: The first position includes a neutral position of the elliptical mask, and wherein the neutral position includes zero movement of the elliptical mask; The second position includes a first lateral position of the elliptical mask, and wherein the first lateral position includes a positive 90-degree movement of the elliptical mask; and The third position includes a second lateral position of the elliptical mask, and wherein the second lateral position includes a negative 90-degree movement of the elliptical mask.

6. The method according to claim 1, wherein the elliptical mask utilizes a machine vision system to obtain the surface imaging data.

7. The method according to claim 6, wherein the machine vision system utilizes at least one imaging sensor, at least one infrared sensor, or a combination thereof to capture red, green, blue (RGB) and infrared (IR) depth (RGBIRD) information of the surface imaging data.

8. The method according to claim 1, the method further comprising tracking the position of an instrument during the cranial surgical procedure, and presenting position information of the instrument relative to the first 3D model, the second 3D model, or a combination thereof.

9. The method according to claim 1, the method further comprising: Use the elliptical mask to detect movement of the face, the skull, the operating table, a fixture holding the skull, or a combination thereof during the cranial surgical procedure; Quantify the movement; And Perform an automatic correction of the imaging registration.

10. The method according to claim 9, wherein the automatic correction of the imaging registration obviates the need to initiate another registration of the second 3D model to the first 3D model.

11. The method according to claim 1, wherein the registration of the second 3D model to the first 3D model meets or exceeds a predetermined accuracy threshold.

12. The method according to claim 11, wherein the registration of the second 3D model to the first 3D model further comprises: Detect features of the face or the portion of the skull in the first 3D model and the second 3D model; And Map the features of the second 3D model to the first 3D model.

13. The method according to claim 12, wherein the detection of the features is performed using: a convolutional neural network (CNN) algorithm, a region-based convolutional network (RCNN) algorithm, a region-based fully convolutional network (R-FCN) algorithm, a feedforward neural network (FNN) algorithm, a Harris corner detection algorithm, a Shi-Tomasi corner detection algorithm, a scale-invariant feature transform (SIFT) algorithm, a speeded-up robust features (SURF) algorithm, a binary large object (BLOB) detection algorithm, one or more feature descriptor algorithms, a histogram of oriented gradients (HOG) algorithm, a binary robust independent elementary features (BRIEF) algorithm, or a combination thereof.

14. The method according to claim 1, wherein the creation of the first 3D model and the creation of the second 3D model are performed using the same volume rendering technique.

15. A system for image registration for a cranial surgical procedure, the system comprising: A movable machine vision system that is mountable to a robotic arm and configured to obtain surface imaging data of a patient's face or a portion of the skull; A fixture that holds the skull, wherein the fixture includes a first arm, a second arm, and a third arm to jointly define the size and position of the skull by utilizing respective reflective spheres visible or trackable by the movable machine vision system; A console configured to: Use the surface imaging data to create a first 3D model of the face or the portion of the skull; and Register the first 3D model to a second 3D model, wherein the second 3D model is created using pre-operative medical imaging data; and A display configured to present the first 3D model, the second 3D model, or a combination thereof to a physician during the cranial surgical procedure.

16. The system according to claim 15, wherein the preoperative medical imaging data is obtained using a CT scan, an MRI scan, a PET scan, a SPECT scan, or a combination thereof.

17. The system according to claim 15, wherein the movable machine vision system is embedded in or located on the elliptical mask, and wherein the elliptical mask at least comprises: A camera, an infrared sensor, a light source, or a combination thereof.

18. The system according to claim 15, wherein each reflective sphere is mounted at the visible or trackable end of each arm of the fixing device, and each reflective sphere comprises a passive infrared (PIR) sensor for determining the position of each arm of the fixing device relative to the patient's skull.

19. The system according to claim 18, wherein the position information determined by using each PIR sensor is used to mechanically or electromechanically adjust each arm of the fixing device to fit the size of the skull.

20. The system according to claim 15, wherein the movable machine vision system tracks the position of the instrument during the cranial surgical procedure, and wherein each instrument comprises one or more reflective spheres.

Images