Patient registration system and method

Abstract

A system for registering a subject (e.g., physical) space to an image space is disclosed. Registration may be automatically performed by a registration system with a registration device. A registration device may acquire an image of a subject's space, generate a point cloud, and compare the point cloud to points in the preoperative image.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 516,413, filed July 28, 2023. This application includes subject matter similar to that disclosed in U.S. Patent Application No. 63 / 516,409 (Attorney Docket No. A0004834US01 / 5074A-000286-US-PS1). The entire disclosure of the above application is incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates to a surgical navigation system, and in particular to a method for registering a patient to image data preoperatively and intraoperatively. BACKGROUND

[0004] The statements in this section merely provide background information related to the present disclosure and can not constitute prior art.

[0005] Guided surgery relies on knowing the position of a patient relative to a device used for surgery. Various forms include image guided surgery (IGS). A registration process is performed to determine a transformation between reference frames, allowing the position of the patient relative to the device to be determined.

[0006] Manual intervention by a surgeon can be required to complete a registration process that can be required prior to the start of a surgical procedure. Reducing manual intervention in the registration process can reduce registration time, improve repeatability, and improve the convenience of registration. SUMMARY

[0007] In one aspect of the present disclosure, a method includes: obtaining a clinical image of a subject; segmenting the clinical image to form a segmented clinical image; determining a clinical point cloud of the segmented clinical image; obtaining a registration image from a registration device; generating a registration point cloud of the subject; selecting at least a sub-portion of both the clinical point cloud and the registration point cloud; and registering a subject space to an image space based on the selected at least sub-portion of both the clinical point cloud and the registration point cloud.

[0008] In another aspect of the present disclosure, a system includes: a registration device that generates an image of a subject; and a controller that: segments a clinical image to form a segmented clinical image; determines a clinical point cloud of the segmented clinical image; obtains a registration image; generates a registration point cloud of the subject; selects at least a sub-portion of both the clinical point cloud and the registration point cloud; and registers a subject space to an image space based on the selected at least sub-portion of both the clinical point cloud and the registration point cloud.

[0009] Images of the subject can be used for diagnosis and treatment of the subject. Such images can be referred to as treatment images or clinical images. Clinical images can be based on image data acquired with an appropriate imaging system, as discussed herein. The clinical images can be projections and / or reconstructions of the acquired image data. Clinical images of the subject can be acquired at any appropriate time, such as, for example, prior to or during a procedure. The clinical images can define a clinical image space. The position of an instrument relative to the subject that has been imaged can be determined with a tracking system. Due to the registration of the subject space to the clinical image space, the position of the instrument can be displayed relative to the acquired clinical images.

[0010] The registration can be performed by determining the position of various points on the subject and correlating these points to points in the clinical image space. This correlation can allow for the determination and generation of a transformation mapping between the physical or subject space of the subject and the clinical image space of the clinical image. Based at least in part on the registration, the tracked position of the instrument can be displayed relative to the clinical image.

[0011] During a procedure, the subject can be registered to the clinical image. The registration can be performed substantially automatically by a registration system. The registration system can acquire a registration image of the subject. Further, during a procedure, the registration can be performed or updated automatically due to a determination that the subject has moved, and the registration system can again register the subject space to the clinical image space.

[0012] Further areas of applicability will become apparent from the description provided herein. It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0013] The drawings described herein are for purposes of illustration only and are not intended to limit the scope of the present disclosure in any way.

[0014] Figure 1 is an environmental view of a surgical navigation system or computer assisted surgical system according to various embodiments; Figure 2A is a high level block diagram of a registration controller of Figure 1 Figure 2B is a detailed graphical block diagram of a registration device of Figure 1 Figure 2C is an environmental view of a navigation system with an imaging system; Figure 3A is a detailed view of a patient with a reference frame according to various embodiments; Figure 3B is a situation of a patient in Figure 3A relative to different scan positions;​​ Figure 3C It comes from Figure 3B The scanned point cloud; Figure 3D It is by Figure 3C An example of a point cloud formed by stitching together points; Figure 3E and Figure 3F It is a facial image with points and segments on it; Figure 3G It is a representation of points determined by the scanning process; Figure 3H These are examples of filtered points that have been cropped around a reference frame and the area of ​​the root of the nose according to various embodiments; Figure 3I This is a high-level flowchart of the methods used to perform the registration process; Figure 3J It is a representation of an optical image of a patient with a reference frame, according to various embodiments; Figure 3K It is a representation of a segmented optical image of a patient with a reference frame, according to various embodiments; Figure 4 This is a flowchart of a method for training a trained classifier; Figure 5 It is a detailed flowchart of at least a part of the non-contact registration process; Figure 6 yes Figure 5 A detailed flowchart of the processing box; Figure 7 yes Figure 6 A detailed flowchart of the clipping frame. Detailed Implementation

[0015] The following description is exemplary in nature and is not intended to limit the content, application, or purpose of this disclosure. Although the following description illustrates and describes surgeries related to a patient's skull, this disclosure should not be construed as limited to such surgeries. For example, surgeries may also be performed with respect to the spine, heart, vascular system, etc. Therefore, unless otherwise specifically stated, discussions of specific areas of anatomy herein are to be understood as applicable to all areas of anatomy.

[0016] As discussed in this article, various systems and components can be used to assist in surgical procedures. For example, clinical image data of a patient can be acquired to help demonstrate the position of instruments relative to the patient. Typically, the clinical image space (i.e., defined by the coordinate system of the image generated or reconstructed from the image data) can be registered to the patient space (i.e., defined by the coordinate system relative to the physical space of the patient, to aid in this display and navigation).

[0017] refer toFigure 1 This document demonstrates a navigation system 10 that can be used in various surgical procedures. The navigation system 10 can be used to track the position of a device 12 (e.g., a pointing probe) relative to a patient 14 to assist in the implementation or execution of surgical procedures. It should be further noted that the navigation system 10 can be used to navigate or track other devices, including catheters, probes, needles, leads, electrode implants, etc. Examples, according to various embodiments, include ablation catheters, deep brain stimulation (DBS) leads or electrodes, microelectrode (ME) leads, or electrodes for recording, etc. Furthermore, the navigation device can be used in any area of ​​the body. The navigation system 10 and various devices can be used in any suitable surgical procedure, such as typically minimally invasive surgery, arthroscopic surgery, percutaneous surgery, stereotactic surgery, or open surgery. Although exemplary navigation systems 10, including image registration systems 16, have been discussed herein, those skilled in the art will understand that this disclosure is for clarity of the discussion only and that any suitable imaging system, navigation system, patient-specific data, and non-patient-specific data can be used. It should be understood that the navigation system 10 can be combined with or used with any suitable preoperative or intraoperative image data.

[0018] The navigation system 10 includes an image registration system 16 for acquiring and comparing preoperative and intraoperative (including real-time) image data of the patient 14. In various embodiments, the system may register to and / or maintain registration with intraoperatively acquired clinical image data. In various embodiments, the system may register to and / or maintain registration with preoperative clinical image data until the end of the surgery or until relative movement (e.g., skull movement) is detected. If movement is detected (e.g., by a distance sensor as discussed herein), registration is maintained and / or re-registration is performed by allowing the collection of additional registration data.

[0019] The registration system 16 may, for example, use visible light, infrared light, electromagnetic energy, light detection and ranging (LiDAR), or thermal sensing technology emitted from and / or received by the registration device 18. In various embodiments, the registration device 18 may include a camera system, such as a stereo camera system, including but not limited to an Intel® RealSense™ D415 or D430 depth camera sold by Intel Corporation, or an Einstar 3D scanner from Shining3D. Therefore, according to various embodiments, the registration device 18 may transmit one or more images, points (point clouds) of line segments with points as data signals, meshes, etc., to the registration controller 20. The registration controller 20 may determine the position of the registration device 18 via a reference locator 26, as further described below in the illustrated examples. The reference locator 26 may be an optional feature.

[0020] As described above, the registration device 18 can be an optical, electromagnetic, and / or lidar device for obtaining one or more registration images or points for registration. That is, the registration device 18 can include sensors for generating points corresponding to the registration images or video stream. If video is captured, registration images from each frame of the video stream can be captured therefrom. The registration device 18 can be fixed relative to the subject and / or movable relative to the subject. In either case, the registration device 18 can be handheld, robotically mounted, or pan-tilt-zoomed to obtain registration images or points. The registration controller 20 ultimately determines or identifies data corresponding to the location of various physical features, distances, or characteristics of the patient for registration, as described in detail below. In various embodiments, this data can relate to markers and / or points on the patient. The registration images from the registration device 18 can include information or data as discussed herein for registration to clinical images of the subject acquired using an imaging system, as discussed herein. The subject's clinical image data can be preoperative or intraoperative image data. The clinical image data can be used to generate the displayed clinical images.

[0021] exist Figure 1 In this example, the longitudinal axis 14A of patient 14 is substantially aligned with the longitudinal axis 22 of operating table 24. In this example, the upper body of patient 14 is elevated, but the longitudinal axes 14A and 22 are aligned.

[0022] The reference markers or point data obtained from the registration images of the registration system 16, which can then be used for registration, can be forwarded to a navigation computer and / or processor controller or workstation 28, which has a user interface 40 and a display device 36 for displaying clinical image data 38. Workstation 28 may also have an auditory device 37 for generating auditory signals, such as a speaker, buzzer, or vibration generator. The display device 36 and / or display can generate visual and / or auditory signals corresponding to the registration or non-registration between the patient space and the clinical image space, as will be described in more detail below. Workstation 28 may also include or be connected to an image processor, a navigation processor, and memory for storing instructions and data. It should also be understood that image data is not necessarily stored in controller 20, but may also be directly transmitted to workstation 28. Furthermore, the processing and optimization of the navigation system and / or registration system 16 can be performed by a single or multiple processors, all of which may or may not be included in workstation 28. For example, registration controller 20 may be incorporated into workstation 28.

[0023] Workstation 28 provides facilities for displaying clinical image data 38 as clinical images on display device 36, and saving, digitally manipulating, or printing hard copies of the received image data. User interface 40 (which may be a keyboard, mouse, stylus, touchscreen, or other suitable device) allows physicians or users 42 to provide input to control image registration system 16 or adjust display settings on display device 36. Workstation 28 can also guide the registration device 18 to adjust its position relative to patient 14.

[0024] Continue to refer to Figure 1 The navigation system 10 may further include a tracking system, such as, but not limited to, an electromagnetic (EM) tracking system 46 or an optical tracking system 46'. Either or both of these tracking systems may be used individually or together in the navigation system 10. The discussion of the EM tracking system 46 herein is to be understood as referring to any suitable tracking system. The optical tracking system 46' may include StealthStation® Treon®, StealthStation® S7, StealthStation® S8, and StealthStation® Tria®, all of which are sold by Medtronic Navigation, Inc. Other tracking system modes may include acoustic, radiation, radar, infrared, etc.

[0025] The EM tracking system 46 includes a coil array or EM locator 48 (e.g., a coil array and / or a second coil array 50), a coil array controller 52, a navigation probe interface 54, a device 12 (e.g., an instrument, tool, catheter, needle, pointing probe, or device, as discussed herein), and a dynamic reference frame (DRF) 44. An instrument tracking device 34a may also be associated with, for example, the device 12 or a guiding device for an instrument, such as being attached to it (or coupled to the registration device 18 as described above). The dynamic reference frame 44 may include a dynamic reference frame holder 56 and a removable tracking device 34b. Alternatively, the dynamic reference frame 44 may include a tracking device 34b, which may be integrally or separately formed with the DRF holder 56.

[0026] Furthermore, the DRF 44 can be provided as a standalone component and can be positioned at any suitable location on the anatomical structure. For example, the tracking device 34b of the DRF 44 can be attached to the skin of the patient 14 with adhesive. Moreover, the DRF 44 can be positioned near the patient 14's legs, arms, etc. Therefore, the DRF 44 does not require a head frame, nor does it require any specific base or holding parts.

[0027] Tracking devices 26, 34, 34a, 34b, or any tracking device as discussed herein, may include sensors, transmitters, or combinations thereof. Further, the tracking devices may be wired or wireless to provide a signal transmitter or receiver within the navigation system. For example, the tracking devices may include electromagnetic coils for sensing the field generated by the EM positioning array formed by EM positioners 48, 50; or reflectors that can reflect signals for reception by the optical tracking system 46'. However, it will be understood that the tracking devices may receive signals, transmit signals, or both, to provide information to the navigation system 10 to determine the position of the tracking devices 34, 34a, 34b. The navigation system 10 can then determine the position of the instrument or tracking device to allow navigation relative to the patient and patient space.

[0028] The coil array or locators 48, 50 may also be supplemented or replaced by mobile locators. The mobile locator may be, for example, the mobile locator described in U.S. Patent Application Serial No. 10 / 941,782 (now U.S. Patent Application Publication No. 2005 / 0085720), filed September 15, 2004, entitled “METHOD AND APPARATUS FOR SURGICAL NAVIGATION,” which is incorporated herein by reference. As understood, the locator array may transmit signals that are received by tracking devices 26, 34, 34a, 34b. Tracking devices 34, 34a, 34b may then transmit or receive signals based on signals received from or transmitted to the array or locators 48, 50.

[0029] The navigation system 10 may further include isolator circuitry or components (not shown separately). Isolator circuitry or components may be included in transmission lines to interrupt lines carrying signals or voltages to the navigation probe interface 54. Alternatively, isolator circuitry included in the isolator housing may be included in the navigation probe interface 54, device 12, dynamic reference frame 44, transmission lines coupling the devices, or any other suitable location. The isolator assembly is operable to isolate any patient-contacting device or patient-fitting device or portion in the event of an unwanted surge or voltage.

[0030] It should be further noted that the entire tracking system 46, 46' or a portion thereof can be incorporated into the registration system 16 (including workstation 28). Incorporation of the tracking system 46, 46' provides an integrated imaging and tracking system. This can be particularly useful when creating a referenceless system that does not have a separate physical or implantable marker attached to the patient. Furthermore, referenceless systems can include tracking devices and contour determination systems, including those discussed herein.

[0031] The EM tracking system 46 uses coil arrays 48 and 50 to generate an electromagnetic field for navigation. The coil arrays 48 and 50 may include multiple coils, each operable to generate a different electromagnetic field in a navigation area of ​​the patient 14, sometimes referred to as the patient space. Representative electromagnetic systems can be found in U.S. Patent No. 5,913,820, issued June 22, 1999, entitled “Position Location System,” and U.S. Patent No. 5,592,939, issued January 14, 1997, entitled “Method and System for Navigating a Catheter Probe,” each of which is incorporated herein by reference.

[0032] The coil array 48 is controlled or driven by the coil array controller 52. The coil array controller 52 drives each coil in the coil array 48 in a time-division multiplexing or frequency-division multiplexing manner. In this respect, each coil can be driven individually at different times, or all coils can be driven simultaneously, with each coil driven by a different frequency.

[0033] When the coils in the coil arrays 48 and 50 are driven by the coil array controller 52, an electromagnetic field is generated in the area within the patient 14 where the medical procedure is being performed; this area is sometimes referred to as the patient space. The electromagnetic field generated in the patient space induces currents in the tracking devices 34, 34a, and 34b positioned above or within the device 12, DRF 44, etc. These induced signals from the tracking devices 34, 34a, and 34b are delivered to the navigation probe interface 54 and subsequently forwarded to the coil array controller 52. The navigation probe interface 54 may also include amplifiers, filters, and buffers for direct interface connection with the tracking device 34b attached to the device 12. Alternatively, instead of direct coupling to the navigation probe interface 54, the tracking device 34b or any other suitable component may employ a wireless communication channel, such as the wireless communication channel disclosed in U.S. Patent No. 6,474,341, issued November 5, 2002, entitled “Surgical Communication Power System,” which is incorporated herein by reference.

[0034] Various parts of the navigation system 10 (e.g., device 12, dynamic reference frame 44) are equipped with at least one (and typically multiple) EM or other tracking devices 34a, 34b, which may also be referred to as positioning sensors. EM tracking devices 34a, 34b may include one or more coils that can operate in conjunction with EM locator arrays 48, 50. Alternative tracking devices may include one or more optical devices 58 and may be used to supplement or replace the electromagnetic tracking devices 34a, 34b. Optical tracking devices may work in conjunction with an optional optical tracking system 46'. Optical tracking devices 58 may include marker or sticker-type devices attached to the patient's skin. However, those skilled in the art will understand that any suitable tracking device may be used in the navigation system 10. Further representative or alternative positioning and tracking systems are described in U.S. Patent No. 5,983,126, issued November 9, 1999, entitled "Catheter Location System and Method," which is incorporated herein by reference. Alternatively, the positioning system may be a hybrid system comprising components from various systems.

[0035] In short, the EM tracking device 34a on device 12 can be located in a handle or insert interconnected with an attachment and can assist in the placement of an implant or the actuation of a component. Device 12 may include a grippable or manipulable portion at its proximal end, and the tracking device 34a may be fixed near or at the distal working end of the manipulable portion of device 12, as discussed herein. The tracking device 34a may include an electromagnetic tracking sensor for sensing an electromagnetic field generated by coil arrays 48, 50, which can induce a current in the electromagnetic device 34a. Alternatively, the tracking device 34a may be actuated (i.e., like the coil arrays above), and the coil arrays 48, 50 may receive signals generated by the tracking device 34a.

[0036] The dynamic reference frame 44 can be fixed to the head 60 of the patient 14 and adjacent to the area of ​​positive navigation, such that any movement of the patient 14 is detected as relative motion between the coil arrays 48, 50 and the dynamic reference frame 44. The dynamic reference frame 44 can be interconnected with the patient in any suitable manner, including those discussed herein. The relative motion is forwarded to the coil array controller 52, which updates the registration and maintains accurate navigation, as further discussed herein. Alternatively, reregistration can be performed when motion is detected. The dynamic reference frame 44 can include any suitable tracking device. Thus, the dynamic reference frame 44 can also be EM, optical, acoustic, etc. If the dynamic reference frame 44 is electromagnetic, it can be configured as a pair of orthogonally oriented coils, each coil having the same center; or it can be configured to employ any other non-coaxial or coaxial coil configuration.

[0037] In short, the navigation system 10 operates as follows: The navigation system 10 creates a point map (which may include all points) from the registered image data generated from the registration device 18. This point map may include external and internal portions corresponding to points in the patient's anatomical structures in patient space. This map generated by the registration device 18 can then be transformed (e.g., a transformation map is created) into clinical image data acquired for the subject 14, such as preoperative or intraoperative image data. After establishing this transformation map, whenever the tracked device 12 is used, the workstation 28, in conjunction with the coil array controller 52, uses the transformation map to identify the corresponding points on the clinical image data and / or atlas model displayed on the display 36. This identification is referred to as navigation or localization. Icons representing the location points of the device are displayed on the display 36 in an appropriate manner relative to the clinical image data; this can be one or more two-dimensional image planes, or three-dimensional and four-dimensional images and models.

[0038] To enable navigation, navigation system 10 must be able to detect both the location of the patient's anatomical structures and the location of device 12 or attachments to device 12 (e.g., tracking device 34a). Knowing the locations of these two items allows navigation system 10 to calculate and display the position of device 12 or any part thereof relative to patient 14. EM tracking system 46 is used to simultaneously track the anatomical structures of device 12 and patient 14.

[0039] If the EM tracking system 46 uses an electromagnetic tracking component, it essentially operates by positioning coil arrays 48, 50 adjacent to the patient 14 to generate a magnetic field, which can be low-energy and is commonly referred to as a navigation field. Because each point in the navigation field, or patient space, is associated with a unique field strength, the electromagnetic tracking system 46 can determine the position of device 12 by measuring the field strength at the location of tracking device 34a. A dynamic reference frame 44 is fixed to the patient 14 to identify the patient's position in the navigation field. The electromagnetic tracking system 46 continuously calculates or measures the relative position of the dynamic reference frame 44 and device 12 during positioning and correlates this spatial information with patient registration data to enable navigation of device 12 within and / or relative to the patient 14. Navigation can include image-guided or image-free navigation.

[0040] The points or portions selected for registration can be image points or point clouds of points from the registration images, compared with points derived from clinical images. These points can be identified at any appropriate time, such as during registration. These points can include landmarks, such as anatomical landmarks, measurements between landmarks, locating elements (e.g., reference markers, multiple DRFs), and combinations thereof, as described in more detail below. The landmarks are identifiable in both the clinical image data and the registration image data, and are identifiable and accessible on patient 14. Landmarks can include individual or distinct points on patient 14 or contours defined by patient 14 (e.g., three-dimensional contours).

[0041] As discussed above, registration of patient space or physical space with clinical image data or clinical image space can utilize the association or matching of intraoperatively observed physical or virtual reference points with image reference points in the clinical images. This can be performed by comparing (e.g., matching) point clouds from the clinical images and the registration images. The point clouds can be based on reference portions (e.g., DRFs) identified in the clinical images and / or registration images. The physical reference point in this example is an example anatomical landmark. Physical reference points can also include contours determined using various techniques (e.g., physical space 3D contours), as well as line segments between points, as discussed herein.

[0042] Now for reference Figure 2AThe details of the registration controller 20 are shown. As described above, the registration controller 20 can be a standalone computer or device, or it can be incorporated into workstation 28. The registration controller 20 can access selected clinical image data (e.g., preoperative clinical image data) and can communicate with the preoperative imaging system 110. The preoperative imaging system 110 can include, but is not limited to, a computed tomography (CT) system that generates computed tomography (CT) images, an X-ray system that generates X-ray images, an O-arm® imaging system, a magnetic resonance imaging (MRI) system that generates MRI images, or an ultrasound system that generates ultrasound images. Examples of preoperative clinical imaging systems are shown below. Figure 2C This will be explained in detail below.

[0043] The preoperative imaging system 110 can acquire preoperative clinical images, which are provided to the registration controller 20 for comparison with registration images. The preoperative clinical imaging system 110 can provide digital image files to the registration controller 20. However, it should be understood that clinical image data can also be acquired during the surgical procedure, and is therefore intraoperative clinical image data or images. CT images and MRI images can serve as clinical images. Similarly, video frames can also be used as clinical images. In this document, the discussion of clinical images or image data is understood to refer to any image data of the subject that can be registered.

[0044] The registration controller 20 can also communicate with network 112. Network 112 (e.g., the Internet) can have a wired or wireless network connection. Various types of data can be transmitted via network 112, including from a remote control device 114 that can be used with the operating system. The remote control device 114 can be a separate component or a component integrated into the system (e.g., workstation 28). The remote control device 114 can include systems for initiating the registration process, acquiring preoperative image data, etc.

[0045] Network 112 communicates with network interface 116. Network interface 116 allows communication from registration controller 20 to network 112 and ultimately to other components (such as workstation 28 or various other devices). Network interface 116 allows network 112 to communicate from remote locations other than the operating room where navigation device 10 is located.

[0046] The registration controller 20 can also communicate with the registration device 18, the display device 36, and the auditory device 37. In this example, the display device 36 and the auditory device 37 are part of the workstation 28. However, especially when the registration controller 20 is located away from the workstation 28, separate display and auditory devices can be provided.

[0047] The registration controller 20 can be a processor, such as a microprocessor, and is programmed to perform various functions. Blocks provided within the registration controller 20 can be individual processors or modules programmed to perform various functions.

[0048] Actuator controller 120 is used to control actuator 152 of registration device 18 when it is in use, such as Figure 2C As described herein, and in more detail below, actuator 152 can be used to scan or move registration device 18. Registration device 18 can also be fixed and / or, for example, manually moved by a user. As discussed herein, registration device 18 may include a physical structure that can move relative to subject 14. Actuator 152 may be a motor or other system for moving registration device 18. Actuator controller 120 may drive the motor based on sensor signals received from registration device 18 or tracking devices 34a, 166, and these sensor signals are received at position sensor input 122. Sensors may also be individual sensors, combined sensors, and include any suitable number of sensors. Sensors may include position sensors, which may be distance sensors for sensing distance from the patient and encoders for sensing the position of the moving actuator. Distance sensors may be infrared distance sensors. Actuator controller 120 and the signals from the position sensors in registration device 18 received at position sensor input 122 are provided to position controller 124. The position controller 124 uses the actuator controller 120 based on the position sensor input 122 to control the actuator at the registration device 18.

[0049] The lighting controller 130 (if selected) is used to control the light source at the registration device 18.

[0050] Image processor 132 receives registration imaging signals from registration device 18. Registration device 18 generates registration image signals from registration image sensors, as will be described in more detail below. Registration device 18 can acquire or generate registration image signals that can be used to register a patient to preoperative image data. Image processor 132 may include a trained classifier 132A. The trained classifier 132A is a system (e.g., a trained machine learning system) for identifying the registration image acquired from registration device 18 or portions thereof (e.g., the root of the nose). The trained classifier 132A may include weights W trained according to the procedures set forth below. Typically, the trained classifier 132A has multiple weights W adjusted using a number of classification images or adjusted over time. The trained classifier 132A can be a convolutional neural network (CNN), an autoencoder algorithm, a recurrent neural network (RNN) algorithm and a transformer neural network algorithm, a generative adversarial network (GAN) algorithm, a linear regression, a support vector machine (SVM) algorithm, a random forest algorithm, a hidden Markov model, and / or any combination thereof. For example, in some embodiments, at least one processor can be configured to utilize a combination of a CNN algorithm or a transformer-based neural network algorithm combined with an SVM algorithm. The trained classifier can also be a machine learning algorithm or a software-based algorithm that uses a metric between points. The image processor 132 can generate points, such as points in a point cloud, from the registered image from the registration device 18. These points are used to identify various facial features or points thereof to perform face recognition, and / or identify various features of a reference frame to be able to define the patient's face and the reference frame. For example, it can identify those points that belong only to one of the patient's face or the reference frame to segment or separate them from other parts of the image or points in the point cloud. Various images or snapshot images (e.g., partial or selected registered image frames) obtained using registration device 18 can be acquired and stitched together to generate an image, which may be a registration image 22. According to various embodiments, one or more methods, such as those for forming the stitched image, can be used, including random sampling or full stitching. In various embodiments of the random sampling method, a snapshot is selected as a reference snapshot. Then, a set of frames is divided into multiple distinct groups. G A random snapshot from each group is selected and registered to a reference snapshot, with data added to the original frame at each registration. In various embodiments, a full-stitching method registers adjacent snapshots. Adjacent snapshots are snapshots acquired in chronological order. Starting with the first snapshot, adjacent snapshots are registered to each other until a goodness-of-fit metric exceeds a certain threshold. The goodness-of-fit metric indicates the degree of registration between two snapshots and is the percentage of fit between two cloud points within a selected percentage or threshold. Thus, the registration data is formed from the identified points within the aforementioned image frames. Finally, the output of image processor 132 is transmitted to registration processor 134.

[0051] The registration processor 134 can perform registration of clinical image data (e.g., preoperative image data, in the form of points or point clouds) with a registration image. This allows registration of the patient space defined by the patient 14 and the physical space associated with the patient 14. As discussed above and further herein, the registration device 18 can acquire an image, referred to as a registration image of at least a portion of the patient 14. The registration image can be converted into points or a point cloud. Common points or reference points between the registration image and the clinical image can be used to perform registration of the patient space with the clinical image space. The positions of points on the patient can be determined based on the poses of the registration device 18 and the patient 14 during the acquisition of the registration image, in order to determine the positions of the points on the patient in the physical space defined by and associated with the patient 14. The registration process can be similar to the process discussed above and includes generating or determining a transformation mapping between the positions of determined points on the patient 14 and the positions of similar or identical points (e.g., head points) in the clinical image data.

[0052] User interface 142, coupled to registration controller 20, is used to provide control signals to various controllers and modules within registration controller 20. Examples of user interface 142 include a keyboard, mouse, or touchscreen.

[0053] The registration controller 20 may also include a timer 144. The timer 144 may record the time of images received from the registration device 18. This may allow the time for determining the location of the registration device, as discussed herein, to be correlated for determining the location of the patient 14 for the registration process.

[0054] Now for reference Figure 2B The registration device 18 is illustrated in further detail. The registration device 18 may have multiple position sensors 150. Each of the actuators 152 and / or arms 153 may have position sensor feedback from its associated position sensor. The position sensors 150 generate multiple position signals, which are ultimately transmitted to the registration controller 20. Control signals from the actuator controller 120 are transmitted to the actuators 152 as signal 120A. The number and type of actuators 152 may vary depending on the type of system.

[0055] Actuator 152 can move a selected portion or the entire registration device 18. Actuator 152 may include only sensors and a light source, or may include more than just sensors, depending on the configuration.

[0056] Distance sensor 156 allows registration device 18 to transmit distance signals to registration controller 20 to determine position and provide position controller 124 with feedback related to that position. Different types of distance sensors can be used, including radar sensors, infrared light propagation time sensors, or laser sensors. Another specific type of distance sensor is a passive infrared (PIR) sensor, which can be used to thermally sense the distance from the mask to the patient. A PIR sensor has a transmitter and a receiver. The transmitter of the PIR sensor can (e.g., omnidirectionally) emit light, and the receiver receives the IR light reflected from the patient. Therefore, each PIR sensor determines the distance. Distance sensor 156 calculates the distance to the head and provides an output based on the distance adjustment required by the movable robotic arm to continue the registration procedure.

[0057] Multiple light sources 160 can be used to illuminate the patient 14 and are controlled by an illumination controller within the registration controller 20. The multiple light sources 160 can surround or be adjacent to the image sensor 154 and are controlled to obtain a useful image. The image sensor 154 can have settable parameters. An image parameter controller 155 can be used to adjust camera settings, such as, but not limited to, aperture, shutter speed, ISO, quality (number of pixels), and white balance. However, the registration device may not require a light source, and ambient light may be sufficient to capture the registered images.

[0058] The registration device 18 may also include a transmitter / receiver 162. The transmitter / receiver 162 may be referred to as a transceiver 162. The transceiver 162 can be used for signal communication with the registration controller 20. The transceiver 162 can communicate, for example, using Bluetooth® wireless communication or another type of wireless technology. The transceiver 162 can also be a wired device. The transceiver 162 communicates with the transceiver 162 located within the registration controller 20. Although... Figure 2A A straight line is shown between the registration controller 20 and the registration device 18, but the transceiver 162 can be used to communicate with the registration device 18 wirelessly or via a wired connection.

[0059] Now for reference Figure 2C It illustrates a diagrammatic view showing an overview of the operating room or facility, similar to... Figure 1 . Figure 1 and Figure 2CThe main difference lies in the further details of the imaging system 180 and the registration device 18, which is mounted on the movable arm 153 and movable with the actuator 152, as described above. Prior to the above process, clinical images can be obtained using any suitable imaging system, including imaging system 180. Finally, the registration images or points from the registration device 18 and imaging system 180 are compared to obtain registration. In various embodiments, the operating room may include a surgical suite with a navigation system 10 that can be used relative to the patient or subject 14. The navigation system 10 may be used to track the position of one or more tracking devices, which may include an imaging system tracking device 162 for tracking imaging system 180. Furthermore, a tool tracking device 166 similar to or the same as tracking device 34a may be included on a tool 168 similar to or the same as device 12. Tools 12 and 168 may be any suitable tool, such as a drill, clamp, catheter, speculum, or other tool operated by user 42. Tool 168 may also include an implant, such as a stent, spinal implant, or orthopedic implant. It should be further noted that the navigation system 10 can be used to navigate any type of instrument, implant, stent, or delivery system, including: guidewires, arthroscopic systems, orthopedic implants, spinal implants, deep brain stimulation (DBS) probes, etc. Furthermore, the instrument can be used to navigate or map any area of ​​the body. The navigation system 10 and various instruments can be used in any appropriate surgical procedure, such as typically minimally invasive surgery or open surgery (including craniocerebral surgery).

[0060] Imaging device 180 can be used to acquire preoperative, intraoperative, or postoperative or real-time clinical image data of a subject (such as patient 14). However, it will be understood that imaging can be performed on any suitable subject, and any suitable surgical procedure can be performed on the subject. In the example shown, imaging device 180 includes an O-arm® imaging device sold by Medtronic Navigation, Inc., which has a business premises in Louisville, Colorado, USA. Imaging device 180 may have a generally annular gantry housing 182 in which an image capture section is movably disposed. The image capture section may include an x-ray source or emitting section and an x-ray receiving or image receiving section, the two sections being generally or practically positioned 180 degrees apart from each other and mounted on a rotor relative to a track or rail. During image acquisition, the image capture section is operable to rotate 360 ​​degrees. The image capture section can rotate about a central point or axis, allowing image data of subject 14 to be acquired from multiple directions or in multiple planes. Imaging device 180 may include those imaging devices disclosed in the following U.S. Patent Numbers: 7,188,998; 7,108,421; 7,106,825; 7,001,045; and 6,940,941; all of which are incorporated herein by reference, or any appropriate portion thereof. In one example, imaging device 180 may utilize flat panel technology having a viewing area of ​​1,720 × 1,024 pixels.

[0061] The position of the imaging device 180 and / or portions thereof (e.g., image capture portions) can be known substantially precisely (e.g., within at least 2 cm, which includes at least 1 cm, and further includes fractions thereof, which include at least 10 micrometers) relative to any other part of the imaging device 180. According to various embodiments, the imaging device 180 can know and recall precise coordinates relative to a fixed or selected coordinate system. This can allow the imaging system 180 to know its position relative to the patient 14 or other reference objects. Additionally, as discussed herein, precise knowledge of the position of the image capture portion can be used in conjunction with a tracking system to determine the position of the image capture portion and image data relative to the tracked subject (e.g., patient 14).

[0062] The imaging device 180 can also be tracked using the tracking device 163. According to various embodiments, the acquired clinical image data defining the clinical image space of the patient 14 can be inherently or automatically registered relative to the object space. This inherent or automatic registration can be complementary to or alternative to registration using the registration device 18 as disclosed herein. The object space or patient space can be the space defined by the patient 14 in the navigation system 10. Automatic registration can be achieved by including the tracking device 163 on the imaging device 180 and / or by determining the precise location of the image capture portion. According to various embodiments, as discussed herein, imageable portions, virtual reference points, and other features can also be used to allow automatic or otherwise performed registration. However, it should be understood that clinical image data of any subject that will define the patient space or subject space can be acquired. The patient space is an exemplary subject space. Registration allows transformations between the patient space and the clinical image space.

[0063] Patient 14 can be anchored within the navigation space defined by navigation system 10 to allow or maintain registration, and / or registration can be obtained and / or maintained using registration device 18. As further discussed herein, registration of the clinical image space with the patient space or subject space allows navigation of devices 12, 168 with reference to clinical image data. When navigating device 168, the position of device 168 can be displayed on display device 36 relative to acquired clinical image data of patient 14, for example, as a graphical representation (e.g., icon) overlay, which represents devices 12, 168 in a selected manner (e.g., simulating device 12, 168). Various tracking systems (e.g., tracking systems including optical locator 48' or electromagnetic (EM) locator 48) can be used to track device 168.

[0064] As discussed, more than one tracking system may be used to track the device 168 in the navigation system 10. According to various embodiments, these tracking systems may include an EM system with an electromagnetic tracking (EM) locator 48 and / or an optical tracking system with an optical locator 48'. Any or both of these tracking systems may be used to track a selected tracking device, as discussed herein. It will be understood that, unless otherwise discussed, a tracking device may be a portion that can be tracked by a selected tracking system. A tracking device does not necessarily refer to the entire component or structure to which it is attached or associated.

[0065] It should be further understood that imaging device 180 may be an imaging device other than an O-arm® imaging device, and may additionally or alternatively include a fluoroscopic C-arm. Other exemplary imaging devices may include fluorometers, such as biplane fluoroscopy systems, ceiling-mounted fluoroscopy systems, catheterization lab fluoroscopy systems, fixed C-arm fluoroscopy systems, isocentric C-arm fluoroscopy systems, 3D fluoroscopy systems, etc. Other suitable imaging devices may also include MRI, CT, ultrasound, etc.

[0066] In various embodiments, the imaging device controller 196 can control the imaging device 80 and can receive image data generated at the image capture section and store these images for later use. The controller 196 can also control the rotation of the image capture section of the imaging device 180. It will be understood that the controller 196 does not need to be integrated with the frame housing 182, but can be separate from the frame housing. For example, the controller 196 can be part of a navigation system 10, which may include a processing and / or control system including a processing unit or processing system 198. However, the controller 196 can be integrated with the frame housing 182 and may include a second separate processor, such as a processor in a portable computer.

[0067] Patient 14 can be secured to operating table 24. According to one example, operating table 104 can be an Axis Jackson® operating table sold by OSI, a subsidiary of Mizuho Ikakogyo Co., Ltd., which has a place of business in Tokyo, Japan, or by Mizuho Orthopedic Systems, Inc., which has a place of business in California, USA. Patient positioning devices can be used with the operating table and include elements such as Mayfield® clamps or those set forth in co-assigned U.S. Patent Application No. 10 / 405,068, filed April 1, 2003, entitled “An Integrated Electromagnetic Navigation and Patient Positioning Device,” which is incorporated herein by reference.

[0068] Navigation system 10 can determine the position of patient 14 relative to imaging device 80. Tracking device 163 can be used to track and position at least a portion of imaging device 180, such as gantry housing 182. Patient 14 can be tracked using dynamic reference frame 44, such as... Figure 1The process under discussion can be invasive and / or non-invasive or minimally invasive. That is, the patient tracking device or dynamic reference device 44 can be used to receive or generate signals that are transmitted to the interface portion 99.

[0069] Therefore, patient 14 relative to imaging device 180 and relative to Figure 1 The position of the registration device 18 can be determined initially and upon detection of movement (such as skull movement). Furthermore, the position of the imaging portion can be determined relative to the housing 182 due to its precisely positioned, substantially rigid rotor, etc., on rails within the housing 182. For example, if the imaging device 180 is an O-Arm® imaging device sold by Medtronic Navigation, Inc., which has a business location in Louisville, Colorado, the imaging device 180 can have known positional accuracy and repeatability within 10 micrometers. Precise positioning of the imaging portion is further described in the following U.S. Patent Nos.: 7,188,998; 7,108,421; 7,106,825; 7,001,045; and 6,940,941; all of which are incorporated herein by reference. According to various embodiments, the imaging device 180 can generate and / or emit x-rays from an x-ray source that propagate through the patient 14 and are received by the x-ray imaging receiving portion. The image capture portion generates image data representing the intensity of the received x-rays. Typically, the image capture section may include: an image intensifier that first converts X-rays into visible light; and a camera (e.g., a charge-coupled device) that converts the visible light into digital image data. The image capture section may also be a digital device that directly converts X-rays into digital image data to form an image, thereby potentially avoiding the distortion introduced by first converting them into visible light.

[0070] Two-dimensional and / or three-dimensional perspective image data that can be captured by imaging device 180 can be captured and stored in imaging device controller 196. Multiple image data captured by imaging device 180 can also be captured and grouped to provide a larger view or image of the entire area of ​​patient 14, rather than just pointing to a portion of a region of patient 14. For example, multiple image data of patient 14's spine can be appended together to provide a complete view of the spine or a complete image dataset. Any or more of these types of image data can be clinical image data.

[0071] Clinical image data can then be forwarded from the imaging device controller 196 to a navigation computer and / or processor system 198, which may be part of the controller or workstation 28. It should also be understood that clinical image data is not necessarily initially stored in the controller 196, but may also be directly transferred to the workstation 28. The workstation 28 can provide facilities for displaying image data as an image 38 on a display 36, saving, digitally manipulating, or printing hard copies of the received image data. The user interface 40 allows the user 42 to provide input to control the imaging device 180 via the imaging device controller 96, or to adjust the display settings of the display 36. The workstation 28 can also guide the imaging device controller 196 to adjust the image capture portion of the imaging device 180 to obtain various two-dimensional images along different planes, in order to generate representative two-dimensional and three-dimensional image data.

[0072] Continue to refer to Figure 2CThe navigation system 10 may further include a tracking system comprising either or both of an electromagnetic (EM) locator 48 and / or an optical locator 48'. The tracking system may include a controller and an interface portion 99. The interface portion 99 may be connected to a processor system 198, which may include a processor contained within a computer. EM tracking systems may include the STEALTHSTATION® AXIEM™ navigation system sold by Medtronic Navigation, Inc., which has a business premises in Louisville, Colorado; or may be the EM tracking system described in the following documents: U.S. Patent No. 7,751,865, entitled “METHOD AND APPARATUS FOR SURGICAL NAVIGATION”, issued July 6, 2010; U.S. Patent No. 5,913,820, entitled “Position Location System”, issued June 22, 1999; and U.S. Patent No. 5,592,939, entitled “Method and System for Navigating a Catheter Probe”, issued January 14, 1997; all of which are incorporated herein by reference. It will be understood that navigation system 10 may also be or include any suitable tracking system, including the STEALTHSTATION® TREON® or S7™ tracking system with an optical locator (which can be used as optical locator 48') sold by Medtronic Navigation, Inc., Louisville, Colorado. Other tracking systems include acoustic, radiation, radar, etc. The tracking system may be used in accordance with techniques generally known or described in the above-incorporated references. Details will not be included herein except for selected operations that are intended to clarify the disclosure of this subject matter.

[0073] Wired or physical connections can interconnect tracking systems 46 and 46', imaging device 180, etc. Alternatively, instead of direct coupling to processor system 198, various components, such as instrument 168, can utilize wireless communication channels, such as those disclosed in U.S. Patent No. 6,474,341, issued November 5, 2002, entitled "Surgical Communication Power System," which is incorporated herein by reference. Furthermore, tracking devices 163, 166 can generate fields and / or signals sensed by tracking systems 46, 46'(s).

[0074] Various parts of the navigation system 10 (such as device 168 and other parts described in detail below) may be equipped with at least one tracking device of tracking devices 166, and typically multiple tracking devices. The device may also include more than one type or form of tracking device 166, such as EM tracking devices and / or optical tracking devices. Device 68 may include a grippable or manipulable portion at its proximal end, and the tracking device may be fixed near the manipulable portion of device 68.

[0075] Another representative or alternative positioning and tracking system is described in U.S. Patent No. 5,983,126, issued November 9, 1999, entitled "Catheter Location System and Method," which is incorporated herein by reference. The navigation system 10 may be a hybrid system comprising components from various tracking systems.

[0076] Now for reference Figures 3A to 3D This illustrates an example of the process used to acquire the registered image. Patient 14 (and in this case, the patient's head) is shown as having a reference frame 44 attached thereto. Reference frame 44 in Figure 3A and Figure 3B Shown separately in [the text]. Figure 3B In the diagram, registration device 18 is shown in various locations. As discussed above, registration device 18 can be or at least includes a point collection and / or determination system. For example, a lidar, stereo camera, depth camera, etc. Registration device 18 allows the generation of image frames of meshes, point clouds, and / or point clouds, which can be stitched together.

[0077] In the first position (scan 1), the registration device generates multiple points 310 corresponding to different points on the patient 14. The registration device 18 generates a second set of points as scan n. Although... Figure 3B The diagram shows two scans, but multiple scans can be performed at multiple angles with multiple speeds and other types of scan parameters. The registration device 18 is scanned along path 314 to obtain multiple images. Points from the registration device (e.g., a handheld camera) are used to obtain a stitched-together registered image, as described above. Each scan frame or scan location can allow the generation of one or more points, such as through processing of depth images or LiDAR scans. Each point can have positional data (e.g., x, y, z data) and additional data (e.g., normal and color information). Each frame can have one or more points that match or overlap with at least one point in another frame, i.e., they have the same x, y, z data. Therefore, different frames can be stitched together to form a registered image.

[0078] Various features, such as patient tracker 44, can be identified in the registered images. In various embodiments, key features of the images can be obtained using face recognition through machine learning-based classification techniques. That is, face recognition can be used as a means of image segmentation to identify key points from the image, or a convolutional neural network (CNN) or transformer-based neural network trained to identify and segment points belonging to the reference frame and the patient's head can be used to identify key points from the image.

[0079] exist Figure 3C In the image processing, the first plurality of points 310 and the second plurality of points 312 are transmitted to the image processor 132. The image processor 132 stitches the plurality of point images together to form a composite image. Figure 3D Point cloud 320 is shown in the image. Multiple points 310 and 312 are also point clouds. However, point clouds from different scans... Figure 3D The data are merged (stitched) together to form the registration data of the final registration point cloud. As discussed above, stitching can be performed according to various embodiments. In any case, as... Figure 3D As shown, stitching allows the generation of point clouds that include selected volumes or regions for performing registration, such as selected volumes or regions of face 332.

[0080] In one example, as the registration device 18 moves along the scan path 314, the registration device allows a new registered image to be formed (e.g., acquired) every 30 milliseconds. Image captures can be referred to as snapshots or frames. Each point can have a “normal” associated with it, allowing each point in the point cloud to have a position (i.e., x, y, z data) and orientation relative to the surface via the normal. Stitching can select random snapshots as reference snapshots. This set of snapshots can be divided into multiple distinct groups G, where the number G is any suitable number and can be based on the total number of snapshots acquired, the volume imaged, etc. Each snapshot in group G can be added to the original reference frame. Additionally or alternatively, according to various embodiments, when snapshots are acquired sequentially around the patient in temporal and spatial order, adjacent snapshots can be stitched together sequentially. Thus, each of the snapshots is stitched to the previous snapshot and can be mathematically adjusted to fit together.

[0081] Now for reference Figure 3E and Figure 3FThe registration features or landmarks (e.g., individual points 320) on the patient 14 in the image can be manually or automatically identified by the comparison module 136. In manual configuration, the user can move the tracked probe or component to track or position the pose of individual points on the patient and / or in the image (e.g., by moving the mouse pointer). In automatic configuration, instructions can be executed by a processor (e.g., in the comparison module 136) to identify points 320. This can be based on a trained machine learning system, a selected algorithm, image segmentation, including those discussed herein.

[0082] Features or points 320 may include such anthropometric locations, such as head points, like the edges of the eyes (eye point), the position of the earlobes (ear point), the chin (chin point), the mouth (mouth point), the nose (nose point), and various other locations. Figure 3E In this context, the segment 322 between points 320 can be used for comparison with measurements in clinical image data. Points 320 and / or distances 322 can be used to determine registration. Points may include a predetermined number of landmark features, such as the root of the nose 344A, the eye 344B, the eyebrow 344C, and the tip of the nose 344D. As discussed above, points in the pre-acquired image can be compared with points 320, distances 322, and / or distances from the image sensor 154 for registration. Figure 3E This represents the entire face and the possible landmark features used in the registration process, such as landmark features defined by points.

[0083] Based on the location, measurement, or both of the points, the comparison module 136 can generate signals indicating whether and / or whether registration can be performed between patient space and image space. During the procedure, the user 42 may occlude a portion of a point or measurement outside the field of view of the camera or image sensor 154. For example, a number of points can be identified in clinical image data, and the same points or all points can be identified on the patient 14 in the registered image data from image sensor 154. In some configurations, only points at a predetermined distance from the reference frame can be used, and only non-moving portions (sub-portions) of the face can be used. This will... Figure 3H The details are described in more detail below. Points in the registration image and their physical positions relative to the patient 14 in patient space can be determined using distances measured by distance sensor 156 and / or using the registration image data. Furthermore, the position relative to the patient 14 can be further determined due to movement of motors associated with actuators 152, which can be driven by electrical signals, thereby allowing… Figure 2A The position controller 124 controls the precise position of the registration device 18. For example, the motor can rotate a predetermined number of rotations based on feedback provided by position signals from position sensors 150, which can be potentiometers, encoders, or part of a motor (as a servo motor).

[0084] Now for reference Figure 3G and Figure 3H The scan can be performed as follows: by moving the registration device 18 relative to the face (e.g., around and / or along the longitudinal axis 334, e.g., Figure 3B The registration device 18 scans the patient's face 332, including one or more landmarks 336. Figure 3G In the process, landmark points 336 are recorded from the registered images. Finally, data points determined based on the scanned images and / or the scanned images are provided to comparison module 136, where the data points and / or the scanned images are compared with clinical images and / or points determined based on the clinical images. The registration device 18 can use any suitable wavelength (including one or more wavelengths, such as infrared, visible, or both) to acquire images of the subject, which may include the locations of unique anthropometric points (e.g., the patient's face) for each patient 14. According to various embodiments, the anthropometric points may be landmark points 336 and may be determined relative to the face 332 in any suitable manner, such as from the top to the bottom of the patient 14's head. Scans performed with the registration device 18 can be performed in less than 10 seconds and can operate at various distances from the head, such as approximately one meter.

[0085] exist Figure 3H (and refer to this) Figure 3J and Figure 3K (Further discussion) In some examples, the registration system 16 may strategically crop the registration image and / or clinical image without taking all landmark points 336 into consideration. According to various embodiments, the reference frame 44 may be mounted at a fixed location within the boundary 340, and points associated with the reference frame 44 may be used for registration. This cropping of points from the image (e.g., the registration image) may allow for faster or more efficient registration. The boundary 340 may be a circle or other shape at a predetermined distance D1 from the reference frame 44. The distance D1 may be selected such that non-overlapping areas and areas with no potential for movement are taken into account. According to various embodiments, the facial region 342 may be defined as the boundary of a non-moving location. For example, region 342 may extend a predetermined distance from the root of the nose 344A (root of the nose point). Region 342 may be circular, elliptical, or other shapes to include a predetermined number of features, such as the root of the nose 344A, eyes 344B (multiple eye points), eyebrows 344C (multiple eyebrow points), and the tip of the nose 344D (nose point). Images from both the registration device 18 and the preoperative images can have the same features and the same boundaries.

[0086] In one example, reference frame 44 can be used for registration. In this case, frame initialization can be performed. The frame initialization input can be a single point on the reference frame located at a radius of a flat plane, along with the surface normal at that point. This radius can be determined based on the size of the reference frame, and this point can be referred to as the Finit point. The Finit point can be detected automatically and / or manually during the scan and is predefined in the model of the reference frame. During the point cloud scan, a sphere surrounding the Finit point is cropped from the scan mesh or the remainder of the points. The size of the sphere can be predetermined and / or manually selected based on the size of the reference frame and / or training of a machine learning system. This results in a segmented registered image with a point cloud data sphere centered on the Finit point at the selected radius. The selected radius can also be based on the size of reference frame 44. The segmented reference frame scan is aligned with the model of the reference frame such that the Finit points in each space are matched. Further, the normals of the scan are matched with the normals of the model of the reference frame (that is, perpendicular to the surface of the CAD model). The segmented registered image of the reference frame is registered to the model using an iterative nearest-point-to-plane method. The segmented reference frame can then be rotated 36 degrees and re-registered to the model. This process is repeated multiple times, for example, 10 times, with the best-fit registration chosen as the final registration orientation.

[0087] Similarly, head or face registration initialization can be performed. In one example, head initialization can take the location of the nasal root in the clinical image and the registration image as input. The nasal root in the clinical image can be automatically determined using a selected known segmentation technique, such as a facial landmark detection algorithm, CNN-based point cloud segmentation, or a transformer-based neural network. The nasal root points in the registration image are automatically determined by a selected face detection algorithm based on: CNN, transformer-based neural networks, and deep learning methods such as YOLO (YOLO Only Once), feature detection and matching based on edges or Haar wavelets, etc. Neural networks and deep learning can be techniques used for segmentation. In the overall segmentation, there can be partial segmentation and keypoint detection (i.e., in the segmented parts), which can be performed using deep learning and neural networks. According to various embodiments, a curvature matching system (such as a machine learning training system (e.g., a neural network)) can be used to identify the nasal root. A selected curvature measured at a selected distance can be identified as the nasal root in the registration image.

[0088] Segmentation can be performed in two stages. Stage (1) can provide segmentation of the entire skin surface or anatomical mask (e.g., “face”). Stage (2) identifies certain areas or selected points (e.g., “nose”, “root of the nose”) within the identified face.

[0089] A sphere of a selected radius is cropped around a selected point (such as the root of the nose, which can be called the Hint point). This cropping or segmentation removes unwanted noise and the reference frame. The registered image of the head is then registered with the clinically segmented head image using a selection process (such as Coherence Point Shift (CPD)). Similar to the reference frame initialization process, the registered image can be rotated 36 degrees around the Hint point and registered on each iteration, with the best match selected for registration.

[0090] Now for reference Figure 3I The flowchart illustrates a high-level method 346 for operating the aforementioned system. In box 348, a clinical image is obtained using one of the aforementioned systems. The clinical image data obtained in box 348 may have one or more landmark points identified therein. As discussed above, the clinical image data may identify a portion or segment of the entire skin anatomical mask in a first stage (optional stage). The skin surface or anatomical mask can then be segmented therein. Landmarks can be identified thereon. In box 352, the registration system 18 can be initiated, for example, via a user interface to start the system. The user interface may be a remote control device, with signals transmitted via… Figure 2A The network 112 shown transmits data from this remote control device. However, the system can be operated via direct wired or wireless communication through the registration controller 20, user interface 142, and / or workstation 28.

[0091] In box 354, the registration device can be positioned relative to a portion of the patient, such as for selective alignment or scanning. In various embodiments, a motor driving the actuator moves the registration device 18. However, as discussed above, the registration device 18 can also be moved manually by the user.

[0092] In box 358, obtain the patient's registration image or data. The registration data can be processed to generate a data point cloud. For example, obtain registration image data for certain landmark features, such as bone position, nasal root, physical features (e.g., corners of the eyes), and distances between physical features (e.g., the distance between the two corners of the eyes). Facial features and their relationships are... Figures 3A to 3G The image is displayed in the image. Within frame 360, the registered image can optionally be stored in memory 138.

[0093] In box 362, registration data corresponding to registration features (such as registration distance in the registered image) is compared with the same features in the clinical image. This comparison determines whether the registration features of the clinical image and the registered image are associated, and whether the patient space of patient 14 can be registered with the clinical image space and / or has been registered. When association is successful in box 364, box 366 can generate an auditory and / or visual indicator via display device 36 or auditory device 37 to indicate to user 42 that association was successful or to provide the user with an indication of successful association. As discussed above, this association can be used to generate a transformation map to allow registration of the patient space with the image space. This registration can then be output in box 370. When association is unsuccessful in box 364, a second auditory and / or visual indicator 368 can be used to indicate to user 42 that association was unsuccessful. This can lead to corrective measures, such as moving the patient or registration device 18 to obtain a second registered image. Errors in the procedure can be checked, and if the error is not within a defined threshold, re-registration can be performed. It should be understood that the indication signal is optional during the registration process.

[0094] Figures 3A to 3H The demonstrated process allows for the complete replacement of manual registration with the automated registration described herein using an electromagnetic tracking system or any suitable tracking system. That is, registration, except for the initial process (e.g., acquiring the registration image by moving the registration device), can be automated. This can increase the speed of registration and potentially improve the level of accuracy, while allowing the user who previously performed manual registration to perform other tasks. In one exemplary embodiment, this registration is significantly faster than a manual process. The system allows for automated registration, is easily remotely controlled, and preventative maintenance on such a system is relatively easy.

[0095] Now for reference Figure 3J and Figure 3K The original or complete image 380A before segmentation is shown. Image 380A can be generated using a registration device and is a complete or whole registered image. Image 380A can be a grid from which its points are generated and / or can be defined by points in a point cloud. According to various embodiments, the registration device can generate a grid, and the output of the registration device can be a 3D scanner output (grid) that is converted into a point cloud. As discussed herein, selected systems (such as CNNs) can be trained to identify and segment points belonging to the head and patient reference frames. The segmented portion can include only the reference frame points detected during frame registration or a subsample of points within a known radius around the detected frame, ignoring other points in the point cloud. As described above, the reference frame... Figure 3HThe radius can be D1, which allows for cropping of the image (including the point cloud). According to various embodiments, only the patient head point during patient registration or a subsample of points within a known radius surrounding the detected head is used, and other points in the point cloud are ignored. Alternatively, both types of points can be used and segmented from other points, such as background points (e.g., patient support or patient hold-up portion 381).

[0096] exist Figure 3K The diagram shows image 380B (e.g., a point cloud) segmented into head region 382 and optical tracking device image 384. However, optical tracking device image 384 can be any suitable patient reference, such as reference 44 discussed above (e.g., EM, optical, acoustic, etc.). Head region 382 and optical tracking device image 384 can be separated from other parts (e.g., black portion 386), as discussed herein. The classifier can be appropriately trained (e.g., CNN) or execute a selected and appropriate algorithm. However, the segmentation process allows at least some data points to be ignored during registration. In other words, black portion 386 is discarded in the segmentation because it is irrelevant to registration. Figure 3I The process can be applied to related segmented parts, such as for registration.

[0097] Now for reference Figure 4 The process for training is described in procedure 410. Figure 2AA method for training a classifier 132A in an image processor 132. The trained classifier can be a machine learning process, such as a convolutional neural network (CNN) or a transformer-based neural network. In block 412, multiple training set images are generated or provided to the system to train the system to recognize points in the registration images. The training data can involve both clinical images and registration images. In various embodiments, a trained classifier can be provided for both clinical images and registration images, or a separate trained classifier can be provided for each. The multiple training sets have multiple images and correct labels for various features in the images. For example, the training sets have known landmarks, selected radii, and classifications for each point belonging to at least one of the patient's head or a patient reference frame; points can also be labeled as background. Typically, after providing sufficient training sets, appropriate labeling is achieved in the acquired registration images during surgery via automatic labeling using the trained classifier. In block 414, an input image is provided to the trained classifier. In box 416, the output of the trained classifier to the test image is compared with known outputs, and / or labeled by an expert (e.g., a surgeon) in conjunction with the trained image and / or independently of it. In box 418, the classifier weights W are adjusted based on the comparison. For example, the weights of a CNN or a transformer-based neural network can be adjusted to achieve selected or “known” labels for selected features in the input image (e.g., the input registration image).

[0098] In box 420, determine if the accuracy of the training and output is within an acceptable range. If the accuracy is not within an acceptable range in box 420, repeat boxes 412 to 418. If the accuracy is acceptable in box 420, end training in box 422. The trained classifier can then be stored and / or accessed to identify selected features in the input image, such as during the program.

[0099] For example, the output of the trained classifier can be a landmark point cloud or a patient's head and reference frame segmented from the remainder of the point cloud identified in the acquired registration image and / or clinical image. These two sets of points can be correlated to perform registration between the image space of the clinical image and the patient space identified by the registration image. Registration allows for the generation of a transformation mapping between the two spaces.

[0100] Now for reference Figure 5 The paper elaborates on a method for performing patient registration. In box 510, the patient is positioned in a fixed location, such as on an operating table or bed. When performing registration imaging, factors such as skin color, age, gender, and facial hair can be considered. For example, lighting, scanner position, and camera settings can be varied based on these factors.

[0101] In box 512, a reference frame (such as the electromagnetic (EM) reference frame 44 described above) is attached to the patient. The reference frame can be attached (e.g., with threaded fasteners), glued, or taped to the patient's skull. In box 514, algorithms such as facial landmark detection, feature detection, or machine learning-based methods can be used to detect the registration model and nasal root position during registration checks based on a skin segmentation model. Instructions for manually holding the handheld registration device 18 can be provided. However, the positions of the reference frame and scanner can vary. It should be noted that automated or robotic scanning can also be performed, or alternatively. The scan volume and scan duration can vary depending on the device used (e.g., scanner). Fixed or random paths can be used during scanning. For example, scanning can be stopped when a suitable amount of data is obtained. In box 516, the scanner can be held at a predetermined distance during scanning. For manual scanners, in box 518, the operator (e.g., a surgeon) presses a button and performs the scan. Scanning can be performed to obtain registration images at different locations. Within frame 520, video can be used to acquire multiple registration images, or a single registration image can be used. The scanner's scanning speed and sweep can be performed at different distances and speeds relative to the patient site being scanned. More stable speeds can be provided by the instrument, such as... Figure 2C The movable arm is included. However, manual scanning is also available.

[0102] In box 522, in some examples, various features (such as the face) and a patient tracker (such as reference frame 44) can be captured. For the face, various features can be identified, such as the eye sockets, nose, forehead, and nasal root. As described in more detail below, certain features within the boundaries can be obtained. In box 524, various image processing can be performed, such as generating points to be used in registration. Finally, registration in box 526 can be performed, as discussed in further detail below.

[0103] Now for reference Figure 6 The processing box 524 is shown in further detail. Input is provided by a user-oriented action 612. The user-oriented action 612 may include: scanning the face and reference frame in box 614 to capture a registration image; and loading CT or other types of clinical images from a clinical application in box 616. In box 526, two different inputs and associated flow paths are executed to achieve registration. The face and reference frame obtained as the registration image in box 614 are provided to box 620 to detect the nasal root from the registration image. Similarly, in box 622, the reference frame is detected in the registration image received from box 614. In box 624, the registration image may be cropped and further denoised mathematically and / or according to the process discussed above. Details of the boundary-based cropping process are provided in... Figure 7The process is described in section 626. In box 626, point cloud stitching is used to stitch the processed registered images together. The stitched point cloud becomes the registration point cloud used for registration. The point cloud stitching process is described above. Figures 3A to 3D The process was demonstrated and described in the document. Ultimately, point cloud stitching uses multiple registered images and performs coordinate processing on the points from these images to find the final registered point cloud.

[0104] Based on the CT image or other type of clinical image obtained in box 616, box 630 establishes a registration model or segmented clinical image from the clinical image. Binary segmentation can be used to form the registration model. Skin segmentation can be performed on the clinical image data. Skin segmentation can be performed because the registration device 18 can generate data about the patient's skin-related surface, such as a point cloud. Skin segmentation can be performed using any suitable technique and can be used to generate a mesh from which the point cloud can be generated. Ultimately, one or more types of segmentation form the segmented clinical image.

[0105] In box 632, the nasal root and / or other facial features can be determined based on the registration model. Similarly, in box 634, the registration model can be transferred to the surgical space. Finally, the registration model is provided to box 526, where registration between two sets of point clouds is determined. The registered point cloud obtained from box 626 is compared with a clinical point cloud from a clinical image. When the two sets of point clouds are successfully registered, an error metric or pass / fail indicator is provided in box 640, where the registration is considered successful if the error metric is within a predetermined range. In addition to providing an error metric, or as an alternative, a pass / fail indicator, such as a visual or auditory indicator, can be provided.

[0106] Now for reference Figure 7 The details of clipping frame 624 are shown. Clipping features can clip patient landmarks (such as the face (e.g., the root of the nose)) and the reference frame together and / or separately. These two processes are shown as sub-processes 624a and 624b. During registration, only one or both of them can be identified in the registration image and the clinical image. According to various embodiments, identifying the reference frame and separating it from the face or head can contribute to the efficiency and / or speed of registration.

[0107] In sub-procedure 624a, in block 710, a reference frame is detected. In block 712, a reference frame boundary is generated around the reference frame, large enough to incorporate the desired reference frame points. As discussed above, the boundary can be a selected or predetermined radius from the points on the reference frame. The boundary can be two-dimensional or three-dimensional. The region or volume within the boundary can then be cropped, at least in the registered image. The cropped portion can then be used in the registration process. Thus, ultimately, the reference frame is registered to the clinical image by including multiple reference frame points at predetermined distances from the reference frame.

[0108] In subprocess 624b, within box 720, the face is detected. The face can be detected by edges or predetermined features (such as the eyes, root of the nose, or nose itself), or by identifying relevant landmarks or segmenting the patient's face or head from the point cloud using a CNN-based (or transformer-based) neural network method. In box 722, facial boundaries are determined. Facial boundaries have been briefly mentioned above. Facial boundaries are generated to provide a region or volume (such as the region or volume of the patient's face) on which points (which may be referred to as head points) can be used for registration. The boundaries can originate from one or more identified landmarks, such as the root of the nose in the image. The boundaries can be selected to provide points from one or more non-moving regions of the face. For example, points in and around the chin may not be selected, as they may move during the surgical procedure. The region or volume within the boundaries can then be cropped, at least in the registered image. The cropped portion can then be used in the registration process.

[0109] Following box 722, box 724 registers faces and points within the boundary. Facial points within the boundary can be cropped for efficient and / or faster registration.

[0110] Therefore, as described above, the cropping process provided in box 624 is used in registration box 526. Cropping points can allow for a reduction in the number of points used for registration, rather than using all points set in the image. Furthermore, cropping can allow for registration of selected portions. For example, a reference frame can be cropped and used for registration without using other points in the image. Similarly, facial points can be cropped separately from other portions (e.g., the reference frame used for registration). Therefore, the registration process can be applied to a subset of points from the image, such as only the reference frame, only facial points, etc. However, those skilled in the art will understand that all or selected sets of points can be used for registration.

[0111] The systems and methods demonstrated above allow for both manual registration and automated, continuous registration with high accuracy comparable to manual registration. The entire automated registration process may take approximately 20 to 30 seconds. Because the device can be remotely controlled, the system can be activated from anywhere in the world.

[0112] Example embodiments are provided to make this disclosure thorough and to fully convey the scope to those skilled in the art. Numerous specific details, such as examples of specific components, devices, and methods, are set forth to provide a thorough understanding of embodiments of this disclosure. It will be apparent to those skilled in the art that the example embodiments may be implemented in many different forms without requiring the specific details, and that neither these specific details nor the example embodiments should be construed as limiting the scope of this disclosure. In some example embodiments, well-known processes, well-known device structures, and well-known techniques are not described in detail.

[0113] Instructions can be executed by a processor and can include software, firmware, and / or microcode, and can refer to programs, routines, functions, categories, data structures, and / or objects. Once selected input or data is received, the execution of instructions can be essentially automated, for example, using a processor. Therefore, the user may not need to provide multiple inputs for the occurrence of a process or result. The term "shared processor circuit" includes a single processor circuit that executes some or all of the code from multiple modules. The term "group processor circuit" includes processor circuits that, in combination with additional processor circuits, execute some or all of the code from one or more modules. References to multiple processor circuits include multiple processor circuits on a discrete die, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term "shared memory circuit" includes a single memory circuit that stores some or all of the code from multiple modules. The term "group memory circuit" includes memory circuits that, in combination with additional memory, store some or all of the code from one or more modules.

[0114] The apparatus and methods described in this application can be implemented, in whole or in part, by a special-purpose computer created by configuring a general-purpose computer to perform one or more specific functions implemented in a computer program. These computer programs include processor-executable instructions stored on at least one non-transitory tangible computer-readable medium. The computer program may also include or depend on stored data. These computer programs may include a basic input / output system (BIOS) that interacts with the hardware of the special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, and applications, etc.

[0115] These computer programs may include: (i) assembly code; (ii) object code generated from source code by a compiler; (iii) source code for execution by an interpreter; (iv) source code for compilation and execution by a just-in-time (JIT) compiler; and (v) descriptive text for parsing, such as HTML (Hypertext Markup Language) or XML (Extensible Markup Language). As an example only, source code may be in C, C++, C#, Objective-C, Haskell, Go, SQL, Lisp, Java®, ASP, Perl, Javascript®, HTML5, Ada, ASP (Dynamic Server Pages), Perl, Scala, Erlang, Ruby, Flash®, Visual Basic®, Lua, or Python®.

[0116] The communications may include the wireless communications described in this disclosure, which may be wholly or partially compliant with IEEE Standard 802.11-2012, IEEE Standard 802.16-2009, and / or IEEE Standard 802.20-2008. In various implementations, IEEE 802.11-2012 may be supplemented by draft IEEE Standard 802.11ac, draft IEEE Standard 802.11ad, and / or draft IEEE Standard 802.11ah.

[0117] The term 'processor' or 'module' or 'controller' can be replaced by the term 'circuit'. The term 'module' can refer to or include some of the following: application-specific integrated circuit (ASIC); digital, analog, or mixed-signal analog / digital discrete circuit; digital, analog, or mixed-signal analog / digital integrated circuit; combinational logic circuit; field-programmable gate array (FPGA); processor circuitry (shared, dedicated, or grouped) that executes code; memory circuitry (shared, dedicated, or grouped) that stores the code executed by the processor circuitry; other suitable hardware components that provide the described functionality; or some or all of the above, such as in a system-on-a-chip.

[0118] For illustrative and descriptive purposes, the foregoing description of embodiments has been provided. This description is not intended to be exhaustive or limiting of the invention. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but are interchangeable where applicable and can be used in selected embodiments, even if not specifically shown or described. It can also be varied in many ways. Such variations should not be considered as departing from the invention, and all such modifications are intended to be included within the scope of the invention.

Claims

1. A method comprising: Receive the patient's registered images from the registration device; At least one of a head point and a reference frame point is detected from the image using facial recognition. The patient's registration data is generated based on the head points and the reference frame points; Segmenting clinical images to generate clinical head points; The registration data is compared with the clinical images; and The comparison of the registration data with the clinical head point enables the physical space to be registered to the image space of the clinical image.

2. The method as described in claim 1, wherein, Generating the registration data includes detecting the head point in the registration image and generating the registration data within the first boundary of the reference frame and the second boundary of the predetermined head point.

3. The method of claim 2, further comprising determining the first boundary by generating the first boundary at a first predetermined distance from the reference frame.

4. The method of claim 1, wherein, Generating the registration data includes generating the nasal root point and the reference frame point.

5. The method of claim 1, wherein, Generating the registration data includes generating registration data from the registration device mounted on the movable arm.

6. The method of claim 1, wherein, Generating the registration data includes generating a point cloud of registration points from data collected using the registration device.

7. The method of claim 1, wherein, Generating the registration data includes generating a final registration point cloud formed from point clouds generated from multiple registration images.

8. The method of claim 1, wherein, Generating the registration data includes generating a final registration point cloud, which is formed by point clouds generated from multiple stitched registration images.

9. The method of claim 1, wherein, The detection of head points includes generating the root of the nose point and at least one of the eye point, tip of the nose point, eyebrow point, or reference frame.

10. The method of claim 1, wherein, Generating the registration data involves using a trained classifier to generate the registration data.

11. The method of claim 1, wherein, Receiving the registered image includes receiving a video stream from the registration device.

12. The method of claim 1, further comprising receiving the clinical image, wherein the clinical image is at least one of a computed tomography image or a magnetic resonance image.

13. A system comprising: A registration device that generates a registration image of the patient; The controller executes the following instructions: Facial recognition is used to detect at least one of a head point and a reference frame point from the image. The patient's registration data is generated based on the head points and the reference frame points. Segmenting clinical images to generate clinical head points, and The registration data is compared with the clinical images; and The comparison of the registration data with the clinical head point image enables the physical space to be registered to the image space of the clinical image.

14. The system of claim 13, wherein, The controller generates the registration data by detecting the head point in the registration image and generating registration data within the first boundary of the reference frame and the second boundary of the predetermined head point.

15. The system of claim 14, wherein, The controller determines the first boundary by generating a first boundary at a first predetermined distance from the reference frame.

16. The system of claim 15, wherein, The controller determines the second boundary by generating a second boundary at a second predetermined distance from at least one of the head points.

17. The system of claim 13, wherein, The controller generates the registration data by generating the nasal root point and the reference frame point.

18. The system of claim 13, wherein, The registration data includes generating the nasal root point.

19. The system of claim 13, wherein, The registration data includes a point cloud of registration points generated from data collected using the registration device.

20. The system of claim 13, wherein, The registration data includes a final registration point cloud based on point clouds generated from multiple registration images.

21. The system of claim 13, wherein, The controller includes a trained classifier that generates the registration data, the generation of which includes detecting at least one of the head point and the reference frame point from the image using face recognition.

22. The system of claim 13, wherein, The image is based on a video stream from the registration device.

Images