Anatomy orientation detection and verification using multi-dimensional imaging

Abstract

An imaging system is disclosed and includes a memory, an orientation module, and at least one processor. The memory is configured to store orientation metadata and images of a subject. The orientation module includes at least one neural network configured to implement at least one artificial intelligence algorithm to analyze a first one or more of the images, and based on the analysis, to determine orientation of at least one anatomical object of the subject. The at least one processor is configured, based on the determined orientation, to at least one of verify the orientation metadata, correct the orientation metadata, analyze a second one or more images, and track at least one of a tool and an implant relative to the patient.

Description

ANATOMY ORIENTATION DETECTION AND VERIFICATION USING MULTI¬DIMENSIONAL IMAGINGFIELD

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 609,453, filed 13 December 2023, the entire content of which is incorporated herein by reference.

[0002] The subject disclosure relates generally to systems for determining orientation of patient anatomy within an imaging volume.BACKGROUND

[0003] This section provides background information related to the present disclosure which is not necessarily prior art.

[0004] Imaging technologies that produce highly detailed two-, three-, and fourdimensional images include computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopic imaging (such as with a C-arm device), positron emission tomography (PET), and ultrasound imaging (US). A subject is scanned and each scan can include the collection of hundreds to thousands of cross-sectional images (or slices) of the subject. The cross-sectional images are stored and can be combined to create 3D images of anatomy of the subject. The images can be, for example, analyzed to detect conditions of the subject. Image guided medical and surgical procedures can also utilize the stated images, which are obtained prior to or during a medical procedure to guide a physician performing the procedure.

[0005] During a navigated procedure, images are acquired by a suitable imaging device for display on a workstation. The navigation system tracks the patient, instruments and other devices in the surgical field and / or patient space. These tracked devices are then displayed relative to the image data on the workstation in an image space. In order to track the patient, instruments, and other devices, the patient, instruments and other devices can be equipped with tracking devices. For example, a tracking device can be coupled to an exterior surface of an instrument, and can provide the surgeon, via a corresponding tracking system, an accurate depiction of the location of that instrument in the patient space.SUMMARY

[0006] This section provides a general summary of the disclosure, and is not a comprehensive disclosure of its full scope or all of its features.

[0007] An imaging system is disclosed and includes a memory, an orientation module, and at least one processor. The memory is configured to store orientation metadata and images of a subject. The orientation module includes at least one neural network configured to implement at least one artificial intelligence algorithm to analyze a first one or more of the images, and based on the analysis, to determine orientation of at least one anatomical object of the subject. The at least one processor is configured, based on the determined orientation, to at least one of verify the orientation metadata, correct the orientation metadata, analyze a second one or more images, and track at least one of a tool and an implant relative to the patient.

[0008] In other features, the determined orientation is relative to at least one of a reference point and a structure supporting at least a portion of the subject.

[0009] In other features, the at least one artificial intelligence algorithm includes at least one of a machine learning algorithm and a deep learning algorithm. In other features, the at least one artificial intelligence algorithm includes at least one of a dense network, an inception network, a vision transformer network, and a fully convolutional network.

[0010] In other features, the orientation module is configured to implement the at least one artificial intelligence algorithm to detect at least one landmark of the at least one anatomical object, and based on the at least one landmark, determine the orientation of the at least one anatomical object.

[0011] In other features, the orientation module is configured to implement the at least one artificial intelligence algorithm to identify the at least one anatomical object, and based on the identification, determine the orientation of the at least one anatomical object. In other features, the at least one artificial intelligence algorithm includes at least one of a machine learning algorithm and a deep learning algorithm. In other features, the at least one artificial intelligence algorithm includes at least one of a dense network, an inception network, a vision transformer network, and a three- dimensional fully convolutional network.

[0012] In other features, the orientation module is configured to implement the at least one artificial intelligence algorithm to determine directly from the images the orientation of the at least one anatomical object.

[0013] In other features, the orientation module is configured to compare the determined orientation to the orientation metadata, and, in response to determiningthat an orientation mismatch exists between the determined orientation and the orientation metadata, correct the orientation metadata.

[0014] In other features, the orientation module is configured to compare the determined orientation to the orientation metadata, and in response to determining that an orientation mismatch exists between the determined orientation and the orientation metadata, indicate a mismatch has been detected via a user interface and wait for approval to change the orientation metadata.

[0015] In other features, the orientation module is configured to determine a confidence level in the determined orientation, and based on the confidence level, correct the orientation metadata in response to determining that an orientation mismatch exists between the determined orientation and the orientation metadata.

[0016] In other features, the at least one processor is configured, based on the determined orientation, to verify the orientation metadata. In other features, the at least one processor is configured, based on the determined orientation, to correct the orientation metadata.

[0017] In other features, the at least one processor is configured, based on the determined orientation, to analyze the second one or more images.

[0018] In other features, the at least one processor is configured, based on the determined orientation, to track the at least one of the tool and the implant relative to the patient.

[0019] In other features, the processor is configured to analyze the first one or more images having a first resolution to determine orientation of the at least oneanatomical object, and analyze the second one or more images having a second resolution based on the determined orientation. The first resolution is less than the second resolution.

[0020] In other features, the first one or more images include a three-dimensional image. In other features, the first one or more images include a two- dimensional image.

[0021] In other features, the orientation module is configured to: at least one of capture and access a first slice in a sagittal view of the at least one anatomical object; at least one of capture and access a second slice in an axial view of the at least one anatomical object; at least one of capture and access a third slice in a coronal view of the at least one anatomical object; and determine the orientation of the at least one anatomical object based on the first slice, the second slice and the third slice. In other features, the first slice, the second slice and the third slice are middle slices.

[0022] In other features, the orientation module is configured to: at least one of capture and access a first maximum intensity projection image in a sagittal view of the at least one anatomical object; at least one of capture and access a second maximum intensity projection image in an axial of the at least one anatomical object; at least one of capture and access a third maximum intensity projection image in a coronal view of the at least one anatomical object; and determine the orientation of the at least one anatomical object based on the first maximum intensity projection image, the second maximum intensity projection image and the third maximum intensity projection image.

[0023] Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are intendedfor purposes of illustration only and are not intended to limit the scope of the present disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.

[0025] Fig. 1 is an environmental view of an example navigation system including an imaging system including an image processor having an orientation module in accordance with the present disclosure;

[0026] Fig. 2 is a functional block diagram of an example control system including an orientation module in accordance with the present disclosure;

[0027] FIG. 3 is a functional block diagram of an example processor including an orientation module in accordance with the present disclosure;

[0028] FIG. 4 is an example diagram for determining orientation of a head of a subject in a first position using an orientation module including an artificial intelligence (Al) neural network implementing at least one of an Al algorithm and a machine learning algorithm in accordance with the present disclosure;

[0029] FIG. 5 is another example diagram for determining orientation of a head of a subject in a second position directly from image data using an orientation module including an Al neural network implementing at least one of an Al algorithm and a machine learning algorithm in accordance with the present disclosure;

[0030] FIG. 6 is an example diagram for determining landmarks of a head of a subject for orientation determination purposes using a landmark module in accordance with the present disclosure;

[0031] FIG. 7 is an example diagram for determining anatomy of a subject for orientation determination purposes using an anatomy module in accordance with the present disclosure;

[0032] FIG. 8 is a 3D representative view of a skull of a subject illustrating an example based on which orientation of the head of the subject may be determined in accordance with the present disclosure;

[0033] FIG. 9 is a 3D representative view of a skull of a subject illustrating an example of 2D slices that may be used to determine orientation of the head of the subject in accordance with the present disclosure;

[0034] FIGs. 10A and 10B (collectively FIG. 10) illustrates an imaging process including orientation verification and correction in accordance with the present disclosure;

[0035] FIG. 11 illustrates an example training process in accordance with the present disclosure;

[0036] FIG. 12 illustrates an example segmentation method in accordance with the present disclosure;

[0037] FIG. 13 illustrates an example method of operating a surgical navigation system in accordance with the present disclosure; and

[0038] FIG. 14 is a cross-sectional image that may be used for determining anatomy of a subject for orientation determination in accordance with the present disclosure.

[0039] Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.DETAILED DESCRIPTION

[0040] Example embodiments will now be described more fully with reference to the accompanying drawings.

[0041] Multi-dimensional imaging (e.g., CT imaging, MRI imaging, etc.) is used by a navigation system (e.g., the commercially available StealthStation® ENT navigation system sold by Medtronic Navigation, Inc. of Littleton, MA) for surgery. In order to obtain correct registration, the orientation in which an image is taken is needed. Orientation metadata is usually provided by a technician and entered in a browser (e.g., the DICOM® browser). The orientation metadata may be entered incorrectly. For example, a patient position may be accidentally entered as prone instead of supine, or may be entered as facing left instead of facing right. Wrong orientation metadata may present inaccuracies for image registration, image analysis, and navigation.

[0042] The examples set forth herein include an imaging system or processor that is configured to determine orientation of an anatomy of a subject based on captured multi-dimensional images. This orientation information may be relied upon and / or used for verification purposes. The orientation information may, for example, becompared with orientation metadata provided by a technician to verify and / or correct the orientation metadata. The orientation information may be used in replacement of orientation metadata from a technician and used for touchless registration purposes. The orientation information allows a processing system to determine a direction that an anatomical feature and / or object is facing in images.

[0043] The examples include use of Al algorithms to automatically detect orientations of anatomical objects (e.g., a head, vertebrae, etc.) based on collected images. This includes determining whether: the anatomical object is in a prone or supine position; whether the anatomical object is facing left or right; whether the anatomical object is superior or inferior; etc. This is determined directly and / or indirectly from the collected image without need for orientation metadata. The automatic determination of orientation of anatomical objects may be compared to orientation metadata to alert a technician to the possibility of wrong orientation metadata when there is a mismatch. The technician can then accept correction of the orientation metadata.

[0044] The examples include neural networks that receive images having different levels of resolution. Lower resolution images may be analyzed to minimize storage requirements and / or processing requirements and time involved. In an embodiment, a low-resolution version of a full 3D image is received and analyzed and the orientation, such as Left, Posterior, Superior (LPS), Right, Anterior, Inferior (RAI), etc., is provided as an output. In another embodiment, one or more slices of a 3D image are analyzed, for example middle slices in sagittal, axial and coronal views, to determine the orientation. In yet another embodiment, maximum intensity projection(MIP) images are analyzed in sagittal, axial and coronal views to determine orientation. MIP includes projecting a voxel with a highest attenuation value of each view onto a 2D image.

[0045] Fig. 1 shows an example navigation system 10 including an imaging system 11 including an image processor 12 having an orientation module 13. The orientation module 13 automatically determines orientation of the anatomy of a subject based on collected images of the anatomy of the subject. The orientation module 13 may be implemented by the image processor 12 or by another processor, such as navigation processor 14. An example of the orientation module 13 is shown in FIG. 3. The orientation module 13 may be configured and operated as any of the orientation modules referred to herein. Operation of the orientation module 13 is further described below at least with respect to FIGs. 2-13.

[0046] The navigation system 10 may be used for various purposes or procedures by one or more users, such as a user 15. The navigation system 10 may be used to determine or track a position of an instrument 16 in a volume. The position may include both a three-dimensional X, Y, Z location and orientation. Orientation may include one or more degree of freedom, such as three degrees of freedom. It is understood, however, that any appropriate degree of freedom position information, such as less than six-degree of freedom position information, may be determined and / or presented to the user 15.

[0047] Tracking the position of the instrument (or tool) 16 may assist the user 15 in determining a position of the instrument 16, even if the instrument 16 is not directly viewable by the user 15. Various procedures may block the view of the user 15, suchas performing a repair or assembling an inanimate system, such as a robotic system, assembling portions of an airframe or an automobile, or the like. Various other procedures may include a surgical procedure, such as performing a spinal procedure, neurological procedure, positioning a deep brain simulation probe, or other surgical procedures on a living subject. In various embodiments, for example, the living subject may be a human subject 20 and the procedure may be performed on the human subject 20. It is understood, however, that the instrument 16 may be tracked and / or navigated relative to any subject for any appropriate procedure. T racking or navigating an instrument for a procedure, such as a surgical procedure, on a human or living subject is merely exemplary.

[0048] Nevertheless, in various embodiments, the surgical navigation system 10, as discussed further herein, may incorporate various portions or systems, such as those disclosed in U.S. Pat. Nos. RE44,305; 7,697,972; 8,644,907; and 8,842,893; and U.S. Pat. App. Pub. No. 2004 / 0199072, all incorporated herein by reference. Various components that may be used with or as a component of the surgical navigation system 10 may include an imaging system 11 that is operable to image the subject 20, such as an O-arm® imaging system, magnetic resonance imaging (MRI) system, computed tomography system, etc. A subject support 26 may be used to support or hold the subject 20 during imaging and / or during a procedure. The same or different supports may be used for different portions of a procedure.

[0049] In various embodiments, the imaging system 11 may include a source 24s.The source may emit and / or generate X-rays. The X-rays may form a cone 24c, such as in a cone beam, that impinge on the subject 20. Some of the X-rays pass thoughand some are attenuated by the subject 20. The imaging system 24 may further include a detector 24d to detect the X-rays that are not completely attenuated, or blocked, by the subject 20. Thus, the image data may include X-ray image data. Further, the image data may be two-dimensional (2D) image data.

[0050] Image data may be acquired, such as with one or more of the imaging systems discussed above, during a surgical procedure or acquired prior to a surgical procedure for displaying an image 30 on a display device 32. In various embodiments, the acquired image data may also be used to form or reconstruct selected types of image data, such as three-dimensional volumes, even if the image data is 2D image data. The instrument 16 may be tracked in a trackable volume or a navigational volume by one or more tracking systems. Tracking systems may include one or more tracking systems that operate in an identical manner or more and / or different manner or mode. For example, the tracking system 44 may include an electro-magnetic (EM) localizer 40, as illustrated in Fig. 1. In various embodiments, it is understood by one skilled in the art, that other appropriate tracking systems may be used including optical, radar, ultrasonic, etc. The discussion herein of the EM localizer 40 and tracking system is merely exemplary of tracking systems operable with the navigation system 10. The position of the instrument 16 may be tracked in the tracking volume relative to the subject 20 and then illustrated as a graphical representation, also referred to as an icon, 16i with the display device 32. In various embodiments, the icon 16i may be superimposed on the image 30 and / or adjacent to the image 30. As discussed herein, the navigation system 10 may incorporate the display device 32 andoperate to render the image 30 from selected image data, display the image 30, determine the position of the instrument 16, determine the position of the icon 16i, etc.

[0051] The EM localizer 40 is operable to generate electro-magnetic fields with a transmitting coil array (TCA) 42 which is incorporated into the EM localizer 40. The TCA 42 may include one or more coil groupings or arrays. In various embodiments, more than one group is included and each of the groupings may include three coils, also referred to as trios or triplets. The coils may be powered to generate or form an electro-magnetic field by driving current through the coils of the coil groupings. As the current is driven through the coils, the electro- magnetic fields generated will extend away from the coils of the TCA 42 and form a navigation domain or volume 50, such as encompassing all or a portion of a head 20h, spinal vertebrae 20v, or other appropriate portion. The coils may be powered through a TCA controller and / or power supply 52. It is understood, however, that more than one of the EM localizers 40 may be provided and each may be placed at different and selected locations.

[0052] The navigation domain or volume 50 generally defines a navigation space or patient space. As is generally understood in the art, the instrument 16, such as a drill, lead, etc., may be tracked in the navigation space that is defined by a navigation domain relative to a patient or subject 20 with an instrument tracking device 56. For example, the instrument 16 may be freely moveable, such as by the user 15, relative to a dynamic reference frame (DRF) or patient reference frame tracker 60 that is fixed relative to the subject 20. Both the tracking devices 56, 60 may include tracking portions that are tracking with appropriate tracking systems, such as sensing coils (e.g., conductive material formed or placed in a coil) that senses and are used tomeasure a magnetic field strength, optical reflectors, ultrasonic emitters, etc. Due to the instrument tracking device 56 connected or associated with the instrument 16, relative to the DRF 60, the navigation system 10 may be used to determine the position of the instrument 16 relative to the DRF 60.

[0053] The navigation volume or patient space may be registered to an image space defined by the image 30 of the subject 20 and the icon 16i representing the instrument 16 may be illustrated at a navigated (e.g., determined) and tracked position with the display device 32, such as superimposed on the image 30. Registration of the patient space to the image space and determining a position of a tracking device, such as with the instrument tracking device 56, relative to a DRF, such as the DRF 60, may be performed as generally known in the art, including as disclosed in U.S. Pat. Nos. RE44,305; 7,697,972; 8,644,907; and 8,842,893; and U.S. Pat. App. Pub. No. 2004 / 0199072, all incorporated herein by reference.

[0054] The navigation system 10 may further include a navigation processor system 66. The navigation processor system 66 may include the display device 32, the TCA 40, the TCA controller and / or power supply 52, and other portions and / or connections thereto. For example, a wire connection may be provided between the TCA controller and / or power supply 52 and a navigation processor 14. Further, the navigation processor system 66 may have one or more user control inputs, such as a keyboard 72 (or other user interface), and / or have additional inputs such as from communication with one or more memory systems such as navigation memory 74, either integrated or via a communication system. The navigation processor system 66 may, according to various embodiments include those disclosed in U.S. Pat. Nos.RE44,305; 7,697,972; 8,644,907; and 8,842,893; and U.S. Pat. App. Pub. No. 2004 / 0199072, all incorporated herein by reference, or may also include the commercially available StealthStation® or Fusion™ surgical navigation systems sold by Medtronic Navigation, Inc. of Littleton, MA.

[0055] Tracking information, including information regarding the magnetic fields sensed with the tracking devices 56, 60, may be delivered via a communication system, such as the TCA controller and / or power supply 52, which also may be a tracking device controller, to the navigation processor system 66 including the navigation processor 14. Thus, the tracked position of the instrument 16 may be illustrated as the icon 16i relative to the image 30. Various other memory and processing systems may also be provided with and / or in communication with the navigation processor system 66, including the navigation memory 74 that is in communication with the navigation processor 14 and / or an image processor 12.

[0056] The image processor 12 may be incorporated into the imaging system 11 , such as the O-arm® imaging system, as discussed above. The imaging system 11 may, therefore, include various portions such as a source and an X-ray detector that are moveable within a gantry 78. The imaging system 11 may also be tracked with a tracking device 80. It is understood, however, that the imaging system 11 need not be present while tracking the tracking devices, including the instrument tracking device 56. Also, the imaging system 11 may be any appropriate imaging system including a MR I, CT, etc.

[0057] In various embodiments, the tracking system may include an optical localizer 82. The optical localizer 82 may include one or more cameras that view orhave a field of view that defines or encompasses the navigation volume 50. The optical localizer 82 may receive light (e.g., infrared or ultraviolet) input to determine a position or track the tracking device, such as the instrument tracking device 56. It is understood that the optical localizer 82 may be used in conjunction with and / or alternatively to the EM localizer 40 for tracking the instrument 16.

[0058] Information from all of the tracking devices may be communicated to the navigation processor 14 for determining a position of the tracked portions relative to each other and / or for localizing the instrument 16 relative to the image 30. The imaging system 11 may be used to acquire image data to generate or produce the image 30 of the subject 20. In an embodiment, the orientation information determined by the orientation module 13 is used for registration purposes and as a basis on which tracking is conducted, such as tracking of the instrument 16. The TCA controller 52 may be used to operate and power the EM localizer 40, as discussed above.

[0059] The image 30 that is displayed with the display device 32 may be based upon image data that is acquired of the subject 20 in various manners and the orientation information determined by the orientation module 13. For example, the imaging system 11 may be used to acquire image data that is used to generate the image 30. The orientation information may be used when orienting the displayed image of the subject 20 on the display device 32 and / or relative to the instrument 16. It is understood, however, that other appropriate imaging systems may be used to generate the image 30 using image data acquired with the selected imaging system. Imaging systems may include magnetic resonance imagers, computed tomography imagers, and other appropriate imaging systems. Further the image data acquiredmay be two dimensional or three-dimensional data and may have a time varying component, such as imaging the patient during a heart rhythm and / or breathing cycle.

[0060] In various embodiments, the image data is a 2D image data that is generated with a cone beam. The cone beam that is used to generate the 2D image data may be part of an imaging system, such as the O-arm® imaging system. The 2D image data may then be used to reconstruct a 3D image or model of the imaged subject, such as the patient 20. The reconstructed 3D image and / or an image based on the 2D image data may be displayed. Thus, it is understood by one skilled in the art that the image 30 may be generated using the selected image data.

[0061] Further, the icon 16i, determined as a tracked position of the instrument 16, may be displayed on the display device 32 relative to the image 30. In addition, the image 30 may be segmented, for various purposes, including those discussed further herein. Segmentation of the image 30 may be used to determine and / or delineate objects or portions in the image. The delineation may include or be made as a mask that is represented on a display. The representation may be shown on the display such as with a graphical overlay of a mask, which may also be referred to as an icon. The icon may be a segmented mask. In various embodiments, the delineation may be used to identify boundaries of various portions within the image 30, such as boundaries of one or more structures of the patient that is imaged, such as the vertebrae 20v. Accordingly, the image 30 may include an image of one or more of the vertebrae 20v, such as a first vertebrae 20vi and a second vertebrae 20vii. As discussed further herein, the vertebrae, such as the first and second vertebrae 20vi, 20vii may be delineated in the image which may include and / or assist in determiningboundaries in images, such as 3D and 2D images. In various embodiments, the delineation may be represented such as with an icon 20vi' or a second icon 20vii'. The boundaries 20vi', 20vii' may be determined in an appropriate manner and for various purposes, as also discussed further herein. Further, the icon may be used to represent, for display, a selected item, as discussed herein, including the delineation of the object, boundary, etc.

[0062] According to various embodiments, the image 30 may be segmented in a substantially automatic manner. In various embodiments, the automatic segmentation may be incorporated into a neural network, such as a convolutional neural network (CNN). The CNN may learn various features such as objects (e.g., vertebrae) or parts or portions of objects (e.g., pedicle), and segmentations or boundaries of these objects and / or portions thereof. The selected segmentations may include identifying a segmentation of selected vertebrae, such as the first vertebrae 20vi and the second vertebrae 20vii. The selected segmentation may be displayed with a selected graphical representation such as a segmentation icon or representation 20vi' and 20vii' for display on the display device 32.

[0063] The icons are displayed alone on the display device 32 and / or superimposed on the image 30 for viewing by a selected user, such as the user 15 which may be a surgeon or other appropriate clinician. Moreover, once identified, the boundaries or other appropriate portion, whether displayed as icons or not, may be used for various purposes. The boundaries may identify a physical dimension of the vertebrae, positions of the vertebrae in space (i.e., due to registration of the image 30 to the subject 20 as discussed above), possible identified trajectories (e.g., forimplantation placement), or the like. Therefore, the image 30 may be used in planning and / or performing a procedure whether the icons 20vi', 20vii' are displayed or the geometry of the boundaries is only determined and not displayed as an icon.

[0064] Fig. 2 shows a control system 200, which may be a tracking control system, and may be used as part of the navigation system 10 of FIG. 1 , such as the image processor 12 and / or the navigation processor 14. The tracking control system 200 includes a controller 201 , which may include a processor 202 implementing a gantry control module 204, a source module 206, a detector module 208, an image capture module 210, an EM transmission module 212, and a tracking module 214, a navigation control module 216 and a degradation module 218. The controller 201 may also implement a browsing module 217 (e.g., a DICOM® browser application) and / or an orientation module 219, which may be similar as the orientation module 13 of FIG. 1. The browsing module 217 may request orientation information from a technician via a user interface such as one of the user interfaces 72, 100 of FIG. 1 and store the orientation information as orientation metadata in memory of the processor 202 and / or elsewhere.

[0065] The orientation information may include indications as to whether the subject and / or anatomical objects thereof are lying prone or supine, whether the subject and / or an anatomical object thereof is facing left or right; whether an anatomical object is oriented in superior, inferior, anterior, and / or posterior directions. The orientation information may be provided relative to a patient coordinate system and / or other coordinate system. Similar information may be determined by the orientation module 219. In an embodiment, the orientation module 219 determines avector for the facing direction of the anatomical object. In an alternative embodiment, the controller 201 is in communication with the image processor 12 of FIG. 1 and may receive orientation information from the orientation module 13.

[0066] The gantry control module 204 may control positioning and determine positions of the gantry of the navigation system 10 of FIG. 1. The gantry control module 204 may control a gantry motor 220. The source module 206 may control an X-ray source and / or positioning thereof including controlling one or more source motors 222 of a source actuator and motor assembly 224. The detector module 208 may control operation of an X-ray detector and / or positioning thereof including controlling one or more detector motors 226 of a detector actuator and motor assembly 228. The image capture module 210 may control capturing of images of a patient and / or space surrounding the patient. The EM transmission module 212 may control generation and emission of electromagnetic signals, via EM coils, such as the EM coils referred to herein. The EM transmission module 212 may supply current at selected frequencies and / or respective signatures to EM coils in, for example, the instrument 16 or localizer 40 of FIG. 1 .

[0067] The tracking module 214 may determine locations of tools and / or instruments and / or portions thereof based on received EM signals detected by EM coils on tools, instruments, localizers and / or tracking devices referred to herein. The localizers and the tracking devices may each include EM coils. The EM coils of the tools, instruments, localizers and tracking devices may each operate as an emitter or as a receiver of EM signals. The EM signals are received and processed via the tracking module 214. Tracking data associated with the EM signals may be stored inmemory of the processor 202 or in other memory separate from the processor 202. The tracking data can include data indicating and / or indicative of the locations of a tip of a tool, a distal end of the tool, and / or a distal end of the corresponding instrument housing (or attachment).

[0068] When the EM coils of the tool or instrument are activated (i.e., emitting EM signals), the controller 201 can receive EM signals and / or sensor data, via a cable, indicative of the EM signals generated by or received by the EM coils. Based on the sensor data, the tracking system 44 of FIG. 1 and / or the navigation control module 216 of the navigation system 10 can display locations of the tip of the tool, the distal end of the tool and / or the distal end of the instrument housing.

[0069] The navigation control module 216 can receive the tracking data from the tracking system 44 as input. The navigation control module 216 can also receive patient image data as input. The patient image data can comprise images of the anatomy of a patient obtained from a pre- or intra-operative imaging device, such as the images obtained by the imaging system 11 of FIG. 1. Based on the tracking data and the patient image data, the navigation control module 216 can generate image data for display on the display device 32 of FIG. 1 . The image data can comprise the patient image data superimposed with an icon of the tool and / or instrument. The icon can provide a graphical representation of the position, orientation and trajectory of the tip of the tool, a distal end of the tool, and / or a distal end of the instrument housing relative to the anatomy of the patient. In addition, the icon can illustrate a starting point of the tool, (e.g., an "X" adjacent to a bone in the anatomy) and can illustrate the trajectory of the tool through the anatomy, (i.e., dashes from the "X"). A currentlocation of the tool can also be displayed by the icon (i.e. , an "0" at the end of the dashes). It should be understood, however, that any suitable symbol, indicia or the like could be employed to graphically represent the location and / or trajectory of the tip of the tool relative to the anatomy.

[0070] The tracking module 214 can receive as an input start-up data from the navigation control module 216 and sensor data from EM coils of tools, instruments, localizers, and / or tracking devices. The navigation control module 216 can receive as input tracking data and patient image data. The tracking data can be indicative of locations of portions of the tools, instruments, localizers, and / or tracking devices in patient space. Based on the tracking data, the navigation control module 216 determines the appropriate patient image data for display on the display device 32, and outputs both the tracking data and the patient image data together as image data.

[0071] The degradation module 218, in response to the tracking module 214 detecting the movement of a distal end of an instrument (e.g., a distal end of instrument 230, a distal end of a tool of the instrument, or distal end of other instrument or tool referred to herein), can detect tool instability and / or breakage for tool and / or instrument replacement. The degradation module 218 actively monitors locations of tool tips (or distal ends of tools) and based thereon actively adjusts speed, torque, stability, dampening, and / or feed rate of the tool via a corresponding motor (e.g., motor 232 or other motor referred to herein) and / or one or more actuators (e.g., actuators 234 or other actuators disclosed herein). The actuators 234 may include attachment blocks, shafts, links, etc. This can occur during a procedure, such as while resecting tissue and / or bone.

[0072] FIG. 3 shows an example processor 300, which may represent a portion of one of the processors 12 and 202 of FIGs. 1 -2. The processor 300 includes an orientation module 302, which may operate similarly as the orientation modules 13, 219 of FIGs. 1 -2. The orientation module 302 may include a feature and pattern module 304, a landmark module 306, an anatomy module 308, and an orientation verification module 310. Each of the modules 302, 304, 306, 308, 310 may be implemented as a neural network and / or implement a machine learning algorithm to learn features, patterns, landmarks, anatomical features, etc. based on image data provided. In an embodiment, image data along with correct features, patterns, landmarks, anatomical features, and / or orientation information is provided to the modules 302, 304, 306, 308 and / or 310 to allow these modules to learn relationships between image data and the correct features, patterns, landmarks, anatomical features, orientation information. Landmarks may refer to distinct features of an anatomical object, such as: a nose or ears of a head; pedicles of vertebrae; a sacrum; first and last ribs next to T1 and T12 vertebrae, respectively; the C2 vertebra odontoid process, the aorta and vena cava; and / or other landmarks of other anatomical objects. The learning may be an ongoing process and further improved over time and usage. The neural networks may be implemented as CNNs. The orientation information once determined may be used for registration, initialization, segmentation, image analysis, merging of images, navigation, etc.

[0073] The feature and pattern module 304 analyzes images to determine features and / or patterns indicative of the orientation of the anatomical objects in the images. The features may include pixel intensities, colors, contrast ratios, etc. The patternsmay include pixel patterns, shapes of detected object features (e.g., nose, ears, etc.), brightness or intensity patterns, pixel brightness levels at certain locations, etc.

[0074] The landmark module 306 is configured to determine landmarks such as a nose (or tip of a nose) and ears (or tips of ears) of a head. The landmarks may refer to anatomical features having a particular well know set of characteristics, such as shape, size, pixel intensity pattern, etc. The anatomy module 308 is configured to identify anatomical objects and / or distinct features such as vertebral bodies, posterior elements, a sacrum, a C2 vertebra odontoid process (an example of which is shown in FIG. 14), etc. In one embodiment, after three landmarks are detected, triangulation is performed based on the three landmarks to determine orientation of the corresponding anatomical object. In another embodiment, an Al learning algorithm is implemented based on the detected landmarks to determine orientation.

[0075] The orientation verification module 310 compares orientation metadata for one or more images to orientation information determined by the orientation module 302 for the one or more images. This may include facing directions, vectors, etc. If there is not a match, the orientation verification module as further described below may indicate that there is a mismatch to a technician and / or automatically correct the orientation metadata. The correction of the orientation metadata may be based on a predetermined criterion, an example of which is described below.

[0076] The processor 300 may store and / or access data stored in a memory 320. This includes image data 322, feature and pattern data 324, landmark data 326, anatomical data 328, orientation metadata 330, and historically determined orientation data 332. The data 324, 326, 328, 330, 332 may be stored along with thecorresponding image data 322 and / or with references to the corresponding image data 322.

[0077] In an embodiment, the orientation module 302 sets imaging capturing at a low resolution for orientation determination purposes. As an example, a first one or more images may be captured at a low resolution to determine orientation of one or more anatomical objects. Once the orientation(s) are determined, the resolution may be increased for subsequent imaging. Subsequent images taken at higher resolutions may be analyzed based on the determined orientation of the one or more anatomical objects. The low-resolution images may be stored, accessed and analyzed to determine the orientation information. The orientation information may be determined using lower resolution images because fine image details are not typically needed for orientation determinations, but rather are needed for performing procedures and / or for detecting irregularities in a patient. In an embodiment, orientation determinations are accurate within a 5-10° range. In another embodiment, orientation determinations are accurate within a 10-20° range. In another embodiment, orientation determinations are accurate within a 20-30° range. The capturing of low-resolution images also minimizes the amount of memory utilized. The low-resolution for orientation determination may be multiple times less than image resolution of images used for other purposes.

[0078] In an embodiment, a low-resolution version of a full 3D image is captured and / or accessed and analyzed and the orientation, such as LPS, RAI, etc., is provided as an output. In another embodiment, one or more slices of a 3D image are analyzed, for example middle slices in sagittal, axial and coronal views are analyzed, todetermine the orientation. The middle slices may extend through a center point of an image volume and / or a center point of an anatomical object being imaged. In yet another embodiment, MIP images are analyzed in sagittal, axial and coronal views to determine orientation. Any of these operations of these example embodiments may be performed to detect and identify landmarks and anatomical objects.

[0079] Orientation information may be collected, verified and / or corrected initially, periodically, and / or subsequent to performing certain operations, analysis, procedures, etc. A patient may, for example, be told to move his or her head and thus this movement would warrant an update in the orientation of the head. This update may occur automatically or in response to a user input. As an example, one or more sensors may be included and detect movement by and / or of a patient. An example motion sensor 340 is shown and may be implemented as a camera, an infrared sensor, an ultrasonic sensor, a radar sensor, and / or other motion and / or object detection sensor. More than one motion sensor may be included and monitored. Orientation information may be determined, verified and / or updated each time movement is detected.

[0080] FIG. 4 shows a diagram for determining orientation of a head of a subject in a first position using the orientation module 302 of FIG. 3. The orientation module 302 may include a traditional software algorithm and / or an Al neural network implementing an Al algorithm and a machine (or deep) learning algorithm. As an example, the orientation module 302 may receive a set of images 400, which may be CT images of a head of a patient taken from different angles while the head is in a fixed position. The orientation module 302 analyzes the set of images and determinesthe orientation of the head. In the example shown, the head is in a left, anterior, superior (LAS) orientation.

[0081] FIG. 5 shows a diagram for determining orientation of a head of a subject in a second position directly using the orientation module 302 of FIG. 3. As an example, the orientation module 302 may receive a set of images 500, which may be CT images of a head of a patient taken from different angles while the head is in a fixed position. The orientation module 302 analyzes the set of images and determines the orientation of the head. In the example shown, the head is in a right, posterior, superior (RPS) orientation.

[0082] In an embodiment, due to symmetry of a head, the orientation module 302 detects superior / inferior and anterior / posterior directions based on the image data directly. The orientation module 302 determines whether the head is facing in the left direction or right direction based on known handedness of image axes. Right- handedness refers to a positive axis (x, y, or z) pointing toward a viewer and lefthandedness refers to a positive axis pointing away from the viewer.

[0083] FIG. 6 shows a diagram for determining landmarks of a head of a subject for orientation determination purposes using the landmark module 306 of FIG. 3. The landmark module 306 receives a set of images 600A, which may be MRI images, and detects landmarks. In the example shown, a tip 602 of a nose and tips 604, 606 of ears of a head 608 are detected. The orientation module 302 may determine the orientation of the head based on the detected landmarks. This may include identification of a left ear versus a right ear. This may be based on the locations of the landmarks, the type of landmarks detected, the shape and / or orientations of thelandmarks, etc. The landmarks may be detected in one or more images. The same landmark may be detected in multiple images. Multiple detections of the same landmark may be used to increase a confidence level in identification and / or orientation of that landmark. The tips of landmarks may be designated as points in the images 600B. Images 600B are modified versions of images 600A.

[0084] FIG. 7 shows a diagram for determining anatomy of a subject for orientation determination purposes using the anatomy module 308 of FIG. 3. The anatomy module 308 may receive a set of images 700 of a portion of a back 701 of a subject. As an example, the images may be CT images of a spinal area of the subject. In the example shown, the set of images 700 include images of a portion of a spinal region of the subject. The anatomy module 308 identifies vertebral body (or vertebrae) 702, posterior elements 704, a sacrum 706, and / or other anatomical objects such as the C2 vertebra odontoid process (example of which shown in FIG. 14). In an embodiment, the anatomy module 308 may shade, color, and / or pattern the bodies, elements, sacrum, and / or other anatomical objects detected as shown. The posterior elements 704 include pedicles to spinous process. The positional relationship between the vertebral body 702, the posterior elements 704, the sacrum 706, and / or other anatomical features such as the C2 vertebra odontoid process indicates the orientation of the subject. For example, the posterior elements 704 are always located posterior of the vertebral body 702, the sacrum is always a lower vertebrae, and the C2 vertebra with the unique odontoid process is always a higher vertebrae. This allows the orientation module 302 to determine the anterior-posterior (AP) direction and the superior-inferior (SI) direction.

[0085] FIG. 8 shows a 3D representative view of a skull 800 of a subject in an imaging volume 802. The orientation module 302 of FIG. 3 may receive a 3D image of the skull 800 and determine the orientation of the skull 800 (or head) of the subject based on this image. In an embodiment, the orientation module 302 receives a 3D image and based on features and / or patterns of the 3D image and / or detected landmarks and / or anatomical features in the 3D image, detected in the image, determines the orientation of the skull 800. This also may be accomplished using a neural network as referred to herein.

[0086] FIG. 9 shows a 3D representative view of a skull 800 of a subject illustrating an example of 2D slices 900 that may be used to determine orientation of the head of the subject. The orientation module 302 may receive the 2D slices 900 and determine the orientation of the skull 800 based on features and / or patterns of the slices 900 and / or based on detected landmarks and / or anatomical features of the skull 800 in the slices 900.

[0087] FIGs. 10A-10B shows an imaging process including orientation verification and correction. The following operations may be performed by at least a browsing module (e.g., the browsing module 217 of FIG. 2) and an orientation module (e.g., one of the orientation modules 13, 219, or 302 of FIGs. 1 -3).

[0088] At 1000, orientation metadata is collected via the browsing module. This may occur prior to or subsequent to locating a subject within an imaging volume. Orientation metadata may be input by a technician and stored in a memory (e.g., the memory 320 of FIG. 3). This may be done via the display device 32 of FIG. 1 or other user interface.

[0089] The image data may include any appropriate image data such as CT image data, MRI data, X-ray cone beam image data, etc. Further, the imager may be any appropriate imager such as the O-arm® imaging system, as discussed herein or other imaging system. The O-arm® imaging system may be configured to acquire image data for 360 degrees around a subject and include 2D image data and / or a 3D reconstruction to provide 3D images based on the 2D image data. Further, the 0- arm® imaging system may generate images with an X-ray cone beam. The 2D image data or the reconstructed 3D image data may be from an imaging system such as the imaging system 24, which may include the O-arm® imaging system. The imaging system 24 may generate two- dimensional image data in the form of slices that may be used to reconstruct a three-dimensional model of one or more anatomical objects (e.g., head, vertebrae, etc.) of the subject. The input image data may also be acquired at any appropriate time such as during a diagnostic or planning phase rather than in an operating theatre, as specifically illustrated in Fig. 1 . Nevertheless, the image data may be acquired of the subject with the imaging system 24 and may be input or accessed by the orientation module.

[0090] At 1002, the orientation module may perform an imaging process to capture one or more images and / or collect one or more images from the memory. The imaging process may include capturing one or more CT images, MRI images, and / or other images of the subject. The one or more images collected from the memory may include one or more CT images, MRI images and / or other images of the subject. At 1004, if one or more images are captured at 1002, the orientation module may storethe images along with the corresponding orientation metadata in the memory or may use the images for orientation determination purposes and then discard the images.

[0091] As an alternative to operations 1000, 1002, and / or 1004, the orientation module may access one or more images and, if available, corresponding orientation metadata stored in memory, as represented by operation 1006. The one or more images may include one or more CT images, MRI images and / or other images.

[0092] At 1008, the orientation module may analyze the one or more images with a first Al algorithm to generate image aspect data. As an example, the image aspect data may include feature data, pattern data, segment data, landmark identifications, anatomy identifications, and / or other data and / or image information referred to herein. This may include as described above use of a first neural network implementing a machine learning algorithm.

[0093] The analyzing of the data may include segmenting an image to identify one or more portions of an image such as identifying landmarks and / or anatomical objects. A neural network (or artificial neural network) may be used to automatically identify features of the portions of the images, such as pixel intensities, contrast ratios of sets of pixels (each set including one or more pixels), voxel intensities, boundaries of features, patterns, etc. This may be done to segment the image data and to focus processing on certain image areas and / or portions of an anatomical object. The artificial neural network may be a CNN. The CNN may analyze the input image data to segment selected portions of the image data.

[0094] The landmarks, identified anatomical features and / or objects, boundaries, segmented portions, etc. may be displayed on the display device 32 of FIG. 1 or otherdisplay either alone and / or in combination with the corresponding image(s). The landmarks, identified anatomical features, boundaries, segmented portions, etc. are stored in memory.

[0095] At 1010, the orientation module may determine orientation of one or more anatomical objects based on the image aspect data. This may be accomplished using the traditional software algorithm, the first Al algorithm and / or a second Al algorithm and a second neural network. A traditional software algorithm may be used to determine orientation based on the first Al algorithm's image-aspect (landmarks) data. The second neural network may be a CNN. The orientation information may include any of the orientation information referred to herein.

[0096] As an alternative to operations 1008, 1010, operation 1012 may be performed by the orientation module to directly determine orientation of one or more anatomical objects based on the one or more images using a neural network and a third Al algorithm.

[0097] During the above operations, the neural networks involved may include: dense neural networks (or neural networks with dense layers); an inception networks (or deep neural networks that consist of repeating blocks where the output of a block acts as an input to a next block); neural networks having a 3D U-Net architectures; and / or other neural networks. A U-Net is a fully convolutional network (FCN). In other embodiments, the neural networks may use a number of machine-learning algorithms. The machine-learning algorithm(s) may include one or more of a CNN algorithm, an autoencoder algorithm, a recurrent neural network (RNN) algorithm, and transformer neural network algorithms, a Swin transformer network, a vision transformer network,a generative adversarial network (GAN) algorithm, linear regression, support vector machine (SVM) algorithm, a random forest algorithm, a hidden Markov model, and / or any combination thereof. For example, in some embodiments, the at least one processor may be configured to utilize a combination of a CNN algorithm in conjunction with an SVM algorithm.

[0098] At 1014, the orientation module may compare the determined orientation(s) of the one or more anatomical objects to the orientation metadata to verify whether the orientation metadata is correct. At 1016, the orientation module determines whether the determined orientation(s) match the orientation metadata. If the determined orientation(s) do not match the orientation metadata, operation 1018 may be performed, otherwise operation 1030 may be performed.

[0099] At 1018, the orientation module may generate one or more confidence level(s) for the determined orientation(s). The confidence level may be based on the resolution of the images analyzed, whether landmarks have been detected, which landmarks have been detected, whether anatomical objects have been identified, which anatomical objects have been identified, types of features detected, etc. The confidence levels may each be a value between 0-1 , where 0 is a 0% confidence level and 1 is a 100% confidence level. As an example, when landmarks and / or anatomical objects are detected and / or identified, the confidence level may be higher than the threshold. The more landmarks detected, the higher the confidence level.

[0100] At 1020, the orientation module may determine whether the confidence level(s) are greater than a predetermined level. As an example, the predetermined threshold may be 70-90%. If more than one confidence level is determined, theconfidence levels may be i) weighted and summed, or ii) averaged. The weighted sum or the average may then be compared to the predetermined amount. If not greater than the predetermined amount, operation 1022 may be performed, otherwise operation 1024 may be performed.

[0101] At 1022, the orientation module may refrain from changing the orientation metadata and / or provide an indication of the orientation mismatch and confidence level(s) of the determined orientations.

[0102] At 1024, the orientation module may indicate that an orientation mismatch has been detected and indicate the confidence level(s) to the technician. This may be via the display device 32 of FIG. 1 or other user interface.

[0103] At 1026, the orientation module may determine whether approval has been received to change the orientation metadata. If yes operation 1028 may be performed, otherwise operation 1030 may be performed. In one embodiment, operation 1026 is not performed and the orientation metadata is automatically changed to match the determined orientation(s). In another embodiment, the orientation module waits for use input indicating whether to maintain current orientation metadata or to change the orientation metadata.

[0104] At 1028, the orientation module changes the orientation metadata to match the determined orientation(s).

[0105] At 1030, the orientation module may proceed with performing image analysis, performing a surgical procedure, and / or performing some other operation based on the determined orientation(s) and / or the orientation metadata.

[0106] FIG. 11 shows an example training process 1150. Although the following example training process includes segmentation, training may be conducted without use of segmentation. The segmentation may be used for orientation determination purposes and / or for navigation purposes. Segmentation may be performed a first time to determine orientation of one or more anatomical objects and a second time to show the one or more anatomical objects during a procedure. In an embodiment, the same segmentation results are used for both orientation and navigation. In another embodiment, segmentation is performed multiple times to provide different segmentation results for orientation determination and navigation.

[0107] The training phase process 1150 may start with an input and may include image data 1152, such as any of the image data referred to herein. The selected image data may include low, medium or high-resolution image data. In an embodiment, when general locations of landmarks are being determined, low resolution data may be used. High resolution data is not necessary when determining orientation of certain objects, such as a head. Inputs may further include a segmentation mask, such as a binary segmentation mask 1156. The segmentation mask 1156 may be a standard or training data segmentation, such as a gold standard or user determined segmentation. For example, the binary segmentation mask may include a user (e.g., trained expert, such as a surgeon) segmentation of a selected structure, such as selection of landmarks and / or anatomical features and / or objects.

[0108] After receiving the image data 1152 and using the mask 1156 selected steps may occur that included selected preprocessing. For example, an optional resizing step in block 1160 may occur to resize the image data to an appropriate orselected size. In various embodiments, voxels may be resampled to a specific resolution, such as about 1.5 mm x 1.5 mm x 1.5mm. Further preprocessing may include zero padding in block 1164. Zero padding may be used to ensure that an image size is achieved after or during a CNN process and also to ensure that selected augmentation maintains all image data within bounds of the image.

[0109] Selected augmentation may also be selectively applied to the image data in block 1168. The augmentation may be of the input data may include offline and / or online image data. Selected offline augmentation may include randomly scaling the images along a selected axis by a selected scale factor. Scale factors may include between about 0.9 and about 1.1 but may also include other appropriate scale factors. Further, images may be randomly rotated around selected axes at a selected amount. Selected amounts of rotation may include minus 10 degrees to about plus 10 degrees of rotation. Online augmentation may include randomly flipping images along different axes or transposing image channels. The augmentation in block 1168 may assist in training the CNN by providing greater variability in the input image data 1152 than provided by the image data set itself. As discussed above, and generally known in the art, the attempt is to have the CNN generate the filters that allow for automatic detection of selected features, such as segmenting boundaries of vertebrae, within the image data without additional input from a user. Therefore, the CNN may better learn or more effectively learn appropriate filters by including data that is more randomized or more highly randomized than provided by the initial image data.

[0110] The image data may then be normalized in block 1172. In normalizing the image data, the variables are standardized to have a zero mean and a unit variance.This is performed by subtracting the mean and then dividing the variables by their standard deviations.

[0111] A cropping or patch wise process may occur in block 1180. In various embodiments to achieve selected results, such as a decrease training time, reduced memory requirements, and / or finer grain detail learning, a selected cropping may occur. For example, image data of a selected size may be cropped, such as in half, to reduce the amount of image data trained at a time. A corresponding portion of the segmentation mask is also cropped and provided in the image and mask in block 1180. The cropped portions may then be combined to achieve the final output. The cropping process in block 1180 may also reduce memory requirements for analyzing and / or training with a selected image data set.

[0112] The image data, whether cropped or not from process 1180 may then be used as an input to the CNN in block 384. The CNN, as discussed above, may then determine filters to achieve the output. The output may include a probability map 1188 and a trained model 1190. The probability map 1188 is a probability of each voxel, or other selected image element, belonging to a selected labeled or delineated portion, e.g., a vertebra, a portion of a vertebra, a screw, or other selected portion in the input image. The input image may include various selectable portions, such as a vertebra, a plurality of vertebrae, a screw, etc. A threshold probability may be selected, in various embodiments, for identifying or determining that a selected image portion is a selected portion or label. It is understood, however, that a threshold is not required and that the probability map may output selected probabilities in the output for the segmentation.

[0113] The trained model 1190 includes the defined filters that may be applied as kernels K and may be based on the probability map 1188. The defined filters, as also discussed above, are used in the various layers to identify the important or significant portions of the image to allow for various purposes, such as segmentation of the image. Accordingly, the trained model 1190 may be trained based upon the input image 1152 and the binary segmentation mask 1156. The trained model may then be stored or saved, such as in a memory system including the navigation memory 74, for further access or implementation such as on the image memory 112 and / or the navigation memory 74. The training process 1150 may include various inputs, such as an amount of padding or a selected voxel size but is generally performed by a processor executing selected instructions, such as the navigation processor 14. For example, the training of the CNN in block 1184 and the training model may be substantially executed by the navigation processor 14.

[0114] It is understood, however, that the trained model may also be provided on a separate memory and / or processing system to be accessed and used at a selected time. For example, the trained model may be used during a planning phase of a procedure, and / or, during a procedure when a subject or portion thereof is in an operating theater during an implantation procedure.

[0115] FIG. 12 shows an example segmentation method 1200. The trained model 1190 from the training method 1150 may be used as an input when attempting to determine a segmentation of an image, such as an image of an anatomical object. Accordingly, image data 1202 may be input with the trained model 1190. As discussed above inputting the image data 1202 and the trained model 1190 may includeaccessing both the image data and the trained model that are stored in a selected memory, such as those discussed above, by one or more of the processor systems including one or more of the referred to processors.

[0116] The image data 1202 may be preprocessed in a manner similar to the image data preprocessed during the training method 1150. For example, the image data 1202 is preprocessed in the same manner as the trained model 1190 is trained. As discussed above, various preprocessing steps are optional and may be performed on the image data 1152 during the training phase. During the segmentation phase 1200, the image data 1202 may be or is selectively preprocessed in a similar manner. Accordingly, the image data 1202 may be resized in block 1160', zero padding may be added in block 1164', the image data may be normalized in block 1172'. It is understood that the various preprocessing steps may be selected and may be chosen during the segmentation phase 1200 if performed during the training phase 1150. The segmentation image data is the same type as the training image data.

[0117] After appropriate preprocessing is performed in blocks 360', 364', and 1172', the image data 1202 may be split or cropped, in block 1210. The splitting of the image in block 1210 is also optional and may be selected based upon processing time, memory availability, or other appropriate features. Nevertheless, the image data 1202 may be split in a selected manner, such as along selected axes. The image data may then be merged, such as in a post processing step 1214 once the segmentation has occurred.

[0118] Once the image data is preprocessed, as selected, the CNN 1184, with the learned weights and / or filters may be used to segment the image data 1202. Thesegmentation of the image data 1202 by the CNN 1184 may create an output 1220 including a probability map 1216 and a selected mask, such as a binary segmentation mask in the outputs 1222. The output 1220 may be an identification of a selected geometry of the segmented portions, such as landmarks and / or anatomical features and / or objects. The CNN 1184, having been taught or learned the selected geometry, landmarks, features, anatomical objects, and / or weights, may segment the portions of the image data.

[0119] In the outputs 1222, the probability map 1216 is a probability of each voxel, or other image element or portion belonging to a selected label or portion, such as a landmark, anatomical object such as a head, spine, vertebra, vertebrae, and / or other object such as a screw or other implanted object. The binary segmentation 1220 is produced from the probability map 1216 by selecting all the voxels or other image portions with a probability greater than a threshold. The threshold may be any selected amount, such as about 30% to about 99%, including about 35%. It is further understood, however, that a threshold may not be required for performing the binary segmentation 1220 based on the probability map 1216.

[0120] The segmentation process 1200 may include various inputs, such as an amount of padding or a selected voxel size but is generally performed by a processor system executing selected instructions, such as the navigation processor system 66. For example, the segmentation with the CNN in block 1184 and the output segmentation in block 1220 may be substantially executed by the processor system. Thus, the segmentation process 1200, or substantial portions thereof, may beperformed substantially automatically with the processor system executing selected instructions.

[0121] The output 1220 may then be stored in a selected memory. Moreover, the output 1220 may be output as a graphical representation, such as one or more icons representing the geometry of the segmented portion. As illustrated in FIG. 1 , the segmented portions may be displayed either alone or superimposed on an image. It is understood that any appropriate number of segmentations may occur, and the illustration of two vertebrae in FIG. 1 is merely exemplary. For example, the image data may be of an entire spine or all vertebrae of the subject. Accordingly, the segmentation mask may include an identification of each of the vertebrae. Moreover, it is understood that the segmentation may be a three-dimensional segmentation such that an entire three-dimensional geometry and configuration of the vertebrae may be determined in the output 1220 and used for various purposes, such as illustration on the display device 32.

[0122] The navigation system 10, as illustrated in FIG. 1 , may be used for various purposes such as performing a procedure on the subject 20. The procedure may be performed based on orientation information received, determined, verified and / or corrected herein. In various embodiments, the procedure may include positioning an implant in the subject based on the orientation of the subject, such as fixing a pedicle screw into one or more of the vertebrae 20v. In performing the procedure, the tool 16 may be an implant, such as a screw. It is understood that various preliminary steps may be required for performing or placing the implant, such as passing a cannula through soft tissue of the subject 20, drilling a hole into the vertebrae 20v, tapping ahole in the vertebrae 20v, or other appropriate procedures. It is further understood that any of the items used to perform various portions of the procedure may be the tool 16 and that the tool 16 may also be the implant. Any one of the portions (e.g., implants or tools or instruments) may be tracked with the respected tracking system, such as simultaneously or in sequence, and navigated with the navigation system 10. During navigation, the navigation system 10 may display a position of the tool 16 as the icon 16i on the display device 32. In a similar manner, other instruments may be navigated simultaneously with the instrument 16 such that the instrument 16 may include a plurality of instruments and all or one may be individually or multiply displayed on the display device 32, according to instructions such as those from the user 15.

[0123] FIG. 13 shows an example method of operating the navigation system 10 of FIG. 1 , therefore, may be used to perform a procedure on the subject 20 by the user 15. Further, the navigation processor 14 may execute instructions that are stored on selected memories for performing or assisting the user 15 in performing the procedure. The method includes various operations performed by one or more of the processors of the navigation system 10. One or more of these operations may be performed based on inputs received from the user 15.

[0124] The method may include a data acquisition or accessing operation including operating the imaging system 24, such as an O-arm® imaging system, in block 1300 to implement an image scan and acquire image data of the subject, represented by block 1302. The image data may then be accessed or received. Image data may be accessed via a processor. Similarly, as discussed further herein, the image data maybe analyzed and segmented such as with the automatic segmentation process described above.

[0125] The processor may select a procedure at block 1304. The procedure may be selected based on an input by the user 15 and / or the system based on identification of the user 15. User (or surgeon) preferences and / or operation that may be specific to the surgeon may be loaded from memory. The surgeon may have preferences that may augment one or more of the following items such as specific instruments to be prepared for a selected procedure, a size of an implant for a selected anatomical structure, or the like. In various embodiments for example, a selected surgeon may select to include an implant, such as a pedicle screw, that has a 3 mm clearance relative to a boundary of vertebrae while another surgeon may select to include a pedicle screw that has a 5 mm clearance. Accordingly, the selected surgeon having the identified preferences may be used by the processor in selecting and / or identifying instruments during navigation.

[0126] At 1306, the processor may automatically suggest an instrument set based upon either one or both of the selected procedure or the identified surgeon. Automatically suggesting an instrument set may include selecting or suggesting instrument tools, implants, or the like. For example, with regard to the placement of a pedicle screw, the processor may suggest an instrument (e.g., a probe, an awl, a driver, a drill tip, and a tap) and / or implant type and or geometry and size (e.g., a screw size and length). The suggestion of an instrument and / or implant set may be based upon a selected algorithm that accesses a database of possible procedures and identifies tools therefrom. Further, a machine learning system may be used to identifyan instrument set based upon various inputs such as the procedure and surgeon, as selected surgeons may select different instruments and / or a surgeon's preference (e.g., pedicle screw size) may vary or change a selected instrument set. Instrument selection may also be made or assisted with heuristics based on the segmentation as one of the inputs. Whether the instrument set is automatically suggested or not, instruments may be verified at 1308. The verification of the instruments may ensure that the instruments are present in an operating theater and / or inputting them into the navigation system 10. For example, the navigation system 10 may be instructed or used to identify a selected set of instruments or types of instrument.

[0127] The instruments in a navigated procedure are generally tracked using a selected tracking system. It is understood that appropriate tracking systems may be used, such as an optical or an EM tracking system as discussed above. In various embodiments, therefore, an instrument tracker may be identified in block 1310. An identification of the instrument tracker may be substantially automatic based upon the tracker being identified by the selected tracking system, such as with the optical localizer 82. For example, the optical localizer may be used to identify or "view" the tracking device, such as the instrument tracking device 56. It is understood that a plurality of instruments may have a plurality of unique trackers on each of the instruments and therefore a viewing of a selected tracker may be used to identify the tracker to the instrument. It is understood, however that trackers may be changeable and therefore an automatic detection may not be possible and therefore a manual identification of the instrument tracker may be selected.

[0128] A tip associated with a selected instrument may be automatically identified in block 1312. As discussed above, the automatic identification of the tip may be used by "viewing" the tip with the optical localizer 82. Accordingly, the processor may use a deep learning system, such as a CNN, to identify the tip relative to the instrument and / or the tracker. By identifying the tip, the process of the procedure and the user may be assisted in identifying selected features. Features may include a geometry of the tip used during navigation and displaying on the display device 32, such as with the instrument icon 16i. It is understood, however, that the tip may also be manually inputted or identified in selected procedures.

[0129] In a navigated procedure the patient 20 may also be tracked with the DRF 60. In block 1314, the DRF 60 may be placed or identified on the patient 20. It is understood that placing the DRF 60 on a patient is generally a substantially manual procedure being performed by the user 15 or at the instruction of the user 15. Nevertheless, the placement of the DRF 60 may also include identifying or tracking of the DRF 60 via the navigation process. Accordingly, the navigation process may include tracking the DRF once placed on the patient 20.

[0130] The DRF allows for registration in block 1316 to the image data input. Registration allows for a subject or physical space defined by the subject 20 to be registered to the image data such that all points in the image data are related to a physical location. Therefore, a tracked location of the instrument may be displayed on the display device 32 relative to the image 30. Further the registration may allow for image portions to be registered to the patient, such as segmented portions. Registration may include optionally receiving orientation inputs from the user 15 andstoring the orientation inputs as orientation metadata. The orientation inputs indicate orientation of one or anatomical objects of the patient 20.

[0131] At 1318, an orientation module, such as any orientation module referred to herein, may perform an orientation determination process to determine orientation of at least one anatomical object of the patient including optionally performing a first segmentation process. At 1320, the orientation module may optionally perform the orientation verification process to verify the orientation metadata. Operations 1318 and 1320 may include, for example, performing the method of FIG. 10 and / or one or more portions thereof. In one embodiment, the user does not input orientation information, orientation metadata is not stored, the processor automatically determines the orientation information based on image data, and the determined orientation information is used as a basis on which the following operations are performed.

[0132] At 1322, the processor may optionally perform a second segmentation of image data. The second segmentation of image data may segment the image data differently than the first segmentation. For example, the first segmentation may be performed to detect and / or identify landmarks and the second segmentation may be performed to identify a portion of an anatomical object where an implant is to at least partially be located.

[0133] At 1324, the second segmented portions (or segmentations) may be displayed on the display device 32. As discussed above, segmentation of selected image portions may be done via a CNN, as discussed above. In addition to or alternative to the second segmentation as discussed above, the second segmentationmay also be manually accomplished by the user 15 physically tracing, with a selected instrument, such as the tracked probe on the display device, on the image 30. Nevertheless, the auto-segmentation in the navigation process may allow the user 15 to not use surgical time or planning time to segment the vertebrae and allow for a faster and more efficient procedure. A faster and more efficient procedure is achieved by saving the surgeon time in manual interaction with the navigation system 10 including various software features thereof, e.g., by automatic selection of the correct tool projection based on the segmentation.

[0134] The second segmentations may also be displayed at 1324, including the display of segmented icons (e.g., icons 20vi' and 20vii'). The segmentation icons may be viewed by the user 15 and verified that they overlay selected vertebrae. In addition to or as a part of a verification, the image portions may also be identified and / or labeled at 1326. The labeling of the image portions may be manual, such as the user 15 selecting and labeling each vertebra in the image 30, including the segmented portions therein. The labeling and / or identification of vertebrae may also be semiautomatic such as the user 15 identifying one or less than all of the vertebrae in the image 30 and the processor labeling all of the other vertebrae relative thereto. Finally, the identification of the vertebrae and labeling thereof 1326 may be substantially automatic wherein the processor executes instructions, such as based upon the CNN, to identify selected and / or all of the vertebrae in the image 30 and display labels therefore relative to the segmented portions, such as the segmentation icons 20vi' and20vii'.

[0135] The navigation process, either during a procedure or during a planning phase, may also automatically select an implant parameter, such as a size (e.g., length and width), at 1328. As discussed above, the vertebrae may be segmented according to selected procedures. Upon segmenting the vertebrae, the dimensions of the vertebrae may be known, including a three-dimensional geometry, including size and shape. This may assist in selecting a size of an implant based upon a segmented size or determined size of the vertebrae. Also, based upon the selected surgeon preferences, a size of a vertebra relative to an implant may also be known and therefore will also assist in automatically selecting the implant parameters to a specific size. The size may be output, such as on the display device 32 for selection and / or confirmation by the user 15. Accordingly, selecting the implant parameters, including size or other geometry, may be made by the processor.

[0136] The procedure may include assistance in preparing and / or placing a selected implant. To place an implant, such as a pedicle screw, an entry point into the patient 20 may be determined relative to an anatomical object. The instrument 16 may include a probe with the instrument tracking device 56 (e.g., an optical tracker). The probe may be moved relative to the subject 20, such as without piercing the subject 20.

[0137] In attempting to determine an entry point at 1330 the probe may be moved relative to the anatomical object. The anatomical object, having been identified and / or labeled at 1326, may be identified based upon a projection from the probe, such as from a tracked distal end of the probe. The probe end need not puncture a soft tissue, such as a skin, of the subject 20 but rather the projection may be determined and / ordisplayed, such as with the instrument icon 16i on the display device 32. The instrument icon 16i may change based upon the selected instrument and may be displayed as a projection of the probe or just a trajectory based upon the position of the probe relative to the anatomical object (e.g., the vertebrae 20v). Based upon the projection of the probe the anatomical object may be identified in an image on the display device 32. The display device 32 may display the image in various manners, such as in a medial and axial view.

[0138] The projection of the instrument icon 16i may be based upon a boundary of the anatomical object, such as based upon the segmentation of the anatomical object, as described above. Nevertheless, the projection may be limited to a boundary of the anatomical object and may be displayed either alone or in combination with a corresponding icon. The projected instrument icon 16i may be based upon a geometry of selected tools such as a drill so that the user 15 may view the physical extent of the drill relative to the image and the segmented anatomical object or portion thereof to ensure that the drill would drill far enough into the anatomical object.

[0139] In various embodiments, the find entry point features may be used to then identify or mark a point on the skin of the subject 20. It is understood that marking the incision point is not required. However, performing an incision to allow other instruments to enter the subject 20 may occur after finding the entry point as discussed above. Once an incision is made, a tool may be navigated including tracking the tool and illustrating the position of the tool on the display device 32, as designated by block 1332. For example, after forming the initial incision, an awl may be navigated to the anatomical object identified at 1330. The tool may also be referred to as an instrument.

[0140] In navigating the awl relative to the anatomical object, the awl may be passed through the incision in the skin and contact the vertebrae. An icon representing the awl or a projection from the tracked location of the awl may be illustrated relative to the anatomical object at a selected time, such as when the awl is within a selected distance to the anatomical object (e.g., less than about 1 mm to about 6 mm, including about 5 mm). Thus, the icon representing the tool may auto display a selected implant size, such as an icon superimposed on the image on the display device 32.

[0141] Automatically display an implant size, or tool size or position, may include determining the size of an implant based upon the boundaries of the segmented anatomical object at 1334. The navigation process may include executing instructions based upon the segmented image geometry including size and shape to automatically select and display the anatomical object and optionally the selected implant having the selected size on the display device 32. The user 15 may confirm and / or change the selected implant size at 1336. If changed, a different size implant may then be displayed relative to the image of the anatomical object and / or the segmentation. The user 15 may then view the automatically displayed implant size and / or a changed or confirmed size.

[0142] Further the user 15 may move the tracked awl relative to the anatomical object to select a position of the implant relative to the anatomical object. For example, a different position of the awl relative to the anatomical object may cause the system to determine or calculate a different size implant. Once the user 15 has selected via an input device an appropriate or selected trajectory, the trajectory may be saved at 1338. The input device may be a verbal command for audio input, a gesture, afootswitch, or the like. In addition, the user's selection may be saved based upon the selected surgeon for further or future reference.

[0143] The projection may be saved for future use and displayed and / or hidden as selected to allow for guiding of the tapping of the anatomical object. The anatomical object may be tapped with a tap while viewing the display device 32 when the tap is navigated. The tap may be navigated as it is moved relative to the anatomical object by the user 15. The tap may be displayed as the icon 16i on the display device 32 relative to an image. Further, at a selected time, such as when the tap is near or in contact with the anatomical object projection of a tapped geometry may be displayed relative to the image including the anatomical object as navigating the tap at 1340 to the anatomical object and displaying at the projection of the tapped area or volume may allow the user 15 to confirm that the selected tapped volume based upon a projection of the tap into the anatomical object matches the implant projection or saved implant projection.

[0144] Once it is confirmed that the tap projection matches the saved implant projection, the tap may be driven into the anatomical object. A reduced or shrunken tap projection at 1342 may allow the user 15 to view the extent of the tapping relative to the projected or selected tap length volume. The shrunken tap geometry may allow the user 15 to understand the extent of the tapping performed so far and the auto tapping remaining. Accordingly, the user may slow down driving of the tap into the vertebrae at a selected period while allowing for a fast and efficient tapping at an initial period.

[0145] It is understood that the navigation processor system 66 may shrink the tapped projection, such as the shrunken tap projection at 1342, substantially automatically based upon navigation of the tap relative to the anatomical object. The tap projection is initially based upon the selected implant projection based upon the implant, such as the automatically selected implant. Therefore, the navigation process may allow for efficient tapping of the anatomical object by allowing the user 15 to view the tapping in process and confirm when the tapping is completed.

[0146] When tapping is completed, a reverse projection may be automatically determined and displayed at 1346. The reverse projection may be substantially equivalent or equal to the tapped depth into the anatomical object and based upon the amount of tapping or depth of tapping by the user 15. Further, the reverse projection of may be substantially equivalent to the initial tapped projection. The reverse tap projection may be maintained for viewing by the user 15 on the display device 32 relative to the anatomical object for positioning of the implant into the anatomical object. Moreover, the instrument icon 16i may be a combination of both the instrument portion and a now fixed or permanent tapped projection. The fixed projection of may be initially equivalent to the reverse projection and allow the user 15 to view both the tapped volume (e.g., width and / or depth) relative to the instrument icon 16i and the image of the anatomical object.

[0147] The reverse projection may be saved at 1346 for various purposes, as discussed above for guiding or navigating the implant. Further the saved reverse projection may be equivalent to the tapped position and may also be saved under the selected surgeon for further reference and / or future reference.

[0148] Once the tapping of the anatomical object is performed, the implant may be placed in the anatomical object The implant may include a screw, such as a pedicle screw, that is positioned within the anatomical object. The screw and a driver may be illustrated as an icon on the display device 32 relative to the image of the anatomical object. The reverse projection may also be displayed to assist in navigating the implant at 1348. The implant may be illustrated as at least a part of the icon such that the icon may be aligned with the reverse projection to allow for driving or placing the screw into the anatomical object along the tapped trajectory and volume as illustrated by the reverse projection. Accordingly, navigating the implant may allow the user 15 to position the implant in the selected and tapped location in the anatomical object.

[0149] Tracking of the screw into the anatomical object may also allow for a saved tracked position of the screw at 1350 for the selected surgeon for future use. Accordingly various features, such as positioning of the tapped location and final position of the screw along with various other features, such as geometry and size of the screw may be saved for reference of a selected surgeon for future use.

[0150] After positioning the screw by navigating the implant and / or saving the tracked screw position a determination of whether further implants (e.g., screws) need be placed may be made at 1352. If no additional implants are to be placed, the method may end. Completing the procedure may include decompressing vertebra, removing instrumentation from the subject 20, closing the incision, or other appropriate features.

[0151] If it is determined that another implant is to be implanted operation 1354 may be performed followed by operation 1328. The determination of whether an additional implant is to be placed may be based upon the selected procedure or basedupon a user input. Accordingly, determining whether a further implant is to be positioned may be substantially automatic or manual.

[0152] At 1354, auto-switching to a further image portion may optionally occur. For example, if a first screw is placed in an L5 vertebra, a second screw may be placed in a second side of the L5 vertebra. Automatically switching to a separate image or view portion of the vertebrae may assist the user 15. Further, if a second implant is positioned in the L5 vertebra and the selected procedure is to fuse an L5 and L4 vertebra, the image may automatically switch to display or more closely display the L4 vertebra for further procedure steps. Accordingly, auto-switching to another image portion may assist the user 15 in efficiently performing the procedure.

[0153] Whether an optional automatic switching to additional image step is performed or not, the determination that further implants are to be placed may include the navigation process looping back to operation 1328 to automatically select parameters of the next implant and continue the procedure. It is understood that various other portions of the process may also be repeated, such as identifying instruments or tips, but such application may not be required, particularly if the instrumentation maintains or remains the same from for multiple implants. Nevertheless, a selected number of implants may be positioned in a subject by continuing the process until there are no further implants to be implanted.

[0154] FIG. 14 shown a cross-sectional image 1400 that may be used for determining anatomy of a subject for orientation determination purposes. This determination may be made via, for example, the above-described anatomy module 308 of FIG. 3. The anatomy module 308 may receive a set of images of a portion of aneck 1401 of a subject. As an example, the image 1400 may be a CT image of a spinal area of the subject. In the example shown, the image 1400 includes a portion of a spinal region of the neck 1401 of the subject. The anatomy module 308 may identify vertebral body (or vertebrae) 1402, the C2 vertebra odontoid process 1404, and / or other anatomical objects. In an embodiment, the anatomy module 308 may shade, color, and / or pattern the bodies, elements, sacrum, and / or other anatomical objects detected as shown. The posterior elements 704 include pedicles to spinous process. The positional relationship between the vertebral body 702, the posterior elements 704, the sacrum 706, and / or other anatomical features such as the C2 vertebra odontoid process indicates the orientation of the subject. For example, the posterior elements 704 are always located posterior of the vertebral body 702, the sacrum is always a lower vertebrae, and the C2 vertebra with the unique odontoid process is always a higher vertebrae. This allows the orientation module 302 to determine the anterior-posterior (AP) direction and the superior-inferior (SI) direction.

[0155] Examples

[0156] An imaging system is disclosed and includes a memory, an orientation module, and at least one processor. The memory is configured to store orientation metadata and images of a subject. The orientation module includes at least one neural network configured to implement at least one artificial intelligence algorithm to analyze a first one or more of the images, and based on the analysis, to determine orientation of at least one anatomical object of the subject. The at least one processor is configured, based on the determined orientation, to at least one of verify theorientation metadata, correct the orientation metadata, analyze a second one or more images, and track at least one of a tool and an implant relative to the patient.

[0157] In other features, the determined orientation is relative to at least one of a reference point and a structure supporting at least a portion of the subject.

[0158] In other features, the at least one artificial intelligence algorithm includes at least one of a machine learning algorithm and a deep learning algorithm. In other features, the at least one artificial intelligence algorithm includes at least one of a dense network, an inception network, a vision transformer network, and a fully convolutional network.

[0159] In other features, the orientation module is configured to implement the at least one artificial intelligence algorithm to detect at least one landmark of the at least one anatomical object, and based on the at least one landmark, determine the orientation of the at least one anatomical object.

[0160] In other features, the orientation module is configured to implement the at least one artificial intelligence algorithm to identify the at least one anatomical object, and based on the identification, determine the orientation of the at least one anatomical object. In other features, the at least one artificial intelligence algorithm includes at least one of a machine learning algorithm and a deep learning algorithm. In other features, the at least one artificial intelligence algorithm includes at least one of a dense network, an inception network, a vision transformer network, and a three- dimensional fully convolutional network.

[0161] In other features, the orientation module is configured to implement the at least one artificial intelligence algorithm to determine directly from the images the orientation of the at least one anatomical object.

[0162] In other features, the orientation module is configured to compare the determined orientation to the orientation metadata, and, in response to determining that an orientation mismatch exists between the determined orientation and the orientation metadata, correct the orientation metadata.

[0163] In other features, the orientation module is configured to compare the determined orientation to the orientation metadata, and in response to determining that an orientation mismatch exists between the determined orientation and the orientation metadata, indicate a mismatch has been detected via a user interface and wait for approval to change the orientation metadata.

[0164] In other features, the orientation module is configured to determine a confidence level in the determined orientation, and based on the confidence level, correct the orientation metadata in response to determining that an orientation mismatch exists between the determined orientation and the orientation metadata.

[0165] In other features, the at least one processor is configured, based on the determined orientation, to verify the orientation metadata. In other features, the at least one processor is configured, based on the determined orientation, to correct the orientation metadata.

[0166] In other features, the at least one processor is configured, based on the determined orientation, to analyze the second one or more images.

[0167] In other features, the at least one processor is configured, based on the determined orientation, to track the at least one of the tool and the implant relative to the patient.

[0168] In other features, the processor is configured to analyze the first one or more images having a first resolution to determine orientation of the at least one anatomical object, and analyze the second one or more images having a second resolution based on the determined orientation. The first resolution is less than the second resolution.

[0169] In other features, the first one or more images include a three-dimensional image. In other features, the first one or more images include a two-dimensional image.

[0170] In other features, the orientation module is configured to: at least one of capture and access a first slice in a sagittal view of the at least one anatomical object; at least one of capture and access a second slice in an axial view of the at least one anatomical object; at least one of capture and access a third slice in a coronal view of the at least one anatomical object; and determine the orientation of the at least one anatomical object based on the first slice, the second slice and the third slice. In other features, the first slice, the second slice and the third slice are middle slices.

[0171] In other features, the orientation module is configured to: at least one of capture and access a first maximum intensity projection image in a sagittal view of the at least one anatomical object; at least one of capture and access a second maximum intensity projection image in an axial of the at least one anatomical object; at least one of capture and access a third maximum intensity projection image in a coronal view ofthe at least one anatomical object; and determine the orientation of the at least one anatomical object based on the first maximum intensity projection image, the second maximum intensity projection image and the third maximum intensity projection image.

[0172] Example 1. An imaging system comprising: a memory configured to store orientation metadata and images of a subject; an orientation module comprising at least one neural network configured to implement at least one artificial intelligence algorithm to analyze a first one or more of the images, and based on the analysis, to determine orientation of at least one anatomical object of the subject; and at least one processor configured, based on the determined orientation, to at least one of verify the orientation metadata, correct the orientation metadata, analyze a second one or more images, and track at least one of a tool and an implant relative to the patient.

[0173] Example 2. The imaging system of example 1 , wherein the determined orientation is relative to at least one of a reference point and a structure supporting at least a portion of the subject.

[0174] Example 3. The imaging system of example 1 , wherein the at least one artificial intelligence algorithm includes at least one of a machine learning algorithm and a deep learning algorithm.

[0175] Example 4. The imaging system of example 1 , wherein the at least one artificial intelligence algorithm includes at least one of a dense network, an inception network, a vision transformer network, and a fully convolutional network.

[0176] Example 5. The imaging system of example 1 , wherein the orientation module is configured to implement the at least one artificial intelligence algorithm to detect at least one landmark of the at least one anatomical object, and based on the at least one landmark, determine the orientation of the at least one anatomical object.

[0177] Example 6. The imaging system of example 5, wherein the at least one artificial intelligence algorithm includes at least one of a machine learning algorithm and a deep learning algorithm.

[0178] Example 7. The imaging system of example 5, wherein the at least one artificial intelligence algorithm includes at least one of a dense network, an inception network, a vision transformer network, and a fully convolutional network.

[0179] Example 8. The imaging system of example 1 , wherein the orientation module is configured to implement the at least one artificial intelligence algorithm to identify the at least one anatomical object, and based on the identification, determine the orientation of the at least one anatomical object.

[0180] Example 9. The imaging system of example 8, wherein the at least one artificial intelligence algorithm includes at least one of a machine learning algorithm and a deep learning algorithm.

[0181] Example 10. The imaging system of example 8, wherein the at least one artificial intelligence algorithm includes at least one of a dense network, an inception network, and a fully convolutional network.

[0182] Example 11. The imaging system of example 1 , wherein the orientation module is configured to implement the at least one artificial intelligence algorithm todetermine directly from the images the orientation of the at least one anatomical object.

[0183] Example 12. The imaging system of example 1 , wherein the orientation module is configured to compare the determined orientation to the orientation metadata, and, in response to determining that an orientation mismatch exists between the determined orientation and the orientation metadata, correct the orientation metadata.

[0184] Example 13. The imaging system of example 1 , wherein the orientation module is configured to compare the determined orientation to the orientation metadata, and in response to determining that an orientation mismatch exists between the determined orientation and the orientation metadata, indicate a mismatch has been detected via a user interface and wait for approval to change the orientation metadata.

[0185] Example 14. The imaging system of example 1 , wherein the orientation module is configured to determine a confidence level in the determined orientation, and based on the confidence level, correct the orientation metadata in response to determining that an orientation mismatch exists between the determined orientation and the orientation metadata.

[0186] Example 15. The imaging system of example 1 , wherein the at least one processor is configured, based on the determined orientation, to verify the orientation metadata.

[0187] Example 16. The imaging system of example 1 , wherein the at least one processor is configured, based on the determined orientation, to correct the orientation metadata.

[0188] Example 17. The imaging system of example 1 , wherein the at least one processor is configured, based on the determined orientation, to analyze the second one or more images.

[0189] Example 18. The imaging system of example 1 , wherein the at least one processor is configured, based on the determined orientation, to track the at least one of the tool and the implant relative to the patient.

[0190] Example 19. The imaging system of example 1 , wherein: the processor is configured to analyze the first one or more images having a first resolution to determine orientation of the at least one anatomical object, and analyze the second one or more images having a second resolution based on the determined orientation; and the first resolution is less than the second resolution.

[0191] Example 20. The imaging system of example 19, wherein the first one or more images include a three-dimensional image.

[0192] Example 21. The imaging system of example 19, wherein the first one or more images include a two-dimensional image.

[0193] Example 22. The imaging system of example 1 , wherein the orientation module is configured to:at least one of capture and access a first slice in a sagittal view of the at least one anatomical object; at least one of capture and access a second slice in an axial of the at least one anatomical object; at least one of capture and access a third slice in a coronal view of the at least one anatomical object; and determine the orientation of the at least one anatomical object based on the first slice, the second slice and the third slice.

[0194] Example 23. The imaging system of example 22, wherein the first slice, the second slice and the third slice are middle slices.

[0195] Example 24. The imaging system of example 1 , wherein the orientation module is configured to: at least one of capture and access a first maximum intensity projection image in a sagittal view of the at least one anatomical object; at least one of capture and access a second maximum intensity projection image in an axial of the at least one anatomical object; at least one of capture and access a third maximum intensity projection image in a coronal view of the at least one anatomical object; and determine the orientation of the at least one anatomical object based on the first maximum intensity projection image, the second maximum intensity projection image and the third maximum intensity projection image.

[0196] Example embodiments are provided. Numerous specific details are set forth such as examples of specific components, devices, and methods, to provide an understanding of embodiments of the present disclosure. It will be apparent to those skilled in the art that specific details need not be employed, that example embodiments may be embodied in many different forms and that neither should be construed to limit the scope of the disclosure. In some example embodiments, well- known processes, well-known device structures, and well- known technologies are not described in detail.

[0197] Instructions may be executed by a processor and may include software, firmware, and / or microcode, and may refer to programs, routines, functions, classes, data structures, and / or objects. The term shared processor circuit encompasses a single processor circuit that executes some or all code from multiple modules. The term group processor circuit encompasses a processor circuit that, in combination with additional processor circuits, executes some or all code from one or more modules. References to multiple processor circuits encompass multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term shared memory circuit encompasses a single memory circuit that stores some or all code from multiple modules. The term group memory circuit encompasses a memory circuit that, in combination with additional memories, stores some or all code from one or more modules.

[0198] The apparatuses and methods described in this application may be partially or fully implemented by one or more processors (also referred to as processormodules) that may include a special purpose computer (i.e., created by configuring one or more processors) to execute one or more particular functions embodied in computer programs. The computer programs include processor-executable instructions that are stored on at least one non-transitory, tangible computer-readable medium. The computer programs may also include or rely on stored data. The computer programs may include a basic input / output system (BIOS) that interacts with hardware of the special purpose computer, device drivers that interact with particular devices of the special purpose computer, one or more operating systems, user applications, background services and applications, etc.

[0199] The computer programs may include: (i) assembly code; (ii) object code generated from source code by a compiler; (iii) source code for execution by an interpreter; (iv) source code for compilation and execution by a just-in-time compiler, (v) descriptive text for parsing, such as HTML (hypertext markup language) or XML (extensible markup language), etc. As examples only, source code may be written in C, C++, C#, Objective-C, Haskell, Go, SOL, Lisp, Java®, ASP, Perl, Javascript®, HTML5, Ada, ASP (active server pages), Perl, Scala, Erlang, Ruby, Flash®, Visual Basic®, Lua, or Python®.

[0200] Communications may include wireless communications described in the present disclosure can be conducted in full or partial compliance with IEEE standard 802.11-2012, IEEE standard 802.16-2009, and / or IEEE standard 802.20-2008. In various implementations, IEEE 802.11-2012 may be supplemented by draft IEEE standard 802.11ac, draft IEEE standard 802.11 ad, and / or draft IEEE standard802.11ah.

[0201] A processor, processor module, module or 'controller' may be used interchangeably herein (unless specifically noted otherwise) and each may be replaced with the term 'circuit.' Any of these terms may refer to, be part of, or include: an Application Specific Integrated Circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system- on-chip.

[0202] Instructions may be executed by one or more processors or processor modules, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor" or "processor module" as used herein may refer to any of the foregoing structure or any other physical structure suitable for implementation of the described techniques. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0203] The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown ordescribed. The same may also be varied in many ways. Such variations are not to be regarded as a departure from the invention, and all such modifications are intended to be included within the scope of the invention.

Claims

CLAIMSWhat is claimed is:

1. An imaging system comprising: a memory configured to store orientation metadata and images of a subject; an orientation module comprising at least one neural network configured to implement at least one artificial intelligence algorithm to analyze a first one or more of the images, and based on the analysis, to determine orientation of at least one anatomical object of the subject; and at least one processor configured, based on the determined orientation, to at least one of verify the orientation metadata, correct the orientation metadata, analyze a second one or more images, and track at least one of a tool and an implant relative to the patient.

2. The imaging system of claim 1 , wherein the determined orientation is relative to at least one of a reference point and a structure supporting at least a portion of the subject.

3. The imaging system of claim 1 , wherein the at least one artificial intelligence algorithm includes at least one of a machine learning algorithm and a deep learning algorithm.

4. The imaging system of claim 1 , wherein the at least one artificial intelligence algorithm includes at least one of a dense network, an inception network, and a fully convolutional network.

5. The imaging system of claim 1 , wherein the orientation module is configured to implement the at least one artificial intelligence algorithm to detect at least one landmark of the at least one anatomical object, and based on the at least one landmark, determine the orientation of the at least one anatomical object.

6. The imaging system of claim 1 , wherein the orientation module is configured to implement the at least one artificial intelligence algorithm to identify the at least one anatomical object, and based on the identification, determine the orientation of the at least one anatomical object.

7. The imaging system of claim 1 , wherein the orientation module is configured to implement the at least one artificial intelligence algorithm to determine directly from the images the orientation of the at least one anatomical object.

8. The imaging system of claim 1 , wherein the orientation module is configured to compare the determined orientation to the orientation metadata, and, in response to determining that an orientation mismatch exists between the determined orientation and the orientation metadata, correct the orientation metadata.

9. The imaging system of claim 1 , wherein the orientation module is configured to compare the determined orientation to the orientation metadata, and in response to determining that an orientation mismatch exists between the determined orientation and the orientation metadata, indicate a mismatch has been detected via a user interface and wait for approval to change the orientation metadata.

10. The imaging system of claim 1 , wherein the orientation module is configured to determine a confidence level in the determined orientation, and based on the confidence level, correct the orientation metadata in response to determining that an orientation mismatch exists between the determined orientation and the orientation metadata.

11. The imaging system of claim 1 , wherein the at least one processor is configured, based on the determined orientation, to verify the orientation metadata.

12. The imaging system of claim 1 , wherein the at least one processor is configured, based on the determined orientation, to correct the orientation metadata.

13. The imaging system of claim 1 , wherein the at least one processor is configured, based on the determined orientation, to analyze the second one or more images.

14. The imaging system of claim 1 , wherein the at least one processor is configured, based on the determined orientation, to track the at least one of the tool and the implant relative to the patient.

15. The imaging system of claim 1 , wherein: the processor is configured to analyze the first one or more images having a first resolution to determine orientation of the at least one anatomical object, and analyze the second one or more images having a second resolution based on the determined orientation; and the first resolution is less than the second resolution.

Images