Anatomical structure orientation detection and verification using multi-dimensional imaging

Abstract

An imaging system is disclosed and includes a memory, an orientation module, and at least one processor. The memory is configured to store orientation metadata and images of a subject. The orientation module includes at least one neural network configured to implement at least one artificial intelligence algorithm to analyze a first one or more images of the images and determine an orientation of at least one anatomical object of the subject based on the analysis. The at least one processor is configured to perform at least one of the following based on the determined orientation: validate the orientation metadata, correct the orientation metadata, analyze a second one or more images, and track a position of at least one of a tool and an implant relative to a patient.

Description

Technical Field

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 609,453, filed on December 13, 2023, the entire contents of which are incorporated herein by reference.

[0002] This disclosure relates in general to a system for determining the orientation of patient anatomical structures within an imaging volume. Background Technology

[0003] This section provides background information in connection with this disclosure, which is not necessarily prior art.

[0004] Imaging techniques that produce highly detailed two-dimensional, three-dimensional, and four-dimensional images include computed tomography (CT), magnetic resonance imaging (MRI), fluorescence fluoroscopy (such as using a C-arm device), positron emission tomography (PET), and ultrasound imaging (US). Subjects are scanned, and each scan can include a collection of hundreds to thousands of cross-sectional images (or slices) of the subject. These cross-sectional images are stored and can be combined to create 3D images of the subject's anatomy. For example, the images can be analyzed to assess the subject's condition. Image-guided medical and surgical procedures can also utilize images of the subject's condition obtained before or during the procedure to guide the physician in performing the procedure.

[0005] During the navigation procedure, images are acquired by a suitable imaging device and displayed on a workstation. The navigation system tracks the patient, instruments, and other devices in the surgical area and / or patient space. These tracked devices are then displayed in image space relative to the image data on the workstation. To track the patient, instruments, and other devices, they may be equipped with tracking devices. For example, a tracking device may be attached to the outer surface of an instrument and can provide the surgeon with an accurate depiction of the instrument's position in the patient space via a corresponding tracking system. Summary of the Invention

[0006] This section provides a general overview of this disclosure and is not a full disclosure of the complete scope or all features of this disclosure.

[0007] An imaging system is disclosed, comprising a memory, an orientation module, and at least one processor. The memory is configured to store orientation metadata and images of a subject. The orientation module includes at least one neural network configured to implement at least one artificial intelligence algorithm to analyze one or more images in the memory and determine the orientation of at least one anatomical object of the subject based on the analysis. The at least one processor is configured to perform at least one of the following based on the determined orientation: verify the orientation metadata, correct the orientation metadata, analyze one or more second images, and track the position of at least one of a tool and an implant relative to the patient.

[0008] Among other features, the determined orientation is relative to at least one of a reference point and a structure supporting at least a portion of the subject.

[0009] Among other features, the at least one artificial intelligence algorithm includes at least one of machine learning algorithms and deep learning algorithms. Among other features, the at least one artificial intelligence algorithm includes at least one of dense networks, Inception networks, visual transformer networks, and fully convolutional networks.

[0010] Among other features, the orientation module is configured to: implement the at least one artificial intelligence algorithm to detect at least one landmark of the at least one anatomical object, and determine the orientation of the at least one anatomical object based on the at least one landmark.

[0011] Among other features, the orientation module is configured to: implement the at least one artificial intelligence algorithm to identify the at least one anatomical object, and determine the orientation of the at least one anatomical object based on the identification. Among other features, the at least one artificial intelligence algorithm includes at least one of machine learning algorithms and deep learning algorithms. Among other features, the at least one artificial intelligence algorithm includes at least one of dense networks, Inception networks, visual transformer networks, and 3D fully convolutional networks.

[0012] Among other features, the orientation module is configured to implement the at least one artificial intelligence algorithm to determine the orientation of the at least one anatomical object directly from the image.

[0013] Among other features, the orientation module is configured to: compare the determined orientation with the orientation metadata, and correct the orientation metadata in response to determining that there is an orientation mismatch between the determined orientation and the orientation metadata.

[0014] Among other features, the orientation module is configured to: compare the determined orientation with the orientation metadata, and in response to determining that there is an orientation mismatch between the determined orientation and the orientation metadata, indicate via a user interface that a mismatch has been detected, and wait for approval to change the orientation metadata.

[0015] Among other features, the orientation module is configured to: determine a confidence level for the determined orientation, and, in response to determining that there is an orientation mismatch between the determined orientation and the orientation metadata, correct the orientation metadata based on the confidence level.

[0016] Among other features, the at least one processor is configured to verify the orientation metadata based on the determined orientation. Among other features, the at least one processor is configured to correct the orientation metadata based on the determined orientation.

[0017] Among other features, the at least one processor is configured to analyze the second or more images based on the determined orientation.

[0018] Among other features, the at least one processor is configured to track the position of the tool and the at least one of the implants relative to the patient based on the determined orientation.

[0019] Among other features, the processor is configured to: analyze the first one or more images having a first resolution to determine the orientation of the at least one anatomical object, and analyze the second one or more images having a second resolution based on the determined orientation. The first resolution is smaller than the second resolution.

[0020] Among other features, the first one or more images include three-dimensional images. Among other features, the first one or more images include two-dimensional images.

[0021] Among other features, the orientation module is configured to: capture a first slice in a sagittal view of the at least one anatomical object and access at least one of the first slices in the sagittal view of the at least one anatomical object; capture a second slice in an axial view of the at least one anatomical object and access at least one of the second slices in the axial view of the at least one anatomical object; capture a third slice in a coronal view of the at least one anatomical object and access at least one of the third slices in the coronal view of the at least one anatomical object; and determine the orientation of the at least one anatomical object based on the first slice, the second slice, and the third slice. Among other features, the first slice, the second slice, and the third slice are intermediate slices.

[0022] Among other features, the orientation module is configured to: capture a first maximum intensity projection image in a sagittal view of the at least one anatomical object and access at least one of the first maximum intensity projection images in the sagittal view of the at least one anatomical object; capture a second maximum intensity projection image in the axial direction of the at least one anatomical object and access at least one of the second maximum intensity projection images in the axial direction of the at least one anatomical object; capture a third maximum intensity projection image in a coronal view of the at least one anatomical object and access at least one of the third maximum intensity projection images in the coronal view of the at least one anatomical object; and determine the orientation of the at least one anatomical object based on the first maximum intensity projection image, the second maximum intensity projection image, and the third maximum intensity projection image.

[0023] Further areas of applicability will become apparent from the description provided herein. The descriptions and specific examples in this overview are intended for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0024] The accompanying drawings described herein are for illustrative purposes only, representing the selected embodiments and not all possible specific implementations, and are not intended to limit the scope of this disclosure.

[0025] Figure 1 This is an environmental view of an example navigation system according to the present disclosure, including an imaging system comprising an image processor having an orientation module; Figure 2 This is a functional block diagram of an example control system including an orientation module according to the present disclosure; Figure 3 This is a functional block diagram of an example processor including an orientation module according to the present disclosure; Figure 4 This is an example diagram of an orientation module for determining the orientation of a subject's head in a first position, according to the present disclosure, the orientation module including an AI neural network implementing at least one of an artificial intelligence (AI) algorithm and a machine learning algorithm; Figure 5 This is another example diagram according to the present disclosure for using an orientation module to determine the orientation of a subject's head in a second location directly from image data, the orientation module including an AI neural network implementing at least one of an AI algorithm and a machine learning algorithm; Figure 6 This is an example diagram of a landmark module used to determine the head of a subject for orientation determination purposes, according to the present disclosure. Figure 7 This is an example diagram based on the present disclosure for using an anatomical structure module to determine the anatomical structure of a subject for orientation determination purposes; Figure 8 It is a 3D representative view of the skull of the subject according to this disclosure, which illustrates an example on which the orientation of the subject's head can be determined; Figure 9 The present disclosure provides a 3D representative view of the skull of a subject, which illustrates an example of a 2D slice that can be used to determine the orientation of the subject's head. Figure 10A and Figure 10B (Collectively referred to as FIG10) illustrates an imaging process according to the present disclosure, including orientation verification and correction; Figure 11 An example training process according to this disclosure is illustrated; Figure 12 An example segmentation method according to this disclosure is illustrated; Figure 13 An example method for operating a surgical navigation system according to this disclosure is illustrated; and Figure 14 It is a cross-sectional image that, according to this disclosure, can be used to determine the anatomical structure of a subject for orientation determination.

[0026] In several views of all the accompanying drawings, the corresponding reference numerals indicate the corresponding components. Detailed Implementation

[0027] The example implementation will now be described in more complete form with reference to the accompanying drawings.

[0028] Multidimensional imaging (e.g., CT imaging, MRI imaging, etc.) is achieved by navigation systems (e.g., the commercially available StealthStation sold by Medtronic Navigation, Inc. of Littleton, Massachusetts). ® ENT navigation systems are used in surgical procedures. To achieve correct registration, the orientation of the captured images needs to be known. Orientation metadata is typically provided by technicians and entered into a browser (e.g., DICOM). ® In the browser, orientation metadata may be entered incorrectly. For example, the patient's posture may be accidentally entered as prone instead of supine, or as facing left instead of right. Incorrect orientation metadata can lead to inaccuracies in image registration, image analysis, and navigation.

[0029] Examples described herein include imaging systems or processors configured to determine the orientation of a subject's anatomical structures based on captured multidimensional images. This orientation information can be relied upon and / or used for validation purposes. For example, the orientation information can be compared with orientation metadata provided by a technician to validate and / or correct the orientation metadata. The orientation information can be used to replace orientation metadata from a technician and for non-contact registration purposes. The orientation information allows the processing system to determine the orientation of anatomical features and / or objects in the image.

[0030] Examples include using AI algorithms to automatically detect the orientation of anatomical objects (e.g., head, vertebrae, etc.) based on acquired images. This includes determining: whether the anatomical object is in a prone or supine position; whether the anatomical object is facing left or right; whether the anatomical object is oriented up or down; etc. This is determined directly and / or indirectly from the acquired images without requiring orientation metadata. The automatic determination of the anatomical object's orientation can be compared with orientation metadata to alert technicians of the possibility of erroneous orientation metadata in the event of a mismatch. Technicians can then accept corrections to the orientation metadata.

[0031] Examples include neural networks that receive images at different resolution levels. Lower-resolution images can be analyzed to minimize storage and / or processing requirements and the time involved. In one embodiment, a low-resolution version of a full 3D image is received and analyzed, providing orientations such as left, back, top (LPS), right, front, bottom (RAI), etc., as output. In another embodiment, one or more slices of the 3D image (e.g., intermediate slices in sagittal, axial, and coronal views) are analyzed to determine orientation. In yet another embodiment, maximum intensity projection (MIP) images are analyzed in sagittal, axial, and coronal views to determine orientation. MIP involves projecting the voxel with the highest attenuation value from each view onto the 2D image.

[0032] Figure 1 An example navigation system 10 is shown, including an imaging system 11 and an image processor 12 with an orientation module 13. The orientation module 13 automatically determines the orientation of the subject's anatomical structures based on acquired images of the subject's anatomy. The orientation module 13 may be implemented by the image processor 12 or by another processor, such as a navigation processor 14. An example of the orientation module 13 is shown in... Figure 3 As shown in the diagram. Orientation module 13 can be configured and operated as any of the orientation modules mentioned herein. The following is at least relative to Figures 2 to 13 The operation of orientation module 13 is further described.

[0033] The navigation system 10 can be used by one or more users (such as user 15) for various purposes or procedures. The navigation system 10 can be used to determine or track the positioning of the instrument 16 in a volume. Positioning can include both three-dimensional X, Y, Z position and orientation. Orientation can include one or more degrees of freedom, such as three degrees of freedom. However, it should be understood that any appropriate degree of freedom positioning information, such as positioning information with fewer than six degrees of freedom, can be determined and / or presented to user 15.

[0034] Tracking the location of instrument (or tool) 16 can assist user 15 in determining the location of instrument 16, even if instrument 16 cannot be directly seen by user 15. Various procedures may obstruct user 15's line of sight, such as performing repairs or assemblies of inanimate systems (such as robotic systems), assembling parts of an aircraft fuselage or automobile, etc. Various other procedures may include surgical procedures, such as spinal procedures, neurological procedures, locating deep brain simulation probes, or other surgical procedures performed on a living subject. In various embodiments, for example, the living subject may be a human subject 20, and the procedure may be performed on a human subject 20. However, it should be understood that for any suitable procedure, instrument 16 can be tracked and / or navigated relative to any subject. Tracking or navigating instruments for procedures (such as surgical procedures) on a human or living subject is merely exemplary.

[0035] However, in various embodiments, as further discussed herein, the surgical navigation system 10 may incorporate various components or systems, such as those disclosed in U.S. Patent Nos. RE44,305; 7,697,972, 8,644,907, and 8,842,893, and U.S. Patent Application Publication No. 2004 / 0199072, all of which are incorporated herein by reference. Various components that may be used with or as part of the surgical navigation system 10 may include an imaging system 11 operable to image the subject 20, such as an O-arm. ® Imaging systems, magnetic resonance imaging (MRI) systems, computed tomography (CT) systems, etc. Subject support 26 can be used to support or hold subject 20 during imaging and / or during the procedure. The same or different supports can be used for different parts of the procedure.

[0036] In various embodiments, the imaging system 11 may include a source 24s. The source may emit and / or generate X-rays. The X-rays may form cones 24c that impact the subject 20, such as in a cone beam. Some of the X-rays pass through the subject 20, while others are attenuated by the subject. The imaging system 24 may also include a detector 24d to detect X-rays that are not completely attenuated or blocked by the subject 20. Therefore, the image data may include X-ray image data. Further, the image data may be two-dimensional (2D) image data.

[0037] Image data may be acquired during a surgical procedure, such as by one or more imaging systems discussed above, or prior to the surgical procedure for displaying image 30 on display device 32. In various embodiments, even if the image data is 2D image data, the acquired image data may be used to form or reconstruct selected types of image data, such as three-dimensional volume. Instrument 16 may be tracked in a trackable or navigable volume by one or more tracking systems. Tracking systems may include one or more tracking systems operating in the same or multiple and / or different ways or modes. For example, tracking system 44 may include electromagnetic (EM) positioner 40, such as… Figure 1 As illustrated herein, those skilled in the art will understand that other suitable tracking systems, including optical, radar, ultrasonic, etc., can be used in various embodiments. The EM locator 40 and tracking system discussed herein are merely exemplary tracking systems that can operate in conjunction with the navigation system 10. The position of the device 16 relative to the subject 20 can be tracked within the tracking volume and then displayed as a graphical representation, also referred to as icon 16i, using the display device 32. In various embodiments, icon 16i may be overlaid on and / or adjacent to image 30. As discussed herein, the navigation system 10 may be incorporated into the display device 32 and operated to render image 30 based on selected image data, display image 30, determine the position of device 16, determine the position of icon 16i, etc.

[0038] The EM locator 40 is operable to generate an electromagnetic field using a transmitting coil array (TCA) 42 incorporated into the EM locator 40. The TCA 42 may include one or more coil groups or arrays. In various embodiments, more than one group is included, and each group may include three coils, also referred to as a triplet or triplet. The coils can be powered to generate or form an electromagnetic field by driving current through the coils of the coil group. When current is driven through the coils, the generated electromagnetic field will extend away from the coils of the TCA 42 and form a navigation domain or volume 50, such as surrounding all or part of the head 20h, spinal vertebrae 20v, or other suitable parts. The coils can be powered by a TCA controller and / or a power supply 52. ​​However, it should be understood that more than one EM locator in the EM locator 40 may be provided, and each EM locator may be placed in a different and selected location.

[0039] A navigation domain or volume 50 typically defines a navigation space or patient space. As is generally understood in the art, an instrument tracking device 56 can be used to track an instrument 16, such as a drill or wire, relative to a patient or subject 20 within the navigation space defined by the navigation domain. For example, the instrument 16 may be capable of free movement, such as by a user 15, relative to a dynamic reference frame (DRF) or a patient reference frame tracker 60, which is fixed relative to the subject 20. Both tracking devices 56 and 60 may include a tracking portion that is tracked using a suitable tracking system, such as a sensing coil (e.g., a conductive material formed in or placed in the coil) for sensing and measuring magnetic field strength, an optical reflector, an ultrasonic transmitter, etc. Because the instrument tracking device 56 is connected to or associated with the instrument 16 relative to the DRF 60, the navigation system 10 can be used to determine the position of the instrument 16 relative to the DRF 60.

[0040] The navigation volume or patient space can be registered to the image space defined by the image 30 of the subject 20, and the icon 16i representing the device 16 can be illustrated by the display device 32 for navigable (e.g., determined) and tracking positioning, such as overlaid on the image 30. Patient space-to-image space registration and positioning of a tracking device (such as device tracking device 56) relative to a DRF (such as DRF 60) can be performed as is known in the art, including those disclosed in U.S. Patent Nos. RE44,305, 7,697,972, 8,644,907, and 8,842,893, and U.S. Patent Application Publication No. 2004 / 0199072, all of which are incorporated herein by reference.

[0041] The navigation system 10 may also include a navigation processor system 66. The navigation processor system 66 may include a display device 32, a TCA 40, a TCA controller and / or a power supply 52, and other components and / or connections thereto. For example, a wired connection may be provided between the TCA controller and / or power supply 52 and the navigation processor 14. Further, the navigation processor system 66 may have one or more user control inputs such as a keyboard 72 (or other user interface), and / or additional inputs such as those from integrated or communication systems with one or more memory systems (such as navigation memory 74). According to various embodiments, the navigation processor system 66 may include those disclosed in U.S. Patent Nos. RE44,305, 7,697,972, 8,644,907, and 8,842,893, and U.S. Patent Application Publication No. 2004 / 0199072, all of which are incorporated herein by reference, or may also include the commercially available StealthStation sold by Medtronic Navigation, Inc. of Littleton, Massachusetts. ®or Fusion ™ Surgical navigation system.

[0042] Tracking information (including information related to the magnetic fields sensed by the tracking devices 56, 60) can be transmitted via a communication system (such as a TCA controller and / or power supply 52, or possibly a tracking device controller) to a navigation processor system 66 including a navigation processor 14. Thus, the positioning of the tracked instrument 16 can be exemplified as icon 16i relative to image 30. Various other memories and processing systems may also be provided with and / or communicate with the navigation processor system 66, including a navigation memory 74 that communicates with the navigation processor 14 and / or image processor 12.

[0043] Image processor 12 can be incorporated into imaging system 11, such as O-arm. ® An imaging system, as discussed above. Therefore, imaging system 11 may include various components capable of movement within gantry 78, such as sources and X-ray detectors. Imaging system 11 may also be tracked by tracking device 80. However, it should be understood that the presence of imaging system 11 is not required when tracking a tracking device including instrument tracking device 56. Moreover, imaging system 11 can be any suitable imaging system, including MRI, CT, etc.

[0044] In various embodiments, the tracking system may include an optical locator 82. The optical locator 82 may include one or more cameras that observe or have a field of view defining or surrounding the navigation volume 50. The optical locator 82 may receive input light (e.g., infrared or ultraviolet light) to determine positioning or track a tracking device, such as an instrument tracking device 56. It should be understood that the optical locator 82 may be combined with and / or alternatively used to track the instrument 16, in conjunction with the EM locator 40.

[0045] Information from all tracking devices can be transmitted to the navigation processor 14 to determine the position of the tracked parts relative to each other and / or to position the instrument 16 relative to the image 30. The imaging system 11 can be used to acquire image data to generate or produce an image 30 of the subject 20. In one embodiment, the orientation information determined by the orientation module 13 is used for registration purposes and serves as the basis for performing tracking (such as tracking the instrument 16). As discussed above, the TCA controller 52 can be used to operate and power the EM locator 40.

[0046] The image 30 displayed using the display device 32 may be based on image data acquired from the subject 20 in various ways and orientation information determined by the orientation module 13. For example, the imaging system 11 may be used to acquire image data for generating the image 30. The orientation information may be used when the displayed image of the subject 20 is oriented on the display device 32 and / or relative to the device 16. However, it should be understood that other suitable imaging systems may be used to generate the image 30 using image data acquired using the selected imaging system. The imaging system may include a magnetic resonance imaging (MRI) scanner, a computed tomography (CT) scanner, and other suitable imaging systems. Furthermore, the acquired image data may be two-dimensional or three-dimensional data and may have a time-varying component, such as imaging the patient during heart rhythm and / or respiratory cycles.

[0047] In various implementations, the image data is 2D image data generated using a cone-beam imaging system (such as O-arm). ® This is part of an imaging system. The 2D image data can then be used to reconstruct a 3D image or model of the imaged subject (e.g., patient 20). The reconstructed 3D image and / or an image based on the 2D image data can be displayed. Therefore, those skilled in the art will understand that selected image data can be used to generate image 30.

[0048] Furthermore, an icon 16i, indicating the location of the tracked instrument 16, may be displayed on the display device 32 relative to the image 30. Additionally, the image 30 may be segmented for various purposes, including those further discussed herein. Segmentation of the image 30 may be used to identify and / or define objects or portions within the image. The definition may include or be made into a mask represented on the display. This representation may be displayed on the display, such as by overlaying a graphic of the mask, and may also be referred to as an icon. The icon may be a segmented mask. In various embodiments, the definition may be used to identify the boundaries of various portions within the image 30, such as the boundaries of one or more structures of the imaged patient (e.g., vertebrae 20v). Thus, the image 30 may include images of one or more vertebrae of vertebrae 20v, such as the first vertebra 20vi and the second vertebra 20vii. As further discussed herein, vertebrae such as the first vertebra 20vi and the second vertebra 20vii may be defined in the image, which may include and / or aid in defining boundaries in images such as 3D and 2D images. In various implementations, the boundary may be represented by icon 20vi' or a second icon 20vii'. The boundaries 20vi', 20vii' may be determined in an appropriate manner and for various purposes, which is also discussed further herein. Furthermore, as discussed herein, icons may be used to represent selected items for display, including the definition of objects, boundaries, etc.

[0049] According to various embodiments, image 30 can be segmented in a substantially automatic manner. In various embodiments, automatic segmentation may be incorporated into a neural network (such as a convolutional neural network (CNN)). The CNN can learn various features, such as objects (e.g., vertebrae) or parts of objects (e.g., pedicles), and the segmentation or boundaries of these objects and / or their parts. Selected segments may include segments identifying selected vertebrae such as first vertebra 20vi and second vertebra 20vii. Selected segments may be displayed on display device 32 along with selected graphical representations (such as segmentation icons or representations of 20vi' and 20vii').

[0050] These icons are displayed individually on display device 32 and / or superimposed on image 30 for observation by a selected user (such as user 15, who may be a surgeon or other appropriate clinician). Furthermore, once identified, boundaries or other appropriate portions, whether or not displayed as icons, can be used for various purposes. Boundaries can identify the physical dimensions of the vertebrae, the spatial positioning of the vertebrae (i.e., as discussed above, due to the registration of image 30 to subject 20), the possible trajectory of the identification (e.g., for implantation placement), etc. Therefore, image 30 can be used to plan and / or execute procedures, whether to display icons 20vi', 20vii', or simply to define the geometry of the boundaries without displaying them as icons.

[0051] Figure 2 A control system 200 is shown, which can be a tracking control system and can be used as... Figure 1 The navigation system 10 is part of a system such as an image processor 12 and / or a navigation processor 14. The tracking control system 200 includes a controller 201, which may include a processor 202, a navigation control module 216, and a degradation module 218 that implements a bench control module 204, a source module 206, a detector module 208, an image capture module 210, an EM transmission module 212, and a tracking module 214. The controller 201 may also implement a browsing module 217 (e.g., a DICOM...). ® (Browser application) and / or orientation module 219, the orientation module may be similar to Figure 1 Orientation module 13. Browsing module 217 can be accessed via a user interface (such as...) Figure 1 The user interface 72, 100 (one of the user interfaces) requests orientation information from the technician and stores the orientation information as orientation metadata in the memory of the processor 202 and / or elsewhere.

[0052] Orientation information may include indications regarding whether the subject and / or their anatomical object is prone or supine, whether the subject and / or their anatomical object is facing left or right, and whether the anatomical object is oriented in the up, down, forward, and / or back directions. Orientation information may be provided relative to a patient coordinate system and / or other coordinate systems. Similar information may be determined by orientation module 219. In one embodiment, orientation module 219 determines a vector of the anatomical object's facing direction. In another embodiment, controller 201 and... Figure 1 The image processor 12 communicates with the image processor and can receive orientation information from the orientation module 13.

[0053] The bench control module 204 can control positioning and determine Figure 1 The navigation system 10 positions the X-ray gantry. The gantry control module 204 controls the gantry motor 220. The source module 206 controls the X-ray source and / or its positioning, including controlling one or more source motors 222 of the source actuator and motor assembly 224. The detector module 208 controls the operation and / or positioning of the X-ray detector, including controlling one or more detector motors 226 of the detector actuator and motor assembly 228. The image capture module 210 controls the capture of images of the patient and / or the space surrounding the patient. The EM emission module 212 controls the generation and emission of electromagnetic signals via an EM coil (such as an EM coil mentioned herein). The EM emission module 212 can select frequencies and / or corresponding characteristics to, for example... Figure 1 The EM coil in the device 16 or positioner 40 is supplied with current.

[0054] Tracking module 214 can determine the position of a tool and / or instrument and / or a portion thereof based on received EM signals detected by EM coils on the tools, instruments, locators, and / or tracking devices mentioned herein. The locators and tracking devices may each include an EM coil. The EM coils of the tools, instruments, locators, and tracking devices may each operate as a transmitter or receiver of EM signals. EM signals are received and processed via tracking module 214. Tracking data associated with the EM signals may be stored in the memory of processor 202 or in other memory separate from processor 202. The tracking data may include data indicating and / or indicating the position of the tip of the tool, the distal end of the tool, and / or the distal end of the corresponding instrument housing (or attachment).

[0055] When the EM coil of the tool or instrument is activated (i.e., emits an EM signal), the controller 201 can receive the EM signal and / or sensor data via a cable, which indicates the EM signal generated or received by the EM coil. Based on the sensor data, Figure 1 The tracking system 44 and / or the navigation control module 216 of the navigation system 10 can display the position of the tip of the tool, the distal end of the tool, and / or the distal end of the instrument housing.

[0056] The navigation control module 216 can receive tracking data as input from the tracking system 44. The navigation control module 216 can also receive patient image data as input. The patient image data may include images of the patient's anatomical structures obtained from preoperative or intraoperative imaging devices, such as those obtained by... Figure 1 The imaging system 11 acquires images. Based on tracking data and patient image data, the navigation control module 216 can generate data for use in... Figure 1 The image data is displayed on the display device 32. The image data may include patient image data overlaid with icons of tools and / or instruments. The icons provide a graphical representation of the location, orientation, and trajectory of the tool tip, distal end of the tool, and / or distal end of the instrument housing relative to the patient's anatomy. Additionally, the icons may indicate the origin of the tool (e.g., an "X" adjacent to a bone in the anatomy) and the trajectory of the tool across the anatomy (i.e., a dashed line starting from an "X"). The current position of the tool may also be indicated by an icon (i.e., an "O" at the end of a dashed line). However, it should be understood that any suitable symbols, markings, etc., may be used to graphically represent the position and / or trajectory of the tool tip relative to the anatomy.

[0057] The tracking module 214 can receive start-up data from the navigation control module 216 and sensor data from the EM coils of the tools, instruments, locators, and / or tracking devices as input. The navigation control module 216 can receive tracking data and patient image data as input. The tracking data can indicate the position of each part of the tools, instruments, locators, and / or tracking devices in the patient space. Based on the tracking data, the navigation control module 216 determines the appropriate patient image data for display on the display device 32 and outputs both the tracking data and the patient image data together as image data.

[0058] In response to the tracking module 214 detecting movement of the distal end of an instrument (e.g., the distal end of instrument 230, the distal end of the tool of the instrument, or the distal end of other instruments or tools mentioned herein), the degradation module 218 can detect tool instability and / or breakage in order to replace the tool and / or instrument. The degradation module 218 actively monitors the position of the tool tip (or the distal end of the tool) and, based on this, actively adjusts the tool's speed, torque, stability, damping, and / or feed rate via a corresponding motor (e.g., motor 232 or other motors mentioned herein) and / or one or more actuators (e.g., actuator 234 or other actuators disclosed herein). Actuator 234 may include an attachment block, shaft, linkage, etc. This can occur during procedures such as when removing tissue and / or bone.

[0059] Figure 3 An example processor 300 is shown, which can represent Figures 1 to 2It is part of one of processors 12 and 202. Processor 300 includes orientation module 302, which is compatible with... Figures 1 to 2 Orientation modules 13 and 219 operate similarly. Orientation module 302 may include feature and pattern module 304, landmark module 306, anatomical structure module 308, and orientation verification module 310. Each of modules 302, 304, 306, 308, and 310 may be implemented as a neural network and / or implement a machine learning algorithm to learn features, patterns, landmarks, anatomical features, etc., based on the provided image data. In one embodiment, image data, along with correct features, patterns, landmarks, anatomical features, and / or orientation information, is provided to modules 302, 304, 306, 308, and / or 310 to allow these modules to learn the relationship between the image data and the correct features, patterns, landmarks, anatomical features, and orientation information. Landmarks may refer to different features of an anatomical object, such as: the nose or ear of the head; the pedicle of a vertebra; the sacrum; the first and last ribs near the T1 and T12 vertebrae, respectively; the odontoid process of the C2 vertebra, the aorta, and the vena cava; and / or other landmarks of other anatomical objects. Learning can be an ongoing process, and it improves further over time and with use. Neural networks can be implemented as CNNs. Once orientation information is determined, it can be used for registration, initialization, segmentation, image analysis, image merging, navigation, and more.

[0060] The feature and pattern module 304 analyzes the image to determine features and / or patterns that indicate the orientation of anatomical objects in the image. Features may include pixel intensity, color, contrast, etc. Patterns may include pixel patterns, the shape of detected object features (e.g., nose, ears, etc.), brightness or intensity patterns, pixel brightness levels at certain locations, etc.

[0061] Landmark module 306 is configured to identify landmarks such as the nose (or the tip of the nose) and ears (or the tip of the ears) on the head. Landmarks may refer to anatomical features with a specific set of well-known characteristics, such as shape, size, pixel intensity pattern, etc. Anatomical structure module 308 is configured to identify anatomical objects and / or different features, such as vertebral bodies, posterior structures, sacrum, and the odontoid process of the C2 vertebra (examples of which are in...). Figure 14 (as shown in the figure). In one embodiment, after detecting three landmarks, triangulation is performed based on the three landmarks to determine the orientation of the corresponding anatomical object. In another embodiment, an AI learning algorithm is implemented based on the detected landmarks to determine the orientation.

[0062] Orientation verification module 310 compares orientation metadata of one or more images with orientation information determined by orientation module 302 for the one or more images. This may include orientation direction, vector, etc. If no match is found, the orientation verification module, as further described below, may indicate a mismatch to a technician and / or automatically correct the orientation metadata. The correction of the orientation metadata may be based on predetermined criteria, examples of which are described below.

[0063] The processor 300 can store and / or access data stored in the memory 320. This includes image data 322, feature and pattern data 324, landmark data 326, anatomical data 328, orientation metadata 330, and historically determined orientation data 332. Data 324, 326, 328, 330, and 332 can be stored together with and / or with reference to the corresponding image data 322.

[0064] In one embodiment, orientation module 302 sets up imaging capture at low resolution for orientation determination purposes. As an example, one or more first images may be captured at low resolution to determine the orientation of one or more anatomical objects. Once the orientation is determined, the resolution can be increased for subsequent imaging. Subsequent images taken at higher resolution can be analyzed based on the determined orientation of one or more anatomical objects. The low-resolution images can be stored, accessed, and analyzed to determine orientation information. Lower-resolution images can be used to determine orientation information because orientation determination typically does not require fine image detail, while performing procedures and / or detecting irregularities in a patient requires fine image detail. In one embodiment, orientation determination is accurate in the range of 5° to 10°. In another embodiment, orientation determination is accurate in the range of 10° to 20°. In yet another embodiment, orientation determination is accurate in the range of 20° to 30°. Capturing low-resolution images also minimizes the amount of memory used. The low resolution used for orientation determination can be many times smaller than the image resolution of images used for other purposes.

[0065] In one implementation, a low-resolution version of the full 3D image is captured and / or accessed and analyzed, providing orientations such as LPS, RAI, etc., as output. In another implementation, one or more slices of the 3D image are analyzed, such as intermediate slices in sagittal, axial, and coronal views, to determine orientation. Intermediate slices may extend through the center point of the image volume and / or the center point of the imaged anatomical object. In another implementation, a MIP image is analyzed in sagittal, axial, and coronal views to determine orientation. Any of these operations in these example implementations may be performed to detect and identify landmarks and anatomical objects.

[0066] Orientation information can be initially, periodically, and / or subsequently acquired, verified, and / or corrected after certain operations, analyses, procedures, etc. For example, a patient can be instructed to move their head, thus ensuring an update of head orientation. This update can occur automatically or in response to user input. As an example, one or more sensors may be included, and these sensors detect movements made by the patient and / or the patient's movements. An example motion sensor 340 is shown, and this example motion sensor can be implemented as a camera, infrared sensor, ultrasonic sensor, radar sensor, and / or other motion and / or object detection sensors. More than one motion sensor may be included and monitored. Each time movement is detected, orientation information can be determined, verified, and / or updated.

[0067] Figure 4 It shows how to use Figure 3 The orientation module 302 determines the orientation of the subject's head in a first positioning diagram. The orientation module 302 may include conventional software algorithms and / or AI neural networks implementing AI algorithms and machine (or deep) learning algorithms. As an example, the orientation module 302 may receive a set of images 400, which may be CT images of a patient's head taken from different angles when the head is in a fixed position. The orientation module 302 analyzes the set of images and determines the head orientation. In the example shown, the head is in a left-front-upper (LAS) orientation.

[0068] Figure 5 It shows the method for direct use Figure 3 The orientation module 302 determines the orientation of the subject's head in a second positioning diagram. As an example, the orientation module 302 may receive a set of images 500, which may be CT images of the patient's head taken from different angles when the head is in a fixed position. The orientation module 302 analyzes the set of images and determines the head orientation. In the example shown, the head is in a right-back-up (RPS) orientation.

[0069] In one implementation, due to the symmetry of the head, the orientation module 302 directly detects the up / down and front / back directions based on image data. The orientation module 302 determines whether the head is facing left or right based on the known rotation of the image axes. Right rotation refers to the positive axis (x, y, or z) pointing towards the observer, while left rotation refers to the positive axis pointing away from the observer.

[0070] Figure 6 It shows how to use Figure 3The landmark module 306 determines landmarks on the subject's head for illustration purposes of orientation determination. The landmark module 306 receives a set of images 600A (which may be MRI images) and detects landmarks. In the example shown, the tip 602 of the nose and the tips 604, 606 of the ears on the head 608 are detected. The orientation module 302 can determine the orientation of the head based on the detected landmarks. This may include the identification of the left and right ears. This can be based on the location of the landmark, the type of landmark detected, the shape of the landmark, and / or its orientation, etc. Landmarks can be detected in one or more images. The same landmark can be detected in multiple images. Multiple detections of the same landmark can be used to increase the confidence level of the landmark's identification and / or orientation. The tip of the landmark can be designated as a point in image 600B. Image 600B is a modified version of image 600A.

[0071] Figure 7 It shows how to use Figure 3 The anatomical structure module 308 determines the subject's anatomical structures for visualization purposes. The anatomical structure module 308 may receive a set of images 700 of a portion of the subject's back 701. As an example, the images may be CT images of the subject's spinal region. In the example shown, the set of images 700 includes images of a portion of the subject's spinal region. The anatomical structure module 308 identifies vertebral bodies (or vertebrae) 702, posterior structures 704, the sacrum 706, and / or other anatomical objects, such as the odontoid process of the C2 vertebra (examples of which are shown in the image). Figure 14 (As shown in the figure). In one embodiment, the anatomical structure module 308 can trace, color, and / or pattern the detected vertebral bodies, structures, sacrum, and / or other anatomical objects as shown. The posterior structure 704 includes the pedicle to the spinous process. The positioning relationships between the vertebral body 702, the posterior structure 704, the sacrum 706, and / or other anatomical features (such as the odontoid process of the C2 vertebra) indicate the orientation of the subject. For example, the posterior structure 704 is always located posterior to the vertebral body 702, the sacrum is always the inferior vertebra, and the C2 vertebra with its distinctive odontoid process is always the superior vertebra. This allows the orientation module 302 to determine the anterior-posterior (AP) orientation and the superior-inferior (SI) orientation.

[0072] Figure 8 A 3D representative view of the subject's skull 800 in imaging volume 802 is shown. Figure 3 The orientation module 302 can receive a 3D image of the skull 800 and determine the orientation of the subject's skull 800 (or head) based on the image. In one embodiment, the orientation module 302 receives a 3D image and determines the orientation of the skull 800 based on features and / or patterns of the 3D image and / or detected landmarks and / or anatomical features in the 3D image. This can also be achieved using neural networks as described herein.

[0073] Figure 9 A 3D representative view of the subject's skull 800 is shown, illustrating an example of a 2D slice 900 that can be used to determine the orientation of the subject's head. Orientation module 302 may receive the 2D slice 900 and determine the orientation of the skull 800 based on features and / or patterns of the slice 900 and / or based on detected landmarks and / or anatomical features of the skull 800 in the slice 900.

[0074] Figures 10A to 10B The imaging process, including orientation verification and correction, is illustrated. The following operations can be performed at least by the browsing module (e.g., Figure 2 The browsing module 217) and the orientation module (e.g., Figures 1 to 3 (Execute one of the orientation modules 13, 219 or 302).

[0075] At 1000, orientation metadata is acquired via a browsing module. This can occur before or after positioning the subject within the imaging volume. The orientation metadata can be entered by a technician and stored in memory (e.g., Figure 3 In memory 320). This can be achieved via Figure 1 The display device 32 or other user interface is used for completion.

[0076] Image information may include any suitable image data such as CT image data, MRI data, X-ray cone-beam imaging data, etc. Furthermore, the imager can be any suitable imager, such as the O-arm discussed herein. ® Imaging systems or other imaging systems. O-arm ® The imaging system can be configured to acquire image data of 360 degrees around the subject and includes 2D image data and / or 3D reconstruction to provide a 3D image based on the 2D image data. Further, O-arm ® The imaging system can generate images using an X-ray cone beam. 2D image data or reconstructed 3D image data can originate from, for example, an imaging system 24 (which may include an O-arm). ® Imaging systems such as imaging systems. Imaging system 24 can generate two-dimensional image data in slice form, which can be used to reconstruct a three-dimensional model of one or more anatomical objects of the subject (e.g., head, vertebrae, etc.). Input image data can also be acquired at any appropriate time, such as during the diagnostic or planning phase, rather than in the operating room. Figure 1 Specifically, as shown. However, image data of the subject can be acquired using imaging system 24, and this image data can be input or accessed by the orientation module.

[0077] At 1002, the orientation module may perform an imaging process to capture one or more images and / or acquire one or more images from memory. The imaging process may include capturing one or more CT images, MRI images, and / or other images of the subject. The one or more images acquired from memory may include one or more CT images, MRI images, and / or other images of the subject. At 1004, if one or more images have been captured at 1002, the orientation module may store the images along with corresponding orientation metadata in memory, or may use the images for orientation determination purposes and then discard the images.

[0078] As an alternative to operations 1000, 1002, and / or 1004, the orientation module may access one or more images and, if available, access corresponding orientation metadata stored in memory, as indicated by operation 1006. The one or more images may include one or more CT images, MRI images, and / or other images.

[0079] At point 1008, the orientation module can use a first AI algorithm to analyze one or more images to generate image morphological data. As an example, image morphological data may include feature data, pattern data, fragment data, landmark markers, anatomical structure markers, and / or other data and / or image information mentioned herein. This may include a first neural network implementing a machine learning algorithm as described above.

[0080] Analysis of the data may include segmenting an image to identify one or more parts of it, such as identifying landmarks and / or anatomical objects. Neural networks (or artificial neural networks) can be used to automatically identify features of different parts of an image, such as pixel intensity, contrast ratio of pixel sets (each set consisting of one or more pixels), voxel intensity, feature boundaries, patterns, etc. This can be done to segment image data and focus processing on certain image regions and / or portions of an anatomical object. An artificial neural network may be a CNN. A CNN analyzes input image data to segment selected portions of the image data.

[0081] Landmarks, identified anatomical features and / or objects, boundaries, segmented parts, etc., can be displayed individually and / or in combination with corresponding images. Figure 1 On the display device 32 or other display. Landmarks, identified anatomical features, boundaries, divisions, etc., are stored in memory.

[0082] At point 1010, the orientation module can determine the orientation of one or more anatomical objects based on image morphological data. This can be implemented using conventional software algorithms, a first AI algorithm, and / or a second AI algorithm and a second neural network. Conventional software algorithms can be used to determine orientation based on image morphological (landmark) data from the first AI algorithm. The second neural network can be a CNN. Orientation information can include any orientation information mentioned herein.

[0083] As an alternative to operations 1008 and 1010, operation 1012 can be performed by the orientation module to directly determine the orientation of one or more anatomical objects based on one or more images using a neural network and a third AI algorithm.

[0084] In the above operations, the neural network involved may include: a dense neural network (or a neural network with dense layers); an Inception network (or a deep neural network consisting of repeating blocks, where the output of one block serves as the input to the next block); a neural network with a 3D U-Net architecture; and / or other neural networks. U-Net is a fully convolutional network (FCN). In other embodiments, the neural network may use a variety of machine learning algorithms. Machine learning algorithms may include one or more of the following: CNN algorithms, autoencoder algorithms, recurrent neural network (RNN) algorithms and transformer neural network algorithms, Swin transformer networks, visual transformer networks, generative adversarial networks (GAN) algorithms, linear regression, support vector machine (SVM) algorithms, random forest algorithms, hidden Markov models, and / or any combination thereof. For example, in some embodiments, at least one processor may be configured to utilize a combination of CNN algorithms and SVM algorithms.

[0085] At 1014, the orientation module can compare the determined orientation of one or more anatomical objects with the orientation metadata to verify the correctness of the orientation metadata. At 1016, the orientation module determines whether the determined orientation matches the orientation metadata. If the determined orientation does not match the orientation metadata, operation 1018 can be performed; otherwise, operation 1030 can be performed.

[0086] At position 1018, the orientation module can generate one or more confidence levels for the determined orientation. Confidence levels can be based on the resolution of the analyzed image, whether landmarks have been detected, which landmarks have been detected, whether anatomical objects have been identified, which anatomical objects have been identified, the type of features detected, etc. Confidence levels can each be values ​​between 0 and 1, where 0 is a 0% confidence level and 1 is a 100% confidence level. As an example, when landmarks and / or anatomical objects are detected and / or identified, the confidence level can be higher than a threshold. The more landmarks detected, the higher the confidence level.

[0087] At 1020, the orientation module can determine whether the confidence level is greater than a predetermined level. As an example, the predetermined threshold could be between 70% and 90%. If more than one confidence level is determined, the confidence levels can be i) weighted and summed, or ii) averaged. The weighted sum or average can then be compared to a predetermined quantity. If it is not greater than the predetermined quantity, operation 1022 can be performed; otherwise, operation 1024 can be performed.

[0088] At 1022, the orientation module can avoid altering the orientation metadata and / or provide an indication of orientation mismatch and the confidence level of the determined orientation.

[0089] At position 1024, the orientation module can indicate that an orientation mismatch has been detected and indicate the confidence level to the technician. This can be done via... Figure 1 The display device 32 or other user interface is used for this purpose.

[0090] At point 1026, the orientation module can determine whether approval for changing the orientation metadata has been received. If so, operation 1028 can be performed; otherwise, operation 1030 can be performed. In one embodiment, operation 1026 is not performed, and the orientation metadata is automatically changed to match the determined orientation. In another embodiment, the orientation module waits for indication of whether to maintain the current orientation metadata or change the usage input of the orientation metadata.

[0091] At position 1028, the orientation module modifies the orientation metadata to match the determined orientation.

[0092] At 1030, the orientation module can continue to perform image analysis, surgical procedures, and / or other operations based on the determined orientation and / or orientation metadata.

[0093] Figure 11 An example training procedure 1150 is illustrated. Although the following example training procedure includes segmentation, training can be performed without segmentation. Segmentation can be used for orientation determination and / or navigation purposes. Segmentation can be performed a first time to determine the orientation of one or more anatomical objects, and a second time to display one or more anatomical objects during the procedure. In one embodiment, the same segmentation result is used for both orientation and navigation. In another embodiment, segmentation is performed multiple times to provide different segmentation results for orientation determination and navigation.

[0094] The training phase process 1150 may begin with input and may include image data 1152, such as any image data mentioned herein. Selected image data may include low-resolution, medium-resolution, or high-resolution image data. In one embodiment, low-resolution data may be used when determining the general location of landmarks. High-resolution data is not required when determining the orientation of certain objects, such as a head. Input may also include a segmentation mask, such as a binary segmentation mask 1156. The segmentation mask 1156 may be a standard data segmentation or a training data segmentation, such as a gold standard or a user-defined segmentation. For example, a binary segmentation mask may include a segmentation of selected structures by a user (e.g., a trained expert, such as a surgeon), such as the selection of landmarks and / or anatomical features and / or objects.

[0095] After receiving image data 1152 and using mask 1156, selected steps including selected preprocessing may occur. For example, an optional resizing step in box 1160 may occur to resize the image data to an appropriate or selected size. In various embodiments, voxels may be resampled to a specific resolution, such as approximately 1.5mm × 1.5mm × 1.5mm. In box 1164, further preprocessing may include zero padding. Zero padding can be used to ensure that the image size is achieved after or during CNN processing, and also to ensure that selected enhancements keep all image data within the image boundaries.

[0096] In box 1168, the selected enhancements can also be selectively applied to the image data. Enhancements to the input data can include offline and / or online image data. Selected offline enhancements can involve randomly scaling the image along a selected axis by a selected scaling factor. The scaling factor can be between approximately 0.9 and approximately 1.1, but other suitable scaling factors may also be included. Further, the image can be randomly rotated about a selected axis by a selected amount. The selected rotation amount can include a rotation angle from -10 degrees to approximately +10 degrees. Online enhancements can involve randomly flipping or transposing image channels along different axes. The enhancements in box 1168 can aid in training the CNN by providing greater variability in the input image data 1152 than the image dataset itself provides. As discussed above and well known in the art, attempts are made to enable the CNN to generate filters that allow the automatic detection of selected features (such as the boundaries of segmented vertebrae) within the image data without additional input from the user. Therefore, the CNN can learn or learn the appropriate filters better or more efficiently by including data that is more random or highly random than the data provided by the initial image data.

[0097] In box 1172, the image data can then be normalized. When normalizing the image data, the variables are standardized to have zero mean and unit variance. This is done by subtracting the mean and then dividing the variable by its standard deviation.

[0098] A batch cropping or patching process may occur in box 1180. In various implementations, selected cropping may occur to achieve chosen results, such as reduced training time, reduced memory requirements, and / or finer-grained detail learning. For example, image data may be cropped to a selected size (e.g., half) to reduce the amount of image data used for training at one time. Corresponding portions of the segmentation mask are also cropped in box 1180 and provided in both the image and the mask. The cropped portions can then be combined to achieve the final output. The cropping process in box 1180 may also reduce memory requirements for analysis and / or training using the selected image dataset.

[0099] The image data, whether cropped from process 1180, can then be used as input to the CNN in box 384. As discussed above, the CNN can then determine filters to achieve the output. The output may include a probability map 1188 and a trained model 1190. The probability map 1188 is the probability that each voxel or other selected image element belongs to a selected marker or defined portion (e.g., a vertebra, a portion of a vertebra, a screw, or other selected portion in the input image). The input image may include various selectable portions, such as vertebrae, multiple vertebrae, screws, etc. In various embodiments, a threshold probability may be selected to identify or determine that a selected image portion is a selected portion or marker. However, it should be understood that a threshold is not required, and the probability map may output the selected probabilities in the output used for segmentation.

[0100] The trained model 1190 includes defined filters that can be applied as kernel K and can be based on probabilistic graph 1188. As discussed above, defined filters are used in various layers to identify important or significant parts of the image for various purposes, such as image segmentation. Therefore, the trained model 1190 can be trained based on input image 1152 and binary segmentation mask 1156. The trained model can then be stored or saved in a memory system, such as including navigation memory 74, for further access to, or implementation thereon, such as image memory 112 and / or navigation memory 74. The training process 1150 can include various inputs, such as padding amount or selected voxel size, but is typically performed by a processor (such as navigation processor 14) that executes selected instructions. For example, training of the CNN in box 1184 and training the model can be performed substantially by navigation processor 14.

[0101] However, it should be understood that trained models can also be provided on separate storage and / or processing systems for access and use at selected times. For example, trained models can be used during the planning phase of the procedure and / or during the procedure when the subject or part of them is in the operating room undergoing the implantation procedure.

[0102] Figure 12 An example segmentation method 1200 is illustrated. When attempting to determine the segmentation of an image (such as an image of an anatomical object), a trained model 1190 from training method 1150 can be used as input. Therefore, image data 1202 can be input along with the trained model 1190. As discussed above, the input image data 1202 and the trained model 1190 may include access to both the image data and the trained model stored in selected memory, such as the memory discussed above, via one or more processor systems in a processor system including one or more of the processors mentioned above.

[0103] Image data 1202 can be preprocessed in a manner similar to that used during training method 1150. For example, image data 1202 can be preprocessed in the same way as when training the trained model 1190. As discussed above, various preprocessing steps are optional and can be performed on image data 1152 during the training phase. During segmentation phase 1200, image data 1202 can be selectively preprocessed, or in a similar manner. Thus, in box 1160', the size of image data 1202 can be adjusted, in box 1164', zero padding can be added, and in box 1172', the image data can be normalized. It should be understood that if performed during training phase 1150, various preprocessing steps can be selected, and various preprocessing steps can be selected during segmentation phase 1200. The type of segmented image data is the same as the type of training image data.

[0104] After performing appropriate preprocessing in boxes 360', 364', and 1172', the image data 1202 can be segmented or cropped in box 1210. Segmentation of the image in box 1210 is also optional and can be selected based on processing time, memory availability, or other suitable characteristics. However, the image data 1202 can be segmented in a selected manner, such as along a selected axis. Once segmentation has occurred, the image data can then be merged, such as in post-processing step 1214.

[0105] Once the image data has been preprocessed, as selected, a CNN 1184 with learned weights and / or filters can be used to segment the image data 1202. Segmentation of the image data 1202 by the CNN 1184 creates an output 1220, which includes a probability map 1216 and a selected mask, such as a binary segmentation mask in output 1222. Output 1220 can be an identifier of selected geometry of the segmented portion, such as landmarks and / or anatomical features and / or objects. The CNN 1184, having been taught or learned with selected geometry, landmarks, features, anatomical objects, and / or weights, can segment portions of the image data.

[0106] In output 1222, probability map 1216 represents the probability that each voxel or other image element or portion belongs to a selected label or portion, such as a landmark, anatomical object (such as the head, spine, or vertebrae), and / or other object (such as a screw or other implanted object). Binary segmentation 1220 is generated from probability map 1216 by selecting all voxels or other image portions with probabilities greater than a threshold. The threshold can be any selected amount, such as approximately 30% to approximately 99%, encompassing approximately 35%. However, it should be further understood that performing binary segmentation 1220 based on probability map 1216 may not require a threshold.

[0107] The segmentation process 1200 may include various inputs, such as padding amount or selected voxel size, but is typically performed by a processor system (such as navigation processor system 66) that executes selected instructions. For example, segmentation using the CNN in box 1184 and output segmentation in box 1220 can be performed substantially by the processor system. Thus, the segmentation process 1200, or most of it, can be performed substantially automatically with the processor system executing selected instructions.

[0108] Then, output 1220 can be stored in a selected memory. Furthermore, output 1220 can be output as a graphical representation, such as one or more icons representing the geometry of the segmented portion. Figure 1 As illustrated, the segmented portions can be displayed individually or overlaid on the image. It should be understood that any appropriate number of segments can occur, and Figure 1 The two vertebrae shown are merely illustrative. For example, the image data could be the entire spine or all vertebrae of the subject. Therefore, the segmentation mask could contain the identifier of each vertebra. Furthermore, it should be understood that the segmentation can be three-dimensional segmentation, such that the entire three-dimensional geometry and structure of the vertebrae can be determined in the output 1220, and can be used for various purposes, such as for illustration on the display device 32.

[0109] like Figure 1The illustrated navigation system 10 can be used for various purposes, such as performing procedures on subject 20. Procedures may be performed based on orientation information received, determined, verified, and / or corrected herein. In various embodiments, procedures may include positioning an implant within the subject based on the subject's orientation, such as fixing a pedicle screw to one or more vertebrae in vertebrae 20v. During procedure performance, tool 16 may be an implant, such as a screw. It should be understood that performing or placing an implant may require various preparatory steps, such as passing a cannula through the soft tissue of subject 20, drilling into vertebrae 20v, tapping a hole in vertebrae 20v, or other appropriate procedures. It should further be understood that any item used for performing the various parts of the procedure may be tool 16, and tool 16 may also be an implant. Any part (e.g., implant, tool, or device) may be tracked by a tracking system of interest, such as simultaneously or sequentially, and may be navigated by navigation system 10. During navigation, navigation system 10 may display the location of tool 16 as icon 16i on display device 32. In a similar manner, other instruments can be navigated simultaneously with instrument 16, such that instrument 16 may include multiple instruments, and all or one instrument may be displayed individually or in multiples on display device 32 according to instructions such as those from user 15.

[0110] Figure 13 The operation is shown Figure 1 The example method of the navigation system 10 is thus applicable to a procedure performed by a user 15 on a subject 20. Further, the navigation processor 14 can execute instructions stored in a selected memory to perform or assist the user 15 in performing a procedure. The method includes various operations performed by one or more processors of the navigation system 10. One or more of these operations may be performed based on input received from the user 15.

[0111] The method may include data acquisition or access operations, which include operating the imaging system 24 (such as O-arm) in block 1300. ® An imaging system is used to perform image scanning, as indicated by box 1302, and to acquire image data of the subject. The image data can then be accessed or received. The image data can be accessed via a processor. Similarly, as further discussed herein, the image data can be analyzed and segmented using an automatic segmentation process such as that described above.

[0112] The processor can select a program at box 1304. The program selection can be based on input from user 15 and / or on the system's identification of user 15. User (or surgeon) preferences and / or operations that may be specific to the surgeon can be loaded from memory. The surgeon may have preferences that can include one or more of the following, such as specific instruments to prepare for the selected program, the size of the implant for the selected anatomical structure, etc. In various embodiments, for example, the selected surgeon may select an implant including a 3 mm gap relative to the boundary of the vertebra, such as a pedicle screw, while another surgeon may select an implant including a 5 mm gap. Thus, the selected surgeon with identified preferences can be used by the processor to select and / or identify instruments during navigation.

[0113] At 1306, the processor can automatically suggest instrument groups based on either or both of the selected procedures or identified surgeons. Automatically suggested instrument groups may include selecting or suggesting instrument tools, implants, etc. For example, regarding pedicle screw placement, the processor may suggest instrument (e.g., probes, awls, actuators, drills, and taps) and / or implant type and / or geometry and size (e.g., screw size and length). Suggestions for instrument and / or implant groups may be based on a selected algorithm that accesses a database of possible procedures and identifies tools from them. Furthermore, a machine learning system can be used to identify instrument groups based on various inputs such as procedures and surgeons, as the selected surgeon may choose different instruments and / or surgeon preferences (e.g., pedicle screw size) may vary or alter the selected instrument group. Instrument selection can also be heuristically based on segmentation as one of the inputs, or assisted by other methods. Regardless of whether an instrument group is automatically suggested, the instruments can be verified at 1308. Verification of the instruments ensures that they are present in the operating room and / or input into the navigation system 10. For example, the navigation system 10 can be used to identify a selected set of instruments or instrument types.

[0114] Typically, a selected tracking system is used to track the instruments in a navigation procedure. It should be understood that suitable tracking systems, such as the optical or EM tracking systems discussed above, can be used. Therefore, in various embodiments, instrument trackers can be identified in block 1310. The identification of instrument trackers can be substantially automatic based on trackers identified by a selected tracking system (such as optical locator 82). For example, an optical locator can be used to identify or “observe” tracking devices, such as instrument tracking device 56. It should be understood that multiple instruments may have multiple unique trackers on each instrument in the instrument, and therefore trackers of instruments can be identified by observing selected trackers. However, it should be understood that trackers can be variable and therefore may not be automatically detectable, and therefore manual identification of instrument trackers may be an option.

[0115] In box 1312, the tip associated with the selected instrument can be automatically identified. As discussed above, automatic identification of the tip can be used by “observing” the tip with optical locator 82. Therefore, the processor can use a deep learning system (such as CNN) to identify the tip relative to the instrument and / or tracker. Identifying the tip assists the procedure and the user in identifying selected features. Features can include the geometry of the tip during navigation and displayed on display device 32 (such as with instrument icon 16i). However, it should be understood that the tip can also be manually entered or identified in the selected procedure.

[0116] In the navigation procedure, the DRF 60 can also be used to track the patient 20. In box 1314, the DRF 60 can be placed or identified on the patient 20. It should be understood that placing the DRF 60 on the patient is typically a largely manual procedure performed by or at the instruction of the user 15. However, the placement of the DRF 60 may also include identifying or tracking the DRF 60 via the navigation procedure. Therefore, the navigation procedure may include tracking the DRF once it is placed on the patient 20.

[0117] DRF allows registration with image data input in box 1316. Registration allows the subject, or a physical space defined by subject 20, to be registered to the image data such that all points in the image data are associated with a physical location. Therefore, the position of the tracked instrument can be displayed on display device 32 relative to image 30. Further, registration can allow image portions to be registered to the patient, such as segmented portions. Registration may optionally include receiving orientation input from user 15 and storing the orientation input as orientation metadata. The orientation input indicates the orientation of one of the patient 20 or an anatomical object.

[0118] At 1318, the orientation module (such as any orientation module mentioned herein) may perform an orientation determination process to determine the orientation of at least one anatomical object of the patient, including optionally performing a first segmentation process. At 1320, the orientation module may optionally perform an orientation verification process to verify orientation metadata. Operations 1318 and 1320 may include, for example, performing the method of Figure 10 and / or one or more portions thereof. In one embodiment, the user does not input orientation information, orientation metadata is not stored, the processor automatically determines orientation information based on image data, and the determined orientation information is used as the basis for performing the following operations.

[0119] At 1322, the processor may optionally perform a second segmentation of the image data. The second segmentation of the image data may segment the image data differently from the first segmentation. For example, the first segmentation may be performed to detect and / or identify landmarks, and the second segmentation may be performed to identify a portion of the anatomical object where the implant will be at least partially located.

[0120] At 1324, the second segmentation (or segmentation) can be displayed on the display device 32. As discussed above, segmentation of the selected image portion can be performed via a CNN, as discussed above. However, in addition to or alternatively to the second segmentation discussed above, the second segmentation can also be physically tracked manually by the user 15 using selected instruments on the image 30 (such as tracked probes on the display device). However, automatic segmentation during navigation allows the user 15 to segment the vertebrae without using surgical or planning time, and allows for a faster and more efficient procedure. A faster and more efficient procedure is achieved by saving the surgeon time spent manually interacting with the navigation system 10, which includes its various software features, such as automatically selecting the correct tool projection based on the segmentation.

[0121] The second segmentation can also be displayed at 1324, including the display of segmentation icons (e.g., icons 20vi' and 20vii'). The segmentation icons can be observed by user 15 and verified that they cover the selected vertebrae. In addition to or as part of verification, at 1326, image portions can also be identified and / or labeled. Labeling of image portions can be manual, such as user 15 selecting and labeling each vertebra in image 30, including the segmented portion therein. Labeling and / or identification of vertebrae can also be semi-automatic, such as user 15 identifying one or fewer of all vertebrae in image 30, and the processor labeling all other vertebrae relative to it. Finally, the identification and labeling of vertebrae at 1326 can be substantially automatic, where the processor executes instructions such as CNN-based instructions to identify selected and / or all vertebrae in image 30 and thus display labels relative to the segmented portions (e.g., segmentation icons 20vi' and 20vii').

[0122] At 1328, during the procedure or planning phase, the navigation process can also automatically select implant parameters, such as size (e.g., length and width). As discussed above, the vertebrae can be segmented according to the selected procedure. When segmenting the vertebrae, the size of the vertebrae, including their three-dimensional geometry, including size and shape, is known. This can assist in selecting the implant size based on the segmented vertebrae size or the determined vertebrae size. Moreover, based on the preferences of the selected surgeon, the size of the vertebrae relative to the implant is also known, and therefore will also assist in the automatic selection of implant parameters for a specific size. The size can be output, for example, on display device 32, for user 15 to select and / or confirm. Thus, implant parameters including size or other geometries can be selected by the processor.

[0123] The procedure may include assisting in the preparation and / or placement of a selected implant. For placement of the implant (such as a pedicle screw), an entry point into the patient 20 may be determined relative to the anatomical object. The device 16 may include a probe with a device tracking device 56 (e.g., an optical tracker). The probe may be movable relative to the subject 20, such as without puncturing the subject 20.

[0124] When attempting to determine the entry point at 1330, the probe may move relative to the anatomical object. The anatomical object already identified and / or marked at 1326 may be identified based on projections from the probe (such as from the distal end of the tracked probe). The probe tip does not need to pierce the soft tissue (such as skin) of the subject 20; instead, the projection may be identified and / or displayed, for example, using an instrument icon 16i on the display device 32. The instrument icon 16i may vary based on the selected instrument and may be displayed as a projection of the probe or simply its trajectory, based on the probe's positioning relative to the anatomical object (e.g., vertebra 20v). Based on the probe's projection, the anatomical object can be identified in the image on the display device 32. The display device 32 may display the image in various ways, such as in intermediate and axial views.

[0125] The projection of the instrument icon 16i may be based on the boundaries of the anatomical object, such as based on the segmentation of the anatomical object, as described above. However, the projection may be limited to the boundaries of the anatomical object and may be displayed alone or in combination with corresponding icons. The projected instrument icon 16i may be based on the geometry of the selected tool (such as a drill), allowing the user 15 to observe the physical extent of the drill relative to the image and the segmented anatomical object or portions thereof, ensuring that the drill penetrates sufficiently into the anatomical object.

[0126] In various implementations, the entry point feature can then be used to identify or mark points on the skin of subject 20. It should be understood that marking incision points is not required. However, as discussed above, after the entry point is found, an incision may be made to allow other instruments to enter subject 20. Once the incision is made, tools can be navigated, including tracking tools, and the location of the tools is illustrated on display device 32, as specified by box 1332. For example, after the initial incision is made, an awl can be navigated to the anatomical object identified at 1330. Tools may also be referred to as instruments.

[0127] When navigating the awl relative to the anatomical object, the awl can penetrate an incision in the skin and contact the vertebrae. At selected times, such as when the awl is within a selected distance from the anatomical object (e.g., less than about 1 mm to about 6 mm, including about 5 mm), an icon representing the awl or a projection from the position of the tracked awl can be displayed relative to the anatomical object. Thus, the icon representing the tool can automatically display the selected implant size, such as an icon superimposed on an image on display device 32.

[0128] At 1334, automatically displaying the implant size or tool size or positioning may include determining the implant size based on the boundaries of the segmented anatomical object. The navigation process may include executing instructions based on the segmented image geometry, including size and shape, to automatically select the anatomical object and display it on display device 32, along with an implant of the selected size, optionally. At 1336, user 15 may confirm and / or change the selected implant size. If a change is made, a different size implant may be displayed relative to the image and / or segmentation of the anatomical object. User 15 can then observe the automatically displayed implant size and / or the changed or confirmed size.

[0129] Furthermore, user 15 can move the tracked cone relative to the anatomical object to select the implant's positioning relative to the object. For example, different positioning of the cone relative to the object can allow the system to determine or calculate implants of different sizes. Once user 15 has selected an appropriate or chosen trajectory via an input device, the trajectory can be saved at 1338. The input device can be a verbal command, gesture, foot switch, etc., for audio input. Additionally, the user's selection can be saved based on a selected surgeon for further or future reference.

[0130] The projection can be saved for future use, and the projection can be selectively shown and / or hidden to allow guidance of tapping the anatomical object. When navigating the tap, the anatomical object can be tapped with the tap while viewing the display device 32. The tap can be navigated when moved relative to the anatomical object by the user 15. The tap can be displayed as icon 16i on the display device 32 relative to the image. Furthermore, at selected times, such as when the tap approaches or contacts the anatomical object, or when the tap is navigated to the anatomical object at 1340, a projection of the geometry of the tapping can be displayed relative to an image including the anatomical object, and the display at the projection of the tapping area or volume allows the user 15 to confirm, based on the projection of the tap onto the anatomical object, that the selected tapping volume matches the implant projection or the saved implant projection.

[0131] Once the tap projection is confirmed to match the saved implant projection, the tap can be driven into the anatomical object. Reducing or shrinking the tap projection at 1342 allows the user 15 to observe the degree of tapping relative to the projected or selected tap length volume. The reduced tap geometry allows the user 15 to understand the degree of tapping performed so far and the remaining automatic tapping. Therefore, the user can slow the tap's penetration speed into the vertebra at selected time intervals, while allowing for rapid and efficient tapping in the initial time interval.

[0132] It should be understood that the navigation processor system 66 can substantially automatically reduce the projection of the tapping based on the navigation of the tap relative to the anatomical object, such as the reduced tap projection at 1342. The tap projection is initially based on the projection of a selected implant (such as an automatically selected implant). Therefore, the navigation process allows for efficient tapping of the anatomical object by allowing the user 15 to observe the tapping in progress and confirm when the tapping is completed.

[0133] Upon completion of tapping, a reverse projection can be automatically determined and displayed at position 1346. The reverse projection can be substantially equivalent to or equal to the tapping depth into the anatomical object and is based on the amount or depth of tapping by user 15. Further, the reverse projection can be substantially equivalent to the initial tapping projection. The reverse tap projection can be maintained for user 15 to observe on display device 32 relative to the anatomical object to position the implant within it. Additionally, the instrument icon 16i can be a combination of the instrument portion and the projection of the now-fixed or permanent tapping. The fixed projection can initially be equivalent to the reverse projection and allows user 15 to observe both the volume (e.g., width and / or depth) of the tapping relative to the instrument icon 16i and an image of the anatomical object.

[0134] For various purposes, the reverse projection at 1346 can be preserved, as discussed above, to guide or navigate the implant. Furthermore, the preserved reverse projection can be equivalent to the location of the tapping and can also be preserved below the selected surgeon for further reference and / or future reference.

[0135] Once tapping has been performed on the anatomical object, an implant can be placed within it. The implant may include a screw, such as a pedicle screw, positioned within the anatomical object. The screw and driver may be illustrated as icons on the display device 32 relative to an image of the anatomical object. A back projection may also be displayed to assist in navigating the implant at 1348. The implant may be illustrated as at least a portion of the icon, such that the icon is aligned with the back projection to allow the screw to be driven or placed into the anatomical object along the trajectory and volume of tapping as illustrated by the back projection. Thus, navigating the implant allows the user 15 to position the implant at a selected and tapped location within the anatomical object.

[0136] Tracking screws that have entered the anatomical object also allows the location of the tracked screw to be saved at 1350 for future use by selected surgeons. Therefore, various features, such as the location of the screw's strike and final position, as well as other features, such as the screw's geometry and size, can be saved for future reference by selected surgeons.

[0137] After locating the screw by navigating the implant and / or preserving the tracked screw position, it can be determined at 1352 whether additional implants (e.g., screws) need to be placed. If no additional implants are needed, the procedure can end. The completion procedure may include vertebral decompression, removal of the device from the subject 20, closure of the incision, or other suitable features.

[0138] If it is determined that another implant needs to be placed, operation 1354 can be performed, followed by operation 1328. The determination of whether to place an additional implant can be based on a selected procedure or on user input. Therefore, determining whether to position an additional implant can be substantially automatic or manual.

[0139] At 1354, automatic switching to another image portion can optionally occur. For example, if the first screw is placed in the L5 vertebra, the second screw can be placed in the second side of the L5 vertebra. Automatic switching to a separate image or view portion of the vertebra can assist the user 15. Further, if the second implant is positioned in the L5 vertebra and the selected procedure is to fuse the L5 and L4 vertebrae, the image can automatically switch to display or more closely display the L4 vertebra for further procedural steps. Therefore, automatic switching to another image portion can assist the user 15 in performing the procedure efficiently.

[0140] Regardless of whether the optional automatic switch to additional image steps is performed, determining the placement of additional implants may include cyclically returning to operation 1328 to automatically select the parameters of the next implant and continue the procedure. It should be understood that various other parts of the procedure, such as marking the instrument or tip, may also be repeated, but such applications may not be necessary, especially if the instrument is maintained or remains unchanged for multiple implants. However, by continuing the procedure until no further implants are needed, a selected number of implants can be positioned within the subject.

[0141] Figure 14 A cross-sectional image 1400 is shown that can be used to determine the anatomical structures of a subject for orientation determination purposes. This determination can be achieved via, for example, as described above. Figure 3The anatomical structure module 308 is used for this purpose. The anatomical structure module 308 can receive a set of images of a portion of the subject's neck 1401. As an example, image 1400 can be a CT image of the subject's spinal region. In the example shown, image 1400 includes a portion of the spinal region of the subject's neck 1401. The anatomical structure module 308 can identify vertebral bodies (or vertebrae) 1402, the odontoid process 1404 of the C2 vertebra, and / or other anatomical objects. In one embodiment, the anatomical structure module 308 can trace, color, and / or pattern the vertebral bodies, structures, sacrum, and / or other anatomical objects detected as shown. The posterior structure 704 includes the pedicle to the spinous process. The positioning relationships between the vertebral body 702, the posterior structure 704, the sacrum 706, and / or other anatomical features (such as the odontoid process of the C2 vertebra) indicate the subject's orientation. For example, the posterior structure 704 is always located posterior to the vertebral body 702, the sacrum is always the inferior vertebra, and the C2 vertebra with its distinctive odontoid process is always the superior vertebra. This allows the orientation module 302 to determine the forward-backward (AP) direction and the up-down (SI) direction.

[0142] Example

[0143] An imaging system is disclosed, comprising a memory, an orientation module, and at least one processor. The memory is configured to store orientation metadata and images of a subject. The orientation module includes at least one neural network configured to implement at least one artificial intelligence algorithm to analyze one or more images in the memory and determine the orientation of at least one anatomical object of the subject based on the analysis. The at least one processor is configured to perform at least one of the following based on the determined orientation: verify the orientation metadata, correct the orientation metadata, analyze one or more second images, and track the position of at least one of a tool and an implant relative to the patient.

[0144] Among other features, the determined orientation is relative to at least one of a reference point and a structure supporting at least a portion of the subject.

[0145] Among other features, the at least one artificial intelligence algorithm includes at least one of machine learning algorithms and deep learning algorithms. Among other features, the at least one artificial intelligence algorithm includes at least one of dense networks, Inception networks, visual transformer networks, and fully convolutional networks.

[0146] Among other features, the orientation module is configured to: implement the at least one artificial intelligence algorithm to detect at least one landmark of the at least one anatomical object, and determine the orientation of the at least one anatomical object based on the at least one landmark.

[0147] Among other features, the orientation module is configured to: implement the at least one artificial intelligence algorithm to identify the at least one anatomical object, and determine the orientation of the at least one anatomical object based on the identification. Among other features, the at least one artificial intelligence algorithm includes at least one of machine learning algorithms and deep learning algorithms. Among other features, the at least one artificial intelligence algorithm includes at least one of dense networks, Inception networks, visual transformer networks, and 3D fully convolutional networks.

[0148] Among other features, the orientation module is configured to implement the at least one artificial intelligence algorithm to determine the orientation of the at least one anatomical object directly from the image.

[0149] Among other features, the orientation module is configured to: compare the determined orientation with the orientation metadata, and correct the orientation metadata in response to determining that there is an orientation mismatch between the determined orientation and the orientation metadata.

[0150] Among other features, the orientation module is configured to: compare the determined orientation with the orientation metadata, and in response to determining that there is an orientation mismatch between the determined orientation and the orientation metadata, indicate via a user interface that a mismatch has been detected, and wait for approval to change the orientation metadata.

[0151] Among other features, the orientation module is configured to: determine a confidence level for the determined orientation, and, in response to determining that there is an orientation mismatch between the determined orientation and the orientation metadata, correct the orientation metadata based on the confidence level.

[0152] Among other features, the at least one processor is configured to verify the orientation metadata based on the determined orientation. Among other features, the at least one processor is configured to correct the orientation metadata based on the determined orientation.

[0153] Among other features, the at least one processor is configured to analyze the second or more images based on the determined orientation.

[0154] Among other features, the at least one processor is configured to track the position of the tool and the at least one of the implants relative to the patient based on the determined orientation.

[0155] Among other features, the processor is configured to: analyze the first one or more images having a first resolution to determine the orientation of the at least one anatomical object, and analyze the second one or more images having a second resolution based on the determined orientation. The first resolution is smaller than the second resolution.

[0156] Among other features, the first one or more images include three-dimensional images. Among other features, the first one or more images include two-dimensional images.

[0157] Among other features, the orientation module is configured to: capture a first slice in a sagittal view of the at least one anatomical object and access at least one of the first slices in the sagittal view of the at least one anatomical object; capture a second slice in an axial view of the at least one anatomical object and access at least one of the second slices in the axial view of the at least one anatomical object; capture a third slice in a coronal view of the at least one anatomical object and access at least one of the third slices in the coronal view of the at least one anatomical object; and determine the orientation of the at least one anatomical object based on the first slice, the second slice, and the third slice. Among other features, the first slice, the second slice, and the third slice are intermediate slices.

[0158] Among other features, the orientation module is configured to: capture a first maximum intensity projection image in a sagittal view of the at least one anatomical object and access at least one of the first maximum intensity projection images in the sagittal view of the at least one anatomical object; capture a second maximum intensity projection image in the axial direction of the at least one anatomical object and access at least one of the second maximum intensity projection images in the axial direction of the at least one anatomical object; capture a third maximum intensity projection image in a coronal view of the at least one anatomical object and access at least one of the third maximum intensity projection images in the coronal view of the at least one anatomical object; and determine the orientation of the at least one anatomical object based on the first maximum intensity projection image, the second maximum intensity projection image, and the third maximum intensity projection image.

[0159] Example 1. An imaging system, the imaging system comprising: A memory configured to store subject orientation metadata and images; An orientation module, comprising at least one neural network configured to: implement at least one artificial intelligence algorithm to analyze one or more first images in the images, and determine the orientation of at least one anatomical object of the subject based on the analysis; and At least one processor is configured to perform at least one of the following based on the determined orientation: verifying the orientation metadata, correcting the orientation metadata, analyzing a second or more images, and tracking the position of at least one of a tool and an implant relative to the patient.

[0160] Example 2. The imaging system according to Example 1, wherein the determined orientation is relative to at least one of a reference point and a structure supporting at least a portion of the subject.

[0161] Example 3. The imaging system according to Example 1, wherein the at least one artificial intelligence algorithm includes at least one of machine learning algorithm and deep learning algorithm.

[0162] Example 4. The imaging system according to Example 1, wherein the at least one artificial intelligence algorithm includes at least one of dense networks, Inception networks, visual transformer networks, and fully convolutional networks.

[0163] Example 5. The imaging system according to Example 1, wherein the orientation module is configured to: implement the at least one artificial intelligence algorithm to detect at least one landmark of the at least one anatomical object, and determine the orientation of the at least one anatomical object based on the at least one landmark.

[0164] Example 6. The imaging system according to Example 5, wherein the at least one artificial intelligence algorithm includes at least one of machine learning algorithm and deep learning algorithm.

[0165] Example 7. The imaging system according to Example 5, wherein the at least one artificial intelligence algorithm includes at least one of dense networks, Inception networks, visual transformer networks, and fully convolutional networks.

[0166] Example 8. The imaging system according to Example 1, wherein the orientation module is configured to: implement the at least one artificial intelligence algorithm to identify the at least one anatomical object, and determine the orientation of the at least one anatomical object based on the identification.

[0167] Example 9. The imaging system according to Example 8, wherein the at least one artificial intelligence algorithm includes at least one of machine learning algorithm and deep learning algorithm.

[0168] Example 10. The imaging system according to Example 8, wherein the at least one artificial intelligence algorithm includes at least one of dense networks, Inception networks, and fully convolutional networks.

[0169] Example 11. The imaging system according to Example 1, wherein the orientation module is configured to implement the at least one artificial intelligence algorithm to directly determine the orientation of the at least one anatomical object from the image.

[0170] Example 12. The imaging system according to Example 1, wherein the orientation module is configured to: compare the determined orientation with the orientation metadata, and correct the orientation metadata in response to determining that there is an orientation mismatch between the determined orientation and the orientation metadata.

[0171] Example 13. The imaging system according to Example 1, wherein the orientation module is configured to: compare the determined orientation with the orientation metadata, and in response to determining that there is an orientation mismatch between the determined orientation and the orientation metadata, indicate via a user interface that a mismatch has been detected, and wait for approval to change the orientation metadata.

[0172] Example 14. The imaging system according to Example 1, wherein the orientation module is configured to: determine a confidence level of the determined orientation, and in response to determining that there is an orientation mismatch between the determined orientation and the orientation metadata, correct the orientation metadata based on the confidence level.

[0173] Example 15. The imaging system according to Example 1, wherein the at least one processor is configured to verify the orientation metadata based on the determined orientation.

[0174] Example 16. The imaging system according to Example 1, wherein the at least one processor is configured to correct the orientation metadata based on the determined orientation.

[0175] Example 17. The imaging system according to Example 1, wherein the at least one processor is configured to analyze the second or more images based on the determined orientation.

[0176] Example 18. The imaging system according to Example 1, wherein the at least one processor is configured to: track the position of at least one of the tool and the implant relative to the patient based on a determined orientation.

[0177] Example 19. The imaging system according to Example 1, wherein: The processor is configured to: analyze the first one or more images having a first resolution to determine the orientation of the at least one anatomical object, and analyze the second one or more images having a second resolution based on the determined orientation; and The first resolution is smaller than the second resolution.

[0178] Example 20. The imaging system according to Example 19, wherein the first one or more images include three-dimensional images.

[0179] Example 21. The imaging system according to Example 19, wherein the first or more images include two-dimensional images.

[0180] Example 22. The imaging system according to Example 1, wherein the orientation module is configured as follows: Capturing a first slice in a sagittal view of the at least one anatomical object and accessing at least one of the first slice in a sagittal view of the at least one anatomical object; Capturing a second slice along the axis of the at least one anatomical object and accessing at least one of the second slices along the axis of the at least one anatomical object; Capturing a third slice in the coronal view of the at least one anatomical object and accessing at least one of the third slice in the coronal view of the at least one anatomical object; and The orientation of the at least one anatomical object is determined based on the first slice, the second slice, and the third slice.

[0181] Example 23. The imaging system according to Example 22, wherein the first slice, the second slice and the third slice are intermediate slices.

[0182] Example 24. The imaging system according to Example 1, wherein the orientation module is configured as follows: Capturing a first maximum intensity projection image in a sagittal view of the at least one anatomical object and accessing at least one of the first maximum intensity projection images in a sagittal view of the at least one anatomical object; Capturing a second maximum intensity projection image in the axial direction of the at least one anatomical object and accessing at least one of the second maximum intensity projection images in the axial direction of the at least one anatomical object; Capturing a third maximum intensity projection image in the coronal view of the at least one anatomical object and accessing at least one of the third maximum intensity projection images in the coronal view of the at least one anatomical object; and The orientation of the at least one anatomical object is determined based on the first maximum intensity projection image, the second maximum intensity projection image, and the third maximum intensity projection image.

[0183] Example implementations are provided. Numerous specific details, such as examples of particular components, apparatus, and methods, are set forth to provide an understanding of embodiments of this disclosure. It will be apparent to those skilled in the art that specific details are not required, example implementations may be embodied in many different forms, and should not be construed as limiting the scope of this disclosure. In some example implementations, well-known processes, well-known apparatus structures, and well-known techniques are not described in detail.

[0184] Instructions can be executed by a processor and can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuitry" covers a single processor circuitry that executes some or all of the code from multiple modules. The term "group processor circuitry" covers processor circuitry that, in conjunction with additional processor circuitry, executes some or all of the code from one or more modules. References to multiple processor circuitry cover multiple processor circuitry on a discrete die, multiple processor circuitry on a single die, multiple cores of a single processor circuitry, multiple threads of a single processor circuitry, or a combination thereof. The term "shared memory circuitry" covers a single memory circuitry that stores some or all of the code from multiple modules. The term "group memory circuitry" covers memory circuitry that, in conjunction with additional memory, stores some or all of the code from one or more modules.

[0185] The apparatus and methods described in this application may be implemented, in part or in whole, by one or more processors (also referred to as processor modules), which may include a dedicated computer (i.e., created by configuring one or more processors) for performing one or more specific functions embodied in a computer program. The computer program includes processor-executable instructions stored on at least one non-transitory, tangible computer-readable medium. The computer program may also include or depend on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of the dedicated computer, device drivers that interact with specific devices of the dedicated computer, one or more operating systems, user applications, background services, background applications, etc.

[0186] Computer programs may include: (i) assembly code; (ii) object code generated from source code by a compiler; (iii) source code for execution by an interpreter; (iv) source code for compilation and execution by a just-in-time (JIT) compiler; and (v) descriptive text for parsing, such as HTML (Hypertext Markup Language) or XML (Extensible Markup Language). As an example only, source code may be in C, C++, C#, Objective-C, Haskell, Go, SQL, Lisp, or Java. ® ASP, Perl, Javascript ® HTML5, Ada, Active Server Pages (ASP), Perl, Scala, Erlang, Ruby, Flash ® Visual Basic ® Lua or Python ® To write it.

[0187] Communication may include the wireless communications described in this disclosure, which may be wholly or partially compliant with IEEE Standard 802.11-2012, IEEE Standard 802.16-2009, and / or IEEE Standard 802.20-2008. In various specific implementations, IEEE 802.11-2012 may be supplemented by draft IEEE Standard 802.11ac, draft IEEE Standard 802.11ad, and / or draft IEEE Standard 802.11ah.

[0188] The terms processor, processor module, module, or “controller” are used interchangeably herein (unless otherwise specifically indicated), and each may be replaced by the term “circuit”. Any of these terms may refer to, be part of, or include: application-specific integrated circuit (ASIC); digital, analog, or mixed-signal analog / digital discrete circuit; digital, analog, or mixed-signal analog / digital integrated circuit; combinational logic circuit; field-programmable gate array (FPGA); processor circuitry (shared, dedicated, or grouped) that executes code; memory circuitry (shared, dedicated, or grouped) that stores code executed by the processor circuitry; other suitable hardware components that provide the described functionality; or some or all of the foregoing, such as in a system-on-a-chip.

[0189] Instructions can be executed by one or more processors or processor modules, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the terms "processor" or "processor module" as used herein may refer to any of the foregoing structures or any other physical structure suitable for implementing the described techniques. Furthermore, these techniques can be fully implemented in one or more circuit or logic elements.

[0190] The foregoing description of embodiments has been provided for illustrative and descriptive purposes. The foregoing description is not intended to be exhaustive or limiting of the invention. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but are interchangeable and may also be used in chosen embodiments where applicable, even if not specifically shown or described. The same element or feature may be varied in many ways. Such variations are not considered to depart from the invention, and all such modifications are intended to be included within the scope of the invention.

Claims

1. An imaging system, the imaging system comprising: A memory configured to store subject orientation metadata and images; An orientation module, the orientation module including at least one neural network, the at least one neural network being configured to: implement at least one artificial intelligence algorithm to analyze a first one or more images in the images, and determine the orientation of at least one anatomical object of the subject based on the analysis; and At least one processor is configured to perform at least one of the following based on the determined orientation: verifying the orientation metadata, correcting the orientation metadata, analyzing a second or more images, and tracking the position of at least one of the tools and implants relative to the patient.

2. The imaging system of claim 1, wherein the determined orientation is relative to at least one of a reference point and a structure supporting at least a portion of the subject.

3. The imaging system according to claim 1, wherein the at least one artificial intelligence algorithm includes at least one of machine learning algorithms and deep learning algorithms.

4. The imaging system according to claim 1, wherein the at least one artificial intelligence algorithm comprises at least one of dense networks, Inception networks, and fully convolutional networks.

5. The imaging system of claim 1, wherein the orientation module is configured to: implement the at least one artificial intelligence algorithm to detect at least one landmark of the at least one anatomical object, and determine the orientation of the at least one anatomical object based on the at least one landmark.

6. The imaging system of claim 1, wherein the orientation module is configured to: implement the at least one artificial intelligence algorithm to identify the at least one anatomical object, and determine the orientation of the at least one anatomical object based on the identification.

7. The imaging system of claim 1, wherein the orientation module is configured to: implement the at least one artificial intelligence algorithm to determine the orientation of the at least one anatomical object directly from the image.

8. The imaging system of claim 1, wherein the orientation module is configured to: compare the determined orientation with the orientation metadata, and correct the orientation metadata in response to determining that there is an orientation mismatch between the determined orientation and the orientation metadata.

9. The imaging system of claim 1, wherein the orientation module is configured to: compare the determined orientation with the orientation metadata, and in response to determining that an orientation mismatch exists between the determined orientation and the orientation metadata, indicate via a user interface that a mismatch has been detected, and wait for approval to change the orientation metadata.

10. The imaging system of claim 1, wherein the orientation module is configured to: determine a confidence level of the determined orientation, and, in response to determining that there is an orientation mismatch between the determined orientation and the orientation metadata, correct the orientation metadata based on the confidence level.

11. The imaging system of claim 1, wherein the at least one processor is configured to verify the orientation metadata based on the determined orientation.

12. The imaging system of claim 1, wherein the at least one processor is configured to: correct the orientation metadata based on the determined orientation.

13. The imaging system of claim 1, wherein the at least one processor is configured to analyze the second or more images based on the determined orientation.

14. The imaging system of claim 1, wherein the at least one processor is configured to: track the position of at least one of the tool and the implant relative to the patient based on the determined orientation.

15. The imaging system according to claim 1, wherein: The processor is configured to: analyze the first one or more images having a first resolution to determine the orientation of the at least one anatomical object, and analyze the second one or more images having a second resolution based on the determined orientation; Furthermore, the first resolution is smaller than the second resolution.

Images