System and method for imaging and registration for navigation

Abstract

A system for assisting in guiding and performing a procedure on a subject is disclosed. The subject may be any suitable subject, such as a non-living subject and / or a living subject. An imaging system may be used to image a subject. The system may include various steerable or movable components, such as robotic systems, and may be used to move and position imaging systems.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 459,306, filed April 14, 2023; U.S. Provisional Application No. 63 / 459,308, filed April 14, 2023; U.S. Provisional Application No. 63 / 459,318, filed April 14, 2023; and U.S. Provisional Application No. 63 / 459,354, filed April 14, 2023. The entire disclosures of the above applications are incorporated herein by reference.

[0003] This application includes subject matter similar to that disclosed in U.S. Patent Application No. 18 / 630,285, filed April 9, 2024; U.S. Patent Application No. 18 / 630,290, filed April 9, 2024; U.S. Patent Application No. 18 / 630,317, filed April 9, 2024; U.S. Patent Application No. 18 / 630,336, filed April 9, 2024; and U.S. Patent Application No. 18 / 630,348, filed April 9, 2024. The entire disclosures of the above applications are incorporated herein by reference. TECHNICAL FIELD

[0004] The subject disclosure relates generally to a tracking and navigation system, and in particular to imaging and registering coordinate systems. BACKGROUND

[0005] This section provides background information relating to the present disclosure, which is not necessarily prior art.

[0006] An instrument can be navigated relative to a subject for various procedures. For example, the subject can include a patient on which a surgical procedure is being performed. During the surgical procedure, the instrument can be tracked in an object or subject space. In various embodiments, the subject space can be a patient space defined by the patient. The position of the tracked instrument can be displayed on a display device relative to an image of the patient.

[0007] The positioning of the patient can be determined using a tracking system. Typically, the patient is registered to an image via tracking an instrument relative to the patient to generate a transformation mapping between the subject or object space (e.g., patient space) and the image space. This typically requires time during the surgical procedure for a user, such as a surgeon, to identify one or more points in the subject space and correlate the typically same points in the image space.

[0008] After registration, the positioning of the instrument can be appropriately displayed on the display device while the instrument is tracked. The positioning of the instrument relative to the subject can be displayed as a graphical representation, sometimes referred to as an icon on the display device. SUMMARY

[0009] This section provides a general summary of the disclosure and is not a comprehensive disclosure of the full scope or all the features of the present disclosure.

[0010] According to various embodiments, an imaging system can be used to acquire image data of a subject. The imaging system can include an ultrasound imaging system that includes an ultrasound (US) probe that typically includes an ultrasound transducer to emit and receive ultrasound frequencies. However, it should be appreciated that the imaging system can include separate components to emit and receive ultrasound frequencies.

[0011] According to various embodiments, the US probe can be moved relative to the subject, such as by a user and / or with a robotic system. The robotic system can include a suitable robotic system, such as the Mazor X® sold by Mazor Robotics Ltd. having a place of business in Israel and / or Medtronic, Inc. having a place of business in Minnesota, USA and / or as disclosed in U.S. Patent No. 11,135,025, which is incorporated herein by reference. The US probe can be moved to achieve acquisition of selected image data. ™ a robotic guidance system, which is incorporated herein by reference. The US probe can be moved to achieve acquisition of selected image data.

[0012] According to various embodiments, the imaging system can be incorporated into various components. For example, the imaging system can be incorporated into a surgical drape. Also, the imaging system can include a US probe that can have a single housing for the transducer and / or can include separate components.

[0013] The imaging system, which can include a US probe, can also be used in various techniques to image or follow selected portions in surgery. For example, the US probe can be associated with a robotic system. The robotic system can move the US probe in a selected manner during surgery. During surgery, the US probe can be moved based on predetermined characteristics to follow or maintain a selected view of a selected portion, such as a portion of the subject or a portion of an instrument, during surgery.

[0014] Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0015] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure.

[0016] Figure 1is a schematic diagram illustrating an overview of a robotic system and navigation system in accordance with various embodiments;

[0017] Figure 2 is a detailed environmental view of a surgical drape with an ultrasound assembly and an optional robotic system with an instrument in accordance with various embodiments;

[0018] Figure 3A is a detailed cross-sectional view of a surgical drape with an ultrasound assembly in accordance with various embodiments;

[0019] Figure 3B is a detailed cross-sectional view of a surgical drape with an ultrasound assembly in accordance with various embodiments;

[0020] Figure 3C is a detailed cross-sectional view of a surgical drape with an ultrasound assembly in accordance with various embodiments;

[0021] Figure 4 is a detailed environmental view of a surgical drape with an ultrasound assembly in accordance with various embodiments;

[0022] Figure 5 is a detailed environmental view of a surgical drape with an ultrasound assembly in accordance with various embodiments;

[0023] Figure 6 is a detailed view of a subject support including an ultrasound transducer in accordance with various embodiments;

[0024] Figure 7 is an environmental view of an operating room with a robotic system holding an ultrasound probe in accordance with various embodiments;

[0025] Figure 8 is a flowchart of a process for moving and acquiring image data with a robotic system in accordance with various embodiments;

[0026] Figure 9 is a detailed view of a robotic system holding an ultrasound probe in a first position in accordance with various embodiments;

[0027] Figure 10 is a detailed environmental view of a robotic system holding an ultrasound probe in a second position in accordance with various embodiments;

[0028] Figure 11A and Figure 11B is a flowchart of a process for acquiring image data of a selected portion of a subject in accordance with various embodiments;

[0029] Figure 12 is a detailed view of an operating room with a robotic system holding an ultrasound probe and a robotic system holding an instrument in a first position in accordance with various embodiments;

[0030] Figure 13 is a detailed view of a surgical theater with a robotic system holding an ultrasound probe and a robotic system holding an instrument in a second position according to various embodiments;

[0031] Figure 14 is a detailed view of a surgical theater with a robotic system holding an ultrasound probe and a robotic system holding an instrument in a third position according to various embodiments;

[0032] Figure 15A and Figure 15B is a flowchart of a process of imaging selected portions of an instrument according to various embodiments;

[0033] Figure 16 is a representative slice of a magnetic resonance (MR) image according to various embodiments;

[0034] Figure 17 is an ultrasound image (echogram) according to various embodiments;

[0035] Figure 18A is a representative slice of a simulated computed tomography (CT) image according to various embodiments;

[0036] Figure 18B is Figure 18A a detailed view of a representative slice of a simulated computed tomography (CT) image of

[0037] Figure 19 is a representative slice of a segmented simulated computed tomography (CT) image according to various embodiments;

[0038] Figure 20 is a segmented model based on selected image data according to various embodiments; and

[0039] Figure 21 is a flowchart of a process of acquiring image data and registering image data according to various embodiments.

[0040] In several of the views of the drawings, corresponding reference characters indicate corresponding parts throughout the several views. DETAILED DESCRIPTION

[0041] Exemplary embodiments will now be described more fully with reference to the accompanying drawings.

[0042] The present subject disclosure relates to example embodiments of surgical procedures on a subject, such as a human patient. However, it should be understood that the systems and methods described herein are merely exemplary and are not intended to limit the scope of the claims included herein. In various embodiments, it should be understood that the systems and methods can incorporate and / or be used for non-living objects. For example, these systems can be used to register coordinate systems between two systems for use on manufacturing systems and maintenance systems, etc. For example, automobile assembly can use one or more robotic systems that include separate coordinate systems that can be registered together for coordinated or joint action. Thus, the example illustration of surgical procedures herein is not intended to limit the scope of the appended claims.

[0043] According to various embodiments, discussed herein are processes and systems for allowing registration between various coordinate systems. In various embodiments, a first coordinate system can be registered to a second coordinate system, such as a robotic coordinate system registered to an image coordinate system or space. The navigation space or coordinate system can then be registered to the robotic system or first coordinate system, and thus to the image coordinate system, without being registered to the image space separately or independently. Similarly, the navigation space or coordinate system can be registered directly or independently to the image coordinate system or space. The robotic system or first coordinate system can then be registered to the navigation space, and thus to the image coordinate system or space, without being registered to the image space separately or independently.

[0044] In various embodiments, different systems used relative to a subject can include different coordinate systems (e.g., positioning systems). For example, a robotic system can move relative to a subject that includes a robotic coordinate system. The robot can be fixed, including removably fixed at a location relative to the object. Thus, due to various features of the robot, a portion of the robot can be known relative to movement of a base of the robot (i.e., a fixed portion of the robot). For example, encoders (e.g., optical encoders, potentiometer encoders, etc.) can be used to determine movement or amounts of movement of various joints (e.g., pivots) of the robot. A position of an end effector (e.g., end) of the robot relative to the base of the robot is known. Given a known positioning of the subject relative to the base and a known positioning of the base relative to the subject, a positioning of the end effector relative to the subject is known during movement of the robot and / or during a stationary period of the end effector. Thus, the robot can define a coordinate system relative to the subject.

[0045] Various other portions can also be tracked relative to the subject. For example, the tracking system can be incorporated into a navigation system that includes one or more instruments that can be tracked relative to the subject. The navigation system can include one or more tracking systems that track various portions associated with the instruments, e.g., tracking devices. The tracking system can include a localizer configured to determine a position of the tracking device in a navigation system coordinate frame. Determination of the navigation system coordinate frame can include those described in various references, including U.S. Patent No. 8,737,708, U.S. Patent No. 9,737,235, U.S. Patent No. 8,503,745, and U.S. Patent No. 8,175,681, all of which are incorporated herein by reference in their entireties. In particular, the localizer can be capable of tracking objects within a volume relative to the subject. The navigation volume in which the tracking device can be tracked can include or be referred to as a navigation coordinate frame or navigation space. Determination or correlation between two coordinate frames can allow or also be referred to as registration between two coordinate frames.

[0046] In various embodiments, a first coordinate frame, which can be a robotic coordinate frame, can be registered to a second coordinate frame, which can be a navigation coordinate frame. As a result of the registration, coordinates in one coordinate frame can then be transformed to a different coordinate frame or the second coordinate frame. The registration can allow for use of and / or switching between the two coordinate frames. For example, during a procedure, the first coordinate frame can be used for a first portion of the procedure or a selected portion of the procedure, while the second coordinate frame can be used during a second portion of the procedure. Further, both coordinate frames can be used to implement or track a single portion of the procedure, e.g., for verification and / or to gather additional information.

[0047] Further, an image of a selected portion of the subject can be acquired. The image can be displayed for viewing by a user, such as a surgeon. The image can have superimposed on a portion of the image a graphical representation of a tracked portion or member, e.g., an instrument. According to various embodiments, the graphical representation can be superimposed on the image at the appropriate location due to registration of the image space, also referred to as the image coordinate frame, to the subject space. Methods of registering the subject space, defined by the subject, to the image space can include those disclosed in U.S. Patent No. 8,737,708, U.S. Patent No. 9,737,235, U.S. Patent No. 8,503,745, and U.S. Patent No. 8,175,681, all of which are incorporated herein by reference in their entireties.

[0048] During a selected procedure, a first coordinate system can be registered to the subject space or subject coordinate system due to the selected procedure, such as imaging of the subject. In various embodiments, the first coordinate system can be registered to the subject by imaging the subject with a fiducial portion fixed relative to the first member or system, such as a robotic system. Due to the image of the subject including the fiducial portion, the subject space can be registered relative to the robotic system using the known position of the fiducial relative to the robotic system. Thus, the position of the robotic system or a portion thereof, such as an end effector, can be known or determined relative to the subject. Due to the registration of the second coordinate system to the robotic coordinate system, additional elements can be allowed to be tracked that are not fixed to the robot relative to the position determined or tracked by the robot.

[0049] Tracking of instruments during a procedure, such as a surgical procedure or surgical treatment, allows for navigation of the procedure. When image data is used to define an image space, the image space can be correlated or registered with a physical space defined by a subject, such as a patient. Thus, according to various embodiments, a patient defines a patient space in which instruments can be tracked and navigated. The image space defined by the image data can be registered to the patient space defined by the patient. Registration can be performed by using fiducials that can be identified in the image data and the patient space.

[0050] Figure 1 is a schematic overview illustrating an operating room or surgical environment. In various embodiments, the operating room can include a surgical field in which a robotic system 20 and a navigation system 26 that can be used for various procedures can be placed. The robotic system 20 can include the Mazor X® sold by Mazor Robotics Ltd. of Caesarea, Israel. The navigation system 26 can include the StealthStation® sold by Medtronic, Inc. of Minneapolis, MN. The robotic system 20 can be used to assist in guiding a selected instrument, such as a drill, a screw, etc., relative to a subject 30. Additionally or alternatively, the robotic system 20 can hold and / or move an imaging system, such as an ultrasound (US) probe 33. The robotic system 20 can include a mount 34 that fixes a portion, such as a robotic base 38, relative to the subject 30. The robotic system 20 can include one or more arms 40, such as including an end effector 44, that can be moved or pivoted relative to the subject 30. The end effector can be any appropriate portion, such as a tube, guide, or passage member. The imaging system, which can be the US probe 33, can be attached to the end effector and / or can replace the end effector. The positioning of the end effector 44 can be known or determined relative to the base 38 using one or more encoders located at one or more joints of the robotic system 20, such as a wrist joint 48 and / or an elbow joint 52. ™ Robotic guidance system. The robotic system 20 can be used to assist in guiding a selected instrument, such as a drill, a screw, etc., relative to a subject 30. Additionally or alternatively, the robotic system 20 can hold and / or move an imaging system, such as an ultrasound (US) probe 33. The robotic system 20 can include a mount 34 that fixes a portion, such as a robotic base 38, relative to the subject 30. The robotic system 20 can include one or more arms 40, such as including an end effector 44, that can be moved or pivoted relative to the subject 30. The end effector can be any appropriate portion, such as a tube, guide, or passage member. The imaging system, which can be the US probe 33, can be attached to the end effector and / or can replace the end effector. The positioning of the end effector 44 can be known or determined relative to the base 38 using one or more encoders located at one or more joints of the robotic system 20, such as a wrist joint 48 and / or an elbow joint 52.

[0051] Navigation system 26 can be used to track the position of one or more tracking devices, including robotic tracking device 54, subject tracking device 58, imaging system tracking device 62, and / or tool tracking device 66. The tool or movable component 68 can be any suitable tool, such as a drill, clamp, or other tool operated by the user 72. Tool 68 may also include implants, such as spinal implants or orthopedic implants. It should be further noted that navigation system 26 can be used to navigate any type of instrument, implant, or delivery system, including: guidewires, arthroscopic systems, orthopedic implants, spinal implants, deep brain stimulation (DBS) probes, etc. Furthermore, these instruments can be used to navigate or map any area of ​​the body. Navigation system 26 and the various instruments can be used in any suitable surgical procedure, such as typically minimally invasive or open surgery.

[0052] Additional or alternative imaging systems 80 may be used to acquire preoperative, intraoperative, or postoperative or real-time image data of a subject (such as subject 30). However, it should be understood that imaging can be performed on any suitable subject, and any suitable surgery can be performed relative to the subject. In the example shown, imaging system 80 includes an O-arm sold by Medtronic Navigation, Inc., which has a business location in Louisville, Colorado, USA. ® Imaging apparatus. Imaging apparatus 80 may have a generally annular rack housing 82 in which an image capture portion is movably disposed. Imaging apparatus 80 may include those disclosed in: U.S. Patent Nos. 7,188,998; 7,108,421; 7,106,825; 7,001,045; and 6,940,941; all of which are incorporated herein by reference in any appropriate portion thereof. It should also be understood that imaging apparatus 80 may additionally or alternatively include a fluorescein C-arm. Other exemplary imaging apparatus may include fluorescein instruments, such as dual-plane fluorescein systems, ceiling-mounted fluorescein systems, catheterization lab fluorescein systems, fixed C-arm fluorescein systems, isocentric C-arm fluorescein systems, 3D fluorescein systems, etc. Other suitable imaging apparatus may also include MRI, CT, ultrasound, etc.

[0053] The positioning of the imaging system 33, 80 and / or portions therein, such as the image capture portion, can be known precisely relative to any other portion of the imaging device 33, 80. According to various embodiments, the imaging device 33, 80 can know and / or call upon precise coordinates relative to a fixed or selected coordinate system. For example, the robotic system 20 can know or determine its positioning and position the SU probe 33 at a selected pose. Similarly, the imaging system 80 can also position the imaging portion at a selected pose. This can allow the imaging system 80 to know its positioning relative to the patient 30 or other reference. Additionally, as discussed herein, the precise knowledge of the positioning of the image capture portion can be used in conjunction with a tracking system to determine the positioning of the image capture portion and image data relative to a tracked subject, such as the patient 30.

[0054] In this document, reference to the imaging system 33 can refer to any appropriate imaging system, unless otherwise indicated. Thus, the US probe 33 as an imaging system is merely exemplary with respect to the subject disclosure. As will be appreciated by those skilled in the art, the US probe 33 can generally emit US waves in a plane and receive echoes relative to any portion engaged by the waves. The received echoes at the US probe 33 or other appropriate received echoes can be used to generate image data and can be used to generate US images also referred to as sonograms.

[0055] The imaging device 80 can be tracked with the tracking device 62. Additionally, the tracking device 81 can be associated directly with the US probe 33. Thus, the US probe 33 can be tracked directly with a navigation system as discussed herein. Additionally or alternatively, the US probe 33 can be positioned and tracked with the robotic system 20. Regardless, according to various embodiments, image data defining an acquired image space of the patient 30 can be registered relative to a subject space (e.g., manually, inherently, or automatically). The target space can be a space defined by the patient 30 in the navigation system 26.

[0056] The patient 30 can also be tracked with a patient tracking device, DRF, or tracker 58 as the patient moves. Alternatively or additionally thereto, the patient 30 can be fixed within a navigation space defined by the navigation system 26 to allow for registration. As discussed further herein, registration of the image space with the patient space or subject space allows for navigation of an instrument 68 using the image data. As the instrument 68 is navigated, the positioning of the instrument 68 can be illustrated on a display device 84 relative to the acquired image data of the patient 30. There can also be additional and / or alternative display devices 84' to display the images. Various tracking systems, such as tracking systems including an optical positioner 88 or an electromagnetic (EM) positioner 92, can be used to track the instrument 68.

[0057] More than one tracking system can be used to track instruments 68 in navigation system 26. These can include an electromagnetic tracking (EM) system with EM localizer 94 and / or an optical tracking system with optical localizer 88, according to various embodiments. As discussed herein, either or both of the tracking systems can be used to track selected tracking devices. It will be appreciated that a tracking device can be the portion that is capable of being tracked with the selected tracking system, unless otherwise discussed. The tracking device does not necessarily refer to the entire member or structure to which the tracking device is attached or associated.

[0058] The positioning of patient 30 relative to imaging device 33 can be determined by navigation system 26. As discussed herein, the positioning of imaging system 33 can be determined. As further discussed herein, patient 30 can be tracked with dynamic reference frame 58. Accordingly, the positioning of patient 30 relative to imaging device 33 can be determined.

[0059] Image data acquired from imaging system 33 or any appropriate imaging system can be acquired at an image device controller 96, which can include a processor module, and / or forwarded from the image device controller to a navigation computer and / or processor system 102, which can be part of a controller or workstation 98 having a display 84 and user interface 106. It will also be appreciated that the image data need not necessarily be first saved in controller 96, but can be transferred directly to workstation 98. Workstation 98 can provide facilities for displaying image data as images 108 on display 84 and for saving, digitally processing, or printing hard copy images of received image data. User interface 106, which can be a keyboard, mouse, touch pen, touch screen, or other suitable input device, allows user 72 to provide input to control imaging device 80 via image device controller 96 or to adjust display settings of display 84. Workstation 98 can also instruct image device controller 96 to adjust the image capture portion of imaging device 80 to obtain various two-dimensional images along different planes to generate representative two-dimensional image data and three-dimensional image data.

[0060] With continued reference to Figure 1 Navigation system 26 can also include a tracking system that includes one or both of an electromagnetic (EM) localizer 94 and / or an optical localizer 88. The tracking system can include a controller and interface portion 110. Controller 110 can be connected to processor portion 102, which can include a processor contained within a computer. The EM tracking system can include a STEALTHSTATION® AXIEM® sold by Medtronic Navigation, Inc., having a place of business in Louisville, Colorado. The optical tracking system can include a POLARIS® optical localizer sold by Northern Digital, Inc., having a place of business in Waterloo, Ontario, Canada. ® AXIEM ™The navigation system; or can be an EM tracking system as described in U.S. Patent Application Serial No. 10 / 941,782, filed September 15, 2004 and entitled "METHOD AND APPARATUS FOR SURGICAL NAVIGATION"; U.S. Patent No. 5,913,820, entitled "Position Location System", issued June 22, 1999; and U.S. Patent No. 5,592,939, entitled "Method and System for Navigating a Catheter Probe", issued January 14, 1997; all of which are incorporated herein by reference. It will be appreciated that the navigation system 26 can also be or include any suitable tracking system, including a STEALTHSTATION ® TREON ® or S7 ™ A tracking system, which can be used as the optical localizer 88, and is sold by Medtronic Navigation, Inc. of Louisville, Colorado. Other tracking systems include acoustic systems, radiological systems, radar systems, etc. The tracking system can be used in accordance with generally known or described techniques in the references incorporated above. Unless selected operations of the subject disclosure are elucidated, details will not be included herein.

[0061] Wired or physical connections can interconnect the tracking system, the imaging device 80, etc. Alternatively, rather than being directly coupled to the controller 110, various portions such as the instrument 68 can employ wireless communication channels such as disclosed in U.S. Patent No. 6,474,341, entitled "Surgical Communication Power System", issued November 5, 2002, which is incorporated herein by reference. In addition, the tracking devices 62, 66, 54 can generate fields and / or signals that are sensed by the localizer 88, 94.

[0062] Various portions of the navigation system 26, such as the instrument 68, and other portions that will be described in detail below, can be equipped with at least one and typically a plurality of tracking devices 66. The instrument can also include more than one type or modality of tracking device 66, such as EM tracking devices and / or optical tracking devices. The instrument 68 can include a graspable or steerable portion at a proximal end, and the tracking devices can be affixed near the steerable portion of the instrument 68.

[0063] Another representative or alternative positioning and tracking system is set forth in U.S. Patent No. 5,983,126, entitled "Catheter Location System and Method," issued November 9, 1999, which is hereby incorporated by reference. The navigation system 26 can be a hybrid system that includes components from various tracking systems.

[0064] According to various embodiments, the navigation system 26 can be used to track the instrument 68 relative to the patient 30. As discussed above, the instrument 68 can be tracked with a tracking system. Image data of the patient 30 or appropriate subject can be used to assist the user 72 in guiding the instrument 68. However, the image data is registered to the patient 30. The image data defines an image space that is registered to a patient space defined by the patient 30. The registration can be performed automatically, manually, or a combination thereof as discussed herein.

[0065] In general, the registration allows for generation of a transformation map of the physical location of the instrument 68 relative to the image space of the image data. The transformation map allows for the tracked position of the instrument 68 to be displayed on the display device 84 relative to the image data 108. A graphical representation 68i, also referred to as an icon, can be used to illustrate the position of the instrument 68 relative to the image data 108.

[0066] With continued reference to Figure 1 , the subject registration system or method can use a tracking device 58. The tracking device 58 can include a trackable portion or member 120, but can also operate as or be used as a fiducial assembly. The fiducial assembly 120 can include a fixture or other fixed portion 124 and an imageable fiducial body 120. However, it should be understood that the member 120 can be separate from the tracking device 58. The fixed portion 124 can be provided to fix any appropriate portion, such as a portion of an anatomical structure. As Figure 1 illustrated, the fiducial assembly 120 can be interconnected with a portion of a spinal column 126, such as a spinous process 130. The fixed portion 124 can be interconnected with the spinous process 130 in any appropriate manner. For example, a pin or screw can be driven into the spinous process 130.

[0067] As Figure 1 illustrated, the imaging device 33 can include a US probe 33 that can be positioned relative to the subject 30, such as by the robotic system 20. Thus, the robotic system 20 can move the US probe 33 to a selected position relative to the subject 30, as discussed herein. According to various embodiments, the imaging system can be positioned relative to the subject in any appropriate manner. For example, reference is made to Figure 2A surgical drape 200 can be positioned on the subject 30. The surgical drape 200 can be a sterile sheet similar to known surgical drapes. The surgical drape 200 can include an opening or surgical portal 204. The surgical portal 204 can allow the user 72 to move the instrument 68 through the surgical drape 200 relative to a portion of the subject 30, such as a spine including one or more vertebrae. The surgical portal 204 can also be referred to as a port or an access, and the drape 200 can be a sheet of material configured to be positioned on and / or over the subject 30. The surgical portal 204 can define a boundary 208 through which the instrument 68 can pass.

[0068] At a selected position relative to the portal 204 can be one or more ultrasound arrays 222 that can operate as the imaging system 33. The arrays 222 can be positioned or placed adjacent or otherwise proximate to the portal 204. As illustrated, the arrays 222 can be placed at a selected distance from the boundary 208, such as one millimeter (mm), 5 mm, 10 mm, 50 mm, or more than 50 mm. Generally, the arrays 222 are positioned to image a selected portion of the subject without impeding access to the subject 30 through the portal 204. Figure 2

[0069] The ultrasound arrays 222 can include any appropriate number of arrays, such as a first array 222a, a second array 222b, and a third array 222c on a first side of the opening 204. Additionally or alternatively, ultrasound arrays 222d, 222e, and 222f can be positioned relative to a second portion of the opening 204. It should be understood that more or less than six ultrasound arrays 222 can be included relative to the subject 30, and the six ultrasound arrays are shown merely as an example.

[0070] The ultrasound arrays can be arranged in groups, such as the three arrays 222a-222c positioned on a first acoustic window portion 226, and a second group of arrays 222 including arrays 222d-222f can be positioned on a second acoustic window 230. The acoustic windows 226, 230 can be formed into the surgical drape 200.

[0071] ​The acoustic windows 226, 230 can be made of or formed from a substantially ultrasonically-coupling material portion that can be formed on or in an ultrasonic opening (also referred to as a coupling port or inlet) that is adjacent to the surgical opening 204. The sheet of the drape 200 can have channels or openings (e.g., cavities) formed therein for the windows 226, 230 that can be at least partially filled with a gel-like material, as discussed herein. Thus, the acoustic windows 226, 230 can be in substantial contact with the subject 30. The windows 226, 230 can have proper acoustic coupling with the subject 30. For example, the acoustic windows 226, 230 can be formed from a gel material that has proper ultrasonic transmission properties to achieve proper imaging or wave coupling within the subject 30. Thus, the arrays 222 positioned on the acoustic windows 226, 230 have proper acoustic coupling with the subject 30.

[0072] Turning to FIGS. 1-3, Figure 3A , Figure 3B and Figure 3C , the ultrasonic arrays 222 can be associated with respective acoustic windows, such as the acoustic windows 226, 226’ and 230, in an appropriate manner. The acoustic windows can be associated with the subject 30 and / or the drape or other covering 200 to help provide acoustic coupling of the arrays 222 with the subject 30. Thus, the acoustic windows can help ensure proper or active image acquisition clarity of the arrays 222 relative to the subject 30.

[0073] The acoustic windows, such as the acoustic windows 226, 230, 226’, can be formed from a selected material. For example, an adhesive can be provided with an acrylic material member, such as an acrylic member having an adhesive to contact the subject 30, the drape 200, and / or the arrays 222. Similarly or alternatively, a silicone material and / or adhesive can also be used as the acoustic window. However, those skilled in the art will appreciate that an appropriate low acoustic impedance material can be provided as the acoustic window. The low impedance acoustic impedance material can be provided as a band (e.g., a thin, flexible member) that includes an adhesive on at least one surface or portion to adhere the band or material to the subject 30 to ensure that a selected pose of the acoustic window relative to the subject 30 is maintained. The low acoustic impedance material can also be provided with adhesive on at least two surfaces to allow for adhesion to the subject 30 and to one or more of the arrays 222.

[0074] For example, referring to FIGS. 1-3, Figure 3AThe acoustic window 226 can include a first surface 226a and a second surface 226b. The first surface 226a can contact or adhere to the array 222. In various embodiments, an adhesive can be applied between the array 222 and the first surface 226a. The adhesive can be pre-applied or applied during surgery. Similarly or alternatively, an adhesive can be applied between the surface 226b and the subject 30. Again, the adhesive can be pre-applied or applied during a selected surgery.

[0075] In various embodiments, the acoustic window 226 can be included within the drape 200. For example, the acoustic window can be formed with and / or incorporated into the drape 200. The drape 200 can have a channel or opening that can be filled or connected with the acoustic window 226 in any suitable manner, such as with an adhesive, splicing, mechanical connection, etc.

[0076] In various embodiments, as Figure 3B illustrated, the acoustic window 226 can have two surfaces 226a and 226b to allow contact with the subject 30. Again, the acoustic window 226 can be incorporated into the drape 200 as selected. However, the array 222 can be incorporated into the acoustic window 226. According to various embodiments, for example, the array 222 can be molded into or formed with the acoustic window 226. As discussed above, various acrylic or silicone materials or compositions including the same can be used to mold or form the acoustic window 226 around the array 222 and / or incorporate into the array. Thus, the array 222 can be incorporated into the window 226.

[0077] The acoustic window can also be provided as a tape or adhesive, such as Figure 3CThe acoustic window 226' is illustrated. The acoustic window 226' can include two surfaces 226a and 226b. As discussed above, both surfaces and / or at least one of the surfaces can be included as an adhesive. Thus, the acoustic window 226' can operate as or function as a band or adhesive member to hold the drape 200 relative to the subject 30. For example, during a procedure, the drape 200 can be positioned relative to the subject and an opening or edge can be covered with the acoustic window 226' to connect or hold the drape 200 relative to the subject 30. The array 222 can also be positioned relative to the acoustic window 226', such as by contacting the first surface 226a. Again, both surfaces 226a and 226b can include a pre-applied adhesive, an adhesive applied during a procedure, or other selected connection mechanism. According to any suitable embodiment, the acoustic window 226' can be provided as a band with a pre-applied adhesive to allow it to adhere or connect with a selected portion, such as the drape 200, the subject 30, the array 222, or any suitable portion. According to various embodiments, whether or not the acoustic window can have a very small acoustic impedance to allow for proper ultrasound emissions through the acoustic window to the subject 30, and collection of ultrasound sonographic information due to a selected or suitable acoustic coupling between the array 222 and the subject 30. According to various embodiments, the array 222 can be provided as a transducer array. Thus, each of the arrays can emit an ultrasound frequency that propagates into the subject 30. As is generally understood by those skilled in the art, when an ultrasound wave contacts a solid material, such as bone, the ultrasound wave can reflect from the material. The reflected wave can then be received at a receiver portion of the transducer 222. Thus, the ultrasound array 222 can operate as both an emitter and a receiver (also referred to as a transducer). However, it will be appreciated by those skilled in the art that portions of the array 222 can operate individually and / or that various of the ultrasound arrays 222 operate as emitters and other of the arrays 222 operate as receivers.

[0078] As Figure 2As illustratively shown, ultrasound waves can be emitted by one or more of the ultrasound arrays 222 as emitted ultrasound waves 240. The emitted waves 240 can engage a portion of the subject 30, such as a vertebrate 244. Upon encountering some object, the emitted waves 240 can then reflect off the vertebra 244 as reflected waves 248. The reflected waves 248 can be received at one or more of the ultrasound arrays 222. For example, the emitted waves 240 can be emitted or launched from the portion 222f. The reflected waves 248 can be received at the ultrasound array 222f. Alternatively or additionally, the reflected waves 248 can be received at another of the arrays 222. As understood by those skilled in the art, each of the arrays 222 generally acquires image data in a plane. The plane can have a field of view relative to the portion of the arrays 222. Via the window, the field of view can include a portion of the subject.

[0079] However, as Figure 2 As shown, each of the arrays can be positioned relative to one another. Each of the arrays 222 can have a known positioning relative to one another, such as provided as a unit with the ultrasound windows 226, 230. The navigation system 26 and / or the imaging system 33 can know the positioning of the ultrasound arrays 222 relative to one another. Further, the imaging system 33 can also have a known positioning of the ultrasound arrays 222 relative to another ultrasound array. Thus, the emission and reception of ultrasound waves can be known based on the known positioning of the ultrasound arrays relative to one another.

[0080] According to various embodiments, the positioning and / or relative positioning of the arrays 222 can be established (e.g., predetermined and saved for recall) based on the type of procedure and / or the anatomical view needed to perform the selected procedure. For example, in a spinal procedure, the arrays can be placed such that detailed images of the pedicles or nerve roots can be visualized. In another example, the arrays can be positioned in a cardiac application to optimize imaging of one or more chambers of the heart.

[0081] The US transducer array 222 can also be associated with one or more tracking portions, such as a first tracking device 252 associated with the first ultrasound window assembly 226 and a second tracking device 256 associated with the second ultrasound window assembly 230. The tracking devices 250, 256 can be and / or include fiducial portions or members that are imaged with an imaging system, such as the imaging system 80 discussed above. The tracking devices 250, 256 can be tracking devices that can be tracked with the navigation system 26. The tracking devices 250, 256 can allow the respective portions, such as the ultrasound window assemblies 226, 230, to be tracked to allow the positioning of the ultrasound array 222 to be determined with the navigation system, such as relative to the subject 30 (e.g., in the subject or in the navigation space). Thus, image data acquired with the respective ultrasound array 222 can be registered to the subject 30 within the positioning information based on the navigation system 26 tracking the respective ultrasound array 222. The tracking devices 252, 256 can be tracked with the navigation system 26 to determine the pose of the ultrasound array 222 at an appropriate time, such as during a selected procedure.

[0082] One or more drape fiducials or tracking devices similar to the devices 252, 256 can be incorporated in a drape 200 that is separate from the window. The drape fiducials or tracking devices can allow the drape 200 and / or portions of the drape 200 to be tracked. The pose of the drape 200, including portions thereof such as the window, can allow the movable imaging system 33 to be moved relative thereto, such as automatically. As discussed herein, the US probe 400 can be moved with the robotic system 20. If the drape 200 or selected portions thereof have a known pose, including the opening 204, the US probe 400 can be automatically moved thereover for imaging.

[0083] Further, as understood by those skilled in the art, image data acquired with one or more of the ultrasound arrays 222 can be registered in the navigation system, such as disclosed in U.S. Patent No. 7,085,400 and U.S. Patent No. 9,138,204, both of which are incorporated by reference herein. The image data acquired within the respective ultrasound array 222 can be image data of the subject 30. When the ultrasound array 222 is registered to the subject 330 using the navigation system 26, the image data required of the subject 30, such as the vertebrae 244, can also have their pose determined in the navigation space within the navigation system 26.

[0084] The ultrasound arrays 222 can include one or more ultrasound arrays positioned relative to one another, such as on an ultrasound window assembly (e.g., ultrasound window 226). The ultrasound arrays 222a, 222b, and 222c can be in a set position relative to one another, such as in a linear position, and the selected distance can be a known distance 260 that is separated substantially along or on the axis 264. The assembly can be connected to the imaging system 33, such as with a connection including a wire connection 268. It should be appreciated that a wireless connection can also or instead be provided. Thus, the position of each of the arrays 222 can be known relative to one another, and image data generated based on signals received by any of the arrays can be known relative to the other arrays. Moreover, if a transmitted wave 240 is sent by one of the arrays, such as array 222a, and received by another of the arrays, such as array 222b, the reflected wave 248 can still be known relative to the first ultrasound array, and an appropriate image can be generated.

[0085] As discussed above, an ultrasound window, such as ultrasound window 226, can be incorporated into the drape 200. Thus, the ultrasound window 226 can be in contact with the subject 30, as discussed above. The ultrasound window 226 can be incorporated into the drape 200 in any appropriate manner, such as welding, adhesive, splicing, etc. Moreover, the connection, such as connection 268, can be a permanent or removable connection. Thus, the ultrasound arrays 222 can be used to acquire image data of selected portions of the subject 30 at appropriate times. In various embodiments, for example, the ultrasound arrays 222 can collect substantially real-time image data during a procedure. The ultrasound arrays 222 can be positioned relative to the surgical access opening 204 such that the user 72 can be able to view the subject 30 substantially in real-time. Moreover, various instruments, such as instrument tool 68, can be echogenic. The instrument 68 can also be visible in images generated using ultrasound image data, such as image 108. The navigation system 26 can allow the instrument 68 to be registered relative to the image 108, and an icon or graphical representation 68i can be displayed relative to the image 108. Additionally or alternatively, the image 108 can include image data of the instrument 68 such that the image 108 includes substantially real-time image data of the instrument 68 to allow the instrument image 68a to be displayed with the image 108. That is, as the instrument is imaged with the subject, an image of the instrument can be included in the image 108. The image 108 can include portions of the subject, such as the vertebra 244, as a vertebra image 244a and an instrument image 68a.

[0086] Turning with reference to Figure 4An example surgical drape 270 is illustrated. The surgical drape 270 can be similar to the surgical drape 200 discussed above. The surgical drape 270 can include an access aperture or surgical opening 274 defined by one or more walls or perforations 278 through the surgical drape 270.

[0087] An ultrasound window portion 282 can be positioned proximate the surgical access 274. The ultrasound window portion 282 can surround or partially surround the access 274. The access 274 can be similar to the access 204 discussed above. The ultrasound window portion can be substantially annular or super-annular, including an outer circumference 286 and an inner circumference 288. The ultrasound window 282 can be formed substantially similar to the ultrasound window 226 as discussed above, except for the shape of the ultrasound window 282. Thus, the ultrasound window 282 can be formed about the surgical access 274 to allow a selected view of the subject 30 when acquiring image data with one or more ultrasound arrays 292. A selected number of ultrasound arrays 292 can be provided, such as a first ultrasound array 292a, a second ultrasound array 292b, a third ultrasound array 292c, and a fourth ultrasound array 292d. Each of the ultrasound arrays 292 can be positioned relative to the surgical access aperture 274 in a selected manner, such as substantially 90 degrees from each other about the ultrasound window 282.

[0088] Likewise, the ultrasound arrays 292 can have known positions relative to each other, such as having a known diametric distance 296 from each other and / or they have a selected or known angular position 298 relative to each other. Thus, each of the ultrasound arrays 292 can have a known position relative to each other. The ultrasound arrays 292 can be cooperatively operated to determine the position of the image data relative to the subject 30 in a similar manner as discussed above.

[0089] Further, the imaging system 33 can include one or more fiducial points or tracking portions 304. The tracking portions 304 can be positioned relative to the ultrasound arrays 292 for tracking with the tracking system 26. Thus, the tracking system 26 can track the position of the ultrasound arrays relative to the subject 30. This allows the image data to be registered to the subject 30. Likewise, the position of the individual ultrasound arrays 292a-d can be known relative to the one or more tracking devices 304. Further, multiple tracking devices can be positioned on the ultrasound assembly 33, such as associating one tracking device with each of the ultrasound array portions 292a-d.

[0090] Turning reference Figure 5 An example surgical drape 320 is illustrated. The surgical drape 320 can be similar to the surgical drape 270 discussed above. It can include the surgical access opening 274 relative to which the ultrasound window 282 is positioned. The ultrasound window 282 can be substantially similar to the ultrasound window 226 discussed above with respect toFigure 4 The ultrasound window 282 is discussed. However, the ultrasound array 324 can include substantially similar geometry as the ultrasound window 282 together with the ultrasound window 282. One or more of the tracking devices 304 can also be provided with the ultrasound window 282.

[0091] The ultrasound array 324 can also be substantially annular or super-annular and include an outer or outer circumference 328 and an inner circumference 332. Thus, the ultrasound array 324 can completely surround the surgical portal 274 and can be substantially coextensive with the ultrasound window 282 in an annular fashion. Thus, the ultrasound array 324 can acquire image data at any suitable positioning about the surgical portal 274. As understood by those skilled in the art, the array 324 can include a plurality of transducers or a plurality of transmitters and a plurality of receivers of a selected size. Thus, at 324, the ultrasound array can include a pre-known or known geometry of the ultrasound portion of the ultrasound array 324 to determine image data relative to the portal 274. Likewise, the tracking portion or member 304 can also be used to track the ultrasound array 324 relative to the subject 30.

[0092] As discussed above, each of the ultrasound assemblies, including those relative to the windows 226-230 and the super-annular window 282, can be formed with a respective drape to include suitable coupling with the subject 30. Thus, the respective ultrasound arrays can generate image data to allow generation or reconstruction of images of the subject 30 in a selected manner. The ultrasound arrays formed with or in the drapes can provide substantially intimate contact and precise positioning of the respective ultrasound arrays relative to the subject 30. Further, as the surgical drapes are substantially maintained during a procedure, such as on the subject 30, the ultrasound arrays can be maintained in the selected positioning throughout the procedure. However, due to the respective tracking portions, such as the tracking portions 252, 256, 304, even the navigation system 26 can be utilized to determine movement of the ultrasound arrays.

[0093] Reference Figure 6, the subject 30 can be positioned on a support, such as the surgical table 104 as discussed above. The surgical table 104 can have various attachments attached thereto, such as a head rack or support, a hip holder or support 357, or other appropriate portions. Regardless, the table 104, hip holder 357, or other portion of the surgical support, such as the surgical table 104, can include or have incorporated therein an ultrasound array or imaging system 360. The ultrasound array 360 can be an imaging system 33, as discussed above. The ultrasound array 360 positioned in the hip holder or patient positioner 357 can allow the ultrasound array 360 to be positioned relative to the subject 30 for the selected portion of the surgery. This can help to maintain the ultrasound array 360 relative to the subject 30 without the need for additional support, such as additional users or technicians.

[0094] The ultrasound array 360 can include similar portions as those discussed above. For example, the ultrasound array 360 can include a window or patient contact portion 362. The window 362 can also be a hole or channel formed in the support 357. Further, the support 357 is merely exemplary and can be any appropriate support for the subject 30. Thus, ultrasound waves from the ultrasound array 360 can be transmitted to the subject 30 to allow for imaging or collection of image data of the subject 30. The ultrasound array 360 can include various array portions similar to the array portions 222 discussed above. The array portions can include a first array portion 364a, a second array portion 364b, and a third array portion 364c. As discussed above, the various ultrasound array portions 364a, 364b, and 364c can also be transducer systems and / or be individual transmitter or receiver portions to allow the array 360 to acquire image data in any appropriate or selected manner. However, the array portions 364 can allow for acquisition of image data of the subject 30.

[0095] The ultrasound array 360 can be provided with the window portion 362 such that the array portions 364 are in a known position relative to one another, such as a known or fixed distance 368 along a selected axis. Thus, as discussed above, the imaging system portion 360 can be capable of acquiring image data of the subject 30 in a selected and known manner. Further, a tracking portion 370 can be associated with the tracking array 360, such as being fixed to the tracking array 360 and / or being fixed to a portion relative to the tracking array. For example, the tracking portion 370 can be fixed to the hip support 357. Similarly, the imaging system 360 can be fixed to the hip support 357. Thus, the tracking member 370 can be in a fixed position relative to the imaging array 360.

[0096] Data can be transmitted from the imaging array 360 over an appropriate connection, such as connection 372. The connection 372 can be a wired connection, a wireless connection, or a combination thereof. The connection 372 can transmit various information, such as image data, tracking information from the tracking device 370, or other appropriate information. Thus, the connection 372 can be connected to any appropriate portion, such as the processor system assembly 102, the communication portion 110, or any other appropriate portion. Thus, a patient support, such as the table 104, can include the ultrasound array 360 to be positioned relative to the subject 30.

[0097] As discussed above, various configurations of ultrasound assemblies including one or more arrays, such as the US arrays 222, 292, or 364, can all include various one or more tracking devices, such as the tracking devices 252, 256, 304, 370, etc., relative to one or more of their transducers. As each of the arrays can be included or positioned relative to one or more of the tracking devices, the arrays can be used or operated in a coordinated manner. For example, a set of image data is provided or an image of a larger area than can be imaged by any particular one of the arrays is allowed to be generated. If the tracking devices are positioned relative to the arrays in a selected or known manner, tracking of the tracking devices relative to one of the arrays can also allow for tracking of all of the arrays.

[0098] For example, returning to Figure 1 , the multiple arrays 222 can work together in a coordinated manner. For example, the arrays 222a, 222b, and 222c associated with the first acoustic window 226 can operate as a unit. For example, each of the arrays 222a-c can be positioned relative to the subject 30 in a substantially fixed manner. Thus, the configuration or pose of each of the arrays relative to one another can be known.

[0099] Each of the arrays can generate or acquire image data within an area or plane, such as the first plane 222c' and the second plane 222b'. Data in the plane 222c' can be occluded or have a shadow created by the vertebra 244. The plane 222b' can not be occluded by the vertebra 244 relative to a selected volume of interest or region of interest within the subject 30. Thus, given the known configuration of the array 222b relative to the array 222c and the respective imaging planes 222b' and 222c', an image can be generated that reduces or eliminates any shadow that can occur if using only the array 222c' to generate image data of the subject 30. Various techniques can be used to reduce or eliminate the shadow, such as stitching or padding based on the known positioning of the arrays and their respective imaging planes relative to the subject 30.

[0100] Accordingly, images can be generated that include reduced shadowing or no shadowing based on multiple arrays and respective known positioning relative to one another. The known positioning can be based on the configuration of the multiple arrays relative to one another, such as with acoustic window 226. However, as discussed herein, two array assemblies (including those associated with acoustic window 226 and those associated with acoustic window 230) can also be used in combination to generate images with reduced shadowing for other features.

[0101] In a similar manner, various arrays, such as arrays 222a, 222c, and / or those associated with first acoustic window 226 and second acoustic window 230, can be operated to generate images or generate images in or in a virtual plane. The virtual plane or virtual image can be based on reconstructing images or generating images from multiple planes. For example, plane 222b' and plane 222c' can each be respective planes generated by respective individual ultrasound arrays 222b and 222c. The virtual plane can be generated based on image data acquired with both planes 222b' and 222c'.

[0102] For example, plane 222x' can be a virtual plane generated between two planes 222b' and 222c'. Virtual plane 222x' can be generated as if an ultrasound array were positioned between two ultrasound arrays 222b and 222c. The virtual plane can be generated for various purposes, such as to generate an image centered on or intersecting a selected anatomical structure or portion of subject 30 that is not directly in the center or plane of either of the ultrasound arrays.

[0103] Accordingly, ultrasound arrays can be positioned on subject 30 and virtual planes or virtual images can be generated using generation techniques that are not based on or are based only on a single one of the ultrasound arrays. Likewise, arrays associated with first acoustic window 226 can work in conjunction with arrays in second acoustic window 230 to generate virtual planes.

[0104] In addition to generating virtual planes or virtual images, a 3D volume can also be reconstructed based on image data acquired from multiple ultrasound arrays. As discussed above, virtual planes, such as virtual plane 222x', can be generated based on multiple imaging planes 222b' and 222c' that are in known poses relative to one another. Additionally, given known poses relative to one another, image data acquired with multiple ultrasound arrays allows for a three-dimensional volume to be reconstructed based on the acquired image data. As discussed above, the ultrasound arrays associated with first acoustic window 226 can all be in known poses relative to one another, as can all of the ultrasound arrays associated with second acoustic window 230. Further, tracking devices 252, 256 can be operated to determine the poses of acoustic windows 226, 230 relative to one another and to the respective ultrasound arrays associated therewith. Thus, image data acquired with each of the ultrasound arrays, which can be scalar ultrasound arrays, can be combined to generate a three-dimensional image based on image data acquired with all of the ultrasound arrays. Given tracking and / or known poses relative to one another, multiple scalar arrays can be generated based on the known poses of each of the ultrasound arrays.

[0105] As indicated above, image data of various portions of subject 30 can be generated that can be external to any single one of the ultrasound arrays associated with subject 30. Multiple arrays can be operated as a unit to generate image data of subject 30 for various purposes. As another example, given a predetermined or known configuration of ultrasound arrays, such as each of ultrasound arrays 222a, 222b, and 222c associated with single acoustic window 226, can be operated as a single multi-scalar array. Those skilled in the art will appreciate that more than one of the US arrays can be operated in a coordinated or selected manner to generate image data of a subject. Additionally, each of the ultrasound arrays can be understood as a scalar array, and multiple arrays or a plurality of arrays can be multi-scalar arrays.

[0106] Further, the ability to track each of the components, such as with tracking device 252 associated with first acoustic window 226 and second tracking device 256 associated with second acoustic window 230, can allow the two components to be operated together. The two components can include more than one array (i.e., more than one scalar array). Thus, two or more components can also be operated in coordination.

[0107] As discussed above, acoustic windows 226, 230 can be tracked with respective tracking devices 252, 256. This can allow two tracked imaging components, including the ultrasound arrays associated with respective acoustic windows 226, 230, to have determined poses relative to one another. This can also be referred to together as registration.

[0108] The respective tracking devices 252, 256 can be tracked relative to each other and the subject 30. Thus, based on tracking with the tracking devices 252, 256, which are in known poses relative to the ultrasound arrays 222 associated with the respective acoustic windows 226, 230, the pose of each of the ultrasound arrays 220 relative to each other can be known. This can allow the two acoustic windows 226, 230 and respective ultrasound arrays 222 to operate as a unit given the known configuration of all of the ultrasound arrays 222. Thus, the two acoustic windows 226, 230 can be associated with the subject 30 and in an appropriate manner, and the tracking devices 252, 256 can be tracked relative to the subject 30, such as relative to the subject DRF 58.

[0109] Tracking of the acoustic windows 226, 230 and / or the DRF 58 allows for triangulation of the pose of each of the ultrasound arrays 222 relative to each other. For example, the ultrasound array 222e can be in a known pose relative to the tracking device 256. Similarly, the ultrasound array 222b can be in a known pose relative to the tracking device 252. The tracking device 252 and the tracking device 256 can be tracked relative to each other, and each tracking device can be tracked relative to the subject DRF 58. Thus, due to the tracking of the various tracking devices 252, 256, 58, the pose of the two ultrasound arrays 222b and 222e can have a known or determined pose relative to each other and the subject 30. Thus, generation of image data with any of the ultrasound arrays can be known or determined relative to each other and / or any other tracking device tracked by the navigation system 26. This allows many ultrasound arrays to operate as a unit given their known or determinable pose relative to each other.

[0110] In addition to the example embodiments illustrated as Figure 2 discussed above, tracking of the ultrasound arrays with the navigation system and / or known poses relative to each other can be determined in any of the multiple ultrasound arrays discussed above, including those illustrated as Figure 2 and Figure 4 to Figure 6 Thus, the various embodiments can operate individually and / or together in a similar manner as discussed above. Thus, for example, an ultrasound array including multiple ultrasound arrays can be included in any of the appropriate embodiments to generate image data according to any of the events discussed above.

[0111] As discussed above, the robotic system 20 can include portions of the instrument 68' and / or can include a robotic arm 68 as Figure 2An additional robotic system 20' is illustrated as an optional system. According to various embodiments, an ultrasound array 222 can be positioned relative to the subject 30. As discussed above, the ultrasound array 222 can be used to acquire image data of the subject to allow for generation of images. The user 72 can manually move or manipulate the instrument 68, as Figure 1 Alternatively or additionally, the robotic assembly 20' can have an end effector 44' to engage the instrument 68'. Thus, the robotic system 20' can be used to move the instrument 68' relative to the subject 30. The imaging system including the ultrasound array 222 can be substantially fixed relative to the subject 30, while the robotic system 20' moves the instrument 68' relative to the subject 30. Thus, the robotic system 20 can move the ultrasound probe 33 and / or alternatively move the instrument 68'. Additionally or alternatively, an additional robotic assembly 20' can be provided to move the instrument 68'. Thus, the instrument 68' can be moved relative to the subject 30 with the robotic assembly 20' while the ultrasound array 222 is positioned relative to the subject, such as with the acoustic windows 226, 230. As discussed above, the robotic system 20 can move a portion of the robotic system 20 relative to the subject 30, such as at least the end effector 44. The end effector 44 can hold the imaging system 33 or be the imaging system, as Figure 1 illustrated above. Additionally referring to Figure 7 , the imaging system 33 can be an ultrasound probe 400. The ultrasound probe 400 can include a patient contact or engagement surface 404 that can be moved along a selected portion of the subject 30, such as an external soft tissue surface (i.e., skin) thereof. The engagement surface 404 can include geometry and / or features to help move and / or protect the subject 30 from pressure (point pressure above a selected threshold). For example, a member with a smooth surface can be formed with the US probe 400 to engage the subject 30, but allow the US probe 400 to couple with the subject 30 to acquire selected image data.

[0112] The ultrasound probe 400 can also include other portions, such as portions connected to the end effector 44 and / or connected to the robotic system 20. The ultrasound probe 400 can also include a housing or other portion to which the ultrasound tracking device 81 can be connected. Thus, the ultrasound tracking device 81 can be tracked with a selected tracking system, such as a portion of the navigation system 26 discussed above. Further, as understood by those skilled in the art, the positioning of the end effector 44 can be determined based on movement of the robotic system 20.

[0113] Generally, the robotic system 20 can be moved or operated by a user 72 and / or by executing selected instructions, such as with a processor component including the processor component 102 and / or the robotic processor component 410. The robotic component 20 can also be provided with a memory portion from which instructions can be called and / or which allows for communication with other systems. As discussed above, the ultrasound probe 400 can be used to collect image data of the subject 30, such as with ultrasound waves. Thus, the ultrasound probe 400 can be moved relative to the subject 32 to acquire image data of the subject 30. The robotic component 20 can move the ultrasound probe 400 based on input instructions from the user 72, substantially automatically based on a predetermined imaging path, and / or based on a determination of a portion of the subject to be imaged, and / or combinations thereof. In short, likewise, the ultrasound probe 400 can include a transducer, a transmitter, and / or a receiver.

[0114] For example, the ultrasound probe 400 can be positioned with the robotic system 20 relative to a selected portion of the subject 30. The user 72 can select to have the ultrasound probe 400 moved over a selected portion of the subject 30. The selected portion of the subject can be identified by the user 72, such as using input including the display 84' and / or other appropriate input. For example, a representation of the subject 30 can be displayed, such as based on general dimensions of the subject 30, and the user can identify an area over which the ultrasound probe 400 should be moved. The robotic system 20 can include sensors such that only a maximum force is applied to the subject 30 at the contact surface 404 that contacts the surface of the subject 30. Thus, the robotic arm 20 can move the ultrasound probe 400 relative to the subject 30 in a selected manner.

[0115] In various embodiments, the robotic arm 20 can move the ultrasound probe relative to a surgical device or implant within the subject. For example, the surgical device can be tracked and its pose can be set as a destination to be imaged, and the robotic arm 20 can move according to its movement parameters to reach the destination. Additionally, the robotic arm can move relative to a particular anatomical structure within the subject in a similar manner. Additionally, the destination can be manually input and the robotic arm 20 can move.

[0116] As generally understood by those of skill in the art, the ultrasound probe 400 can acquire image data of the subject 30. The generated or determined images can then be displayed on the display device 84, such as the image 108. The image data or images generated with the ultrasound probe 400 can also be displayed and / or saved for further analysis in any appropriate manner. Thus, the ultrasound probe 400 can be moved with the robotic system 20 to acquire image data of the subject 30.

[0117] The positioning of the ultrasound probe 400, which can include and / or be connected to the end effector 44, can have a position that is known relative to the subject 30 based on the movement of the robotic system 20. As discussed above, the robotic system 20 can have a base 34 that is fixed relative to the subject 30. The base 34 can be fixed to the patient support 104 and / or positioned on a floor surface relative to the subject 30. However, the origin of the base 34 relative to the subject 30 can be known and / or determined. The origin or position of the base 34 relative to the subject can be determined in any appropriate manner, such as by tracking a portion of the base 34 relative to the subject 30, tracking the robotic system tracking device 54 relative to the subject 30, or other appropriate determination, including manual input. As another example, the robotic system 20 can include various encoders that can determine the amount of movement of the end effector 44 relative to the base 34 to allow for determination of the position of the ultrasound probe 400 relative to the subject 30. Thus, the position of the ultrasound probe 400, and the image data collected with it, can be known or determined relative to the subject 30. This allows the image data to be registered relative to the subject 30.

[0118] As discussed above, the ultrasound probe 400 can also be tracked with a navigation system due to the tracking device 81. The position of the ultrasound probe 400 can be known based on the position of the ultrasound probe 400 tracked with the tracking device 81. By knowing the position of the ultrasound probe 400 relative to the subject 30, image data of a selected portion of the subject 30 relative to previously acquired portions of image data can be determined and made by moving the ultrasound probe 400 with the robotic system 20. Further, the robotic system 20 can be able to position the ultrasound probe 400 accurately relative to a previous position of the ultrasound probe 400 and / or repeat a previous position of the ultrasound probe 400. Thus, the robotic system 400 can be able to place the ultrasound probe 400 accurately relative to the subject 30 to acquire image data relative to any selected previously acquired image data and / or acquire image data at the same position as the previously acquired image data.

[0119] As discussed above, an imaging system, such as the ultrasound probe 400, can be moved with the robotic system 20 relative to the subject 30. This movement can be based on a registration of the robotic coordinate system to the subject coordinate system. Additionally, a registration of previously acquired image data to the subject or navigation system space or coordinate system can allow or ensure that a selected region is imaged, such as a region of interest identified in previously acquired images. Further or alternatively, the robotic system 20 can include various sensors to measure and / or ensure that an appropriate amount of pressure is applied when using the ultrasound probe 400.

[0120] As understood by those skilled in the art, the ultrasound probe 400 includes a contact surface 404 that engages the subject 30. When in a selected position or optimal position relative to the subject 30, the ultrasound probe 400, including the selected transducer, can acquire image data of the subject 30, which can include selecting or optimizing the pressure applied to the subject 30 with the ultrasound probe 400.

[0121] In various embodiments, for example, the robotic system 20 can include one or more sensors, such as the sensor 414 and / or the sensor 416. It should be understood that more than one sensor is not required, and the two sensors are exemplified as examples for various positions of the sensors. In various embodiments, the sensors 414, 416 can be positioned at various joints of the robotic system 20. Discussion herein regarding one or more of the sensors 414, 416 can be understood as or refer to any appropriate sensor that can be positioned at any appropriate portion of the robotic system 20. The sensor 414 can include a pressure sensor that can be used to sense an amount of pressure applied to the subject 30 with the ultrasound probe 400. The pressure can be measured to determine whether an appropriate amount of pressure or a selected amount of pressure is being applied to the subject 30 with the ultrasound probe 400.

[0122] When the ultrasound probe 400 is applied or moved over a selected tissue, such as a hard tissue, a selected amount of pressure can be applied in an attempt to obtain or obtain optimal image data acquisition. Thus, the pressure sensor 414 can be used to measure the amount of pressure applied and / or tissue resistance. It can be determined whether additional pressure can be applied or should be applied to obtain optimal image data.

[0123] Further, as discussed above, due to the registration of the robotic coordinate system to the subject coordinate system and any previously required image data coordinate system, the determination of the ultrasound probe 400 being positioned over a selected cardiac tissue or portion (e.g., bone) relative to the subject can be used to assist and determine the amount of pressure to be applied. The pressure sensor 414 can then determine the amount of pressure applied and determine whether a change in the amount of pressure is appropriate, such as by comparing the measured pressure to a predetermined pressure. In various embodiments, for example, as opposed to a portion over a hard portion within the subject 30, a greater amount of pressure can be applied over a portion of the subject that does not include a hard portion, such as tissue on the abdomen. Various sensors can be used to measure the pressure applied to the subject 30 by the ultrasound probe 400 and can be compared to a predetermined pressure and / or determine whether additional or less pressure can be applied to achieve optimal image data acquisition.

[0124] Furthermore, the amount of pressure applied can be measured to determine whether tissue deformation is possible. If a large or selected amount of pressure is applied using the ultrasound probe 400, tissue or organ deformation may occur. Deformation can alter or change the image acquisition. For example, if a selected threshold pressure or pressure greater than the selected threshold pressure is applied, the positioning of an organ (such as a kidney or intestine) may differ from a previously acquired image with no pressure applied or very little pressure applied. Therefore, pressure sensor 414 can be used to determine whether the amount of pressure applied and its effect on the positioning of an internal object (such as an organ within subject 30) can be altered due to the pressure applied at the ultrasound probe 400. Pressure sensor 414 can measure pressure and allows determination of whether the pressure should be increased or decreased and / or whether the applied pressure may cause deformation or movement of an internal organ. If the measured pressure is greater than the selected threshold, it can be determined that the organ may have moved or tissue deformation may have occurred, and thus the image may be analyzed or understood to differ from previously acquired images. This can also allow for the acquisition of additional or alternative image data with different amounts of applied pressure, particularly when the ultrasound probe moves with the robotic system 20, to aid in image data acquisition using the ultrasound probe.

[0125] refer to Figure 8 The following illustrates process 430. Process 430 can be used to generate image data of subject 30 in a selected manner. For example, robotic system 20 can move ultrasound probe 400 relative to subject 30. While moving ultrasound probe 400 relative to subject 30, image data can be collected at various poses (e.g., viewing angles) relative to subject 30. Various viewing angles can be used with appropriate ultrasound probes to generate selected images of subject 30, such as three-dimensional or volumetric images.

[0126] The process 430 can begin in start block 434. The process 430 can then acquire image data in a first position in block 438. The image data can be acquired with respect to a subject in any appropriate position. The ultrasound probe 400 can be moved to the selected position by the robotic system 20, positioned manually by the user 72, or in another appropriate manner including combinations thereof. The image data of the subject 30 can be acquired in any appropriate manner. After the image data is acquired or at any appropriate time, a call or selection of an image to be generated can be made in block 442. The selection of the image to be generated can include a selection of a vertebra, a plurality of vertebrae, an organ or other appropriate portion of the subject to be imaged. For example, the ultrasound probe can be used to generate image data of soft tissue with respect to one or more vertebrae, image data or images of the vertebrae, or an organ of the subject such as the heart of the subject. Regardless of how the image data is after the selection of the portion to be imaged, an identification of a structure in the image data can occur in block 448. The identification of the structure can include segmenting the image data acquired in block 438. The segmentation of the image data can include finding edges or other appropriate portions in the image.

[0127] After the image is segmented, the structure can be identified. The structure in the image data can be identified as a portion or a part of the subject to be imaged. The identified structure can be used to identify where to move the ultrasound probe 402 to acquire image data to generate the image data to generate a selected image of the subject. For example, a heart wall or edge of the structure can be segmented and an identification of the structure can be made. The identification of the structure can be automatic, such as with the processor assembly 102, manually by the user 72, or in any appropriate manner. In various examples, a look up table of structures can be used to compare to the segmented portion. Additionally, various machine learning systems can be trained to identify features of the segmentation. Another example is a statistical atlas of anatomy that has been generated from a population of subjects, registered to the image view and used as a roadmap / guide to segment key features within the current image.

[0128] As indicated above, the robotic system 20 can include one or more sensors 414, 416, which can be pressure sensors. Thus, the process 430 can optionally include sensing or measuring pressure based on the pressure applied to the subject 30 with the US probe 400. The pressure sensed or measured in block 449 can then be used to determine an impact of the pressure in block 450. As discussed above, the impact of the pressure can be deformation of tissue, movement of organs, etc. In addition, the registration of the coordinate spaces of the robotic system 20 and the subject 30, as well as any previously acquired images, can be compared to the sensed pressure to determine a pose of the US probe 400, such as over or near a hard portion, such as a bone. For example, a high measured pressure can occur when the US probe 400 is pressed onto a portion near a bone. Thus, this determination can be to change the applied pressure, move the probe 400, analyze the acquired image data based on the possible deformation, etc.

[0129] Based on the acquisition of the image in block 438 and the identification of the structure in block 448 and based on the called out portions of the subject for which images are to be generated in block 442, the robotic system 20 can move the ultrasound probe 400 to a subsequent or second position to acquire subsequent or second image data at block 452. For example, in block 438, the ultrasound probe 400 can be moved to acquire an anterior to posterior (AP) image of the subject 30. The second position can include acquiring a lateral image through the heart of the subject 30. Thus, the robotic system 20 can move the ultrasound probe 400 to the appropriate position to acquire additional image data.

[0130] The movement of the US probe 400 with the robotic system can be substantially automatic. For example, the robotic system 20 can be tracked relative to the subject to allow the pose of the US probe 400 to be known based on tracking the subject 30 and the robotic system 20. The US probe 400 can be automatically moved to this pose. In addition, as discussed herein, portions to be imaged can be identified. These portions can be partially imaged in a first image. The system can determine additional portions for which additional image data is needed and can move the robotic system 20 accordingly to move the US probe 400 to acquire the additional image data. For example, if the heart is selected to be imaged, the first image can be segmented and a portion of the heart can be identified (e.g., the lower portion of the right atrium). The robotic system 20 can then move the US probe 400 in a manner based on known anatomical structures (e.g., an atlas of the heart) to acquire image data of additional portions of the heart (e.g., the upper portion of the right atrium).

[0131] After acquiring additional image data, a determination can be made in block 454 whether additional image data is needed. If additional image data is needed, the "yes" path 458 can be followed to block 452 to move the ultrasound probe 400 with the robotic system 20 and acquire additional image data. The process 430 can iterate to move the ultrasound probe 400 with the robotic system 20 and acquire additional image data to allow the selected images to be generated.

[0132] In addition to or in place of the robotic arm 20 physically moving the US probe 400 and / or the contact surface 404, various beam shaping or steering techniques can be used to move the area imaged with the US probe 400. It should be appreciated that beam shaping or steering can be used with any appropriate US system, including the one or more arrays 222 discussed above or in accordance with any appropriate embodiment.

[0133] In beamforming or beam steering, various ultrasound transducer arrays can be used to generate selected beam portions, such as beam shapes or beam parameters. Selected portions of the array can be energized individually or simultaneously to generate different beam configurations and / or angles relative to the fixed positioning of the ultrasound probe 400. Beam steering can allow the area being imaged or the area where image data is collected to change without physically moving the ultrasound probe 400. Thus, the image data can appear to be moving and / or being imaged at different positions relative to the ultrasound probe 400 without physically moving the ultrasound probe 400. Beam steering or beamforming can be performed substantially automatically by operating or controlling the ultrasound probe 400 and the transducers therein, such as with a selected processor system as discussed above. Beam steering or beamforming can be used to collect image data at selected positions relative to the subject without physically moving the ultrasound probe 400. Thus, image data can be acquired at different poses relative to the subject without physically moving the ultrasound probe 400.

[0134] The determination in block 454 can be made substantially automatically by the system. For example, the system can include a processor 410 or any appropriate processor. The determination can be made based on a look-up table regarding segmented portions of the current image, a machine learning system trained to determine portions selected for imaging, etc. In this way, the system can automatically determine whether sufficient image data has been collected.

[0135] Accordingly, if it is decided in block 454 that additional image data is not needed, then the "NO" path 462 can be followed. An image can then be generated in block 464. As understood by those skilled in the art, an image can be generated based on the image data acquired with the ultrasound probe. The generated image can be a three-dimensional image, such as a three-dimensional image of the heart of the subject 30. Additionally, other image data can be included, such as flow velocity or pressure within the subject. Accordingly, the image 108 that can be generated in block 464 can include any appropriate image data or selected data.

[0136] After the image is generated in block 464, the image can be output in block 466. Outputting the image in block 466 can include various selected or optional portions, such as saving the image in block 468 and / or displaying the generated image in block 470. As discussed above, the image 108 can be the generated image that is output in block 466.

[0137] The process can then end in block 474. Ending the process 430 in block 474 can end the collection of image data to generate a selected image of the subject 30. A procedure can occur with respect to the subject 30 based on the acquired image data and / or the generated image. Accordingly, additional image data can be acquired at a later time, and the process 430 can be repeated. The robotic system 20 can allow the ultrasound probe 400 to be repositioned at substantially the prior position of the ultrasound probe. Accordingly, the robotic system 20 can move the ultrasound probe 400 to the same position of the process 430 to acquire later or subsequent image data of the subject 30. The process 430 can also be utilized to acquire confirmation or subsequent image data, such as for confirming a planned procedure and / or determining additional procedure portions to be performed.

[0138] In addition to or instead of the above, the image data acquired in block 454 can be collected over time, and the image generated and / or output in blocks 464 and 466 can be four-dimensional (4D) image data. According to various embodiments, the robotic arm 20 can be moved to acquire 4D image data. The creation of 4D data can be used for cardiac use cases where the heart is moving. In such instances, the image data or image can be 3D, and it can change over time, thus being 4D data. The 4D image data can be displayed, such as a movie. In various embodiments, the 4D image data and / or image can be segmented. The segmented image can be registered to a preoperative multi-phase (4D) cardiac CT or other appropriate image. Accordingly, the generated image can be a 4D image, and it can be registered to other 4D images.

[0139] As such, the robotic control system can move the ultrasound transducer system, such as by executing instructions with a processor, to acquire image data to generate 4D image data and / or images of the subject. The acquired 4D image data and / or images can also be segmented and can be registered to the subject and / or pre-acquired images, such as to identify selected structures or to verify predetermined results, among others.

[0140] As discussed above, the imaging system 33 can include an ultrasound imaging probe that can include a housing that is moved relative to the subject 32 to acquire image data of the subject 30. The image data can be used to generate and display images, such as the images 108. However, the displayed images can be based on the positioning of the ultrasound probe relative to the subject 30. In particular, if the ultrasound probe is manipulated by a robotic system, such as the robotic system 20, a user can select to have a particular region or zone imaged substantially continuously. The system can determine movements of the ultrasound probe to position the probe relative to the subject to maintain the selected image for display to and by the user 72. That is, the ultrasound images can generally be collected to be displayed in substantially real-time. Further, the user 72 can make an identification to maintain a portion of an instrument, for example at the tracked instrument 68, within a portion of the image generated from the image data from the ultrasound probe. Thus, with reference to the flowcharts above and below, the system can be used to determine a portion of the subject that is imaged to maintain a selected image, determine a positioning of an instrument and ensure that a portion of the instrument is within the image, and to move the ultrasound probe substantially automatically to maintain the selected image or instrument within the image. Additionally or alternatively, once achieved, the imaging system 33 can be fixed or frozen at the selected pose, as discussed herein.

[0141] Referring first to Figure 9 and Figure 10 , the robotic system 20 can position the ultrasound probe 400 relative to the subject 30. The subject 30 can be positioned in any appropriate portion, such as on a table or support 104. The ultrasound probe 400 can be moved relative to the subject 30 in any appropriate manner, such as in a selected spatial position (including three degrees of freedom) and orientation (including a selected, such as three degrees of freedom). Thus, the ultrasound probe 400 can be moved in selected degrees of freedom, which can be limited by the robotic system 20, or include any appropriate degrees of freedom of motion, such as six degrees of freedom including three degrees of freedom of position and three degrees of freedom of orientation. The robotic system 20 can move the ultrasound probe relative to the subject to acquire various images, such as the first image 10a as illustrated in Figure 9 or the second image 10b as illustrated in Figure 10The second image 108b is illustrated. The images 108a, 108b can be displayed on a corresponding display, such as the display 84'. The images 108a, 108b can be displayed on the display 84' based on the positioning of the ultrasound probe 400 and / or the anatomy of the subject 30. However, it should be understood that the ultrasound probe 400 as the imaging system 33 can image any appropriate subject, and the human subject is merely illustrative.

[0142] However, the images 108a, 108b can be images of a heart of a subject that can be at a selected phase. For example, the image 108a can illustrate an anterior-posterior view of the heart of the subject 30 and / or a systolic phase of the heart. The image 108b can be a lateral or superior-inferior view of the heart and / or a display of the heart at a diastolic phase. However, it should be understood that any appropriate portion of the subject 30 can be displayed with the selected image. Further, the portion of the subject can have a phase that includes or changes the physical dimensions or geometry of the portion within the subject over time, such as rhythmically. Those skilled in the art understand that the heart phase changes the shape of the heart in a substantially rhythmic manner over time. Other portions of the anatomy of the subject 30 can also include similar changes.

[0143] With continued reference to Figure 9 and Figure 10 With further reference to FIG. 11, the process 500 is illustrated. The process 500 is a process for moving an imaging system including the ultrasound probe 400 relative to the subject 30 to generate image data of the subject 30. The process 500 also allows for ensuring that a selected portion of the subject is maintained in an image generated with the image data from the ultrasound probe 400. As discussed above and understood by those skilled in the art, the ultrasound probe 400 as the imaging system 33 can collect or acquire image data of the subject 30. The image data can be analyzed, evaluated, and processed to allow for generation of images, such as the images 108a, 108b. The process 500 can be incorporated into instructions executed by any appropriate processor, such as the processor 410. The instructions can be formulated into an appropriate algorithm.

[0144] The process 500 can begin in a start block 510. The start block 510 can be any appropriate beginning portion of the process 500, such as the user 72 initiating use of the ultrasound probe 400, which can allow for analysis and / or evaluation of image data. The image controller 96 can include a processor and can process image data from any appropriate imaging system, such as the imaging system 80 or an imaging system including the ultrasound probe 400. Thus, acquisition of image data at a first position can be made in a block 514.

[0145] Acquiring image data at the first position can include acquiring image data at the first position as Figure 9The illustrated positioning acquires image data. The ultrasound probe 400 can be positioned in a first position and orientation relative to the subject 30. For example, the subject 30 can extend along the long axis 516. The ultrasound probe 400 can be positioned by the robotic system 20 along the axis 520. Thus, the ultrasound probe can be in the first position at a first angle 524 relative to the long axis 514. Image data can be acquired with the ultrasound probe 400 at any appropriate rate, such as at a selected frame rate and / or over a selected time period.

[0146] At any appropriate time, such as after the image data is acquired in the first position in block 514, a call for a selected portion to be imaged or to be in the image can be made in block 528. The call for the selected portion of the image can be any appropriate call, such as to ensure that the selected portion, such as the right atrium, is in the image. The controller can access a memory and / or the user 72 can input the selected portion to be in the image. In block 532, the image data acquired in block 514 can be evaluated.

[0147] The evaluation of the image data in block 532 can be any appropriate processing and / or evaluation. For example, the image data can be segmented or analyzed for various features. Further, the image data can be normalized or otherwise processed for further processing in the process 500.

[0148] A determination is made in block 538 whether the selected portion is in the image. The determination in block 538 can be based on various appropriate processes. According to various embodiments, the user 72 can view the image and determine whether the selected portion is present in the image. The evaluation of the acquired image data in block 532 can also include generating and displaying an image based on the acquired image data. The determination of block 538 can also include executing a selected algorithm or instructions based on an algorithm, such as a comparison to a lookup table, a comparison to a predefined atlas, spline analysis, or edge and shape detection in the image data. The selected algorithm can identify various features and determine whether they match the selected portion called from block 528. Further, a machine learning system can be used to analyze the image data to determine whether the selected portion is present in the image. Regardless, a determination can be made as to whether the selected / recalled portion to be imaged is in the image data (block 538).

[0149] If the selected portion is not in the image data, then the "no" path 540 can be followed to acquire new image data in a subsequent position in block 542. The image data acquired at the subsequent position can be image data other than the image data acquired at the first position. Thus, the ultrasound probe 400 can be used to acquire additional data. For example, as Figure 10The ultrasound probe 400 can be positioned at a second axis 556 relative to the long axis 516 of the patient, as illustrated. Thus, the ultrasound probe axis 556 can be at a second angle 558 relative to the long axis 516 of the subject 30. The image data acquired at the subsequent position can be as Figure 10 The subsequent position, as illustrated.

[0150] After the subsequent image data is acquired at the subsequent position, the process can loop to the evaluate block 532. Thus, the process 500 can loop to ensure that the image data acquired includes the selected portion. The user can use the ultrasound probe 400 to acquire any appropriate image data at any appropriate position. The ultrasound probe 400 can be moved to a selected random position using the robotic system 20 and / or based on input from the user 72. For example, the robotic system 20 can move in a grid search pattern relative to the initial position. Further, as discussed above, the ultrasound probe 400 can be held by the robotic system 20 that is tracked using the navigation system, and the subject 30 can also be tracked using the patient tracker 58. Thus, the portion to be imaged can be determined relative to the patient tracker 58, and the probe 400 can be moved relative thereto to acquire image data, and / or a search pattern can be conducted relative to the initially determined position.

[0151] If it is determined that the selected portion is in the image data, the "yes" path 562 can be followed. In following the "yes" path 562, the pose of the imaging system can be stored in block 568. The pose of the imaging system can be performed in various ways. As discussed above, the controller 96 can include a processor and / or memory. The pose of the imaging system can be stored in the controller. The pose can be based on the navigation tracked pose of the imaging system, including the ultrasound probe 400. A tracking device 81 can be positioned on the probe 400 and tracked to determine the pose of the probe 400. Additionally or alternatively, the robotic system 20 can include an encoder to determine the pose of the ultrasound probe 400 and / or the end effector 44. The pose of the end effector 44 can be saved and selected portion and / or accessed using the robotic processor 410. Regardless, the pose of the imaging system can be stored in block 568.

[0152] In block 572, it can be determined whether a freeze command has been received. The freeze command can include a command to hold the ultrasound probe 400 at the pose stored in block 568. By freezing the ultrasound probe 400 at the stored pose, the acquired images of the subject 30 can be maintained relative to or based on the stored (e.g., last) pose of the ultrasound probe 400. For example, the ultrasound probe can be positioned at Figure 9 the position, as illustrated, and a freeze command can be issued to ensure that the ultrasound probe is held at that pose. The pose can be the pose that includes the selected portion, as discussed above.

[0153] If a freeze command has been received, then the "Yes" path 564 can be followed to determine if an unfreeze command has been received in block 578. If an unfreeze command has not been received, then the "No" path or loop 580 can be followed. The "No" path loop 580 ensures that the ultrasound probe 400 is held at the freeze command pose, which includes the stored pose from block 568. Thus, once frozen, the ultrasound probe 400 will hold its pose relative to the subject 30 or other appropriately selected portion until unfreezing. An unfreeze command can then follow the "Yes" path 584 to determine if a lock command has been received in block 586. Similarly, when a freeze command has not been received in block 572, the "No" path 588 can be followed to the lock command receipt determination. Thus, whether a lock command has been received in block 586 can be determined based on both paths as discussed above.

[0154] The process 500 can also determine if a lock command has been received in block 586. In brief, a lock command locks the ultrasound probe 400 image data acquisition pose to ensure that the selected view is maintained. For example, as illustrated, the ultrasound probe is positioned to generate image data to generate the image 108b. While the ultrasound probe 400 can be frozen to ensure that the ultrasound probe 400 does not move relative to another over time, the lock command maintains the selected view. As discussed above, in block 528, the selected portion to view or call in the image can be made. The lock command can ensure that the ultrasound probe 400 maintains that view, such as the view of the heart of the subject 30 in the image, regardless of movement of the heart or subject. Thus, the image 108b can be maintained regardless of movement of the subject 30, movement relative to the portion of the subject 30, or even movement of the ultrasound probe 400. The system can operate substantially automatically to maintain the selected view, such as the image 108b. Figure 10

[0155] ​Accordingly, if a lock command is received, then the "yes" path 590 can be followed. Then, in block 594, the "yes" path 590 can be followed to determine if one or more phases are selected. It should be appreciated that selection of one or more phases is optional and, thus, need not be selected in the process 500. However, if selected and determined in block 594, the "yes" path 598 can be followed to display only the selected phase in block 602. The selected phase can be a phase selected or invoked in block 594 and / or block 528. For example, the diastolic or systolic phase of the heart can be selected for display. The phase of the heart can change even though the imaging system is moved over time and / or held in a selected position to ensure the selected viewing image. While the image data is collected with the ultrasound probe 400 over time, only the selected phase image can be displayed, such as one of the phases or the selected phase of the heart. In block 602, the system can be operated or commanded to display only the selected phase and the process 500 can then continue to estimate movement of the portion in block 606.

[0156] However, if the selected phase is not selected in block 594, then the "no" path 610 can be followed so that the displayed image is only the last image or selected image in block 614. Thus, as discussed above, the ultrasound probe 400 can collect image data substantially in real-time and continuously. Accordingly, the image 108b can be the last image collected with the ultrasound probe 400. For example, the ultrasound probe 400 can collect image data at about 30 frames per second. Thus, the image 108b is the last image frame collected with the ultrasound probe 400. Again, after the image display of block 614, the process 500 can continue to estimate movement of the portion in block 606.

[0157] The process 500 can continue to estimate movement of the selected portion, such as based on subject motion, instrument motion relative to the portion, or other characteristics. For example, the image data can illustrate that the selected portion is being stressed via an instrument, such as for resection, implantation, etc. Possible or average motion can be used to estimate movement of the portion. Further, the system can invoke predetermined or known motion of a portion relative to a subject, such as due to beating of the heart, to estimate motion of the selected portion. Accordingly, in the process 500, the system can estimate movement of the portion in block 606.

[0158] After the estimated movement of the portion, an unlock command receipt determination can be made in block 610. If an unlock command is received, then the "yes" path can be followed in block 615. If the imaging system including the ultrasound probe 400 is commanded not to lock onto any particular image portion, then the selected positioning can be moved and / or maintained. Thus, if an unlock command is received, then the "yes" path 615 can be followed to end the process in block 620. The ending of the process 500 will be further discussed below.

[0159] If an unlock command is not received, then the "no" path 624 can be followed. The "no" path 624 can then move or arrive at a command to move the imaging device in block 630. The command to move the imaging device can include moving the imaging device in any appropriate manner and moving the imaging device (including the US probe 400) to maintain the selected image based on the estimated movement of the portion from block 606. Thus, based on the maintain command of the lock, a move command can be issued. The move command can include a movement of the robotic system 20 in a selected three-dimensional spatial position and / or three-dimensional orientation for movement of the imaging system 400. The move command can include a selected movement in any appropriate axis and / or orientation to position the imaging system including the ultrasound probe 400 relative to the subject 30 to maintain the imaging of the selected portion. After the move command is issued, the imaging system can be moved and the acquisition of image data can be made at the subsequent positioning in block 552. The movement in block 552 can include beam steering and / or shaping as discussed above. Thus, the process 500 can loop or iterate to maintain the acquisition of image data of the selected portion of the subject, such as a locked feature within the subject 30.

[0160] Referring back to decision block 586, the received lock command can follow the "no" path 640. The "no" path 640 can also arrive at the end block 620. Thus, if no lock command is received, the process can end in block 620, allowing the imaging system to acquire the selected image. In this way, the user 72 can manually move the ultrasound probe 400 and / or manually command the robotic system 20 to move the imaging system including the ultrasound probe 400. This allows the ultrasound probe 400 to be moved in any appropriate manner to achieve image data of the subject 30 as selected by the user 72.

[0161] In addition to imaging selected features of the subject 30, such as various portions of anatomy, including the heart, the ultrasound probe 400 can also be used to image instruments or portions that move relative to the subject 30. For example, as discussed above, the ultrasound probe 400 can be used to image a catheter 50 that is moved relative to the subject 30. In this way, the ultrasound probe 400 can be used to image the catheter 50 as it is moved relative to the subject 30. Thus, the ultrasound probe 400 can be used to image the catheter 50 as it is moved relative to the subject 30. Figure 12 、 Figure 13 and Figure 14The ultrasound probe 400 can be moved with the robotic system 20 relative to the subject 30, as illustrated. The ultrasound probe 400 can acquire image data of various portions of the subject, as discussed above. Based on the acquired image data, various images can be displayed on the display device 84' and / or any appropriate display device. The images 1108a, 1108b, and 1108c are illustrated in FIG. 11. As discussed above, the images 1108 can be similar to the images 108. However, for clarity with respect to the following discussion, the images 1108 are discussed herein. Figure 12 to Figure 14

[0162] The ultrasound probe 400 can be moved to acquire image data at various positions relative to the subject 30 to allow for generation or reconstruction of the images 1108. The instrument 68' can be moved relative to the subject 30. The instrument 68' can be substantially similar to the instrument 68 discussed above. According to various embodiments, the instrument 68' can be held and / or moved with the robotic system 20'. The robotic system 20' can be substantially similar to the robotic system 20. The system 20' can include an end effector 44' that engages the instrument 68'. Thus, the instrument 68' can be moved with the robotic system 20' to various positions relative to the subject 30. According to various embodiments, the position of the instrument 68' can be tracked. As discussed above, a navigation system can track a tracking device associated with the instrument 68'. In addition and / or alternatively, the robotic system 20' can have a position known relative to the robotic system 20 and / or the subject 30 to allow for tracking or knowing the position of the instrument 68' relative to the subject 30. Thus, according to various embodiments, including those otherwise disclosed and not merely illustrated in FIG. 11, the robotic system 20 can hold and / or position (e.g., move) the US probe 400 separate from the instrument 68' and / or the robotic system 20' can hold and position (e.g., move) the instrument. However, in various embodiments, the robotic system 20 that moves the ultrasound probe 400 can have a position unknown relative to the robotic system 20' that moves the instrument 68'. However, the robotic system 20 can move the ultrasound probe 400 to capture images of at least a portion of the instrument 68' and it can also move the ultrasound probe 400 to hold at least a portion of the instrument 68', including a selected portion, in the images 1108 generated with image data from the ultrasound probe 400. Figure 12 to Figure 14

[0163] In short, as Figure 12 illustrated, the ultrasound probe 400 can capture or acquire image data from a region 700 of the subject 30. The region 700 can not include a portion of the instrument 68', such as a distal tip 704 of the instrument 68'. Turning to the reference Figure 13 ​​, the ultrasound probe 400 and / or the instrument 68' can be moved to image a region 700 of the US probe 400 and / or to encompass at least the distal tip 704 of the instrument 68'. Thus, the images 1108a-c can include an icon or graphical representation of the instrument 68' as a graphical representation 68i'. As discussed further herein, according to various procedures or methods, the ultrasound probe 400 can be moved to a second or different position relative to the subject and / or the instrument 68' can be moved relative to the subject such that the image region 700 of the ultrasound probe 400 continues to ensure that at least the distal tip 704 of the instrument 68' is within the volume 700 being imaged and included in the image 1108c. Thus, the image 1108c can include the graphical representation 68i' of the instrument 68', including its distal tip 704.

[0164] Turning to FIG. 15, the system can perform a procedure 730 to track or illustrate a portion of an instrument, such as the instrument 68', in the image 1108. The procedure 730 can include similar portions as the procedure 500 discussed above. These portions will not be discussed in detail herein. However, the procedure 730 can be used to generate the image 1108 including at least a portion of the instrument 68', such as the distal tip 704. Thus, the image 1108 can be illustrated to help the user 72 understand the pose of the instrument 68' relative to the subject 30 and / or other portions within the image 1108. For example, the instrument 68' can be moved relative to the subject 30 to help perform a procedure on a selected portion of the patient, such as the liver, heart, etc. The instrument 68' can be moved by the robotic system 20' and / or the instrument 68 can be moved by the user 72, regardless, the imaging system including the ultrasound probe 400 can be moved to maintain the selected portion of the instrument 68, 68' within the image 1108. The procedure 730 can be incorporated into instructions executed by any appropriate processor, such as the processor 410. The instructions can be formulated into an appropriate algorithm.

[0165] Initially, the procedure 730 can begin in a start block 734. The procedure 730 can then acquire image data in a first position in a block 738. As discussed above, the first image data can be any appropriate image data and acquired in any position that can be understood or labeled as a first position. However, as discussed further herein, the imaging system 400 can be moved.

[0166] At any appropriate time, such as before or after acquiring the first image data, an instruction can be received in a block 742 whether or not to image the instrument. Similar to determining or receiving an input of two image selected portions of the subject's anatomy or any appropriate portion of the subject, an instruction to image the selected portion or to ensure a portion of the instrument is in the image can be received in the block 742.

[0167] If the instruction has been received, then the "yes" path 746 can be followed. The "yes" path can then follow the decision block in block 752 of whether the instrument is tracked. As discussed above, various systems can be utilized to track the instrument 68, 68'. For example, a navigation system can be utilized to determine the pose of the instrument 68'. Additionally, as indicated above, the robotic system 20' can move the instrument 68'. The robotic system 20' can be tracked with a navigation system and / or understand its pose based on various systems of the robotic system, such as in its encoders. Thus, according to various embodiments, the instrument can be tracked. If the instrument is tracked, then the "yes" path 754 can be followed.

[0168] If the instrument is tracked, then a command to move the imaging system to the tracked pose of the instrument can be sent in block 758. The command to move the imaging system can be a command to move together with the imaging system to a selected position and / or orientation in three-dimensional space relative to the subject 30. For example, the instrument 68, 68' can be tracked relative to the subject 30. However, it should be understood that the coordinate system of the robotic system 20 holding the ultrasound probe 400 can be registered or coordinated with any other appropriate coordinate space to allow for commands to move the robotic system 20 to move the ultrasound probe 400 to any appropriate pose.

[0169] The command can be to move the robotic system 20 to move the ultrasound probe 400 to any appropriate pose. At the appropriate pose, based on the command to move the imaging system including the ultrasound probe 400, the ultrasound probe can acquire subsequent image data in block 762. If the instruction is received as discussed above, then additional image data can be selected to include or attempt to include the selected portion of the instrument. Regardless, the acquisition of subsequent image data will be image data collected after the initial or previous image data as understood by those skilled in the art.

[0170] The process 730 can include various other paths, including a path in the event that the instrument is not imaged, then the "no" path 766 can be followed. The "no" path 766 can also be followed in block 762 to acquire additional or subsequent image data. Thus, if no instruction to image the instrument is received, then the imaging system including the ultrasound probe 400 can be utilized to acquire any appropriate subsequent image data, such as manually selected by the user 72, automatically selected by the system, or to image a selected portion of the anatomy or subject 30 as discussed above.

[0171] Further, if the decision block 752 identifies or includes that the instrument is not tracked, then the "NO" path 770 can be followed. If the instrument is not tracked, then a send and / or receive command can be issued in block 774 to move the imaging system to image the instrument. The movement in block 774 can include beam steering and / or shaping as discussed above. If the user inputs a direction or command, then optional receive input from the user can be followed in block 776. Receiving input from the user can include the user identifying a direction or amount of movement for the robotic system 20 to move the ultrasound probe 400. Alternatively, a command can be sent to move the robotic system 20 to image the subject with any appropriate or selected manner (e.g., a planned search or movement pattern). In this way, the ultrasound probe 400 can be moved based on instructions from a user, such as the user 72. Thus, it should be understood that the ultrasound probe 400 can be moved based on manual input, as understood by those skilled in the art.

[0172] Additionally or alternatively, the input or command can include an evaluation of the image data at the first location in block 780. The evaluation of the image data at the first location can include an evaluation of the type of instrument 68', the selected portion of anatomy, or an identification of the anatomy, etc. For example, if the image data at the first location is evaluated to include a heart as the portion being imaged, then the system can invoke or understand the beating or periodic motion of the heart and its possible movements to determine future locations of the instrument (e.g., such as the movement of the instrument with the heart motion). Further, the image data can be evaluated to determine possible movements of the instrument based on the type of instrument and / or a predetermined plan. For example, the system can segment the image data and identify that the instrument is an ablation probe. Thus, the system can invoke the moving portion to be ablated and understand or determine possible future movements of the instrument 60'. Regardless, the image data at the first location can be evaluated to select or provide a guess of future locations of the instrument.

[0173] Thus, based on the evaluation of the image data, a command to move the imaging system, such as the mobile robotic system 20, can be generated and / or sent. Based on the selected command to move the imaging system, the imaging system including the ultrasound probe 400 can be moved. After the command is sent and / or the imaging system 400 is moved, the imaging system can acquire additional or subsequent image data in block 762. This allows the imaging system to acquire further image data to attempt to image the instrument as indicated above. Thus, the imaging system including the ultrasound probe 400 can be moved based on various inputs and / or analysis to acquire subsequent image data.

[0174] After acquiring the subsequent image data in block 762, following any of the paths described above, a determination can be made in block 790 whether the instrument is in the subsequent image data. The determination of whether the instrument is in the image data can be made based on various appropriate processes. For example, the image 1108 can be displayed and the user 72 can identify or confirm that the instrument is in the image data. In addition, various image analysis can be performed. For example, segmentation of the image data can occur and a comparison of the image data to a lookup table can be performed to determine whether the instrument is in the image. In addition, according to various embodiments, a machine learning system can have been trained to identify the instrument 68, 68'. Thus, the process or system can execute the machine learning system to evaluate and / or confirm that the instrument is in the image.

[0175] Regardless, the determination block 790 can determine whether the instrument is in the image data. If the instrument is not in the image data, then the "no" path 794 can be followed. The "no" path can follow at least two processes, a first path 794 in which a command is sent to move the imaging system in block 784, as discussed above. An alternative path can include commanding the imaging system to move in block 758. Thus, the "no" path 794 can include a path based on prior instructions, such as whether the imaging system is to identify or move based on manual input and / or based on image analysis.

[0176] If the determination in block 794 is that the instrument is in the image data, then the "yes" path 800 can be followed to store the imaging system pose in block 804. The imaging system pose stored in block 804 can be similar to the imaging system pose described above in block 568. The pose of the imaging system, such as the ultrasound probe 400, can include the positioning of the imaging system relative to the subject 30. The pose of the imaging system can include the pose of the imaging system, such as the ultrasound probe 400, in any appropriate coordinate system. Regardless, the positioning of the imaging system can be stored in block 804.

[0177] A freeze command can then be determined in block 808. If a freeze command is received in block 808, then the "yes" path 810 can be followed. In block 812, the "yes" path 810 can be followed to determine whether an unfreeze command has been received. As discussed above, the freeze command can include freezing the imaging system, such as the ultrasound probe 400, at a selected position in space. Thus, the freeze command can include freezing the ultrasound imaging probe 400 or ensuring that the ultrasound imaging probe does not move. If an unfreeze command is not received, then the "no" loop path 814 can be followed. If a freeze command is first received in block 808, then the "no" loop command 814 can loop the determination block of whether an unfreeze command is received until an unfreeze command is received. If an unfreeze command is received, then the "yes" path 820 can be followed.

[0178] If no freeze command is received, the "NO" path 824 can be followed. Thus, in block 830, it can be determined whether a lock command is received after no freeze command or unfreeze command. If a lock command is received, the "YES" path 834 can be followed. The lock command 830 can be similar to the lock command discussed above in block 586. Thus, if the instrument is imaged or selected for imaging, the lock command can include determining or ensuring that the portion of the image includes the instrument 68, 68'.

[0179] Thus, if the received lock command follows the "YES" path 834, an estimate of the movement of the instrument can occur in block 836. The estimate of the movement of the instrument can include an estimate of the motion of the instrument 68, 68'. The estimate can include calling from a selected look-up table whether the instrument is to move a large amount or a small amount. Further, the estimate of the movement can include receiving information from the robotic system 20' regarding the movement of the robotic system 20' holding or moving the instrument 68'.

[0180] After the estimate of the movement of the instrument, it can be determined whether an unlock command is received in block 840. The unlock command can include unlocking the imaging system, such as the ultrasound probe 400, to prevent imaging of the instrument 68, 68'. If no unlock command is received, the "NO" path 866 can be followed to send a movement command to the imaging system in block 850. The movement command can be based on the selected or determined movement of the instrument or the estimated movement of the instrument. The movement command can be sent to the robotic system 20 holding or moving the ultrasound probe 400. The movement command can include an amount of movement or a final position and / or orientation of the ultrasound probe 400 to which the robotic system 20 can move the US probe 400. The amount of movement and / or the final position of the ultrasound probe can be sent as the movement command. The movement command is sent to move the ultrasound probe 400 or any appropriate imaging system, such as the imaging system 33.

[0181] Then, after the movement of the US probe 400, a subsequent image data can be acquired in block 854. Then, the process can end in block 856. Ending the process 730 in end block 856 can include any appropriate ending procedure. Ending the process can include stopping the automated movement of the ultrasound probe 400. Ending the procedure in block 856 can also include saving and / or displaying the selected image data, such as the image data acquired with the ultrasound probe 400. Regardless, the procedure can end in block 856, which can include ensuring that the image data of the selected portion is acquired, including the subsequent or later position of the instrument 68, 68' which can be selectively included.

[0182] If a lock command is not received, then the "NO" path 860 can be followed to acquire subsequent image data in block 854 and further end in block 856. Again, ending the procedure in block 856 can be similar to the procedures discussed above. Also, if an unlock command is received, then the "YES" path 866 can also be followed to acquire subsequent image data and perform block 854. Acquiring subsequent image data in block 854 can include moving the ultrasound probe 400 in any appropriate manner. For example, the ultrasound probe 400 can be moved in a selected random or search pattern to acquire image data selected according to a predetermined procedure. Further, the user 72 can enter instructions regarding moving the ultrasound probe 400 to acquire additional image data.

[0183] Regardless, the ultrasound probe 400 can be used to acquire image data of the subject 30. The ultrasound probe 400 can be moved based on input from a user and / or based on instructions according to various procedures, including the procedure 500 and / or the procedure 730 as discussed above. Thus, the ultrasound probe 400 can acquire image data of the subject, a particular portion of the subject, a portion of an instrument, or a combination thereof to be displayed for viewing by a user and / or for further analysis as discussed above.

[0184] As discussed above, according to various embodiments, the acquired data of the subject 30, such as image data, can be acquired with an ultrasound probe. For example, the ultrasound probe can be an imaging device 33 that is held and moved with the robotic system 20. In various embodiments, the ultrasound array can include the ultrasound array 222, which can include a scalar ultrasound array in various embodiments. Thus, one or more of the transducer assemblies (e.g., the ultrasound array 222) can be positioned at a selected position and / or an immovable position relative to the subject to acquire data of the subject. The acquired data of the subject can be referred to as data or ultrasound data that can be used to generate an image, such as an acoustic map of the subject 30. Also as discussed above, various ultrasound transducers or probes can be tracked with the navigation system 26 to allow for determining a pose of any data acquired in a navigation space (e.g., a subject space or relative to the subject 30) with the ultrasound probe. Further, previously acquired image data or other image data of the subject 30, such as MR image data, can also be acquired. The previously acquired image data can include any appropriate image data, such as the MR image 900, as Figure 16 illustrated. According to various embodiments, the MR image can be displayed on the display device 84. However, as discussed further herein, the MR image 900 need not be displayed, but can be registered to the subject 30 and / or other data, such as data acquired with the ultrasound probe.

[0185] In various embodiments, the MR image 900 can include data regarding the subject 30. For example, various structures such as the first structure 910 can be included in the MRI image 900. The structure 910 can include a portion of a vertebral body of the subject 30. However, it should be understood that the structure 910 can be any appropriate structure, such as any hard or rigid structure within the body of any subject, such as a non-human subject. Further, various portions of the structure can be identified, such as the edges or boundaries 914.

[0186] The MR image 900 can also include a non-rigid or soft region 920, which can include various structures or boundaries 924 relative thereto. The non-rigid structure can be soft tissue of the subject 30, such as the spinal cord, muscle tissue, vasculature, or other appropriate structure.

[0187] However, the MR image 900 can be acquired with the subject. The MR image 900 can include any appropriate type of image data, which can include three-dimensional data of the subject 30. Figure 16 The illustrated MR image 900 can be a two-dimensional rendering or slice of a three-dimensional image data acquisition, and thus is understood to not limit the type of acquired MR image data of the subject 30. The MR image 900 can include a selected extension, such as all or a majority of the vertebral bodies within the subject 30. Thus, each of the original structures 910 can relate to one or more of the vertebral bodies and related bone structures, such as the pelvis, ribs, etc. The soft tissue portion 920 relative to the hard tissue portion can also be included in the MR image, and provide an illustration or representation of an extension or portion of the subject 30. This allows the MR image 900 to include a region or volume that can provide context to any particular portion included in the MR image, such as one or more of the vertebral bodies relative to and including soft tissue, as well as other portions surrounding the vertebral bodies.

[0188] According to various embodiments, one or more ultrasound probes can generate ultrasound images 930, as Figure 17 illustrated. The ultrasound images 930 can be illustrated or displayed on the display device 84 for viewing by the user 72. However, as discussed further herein, the display of ultrasound images is not required for various processes, such as analysis of the subject 30, registration to other data such as the MR image 900, or other appropriate purposes. However, according to various embodiments, the ultrasound images 930 can include data that can be used for various purposes.

[0189] The ultrasound image 930 can be an ultrasound image of one or more vertebrae in the spine of the subject 30. According to various embodiments, the spine of the subject can be imaged with an ultrasound probe to acquire data thereof. For example, as exemplified by the image 930, a vertebra 934 can be included in the data 930. The data 930 can be analyzed, such as with a selected trained machine learning algorithm, manual identification, other automatic algorithm detection, etc., to identify or determine various features in the ultrasound image 930. As generally understood by those skilled in the art, the ultrasound image 930 can include portions that are echogenic under the selected settings of the ultrasound transducer. Thus, the image 930 can include portions of the vertebra 934 that can be identified, such as various features thereof. The features can include one or more boundaries of the vertebra, such as a first boundary 938 and a second boundary 940. The boundaries 938 and 940 can be used to identify the vertebra 934 and / or various portions of a particular vertebra. As discussed further herein, this identification can allow for registration to other image data. In addition, other portions of the anatomy can be identified, such as a joint region 944 that can be between two or more vertebrae in the ultrasound image 930.

[0190] The ultrasound image 930 of the subject 30 can be acquired, and various echogenic features of the subject 30 can be identified in the ultrasound image 930, such as various edges or contacts or junctions between bone structures, such as the vertebrae 934 in other tissue. These features can be compared to other image data or images to help register or correlate the ultrasound image 930 and the other images. As discussed above, the ultrasound probe can be tracked in a navigation system, and thus the pose of the identified portions in the ultrasound image can be known in the navigation space, and once the ultrasound image 930 is registered to other images, such as the MR image 900, the navigation space is allowed to be registered to the other images.

[0191] To help registration of the MR image 900 to the ultrasound image 930, initial processing of the MR image can be performed. The processing of the MR image or any first image data can include generating intermediate or correlatable image data or images. The intermediate image data can be any appropriate image data, and various embodiments are discussed herein. The intermediate image data can help registration to second image data, and various embodiments are discussed herein.

[0192] In various embodiments, the MR image processing can include generating an echogenic feature map or space of the MR image. In various embodiments, for example, identification of echogenic edges or surfaces in the MR image can be determined. For example, the edge 914 of the vertebra 910 can be identified or processed in the MR image 900. Similarly or alternatively, edges or surfaces of soft tissue regions, such as the surface 924, can be identified or processed in the MR image 900.

[0193] In various embodiments, the MR image 900 can also be used to generate or processed to generate simulated computed tomography (simulated CT) images or data spaces. The simulated CT can be an example data or feature space generated based on the MR image 900 to allow the MR image 900 to be registered with ultrasound images or data, such as the US image 930. As discussed further herein, the registration can allow for contextual understanding of various portions of the subject. The contextual image portions can allow for identification of features that can not be effectively identified in the ultrasound images. Further, due to or by tracking the ultrasound probe relative to the subject 30, the processed MR image, such as the simulated CT, can allow the MR image to be registered to the subject space. Thus, as Figure 17 illustrated, various features in the ultrasound can be identified, such as edges in the ultrasound sound wave image. It should be understood that an appropriate number, such as more than one, of image data projections can also be combined into a three-dimensional ultrasound sound wave image. Thus, for example, multiple arrays 222 associated with the subject 30 can be used to generate image data to allow for generation of a three-dimensional image based on the ultrasound image data. Alternatively or additionally, moving the ultrasound probes, such as the ultrasound probes 400, relative to each other into a number of positions can allow for generation of a three-dimensional image based on the acquired image data. Thus, as Figure 17 illustrated, the two-dimensional images are merely examples, as Figure 16 illustrated, the MR image 900, which can be a slice of a three-dimensional MR image. However, processes can be applied to generate images or image data that can be used to register or correlate the US image 930 and the MR image 900.

[0194] Turning to reference Figure 18A , for example, a simulated CT image or image portion 970 can be generated. The simulated CT image 970 can include simulated or images portions having features similar to CT images. For example, segmentation and identification of hard tissue, such as a vertebral body 974, including its edges 978, can be generated, as Figure 18B illustrated. Additional features, such as a spinous process 982 and its associated boundaries or edges 986, can also be identified. Various portions, such as the vertebral body 974 and the spinous process 982, can be identified or generated as simulated CT image portions based on the MR image 900. In various embodiments, a trained algorithm, such as a machine learning or artificial intelligence algorithm, can be trained on a plurality of MR image data to generate simulated CT images or image data. In various embodiments, the simulated CT image data can or can not be displayed. In various embodiments, the simulated CT can be generated only for further analysis and comparison with ultrasound images, as discussed further herein.

[0195] In addition to or as an alternative to simulated CT, other image data or portions that substantially resemble CT images can be generated, such as including the identification of echo features in the MR image. For example, the identification of one or more echo boundaries (such as the boundary between soft and hard tissue) can be made in the MR image 900. Echo boundaries can be determined in a process similar to that of simulated CT generation, such as by an algorithm or a trained algorithm. Furthermore, echo boundaries can allow the generation of image data that does not require the entire generation of a simulated CT image based on MR image data.

[0196] Alternatively or otherwise, manual identification or segmentation processes can be used to determine various boundaries of MR images. For example, manual identification of echo or ultrasound-imageable portions in an MR image can be performed by the user. Manual identification can be saved relative to the MR image for later association with the ultrasound image. Ultrasound image data can be acquired at any appropriate time, such as during surgery as discussed above. Ultrasound data may include echo features of subject 30 identified in MR image 900 to allow for association with that subject.

[0197] The generated image data (e.g., simulated CT image data) can then be segmented into individual parts, as such Figure 19 The illustrated segmented image data 990. As discussed above, the image data may include the vertebrae of the subject 30. Therefore, the vertebrae (several vertebrae) including the vertebral body 974 can be individually segmented into individual vertebrae, such as vertebrae 974a to 974o. Other anatomical features, such as the sacrum 1010, can also be identified or segmented in the simulated CT image data or data. However, it should be understood that any appropriate portion can be segmented or identified in the simulated CT data. Therefore, various processes associated with each vertebra or other portion within the vertebral body can also be identified in the simulated CT.

[0198] In various implementations, each vertebra can be understood as a distinct or separate entity, such as a rigid structure. However, more than one element can be understood as a single or a single rigid member, such as vertebral bodies 974o and 974n. Based on various characteristics, such as the anatomical structure surrounding the vertebral bodies, the two vertebral bodies 974o and 974n can be substantially immobile relative to each other. However, each vertebral body can be segmented and identified as a separate rigid member, such as... Figure 19 exemplified.

[0199] Further, each of the components can then be registered or correlated to other image data. As discussed above, the ultrasound image 930 can include various features or have identifying features therein that can be registered to the segmented portions in the segmented image 990. Likewise, the segmented image or image data 990 can be segmented simulated CT image data. Thus, the segmented image data 990 can or can not be displayed. However, the segmented image 990 can be registered to one or more image data acquisitions of the ultrasound image data. Thus, the segmented image 990 can be registered to the ultrasound image data (sonogram) and thereafter also to the navigation or patient space of the subject 30.

[0200] In various embodiments, for example, an ultrasound imaging system, such as the ultrasound probe 440 or the ultrasound array 222, can acquire ultrasound data or echo data from various portions of the subject 30, such as the subject's vertebrae. The ultrasound data can then be correlated or registered to the segmented portions in the segmented simulated CT 990. The correlation can then allow the segmented simulated CT image 990 to be registered to the ultrasound or echo data collected with the ultrasound probe.

[0201] The segmented image 990 that can be registered to the tracked ultrasound data can be used to generate a model 1020 as illustrated. Figure 20 The model 1020 can be a selected model, such as a two-dimensional model or a three-dimensional model. The model 1020 can be an entire structure, such as an entire or selected portion of a spine including a plurality of vertebral elements or several individual elements or components. For example, the model 1020 can include each individual element or component, such as each of the vertebrae or vertebral bodies 974a' through 974o'.

[0202] Each of the individual components can be individually registered and / or trackable based on a local rigid registration. The registration can be based on registration of data collected with ultrasound, which can be two-dimensional data registered to three-dimensional data, such as from the MR image 900. The generation or creation of simulated CT image data or simulated CT data can allow registration of the ultrasound data to the three-dimensional data of the MR image 900. Thus, the generated model 1020 can be a three-dimensional model registered to the collected ultrasound data of the subject 30.

[0203] The collected ultrasound data of the subject 30 can be substantially real-time image data. Registration of the real-time image data allows for real-time registration of each individual member of the segmented image 990 in the model 1020. According to various embodiments, the individual or local registration can be based on processes such as those disclosed in U.S. Patent No. 10,262,424 or U.S. Patent No. 11,657,518, both of which are incorporated by reference herein. This allows the model 1020 to instantiate a registered pose of each of the members identified and / or segmented in the simulated CT image, such as from the segmented simulated CT image 990. Thus, a pose of each of the individual members, such as each of the vertebral bodies 974a’-974o’ and / or sacrum 1010’, can be instantiated in the model 1020.

[0204] Due to the registration of the MR image 900 to the ultrasound image and the generation of the model 1020 based thereon, registration and instantiation of the current pose of portions of the MR image 900 can be displayed. Further, various portions in addition to the registered portions can be instantiated. The additional portions can include image data in addition to the image data of the individual vertebrae 974. The additional image data can include non-segmented portions, such as soft tissue or other tissue. These can be instantiated with respect to the model 1020, such as on the display 84. Thus, the model 1020 can be instantiated individually and / or with respect to other image data, such as soft tissue image data based on the MR image 900. The user 72 can view the registered model 1020, the segmented simulated CT 990, and / or the registered model 1020 registered to the subject’s MR image 900 along with related portions or features, such as anatomical portions including soft tissue, and other portions, such as vertebrae.

[0205] As discussed above, according to the process 1100 as Figure 21 instantiated, various image data can be processed and registered images and / or generated models 1020 can be generated that can be registered. The process 1100 can begin in a start block 1110. In the process of initiating the start block 1110, the user 72 or any appropriate individual can operate a processor module or system, including those discussed above, such as the navigation system 26 including the processor module associated therewith. Thus, the process 1100 can be understood as a process that is substantially automatically executed upon initiation by the user by execution of the instructions of the processor module. However, the process can also include various inputs, such as from the user 72 or imaging systems or image data, as further discussed herein.

[0206] The process 1100 can acquire first image data in block 1114. The first image data can include any appropriate image data, such as the MR image data 900. As discussed above, the first image data can be two-dimensional image data or three-dimensional image data. If three-dimensional image data, slices or portions of the 2-dimensional image data can be created therefrom, if selected, for registration or correlation to other data. The image data acquired in block 1114 can be acquired at any appropriate time of the subject. In various embodiments, the image data can be pre-acquired image data, such as image data acquired prior to a current or real-time procedure. Alternatively or additionally, the image data of the subject can be acquired substantially in real-time. In various embodiments, for example, the image data can be acquired with an intraoperative MR scanner to generate image data and images of the subject during surgery, such as when an instrument is positioned within or relative to the subject 30. Thus, the process 1100 allows for the first image to be acquired in block 1114 in a selected manner.

[0207] Then, in block 1118, ultrasound-correlatable image data can be generated from the first image data. The generation of the correlatable image data can include various processes that can generally allow for the generation, creation or identification of data or portions of the first image data that can be related to the echo data. For example, various edges or surfaces can be identified in the first image data that can also be acquired in the ultrasound image or data acquisition. In various embodiments, the correlatable image data can be a simulated CT, such as the simulated CT 970.

[0208] The simulated CT can be generated on or based on the first image data with an appropriate process, such as with a trained machine learning system. The trained machine learning system can analyze or process the first image data to identify or generate an image or image data that simulates a computed tomography scan of the features included in the image data. Thus, in various embodiments, the simulated CT image or image data can be generated in block 1118 based on the first image data.

[0209] The generation of the simulated CT can also be performed substantially automatically, such as after the first image data is acquired in block 1114. Additionally or alternatively, the generation of the simulated CT image data can be generated at any appropriate time. Moreover, in accordance with the discussion herein, the simulated CT image data can be understood to relate to a particular type of data or image data, but can also relate to a general process or data that can be correlated with ultrasound image acquisition. Thus, the discussion herein can be understood to encompass the general type of image data that can be generated in block 1118, unless specifically indicated otherwise. Thus, the simulated CT image data can be understood to be exemplary image data generation, unless specifically indicated otherwise.

[0210] The correlatable image data can then optionally be segmented in block 1122. Segmentation of the correlatable image data can allow for segmentation of various features in the correlatable image data, such as particular portions of the vertebrae as discussed above. The correlatable image data can be segmented to identify or determine various portions in the image data, such as the vertebral body 974 or the spinous process 982 as indicated above. As discussed above, additional various portions can be segmented to identify edges or surfaces thereof. The process can be based on various generally known processes, such as segmentation of computed tomography image data.

[0211] The process 1100 can then also allow for acquisition of second image data in block 1126. The acquisition of second image data in block 1126 can be image data generated or needed with an ultrasound array, as discussed above. According to various embodiments, an ultrasound array, such as the array 222, can be used to generate ultrasound image data of the subject 30. Additionally or alternatively, an ultrasound probe, such as the probe 400, can be used to acquire the second image data. The second image data of the subject 30 can be acquired in an appropriate manner. The second image data can include ultrasound image data, as discussed above.

[0212] During acquisition of the second image data, the imaging system can be optionally tracked in block 1130. Tracking of the acquisition of the second image data can include tracking the ultrasound probe 400, tracking the array 222, or any appropriate tracking process. As discussed above, various ultrasound systems can be tracked in a navigation space. The navigation space can include, be registered to, or be defined relative to the subject 30. Thus, the ultrasound array can generate image data in a known or registered pose relative to the subject 30. The imaging system and the acquired image data can be registered to the subject 30, such as for navigation and / or registration of other coordinate systems, such as navigation and / or registration of the first image data according to the process 1100 as further discussed herein.

[0213] In block 1134, features can be identified in the second image data. The features in the second image data can include various features, such as including particular portions of the vertebrae and / or edges or features, such as the edge 938 or the articular surface 944 as illustrated and discussed above. The various identified features can be used for correlation with the correlatable image data, such that in block 1140, the second image data can be correlated with the correlatable image data. Figure 17

[0214] ​In block 1140, the association of the second image data with the correlatable image data can be based on the generated correlatable image data and the identified features from block 1134. The association can include an identification or selection of features or portions that exist in both the second image data and the correlatable image data. For example, as illustrated, the joint surface 944 identified in the ultrasound image can also be identified in the correlatable image data, such as the simulated CT Figure 17 Figure 18A and Figure 18B as illustrated. Thus, in block 1140, the second image data and the correlatable image data can be associated by allowing a matching or correlating of various features in the two image data sets. In other words, the association in block 1140 can be associating the intermediate image data with the second image data by correlating features identified in both the second image data and the intermediate image data.

[0215] Then, in block 1144, the second image data can be registered to the first image data. The registration of the first image data to the second image data can be based on the association of the second image data features with the correlatable image data in block 1140. Since the correlatable image data is based on or generated from the first image data, the pose or coordinates of the features in the first image data can be identified in the first image data based on the correlatable image data. Thus, once the association of the second image data with the correlatable image data is made in block 1140, a registration can be made between the second image data and the first image data. As discussed herein, this can allow for a registration of the first image data and the second image data for various purposes. According to various embodiments, for example, a selectable generation of a model can be based on the registration in block 1150. The model can include a model 1020 as discussed above in the model.

[0216] As discussed above, the model can allow for the display of the model 1020 on the display device 84. In various embodiments, the generation of the model can enable a clarity of the display, an ease of manipulation of the display, or other appropriate purposes. However, the model can be generated in block 1150. The model can be an image generated based on the registration, but entirely for the display. However, in various embodiments, the registration of the second image data to the first image data can allow for an updating of the first image data, such as portions of a segmentation thereof, to illustrate a current or real-time pose of portions in the first image data to match a current pose of the subject 30.

[0217] ​In block 1160, a determination can be made as to whether to register the first image data to the navigation space. If a determination is made that no registration will occur, then an "NO" path 1164 can be followed to an output in block 1170. The output in block 1170 can be of various types, such as an output to memory for later recall, an output of the generated model from block 1150 if generated or selected, or an output of the registered image based on the registration from block 1144. Thus, the user 72 can save and / or allow display of the output from block 1170 for various purposes. The process 1100 can then end in an end block 1174. However, the ending of the process in block 1174 can allow various other actions to occur, such as illustration of the image data of the subject, planning of a procedure with respect to the subject, or other various purposes, including those also discussed herein.

[0218] According to various embodiments, a determination can be made in block 1160 that registration occurs, and then a "YES" path 1180 can be followed. The "YES" path 1180 can allow the tracked pose of the second image data to be associated with the first image data via the registration in block 1184. Thus, the output can then follow the association in block 1170 and include a registered or updated pose of the first image data based on the tracked pose in the second image data. Thus, the output can include display of various portions (such as the model, images, etc.) to illustrate the portions in a current or real-time pose based on the acquisition of the tracked second image data in block 1130 and its association in block 1184. Thus, for example, as illustrated in FIG. 11B, the model 1020 can be illustrated in a current or real-time pose based on the tracked second image data and its association in block 1184. Figure 20 The illustrated model 1020 can illustrate the current or real-time pose of various display elements, such as the segmented vertebrae. Further, as discussed above, each of the individual vertebrae can be tracked individually based on the second image data and displayed based on its local registration. This can allow the user 72 to understand the real-time pose of various features imaged in the first image data based on the tracked portions and the second image data. Additionally, as discussed above, various features can be illustrated with respect to the model portions 1020 and / or the first image data can be displayed with the features to which it is registered to allow the user 72 to understand the context of the tracked or registered portions. Further, the first image data can include a greater context or volume than the second image data, such as can be included in MR image data as compared to ultrasound image data. Thus, the registration or association in block 1184 can allow real-time illustration or display based on or updated from real-time image data acquired with the ultrasound system, such as the MR image 900.

[0219] Accordingly, the process 1100 can allow for registration of the first image data and the second image data. Further, the registration of the first image data and the second image data can allow for generating and displaying real-time poses of features in the first image data based on the second image data. This can allow the user 72 to better understand the current or real-time pose of various portions, such as based on acquiring image data of the subject with the second image data as the imaging system is tracked.

[0220] Further, the end block 1174 can be the end of an iteration of the process 1100. Accordingly, the user 72 or other appropriate user can select to perform the process 1100 again. In various embodiments, the process 1100 can also be performed automatically based on the collection of the second image data. Accordingly, the display of the real-time poses and the registration or updates can be continuously or at selected intervals by the operations of the process 1100. Thus, the process 1100 can be understood as a single iteration and can be performed at a selected rate and / or a selected number of times.

[0221] According to various embodiments, the ultrasound probe can emit or transmit ultrasound waves in a selected pattern or plane. The plane can be a shape as understood by one of skill in the art. The plane is generally capable of acquiring data in a field of view to generate an image, also referred to as an echogram, when generating an image based on the ultrasound data.

[0222] Exemplary embodiments are provided so as to convey the scope of the disclosure to those skilled in the art. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. Those skilled in the art will recognize, however, that the exemplary embodiments can be practiced without the specific details

[0223] Instructions can be executed by the processor, and can be included in firmware, software, and / or microcode, and can refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuitry" encompasses a single processor circuitry executing some or all code from multiple modules. The term "grouped processor circuitry" encompasses a processor circuitry executing some or all code from one or more modules, collectively. A reference to multiple processor circuitry encompasses a single processor circuitry on a single die, a single processor circuitry that is a multitude of processor cores, a single processor circuitry that is a multitude of threads, or a combination thereof. The term "shared memory circuitry" encompasses a single memory circuitry that stores some or all code from multiple modules. The term "grouped memory circuitry" encompasses a memory circuitry that stores some or all code from one or more modules, collectively.

[0224] The apparatus and methods described in this application can be implemented partly or entirely by a processor (also referred to as a processor module), which can include a special purpose computer (e.g., created by configuring the processor) and / or a general purpose computer, for executing one or more specific functions embodied in computer programs. The computer programs include processor-executable instructions stored on at least one non-transitory, tangible computer-readable medium. The computer programs can also include or rely on stored data. The computer programs can include a basic input / output system (BIOS) that interacts with hardware of the special purpose computer, device drivers that interact with particular devices of the special purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0225] The computer programs can include: (i) assembly code, (ii) object code generated from source code by a compiler, (iii) source code for execution by an interpreter, (iv) source code for compilation and execution by a just-in-time compiler, (v) descriptional text for parsing, such as HTML (HyperText Markup Language) or XML (Extensible Markup Language), etc. By way of example only, source code can be written using C, C++, C#, Objective-C, Haskell, Go, SQL, Lisp, Java ® , ASP, Perl, Javascript ® , HTML5, Ada, Active Server Pages (ASP), Perl, Scala, Erlang, Ruby, Flash ® , Visual Basic ® , Lua, or Python ® .

[0226] Communications can include wireless communications described in the present disclosure, which can be conducted entirely or partially in compliance with IEEE Standard 802.11-2012, IEEE Standard 802.16-2009, and / or IEEE Standard 802.20-2008. In various implementations, IEEE 802.11-2012 can be supplemented by draft IEEE Standard 802.1 lac, draft IEEE Standard 802.1 lad, and / or draft IEEE Standard 802.1 lah.

[0227] Processor, processor module, module, or "controller" are used interchangeably herein (unless specifically indicated otherwise) and each can be replaced with the term "circuit." Any of these terms can refer to, be part of, or include: an Application Specific Integrated Circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip.

[0228] Instructions can be executed by one or more processors or processor modules, such as one or more digital signal processors (DSPs), general purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor" or "processor module" as used herein can refer to any of the foregoing structure or any other physical structure suitable for implementation of the described techniques. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0229] The foregoing description of implementations has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Individual elements or features of a particular implementation are generally not limited to that particular implementation, but, where applicable, are interchangeable and can be used in a selected implementation, even if not specificaily shown or described. The same element or feature can be varied from implementation to implementation. Such variations are not to be regarded as a departure from the scope of the invention, and all such modifications are intended to be included within the scope of the invention.

Claims

1. A method of registering first image data to second image data via intermediate image data, the method comprising: acquiring first image data; acquiring second image data; generating the intermediate image data from the first image data that is correlatable with the second image data; correlating the intermediate image data with the second image data; and registering the first image data to the second image data.

2. The method of claim 1, wherein acquiring the first image data comprises acquiring magnetic resonance image data.

3. The method of claim 2, wherein generating the intermediate image data from the first image data that is correlatable with the second image data comprises generating simulated computed tomography image data based on the first image data.

4. The method of claim 2, wherein generating the intermediate image data from the first image data that is correlatable with the second image data comprises identifying echo features in the first image data.

5. The method of any one of claims 1-4, wherein generating the intermediate image data from the first image data that is correlatable with the second image data is automatically generated by executing instructions of an algorithm with a processor module.

6. The method of any one of claims 1-5, wherein generating the intermediate image data from the first image data that is correlatable with the second image data is automatically generated by processing the first image data with a trained model.

7. The method of any one of claims 1-6, wherein correlating the intermediate image data with the second image data comprises correlating features identified in both the second image data and the intermediate image data.

8. The method of claim 7, wherein registering the first image data to the second image data comprises defining a translation between a first image coordinate system and a second image coordinate system based on the correlation of the intermediate image data with the second image data.

9. The method of claim 8, further comprising: registering the first image coordinate system to a subject coordinate system.

10. The method of claim 9, further comprising: tracking an imaging system in the subject coordinate system; and wherein the acquired second image data is acquired with the imaging system while the imaging system is tracked.

11. The method of any one of claims 1-10, further comprising: tracking an imaging system in a subject coordinate system while the second image data is being acquired with the imaging system being tracked; registering the first image data to the subject coordinate system based on the registration of the first image data to the second image data.

12. A system of registering first image data to second image data via intermediate image data, the system comprising: a processor module configured to execute instructions to: acquire first image data; acquire second image data; ​ ​ generating the intermediate image data from the first image data that is correlatable with the second image data; correlating the intermediate image data with the second image data; and registering the first image data to the second image data.

13. The system of claim 12, wherein the first image data is magnetic resonance image data; wherein the processor module is configured to execute further instructions to generate simulated computed tomography image data based on the first image data as the intermediate image data from the first image data that is correlatable with the second image data.

14. The system of any one of claims 12 or 13, wherein the first image data is magnetic resonance image data; wherein the processor module is configured to execute further instructions to identify echo features in the first image data to generate the intermediate image data from the first image data that is correlatable with the second image data.

15. The system of any one of claims 12 to 14, further comprising: a memory system having a trained model stored thereon; wherein the intermediate image data from the first image data that is correlatable with the second image data is automatically generated by processing the first image data with the trained model.

16. The system of any one of claims 12 to 15, wherein the processor module is configured to execute further instructions to: relate features in the second image data to features in the intermediate image data to correlate the intermediate image data with the second image data; define a translation between a first image coordinate system and a second image coordinate system based on the correlation of the intermediate image data with the second image data to register the first image data to the second image data.

17. The system of claim 16, wherein the processor module is configured to execute further instructions to: register the first image coordinate system to a subject coordinate system.

18. The system of claim 17, further comprising: a tracking system to track an imaging system in the subject coordinate system; and wherein the acquired second image data is acquired with the imaging system while the imaging system is tracked.

19. The system of any one of claims 12 to 18, further comprising: an imaging system to acquire the second image data; and a tracking system to track the imaging system in a subject coordinate system while the imaging system is acquiring the second image data; wherein the processor module is configured to execute further instructions to register the first image data to the subject coordinate system based on the registration of the first image data to the second image data.

20. The system of claim 12, further comprising: a first imaging system to acquire the first image data; a second imaging system for acquiring the second image data; a navigation system for tracking at least the second imaging system in a subject coordinate space when acquiring the second image data to allow registration of the first image data to the subject coordinate system.

Images