System for surgical planning, surgical navigation and imaging

Abstract

A computer-implemented method for adjustable three-dimensional (3D) rendering of a location in a subject includes the step of receiving image data for white matter tracts in a subject with discrete spatial resolution, wherein the image data is formed by diffusion imaging. A first stereoscopic image is rendered on a display from the image data. The first stereoscopic image includes image parallax that allows for three-dimensional viewing. Input signals from a pointing device or gestures that allow manipulation of the first stereoscopic image while maintaining correct three-dimensional spatial relationships are received. The pointing device interacts with the first stereoscopic image in three-dimensional space with six degrees of freedom.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 826,857, filed March 29, 2019, the disclosure of which is hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] In at least one aspect, the present disclosure relates to medical imaging systems and their application for surgical planning and surgical navigation. BACKGROUND

[0004] Medical imaging using technologies such as magnetic resonance imaging, CAT scans, and PET scans is an important diagnostic tool. With the development of imaging and computer technology, additional diagnostic information can be obtained through these technologies. SUMMARY

[0005] In at least one aspect, the present disclosure provides a computer-implemented method for adjustable three-dimensional (3D) rendering of a location in a subject. The computer-implemented method includes the step of receiving image data for white matter tracts in a subject having a discrete spatial resolution, wherein the image data is formed by diffusion imaging. A first stereoscopic image is rendered on a display or one or more head-mounted displays from the image data. The first stereoscopic image includes image parallax that allows for three-dimensional viewing. Input signals from a pointing device or gestures that allow manipulation of the first stereoscopic image while maintaining correct three-dimensional spatial relationships are received. The pointing device interacts with the first stereoscopic image in three-dimensional space with six degrees of freedom.

[0006] In another aspect, a computer-implemented method for adjustable rendering of a location in a subject is provided. The computer-implemented method includes the step of receiving image data for a location in a subject having a discrete spatial resolution. One or more stereoscopic images are rendered on a display or one or more head-mounted displays from the image data, wherein the stereoscopic images include image parallax. One or more two-dimensional images corresponding to predetermined slices in the first stereoscopic image are also rendered on the display. The stereoscopic images are updated to reflect changes made to the two-dimensional images. Similarly, the two-dimensional images are updated to reflect changes made to the stereoscopic images. In particular, changes are made to the first stereoscopic image with the aid of a pointing device that interacts with the first stereoscopic image in three-dimensional space with six degrees of freedom.

[0007] In another aspect, a computer-implemented method for adjustable three-dimensional (3D) rendering to assist surgery is provided. The computer-implemented method includes the step of receiving image data for medical images in an object having a discrete spatial resolution. At least one stereoscopic image is rendered from the image data on a display or one or more head-mounted displays. The stereoscopic image includes image parallax to allow 3D viewing. An input signal from a probe is received that allows rendering of the probe in the stereoscopic image. The probe also allows manipulation of the stereoscopic image in three-dimensional space while maintaining correct three-dimensional spatial relationships. In particular, the probe interacts with the first stereoscopic image in three-dimensional space with six degrees of freedom.

[0008] In yet another aspect, a computer imaging system for implementing the computer-implemented methods set forth herein is also provided. The computer imaging system includes a computer processing component and a display. The computer processing component is operable to perform the steps of the computer-implemented methods.

[0009] In yet another aspect, the stereoscopic images rendered by the methods herein can be viewed through head-mounted displays and / or stereoscopic glasses for virtual or augmented reality or stereoscopic display, the stereoscopic glasses having fiducial points that allow position and orientation tracking of the stereoscopic glasses. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a schematic diagram of a computer system for implementing the method of providing adjustable three-dimensional (3D) rendering of locations in an object.

[0011] Figure 2 The rendering content of the stereoscopic image is provided relative to Figure 1 the stereoscopic image is rotated.

[0012] Figure 3A , 3B and 3C represent the rendering content of the user panning the reference plane.

[0013] Figure 4A and 4B shows the selection of a sub-component of the first stereoscopic image that can be highlighted, segmented to form a stereoscopic sub-image that can be edited independently at the voxel level.

[0014] Figure 5A and 5B represents the use of a tracking container that performs operations on the rendered stereoscopic image. Figure 5B shows the application of an orientation container in the form of a rectangular box.

[0015] Figure 6A and 6B shows the use of two containers 60 1,60 2 Result of an "AND" operation when placed on an image.

[0016] Figure 7A and 7B Illustrates the combined action of two containers to define the "OR" operation shown.

[0017] Figure 7C Illustrates the "NOT" operation.

[0018] Figure 8A Indicates a scenario where the container is fixed to a third reference plane.

[0019] Figure 8B Indicates a scenario where the container is fixed at the intersection of three reference planes.

[0020] Figure 9 Is a schematic representation of a scenario where an additional container deletes a portion of the fiber 70.

[0021] Figure 10 A schematic of an imaging system for implementing a method for providing adjustable two-dimensional views and three-dimensional volume rendering of the coexistence of locations in an object is provided.

[0022] Figure 11 A schematic of a surgical navigation system is provided.

[0023] Figure 12A A flowchart of a workflow for representing white matter fiber visualization in a 3D scene is provided.

[0024] Figure 12B A view of imaging data with structural imaging data is provided.

[0025] Figure 12C A flowchart representing rendering of fibers (line sets) by providing a pipeline with coordinates and colors is provided.

[0026] Figure 13A A view representing fiber tracking using a stylus is provided.

[0027] Figure 13B A level conversion of a rendered scene is provided.

[0028] Figure 14 A flowchart of a hybrid rendering that renders both structural and fiber data into a scene is shown.

[0029] Figure 15 A flowchart of a program representing how structural and fiber data are co-registered and co-rendered is shown. DETAILED DESCRIPTION

[0030] Reference will now be made in detail to the presently preferred embodiments and methods of the application, which constitute the best present understanding of the inventors of the optimal ways to implement the application. The drawings are not necessarily to scale. It is understood that the disclosed embodiments are merely examples of the application and can be practiced in various and alternative forms. Therefore, specific details disclosed herein are not to be interpreted as limiting, but rather as being representative of the many possible embodiments of the application.

[0031] It is also to be understood that the application is not limited to the specific embodiments and methods described below, as specific components and / or conditions can necessarily be modified. Furthermore, the terminology used herein is for the purpose of describing particular embodiments of the application only and is not intended to be limiting in any manner.

[0032] It must also be understood that, as used in the specification and the appended claims, the singular form "a", "an" and "the" include plural references unless the context clearly dictates otherwise. For example, a reference to an element by the singular form includes a plurality of elements.

[0033] The words "comprise", "comprising", "have", "having", "include", "including", "contain", "containing", "characterized by" and the like are synonymous. These words are inclusive and open-ended, and do not exclude additional, unrecited elements or method steps.

[0034] The phrase "consisting of" excludes any element, step, or ingredient not specified in the claim. When this phrase appears in the body of a claim, it is limiting only to the elements set forth in that clause; other elements are not excluded from the claim as a whole.

[0035] The phrase "consisting essentially of" limits the scope of a claim to the specified materials or steps and those that do not materially affect the basic and novel characteristic(s) of the claimed subject matter.

[0036] With respect to the words "comprise", "consist of", and "consisting essentially of", where one of these words is used in the following claims, the present disclosure and claimed subject matter can include the use of any of the other two words in place of the used word.

[0037] The word "server" refers to any computer, computing device, mobile phone, desktop, laptop or portable computer, distributed system, blade server, gateway, switch, processing device, or combination thereof adapted to perform the methods and functions described herein.

[0038] When a computing device is described as performing an action or method step, it should be understood that the computing device can generally operate to perform the action or method step by executing one or more lines of source code. The action or method step can be written into non-transitory memory (e.g., a hard drive, an optical drive, a flash drive, etc.).

[0039] The term "computing device" generally refers to any device (e.g., a computer) that can perform at least one function that includes communicating with another computing device.

[0040] In this application, where a publication is referenced by a number, it is understood that this reference is to the complete publication, which is incorporated by reference in its entirety for all purposes.

[0041] The term "stereoscopic image" refers to an image having at least two views, one of which corresponds to a left-eye view and the other of which corresponds to a right-eye view. When the images are viewed using appropriate equipment, a three-dimensional rendering is achieved in the user's brain.

[0042] It should be understood that any manipulation of a stereoscopic image can be applied to each image in a set or multiple sets of stereoscopic images.

[0043] The programs, methods, or algorithms disclosed herein can be implemented in a computing device, controller, or computer that can include any existing programmable electronic control unit or a specially designed electronic control unit, or by such a device, controller, or computer. Similarly, the programs, methods, or algorithms can be stored as data and instructions in any form suitable for being executed by a controller or computer, including but not limited to information permanently stored on non- writable storage media such as ROM devices, and information variably stored on writable storage media such as floppy disks, tapes, CDs, RAM devices, and other magnetic and optical media. The programs, methods, or algorithms can also be implemented in software executable objects. Alternatively, the programs, methods, or algorithms can be implemented in whole or in part using appropriate hardware components, such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0044] In this application, where a publication is referenced by a number, it is understood that this reference is to the complete publication, which is incorporated by reference in its entirety for all purposes.

[0045] Abbreviations:

[0046] "AR" refers to augmented reality.

[0047] "DTI" refers to diffusion tensor imaging.

[0048] “CPU” refers to a central processing unit.

[0049] “CT” refers to computed tomography.

[0050] “CTA” refers to computed tomographic angiography.

[0051] “ID” refers to identification.

[0052] “fMRI” refers to functional magnetic resonance imaging.

[0053] “MRA” refers to magnetic resonance angiography.

[0054] “MRI” refers to magnetic resonance imaging.

[0055] “PET” refers to positron emission tomography.

[0056] “PWI” refers to perfusion weighted imaging.

[0057] “SWI” refers to susceptibility weighted imaging

[0058] “VR” refers to virtual reality.

[0059] Reference is made to Figure 1 and 2 schematically illustrating a computer imaging system and a method of providing an adjustable three-dimensional (3D) rendering of a location in a subject implemented by the computer imaging system. Figure 1 A schematic of an imaging system that can implement the computer implemented method is provided. The computer imaging system 10 includes a display 12 for displaying a rendered stereoscopic image 14 of a portion of a subject (e.g., a patient) that can be viewed through stereoscopic viewing glasses 16. Alternatively, the stereoscopic image can be viewed using an AR or VR headset. It will be appreciated that while the images in this document are shown in black and white, the actual stereoscopic images can be rendered in color. White matter fibers can be edited to any color based on their fiber orientation, location, or sequential numbering as desired by the user. Common color coding of white matter tracts and fibers is green for tracts and fibers that are primarily aligned along the anterior / posterior direction, red for tracts and fibers that are primarily aligned along the left / right direction, and blue for tracts and fibers that are primarily aligned along the superior / inferior direction.

[0060] The computer imaging system 10 also includes a computer processor assembly 18 that includes a CPU 20, input / output interfaces 22, memory 24, and storage devices 26. The computer processor assembly 18 operatively performs the steps of the computer-implemented method. In one refinement, the computer system 10 also includes a tracking system 28 that is optionally mounted to the display 12 that can monitor the position and orientation of the stereoscopic viewing glasses 16, a pointing device 30, and optional gestures from the user. The motion and actions of the pointing device 30 are reflected on the display 12 as a pointer cursor 32. A keyboard 34 can be used to input commands and other information into the computer imaging system. The computer-implemented method includes the step of receiving image data for white matter tracts and / or fibers in a subject (e.g., a patient) having a discrete spatial resolution. Typically, the image data is generated by diffusion imaging, particularly by magnetic resonance diffusion imaging. With this imaging, the first stereoscopic image 14 provides white matter fiber tracking of brain and spinal cord tissue. The first stereoscopic image 14 is rendered on the display 12 based on the image data. The first stereoscopic image includes image parallax to provide a 3D rendering. Thus, the first stereoscopic image can be viewed through stereoscopic or polarized glasses 16 that selectively have fiducials that allow position and orientation tracking of stereoscopic glasses. While the glasses are moved, the stereoscopic image will be dynamically updated to display different perspectives of the rendered view to achieve a 3D viewing effect. Alternatively, the stereoscopic image can be viewed through a head-mounted AR or VR display that is capable of eye tracking or head motion tracking. While the user's visual focus or head is moved, the stereoscopic image will be dynamically updated to reflect corresponding perspectives of the rendered view to achieve a 3D viewing effect. Input signals from the pointing device 30 that allow manipulation of the first stereoscopic image while maintaining correct three-dimensional spatial relationships are received. In particular, the pointing device 30 interacts with the first stereoscopic image in three-dimensional space with six degrees of freedom (i.e., three translational degrees of freedom and three Euler angles). The ability of the pointing device 30 to interact with six degrees of freedom allows the user to reach behind the stereoscopic image 14 and perform manipulations, which are set forth in greater detail below. In this regard, the pointing device 30 facilitates rotation, scaling, translation, editing, highlighting, adding (e.g., increasing), segmenting, and / or deleting of the first stereoscopic image. Typically, the pointing device 30 includes one or more buttons 35 for initiating commands. Figure 2 The rendered content of the stereoscopic image 14 is provided by applying rotation with the pointing device 30.

[0061] In one variation, a first reference plane 36 and a second reference plane 38 are also rendered on the display 14. The first reference plane 36 and the second reference plane 38 each independently intersect the first stereoscopic image 14, rendering a first reference image 40 on the first reference plane 36 and a second reference image 42 on the second reference plane 38. Thus, the first reference plane 36 and the second reference plane 38 combine with the first stereoscopic image 14 as a second stereoscopic image. The first reference plane 36 is oriented at a first angle Al, typically 30 to 150 degrees, relative to the second reference plane 38. Input signals from the pointing device 30 can be received that select a position of the first reference plane or the second reference plane or rotate them at any angle, with the first two-dimensional image and the second two-dimensional image being updated as their position or orientation changes (e.g., rotated at any angle).

[0062] Reference Figure 1 , 2 , 3A and 3B, a third reference plane 44 that intersects the first stereoscopic image 14 can also be displayed intersecting the first stereoscopic image 14. In particular, a third reference image 46 is rendered on the third reference plane 44. The third reference plane is oriented at a second angle A2 relative to the first reference plane 36 and at a third angle A3 relative to the second reference plane 38. In one refinement, the second angle A2 and the third angle A3 are each about 30 to 150 degrees. Input signals from the pointing device 30 can be received that select a position of the first reference plane 36 or the second reference plane 38 or the third reference plane 44 or rotate them at any angle. The reference planes are updated as they are moved or rotated in this manner (i.e., as the position or orientation changes). Advantageously, the user can translate or rotate the first reference plane 36 along a first axis 50 or the second reference plane 38 along a second axis 52 or the third reference plane 44 along a third axis with the two-dimensional reference image associated with the translating reference plane being updated during the translation. Figure 3A and 3B The rendered content is shown after the user has translated the second reference plane 38 along the axis 52. It is observed that the second reference image has changed as the reference plane 38 has moved from position Pl to position P2. It is also understood that the reference image is continuously and dynamically updated as the reference plane moves. Figure 3C A stereoscopic image of the fiber bundle options highlighted in Figure 3A and 3B is provided. Details of making such a selection are set forth below.

[0063] As described above, a first reference image 40 is rendered on a first reference plane 36, a second reference image 42 is rendered on a second reference plane 38, and a third reference image 46 is rendered on a third reference plane 44. In general, the reference images are two-dimensional or three-dimensional images of organs, anatomical structures, or pathological features (e.g., tumors, aneurysms, etc.) of the subject, respectively and independently. In one refinement, the respective display content of the reference images corresponds to a slice of the first volume image 14 formed by the intersection of the reference plane with the first volume image 14. The reference images can be two-dimensional images of the anatomical structure of the subject formed, for example, from CT data, MRI data, digital subtraction angiography, white matter tractography data, ultrasound, mammography, PET, optoacoustic images, or any data further derived from these data sets. In another refinement, the user can initiate a program to display reference images based on previously generated image data for display with the first volume image. Examples of previously generated image data also include, but are not limited to, CT data, MRI data, digital subtraction angiography, white matter tractography data, ultrasound, mammography, PET, optoacoustic images, or any data further derived from these data sets.

[0064] Figure 4A and 4B A segmented subcomponent of the first volume image is shown highlighted and selected to form a volume subimage that can be independently edited, manipulated, or measured at the voxel level. The volume image 14' includes a subcomponent 66 that can be selected (reflecting the motion of the pointing device) by the pointing device 32. The subcomponent 50 can then be rendered so that it is isolated from the remainder of the volume image 14'. This isolated subcomponent 50 can now be edited at the voxel level.

[0065] Referring Figure 5A and 5B A screen shot is provided showing the execution of an operation on the rendered volume image using a tracking container. The user initiates the rendering of a three-dimensional tracking container 60 on the first volume image 14 by the pointing device 30 or a gesture. Figure 5A The entry into edit mode is indicated because the container 60 has been rendered at the head of the cursor 32. Figure 5A It is also indicated that the container 60' has been placed on the volume image 14 in this manner. The three-dimensional tracking container 60 1White matter tracts and / or fibers 62 that pass through the three-dimensional tracking container 60 are highlighted. In one refinement, white matter tracts and / or fibers that do not pass through the three-dimensional tracking container 60 are removed or hidden from the first stereoscopic image 14. The current variation is not limited by the shape of the three-dimensional tracking container 60. Suitable shapes include, but are not limited to, a sphere, a disk, a cylinder, an ellipsoid, a tube, a cube, a parallelepiped, an image slice, a block of a 3D dissected organ, a hand-drawn shape, or a surgical corridor, or any shape imported from another source. In another refinement, the three-dimensional tracking container 60 is moved in response to user input (e.g., by the pointing device 30) so that the highlighted fibers that pass through the container are dynamically updated as the three-dimensional tracking container moves.

[0066] In one refinement, the method further includes freezing the results of the application of one or more three-dimensional tracking containers.

[0067] Figure 5B The application of a container in the form of a rectangular box is shown. In this example, the application container 60 is used to selectively highlight fibers 62 that pass through the rectangular box. When the thickness of the rectangular box is reduced to a thin slice or plane, those fibers with a principal orientation that crosses the slice will be displayed. Such a rectangular box can also be a block or slice of a dissected structure. By adjusting the orientation of the slice or plane, the user can select different fiber orientations.

[0068] Multiple three-dimensional tracking containers can be rendered on the first stereoscopic image 14 to perform complex manipulations of the rendered stereoscopic image. In one refinement, the fibers are displayed or highlighted according to any combination of Boolean operations associated with each of the multiple three-dimensional tracking containers. Figure 6A and 6B The result of an "AND" operation is shown when two containers 60 1 , 60 2 are placed on the image. Since each container highlights only fibers that pass through it, the result of the "AND" operation is that only fibers 64 that pass through both containers are highlighted. Figure 7A and 7B The combined action of two containers to define the "OR" operation shown is illustrated. In this case, fibers 66 that pass through either container 60 1 or 60 2 are highlighted. In one refinement, the fibers are displayed or highlighted in the first stereoscopic image according to any combination of Boolean operations associated with each of the multiple three-dimensional tracking containers. It should be appreciated that complex Boolean operations can be implemented on the stereoscopic image 14 by applying a combination of "AND" and "OR" operations. Figure 7CThe structure for the "NOT" operation is shown. In this variation, the fiber 66 passing through container 60 is not shown or is removed.

[0069] In one improvement, the 3D tracking container 60 is fixed to a first reference plane 36 and / or a second reference plane 38, such that the 3D tracking container 60 can move in accordance with user-guided movements (e.g., by means of a pointing device 30 or gestures) of the first reference plane 38 and / or the second reference plane 38 and / or the third reference plane 44. Figure 8A Provides container 60 1 An example of being fixed to the third reference plane 44. Furthermore, the 3D tracking container 60 can be fixed at the intersection of the first reference plane 36, the second reference plane 38, and the third reference plane 44, such that the 3D tracking container 60 can move in unison with user-guided movements (e.g., by means of the pointing device 30 or gestures) of any one of the first reference plane 36, the second reference plane 38, and the third reference plane 44. Figure 8B Container 60 is shown 1 It is fixed at the intersection of the first reference plane 36, the second reference plane 38, and the third reference plane 44. Similarly, in each of these cases, the reference plane will move in sync with the user-guided motion of the 3D tracking container 60.

[0070] In another improvement, the results of operations on one or more 3D fiber bundles can be frozen (i.e., locked while other editing functions are performed). Frozen fibers can also be unfrozen, allowing the user to further edit them. Frozen fiber groups can be saved to a file and exported. Furthermore, previously saved or imported frozen fibers can also be loaded into the computer.

[0071] exist Figure 9 In another improvement shown, once container 60 is applied... 1 To generate subgroups of fiber 70, other containers 60 can also be further applied. 2 and 60 3 This is used to generate subgroups for manipulating fibers at a finer level. Figure 9 This indicates a case where an additional container removes some portions of fiber 70. In another improvement, the user can selectively show or hide fibers that extend primarily in a specific direction (e.g., front-back, left-right, up-down) to generate subgroups of fibers.

[0072] It should be understood that the fiber tracing scene described herein (including all fibers and their orientations) can be saved to a file and exported. Previously saved or imported fiber tracing scenes can also be loaded into a computer to reproduce a scene exactly the same as the previous one.

[0073] In another refinement, during the tracking of specific fiber pathways of the brain or spinal cord of a subject, the system can provide the user with a reference fiber pathway to identify the shape, orientation and location of fibers. In an advantageous application, the fiber tracking environment can also be co-registered with CT, MRI, DSA, PET, ultrasound, mammography or optoacoustic data of the subject to form a composite organ image. The user can plan a surgical pathway or surgical corridor in the composite 3D organ. During the movement or adjustment of the surgical pathway or surgical corridor, all fibers passing through the surgical pathway can be dynamically highlighted. The computer-implemented method of providing an adjustable three-dimensional (3D) rendering of a location in a subject described above can be advantageously integrated into a surgical navigation system to be described in detail below.

[0074] In another embodiment, a computer imaging system and a method implemented by the computer imaging system of simultaneously providing adjustable two-dimensional and three-dimensional (3D) renderings of a location in a subject are provided. Figure 10 A schematic diagram of an imaging system that can implement the computer-implemented method is provided. The computer system 10 includes a display 12 or one or more head-mounted displays for displaying rendered stereoscopic images 14 of a portion of a subject that can be viewed through stereoscopic or polarized viewing glasses 16. It is understood that while the images are shown here in black and white, the actual stereoscopic images can be rendered with color as described above. The computer system 10 also includes a computer processor assembly 18 including a CPU 20, input / output interface 22, memory 24 and storage 26. In a refinement, the computer system 10 also includes a tracking system 28 selectively mounted to the display 12 that can monitor the position and orientation of the stereoscopic viewing glasses 16, a pointing device 30 and optional gestures from the user. The motion and actions of the pointing device 30 are reflected on the display 12 as a pointer cursor 32. A keyboard 34 can be used to input commands and other information into the computer system. The method implemented by the computer system 10 includes the step of receiving image data for a location in a subject (e.g., white matter tracts) having a discrete spatial resolution. At least one stereoscopic image 90 is rendered on the display 12 based on the image data. Figure 10The rendering of two stereoscopic images 90, 92 is shown. The stereoscopic images include image parallax to provide 3D rendering. Thus, the stereoscopic images can be viewed through stereoscopic glasses 16 or a stereoscopic display having fiducials that allow position and orientation tracking of the stereoscopic glasses. In a refinement, the stereoscopic images can be displayed on a polarized stereoscopic screen or a head mounted display. Alternatively, a head mounted VR or AR display can be used. The head mounted display has a camera for tracking the orientation or position of the visual focus or a pointer or gesture for interaction. One or more two-dimensional images 94, 96 corresponding to predetermined slices in the first stereoscopic image are also rendered on the display(s) 12 (e.g., a polarized stereoscopic screen or a head mounted display). Input signals from the pointing device or gesture are received that allow manipulation of the stereoscopic images and the two-dimensional images while maintaining the correct three-dimensional spatial relationships.

[0075] Still referring to Figure 10 , the pointing device 30 or gesture interacts with the first stereoscopic image in three-dimensional space with six degrees of freedom (e.g., three translational degrees of freedom and three Euler angles). The pointing device 30 or gesture assists in rotating, scaling, translating, editing, highlighting, registering, adding, segmenting, and / or deleting the stereoscopic images (i.e., in 3D space) and / or the two-dimensional images. When changes are made to the stereoscopic images (i.e., in 3D space), the corresponding 2D slices are dynamically updated and reflect the changes. Conversely, when changes are made to one or more 2D slices, the corresponding 3D images are dynamically updated and reflect the changes. For example, the images can be highlighted using the pointing device that interact with a simulated virtual surgical path in 3D space. The user can also select different anatomical components that are independently transparentized or semi-transparentized or independently cut or perforated into shapes for simulating a surgical procedure. Advantageously, a microscopic view of the anatomical structure can be simulated with the cut or perforated path. Typically, the pointing device 30 includes one or more buttons 35 for initiating instructions. In a refinement, the pointing device 30 has fiducials that allow position and orientation tracking of the pointing device.

[0076] In a variation, the stereoscopic images and / or the two-dimensional images are edited or manipulated using the pointing device 30 or gesture. For example, the user can highlight and / or select segmented subcomponents of the stereoscopic images to form stereoscopic subimages that can be independently edited at the voxel level. In a refinement, the user initiates rendering of a three-dimensional container on the first stereoscopic image through the pointing device or gesture so that editing functions including highlighting, segmenting, and deleting can be applied to portions of the stereoscopic image inside the container or portions of the stereoscopic image outside the container at the voxel level. Advantageously, the user selects, manipulates, and aligns the two stereoscopic images together to help them co-register together to form a corrected stereoscopic image.

[0077] In one refinement, the image data is collected by functional magnetic resonance imaging, Tl or T2 weighted magnetic resonance imaging, computed tomography, diffusion tensor imaging, computed tomographic angiography, magnetic resonance angiography, perfusion weighted imaging, susceptibility weighted imaging, digital subtraction angiography, ultrasound, mammography, photoacoustic imaging, positron emission tomography, and combinations thereof. In another refinement, the first stereoscopic image is a composite image rendered from image data collected by two or more imaging techniques including functional magnetic resonance imaging, Tl or T2 weighted magnetic resonance imaging, computed tomography, diffusion tensor imaging, computed tomographic angiography, magnetic resonance angiography, perfusion weighted imaging, susceptibility weighted imaging, ultrasound, mammography, photoacoustic imaging, and positron emission tomography.

[0078] In one variation, the composite image can be constructed as described above. The composite image includes the rendered results of different segments of an organ that are co-registered. Image data from different imaging methods are co-registered and / or segmented to form the composite image such that different anatomical structures come from different imaging methods. The contribution of each imaging method can be adjusted to form the composite image. In one refinement, a user selectively adjusts the opacity or cuts any particular portion or layer of the composite stereoscopic image to form a corrected stereoscopic image. The method of providing adjustable two-dimensional and three-dimensional (3D) renderings of locations in a subject as described above can advantageously be incorporated into a surgical navigation system, such as the system described in detail below. In this regard, a user can selectively adjust the opacity or cut any particular portion or layer of the composite stereoscopic image to form a corrected stereoscopic image. In one refinement, the user applies a slice plane, a simulated surgical drill, a scalpel, or any simulated surgical tool alone or in combination to cut any particular portion of the composite stereoscopic image into any shape, among other things. In particular, a microscopic view of an anatomical structure can be simulated together with a cutting or drilling path.

[0079] In yet another embodiment, a surgical navigation system is provided as well as a method of providing adjustable three-dimensional (3D) renderings for assisting surgery by the surgical navigation system. Figure 11A schematic of a surgical navigation system is provided. The surgical navigation system 100 includes a display 102 for rendering a stereoscopic image 104 (or one or more stereoscopic images) of a portion of a subject 105 (e.g., a patient) viewable through stereoscopic viewing glasses 112. The surgical navigation system 100 also includes a computer processor assembly 108, which includes the CPU, input / output interfaces, memory, and storage devices described above. In one refinement, the surgical navigation system 100 also includes a tracking system 110 optionally mounted to the display 102, which can monitor the position and orientation of the stereoscopic viewing glasses 112, a probe 120, and optional gestures from a user 122 (e.g., a surgeon or other medical professional). The motion and actions of the pointing device 120 are reflected as a pointer cursor on the display 102, as will be described below. A keyboard 124 can be used to input commands and other information into the computer system. The surgical navigation method includes the step of receiving image data for medical images in the subject 105 having a discrete spatial resolution. The image data can include CT, MRI, PET, mammography, ultrasound, or optoacoustic image data. A first stereoscopic image 104 (or one or more stereoscopic images) is rendered on the display 102 from the image data. The first stereoscopic image (or one or more stereoscopic images) includes image parallax. Input signals from the probe 120 allow the probe to be positioned in the first stereoscopic image 104. It should be understood that the probe 120 is a type of pointing device. The probe 120 also allows the first stereoscopic image 104 to be manipulated in three-dimensional space while maintaining the correct three-dimensional spatial relationships, the probe interacting with and manipulating the first stereoscopic image in three-dimensional space with six degrees of freedom to view its spatial relationship with different organ structures. In one refinement, the probe 120 includes fiducial points that allow the position and orientation of the probe to be determined, or the system uses gestures as the pointing device. The probe 120 can also have button(s) and / or other means for direct communication with the computer through a wired or wireless connection. In another refinement, the head motion and / or eye motion of the operator will be tracked and the stereoscopic image will be updated accordingly. In another refinement, the 3D stereoscopic image can be registered to and superimposed on the image of the actual subject on a stereoscopic display or a head-mounted display.

[0080] In one variation, a user simulating a surgical procedure can determine an optimal surgical path. In this regard, the user can also define a potential surgical path or surgical corridor in the first stereoscopic image 104 (or one or more stereoscopic images). Advantageously, white matter fibers that pass through the surgical path or corridor will be highlighted. As the user adjusts the position of the surgical path or surgical corridor, the highlighting of the fibers that pass through the surgical path or corridor is updated. In another refinement, the system alerts the user that the surgical path or surgical corridor intersects with a pre-defined white matter tract, blood vessel or target region through a sound or motion alarm 105 on the monitor 102. In one refinement, a program is initiated that alerts the operator (e.g., user) if the surgical operation deviates from the pre-defined surgical path or surgical corridor. Further, a program is initiated that alerts the operator if the surgical operation interacts with a pre-defined structure overlaid on the surgical view. During the planning of the surgical path, the user can follow the surgical path or surgical corridor to mimic a microscopic view inside the patient's body and / or organs. Advantageously, during the planning of the surgical path, the user can measure the size, distance, volume, diameter and area of any portion of the organ in 3D space.

[0081] In one variation, the method further comprises the step of detecting the position, orientation and body posture of the subject so as to register the first stereoscopic image to the actual physical location in the subject corresponding to the first stereoscopic image. The position of the subject can be registered to the position in the stereoscopic image by any technique known to those skilled in the art. For example, fiducial markers can be attached to the head of the subject and then a CT scan is performed. The tracking system 110 detects the position of these fiducial markers in real time and then registers them to the displayed image. To this end, the tracking system 110 comprises a video camera. The position of the subject's body in the actual space is mapped to the displayed image in the imaging space. In another registration method, a video camera detects the position and orientation of the subject's body. A head-mounted scanner is then used to scan the forehead or other anatomical landmark locations so that the computer can register the subject's image to his / her actual position.

[0082] In another refinement, the probe 120 is positioned in the subject during surgery to access the actual position of the probe in the subject and to further register that position to the stereoscopic images. The stereoscopic images are superimposed on the images of the actual surgical view to form corrected stereoscopic images. These corrected stereoscopic images are displayed on a stereoscopic screen viewed through polarized glasses, a head mounted display, or a surgical microscope. During surgery, the user can interact with and manipulate the 3D stereoscopic subject to view the spatial relationships between different organ structures. In addition, the user can also simulate the surgical procedure on the 3D stereoscopic subject to determine the optimal surgical path. With the aid of the probe, the actual surgical path taken is determined and reconstructed on the stereoscopic images. Thus, adjustments to the pre-planned or suggested surgical path can be provided. To overcome the misalignment between the actual body shape and the 3D stereoscopic images due to body structure deformation during surgery, the probe 120 allows the user to map the actual anatomical locations on the actual subject to the locations on the 3D stereoscopic images to help the computer register the 3D stereoscopic images to the actual subject. In one refinement, the anatomical locations include anatomical landmarks, important bone structures, typical blood vessel structures, typical sulci, and gyri of the brain.

[0083] This registration correction can be performed repeatedly as the surgical procedure progresses. For example, as the patient's body or skull is opened, the brain or body organs will deform, thereby causing a deviation from the original images. Application of the probe 120 at different points on the actual location allows mapping to the corresponding points on the images. The navigation system 100 can then deform the images to register to the actual body shape.

[0084] In one variation, the probe can be a natural hand(s) that allows determination of hand gestures and positions of the hand and fingers.

[0085] In one variation, the probe has wired or wireless buttons and power for communication with the computer for duplex communication.

[0086] In another variation, the user initiates a program in communication with the surgical microscope to retrieve the surgical view in real time. Typically, the surgical view is co-registered with the stereoscopic images or a predetermined surgical path or surgical corridor to form a set of corrected stereoscopic images. In one refinement, the user simulates the surgical procedure in a 3D space with six degrees of freedom to determine the surgical path or surgical corridor or loads a pre-defined surgical path or surgical corridor in the set of corrected stereoscopic images. In another refinement, the surgical path or surgical corridor can be superimposed on the set of corrected stereoscopic images for the first stereoscopic image or any additional stereoscopic images. In another refinement, the user selects important structures of interest including blood vessels, white matter fiber tracts to be superimposed on the corrected stereoscopic images with the surgical path or surgical corridor to form a set of integrated stereoscopic images. In yet another refinement, the highlighting of white matter fiber tracts and / or blood vessels passing through the surgical path or corridor is updated while the user adjusts the position or shape of the surgical path or corridor in the second or third stereoscopic images. The corrected stereoscopic images are displayed on a stereoscopic screen viewed through polarized glasses, a head mounted display or a surgical microscope.

[0087] In another variation, the user initiates a program to extract the surgical view from the microscope in real time and sends the further processed surgical view back to the microscope 126 for display in real time in a 3D stereoscopic space.

[0088] In yet another variation, the user can send a portion of the third or second set of integrated images back to the surgical microscope or endoscope for display in a 3D stereoscope.

[0089] In another variation, the user initiates a program in communication with an existing surgical navigation system to retrieve information of the current position of the surgical procedure and the surgical progress.

[0090] In some variations, the user initiates a program to co-register and integrate the medical images of the subject with the surgical position information of the same person extracted from the surgical navigation system and the surgical view extracted from the microscope to form a new set of images. In one refinement, the user simulates the surgical procedure or adjusts a pre-loaded surgical corridor to determine the optimal surgical path or surgical corridor and superimposes the surgical corridor on the existing stereoscopic images.

[0091] The following examples represent a number of embodiments of the present application. Those skilled in the art will appreciate a variety of modifications that are within the spirit and scope of the present application and the claims.

[0092] 1. Visualization of fibers into 3D stereoscopic images

[0093] Figure 12AA workflow for white matter fiber visualization in 3D scene is shown. The procedure is described below. In this regard, a "line set" is a set of white matter fibers or fiber tracts in 3D space. In step 1, which is represented by block 200, diffusion imaging data is co-registered with structural imaging data, and then white matter fiber tracts of the whole brain or central nervous system are reconstructed based on water molecule diffusion characteristics between each voxel and voxel connectivity. These fiber tracts will be defined as line sets for subsequent 3D visualization. The position or coordinates of the fiber tracts are perfectly aligned with their corresponding structural images (e.g., T1 / T2 / FLAIR / SWI / PWI data).

[0094]

[0095] As shown in block 202, in step 2, line sets are loaded from the reconstructed fiber tract file and data is stored into a dedicated data structure. Each line set is loaded into a defined data structure with all vertices. Table 1 provides a pseudo code for loading reconstructed fibers.

[0096] Figure 12B A schematic diagram of the data structure is provided. In this data structure, a plurality of line sets, orientation matrices, coordinates of each vertex, and the like are stored. As shown in block 204, in step 3, the color of each line set (white matter fiber) is calculated based on the orientation of each line. Here, white matter fibers can be colored based on their spatial orientation. The orientation (direction) of each segment of a line set is calculated as:

[0097]

[0098] The average direction in all segments is calculated as:

[0099]

[0100] The difference between each line set direction and the average direction is calculated as:

[0101]

[0102] Finally, the color of the current line set (fiber) is calculated as:

[0103]

[0104] Table 2 provides an example of a pseudo code for calculating fiber color.

[0105]

[0106] Reference Figure 12C, the fiber (line set) is rendered by providing a pipeline with coordinates loaded at step 2 and colors loaded at step 3, as shown in block 206. The coordinates of the fiber have been co-registered with a structural image such as a Tl volume, MRI data, or any other structural image. The results are rendered on a display screen according to the pseudo code of Table 3, as shown in block 208.

[0107]

[0108]

[0109] 2. Implementing fiber tracking in 3D stereoscope using a stylus

[0110] Reference Figure 13A , a schematic diagram representing fiber tracking using a stylus is shown. The local transformation of the physical stylus is always tracked by two cameras, and data can be extracted by tracking the position of the stylus. The stylus has a marker that is identifiable by the cameras' infrared. A virtual 3D space is defined at the front space of the display. The position of the stylus is converted into this virtual space to determine its interaction with other objects in the same space. In Figure 13B the hierarchical transition of the scene is defined. In this regard, the workflow is as follows:

[0111] 1. Obtain the local transformation matrix of the stylus by extracting the camera tracking information.

[0112]

[0113] 2. Compute the transformation matrix from the camera space to the local space of the fiber.

[0114] M result =M stylus *M window *M scene *M fibers

[0115] 3. Compute the tip and head end of the stylus beam based on its length.

[0116] Coordinate tip =M result *Coordinate initialTip

[0117] 4. Use the computed position of the stylus beam to determine if it interacts with any fiber in the scene, and if so, trigger the corresponding event. The algorithm for the interaction test and triggering events is provided in Table 4.

[0118]

[0119]

[0120] 3. Boolean operations (AND, OR, NOT) of fiber tracking implemented in 3D stereoscopic glasses

[0121] Boolean operations (AND, OR, NOT) of fiber tracking implemented as follows:

[0122] i) When loaded, each line set is assigned a unique ID along with its coordinates

[0123]

[0124] ii) When fibers are extracted through a region of interest (ROI), which can be a spherical or any shaped container. The ROI has in its data structure the IDs of the line sets that pass through it.

[0125]

[0126] iii) Each ROI will have a reference to the whole fiber dataset (all_uIds).

[0127] 3.1 OR operation

[0128] Figure 7B It is shown that fibers that pass through either ROI will be highlighted / displayed together on the scene.

[0129] In the algorithm implementation provided in Table 5, the program combines the IDs from both RoIs into a new object.

[0130]

[0131]

[0132] 3.2 AND operation

[0133] Figure 6A It is shown that fibers that pass through both RoIs will be visualized in the scene. In the algorithm implementation provided in Table 6, the common IDs between the two RoIs can be found and a new RoI is generated through these IDs.

[0134]

[0135] 3.3 NOT operation

[0136] Figure 7ANOT operation of fiber tracking is shown. On the left panel, both dark and light fibers (in real application, these fibers have different colors) pass through the first ROI. On the right panel, the fibers passing through the second ROI are deleted from the scene, meaning that the fibers passing through the first ROI but not the second ROI will be shown in the scene. In the algorithm implementation provided in Table 7, the program will calculate the IDs in one ROI but not the other, and then generate the RoI with these IDs.

[0137]

[0138]

[0139] 3.4 Summary of binary operation

[0140] As shown in Figure 5A , the user can track the white matter fibers in the following steps by using the tracking stylus.

[0141] 1. Move the fiber to the desired position in the 3D space.

[0142] 2. Change the current region of interest (ROI) type to a specific Boolean function, including AND, OR, or NOT, using the function key.

[0143] 3. Calculate the result between the current bundle (which is actually an ROI) and the current label (which is another ROI) by using the algorithm mentioned in the binary operation.

[0144] 4. Generate a set of temporarily extracted fibers for preview.

[0145] 5. Add the temporarily extracted fibers to the result after the user clicks the main button.

[0146] In Figure 5A , the fibers passing through the tracking ball are highlighted in white; right: stylus beam (red) for placing the tracking ball.

[0147] 4. Co-visualization of structural images and white matter fibers

[0148] The structural imaging data (e.g., T1 / T2 / FLAIR / SWI / PWI images) is visualized by using volume rendering and ray casting techniques. The white matter fibers are rendered using the traditional rasterization pipeline. The structural imaging data and fibers can be rendered in the scene of the 3D space using hybrid rendering techniques. Since the fiber data and structural images are pre-co-registered, these two datasets will be perfectly aligned and visualized in the 3D scene. Thus, the fibers are overlaid on the structural images. Figure 14 Hybrid rendering of structural and fiber data in the scene is shown.

[0149] 4.1 Using structural images as reference for fiber tractography

[0150] Since fibers can be overlaid on structural images by means of perfect alignment after co-registration, users can use structural landmarks as reference for performing fiber tractography. Figure 15 A procedure that represents how to co-register structural and fiber data and render them together.

[0151] 4.2 Using 2D image slices as reference planes for fiber tractography in 3D orthogonal planes (orthoslices)

[0152] Figure 3A The workflow for orthoslices is shown in 3B

[0153] 1. Render orthoslices

[0154] a. Position the image acquisition center of the organ, which will also be used as the center of the orthoslices. This will serve as the origin of the x, y, z coordinates.

[0155] b. Render slices to the xz, xy, and yz planes based on the coordinates of the center.

[0156] Algorithm: Render Orthogonal Slice

[0157]

[0158]

[0159] 2. Render fibers as described above.

[0160] 3. Render orthoslices and fibers together (see Figure 3A and 3B ). Since fibers and structural images are co-registered, fibers are overlaid on the 3D orthoslices.

[0161] 4. Move orthoslices. When the user points the stylus to any orthoslice, the user can drag the orthoslice to move it along its axis perpendicular to the plane. The intersection between fibers and orthoslices represents the location of the fibers. Thus, orthoslices can be used to help locate anatomical landmarks for fiber tractography.

[0162] While the foregoing describes exemplary embodiments, these embodiments should not be considered as describing all possible forms of the application. Rather, the words used in the specification are words of description rather than limitation, and it is understood that various changes can be made without departing from the spirit and scope of the application. Additionally, features of the various embodiments can be combined to form further embodiments of the application.​

Claims

1. A computer-implemented method of providing adjustable three-dimensional (3D) rendering of a location, the method comprising: receiving image data for white matter tracts in a subject with discrete spatial resolution, the image data formed by diffusion imaging; rendering a first stereoscopic image on a display from the image data, the first stereoscopic image comprising image parallax; receiving input signals from a pointing device or gesture that allow manipulation of the first stereoscopic image while maintaining correct three-dimensional spatial relationships, the pointing device interacting with the first stereoscopic image in three-dimensional space with six degrees of freedom; displaying a first reference plane and a second reference plane that each independently intersect the first stereoscopic image, rendering a first reference image on the first reference plane and a second reference image on the second reference plane; receiving input signals from the pointing device or gesture that make location selections of the first reference plane or the second reference plane, wherein the first reference image and the second reference image update as their location or orientation changes; and a user initiates rendering of a three-dimensional tracking container on the first stereoscopic image through the pointing device or gesture, wherein the three-dimensional tracking container is fixed to the first reference plane and / or the second reference plane such that the three-dimensional tracking container can move in unison with user-directed movement of the first reference plane and / or the second reference plane.

2. The computer-implemented method of claim 1, wherein, The diffusion imaging is magnetic resonance diffusion imaging.

3. The computer-implemented method of claim 1, further segmenting components of the first stereoscopic image to form stereoscopic sub-images that are independently editable at a voxel level.

4. The computer-implemented method of claim 1, wherein, The pointing device facilitates rotation, scaling, translation, highlighting, cutting, deleting, segmenting, and / or adding to the first stereoscopic image.

5. The computer-implemented method of claim 1, wherein, The pointing device facilitates editing of the first stereoscopic image.

6. The computer-implemented method of claim 1, further comprising: displaying a third reference plane that intersects the first stereoscopic image, rendering a third reference image of anatomy of a subject that intersects the third reference plane on the third reference plane; and receiving input signals from the pointing device or gesture that select a location or orientation of the third reference plane or the second reference plane, wherein the third reference image updates as their location or orientation changes.

7. The computer-implemented method of claim 6, wherein, The first reference image, the second reference image, and the third reference image are each independently two-dimensional images or three-dimensional images of organs, anatomical structures, or pathological features of a subject.

8. The computer-implemented method of claim 7, wherein, The first reference image is a first two-dimensional image of anatomy of a subject that intersects the first reference plane, the second reference image is a second two-dimensional image of anatomy of the subject that intersects the second reference plane, and the third reference image is a third two-dimensional image of anatomy of the subject that intersects the third reference plane.

9. The computer-implemented method of claim 6, wherein, The first reference plane is oriented at a first angle relative to the second reference plane, and the third reference plane is oriented at a second angle relative to the first reference plane and at a third angle relative to the second reference plane.

10. The computer-implemented method of claim 6, wherein, The user can move or rotate the first reference plane along a first axis, move or rotate the second reference plane along a second axis, and move or rotate the third reference plane along a third axis.

11. The computer-implemented method of claim 6, wherein, The user initiates a program to display a reference image based on previously generated image data displayed with the first stereoscopic image.

12. The computer-implemented method of claim 11, wherein, The previously generated image data includes CT data, MRI data, digital subtraction angiography, white matter tract data, ultrasound, mammography, PET, photoacoustic images, or any data further derived from these data sets.

13. The computer-implemented method of claim 1, wherein, The three-dimensional tracking container is a sphere, a disc, a cylinder, an ellipsoid, a tube, a cube, a parallelepiped, an image slice, a block of a 3D dissected organ, a hand-drawn shape, or a surgical corridor, or any shape imported from other sources.

14. The computer-implemented method of claim 1, wherein, White matter tracts and / or fibers and / or blood vessels that pass through the three-dimensional tracking container are highlighted.

15. The computer-implemented method of claim 1, further comprising moving the three-dimensional tracking container in response to a user input signal, wherein the highlighted fibers or blood vessels are dynamically updated as the three-dimensional tracking container moves.

16. The computer-implemented method of claim 1, wherein, White matter tracts and / or fibers that do not pass through the three-dimensional tracking container are removed or hidden from the first stereoscopic image.

17. The computer-implemented method of claim 1, wherein, A plurality of the three-dimensional tracking containers are rendered on the first stereoscopic image.

18. The computer-implemented method of claim 17, wherein, Fibers are displayed or highlighted in the first stereoscopic image according to any combination of Boolean operations associated with each of the plurality of three-dimensional tracking containers.

19. The computer-implemented method of claim 6, wherein, The three-dimensional tracking container is fixed at an intersection of the first reference plane, the second reference plane, and the third reference plane, such that the three-dimensional tracking container can move in unison with user-directed motion of any of the first reference plane, the second reference plane, and the third reference plane.

20. The computer-implemented method of claim 1, further comprising freezing the application results of one or more of the three-dimensional tracking containers.

21. The computer-implemented method of claim 1, wherein, The white matter tracts are from the brain or spinal cord of a subject.

22. A computer-implemented method of providing adjustable rendering of locations, the method comprising: receiving image data for locations in a subject having discrete spatial resolution; rendering at least one stereoscopic image on a display from the image data, at least one of the stereoscopic images including image parallax; rendering one or more two-dimensional images corresponding to predetermined slices in the stereoscopic image; updating the stereoscopic image to reflect changes made to one or more of the two-dimensional images and / or updating the two-dimensional images to reflect changes made to the stereoscopic image, the changes made to the stereoscopic image by means of a pointing device interacting with the stereoscopic image in a three-dimensional space having six degrees of freedom; displaying a first reference plane and a second reference plane that each independently intersects the stereoscopic image, thereby rendering a first reference image on the first reference plane and a second reference image on the second reference plane; receiving input signals from the pointing device or gesture that positionally select the first reference plane or the second reference plane, wherein the first reference image and the second reference image are updated as their position or orientation changes; and the user initiates rendering of a three-dimensional tracking container on the stereoscopic image through the pointing device or gesture, wherein the three-dimensional tracking container is fixed to the first reference plane and / or the second reference plane, such that the three-dimensional tracking container moves in unison with user-directed movement of the first reference plane and / or the second reference plane.

23. The computer-implemented method of claim 22, wherein, The image data is collected by functional magnetic resonance imaging, T1 or T2 weighted magnetic resonance imaging, computed tomography, diffusion tensor imaging, computed tomography angiography, magnetic resonance angiography, perfusion weighted imaging, susceptibility weighted imaging, digital subtraction angiography, ultrasound, mammography, photoacoustic imaging, positron emission tomography, and combinations thereof.

24. The computer-implemented method of claim 22, wherein, The stereoscopic image is a composite image rendered from image data collected from two or more of the image data collected by functional magnetic resonance imaging, T1 or T2 weighted magnetic resonance imaging, computed tomography, diffusion tensor imaging, computed tomography angiography, magnetic resonance angiography, perfusion weighted imaging, susceptibility weighted imaging, digital subtraction angiography, ultrasound, mammography, photoacoustic imaging, and positron emission tomography.

25. The computer-implemented method of claim 22, wherein, The image data is registered and / or segmented to form a composite image such that different anatomical structures are from different imaging methods, the contribution of each image is adjustable to form a composite stereoscopic image.

26. The computer-implemented method of claim 25, wherein, The user selectively adjusts the opacity or cuts any particular portion or layer of the composite stereoscopic image to form a corrected stereoscopic image.

27. The computer-implemented method of claim 26, wherein, The user applies one or more of a slicing plane, a simulated surgical drill, a scalpel, or any simulated surgical tool to cut any particular portion of the composite stereoscopic image into any shape, alone or in combination.

28. The computer-implemented method of claim 22, wherein, The stereoscopic image is viewable through a head-mounted display for virtual or augmented reality or stereoscopic display and / or stereoscopic glasses with fiducial points that allow position and orientation tracking of the stereoscopic glasses.

29. The computer-implemented method of claim 28, wherein, One or more of the head-mounted displays have cameras that track eye movement and head movement or position, thereby causing the 3D stereoscopic image to update accordingly.

30. The computer-implemented method of claim 22, wherein, The pointing device or gesture is used to edit or manipulate at least one of the stereoscopic image and / or the two-dimensional image.

31. The computer-implemented method of claim 30, wherein, The pointing device has fiducial points that allow position and orientation tracking of the pointing device.

32. The computer-implemented method of claim 30, wherein, The user highlights and / or selects segmented subcomponents of the stereoscopic image to form stereoscopic subimages that are independently editable at the voxel level.

33. The computer-implemented method of claim 30, further comprising a user initiating rendering of a three-dimensional container on the stereoscopic image through the pointing device or gesture, such that editing functions including highlighting, segmenting, and deleting can be applied to portions of the stereoscopic image inside the three-dimensional container or portions of the stereoscopic image outside the three-dimensional container on a voxel level.

34. The computer-implemented method of claim 30, wherein, The user selects, manipulates, and aligns the two stereoscopic images together, thereby facilitating their co-registration together to form a corrected stereoscopic image.

35. The computer-implemented method of claim 22, wherein, The stereoscopic images render white matter fiber tracts.

36. The computer-implemented method of claim 22, wherein, The composite image includes a rendering of the co-registration of different segments of an organ.

37. The computer-implemented method of claim 22, further comprising highlighting image segments that interact with a simulated virtual surgical path in 3D space.

38. The computer-implemented method of claim 22, wherein, The user selects different anatomical components that are independently transparentized or translucified or independently cut or perforated into shapes for simulating a surgical procedure.

39. The computer-implemented method of claim 38, wherein, Microscopic perspectives of the subject's anatomy and cutting or drilling paths are simulated together.

40. A computer-implemented method of providing adjustable three-dimensional (3D) rendering for assisting surgery, the method comprising: receiving image data for medical images in a subject having discrete spatial resolution; rendering a first stereoscopic image on a display from the image data, the first stereoscopic image including image parallax; receiving input signals from a probe that allows rendering of the probe in the first stereoscopic image, the probe also allowing manipulation of the first stereoscopic image in three-dimensional space while maintaining correct three-dimensional spatial relationships, the probe interacting with the first stereoscopic image in three-dimensional space with six degrees of freedom; displaying a first reference plane and a second reference plane that each independently intersect the first stereoscopic image, thereby rendering a first reference image on the first reference plane and a second reference image on the second reference plane; receiving input signals from the probe that position-select the first reference plane or the second reference plane, wherein the first reference image and the second reference image update as their position or orientation changes; and a user initiates rendering of a three-dimensional tracking container on the first stereoscopic image through the probe, wherein the three-dimensional tracking container is fixed to the first reference plane and / or the second reference plane, such that the three-dimensional tracking container can move in unison with user-directed motion of the first reference plane and / or the second reference plane.

41. The computer-implemented method of claim 40, wherein, The image data can include CT, MRI, PET, mammography, ultrasound, or optoacoustic image data.

42. The computer-implemented method of claim 40, wherein, The probe includes fiducial points that allow determination of the probe's position and orientation.

43. The computer-implemented method of claim 40, wherein, The probe can also be one or more natural hands that allow determination of hand and finger gestures and positions.

44. The computer-implemented method of claim 40, further comprising a user simulating a surgical procedure to determine a surgical path or surgical corridor in the first stereoscopic image.

45. The computer-implemented method of claim 44, wherein, Highlighting white matter tracts and fibers and / or blood vessels passing through the surgical path or the surgical corridor.

46. The computer- implemented method of claim 44, wherein, Updating the highlighted white matter tracts and fibers and / or blood vessels passing through the surgical path or the surgical corridor as the user adjusts the position of the surgical path or the surgical corridor.

47. The computer-implemented method of claim 44, wherein, Simulating a microscopic view of the subject's body and / or organs while the user follows the surgical path or the surgical corridor.

48. The computer implemented method of claim 40, further comprising detecting the position, orientation and body pose of the subject to register the first stereoscopic image to the actual physical location in the subject corresponding to the first stereoscopic image.

49. The computer-implemented method of claim 40, wherein, Positioning the probe in the subject during surgery, thereby accessing the actual location of the probe in the subject and further registering the location onto the first stereoscopic image or any additional stereoscopic image.

50. The computer-implemented method of claim 40, wherein, The probe allows the user to map actual anatomical locations on the subject to locations on the stereoscopic images to register the stereoscopic images.

51. The computer-implemented method of claim 40, wherein, Anatomical locations include anatomical landmarks, important bony structures, typical vascular structures, typical sulci and gyri of the brain.

52. The computer-implemented method of claim 40, wherein, The first stereoscopic image can be displayed on a polarized stereoscopic screen or a head mounted display.

53. The computer-implemented method of claim 40, wherein, A camera is provided as part of the display or independently to track the position, orientation and motion of the probe and the body pose, orientation and position of the subject undergoing surgery.

54. The computer-implemented method of claim 40, wherein, The probe has wired or wireless buttons and power source for communication with a computer for duplex communication.

55. The computer-implemented method of claim 40, wherein, The user can initiate a program in communication with a surgical microscope to retrieve a surgical view in real time.

56. The computer- implemented method of claim 55, wherein, The surgical view is co-registered with the stereoscopic images or a predetermined surgical path or surgical corridor to form a corrected set of stereoscopic images.

57. The computer implemented method of claim 56, further comprising the user simulating a surgical procedure to determine a surgical path or surgical corridor in the corrected set of stereoscopic images in a 3D space with six degrees of freedom or loading a predetermined surgical path or surgical corridor.

58. The computer- implemented method of claim 57, wherein, The surgical path or the surgical corridor can be superimposed on the corrected set of stereoscopic images.

59. The computer implemented method of claim 57, further comprising the user selecting important structures of interest including blood vessels, white matter fibers to superimpose them on the corrected set of stereoscopic images with the surgical path or the surgical corridor to form an integrated set of stereoscopic images.

60. The computer- implemented method of claim 57, wherein, Updating the highlighted white matter tracts and fibers and / or blood vessels passing through the surgical path or the surgical corridor as the user adjusts the position or shape of the surgical path or the surgical corridor in the corrected set of stereoscopic images or a third integrated set of stereoscopic images.

61. The computer- implemented method of claim 58, wherein, Displaying the corrected set of stereoscopic images on a stereoscopic screen for viewing through polarized glasses, a head mounted display or a surgical microscope.

62. The computer- implemented method of claim 59, wherein, The user can send back to the surgical microscope or endoscope a portion of the corrected stereoscopic image set or the integrated 3D stereoscopic image set for display in the 3D stereoscope.

63. The computer-implemented method of claim 40, wherein, A program to alert the user is initiated if the surgical procedure deviates from the predetermined surgical path or corridor.

64. The computer-implemented method of claim 40, wherein, A program to alert the user is initiated if the surgical procedure interacts with the predetermined structure superimposed on the surgical view.

65. The computer- implemented method of claim 43, wherein, The user initiates a program to communicate with the existing surgical navigation system to retrieve information of the current location of the surgical procedure and the progress of the procedure.

66. The computer- implemented method of claim 43, wherein, The user initiates a program to co-register and integrate the medical images of the subject with the surgical location information extracted from the surgical navigation system and the surgical view extracted from the microscope to form a new image set.

67. The computer implemented method of claim 66, further comprising the user simulating the surgical procedure or adjusting the preloaded surgical corridor to determine the optimal surgical path or corridor and superimpose the surgical corridor on the existing stereoscopic images.

68. The computer- implemented method of claim 40, wherein, The user initiates a program to extract the surgical view from the microscope in real time and sends the further processed surgical view back to the microscope for display in the 3D stereoscopic space in real time. A program to alert the user is initiated if the surgical procedure deviates from the predetermined surgical path or corridor. A program to alert the user is initiated if the surgical procedure interacts with the predetermined structure superimposed on the surgical view. The user initiates a program to communicate with the existing surgical navigation system to retrieve information of the current location of the surgical procedure and the progress of the procedure. The user initiates a program to co-register and integrate the medical images of the subject with the surgical location information extracted from the surgical navigation system and the surgical view extracted from the microscope to form a new image set.

67. The computer implemented method of claim 66, further comprising the user simulating the surgical procedure or adjusting the preloaded surgical corridor to determine the optimal surgical path or corridor and superimpose the surgical corridor on the existing stereoscopic images. The user initiates a program to extract the surgical view from the microscope in real time and sends the further processed surgical view back to the microscope for display in the 3D stereoscopic space in real time.

Images