Systems and methods for viewing a subject

Abstract

A viewing system or imaging system is disclosed that includes optics for viewing a subject. The viewing system can include features that allow for an augmented mixed view through the eyepiece of the viewing system. The mixed view can include graphical representations that are acquired or determined using information separate from the viewing system.

Description

[0001] Cross-references to related applications

[0002] This application includes subject matter relating to U.S. Patent Application No. 16 / 861,328, also filed on April 29, 2020. The entire disclosure of the aforementioned application is incorporated herein by reference. Technical Field

[0003] This disclosure relates to visualization, and more specifically to a hybrid visualization during surgery. Background Technology

[0004] This section provides background information in connection with this disclosure, which is not necessarily prior art.

[0005] During surgery (such as surgery to examine the internal parts of an object), various visualization and viewing systems can be used. For example, microscopes or visualization enhancement systems can be used to examine the internal components of an object. Viewing systems typically allow for direct and magnified examination of internal components through various optical lenses during surgery. Summary of the Invention

[0006] This section provides a general overview of this disclosure and is not a full disclosure of the complete scope or all features of this disclosure.

[0007] Visualization systems may include microscopes or other direct visualization systems. Direct visualization systems may include an optical path that includes various optical lenses, such as magnifying lenses, focusing lenses, etc. Lenses are used to transmit light from an object to an viewing area that may include an eyepiece. The user of a microscope (which may define a focal plane) can view the image through the eyepiece. The user can view or see the focal plane (also called the viewing plane) through the eyepiece.

[0008] In the viewing area, the user can view the object on which the surgery is being performed. A display showing transmitted light from the work area may be provided in the viewing area for the user to view. Various enhancements may be provided in and / or before the viewing area to enhance the view of the object. The view provided to the user and / or the view observed by the user may be a hybrid view, which includes an immediate view or image of the subject and other views or images superimposed on the displayed features (e.g., a graphical representation of a hidden or obstructed object).

[0009] In various implementations, portions can be highlighted in the view, and graphical representations and / or visualizations of various items or portions can be overlaid on the current view of the object. In various implementations, for example, a projection can be placed into the optical path that shows or provides augmented reality (AR), also known as a hybrid view, for the user to view. Thus, the user can view a live display or view of the object (e.g., an instant view) while also viewing enhanced or additional portions (e.g., graphical representations) relative to the live image display.

[0010] In various implementations, the object can be a living object or a non-living object, and may include a patient.

[0011] Further areas of applicability will become apparent from the description provided herein. The descriptions and specific examples in this overview are intended for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0012] The accompanying drawings described herein are for illustrative purposes only, representing selected embodiments and not all possible specific implementations, and are not intended to limit the scope of this disclosure.

[0013] Figure 1 These are environmental views of a microscope system according to various implementation schemes;

[0014] Figure 2 It is a detailed view of the microscope system and the selected user;

[0015] Figure 3 It is the real-time view provided by the microscope system and the selected information provided relative to that real-time view;

[0016] Figure 4A It is an instant view through a microscope system;

[0017] Figure 4B It is a superimposed view through a microscope;

[0018] Figure 4C It is a combination of real-time and overlay views through a microscope system;

[0019] Figure 5A It is achieved through real-time views and mixed overlay contours from a microscope system;

[0020] Figure 5B It is an instant view of fluorescence through a microscope system;

[0021] Figure 5C It is a mixed view through a microscope system;

[0022] Figure 6A and Figure 6BIt is a combination of real-time view and overlay graphics that change;

[0023] Figure 7 This is a schematic diagram of the target object;

[0024] Figure 8A This is the first view at the initial time of the microscope system;

[0025] Figure 8B It is an instant view through a microscope system and an overlay of previously acquired images;

[0026] Figure 9A It is a real-time view and a composite overlay image obtained through a microscope system;

[0027] Figure 9B It is a hybrid real-time view and an overlaid graphical representation view of a microscope system;

[0028] Figure 10 It is a microscope system with a distance measurement component; and

[0029] Figure 11 It is a process used to display previously acquired information.

[0030] In several views of all the accompanying drawings, the corresponding reference numerals indicate the corresponding parts. Detailed Implementation

[0031] Exemplary embodiments will now be described more fully with reference to the accompanying drawings.

[0032] The surgery can be performed in a selected location (such as in operating room 20). However, it should be understood that the operating room can be any suitable operating room of the selected location, and the operating room is merely exemplary. Furthermore, it should be understood that any suitable surgery can be performed relative to a living or non-living subject. For example, surgery can be performed on a complex mechanical system on which it is undesirable to disassemble the entire system. Moreover, the surgery can be performed on a living subject (such as a human subject) for the selected purpose.

[0033] refer to Figure 1 and Figure 2In various embodiments, the surgical operating room may include an observation or optical system, such as a surgical microscope system 30. The observation system 30 (which may also be referred to herein as a microscope) may include various components for performing surgery on the subject 34. In various embodiments, supports (such as patient supports or operating tables 38) may be used to hold or position the subject 34. The surgical microscope system 30 may include microscope components (also referred to as head assembly 32), which may include one or more observation ports or inlets (also referred to as eyepieces or eyepiece units 44). The microscope 30 may be a binocular microscope, such that multiple (such as two) eyepiece units 44a, 44b are available. Alternatively, more than one eyepiece unit may be available, such as a second or auxiliary eyepiece unit 48, which may also be stereoscopic or binocular. Thus, due to the binocular systems 44, 46, more than one individual can simultaneously observe the same parts of the subject 34 and can observe the subject in three dimensions.

[0034] During the selected surgical procedure, the subject 34 can be viewed through the eyepiece 44. One or more objectives 52 may be positioned near or movable near the subject 34, and may also be formed or attached to the head portion 32. Thus, objectives 52 can be positioned near the subject 34. Objectives 52 collect light (e.g., reflected light) from the subject 34 and allow it to pass through the head portion 32. Thus, the various lens systems in the microscope 30 allow for an enhanced or magnified view of the subject 34 between the objectives 52 and the eyepiece 44.

[0035] In various embodiments, the surgical microscope 30 may include: a microscope head assembly 32, which may include optical components such as an eyepiece 44 and a lens 52; and one or more arms or support members 60. The support members may include a first support arm 62 and a second support arm 64. The second support arm 64 may be mounted to a base plate or main support member 70, which may be secured in a selected position and / or movable via one or more wheels 72. It should also be understood that the wheels 72 may be locked in a selected pose to secure the microscope 30 in a selected pose relative to the subject 34.

[0036] In various embodiments, one or more motors may be provided in the one or more arms 60 and / or one or more motors may be provided to move the one or more arms. For example, a first motor 74 may be provided to move a first arm 62 relative to a second arm 64, and a second motor 76 may be provided to move a head 32 relative to the first arm 62. Thus, the head 32, including the lens 52, may move relative to the support member 70 and / or the subject 34 in a selected manner. Motors 74, 76 may be suitable motors, such as electric motors, which may include encoders and / or be operated with encoders to determine precise movement (e.g., determining an error in movement or position by a selected amount, which may be within a selected tolerance range, such as less than about 5 millimeters (mm), less than about 1 mm, less than about 0.1 mm, etc.). Motors 74, 786 may include DC stepper motors or other suitable motors.

[0037] In various embodiments, motors 74, 76 may be controlled by user 80. Additionally, one or more control or processor modules or systems 84 may be provided to control selected motors 74, 76 and / or other components of the surgical microscope 30. For example, processor system 84 may include one or more inputs 86 to control the microscope section 32 to perform selected movements and / or store selected poses of the microscope 32. Accordingly, after storing the selected pose of the microscope 32, the microscope 32 may be moved back to or returned to the selected pose (such as relative to support member 70) for viewing after a selected surgery following a selected elapsed time.

[0038] Therefore, the processor system 84 may include a processor module 88 and one or more memory modules 90. The processor 88 and memory 90 may be any suitable processor or memory, such as those discussed further herein. Additionally, one or more display devices, such as LCD or LED display devices 94, may be provided for viewing various inputs, statuses, and / or images viewed with the microscope 32.

[0039] User 80 can use or operate microscope system 32 to perform surgery on subject 34 using one or more instruments 110. Instrument 110 may include one or more tracking devices 112, which may be tracked using one or more positioning systems 114. Positioning system 114 may include one or more of optical positioning systems, electromagnetic positioning systems, acoustic positioning systems, or other positioning systems, which can be used to track the tracking devices and the parts to which the tracking devices are mounted. Generally, positioning system 114 can be used to track the position of tracking device 112 associated with instrument 110 relative to subject 34. Microscope or observer tracking device 118 may also be associated with surgical microscope system 30, including or directly associated with head component 32. Subject 34 may also have tracking device 120 attached thereto. Thus, using the corresponding tracking devices and tracking locators and system 114, instrument 110, subject 34, and microscope 32 can all be tracked relative to each other, including simultaneously. Other suitable pose systems may also be used to assist and / or perform tracking, such as tracking devices associated with motors used to move microscope 32 and fix subject 34 in a known pose. As discussed above, encoders may be included in the joints of motors 74, 76 and / or components 62, 64 to determine the movement and / or pose of microscope 32, such as in robotic systems or microscopes. In various embodiments, the pose of lens 52 relative to subject 34 is likely to be most relevant for viewing various features relative to subject 34. As discussed herein, the pose of parts (e.g., microscope 32, subject 34, and / or instrument 110) may include all position and orientation information, such as six degrees of freedom, such as translational (x, y, z) coordinates and orientation (yaw, pitch, roll) coordinates.

[0040] Therefore, in various embodiments, the pose of the device 110 can be shown as a graphical representation superimposed on a view through the eyepiece 44 of the microscope 32. Various tracking systems capable of registering previously acquired images or other data to the subject include those sold by Medtronic Navigation, which has a presence in Colorado. The S8 surgical navigation system, and / or the system disclosed in U.S. Patent Application Publication No. 2019 / 0328461, published on October 31, 2019, which is incorporated herein by reference.

[0041] Registration may include registering previously acquired information (such as image data) to subject 34 to determine the pose relationship between the previously acquired image data and subject 34 in real time. Similarly, a tracking system may be used to show the pose of the tracked system to previously acquired information.

[0042] In various embodiments, the system including locator 114 may comprise various components or systems, such as those disclosed in U.S. Patent Nos. RE44,305, 7,697,972, 8,644,907, and 8,842,893, U.S. Patent Application Publication No. 2004 / 0199072, and U.S. Patent Application Publication No. 2019 / 0328460, all of which are incorporated herein by reference. The locator 114 of the navigation system can be used to track the pose of an object, as discussed herein. This pose can then be displayed for viewing by user 80, as also discussed herein. Additionally, tracking information (including information about magnetic fields and / or light signals sensed by the tracking device) can be delivered via a communication system to a selected portion of the microscope system 20, such as processor 88. Using a suitable system, the communication can be wired or wireless.

[0043] In addition to the live view as discussed herein, image data may be acquired before and / or during surgery, such as using one or more imaging systems, to display the image or portions thereof for viewing by user 80 through eyepiece 44. Additional information or data may be viewed by user 80 along with the live view, as discussed herein.

[0044] A navigation or tracking domain or volume generally defines a navigation space or patient space, in which objects can be moved and tracked, for example, using locator 114. Due to tracking devices 112, 118, 120, the various portions can be tracked relative to each other. The navigation volume or patient space can be registered to an image space defined by additional information of subject 34 (e.g., previously acquired or selected images). This registration allows the determined positions of various objects relative to each other and / or the image space to be shown and displayed. In various embodiments, registration can be performed by identifying identical points or portions (e.g., registration points or landmarks) in the image space and navigation space. Registration points can be natural (e.g., anatomical) or artificial (e.g., fixed to the subject). However, once identified in both spaces, registration can be performed by determining a transformation between the two spaces using the registration points. Transformation and registration can be performed by executing instructions using a suitable processor (such as processor 88). Registration from patient space to image space can be performed, as well as determination of the position of a tracking device (such as tracking device 118) relative to a DRF (such as DRF 120), as is known in the art, including as disclosed in U.S. Patent Nos. RE44,305, 7,697,972, 8,644,907, and 8,842,893, and U.S. Patent Application Publication No. 2004 / 0199072, all of which are incorporated herein by reference.

[0045] Continue to refer to Figure 1 and Figure 2 And additional references Figure 3 The view through eyepiece 44 may include a display or image 130. Image 130 may include or display various information, such as a direct or immediate view (also called a live view) 132 of a selected part of the subject (such as the brain 134). The live view 132 may be viewed directly through microscope 32 (such as through the overall direct light transmission through eyepiece 44 for viewing by user 80). The live view may represent the area of ​​view or surgical area at the visual plane. The visual plane is generally understood as the area or plane that the microscope is focused on and that user 80 can view through eyepiece 44. The live view may be a current or real-time view.

[0046] In addition to the live view 132 in view 130, additional information may be overlaid on and / or displayed adjacent to the live view 132 due to, for example, tracking of the subject 34 using the subject tracking device 120 and / or tracking of the microscope 32 using the microscope tracking device 118. In various embodiments, one or more targets may be identified. These one or more targets may be referred to herein as a single target or a target, but it should be understood that more than one target may be identified. Furthermore, a target may be any suitable portion (such as anatomical features, abnormalities, etc.) and / or a selected location (e.g., including selected specific features, such as the volume of a tumor). For example, acquired image data (such as MRI image data) may be used to identify a tumor as a target, and to identify the outline of the tumor that can be displayed as a target graphic representation 140. Furthermore, due to the tracking of the subject 34 and the instrument 110, the instrument 110 may be shown as an instrument illustration or graphic representation (also referred to as an icon) 144. Thus, the pose of the instrument 110 can be directly viewed through the eyepiece 44 of the microscope 32, and / or the representation of the instrument 110 can be shown as a graphic representation 144.

[0047] like Figure 3 As shown, in addition to the real-time image 130, an additional overlay or graphic view 150 may be displayed through eyepiece 44. Overlay 150 or any suitable graphic representation may be opaque, transparent, or at least partially transparent. In a partially transparent overlay, the real-time image can be seen through overlay 150. The mixed or enhanced view 150 may include additional or selected information, such as a graphic representation 144 of an instrument and / or a representation 140 of a tumor or selected target. The enhanced view 150 may additionally include information such as depth lines or indicators 154 from surface 158. The plane or surface representation 158 may be a surface of the brain 132 in the real-time view 130 or another selected portion of the subject 34. This surface may also be a view plane seen through eyepiece 44.

[0048] As discussed above, the pose of the subject 34 can be determined using the subject tracking device 120, which may include the pose of selected portions of the subject (such as the surface of the brain 132). Additionally, the subject 34 can be secured in the selected position using a holding or fixing device (such as a head clamp 39). Therefore, the external or surface line 158 can be determined and shown in the enhanced view 150. The instrument 110 can be tracked using the tracking device 112 and is shown in the enhanced view 150 by the graphical representation 144.

[0049] Additionally, information about the subject 34, such as fiber tracer maps or tracer map representations 162, can also be displayed in an enhanced view 150 relative to the live view 130 of the subject 34. Additional (including previously acquired) information can be used to identify fiber bundles. As discussed above, previously or additionally acquired information can be registered to the subject due to tracking of the subject 34. Furthermore, the pose of the microscope 32 may be known relative to the subject 34. Therefore, additional registered information (e.g., tracer maps) can be displayed in the mixed view. Thus, the user 80 can view the subject 34 through the microscope 32 and can view additional information besides the live view 130 of the subject 34, including the enhanced view 150. The outline 140 of the target, the graphical representation 144 of the instrument, and selected depth or target information can also be displayed. This allows the user 80 to simultaneously view the depth or three-dimensional representation of the selected target within the subject's body through the microscope view port 44 and the live view. The enhanced image 150 allows the user 80 to better understand the view or representation of the image of the subject 34, rather than just or merely viewing an instant view through the microscope 32.

[0050] As discussed above, when viewed through eyepiece 44, the live view 130 allows user 80 to directly view a live or current view of one or more portions of subject 34. Additional data, such as a graphical representation 154 of a tumor or target, a graphical representation 144 of an instrument, a representation 158 of a surface or external extent, and / or tracer information 162, may be based on various previously acquired image data and / or analyses of subject 34. The positioning of the graphical representation in the live view and / or in the additional or enhanced view 150 may be based on tracking subject 34 using subject tracking device 120 and tracking microscope 32 using microscope tracking device 118.

[0051] As is commonly understood by those skilled in the art, additional information can be registered to subject 34 due to a selected registration, such as image registration. For example, tracking locator 114 can be used to track multiple parts, such as tracking subject 34 using subject tracking device 120 and / or tracking microscope 32 using microscope tracking device 118. One or more selected processors, such as processor 88, can be used to perform registration of the subject space defined by subject 34 relative to microscope 32 and / or additional image data space. The tracking system can be compatible with Stealth, sold by Medtronic Navigation, Inc., which has a business location in Colorado. A navigation system similar to a tracking system. Various navigation or tracking systems can be used to determine the pose of the subject 34 relative to the microscope 32 using a corresponding tracking device, and thus determine the pose of the graphic representation, such as the graphic representation 144 of the instrument, placed in the enhanced view 150. The enhanced view can be displayed using a translucent screen or plate (such as those included in the Kinevo 900 microscope sold by Carl ZeissMeditec AG, which has a business location in Germany, and / or microscopes with screens of selected resolution). Additionally, the user 80 can select, for example, to determine or identify the graphic representation displayed in the enhanced view 150 using the microscope system's input 86. Accordingly, the user 80 can choose to display or not display various items, such as surface lines 158, selected contours 140, or other selected graphic representations.

[0052] Furthermore, various graphical representations can be displayed in appropriate ways, such as in two-dimensional representations, three-dimensional representations, or in changing representations, such as time variations based on various information, such as gating relative to the heartbeat of subject 34. At least in part due to the binocular eyepiece 44, a three-dimensional rendering can be displayed to user 80 as a three-dimensional image. Thus, user 80 can view an image of subject 34 enhanced with a selected graphical representation, as discussed above. In particular, the enhanced view 150 can display or show the depth of a selected surface (such as at surface line 15). The enhanced display can also show the position or depth of device 110, such as using graphical representation 144, including viewing the distal end 144d of device 110.

[0053] Continue to refer to Figure 1 and Figure 2 And additional references Figure 4A , Figure 4B and Figure 4CThe image shows an instantaneous view 170 through microscope 32. The instantaneous view 170 allows viewing of selected portions of subject 34, such as the surface of brain 174. In addition to the surface of brain 174, various other items, such as skull 176 and instruments 34, can also be viewed. Eyepiece 44 allows direct visualization of portions of subject 34 (such as instruments 110 and / or portions of subject 34, such as brain 174) in the instantaneous or real-time view 170. The direct visualization for user 80 to view can represent any item within the visual range of microscope 32 relative to subject 34.

[0054] As discussed above, subject tracking device 120 can be used to track subject 34, and microscope tracking device 118 can be used to track microscope 32. Instrument tracking device 112 can be used to track instrument 110. According to various embodiments, the display (such as a mixed or overlaid view through eyepiece 44) includes a three-dimensional graphical representation or graphical overlay, such as... Figure 4B As shown. The graphical representation may include a three-dimensional graphical representation 180 of the instrument 110 relative to parts of the subject (such as a graphical representation 184 of the brain surface and a representation 188 of selected parts or targets (such as an identified tumor). Additionally, various anatomical features, such as vascular structures 192 and / or other anatomical structures such as ventricles, fiber bundles, optic nerves, eyeballs, etc., may also be shown in the graphical representation.

[0055] Therefore, the display through eyepiece 44 can represent a three-dimensional representation based on various information, such as previously acquired image data of subject 34 and / or a graphical representation of instrument 110, such as graphical representation 180. The positioning of various representations (including representations 188 and / or instrument representation 180 belonging to the selected target or tumor) can be based on tracking subject 34 using subject tracking device 120 and tracking instrument 110 using tracking device 112. Accordingly, a three-dimensional representation can be displayed for user 80 to view using microscope 32, including previously acquired information or information that cannot be directly viewed using microscope 32. While microscope 32 can provide or allow a magnified view as an immediate view of subject 34, such as... Figure 4A As shown, however, the representation of additional information for display relative to subject 34 (such as using the overlay and / or display of instant image 170) allows user 80 to form or understand additional concepts about subject 34 and / or parts of that subject.

[0056] During surgery or any appropriate selected procedure, the user 80 may choose to display or not display various information, including such as Figure 4B The three-dimensional representation is shown. Users can use input 86 or indicator input 86 (e.g., instruction assistant, voice control, etc.) to display or not display various three-dimensional features.

[0057] Additionally and / or alternatively, two-dimensional representations may be displayed, overlaid, and / or layered on the instantaneous image 170, such as Figure 4C As shown. Real-time image 170 can display various parts of subject 34, as directly shown or directly observed by user 80 through microscope 32. Thus, the real-time view can display various parts, such as bone structures 176 and / or other features. However, representations of various structures and / or features, such as previously identified tumors or targets 200, can be displayed or overlaid on the real-time image. Additionally, various anatomical structures, such as the vascular system or bones, can be displayed via overlays (such as anatomical overlay 204). Similarly, overlays can be displayed relative to real-time image 170 due to registration and / or tracking of subject 34 and microscope 32, and registration to previously acquired images. In various embodiments, the user or other suitable user can identify selected features, such as anatomical features and / or targets (e.g., tumors), in previously acquired image data (e.g., magnetic resonance imaging (MRI)). These previously acquired and / or identified features can then be registered to subject 34 and overlaid on real-time image 170 during the selected portion of the surgery, as shown. Figure 4C As shown. This allows user 80 to understand or recognize the pose of the identified features as well as alternative or supplementary images (e.g., MRI) used for display on the live image 170 of subject 34. Thus, user 80 can view both the live image and / or the enhanced image to understand the additional features identified in the alternative image data.

[0058] Similarly, the user can use the input 86 of the microscope system 30 to determine whether to display various features. The user can use the microscope 32 to directly view the subject 34 and / or enhance the view of the subject 34 during a selected procedure. The user 80 can use additional information to identify and / or understand the representation or pose of previously identified or determined features (such as tumors or selected anatomical targets or features).

[0059] Continue to refer to Figure 1 and Figure 2 And additional references Figure 5A , Figure 5B and Figure 5C The display or image view for user 80 may include a general, real-time view 220 of subject 34 (including the surface 224 of a selected portion of the subject, such as the brain surface). As discussed above, various additional information may be overlaid and / or displayed on the real-time image 220, such as the outline 228 of the selected target or portion (e.g., a tumor). The target image display 228 may be based on previously acquired or viewed information, such as MRI data.

[0060] During the selected procedure, additional surgical or information may be obtained or utilized. Various digital and / or procedural techniques may include fluorescence of selected anatomical features or portions of the subject 34. For example, such as... Figure 5B As shown, fluorescent materials and / or multiple fluorescent materials can be provided to subject 34. The material can then fluoresce and / or can be induced to fluoresce in a selected manner. Figure 5B As shown, for example, the selected region may have a first fluorescence 230, and the second region may have a second fluorescence 234. The image or view may be an instantaneous view 220 of the subject 34 through a microscope, and the user 80 may directly observe the selected fluorescence. Alternatively, the image may be viewed or analyzed via the microscope system 30. The system can identify or distinguish between high or bright fluorescence 230 and low fluorescence 234. Based on the differentiation of fluorescence, the processor system 88 can identify or segment the bright region 230 from the darker region 234. In various embodiments, the high or bright fluorescence 230 region may be referred to as the bright region and may have a brightness or luminance that is at least about 10% brighter than the low fluorescence 234 region. In various embodiments, the bright region may have a brightness or luminance that is at least about 50%, about 200%, or about 400% brighter. The bright region may be a two-dimensional region and / or a three-dimensional volume.

[0061] The microscope system 30, including the processor, can then segment a view or image of the subject 34 based on fluorescence. The areas of bright fluorescence 230 and low fluorescence 234, or the lines between them, can be determined in any suitable manner (such as by segmentation). Thus, the segmented region can be used to determine the current or updated volumetric region of the target. The updated region can then be displayed, as discussed herein.

[0062] Therefore, as Figure 5C As shown, the live view 220 may have a mixed or enhanced contour graphic representation 240 overlaid thereon, which may be based on the segmentation of bright fluorescent regions 230 and darker fluorescent regions 234. In various embodiments, for example, a second or enhanced contour 240 may be displayed along with fluorescence and / or after fluorescence has ceased. Therefore, it should be understood that the secondary or enhanced representation 240 may also be displayed without fluorescence. For example, during a selected surgical procedure, fluorescence may be emitted during illumination of the subject 34 using light or energy of a selected wavelength. At this time, fluorescence may be present within the subject 34, and the image may be analyzed by the microscopy system 30. After image analysis, the enhanced contour 240 may be identified. The contour may then be displayed for viewing by the user 80, such as by overlaying or superimposing the contour 240 on the live view 220.

[0063] Enhanced or updated contours can be used by user 80 to determine the progress of the selected procedure. For example, during the selected procedure, ablation or removal of selected portions of the subject may be performed, such as removal of a tumor or removal or termination of an aneurysm. Therefore, user 80 can view the updated contour 240 to help determine the progress of the procedure. Thus, the contour, or any appropriate representation, may allow geometry (e.g., shape and size) to be shown and changed relative to the initial or first contour or representation 228. This can be in addition to its pose (i.e., x, y, z position and orientation).

[0064] It should be understood that fluorescence in subject 34 may occur more than once and at various times during the procedure. Fluorescence can be used to identify or determine updates or changes in the subject's anatomical structures or selected portions thereof. Therefore, users can understand or better visualize changes in the subject or portions of subject 34.

[0065] In various embodiments, the wavelength of fluorescence illumination of selected portions of the anatomical structure may be within the invisible wavelength range. Accordingly, fluorescence that can be bright fluorescence 230 relative to darker fluorescence 234 cannot be visually recognized by user 80. However, a microscope system or suitable system can analyze images of fluorescence in invisible wavelengths to identify differentiation between anatomical structures or portions of the subject. Thus, the contour 240 that can be superimposed on the instantaneous image 220 can be the only visual representation of fluorescence differentiation for user 80. Accordingly, microscope system 30 can be used to display differences between initial contour 228 and updated or second or later contour 240. As discussed above, initial contour 228 may be based on previously acquired images or information of subject 34, such as having image data (including MRI image data). Changes in the subject or portions of the subject during surgery can be identified and determined and displayed (e.g., using enhanced contours) for use by user 80.

[0066] Continue to refer to Figure 1 and Figure 2 And additional references Figure 6A and Figure 6BThe live view image 260 of subject 34 may include a direct view of a portion of the subject, such as the brain surface 264. Additionally, an outline or graphical representation 268 of a selected portion or target (such as a tumor) may be displayed. Targets can be identified or selected in the image or live view in various ways, such as by manual identification (e.g., a user drawing the outline of the target), image segmentation (e.g., automatic segmentation based on a selected algorithm (such as color or edge detection)), and / or retrieval of previously identified or drawn targets (e.g., access and retrieval from memory). As discussed above, the identification of a tumor, as shown as graphical representation 268, may be based on previously acquired information, such as previously acquired MRI data of subject 34. Partly due to tracking subject 34 using tracking device 120, graphical representation 268 may be based on registration of previously acquired images of subject 34 and / or identification of a tumor or target. Also as discussed above, additional graphical representations may be displayed relative to the live image 260, but these are not shown here for clarity of the present discussion.

[0067] Graphical representation 268 can show selected or defined boundaries of a tumor relative to real-time image 260. Real-time image 260 can be viewed by user 80 through eyepiece 44. Accordingly, even a binocular three-dimensional view of the brain surface 264 can be enhanced using graphical representation 268 to show the depth, shape, etc., of the selected target. However, in addition to static boundaries, pulsating or moving boundary representations 272 can also be shown. (Continue to the next section) Figure 6A And additional references Figure 6B The real-time view 260 can show generally unchanging or immobile parts of the subject, such as viewing the brain surface 264 in a fixed position. Additionally, the static boundary 268 of the selected target or tumor can also be displayed relative to the brain surface 264. However, a pulsating or altering representation 272 can be movable or appear to be moving, such as changing the highlighted portion or color of the tumor boundary over time. For example, Figure 6A The pulsating shell or grid 272 can indicate the uppermost or closest part of the tumor's surface boundary. At a second time, such as a period of time lasting a fraction of a second or more (e.g., about 0.1 seconds to about 0.5 seconds), the pulsating grid 272 can change shape and position to the second grid position 274, as... Figure 6B As shown.

[0068] At the selected time point, user 80 can choose to display the geometry of the tumor or target on subject 34 in a changing or pulsating manner. Accordingly, in Figure 6A In the image, the first portion of the pulsating mesh 272 can be highlighted or shown. Changes in the highlighted portion of the tumor's mesh or outline within the same view or plane, such as... Figure 6BThe difference is shown in the second part 274. This allows the user 80 to observe dynamic changes in a selected portion of the view without moving the microscope 32.

[0069] It should be understood that any appropriate number of highlighted portions can be shown (e.g., sequentially) over a selected time period. For example, a tumor or target with a static graphical representation of 268 may have a selected depth or external geometry, such as... Figure 7 As exemplarily shown in the diagram. During time zero or at time zero (T0), the mesh shell 272 may show the boundary from the surface or at a depth of zero millimeters into the tumor or target 268i, which is formed by... Figure 6A and Figure 6B The outline 268 is graphically represented in the image. At selected time intervals, such as approximately 0.1 seconds to approximately 0.5 seconds, the outline at a selected depth (such as 1 mm depth) can be shown by highlighting the grid or changing its color, for example, as shown by different grid portions 274. Figure 6B As shown. It should be understood that any suitable number of sizes or times can be used, and as shown... Figure 7 The 1 mm dimension shown at the selected times T0 to T5 is merely exemplary. However, at the selected times, for example, different portions at T0 to T5 can be highlighted and shown sequentially in the display, such as overlaid on the live view 260 for the user 80 to view.

[0070] As selected portions or different portions of a target (such as tumor 268i) are highlighted, user 80 can understand the contour of tumor 268i at different depths relative to the surface or zero millimeters. Therefore, when viewing the live image 260 using microscope 32, user 80 can understand the contour or shape of tumor 268i. Pulsating geometric contours 272, 274 can be initiated or stopped by user 80 at selected times to understand the predetermined or identified shape of tumor 268i. In other words, user 80 can update the portion or location of the pulsation (e.g., pulsating plane) to be highlighted or specified. Thus, user 80 can select the portion or plane to be highlighted compared to spontaneous pulsation. The contour or pulsating shape can be displayed to aid user 80's understanding of the target within subject 34.

[0071] Continue to refer to Figure 1 and Figure 2 And additional references Figure 8A and Figure 8BMicroscope 32 can be used to view instantaneous images 300 of subject 24, as discussed above. Instantaneous images 300 may include various parts, such as the surface 304 of the subject's brain. In addition to the surface of organs (such as brain 304), other features can also be observed or displayed through the eyepiece 44 of microscope 32. For example, bone structures 308 and one or more other soft structures or soft tissues, such as a vascular system including one or more arteries 312, can be viewed. Although references are made herein, for example... Figure 8A and Figure 8B The discussion may refer to artery 312, but it should be understood that various features may be shown and analyzed in addition to artery 312 as further discussed in this article.

[0072] As discussed above, microscope 32 may be fixed and / or tracked relative to subject 34. In various embodiments, for example, base support 70 may be fixed relative to subject 34, and subject 34 may be fixed relative to microscope 32 (such as head clamp or holding frame 39). Accordingly, microscope 32 may be moved relative to subject 34, such as to gain access to portions of subject 34, to examine more than one area of ​​subject 34, or for other suitable reasons. When microscope 32 moves, eyepiece 44 also moves. Generally, the user 80's examination is performed using eyepiece 44, and movement of eyepiece 44 may be determined or most relevant to movement relative to microscope 32. Therefore, movement of microscope 32 can be understood as referring to or involving movement of eyepiece 44. In any case, microscope 32 (including microscope system 30) can be used to acquire images of subject 34 at various time points (e.g., at a first time point and a second time point).

[0073] During surgical procedures, the microscope can move from a first or original pose at a first time point to a second pose at a second time point, and then return to the original or first pose at a third or selected time point. Various poses of the microscope 32 can be determined by tracking the microscope 32 using selected tracking devices (including microscope tracking device 118 and patient tracking device 120). Furthermore, as discussed above, selected arms or robotic components of the microscope system 30 (including arms 62, 64) can move with considerable precision due to selected motors 74, 76. In any case, the microscope 32 can be moved from the first or original pose for the purpose of examining the subject 32 after a selected time period.

[0074] Accordingly, for example, such as Figure 8AAs shown, a first view of the subject 34 in the first pose of the microscope 32 can be directly viewed through the microscope 32. At this time, an image or snapshot of the subject 34 can be acquired through the microscope 32. In various embodiments, a camera or image acquisition system may be associated with the microscope 32. The images may be stored in a selected memory system (such as memory 90) for retrieval at a later time. The images may include various parts, such as the surface 304 of the brain, bones 308, and arteries 312. Snapshots can be displayed relative to a line view or an instantaneous view 300 at selected time periods (including a first time T1 and a second time T2).

[0075] refer to Figure 8B Therefore, after a selected time period, such as at time T3, the microscope can return to or move to the pose at time T1. Due to tracking of the microscope 32, the pose at time T1 can be determined or saved. In various embodiments, a robotic arm (including motors 74, 76) can be used to automatically or based on input commands move the microscope to a previous pose (i.e., the pose at T1 when the snapshot image was acquired). Accordingly, the instantaneous view 320 can represent the subject 34, including the brain surface 304a. Images acquired at time T1 (e.g., ...) Figure 8A (As shown) can be overlaid on an instant view 320 including the brain surface 304a. Due to various analyses and / or recognitions of the image performed by the user 80, various vascular systems or parts can be identified in the image overlaid on the instant view 304a. For example, as Figure 8B As shown, the original pose of the vascular system 312 can be displayed relative to a new or altered pose 324. Additionally, the microscope system 30 may include indicators or arrows 328 to indicate the determined orientation or change of the subject 34 (such as brain surface 304a) at the current time relative to the original time when the overlaid view is formed. Arrows 328 may include: the original portion at the original or T1 pose of the selected structure (e.g., the vascular system 312), and the head or end at the current or instantaneous view pose of the selected structure (e.g., the vascular system 324).

[0076] Any appropriate number of previous time snapshots can be displayed or retrieved. For example, multiple snapshots 340 can be generated at selected times (such as time T1, time T2, and time T3). Each snapshot in the snapshots 340 can be displayed as an overlay on an instant view of the current surface of the brain 304a for the user 80 to view. Accordingly, the user 80 can select one or more snapshots in the snapshots 340 to display to enhance the current instant view of the surface of the brain 304a.

[0077] Since the microscope 32 is repositioned to its previous position at a previous time (e.g., time T1), previous snapshots can be overlaid on the current live view. Accordingly, selected portions of anatomical structures (e.g., the vascular system) can be viewed and displayed as changes or possible changes relative to the initial or first view acquired during the movement of the microscope 32 at a previous or first time point in the surgery. In any case, the user 80 can view the overlay or superimposed live view with previously acquired snapshots to see the results of changes relative to the subject 34 at time and / or within a portion of the surgery.

[0078] Continue to refer to Figure 1 and Figure 2 And additional references Figure 9A and Figure 9B The user can use microscope 32 to examine subject 34 to view an immediate or real-time view 420 of subject 34. As discussed above, microscope 32 can be moved by user 80 in an appropriate manner, such as from a first pose to a second pose. Alternatively, microscope 32 can be tracked using a tracking system including locator 114 and microscope tracking device 118, relative to tracking subject 34 using subject tracking device 120, or due to various motion-determining components (such as those including motors 74, 76 and / or robotic features). Thus, in various embodiments, user 80 can determine or track the pose or know the pose of microscope 32 relative to subject 34. Furthermore, user 80 can return microscope 32 to a previous pose by tracking or saving the pose of microscope 32 relative to subject 34. Therefore, microscope 32 can be used to examine subject 34 at a first time and a second time, and to compare between the two times, for example, as discussed above. However, in addition to this, the user 80 can also determine or know the orientation of the microscope in order to perform the following operations: observe the subject 32, and analyze or evaluate the orientation of selected parts of the subject 32 at different time points or over a period of time.

[0079] refer to Figure 9A For example, user 80 can identify various checkpoints or landmarks in subject 34. For example, reference Figure 9A User 80 can identify one or more physical or real landmarks in the instantaneous view or view 420 using a microscope. Physical landmarks may include markings (e.g., optical markings), such as X or ink markings, including a first physical landmark 424, a second physical landmark 428, a third physical landmark 432, and a fourth physical landmark 436. Physical landmarks 424 to 434 can be identified by user 80, such as identifying physical landmarks within a subject or placing physical landmarks on the subject, such as ink markings. Landmarks can be identified by user 80, such as identifying portions shown or visible in an image.

[0080] Microscope system 30 can identify physical landmarks, such as through analysis of images viewed using microscope 32. The analysis may include machine learning algorithms or other appropriate identification or determination measures for selecting features to identify the physical landmarks. For example, such as... Figure 9A As shown, the intersection of the two ink marks (i.e., “X”) can be used as an indication of specific finite or fixed inspection points 424i to 434i. Inspection points can be identified in or for the microscope system 30, for example by automatic identification, manual identification (e.g., identification by user 80 for the microscope system 30 using various inputs (including input 86)) or other suitable determination methods.

[0081] Additionally, as discussed above, the microscope system 30 can acquire or determine an image of the subject 34 at a first time point. A second image or determination of checkpoints 420 to 434 can be formed after a selected time period, such as after one minute, two minutes, or after a time selected based on a decision by the user 80. As discussed above, the pose of the microscope 32 relative to the subject 120 and / or due to the robotic or motor mechanisms 74, 76 in the microscope system 30 can be known or determined using a tracking system including a microscope tracking device 118. Therefore, the microscope 32 can be moved to view the subject 34 in a pose substantially the same as during the initial determination of checkpoints 424 to 434. This allows analysis to be performed between the initial checkpoint and the current checkpoint.

[0082] like Figure 9A As shown, the current checkpoint can be considered a physical marker, and previous checkpoints can be indicated graphically, such as by "CP" followed by a number. Additionally, various indications regarding the initial checkpoint pose can be provided, such as color, indicators, or key information, such as... Figure 9A As shown. Similarly, key information can be an enhanced view through eyepiece 44 relative to image 420 displayed for user 80 to view.

[0083] In any case, for example, CP1 can be shown in green, and key information can indicate a distance change of less than one millimeter that has occurred or been determined between the previous pose determination and the current pose determination. Alternatively or otherwise, CP3 can be indicated as “unplanned” or greater than a predetermined or set limit. Additional checkpoints CP3 and CP4 can also be shown in selected colors (such as yellow and red) respectively, which are related to the distance of the change in pose between the initial illustration or determination and the current pose. Thus, the microscope system 30 can be used to show the amount of change in the pose of the subject 34 from a first time to the current time. In various embodiments, this can be used to determine whether reregistration is required and / or whether reregistration or realignment of the microscope is possible.

[0084] In various implementations, for example, the checkpoint may be placed on a generally rigid structure (such as the bone structure 450 of subject 34). Thus, it can be determined that the rigid bone structure has moved between a first time and the current time, and this can indicate a re-registration or re-determination of the pose of subject 34.

[0085] Additionally, the pose of various soft tissues or organs can be determined relative to the selected checkpoints. For example, referencing... Figure 9B The microscope 32 can display checkpoints 424 to 434 relative to the soft tissue 460 of the subject 34. Additionally, various soft tissue portions, such as functional indicators, including a first functional pose 464 and a second functional pose 468, can be displayed. Functional poses 464 and 468 can be displayed or marked on the brain 460 using appropriate markers (such as biocompatible markers or physical parts). Furthermore, due to the determined or trackable pose of the microscope 32 relative to the subject 34, functional poses 464 and 468 can be displayed as overlaid portions or features on the brain 460 for the user 34 to view in view 470. Therefore, the instantaneous view 470 can have indicators of functional poses 464 and 468 overlaid thereon.

[0086] Functional poses 464 and 468 can be determined by the user 80 probing the brain 460. Accordingly, functional poses 464 and 468 may not include physical or identifiable landmarks on the brain 460, but may be based on functional indications determined by the user 80. However, the microscope system 30 can view the functional poses 464 and 468 relative to checkpoints 424 and 434. However, the functional poses 464 and 468 can be displayed as a graphical representation for the user 80.

[0087] The determination of movement or absence can be used to determine whether there is brain displacement of brain 460 relative to a fixed or rigid part of subject 34 (such as bone 450). For example, in a manner similar to that discussed above, microscope system 30 can identify or determine examination point poses 424 to 434. Functional poses 464, 468 can then be determined relative to examination points 424 to 434 at a first time and a second time, or at a time later than the first time. Thus, an indication of movement or absence can then be given to user 80 in the view.

[0088] For example, relative to the live view 470, the user 80 can view an enhanced portion including key information through eyepiece 44. For instance, the numbers 1 and 2 can be used to identify different functional poses 464 and 468 respectively, and indications can be made regarding whether they are in or far from the initial pose. Key information may indicate that functional point 1 is in the same or similar pose, and functional point 2 is at a distance greater than two millimeters relative to the initial pose. Therefore, the user can identify or understand the current pose, including the amount of change (if any) in the current pose of the brain 460 relative to the checkpoint between two time periods. Similarly, the user 80 can then determine whether the surgery can continue or be enhanced due to possible shifts in the brain 460. Furthermore, the determination of the pose of functional points can be reassessed based on indications of possible changes.

[0089] Continue to refer to Figure 1 and Figure 2 And additional references Figure 10 In addition to the tracking device 118 associated with the microscope 32 (such as... Figure 2 As shown, the microscope may also include additional or alternative tracking or positioning systems or other components. For example, a relative distance measuring system may include one or more rangefinders, such as a first distance measuring system or part 500 and a second distance measuring system or part 510. Each of the respective distance measuring parts 500, 510 may transmit or include distance measuring beams or features 514, 520. Beams 514, 520 may be signals (e.g., optical or acoustic signals) that are transmitted and received for distance measuring. The distance measuring system may be any suitable system, such as one for determining linear distances or for determining three-dimensional (3D) scans and correlated distances.

[0090] Distance measuring features 514, 520 can be emitted toward subject 34, who is supported on a selected support structure (such as support 38). Distance measuring features 500, 510 can be incorporated into and / or mounted to microscope 32. Accordingly, distance measuring features 500, 510 can be removably mounted to microscope 32 for use in selected surgical procedures and / or surgical procedures.

[0091] The distance measuring features 514, 520, which can be emitted and / or received by the distance measuring units 500, 510, can be selected distance measuring features, such as optical distance measuring features. In various embodiments, for example, the distance measuring units 500, 510 can form or be incorporated into a binocular or three-dimensional imaging distance measuring system that can triangulate the pose (including distance) relative to the respective distance measuring units 500, 510 and the microscope 32 to which these distance measuring units are fixed. In various embodiments, for example, the distance measuring units 500, 510 can generate or visualize images of the subject 34 (such as its brain portion) to determine the pose of the microscope 32 relative to the subject 34 (and / or a portion thereof), or to determine the movement of the subject 34 (such as the brain therein) relative to the microscope 32 due to the known or fixed pose of the microscope 32 relative to a portion of the subject 34 and / or the support 38.

[0092] The ranging or distance measuring components 500, 510 may also be other suitable distance measuring features, such as optical (visible and infrared) ranging systems, acoustic ranging systems, ultrasonic ranging systems, radar ranging systems, or other suitable ranging systems. In any case, the ranging system may be incorporated into and / or attached to the microscope 32 for use in selected surgical or surgical procedures. Therefore, the ranging systems 500, 510 can be used to identify the pose of the microscope 32 relative to the subject 34 and / or the pose of a portion of the subject 34 relative to other portions (e.g., brain offset).

[0093] In various embodiments, ranging system portions 500, 510 may emit corresponding beams 514, 520. Beams 514, 520 may be reflected from subject 34 or a portion thereof. The reflected portion may be received by ranging system portions 500, 510 or other suitable portions. This emission and reception can be used to determine the pose of subject 34 relative to microscope 32, which includes ranging system portions 500, 510. As described above, the pose of instrument 110 may also be determined relative to microscope 32 and subject 43.

[0094] Additionally, the ranging system components 500 and 510 may include various features such as voice activation or control. For example, user 80 may provide audible input or commands to measure or determine distances such as brain offset, microscope pose, or the pose of the subject 34 relative to the view from the microscope. Thus, user 80 may interact with the distance measuring systems 510 and 520 and provide selected inputs to the system.

[0095] Similarly, microscope 32 can also examine instrument 110, which is movable relative to subject 34. Therefore, ranging systems 500, 510 can determine the pose of at least a portion of instrument 110 relative to subject 34 and can view or display information in the view via the imager 44 of microscope 32. Accordingly, rangefinders 500, 510 can be used to measure distances relative to selected portions of subject 34 and / or instrument 110 (relative to microscope 32). Information can be displayed for user 80 to view in the viewport or eyepiece 44, such as determining the pose or movement and / or pose of subject 34 or determining the pose of instrument 110.

[0096] Accordingly, as discussed above, the microscope system 30 can be used to provide information to the user 80 to assist in the selected procedure. The pose and / or overlay of instruments (such as instrument 110) can be displayed in the eyepiece 44 for the user 80 to view. When the user 80 views the subject 34 through the eyepiece 44, the user 80 can view the overlaid information (e.g., target location or size, tracked instruments). Thus, the user 80 can include a view of the subject 34 along with additional information, which can be obtained separately from the microscope 32 and / or obtained before using the microscope 32. The microscope system 30 can allow the user 80 to provide input for selecting and / or determining various features, such as instrument pose, selected target, or identified features (e.g., tumor, anatomical structure, etc.). Thus, the user 80 can directly view the subject 34 and / or additional information relative to the subject 34, which can be obtained or determined separately from the microscope or the microscope system 30.

[0097] As discussed above, according to various embodiments, images can be viewed through eyepiece 44 using microscope 32. Images viewed through the microscope may include an immediate or actual view of the subject, also referred to as a real-time view. An immediate view can be generated by direct visualization of the subject 34. As discussed above, direct visualization can occur due to reflected or direct light passing through the optics of microscope 32 (such as through objective lens 52, through internal optics, and ultimately through eyepiece 44). Therefore, the immediate view for user 80 to view can represent the subject 34 in real time. Additionally, as discussed above, various additional, enhanced, or mixed view information can be displayed as overlays on and / or adjacent to the immediate view. Accordingly, the view of user 80 through eyepiece 44 can be a mixed view including both the immediate view of the subject 34 and / or additional information displayed relative to it. Furthermore, as discussed above, various views can be displayed as overlays on the immediate view, substantially obscuring the immediate view, such as a three-dimensional representation of the subject 34.

[0098] Additionally, as discussed above, the microscope 32 included in the microscope system 30 can be a robotic microscope, which can be moved or controlled using various motors (such as motors 74, 76) and associated with (e.g., housed in) a corresponding arm or component 62, 64. Thus, the microscope system 30 can be positioned near the subject 34 and moved relative to the subject 34 in a generally precise and known manner. Alternatively and / or in addition, the microscope 32 can be tracked relative to the subject 34 using a selected tracking system (including locator 114). Furthermore, as discussed above, in addition to or alternatively including distance measuring systems comprising distance measuring sections 510, 520, the tracking system can track the subject and / or instrument 110.

[0099] Accordingly, the position of the subject 34 may be known relative to the microscope 32 (including the viewing plane of the microscope 32). Therefore, due to the known position of the microscope 32 relative to the subject 34 and / or the instrument 110, various information can be displayed for the user 80 to view through the eyepiece 44 to indicate the pose of the instrument and / or part relative to the subject 34 (e.g., including the target of the tumor), such as depth, orientation, etc.

[0100] Please refer to the above attached figures and additional references. Figure 11 The method or process 600 can be used to identify or determine the position of supplementary information relative to the subject 34 for display by the user 80 using the eyepiece 44. The supplementary information may be previously acquired image data or other data, such as tracer maps. Generally, the supplementary data will be information that is not visible in an instant view using an optical microscope.

[0101] Process 600 may begin in start box 610. After the process begins in box 610, various steps or procedures may be performed to determine the position of subject 34 relative to other acquired information. For example, the determination of the subject's pose relative to the microscope eyepiece and / or view plane in determination box 614. As discussed above, the pose of microscope 32 and / or eyepiece 44 and / or view plane of microscope 32 may be determined. The pose of microscope 32 relative to subject 34 may be determined by determining the initial position of the microscope relative to subject 34 and / or by tracking the microscope relative to subject 34. As discussed above, various tracking systems may be used to track or determine the pose of microscope 32 at appropriate times. For example, at a first time or time T1, the pose of microscope 32 relative to subject 34 may be determined, such as using a robotic system including motors 74, 76 and associated sensors or encoders and / or tracking devices 118. Subsequent movements of microscope 32 may also be tracked using similar systems to determine a second or later pose of microscope 32 relative to subject 34.

[0102] At an appropriate time, and without needing to determine the subject's pose relative to the microscope in box 614, additional information can be acquired in box 618, and registration of previously acquired information relative to the subject can be performed in box 620. The additional information can be any suitable information, such as previously acquired image data or other appropriate data. The additional information can be stored for retrieval from a selected memory system (such as memory 90). However, the additional information can also be generated substantially in real time, such as thereby showing or identifying functional locations on the brain.

[0103] Additional information may include: information acquired using the imaging system prior to the subject's movement relative to the microscope 32; images acquired using the microscope 32 (e.g., optically acquired images); tracer data, etc. As discussed above, the pose of the subject 34 can be determined, for example, by tracking the subject 34 using a distance measuring system and / or tracking device 120. Additionally, registration can be performed between the acquired images and the immediate view or view through the microscope 32, based on generally known techniques, such as identifying landmarks in the subject 34 and recognizing similar or identical landmarks in previously acquired images.

[0104] As discussed above, images acquired using microscope 32 may include markers that can be identified and later compared with the same location on subject 34 to determine possible movement of subject 34 relative to its initial pose. Similarly and / or alternatively, previously acquired images may be registered to subject 34 to show the appropriate portions identified in the previously acquired image data (and / or appropriate other data) relative to subject 34 for viewing using microscope 32. Thus, registration of acquired information to subject 34 can be performed at any appropriate time, such as after tracking or registering subject 34 relative to microscope 32 in a navigation space and / or after registering subject 34 to previously acquired image data, the navigation space may include movement and / or volume for moving instrument 110 and / or microscope 32. In any case, registration of previously acquired information in box 620 allows for determination of the appropriate pose of subject 34 for information in the previously acquired information during surgery.

[0105] Then, in box 624, it can be determined whether the registered additional information should be displayed relative to the subject's immediate view (such as through eyepiece 44). As discussed above, the microscope system 20 may include various inputs that allow selection by the user 80. Accordingly, the user 80 can use input 86 to determine whether the acquired information should be displayed. If it is determined that the information should not be displayed, the “No” box or path 628 can be followed to the start box 610. Therefore, the determination of pose in box 614 and the registration of information in box 620 may not need to be displayed for the user 80 to view.

[0106] However, if it is determined that additional information should be displayed, then the "Yes" path 632 can be followed. After following the "Yes" path 632, the additional information can be displayed or shown in an appropriate pose relative to the view plane in box 640. As discussed above, the view plane is available for viewing by user 80 of subject 34. Accordingly, the additional information can be displayed relative to the view plane in an appropriate representation to user 80.

[0107] For example, as discussed above, a tumor or target may be identified in previously acquired information, which may be substantially below the surface of the external portion of the subject 34, such as the outer surface of the brain. Accordingly, a representation (such as a graphical representation) of the tumor may be displayed in an appropriate pose (including depth) relative to the viewing plane to indicate the appropriate pose of the tumor for the user 80 to view through eyepiece 44 relative to the viewing plane. The determination of the appropriate pose may be based on the registration of previously acquired information to the subject 34 and / or the determined pose of the subject relative to the microscope eyepiece or viewing plane in frame 614. Determining the pose of the plane relative to the subject 34 allows for the appropriate display of the previously acquired information relative to the subject 34 in an appropriate pose.

[0108] After display 640, process 600 may end in box 650. Ending the process in box 650 may include: termination of the surgery, moving microscope 32 (including surgical microscope system 30 if selected), determining additional information to be displayed, or restarting process 600. Accordingly, end box 650 can be understood as the end of a specific part of the surgery, such as displaying previously acquired information as selected according to process 600.

[0109] The microscope system 30 can be used to display appropriate information for viewing by the user 80. The information can be displayed based on determinations of the pose of the microscope 32 relative to the subject 34, and previously acquired information regarding the registered or known pose of the subject 34. This allows the previously acquired information to provide appropriate perspective information to the user 80 by displaying the appropriate pose of the subject 34 relative to the previously acquired information.

[0110] Information can be displayed as graphical representations of various parts, such as instruments, targets (e.g., tumors or anatomical parts), identified features or portions, etc. Graphical representations can be overlaid on live images, as discussed above. Depending on the embodiments, the graphical representation can be completely opaque, partially opaque, or transparent. A completely opaque graphical representation can completely obscure the live image or any portion overlaid thereon. A partially opaque graphical representation allows at least partial viewing of the portion of the live image overlaid thereon. A transparent graphical representation may include, for example, only an outline and allows viewing of the live image within the outline.

[0111] It should be understood that the various aspects disclosed herein can be combined in combinations different from those specifically given in the specification and drawings. It should also be understood that, depending on the example, certain actions or events of any process or method described herein may be performed in a different order, or may be completely added, combined, or omitted (e.g., performing the described technique may not require all the described actions or events). Furthermore, although for clarity some aspects of this disclosure are described as being performed by a single module or unit, it should be understood that the techniques of this disclosure can be performed by combinations of units or modules associated with, for example, a medical device.

[0112] In one or more examples, the described techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium (e.g., a memory module) and executed by a hardware-based processing unit. Computer-readable media may include non-transitory computer-readable media, which correspond to tangible media such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer).

[0113] Instructions may be executed by one or more processors (e.g., processor modules), such as one or more digital signal processors (DSPs), general-purpose microprocessors, graphics processing units (GPUs), application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein may refer to any of the foregoing structures or any other physical structures suitable for implementing the described techniques. Furthermore, this technique may be fully implemented in one or more circuit or logic elements.

[0114] Exemplary embodiments are provided to make this disclosure thorough and to fully communicate the scope of this disclosure to those skilled in the art. Numerous specific details, such as examples of particular components, apparatus, and methods, are set forth to provide a thorough understanding of embodiments of this disclosure. It will be apparent to those skilled in the art that specific details are not required, that exemplary embodiments may be embodied in many different forms, and should not be construed as limiting the scope of this disclosure. In some exemplary embodiments, well-known processes, well-known apparatus structures, and well-known techniques are not described in detail.

[0115] The foregoing description of embodiments has been provided for illustrative and descriptive purposes. The foregoing description is not intended to be exhaustive or limiting of this disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but are interchangeable and may also be used in selected embodiments where applicable, even if not specifically shown or described. The same element or feature may be varied in many ways. Such variations should not be considered as departing from this disclosure, and all such modifications are intended to be included within the scope of this disclosure.

Claims

1. A system for viewing a subject in a view through the eyepiece of a microscope, comprising: Processor system, the processor system being configured to access memory system and execute instructions to: Determine the first pose of the visual plane viewed through the eyepiece relative to the subject; At the first time point, acquire a first image of the subject at the visual plane in the first pose; Store the first image; The first image is displayed in the view plane, superimposed on the subject's live image, and The display includes a graphic representation of an arrow having an initial end at a point in the first image and a head at a point in the instantaneous image, the head indicating the direction and extent of the subject's movement relative to the view plane.

2. The system of claim 1, wherein the processor system is further configured to execute instructions to: Analyze the first image; Analyze the real-time images; and Determine whether the position of the instantaneous image relative to the first image has changed.

3. The system of claim 2, wherein the processor system is further configured to execute instructions to: The output determines whether a positional change has occurred in the instantaneous image relative to the first image, wherein the determined pose is output as follows: Generate a graphical representation showing the determined changes; as well as The graphic representation is displayed and superimposed on the instantaneous image.

4. The system of claim 3, wherein the graphic representation is partially opaque to allow the real-time image to be viewed through the graphic representation.

5. The system according to any one of claims 1 to 4, wherein the processor system is further configured to execute instructions to: The microscope head is tracked relative to the subject using the visual plane at the first pose.

6. The system of claim 5, wherein the head includes an objective lens configured to receive light from the subject.

7. A system for viewing a subject in a view through the eyepiece of a microscope, comprising: The processor system is configured to locate the memory system and execute instructions to: Determine the first pose of the visual plane viewed through the eyepiece relative to the subject; At the first time point, acquire a first image of the subject at the visual plane in the first pose; Store the first image; The first image is displayed in the view plane and superimposed on the subject's real-time image. The processor system is further configured to execute instructions to: After moving the microscope head from the viewing plane at the first pose and returning the head to the viewing plane, the instantaneous image at the second time point is compared with the first image; Determine whether a change in the position of the instantaneous image relative to the first image has occurred between the first time and the third time; as well as Output a determination result of whether the position of the instantaneous image relative to the first image has changed between the first time and the third time.

Images