Systems and methods for predictive fusion

The position and direction of the imaging device are tracked through the spatial tracker, and the image data is updated using the feature information in the previously acquired image data, which realizes predictive fusion between MRI and ultrasound images, solving the calculation difficulties and coordination caused by modal differences. The problem of incomplete accuracy improves the effect of the image guidance process.

CN110892452BActive Publication Date: 2024-11-22MIM SOFTWARE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201880043083.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2017-05-04
Filing Date
2018-05-04
Publication Date
2024-11-22
Estimated Expiration
2038-05-04

AI Technical Summary

Technical Problem

In medical imaging, modal differences between MRI and ultrasound images lead to computational difficulties and imperfect registration, especially when the direction of the prostate changes significantly, making it difficult to achieve effective image fusion.

Method used

The position and direction of the imaging device are detected by a spatial tracker, a desired attitude is determined using feature information in the previously acquired image data, and an image data and feature information are updated based on the tracking information, maintaining alignment between the real-time image and the previously acquired image, Achieve predictive fusion of images of different modalities.

Benefits of technology

Accurate alignment and fusion between different modal images is achieved, the execution effect of the image guidance process is improved, the recognition ability of anatomical features is enhanced, and the calculation difficulties and incomplete registration are solved due to modal differences.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110892452B_ABST
    Figure CN110892452B_ABST
Patent Text Reader

Abstract

The image fusion system provides predictive alignment between images of different modalities and synchronization of the alignment once acquired. The spatial tracker detects and tracks the position and orientation of the imaging device in the environment. The predicted pose of the anatomical feature can be determined relative to the expected position and orientation of the imaging device based on previously acquired image data. When the imaging device is moved to the expected position and orientation, a relationship is established between the pose of the anatomical feature in the image data and the pose of the anatomical feature imaged by the imaging device. This relationship is maintained even if the imaging device is moved around during the imaging process based on the tracking information provided by the spatial tracker.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to U.S. Provisional Application No. 62 / 501,329, filed May 4, 2017, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to image registration and fusion, and more particularly, to systems and methods for facilitating image registration and fusion via spatial tracking of imaging devices. Background Art

[0004] Image fusion generally involves combining information from different images into a single composite image. For example, in medical imaging, fusion can involve registering and combining different images in some way to produce a composite image. The composite image can provide improved image quality or enhance the usability of the image for diagnosis, treatment planning and evaluation, tracking disease progression, etc. In medical imaging, the two or more images that are fused can be of the same imaging modality or different imaging modalities. Multiple images of the same modality can be fused to determine disease progression or treatment effects. Images of different modalities can be combined to take advantage of the strengths of the different modalities or for convenience.

[0005] For example, magnetic resonance imaging (MRI) provides good soft tissue contrast. Therefore, MRI can distinguish healthy tissue from lesions or other abnormalities relatively easily. Therefore, MRI performs well in detection and planning. In image-guided procedures, MRI may be inconvenient due to the cost and non-portability of the imaging machine. For example, biopsies of the prostate are usually guided by ultrasound, which is portable and provides high spatial resolution. However, ultrasound provides less tissue differentiation than MRI. Therefore, MRI-ultrasound fusion can combine information from each modality to improve the execution of image-guided procedures. Summary of the invention

[0006] A simplified overview is provided here to help achieve a basic or general understanding of various aspects of the exemplary non-limiting embodiments in the following detailed description and drawings. However, this overview is not intended to be a broad or exhaustive overview. Instead, the sole purpose of this overview is to present some concepts related to some exemplary non-limiting embodiments in a simplified form as a prelude to a more detailed description of the various embodiments that follow.

[0007] In various non-limiting embodiments, once obtained, the image fusion system provides predicted alignment between images of different modalities and synchronization of the alignment. The spatial tracker detects and tracks the position and orientation of the imaging device in the environment. The imaging device is, for example, a suitable device for intra-procedural imaging. Based on image data of a modality different from the imaging device, a predicted pose of the anatomical feature can be determined relative to the expected position and orientation of the imaging device. When the imaging device is moved to the desired position and orientation, a relationship is established between the pose of the anatomical feature in the image data and the pose of the anatomical feature imaged by the imaging device. Based on the tracking information provided by the spatial tracker, this relationship is maintained even if the imaging device is moved to various positions during the imaging process.

[0008] These and other embodiments are described in more detail below. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Various non-limiting embodiments are further described with reference to the accompanying drawings, in which:

[0010] Figure 1 is a block diagram of an exemplary non-limiting embodiment of an image fusion system according to one or more aspects;

[0011] Figure 2 is with Figure 1 A schematic block diagram of an exemplary non-limiting embodiment of a computing device associated with an image fusion system;

[0012] Figure 3 is a schematic block diagram of an exemplary non-limiting embodiment of medical image fusion according to one or more aspects;

[0013] Figure 4 is a flow chart of an exemplary non-limiting method for predicting fusion between at least two image modalities;

[0014] Figure 5 is a flow chart of an exemplary non-limiting method for generating a fusion between a real-time image and a previously acquired image;

[0015] Figure 6 is an exemplary image having a contour overlay to indicate a predicted pose of an anatomical feature at an expected position and orientation relative to an imaging device;

[0016] Figure 7 is an exemplary image captured by an imaging device after an operator positions the imaging device to align an imaged anatomical feature with an overlaid contour;

[0017] Figure 8is an exemplary image captured after the imaging device is moved and after the coverage contour is updated in response to the movement of the imaging device; and

[0018] Fig. 9 is a block diagram that represents an exemplary non-limiting computing system or operating environment in which one or more aspects of the various embodiments described herein may be implemented. DETAILED DESCRIPTION

[0019] General Overview

[0020] As discussed in the background, medical image fusion can take advantage of the strengths of different imaging modalities and produce combined information with a wide range of applications. For example, the fusion of ultrasound images and MRI of the prostate can provide effective intra-procedural images and enable accurate identification of anatomical features. However, due to many factors, the various orientations of the prostate in the MRI image and the intra-procedural ultrasound image can vary significantly. These significant differences lead to computational difficulties and imperfect registration between the two modalities.

[0021] In various non-limiting embodiments, a system and related methods for image fusion are provided. Based on a real-time image provided by a spatial tracking imaging device, it can be determined whether the imaging device is aligned with a desired position and orientation relative to an anatomical feature. Specifically, the desired position and orientation can be specified based on feature information associated with previously acquired image data, which can be a different modality than the real-time image. The feature information specifies the position and orientation of the anatomical feature in the previously acquired image data. The image data and feature information can be transformed to indicate the pose of the feature observed at the desired position and orientation from the perspective of the imaging device.

[0022] Once aligned with the desired position and orientation, a relationship between the real-time image and the previously acquired image data can be established. After locking, tracking information from the spatial tracker can inform the processing of the previously acquired image data. Specifically, the image data and / or feature information are transformed in response to the movement of the imaging device to maintain the alignment. For example, a spatial transformation between the current spatial position and a reference spatial position (i.e., the desired position and orientation) can be determined based at least in part on the tracking information. A corresponding transformation based on the spatial transformation can then be applied to the previously acquired image and / or feature information.

[0023] In one embodiment, a system is provided that includes a processor coupled to a memory storing computer executable instructions. When executed by the processor, the instructions configure the processor to: obtain feature information indicating a pose of a feature in a first image of a first modality; obtain tracking information from a spatial tracker with at least three degrees of freedom, the spatial tracker configured to identify a position and orientation of a handheld imaging device that produces an image of a second modality; and determine when a pose of the imaging device matches a predetermined pose based on at least one of the feature information and the tracking information, the predetermined pose being suitable for producing a second image of the feature having a pose that matches a reference pose of the feature. According to one example, the feature information specifies an orientation of the feature in the first image relative to a given orientation of the imaging device. For example, the feature information may include an outline of the feature in the first image.

[0024] The processor may be further configured to update the indication of the posture of the feature in the second image generated by the imaging device based on at least one of the tracking information and the feature information in response to the movement of the imaging device. The processor may be further configured to obtain the relationship between the posture of the imaging device in space and the posture of the imaging plane of the imaging device, and update the indication of the posture of the feature in the second image based on the relationship. In addition, for example, the spatial relationship between the current posture of the imaging device and the reference posture of the imaging device is also determined based on the tracking information. The reference posture of the imaging device may correspond to the above-mentioned predetermined posture, for example. In order to update the indication of the posture of the feature, the processor may be configured to re-slice the first image along the virtual imaging plane. The processor may be configured to determine the virtual imaging plane based on the tracking information. In addition, the processor may be configured to interpolate the image data of the first image corresponding to the virtual imaging plane. In another example, the processor may be configured to identify the transformation between the first image and the second image, and apply the transformation to the feature information.

[0025] Furthermore, the processor may be configured to display a real-time image acquired from the imaging device. The processor may also be configured to display an overlay on the real-time image based on the feature information. Additionally, the processor may be configured to update the overlay based on an indication of a pose of the feature in a second image acquired by the imaging device.

[0026] According to another aspect, a method is described. The method includes acquiring a real-time image of a first modality from an imaging device. Additionally, the method may include acquiring tracking information indicating a position and orientation of the imaging device from a spatial tracker. Furthermore, the method may also include determining when the imaging device is in a desired pose relative to a feature in the real-time image, thereby establishing a relationship between the pose of the imaging device and the pose of the feature in a second image of a second modality, wherein the imaging device is determined to be in the desired pose based on at least one of the tracking information indicating the pose of the feature in the second image and the feature information.

[0027] According to one example, the method may include updating an indication of a feature in the real-time image in response to movement of the imaging device based on at least one of the tracking information and the established relationship. The indication of the feature may be initially generated based on feature information indicating a posture of the feature in the second image relative to an expected posture of the imaging device relative to the feature. For example, the feature information may be a contour. Updating the indication of the feature may include re-slicing the second image along a virtual imaging plane, wherein the virtual imaging plane is determined based on the tracking information. Subsequently, the re-slicing may include interpolating image data of the second image corresponding to the virtual imaging plane. In another example, updating the indication of the feature may include determining a transformation between an expected posture of the imaging device and a position and orientation of the imaging device provided by the tracking information, and applying the transformation to the indication of the feature.

[0028] In yet another embodiment, a computer-readable storage medium is described. The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, configure the processor to: obtain feature information indicating a feature in a first image of a first modality and a pose of the feature in the first image relative to a desired pose of an imaging device, wherein the imaging device is configured to produce an image of a second modality; obtain tracking information from a spatial tracker configured to track a position and orientation of the imaging device in space with at least three degrees of freedom; and determine when the imaging device is aligned with the desired pose based on at least one of the feature information, the tracking information, and a real-time image produced by the imaging device.

[0029] In one example, the computer executable instructions stored on the computer readable storage medium further configure the processor to update the indication of the feature in the real-time image based on at least one of the tracking information and the feature information. For example, the medium may include instructions that configure the processor to re-slice the first image along a virtual imaging plane determined based on the tracking information, and insert the feature information along the virtual imaging plane to generate an updated indication of the pose of the feature in the real-time image. In another example, the computer executable instructions stored on the computer readable storage medium further configure the processor to determine a transformation between the desired pose and the position and orientation of the imaging device provided by the tracking information; and apply the transformation to the feature information to update the indication of the pose of the feature in the real-time image. In addition, the medium also stores instructions that configure the processor to display an initial indication of the pose of the feature in the real-time image, wherein the initial indication is generated based on the feature information and provides guidance for aligning the imaging device with the desired pose.

[0030] An overview of some embodiments of predictive fusion has been given above. As a next development roadmap, predictive image fusion is described in more detail. Afterwards, an exemplary computing device and computing environment in which such embodiments and / or features described above can be implemented are described. The above features and embodiments will be described with reference to the accompanying drawings, in which like reference numerals are used throughout to refer to like elements.

[0031] Predictive Image Registration and Fusion

[0032] As described above, in various embodiments, even in cases where the imaging devices provide real-time images and are able to move freely in space, the fusion of images of different modalities can be determined and the alignment between the images can be locked.

[0033] Figure 1 1 is a block diagram illustrating an exemplary, non-limiting embodiment of an image fusion system 100. As shown, the image fusion system 100 may include a computing device 110, an imaging device 120, and a spatial tracker 130. The computing device may include a processor and various computer-readable storage media (e.g., volatile and non-volatile). The computer-readable storage media may store computer-executable instructions that implement at least a portion of the functional modules described herein, such as the fusion engine 112. When the computer-executable instructions are executed by the processor, the image fusion system 100 is thus configured to perform the operations described herein.

[0034] The computing device 110 may further include various hardware devices (not shown) for implementing various parts of the fusion engine 112. For example, the computing device 110 may include a graphics device having a graphics processing unit (GPU), dedicated memory, and / or a hardware interface for coupling the graphics device to a display. In addition, the computing device 110 may include physical hardware ports and / or wireless interfaces (e.g., Bluetooth, wireless USB, etc.) to couple the computing device 110 to various devices of the image fusion system 100, such as, but not limited to, the imaging device 120 and the spatial tracker 130.

[0035] As shown, the imaging device 120 may include an imaging probe 122 and an image processor 124. In one aspect, the imaging device 120 may be a relatively inexpensive and portable device suitable for intra-procedural imaging, such as an ultrasound imaging device. Nevertheless, it should be understood that the features and aspects described and claimed herein are not limited to ultrasound applications and may be readily adapted for use with other imaging modalities.

[0036] In the ultrasound example, the imaging probe 122 may include one or more transducer arrays configured to emit ultrasound pulses and receive echoes. The echoes may be converted into electrical signals and provided to the image processor 124 to generate an ultrasound image. Although the image processor 124 is shown as being separate from the computing device 110, it should be understood that the processing of the echo signals may be performed by the computer device 110. For example, a separate software application or module of the fusion engine 112 may be configured to process signals from the imaging probe 122. In addition, although the imaging device 120 is shown as including the imaging probe 122 and the image processor 124, the term "imaging device" used herein may refer to all components that interact together to generate an image, or, depending on the context, may refer to a portion (i.e., a probe) that houses the transducer array. For example, it should be understood that when described in conjunction with spatial tracking, the term "imaging device" refers to the portion of the entire imaging device that can be manipulated to guide or aim at the final imaging object.

[0037] In one aspect, the sensor 132 may be coupled to or integrated with the imaging probe 122. The sensor 132 cooperates with the spatial tracker 130 to generate tracking information indicating the position and orientation of the imaging probe 122 in space. According to an example, the spatial tracker 130 may be an electromagnetic (EM) tracking system that includes an EM source that generates an EM field, which establishes a three-dimensional reference system. According to this example, the sensor 132 may include orthogonally aligned inductive coils or other devices, and generates a signal indicating the strength of the received EM field. The signal enables the position and orientation of the sensor 132 in the three-dimensional reference system established by the EM source to be determined. The signal may be transmitted to the computing device 110 via a wired or wireless connection for processing and position determination. For example, in an EM tracking system, the signal is typically transmitted by the sensor 132. However, alternatively, the signal may be received by the spatial tracker 130 and then forwarded to the computing device 110 by pre-processing or in a raw format. In addition, although the above example contemplates an EM-based positioning system, it should be understood that other trackers may be utilized. For example, the spatial tracker 130 can be an accelerometer / gyroscope based tracker, an infrared tracker, an optical tracker, an acoustic tracker, a laser tracker, an RF based tracker, or substantially any other type of tracking and positioning system capable of identifying the position and orientation of a sensor within a reference frame, such as but not limited to mechanical spatial tracking (e.g., a probe stepper or gantry).

[0038] On the other hand, the sensor 132 is coupled to or integrated with the imaging probe 122 so as to establish a known relationship of the position of the sensor 132 relative to the imaging plane of the probe 122. Therefore, based on this relationship, the position and orientation of the imaging plane (and the generated image) within the spatial reference system provided by the spatial tracker 130 are also known. The relationship between the position of the sensor 132 and the imaging plane can be determined by measuring the relationship (i.e., explicitly designed by the probe manufacturer) or by calibration. By way of illustration, a technique for calibrating or establishing a known relationship between the position of the sensor 132 in the spatial reference system and the imaging plane involves imaging a control object. For example, an object (e.g., the intersection of multiple threads) can be imaged in a bucket from various positions and / or orientations. A spatial transformation is then determined that solves the relationship between the pixels identified as the object in the image and the position of the sensor in the spatial reference system. For example, the spatial transformation can be the solution of a linear system of equations that includes several unknown variables (translations and rotations in space) and multiple images acquired from the object. The linear system of equations can be determined by a variety of factors so that the number of acquired images is greater than the number of unknowns. It should be understood that the above techniques are exemplary only and that other calibration techniques may be utilized.

[0039] The fusion engine 112 utilizes this relationship to facilitate fusion between the image produced by the imaging device 120 and the image data 140, which may include, for example, previously acquired image data of a different modality. To illustrate this, consider an image-guided biopsy of the prostate. The image data 140 may include imaging of the prostate in a modality different from the modality produced by the imaging device 120. The modality of the image data 140 may provide better tissue discrimination, so that the prostate can be easily identified and healthy tissue of the prostate can be distinguished from abnormal tissue.

[0040] The position or posture of the prostate in a previously acquired image and the position or posture of abnormal tissue can be specified by feature information. In one example, the feature information may include the outline of the prostate and the separate outline of the abnormal tissue. However, in general, the term "feature information" relates to imaging data or other data used to specify the posture of a feature or object in a medical image. As used herein, the term "posture" refers to the position and orientation of an object in a given reference system, where the given reference system can be defined relative to another object. For example, the posture of a feature in an image is related to the position and orientation of the feature shown in the image or in an imaging space or volume. The posture of an imaging device relates to the position and orientation of the imaging device relative to an identified reference (e.g., a feature, a tracking volume, etc.), and may also refer to the direction of an imaging plane of the imaging device relative to the same identified reference.

[0041] Using the pose of a feature (e.g., a prostate) determined in the image data 140, the fusion engine 112 can determine a reference position and orientation (or pose) of the imaging device 120 relative to the feature. The reference position can be indicated on a display as, for example, a contour line of the feature from the viewpoint of the imaging device in the reference pose. More specifically, the imaging device in the reference pose defines an imaging plane. The image data 140 and / or feature information is resampled along the imaging plane to obtain a contour line. That is, voxel data or contour data that intersects the imaging plane is used to generate a contour line or other indication of the feature.

[0042] Imaging device 120 may generate a real-time image that may be displayed by computing device 110. Fusion engine 112 is configured to overlay the contours of the features on the real-time image to achieve alignment. Figure 6 An exemplary initial image captured by the imaging device 120 is depicted and displayed along with the contour lines generated by the fusion engine 112 as described above. The operator manipulates the imaging device 120 until the features shown in the real-time image correspond to the contour lines. For example, Figure 7 An exemplary image is depicted after an operator manipulates the imaging device 120 to align the imaged features with the contour lines.

[0043] Once correspondence is achieved, the fusion engine 112 can establish a relationship between the pose of the imaging device 120 in space (i.e., the reference frame provided by the spatial tracker 130) and the pose of the feature in the image data 140. The fusion engine 112 uses this relationship to lock the fusion between the real-time image and the image data 140. In particular, the fusion engine 112 maintains the alignment between the displayed feature information (e.g., contours) and the pose of the feature in the real-time image. For example, as the operator moves the imaging device 120, the image data 140 is processed and transformed to provide corresponding updated contours of the feature displayed on the changing real-time image (i.e., updating the fusion). For example, Figure 8 Depicted is an exemplary image captured by the imaging device 120 after an operator moves (e.g., tilts) the imaging device 120. Figure 8 In the embodiment of the present invention, the contours of the features are updated in response to movement of the imaging device to maintain correspondence.

[0044] The spatial tracker 130 generates tracking information indicating the posture of the imaging device 120 in space. Based on the tracking information, the fusion engine 112 determines the position and orientation of the corresponding imaging plane and re-slices the image data 140 along the imaging plane. Feature information (e.g., contours) may also be re-sliced ​​along the imaging plane to generate updated contour lines. Therefore, as the operator moves the imaging probe 122 and the real-time image changes, the fusion engine 112 responsively updates the feature information obtained from the image data 140 to provide a composite image based on the real-time image and the image data 140.

[0045] If the patient moves position with or without corresponding movement of the imaging device 120, the relationship between the imaging plane and the feature may change. Therefore, the relationship (i.e., the initial alignment) may need to be updated. On the other hand, the computing device 110 may enable the operator to manually adjust the contour line or contour to correct the relationship. Alternatively, the fusion engine 112 may relock the relationship. For example, the fusion engine 112 may determine a new reference pose and identify when the feature imaged by the imaging device 120 corresponds to the new reference pose.

[0046] Figure 2 Shown with Figure 1 A schematic block diagram of an exemplary non-limiting embodiment of a computing device 110 associated with the image fusion system 100 is shown. Figure 2As shown, the computing device 110 includes one or more processors 202 configured to execute computer-executable instructions such as instructions that constitute the fusion engine 112. Such computer-executable instructions may be stored on one or more computer-readable media, including non-transitory computer-readable storage media such as memory 208. For example, the memory 208 may include non-volatile memory to permanently store the fusion engine 112 and / or data 212 (e.g., image data, feature information, tracking information, captured image data, configuration information, working data, etc.). The memory 208 may also include volatile memory that stores the fusion engine 112 and other data 212 (or portions thereof) during execution by the processor 202.

[0047] The computing device 110 includes a communication interface 206 to couple the computing device 110 to various remote systems (e.g., image data storage, imaging devices, etc.). The communication interface 206 may be a wired or wireless interface, including but not limited to a WiFi interface, an Ethernet interface, a fiber optic interface, a cellular radio interface, a satellite interface, etc. An I / O interface 210 is also provided to couple the computing device 210 to various input and output devices, such as a display, a touch screen, a keyboard, a mouse, a touchpad, etc. As examples, the I / O interface 210 may include a wired or wireless interface, such as but not limited to a USB interface, a serial interface, a WiFi interface, a short-range RF interface (Bluetooth), an infrared interface, a near field communication (NFC) interface, etc. Various peripheral devices used to couple the computing device 110 to the fusion engine 112 by wire or wireless are also shown in the peripheral interface 204. For example, peripherals interface 204 may couple computing device 110 to sensor 132 to receive signals, to imaging device 120 to receive real-time images, to imaging probe 122 to receive raw signals for processing, and / or to spatial tracker 130 to receive tracking information.

[0048] Now go to Figure 3 , depicts a block diagram of an exemplary non-limiting fusion engine 112. Figure 3 As shown, the fusion engine 112 may include various functional modules implemented by computer executable instructions, such as an initialization module 302 , a synchronization module 304 , a fusion module 306 , and a display module 308 .

[0049] The initialization module 302 receives user input 314 to configure the fusion engine 112 to perform the medical image fusion disclosed herein. For example, the initialization module 302 can load image data 316 from a local or remote memory based on the selection included in the user input 314. The initialization module 302 can also establish a reference pose for the imaging device, which can be a default pose or a pose provided in the user input 314. As described above, the reference pose can be a desired pose, which can be an initial target pose of the imaging device to achieve image fusion according to one or more aspects of the present invention. Based on the reference pose, the initialization module 302 can process the image data 316 and the feature information 318 to generate initial fusion information (e.g., display data, etc.), which is combined with the real-time image data 322 by the display module 308 to generate a composite image 324 that can be displayed on a display.

[0050] The synchronization module 304 determines when the imaging device is aligned with the reference pose and when the relationship between the pose of the imaging device in space and the position and orientation of the feature in the image 316 (such as the position and orientation provided by the feature information 318) is locked. The synchronization module 304 can determine the alignment based on the user input 314. For example, the operator can provide input when the initial fusion information synthesized with the real-time image data 322 visually shows the alignment between the feature in the real-time image and the feature from the image data 316. Alternatively, the registration engine or other image processing module can evaluate the synthesized image 324, the image data 316, the feature information 318, and / or the real-time image data 322 to computationally determine the alignment.

[0051] Once the relationship is established, the fusion module 306 maintains the alignment. For example, the fusion module 306 includes a pose determination module 310 that determines the pose of the imaging device based on tracking information 320 from the spatial tracker. The transformation module 312 processes the image data 316 and / or the feature information 318 to update the fused information that is combined with the real-time image data 322 to generate the synthesized image 324. As described above, the transformation module 312 can re-slice the image data 316 and / or the feature information 318 according to a virtual image plane, wherein the virtual image plane corresponds to the imaging plane of the imaging device in the pose specified by the tracking information 320. In another example, the transformation can be determined based on the tracking information 320, and the determined transformation can be directly applied to the fused information synthesized with the real-time image data 322.

[0052] Figure 4 A flow chart of an exemplary non-limiting method for predicting fusion between at least two image modalities is shown. Figure 4The method can be performed, for example, by the image fusion system 100 described above and / or the fusion engine 112 executed on the computing device 110. At 400, the position and orientation of a feature in a first image of a first modality are identified. For example, a contour can be defined that indicates the feature in the first image. At 402, tracking information specifying the position and orientation of an imaging device of a second modality is obtained. The tracking information can be obtained by spatial tracking or other positioning systems. At 404, it is determined when the imaging device is placed in a predetermined posture relative to the feature. The determination can be based on tracking information, feature information (e.g., contour), image data of the first image, or user input. At 406, the first image data and / or associated contours are transformed in response to the tracking information to maintain alignment between the first image and the image generated by the imaging device.

[0053] Reference now Figure 5 , a flow chart illustrating an exemplary non-limiting method for producing a fusion between a real-time image and a previously acquired image. Figure 5 The method may be performed, for example, by the image fusion system 100 described above and / or a fusion engine 112 executing on a computing device 110. At 500, a real-time image of a first modality is acquired from an imaging device. At 502, an indication of a feature from a second image having a different modality is displayed over the real-time image. At 504, when it is determined that the imaging device is in a desired pose relative to the feature, a relationship is established between the real-time image and the second image. At 506, tracking information is acquired. The tracking information specifies a position and orientation of the imaging device in a reference space. At 508, the second image is processed based on the tracking information to update an indication of the feature and possible other features displayed on the real-time image.

[0054] The above exemplary embodiments are given in the context of image fusion. It should be understood that these concepts can be extended to other contexts, such as image correlation. For example, instead of combining or compositing (i.e., overlapping) portions of images, images or portions of images can be displayed side by side or in another arrangement for comparison.

[0055] Exemplary Computing Devices

[0056] As mentioned, advantageously, the techniques described herein may be applied to any device that desires to provide predictive fusion and real-time tracking of images of different modalities. Thus, it will be appreciated that handheld, portable, and other computing devices and various computing objects may be contemplated for use in conjunction with various embodiments of the registration visualization system. Fig. 9 The following general-purpose computer described in is just one example of a computing device.

[0057] Embodiments may be implemented in part via an operating system for use by developers of services for devices or objects, and / or included within application software that operates to perform one or more functional aspects of the various embodiments described herein. Software may be described in the broad context of computer-executable instructions (such as program modules) executed by one or more computers, such as client workstations, servers, or other devices. Those skilled in the art will appreciate that computer systems have a variety of configurations and protocols that can be used to communicate data, and therefore, no particular configuration or protocol is considered limiting.

[0058] therefore, Fig. 9 An example of a suitable computing system environment 900 is shown in which one or more aspects of the embodiments described herein may be implemented, although as noted above, the computing system environment 900 is only one example of a suitable computing environment and is not intended to suggest any limitation as to scope of use or functionality. Additionally, the computing system environment 900 is not intended to be interpreted as having any dependency related to any one or combination of components shown in the exemplary computing system environment 900.

[0059] refer to Fig. 9 , an exemplary device for implementing one or more embodiments includes a general purpose computing device in the form of a computer 910. Components of the computer 910 may include, but are not limited to, a processing unit 920, a system memory 930, and a system bus 922, where the system bus 922 couples various system components including the system memory to the processing unit 920.

[0060] The computer 910 typically includes a variety of computer-readable media, and may be any available media that the computer 910 can access. The system memory 930 may include computer storage media in the form of volatile and / or non-volatile memory, such as read-only memory (ROM) and / or random access memory (RAM). As an example and not limitation, the system memory 930 may also include an operating system, application programs, other program modules, and program data. According to another example, the computer 910 may also include various other media (not shown), which may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technology, compact disk (CD) ROM, digital versatile disk (DVD) or other optical disk storage, or other tangible and / or non-temporary media that can be used to store the desired information.

[0061] A user may enter commands and information into the computer 910 through input devices 940. A monitor or other type of display device is also connected to the system bus 922 via an interface, such as output interface 950. In addition to the monitor, computers may also include other peripheral output devices, such as speakers and printers, which may be connected via output interface 950.

[0062] The computer 910 may include a network interface 960 to operate in a networked or distributed environment using logical connections to one or more other remote computers, such as remote computer 970. The remote computer 970 may be a personal computer, a server, a router, a network PC, a peer device or other public network node or any other remote media consumption or transmission device, and may include any or all of the elements described above with respect to the computer 910. Fig. 9 The logical connections described in the 970 ...

[0063] As noted above, although exemplary embodiments have been described in conjunction with various computing devices and network architectures, the underlying concepts may be applied to any network system and any computing device or system in which it is desired to implement an image fusion system.

[0064] Furthermore, there are multiple ways to implement the same or similar functionality, such as appropriate APIs, toolkits, driver code, operating systems, controls, standalone or downloadable software objects, etc., that enable applications and services to take advantage of the technology provided herein. Thus, the embodiments herein are considered from the perspective of an API (or other software object) and from the perspective of a software or hardware object that implements one or more embodiments described herein. Thus, the various embodiments described herein may have aspects that are entirely in hardware, partially in hardware and partially in software, and entirely in software.

[0065] As used herein, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless specified otherwise, or clear from context, the phrase "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, the phrase "X employs A or B" is satisfied by any of the following examples: X employs A; X employs B; or X employs both A and B. Furthermore, the articles "a" and "an" used in this disclosure and the appended claims should generally be construed to mean "one or more" unless specified otherwise or clear from context to be directed to a singular form.

[0066] Additionally, as used herein, the term "exemplary" is intended to mean "serving as an illustration or example of something."

[0067] Illustrative embodiments have been described above. It will be apparent to those skilled in the art that the above apparatus and methods may include changes and modifications without departing from the overall scope of the claimed subject matter. It is intended that all such modifications and substitutions are included within the scope of the claimed subject matter. Furthermore, to the extent that the term "comprising" is used in the detailed description or claims, such term is intended to be included in a manner similar to the term "comprising" as "comprising" is interpreted when used as a transitional word in a claim.

Claims

1. A system comprising: a processor coupled to a memory, the memory storing computer executable instructions that, when executed by the processor, configure the processor to: Obtaining feature information indicating a feature in a three-dimensional first image dataset of a first modality, the feature information comprising contour data of the feature in the first image dataset; determining a reference pose of a handheld imaging device that generates an image of a second modality, the reference pose of the handheld imaging device being a predicted position of the handheld imaging device and being suitable for generating a second image of the feature having a pose that matches the reference pose of the feature; Displaying a second image acquired in real time from the handheld imaging device according to the feature information and a reference posture of the handheld imaging device, wherein the second image displays a contour of the feature from a viewpoint of the handheld imaging device in the reference posture; determining when a posture of the handheld imaging device matches a reference posture based on the feature information and a contour of the feature; When the posture of the handheld imaging device matches the reference posture, obtaining a relationship between the posture of the handheld imaging device in space and the posture of the feature in the first image data set; obtaining tracking information in at least three degrees of freedom from a spatial tracker configured to identify a position and orientation of the handheld imaging device; and In response to the movement of the handheld imaging device, a spatial relationship between a reference pose of the handheld imaging device and a position of the handheld imaging device is detected based on the tracking information, and based on the tracking information and the feature information, the first image data set is re-sliced ​​along a virtual imaging plane, and a contour line covering the feature is updated according to the feature information acquired by the handheld imaging device based on the virtual imaging plane to update an indication of the pose of the feature in the second image generated by the handheld imaging device, the virtual imaging plane being determined based on the tracking information; and, during the movement of the handheld imaging device, an alignment between the contour line and the pose of the feature in the real-time second image is maintained by utilizing the relationship between the pose of the handheld imaging device in space and the pose of the feature in the first image data set.

2. The system according to claim 1, wherein: The processor is further configured to interpolate image data of the first image corresponding to the virtual imaging plane.

3. The system according to claim 1, wherein: The processor is also configured to identify a transformation between the first image and the second image.

4. The system according to claim 3, wherein: The processor is further configured to apply the transformation to the feature information in order to update the identified position of the feature.

5. The system according to claim 1, wherein: The feature information also includes information specifying an orientation of the feature in the first image relative to a given orientation of the handheld imaging device.

6. A method comprising: Obtaining feature information indicating a feature in a three-dimensional first image dataset of a first modality, the feature information comprising contour data of the feature in the first image dataset; determining a reference pose of a handheld imaging device that generates an image of a second modality, the reference pose of the handheld imaging device being a predicted position of the handheld imaging device and being suitable for generating a second image of the feature having a pose that matches the reference pose of the feature; Displaying a second image acquired in real time from the handheld imaging device according to the feature information and a reference posture of the handheld imaging device, wherein the second image displays a contour of the feature from a viewpoint of the handheld imaging device in the reference posture; determining when a posture of the handheld imaging device matches a reference posture based on the feature information and a contour of the feature; When the posture of the handheld imaging device matches the reference posture, obtaining a relationship between the posture of the handheld imaging device in space and the posture of the feature in the first image data set; obtaining tracking information in at least three degrees of freedom from a spatial tracker, the spatial tracker indicating tracking information of a position and orientation of the handheld imaging device; and In response to the movement of the handheld imaging device, a spatial relationship between a reference pose of the handheld imaging device and a position of the handheld imaging device is detected based on the tracking information, and based on the tracking information and the feature information, the first image data set is re-sliced ​​along a virtual imaging plane, and a contour line covering the feature is updated according to the feature information acquired by the handheld imaging device based on the virtual imaging plane to update an indication of the pose of the feature in the second image generated by the handheld imaging device, the virtual imaging plane being determined based on the tracking information; and, during the movement of the handheld imaging device, an alignment between the contour line and the pose of the feature in the real-time second image is maintained by utilizing the relationship between the pose of the handheld imaging device in space and the pose of the feature in the first image data set.

7. The method according to claim 6, wherein: With the imaging device being in a pose relative to a reference pose of the feature, an indication of the feature is initially generated based on feature information indicative of the feature relative to the pose in the second image.

8. The method according to claim 6, wherein: The re-slicing further includes interpolating image data of the second image corresponding to the virtual imaging plane.

9. The method according to claim 6, wherein: The instruction to update the pose of the feature also includes: determining a transformation between a reference pose of the handheld imaging device and a position and orientation of the handheld imaging device provided by the tracking information; and The transformation is applied to the feature information.

10. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, configure the processor to: Obtaining feature information indicating a feature in a three-dimensional first image dataset of a first modality, the feature information comprising contour data of the feature in the first image dataset; determining a reference pose of a handheld imaging device configured to generate an image of a second modality, the reference pose of the handheld imaging device being a predicted position of the handheld imaging device and being suitable for generating a second image of the feature having a pose matching the reference pose of the feature; Displaying a second image acquired in real time from the handheld imaging device according to the feature information and a reference posture of the handheld imaging device, wherein the second image displays a contour of the feature from a viewpoint of the handheld imaging device in the reference posture; determining when a posture of the handheld imaging device matches a reference posture based on the feature information and a contour of the feature; When the posture of the handheld imaging device matches the reference posture, obtaining a relationship between the posture of the handheld imaging device in space and the posture of the feature in the first image data set; obtaining tracking information from a spatial tracker configured to track a position and orientation of the handheld imaging device in space in at least three degrees of freedom; In response to the movement of the handheld imaging device, a spatial relationship between a reference pose of the handheld imaging device and a position of the handheld imaging device is detected based on the tracking information, and based on the tracking information and the feature information, the first image data set is re-sliced ​​along a virtual imaging plane, and a contour line covering the feature is updated according to the feature information acquired by the handheld imaging device based on the virtual imaging plane to update an indication of the pose of the feature in the second image generated by the handheld imaging device, the virtual imaging plane being determined based on the tracking information; and, during the movement of the handheld imaging device, an alignment between the contour line and the pose of the feature in the real-time second image is maintained by utilizing the relationship between the pose of the handheld imaging device in space and the pose of the feature in the first image data set.

11. The non-temporary computer-readable storage medium of claim 10, further storing instructions for configuring the processor to: display an initial indication of the posture of the feature in the second image, wherein the initial indication is generated based on the feature information, and provide guidance for aligning the handheld imaging device with the reference posture.

12. The non-transitory computer-readable storage medium of claim 10, further storing instructions for configuring the processor to interpolate the feature information along the virtual imaging plane to generate an updated indication of the pose of the feature in the second image.

13. The non-transitory computer-readable storage medium of claim 10, further storing instructions for configuring the processor to: determining a transformation between the reference pose and the position and orientation of the handheld imaging device provided by the tracking information; and The transformation is applied to the feature information to update an indication of a pose of the feature in the second image.