Ultrasound system and method for correcting motion-induced misalignment in image fusion

Abstract

The present disclosure describes ultrasound imaging systems and methods that can enable automatic identification of image planes in a preoperative volume of a real-time image corresponding to a moving region of interest. An example method includes: receiving real-time ultrasound image data from a probe associated with a position tracking sensor; generating a real-time image based on the real-time ultrasound data; and deriving a motion model from the real-time ultrasound image data. The method may also include automatically identifying an image plane in a preoperative data set to correspond to the real-time ultrasound image by correcting for motion-induced misalignment between the real-time data and the preoperative data.

Description

Technical Field

[0001] The present disclosure generally relates to ultrasound systems and methods for correcting motion-induced misalignment when performing image fusion, such as when fusing real-time images (e.g., during real-time ultrasound) and pre-operative images (e.g., images obtained using a different imaging modality such as magnetic resonance (MR) or computed tomography (CT), or pre-operative 3D ultrasound (US) data). Background Art

[0002] Fusion of a patient's live ultrasound image with a preoperative 3D volume of the same patient but from another modality (usually but not limited to MR or CT) requires spatial calibration. For this purpose, ultrasound probes are usually equipped with position tracking (e.g., electromagnetic (EM) tracking) sensors. One such medical imaging system is described in U.S. Patent Application Publication US 2011 / 028843A1. In order to align images from the two modalities in space, the tracking system is first calibrated. The calibration step provides a strict transformation between the live ultrasound data and the preoperative data set so that spatial synchronization can be automatically maintained. However, even if the calibration step is performed very accurately, due to the patient's movement, such as due to breathing or cardiac movement, the fused image can also experience misalignment errors under real-time tracking conditions. In order to reduce this fusion error, the "default" preoperative 2D plane used for fusion (as indicated by the EM tracker) can be replaced by another plane P representing different respiratory states. However, to do so, multiple data sets of preoperative data associated with different respiratory states will need to be available for applying the solution. More often, only a single set of preoperative data is available. Therefore, it is more often necessary to select an alternative plane P from a single available preoperative data set, and existing systems vary in the way they identify this alternative plane P to use. For example, in one known system, a live ultrasound image and a preoperative CT image are aligned via image processing performed on the live image and the CT image, thereby matching features in the two sets of image data. Another known system uses an additional respiratory tracking system with a sensor that is positioned on the patient's chest to derive a model for correcting for respiratory-induced misalignment.

[0003] EP 3056151 A1 relates to a method for fusing at least one ultrasound image with a pre-stored modality image such as a CT or MR image. The ultrasound fusion imaging method comprises a selection step, a registration step and a fusion step. The selection step is used to select at least one ultrasound image from at least one previously stored ultrasound video data slice, which includes an ultrasound image obtained by acquiring a target object in at least one plane, and position indication information corresponding to each ultrasound image. The position information is generated by a position sensor fixed to an ultrasound probe. The registration step is used to register the selected at least one ultrasound image with the modality image using the position of the position indication information of the at least one ultrasound image. The fusion step is used to fuse the registered ultrasound image with the modality image.

[0004] WO 2017 / 109685 A1 discloses a medical imaging device for examining the volume of an object. The medical imaging device includes an ultrasound acquisition unit, including an ultrasound probe, for acquiring ultrasound image data of an object; an image interface for receiving medical image data of the object; and a position determination unit for determining the position of the ultrasound probe. An alignment unit is provided, which is used to align ultrasound image data and medical image data based on anatomical features of the object and a detection position of the ultrasound probe, and is used to adjust the alignment of the ultrasound image data and the medical image data based on a motion model. The motion model is defined based on the position of the ultrasound probe and the change of a repeated pattern of common anatomical features throughout multiple ultrasound images. An image processing unit is provided for processing ultrasound image data and medical image data to fuse the ultrasound image data and the medical image data based on alignment with the combined image data.

[0005] However, currently known solutions may not be as robust as might be desired, for example because they either rely on the image quality of both pre-operative and live ultrasound data, because they rely on image features such as diaphragm position, which may not always be distinguishable for a good match or even not appear in the image, depending on the orientation of the probe towards the anatomy of interest (e.g., the liver), and because in some cases they require additional hardware (e.g., a respiratory motion tracker) and introduce additional complexity to the system. Therefore, alternative methods for correcting motion misalignment can be desired. Summary of the invention

[0006] Examples according to the present disclosure may provide improved systems and methods for correcting motion-induced misalignment when performing image fusion.

[0007] An ultrasound imaging system according to the principles of the present disclosure may include an ultrasound probe and a position tracking sensor connected to the probe or operatively associated therewith, the ultrasound probe being used to image an object in real time. The system may include a processor communicatively coupled to the ultrasound probe and a source of previously acquired image data, the previously acquired image data including a 3D data set corresponding to an imaging volume of the object. The processor may be configured to receive real-time ultrasound image data and generate a real-time ultrasound image (e.g., a 2D or 3D image) based on a current image frame from the real-time ultrasound data. The processor may also be configured to derive a motion model based on the real-time ultrasound image data and resynchronize the motion model with the movement of anatomical features of the object (e.g., automatically or in response to user input). The processor may also be configured to identify a motion-corrected image plane in the imaging volume based on position information from the position tracking sensor and the motion model to generate a motion-compensated preoperative image, and fuse the real-time ultrasound image with the motion-compensated preoperative image to produce a combined image.

[0008] A method according to the principles of the present invention may include: receiving real-time ultrasound image data in response to ultrasound waves transmitted by a probe toward an object, wherein the probe is associated with a position tracking sensor, and generating a real-time ultrasound image based on a current image frame from the real-time ultrasound data. The method may also include: deriving a motion model based on the real-time ultrasound image data; resynchronizing the motion model with movement of anatomical features of the object; accessing a previously acquired image data set corresponding to an imaging volume of the object; identifying a motion-corrected image plane of the imaging volume based on position information from the position tracking sensor and the motion model to generate a motion-compensated preoperative image, and fusing the real-time ultrasound image with the motion-compensated preoperative image to produce a combined image.

[0009] Any features or preferred embodiments described herein with respect to the method of the present invention may also be applied to the ultrasound imaging system of the present invention.

[0010] According to the principles of the present invention, no respiratory motion tracker is used in either the ultrasound imaging system or the method, ie the motion model is derived without using data from a respiratory motion tracker. This reduces the hardware complexity of the system and method of the present invention.

[0011] Furthermore, according to the principles of the present invention, the motion-corrected image planes in the imaging volume are identified purely based on the position information from the position tracking sensor and the motion model. Thus, preferably, no registration or alignment is performed, for example, based on the alignment of anatomical features between the real-time ultrasound image and the pre-operative image data set, and thereby the present invention is independent of the image quality of the real-time ultrasound image and the pre-operative image data set, and is therefore very robust.

[0012] According to a preferred embodiment, the processor may be configured to generate long term correlation (LTC) data for each of a plurality of incoming frames of real-time ultrasound image data; and determine a periodic frequency and phase associated with each incoming frame by identifying at least two local minima of an LTC curve corresponding to the LTC data associated with each incoming frame. The LTC curve is generated by subtracting the incoming frame from each previous frame (or vice versa) and summing the subtraction results for all pixels, as described in equation (1) below. Preferably, at least previous frames from one or two previous cycles are used to calculate the LTC curve. Thus, the LTC curve for the incoming frame will have a minimum when the subject was last in the same respiratory position in the past, which would be one cycle length ago, but there may also be a minimum in between, i.e., when the subject is currently in an intermediate respiratory position, i.e., a respiratory position that is passed through twice in each respiratory cycle. Thus, information about the periodic frequency and phase of the motion may be derived from the LTC curve. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A block diagram of an ultrasound imaging system according to an embodiment of the present disclosure is shown.

[0014] Figure 2 A processing component of an ultrasound imaging device according to the principles of the present disclosure is shown.

[0015] Figure 3 A functional block diagram of an algorithm for correcting motion-induced misalignment when performing image fusion between real-time data and pre-operative data in accordance with the principles of the present disclosure is shown.

[0016] Figure 4 Long term correlation curves and local minima for determining parameters of a motion model according to the principles of the present disclosure are shown.

[0017] Figure 5 An example breathing-induced motion model of the liver as derived from real-time ultrasound data in accordance with the principles of the present disclosure is shown.

[0018] Figure 6 Mapping of long-term correlation data to an error array to perform error minimization in accordance with the principles of the present disclosure is illustrated.

[0019] Figure 7 A functional block diagram of an algorithm for correcting motion-induced misalignment according to the principles of the present disclosure is shown, the algorithm including a static update block.

[0020] Fig. 8A A functional block diagram of another algorithm for correcting motion-induced misalignment when performing image fusion between real-time data and pre-operative data according to further principles of the present disclosure is shown.

[0021] Figure 8B FIG. 1 shows a schematic diagram of a system with a static update box added according to the principles of the present disclosure. Fig. 8A Functional block diagram of the algorithm in .

[0022] Fig. 9 A block diagram of an ultrasound imaging system according to the principles of the present disclosure is shown.

[0023] Fig.10 is a flow chart of a method for real-time imaging according to the principles of the present disclosure. DETAILED DESCRIPTION

[0024] The following description of specific exemplary embodiments is merely exemplary in nature and is in no way intended to limit the present disclosure or its application or use. In the following detailed description of embodiments of the present system and method, reference is made to the accompanying drawings forming a part thereof, and in which specific embodiments are illustrated by way of illustration, in which the described system and method can be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the currently disclosed system and method, and it should be understood that other embodiments may be utilized and that structural and logical changes may be made without departing from the spirit and scope of the present system. In addition, for the sake of clarity, when it is obvious to those skilled in the art, the detailed description of certain features will not be discussed so as not to obscure the description of the present system. Therefore, the following detailed description should not be regarded as having a limiting meaning, and the scope of the present system is limited only by the appended claims.

[0025] According to the principles of the present invention, an ultrasound imaging system may be communicatively connected to a source of real-time ultrasound image data (e.g., an ultrasound probe) and a source of previously acquired image data (interchangeably referred to herein as preoperative data). For example, the system may include an ultrasound probe or be operatively connected to an ultrasound probe for real-time acquisition of ultrasound image data. The ultrasound imaging system may also be connected to a preoperative data source, for example, a data storage device such as a picture archiving and communication system (PACS) server. The preoperative data may be a 3D data set corresponding to the imaging volume of the object, which may have been acquired using any imaging modality (e.g., CT, MRI, PET, or 3D US). The system may also include a position tracking system or be operatively associated with a position tracking system. For example, the probe may be equipped with a position tracking sensor, such as an electromagnetic (EM) sensor or other type of sensor, a field generator, and a processor configured to determine the position of the probe in 3D space. The system may also include a calibration unit (which may also be referred to as a registration unit) configured to maintain alignment or registration between the real-time ultrasound image and the preoperative data so that a fused image including the ultrasound image data and the preoperative data may be displayed to the user in real time. As will be further described with reference to the accompanying drawings, the system may include a motion correction unit that enables the system to automatically identify different (motion-adjusted or motion-compensated) candidate image planes from the pre-operative data for fusion with the real-time data, rather than otherwise selecting based purely on position information from the position tracking system.

[0026] Figure 1 A diagram of a system 100 according to some embodiments of the present disclosure is shown. The system 100 includes a medical imaging device (e.g., an ultrasound imaging device 110) that is operatively (i.e., communicatively, such as via a wired or wireless communication link) connected to a probe 112. The probe 112 can be used to image a volume (e.g., an anatomical structure, such as the liver, heart, kidney, or a portion thereof) of an object 101. The volume to be imaged may include a region of interest (e.g., an anatomical part or structure of interest, such as the liver, heart, kidney, or any other tissue or organ). In the illustrated example, the probe 112 is an ultrasound probe that includes an array of transducer elements that are operable to transmit ultrasound toward the volume to be imaged and are also operable to detect echoes in response to the transmitted ultrasound. The transducer elements may be arranged in a 1D or 2D array. The ultrasound probe 112 may be used to obtain real-time ultrasound image data. The real-time ultrasound image data may be used to generate a real-time ultrasound image (also referred to as a live ultrasound image). The terms real-time and live are generally used to describe image data acquisition and display of images occurring during acquisition of the image data, as opposed to post-acquisition display and evaluation of images generated from previously acquired image data.

[0027] The imaging device 110 may include a control unit 120 connected to the ultrasound probe 112 for controlling the acquisition of ultrasound data, for example, for providing beam steering and / or beam forming functions. The control unit 120 may include a processing unit 122, which is operable to generate an ultrasound image based on the detected echoes. The processor 122 is also configured to receive preoperative data, automatically spatially register the incoming live ultrasound image with the preoperative data, and also adjust the spatial registration to compensate for motion-induced misalignment before fusing the image data of the two data sets (i.e., the real-time data and the preoperative data).

[0028] The imaging device 110 may also include a memory that may be configured to store image data (e.g., real-time ultrasound image data) and / or processor-executable instructions for configuring a processor to perform functions associated with spatial registration and image fusion, among other things. As described, the processor 122 may be configured (e.g., in response to processor-executable instructions stored in the memory) to automatically spatially register the live ultrasound image to the preoperative data, e.g., to extract corresponding slices from the preoperative 3D data set. According to some embodiments, the spatial registration may involve motion correction. Motion correction may be performed by applying a motion model to the preoperative data to identify an alternative slice in the preoperative data (different from a default slice selected based solely on the position data) that better corresponds to the live image after motion-induced misalignment has been accounted for by the motion model.

[0029] The imaging device 110 is communicatively connected to a preoperative data source 150, such as a storage device (e.g., a PACS server) or another medical imaging device (e.g., an MRI scanner, a CT scanner, an ultrasound scanner, or any other device using any type of imaging modality or any combination thereof). The preoperative data may be a 3D data set of medical image data, which may be obtained by any imaging modality. In some embodiments, the preoperative data may be obtained by an imaging modality different from ultrasound, for example, it may be a 3D data set (also referred to as a CT volume, an MRI volume, a PET volume, respectively) obtained by computed tomography (CT), magnetic resonance imaging (MRI), or positron emission tomography (PET), or the preoperative data may be previously acquired ultrasound data, for example, a 3D US data set (or US volume) obtained from the same patient. As the name implies, the preoperative data is obtained prior to real-time imaging (e.g., live ultrasound imaging, which may occur during an invasive procedure such as a biopsy). Therefore, the preoperative data is interchangeably referred to herein as previously acquired image data. In the illustrated example, the processor 122 includes a calibration module 124 for spatially registering the live image to the preoperative data, a motion correction unit 126 for adjusting the spatial registration to account for motion-induced misalignment, and a fusion unit 128 operable to fuse the motion-compensated preoperative image with the live ultrasound image to produce a combined image. The imaging device 110 is operatively associated with a user interface 130, which includes, for example, a display unit 132 for displaying the combined image in real time. The user interface may also include a user input device 134 for receiving user input, for example, for controlling the image acquisition process. The input device 134 may include, for example, but not limited to, a keyboard, a mouse, a trackball, one or more buttons provided on a control panel, and / or one or more soft controls provided on a touch-sensitive display.

[0030] The imaging device 110 is operatively associated with a position tracking system 140 (e.g., an electromagnetic tracking (EM), optical, or other type of position tracking system). The position tracking system 140 may include a position tracking sensor 142 (e.g., an EM sensor) attached to a probe and a processor configured to determine the position of the sensor 142 and, therefore, the position of the probe with reference to a 3D coordinate system. For example, the position tracking system may correlate the spatial position of the EM sensor (and, therefore, the spatial position of the probe) with the coordinate system of the EM tracking field generator. The calibration unit 124 may then register or correlate the coordinate system of the tracking field generator with the coordinate system of the preoperative image data, thereby enabling spatial registration between the preoperative image data and the live ultrasound data. In this manner, the position of the ultrasound probe 112 may be used to identify a corresponding image plane within the preoperative volume. In some examples, the functionality of the processor may be performed by the calibration unit 124, or may be typically implemented within the processor 122 of the imaging device 110.

[0031] The fusion unit 128 may be used to fuse the live image with the preoperative image to produce a combined image for display. The live image may overlap with the preoperative image, or alternatively the images may be displayed side by side to produce a fused image or a combined image. However, as described above, image registration or alignment based solely on position tracking information may not address motion-induced misalignment issues, and thus may reduce the quality of the image fusion process. Therefore, as described herein, the processor 122 may also be configured with additional functionality to account for and correct such motion-induced misalignment.

[0032] According to an embodiment of the present disclosure, the processor 122 may include a motion correction unit 126. The motion correction unit 126 may utilize a motion model to correct any motion-induced misalignment between the live ultrasound image and the preoperative data. The motion model may be a mathematical model (e.g., a mathematical representation, such as a linear model) of the motion trajectory of the anatomical feature of interest within the imaging volume. Figure 5 ). In some embodiments, a motion model can be derived based on image data including only ultrasound image data and specifically from real-time ultrasound data. That is, unlike existing solutions in which images from both ultrasound and preoperative (e.g., CT data) are image processed to match features in the images and thus facilitate alignment, the current motion model does not rely on preoperative data. Instead, and for example in the context of respiratory-induced motion, real-time ultrasound data is acquired for a time window greater than one respiratory cycle of the object, and respiratory cycle parameters are extracted only from the real-time ultrasound image data. In some examples, image processing can be performed on the real-time ultrasound data to identify repetitive features or patterns within the acquired frames, so that the phase, rate and / or amplitude of the motion can be determined. The phase, rate and amplitude of the motion can then be used to select an alternative image plane from the preoperative data set (except for a default or purely position-based image plane). In this way, the motion correction unit enables a live image to be more accurately aligned or registered to a corresponding image plane in the preoperative image data, even if the anatomical feature of interest is in motion, for example due to vital motion of the object such as breathing. In some embodiments, such as when imaging the liver, the motion may be considered to be generally in the craniocaudal direction, and thus the motion model may also be referred to as a craniocaudal motion model.

[0033] Figure 2 FIG. 2 shows a functional block diagram of a processor 200 of an ultrasound imaging system according to the present disclosure. The processor 200 may be used to implement Figure 1208, and applies a motion model 212 to a pre-operative data set 250 to extract motion-corrected image planes from the pre-operative set (as shown in block 220) to generate motion-corrected (or motion-compensated) pre-operative images corresponding to the live images. The motion-corrected pre-operative images and the live images are then combined by an image fusion unit 230 to produce a combined image 232 for display on a display unit 240.

[0034] like Figure 2 As shown, the processor 200 is communicatively coupled to a real-time ultrasound data source (e.g., an ultrasound probe) and a preoperative data source. The processor 200 can be configured to derive a motion model 212 based in part on imaging data including only ultrasound data (e.g., without reference to any other type of medical imaging data), and preferably without reference to respiratory motion tracking data. In some embodiments, the processor 200 can be configured to derive the motion model 212 based also on user input. For example, in the case of simple periodic respiration-induced motion, the motion of certain anatomical structures (e.g., the liver) can be characterized as a simple periodic motion in the cranio-caudal direction, and thus the motion model 212 can be a cranio-caudal model, i.e., an estimate of the respiration-induced motion of anatomical features along the cranio-caudal direction of the object as a function of time, defined based on the respiratory rate, phase, and amplitude of the respiratory cycle. Such a motion model assumes that the motion remains stable; however, in reality, the patient's respiration may change, and thus one or more parameters of the respiratory cycle (e.g., respiratory rate, phase, and / or amplitude) may be out of sync with the motion model. Thus, in some examples, the processor may also be configured to dynamically update the motion model to account for any perturbations in the motion (as shown in block 214), and thereby maintain synchronization between the motion estimated by the motion model and the actual motion of the anatomical feature (e.g., the patient's liver). The term dynamically implies that the processor may be configured to continuously and automatically update the motion model (e.g., at each incoming frame or after several new frames) to maintain synchronization between the motion model and the organ motion. In some examples, resynchronization of the model may be performed in response to user input, such as in response to receiving an indication to initiate synchronization.

[0035] In addition, in some examples, the processor 200 can be configured to monitor or detect the motion of the probe. For example, the processor 200 can be configured to determine whether the probe is stationary (as shown in box 216), for example, based on the position tracking data. When it is determined that the probe is stationary, the processor 200 can continue to update (for example, as shown in box 218) or synchronize the motion model with the actual motion of the anatomical structure. However, if it is determined that the probe is not stationary (for example, the probe motion is detected to exceed a given threshold), the processor 200 can suspend synchronization and use the version of the motion model that was most recently updated before the probe motion was detected. In the latter case, the registration or alignment of the motion correction of the live image and the preoperative image continues, but based on the static version of the motion model instead of the dynamically updated motion model until the probe motion stops and the processor 200 resumes the motion model synchronization. In addition to motion estimation and / or correction, the processor 200, alone or in combination with other processors of the medical imaging device, can be further configured to perform other functions of the medical imaging device, for example, a function for generating an ultrasound image based on real-time data and / or a function for generating an image based on preoperative data.

[0036] Figure 3 A functional block diagram of an example algorithm or process 300 for spatial registration with motion correction is shown, which may be implemented (in hardware and / or software) in a processor of a medical imaging device, such as the ultrasound imaging device 110 . Figure 3 The example algorithm in is based on long-term correlations computed over successive frames or portions thereof (also referred to as image frames) over an amount of time greater than one motion cycle (e.g., one breathing cycle). The description will be made with reference to the motion caused by breathing. Figure 3 However, the principles described can be applied to other types of motion. The algorithm or process 300 exploits the repetitive patterns that can be found in images through motion cycles. For example and also with reference to Figure 5 , which shows a liver motion model 504 represented by a trajectory or displacement curve 504 derived according to an embodiment of the present disclosure, it can be observed that the position of the anatomical feature during the inspiration phase can be very similar to the position of the anatomical feature during the expiration phase. D(t) or d(t) indicates the displacement d as a function of time. As an example, the position of feature 507 on inspiration 507-I is similar to the position of feature 507 on inspiration 507-E, and similarly, the position 509-I and 509-E of feature 509 on inspiration and expiration are similar. Therefore, by analyzing a series of temporally consecutive frames 505 (for example, by calculating long-term correlations between frames or by performing motion estimation), parameters defining periodic respiratory motion can be extracted separately from the ultrasound data, and thus a motion model can be defined over time based on the parameters extracted from the ultrasound data.

[0037] Return to reference Figure 3 , upon starting the algorithm or process 300, certain parameters of the algorithm are initialized, such as the frequency range, the number of frequency states (freqStateN), the number of phase states (phaseStateN), and the range of cumulative costs. In the case of respiratory motion, the actual respiratory cycle may be between 3 and 5 seconds for an adult and between 2 and 3 seconds for a child. The frequency range may be initialized at a default frequency range, such as approximately 12 to 20 cycles per minute for an adult or approximately 20 to 30 cycles per minute for a child, which may be determined at least in part based on patient age information such as input by a user, such as before starting live ultrasound. In some examples, the frequency range may be adjustable in response to user input, such as after defaulting to a preprogrammed range, or the system may require that the range be input by the user. Similarly, the number of frequency and phase states may be defaulted to given values, such as 32 frequency states and 64 phase states. These numbers are purely illustrative, and other numbers of states (less or greater than the specific example) may be used, which may be pre-programmed and / or adjustable by the user. For example, the number of phase states may be optimized to improve visual quality (eg, visual twitches may result from under-sampled respiratory cycles).

[0038] Once initiated, the phase is automatically updated by the algorithm taking into account the timing information, resulting in a new plane of the preoperative 3D volume being selected for each incoming live US image. Additionally, in some cases, resynchronization is performed whenever a disturbance or drift occurs in the pace of anatomical motion.

[0039] The process begins at block 304, where the algorithm receives multiple frames of real-time image data and timing information. The frames may also be coupled to a long-term storage device 308 to free up buffer space for new incoming frames. For each incoming frame, as shown in block 310, the processor 302 may generate long-term correlation (LTC) data.

[0040] The long-term correlation of each incoming frame can be calculated according to Equation 1:

[0041] Long-term correlation 帧(i) (t) = SUM 全部像素 (frame(i)-frame(t))(Equation 1)

[0042] Thus, the LTC curve for an incoming frame (Frame (i)) is generated by subtracting each previous frame (Frame (t)) from the incoming frame (or vice versa, by subtracting the incoming frame from the previous frame) and summing the results of the subtractions over all pixels. Preferably, at least one or two previous (temporally consecutive) frames are used to calculate the LTC curve. Figure 4Two example LTC curves are shown, given in arbitrary units over time, where the number on the abscissa indicates the number of previous frames used in the subtraction. Figure 4 The LTC curve 403 in corresponds to the LTC curve generated for the incoming frame at position 507-I in the respiratory cycle, as shown in FIG. Figure 5 As indicated. Figure 4 The LTC curve 405 in corresponds to the LTC curve generated for the incoming frame at position 509-I in the respiratory cycle, as shown in FIG. Figure 5 As indicated.

[0043] In some examples, an LTC curve 311 may be generated for each incoming live frame. The periodic frequency and phase associated with a given live frame may be identified by determining a local minimum from the LTC data. For example, referring also to FIG. 400 , which shows two LTC curves 403 and 405 Figure 4 , the algorithm can look for at least two strong minima in each curve. The periodic frequency can be identified by looking for a strong local minimum M1 at approximately the same location on each LTC curve. This minimum arises from the fact that the respiratory position - or displacement d(t) - one full cycle ago is similar to the current displacement, and therefore the incoming frame is similar to the frame acquired one respiratory cycle ago, so that subtracting the frames from each other produces a minimum. The phase (or position in the respiratory cycle) associated with this particular image frame can be found by identifying another strong minimum (e.g., M2, M3), which also relies on the regularization principle - that the phase of each new frame will be at a minimum that is consistent with the phase of the previous frame. In Figure 4 and Figure 5 In the example of FIG. 4 , curve 403 has a minimum at approximately half of the breathing cycle, which is an indication that curve 403 corresponds to a phase with an intermediate displacement d(t) in the breathing cycle, as shown in FIG. Figure 5 507-I in the respiratory cycle. Therefore, the respiratory motion was at the same position d(t) about half a cycle ago, resulting in a local minimum, and this gives an indication for the indication. For curve 405, the second minimum M2 is much closer to the origin of the LTC curve, indicating that the LTC curve corresponds to a position closer to the peak displacement d(t) within the respiratory cycle (e.g., maximum inspiration or maximum exhalation), for example Figure 5 The position of point 509_I in . Figure 4 The two LTC curves 403 and 405 in are associated with frames that are further apart in time, which is why the minima corresponding to the phase are spaced further apart than is the case for frames that are consecutive in time.

[0044] exist Figure 3The algorithm in the example utilizes an error minimization technique to determine the associated minimum from the LTC data. For example, as shown in blocks 312, 314, and 318, the cost (or error) associated with each state is calculated from the number of states (stateN) using the LTC data obtained in block 310. This essentially provides a mapping of the LTC data to an M x N array 313 (e.g., freqStateN xphaseStateN), where M is the number of frequency states and N is the number of phase states. This M x N array 313 is also referred to as an error array. The mapping may be performed using any known technique, for example, by calculating the area under the LTC curve for each increment for the number of frequency states for those frequencies within a set frequency range. In other words, and also with reference to Figure 6 The example LTC curve 601 in FIG. 601 may be divided into equal frequency state increments, such as in the frequency range 609 between 3-5 seconds, as shown by the dashed lines. For clarity, and to avoid cluttering the image, a small number (in this case, 6) of frequency states are shown, but it will be appreciated that a different number, such as 30, 32, 36, etc., may be used. A similar process may be used to calculate errors related to phase. For example, since phase is frequency-dependent, for any given frequency, the portion of the curve from time 0 to the time corresponding to the given frequency is divided into equal increments as the number of phase states, and the area under the curve for each increment is calculated, thereby obtaining an error value for the error array. Figure 6 Minimum values ​​605 and 607 corresponding to frequency and phase, respectively, associated with the live frame used to generate curve 601 are shown, as well as an M x N error array 603 to which the cost values ​​of curve 601 may be mapped according to this example.

[0045] In some examples, as shown in block 318, before the state associated with the minimum cost or error is selected, the cumulative cost of each state S from the number of states (stateN) can be calculated to smooth the data, e.g., as shown in block 314. For example, the algorithm can be configured to assume that the respiratory rate remains stable over a fraction of a cycle ("fraction of a cycle"), such as during one quarter of a respiratory cycle ("fraction" = 4). Given the corresponding array of stateN errors associated with past frames, the algorithm can then calculate a sum over that "fraction of a cycle", taking into account a "fraction" of phase change and a duration of the "fraction of a cycle" associated with the frequency state.

[0046] For each incoming frame, the algorithm outputs a respiratory rate (or frequency) and phase within a set range, such as a value between 0-1, which can be selected based on the state with the lowest cost, as shown in block 322. The cost array calculated at block 312 can also be provided to a long-term storage device 316, for example to free up buffer space, which can be configured to temporarily store only a subset of all frames acquired within a period of at least one motion cycle.

[0047] As described, processor 302 may receive probe information, such as position information of the probe or an indication of whether the probe is stationary or moving (e.g., as shown in block 306). Upon determining that the probe is moving, synchronization of the phase and rate with each incoming frame may be suspended (until the probe is stationary again), such as by resetting the cost value, as shown in block 320. In this case, spatial registration between the live image and the preoperative data may proceed using the motion model version from the last update without requiring dynamic updating for each new frame. To this end, reference is also made to Figure 7 , the processor 302' may include a static update block 330. The probe information and timing information may be additionally coupled to the static update block 330. In response to an indication that the probe is moving, the static update block 330 may output a frequency that corresponds to an average frequency over a period of time, since it would be expected that dynamic synchronization may not be reliable when the probe is moving. In this way, dynamic synchronization is suspended while the probe is moving, and the average frequency and next consecutive phase value are output at block 322 for each new frame.

[0048] like Figure 5 As shown in the example of , in the case of respiratory-induced liver motion, the motion model can be defined based on the rate, phase and amplitude of the respiratory cycle, for example according to Equation 2:

[0049] Motion model d(t) = amplitude*cos2(rate*t+phase) (Equation 2)

[0050] Or according to equation 3:

[0051] Motion model d(t) = amplitude * cos 2 (rate*t+phase) (Equation 32)

[0052] You can refer to Figure 3 The described dynamic synchronization algorithm dynamically updates the motion model in the context of each live frame, for example, by obtaining updated values ​​of rate and phase for each frame. The magnitude may be set in response to user input, or automatically by the system, for example based on motion estimation, for example, as described below with reference to Fig. 8AThe amplitude is set automatically by the system, for example based on a default value. In a preferred embodiment, the amplitude is also adjustable by user input, for example in the case where the combined image is not visually satisfactory.

[0053] Reference now Fig. 8A and Figure 8B Another technique for deriving or updating a motion model based solely on ultrasound data (ie, without relying on previously acquired image data) may utilize image processing performed on the ultrasound data to estimate motion. Fig. 8A and Figure 8B The algorithm in the example of relies on alignment of the ultrasound beam with the axis of motion (e.g., cranio-caudal in the case of respiratory motion), as shown in block 810. This can be accomplished manually, by positioning the probe so that the beam is aligned with the axis of motion, or by automatically steering the beam to achieve such alignment. In some cases, such as when manual positioning is used, the system can provide feedback to the user to indicate when proper alignment has been achieved. Fig. 8A and Figure 8B In the example of , synchronization (or model update) is performed only in response to user input. That is, the user can indicate to the system that a resynchronization is desired, for example by providing an indication of the start of a synchronization period. Fig. 8A and Figure 8B As shown, processor 802 receives an indication of the start of a synchronization period and the end of a synchronization period. As shown in box 812, processor 802 performs image data acquisition on a number of breathing cycles (2, 3 or more). As shown in box 812, processor 802 uses frames spanning the synchronization period to perform motion estimation. Motion estimation can be performed according to any currently known or later developed motion estimation technique, such as using a block matching motion estimation algorithm. A block matching algorithm is a method for locating matching macroblocks in a sequence of frames for the purpose of motion estimation. For example, a block matching algorithm involves dividing the current frame in a sequence of frames into macroblocks and comparing each macroblock with a corresponding block in a nearby frame (sometimes just the previous frame) and its adjacent neighbors. The output of motion estimation is an estimated trajectory of motion, as shown in 813. The trajectory is provided to box 814, in which the rate, phase and amplitude of the motion can be extracted from the estimated trajectory, as shown by 815. One way to obtain rate, phase and amplitude from the trajectory is to search for {phase, rate} pairs such that:

[0054] (phase, rate) = argmax(sum(abs(C[phase+k*rate]–C[phase+(k+1)*rate])) / (kmax+1)

[0055] As long as there is a value of [phase+(k+1]*rate] in the curve, the sum can be calculated, so kmax is the last possible value of k. The maximum independent variable provides the amplitude. Other techniques can be used to obtain the rate, phase, and amplitude. The rate (or frequency), phase, and amplitude are output as shown in box 816 and are used to identify motion compensated image planes from the pre-operative data for fusion with the real-time frame. Figure 8B The processor 802' in is similar to the processor 802, but differs in that it includes a processor similar to Figure 7 The static update box 830 of the example static update box in FIG.

[0056] Fig. 9 910 is a block diagram of an ultrasound imaging system constructed according to the principles of the present disclosure. Some or all of the components of the ultrasound imaging system 910 may be used to implement an ultrasound imaging system according to any example herein, such as Figure 1 The system 100 in FIG. Fig. 9 The ultrasound imaging system 910 in the embodiment includes: an ultrasound probe 912, a transducer array 914, a beamformer 922 and optionally a microbeamformer 916, a transmit / receive (T / R) switch 918, a transmit controller 920, and one or more processing components for generating an ultrasound image from detected echoes. For example, the system 910 may include a signal processor 926, a B-mode processor 928, a Doppler processor 960, a scan converter 930, a multi-plane reformatter 932, a volume renderer 934, an image processor 936, a graphics processor 940, etc. The system may also include a user interface 924, an input device 952, and an output device 938. Fig. 9 The components shown are illustrative only, and other variations, including eliminating components, combining components, rearranging components, and substituting components are all contemplated.

[0057] As shown, the ultrasound imaging system 910 includes an ultrasound probe 912, which includes a transducer array 914 for transmitting ultrasound waves and receiving echo information. Various transducer arrays can be used, for example, linear arrays, curved arrays, or phased arrays. The transducer array 914 can, for example, include a two-dimensional array of transducer elements (as shown in the figure), which can be scanned in both elevation and azimuth dimensions for 2D and / or 3D imaging. The transducer array 914 can be coupled to a microbeamformer 916, which can be located in the ultrasound probe 112. The microbeamformer 916 controls the transmission and reception of signals by the transducer elements in the array 914. In the illustrated example, the microbeamformer 916 is coupled to a transmit / receive (T / R) switch 918, which switches between transmission and reception and protects the main beamformer 922 from high-energy transmission signals. In some embodiments, such as in a portable ultrasound system, the T / R switch 918 and other elements of the system may be included in the ultrasound probe 912 rather than in a separate ultrasound system base. The ultrasound system base typically includes software and hardware components, including circuits for signal processing and image data generation, and executable instructions for providing a user interface.

[0058] Under the control of the microbeamformer 916, the transmission of ultrasound pulses from the transducer array 914 is directed by a transmit controller 920 coupled to the T / R switch 918 and the beamformer 922, which can receive input from the operation of a user of the user interface 924. The user interface 924 may include one or more input devices, such as a control panel 952, which may include one or more mechanical controls (e.g., buttons, encoders, etc.), touch-sensitive controls (e.g., touch pads, touch screens, etc.), and other known input devices. The user interface 924 may include one or more output devices, such as a display 938 configured to display an image (e.g., a fused image of real-time data and preoperative data). Another function that can be controlled by the transmit controller 920 is the direction of beam steering. The beam can be steered straight ahead (orthogonal) from the transducer array 914, or steered at different angles for a wider field of view. The partial beamformed signal produced by the microbeamformer 916 is coupled to a main beamformer 922 where the partial beamformed signals from the individual patches of transducer elements are combined into a fully beamformed signal. The beamformed signal is coupled to a signal processor 926.

[0059] The signal processor 926 can process the received echo signals in various ways, such as by bandpass filtering, extraction, I and Q component separation, and harmonic signal separation. The signal processor 926 can also perform additional signal enhancement, such as speckle reduction, signal compounding, and noise elimination. The processed signal can be coupled to the B-mode processor 928 to generate B-mode image data. The B-mode processor 928 can use amplitude detection to image structures in the body. The signal generated by the B-mode processor 928 can be coupled to the scan converter 930 and the multi-plane reformatter 932. The scan converter 930 is configured to arrange the echo signals in the spatial relationship from which they are received in the desired image format. For example, the scan converter 930 can arrange the echo signals into a two-dimensional (2D) fan format, or a three-dimensional (3D) format of a pyramid or other shape. The multi-plane reformatter 932 can convert the echoes received from points in a common plane in a volumetric region of the body into an ultrasound image (e.g., a B-mode image) of the plane, such as described in U.S. Pat. No. 6,443,896 (Detmer). When , the volume renderer 934 can generate an image of the 3D data set as viewed from a given reference point, for example, as described in US Pat. No. 6,530,885 (Entrekin et al.).

[0060] The system 910 may also include a Doppler processor 960. The signal from the signal processor 926 may be coupled to the Doppler processor 960, which may be configured to estimate the Doppler shift and generate Doppler image data. The Doppler image data may include color data, which may be overlapped with the B-mode (or grayscale) image data for display. The Doppler processor 960 may be configured to estimate velocity and power according to known techniques. For example, the Doppler processor may include a Doppler estimator, such as an autocorrelator, in which the velocity (Doppler frequency) estimate is based on the independent variable of the lag one autocorrelation function, and the Doppler power estimate is based on the magnitude of the lag zero autocorrelation function. Motion may also be estimated by known phase domain (e.g., parameter frequency estimators such as MUSIC, ESPRIT, etc.) or time domain (e.g., cross-correlation) signal processing techniques. Instead of or in addition to the velocity estimator, other estimators related to the temporal or spatial distribution of velocity (such as estimators of acceleration or temporal and / or spatial velocity derivatives) may be used.

[0061] The output (e.g., images) from the scan converter 930, the multi-planar reformatter 932, and / or the volume renderer 934 may be coupled to an image processor 936 for further enhancement, buffering, and temporary storage before being displayed on an image display 938. In some embodiments, for example, when performing image fusion of 2D real-time ultrasound data with preoperative image data, the system may include or be communicatively coupled to a preoperative data source 968. The 2D images from the scan converter 930 may first pass through a registration and fusion processor 964, which may be configured to correct motion-induced misalignments in real time before the combined images are fused and sent downstream to, for example, an image processor and / or a graphics processor. The graphics processor 940 may generate graphic overlays for display with the images. These graphic overlays may contain, for example, standard identification information, such as the patient's name, the date and time of the image, imaging parameters, etc. For these purposes, the graphics processor may be configured to receive input from the user interface 924, such as a typed patient name or other annotations. In some embodiments, the system 100 may be configured to receive user input via the user interface 924 for setting parameters of the algorithms described herein. In some embodiments, rather than the specific functions described with reference to each of these components being performed by a discrete processing unit, one or more functions of at least one of the graphics processor, image processor, volume renderer, and multiplanar reformatter may be combined into an integrated image processing circuit (the operation of which may be divided among multiple processors operating in parallel). In addition, although the processing of echo signals, for example, for the purpose of generating a B-mode image or a Doppler image, is discussed with reference to a B-mode processor and a Doppler processor, it will be understood that the functions of these processors may be integrated into a single processor.

[0062] Fig.10A flow chart of a method 1000 according to the principles of the present disclosure is shown. The method may include receiving real-time ultrasound image data as shown in the box and preoperative image data (e.g., CT or MRI data) as shown in box 1010. A real-time ultrasound image may be generated for each incoming frame and spatially registered with the preoperative data for image fusion. Spatial registration according to the principles of the present disclosure may involve spatially aligning the real-time image with the preoperative data based on position data, and also correcting for motion-induced misalignment. To this end, the method 1000 may include deriving a motion model based on the real-time ultrasound image data, as shown in box 1012. The motion model may be used to correct for motion-induced misalignment by allowing the system to automatically select motion-compensated image planes. The method may also involve dynamically updating the motion model, for example, to maintain synchronization between the motion model and the organ motion, as shown in box 1014. As shown in box 1016, the motion model 1016 may be applied to the preoperative data to identify motion-compensated preoperative slices to be fused with the real-time image data, as shown in box 1018, and then the combined image may be provided on a display 1020. Methods according to examples herein may include any combination of steps or functions associated with the execution of the algorithms described herein.

[0063] In view of the present disclosure, it should be noted that the various methods and devices described herein can be implemented in hardware, software and firmware. In addition, various methods and parameters are included only by example without any limiting meaning. In view of the present disclosure, those of ordinary skill in the art can implement this teaching when determining their own technology and the required equipment affecting these technologies, while still within the scope of the present disclosure. The functions of one or more of the processors described herein can be incorporated into a smaller number or a single processing unit (e.g., CPU or GPU), or alternatively, it can be distributed in the middle of a larger number of processing units, and can be implemented using an application-specific integrated circuit (ASIC) or a general-purpose processing circuit, which is programmed to perform the functions described herein in response to executable instructions. Computer programs (e.g., executable instructions) can be stored / distributed on any suitable computer-readable medium, such as optical storage media or solid-state media provided together with other hardware or as part of other hardware, but computer programs can also be distributed in other forms, such as distributed via the Internet or other wired or wireless telecommunications systems.

[0064] It will be understood that, depending on the present system, device and method, any of the examples, embodiments or processes described herein may be combined with one or more other examples, embodiments and / or processes, or may be separated and / or performed in the middle of a separate device or device portion. Finally, the above discussion is intended only to illustrate the present system and should not be interpreted as limiting the claims to any particular embodiment or group of embodiments. Therefore, although the present system has been described in detail with reference to the exemplary embodiments, it should be appreciated that a person of ordinary skill in the art may design many modifications and alternative embodiments without departing from the broader and intended spirit and scope of the present system as set forth in the claims. Therefore, the specification and drawings are to be viewed in an illustrative manner and are not intended to limit the scope of the claims.

Claims

1. An ultrasound imaging system operatively associated with an ultrasound probe for imaging an object in real time and with a position tracking sensor connected to the probe, the system comprising: a processor communicatively coupled to the ultrasound probe and to a source of previously acquired image data, wherein the previously acquired image data comprises a 3D data set corresponding to an imaging volume of the object, and wherein the processor is configured to: receiving real-time ultrasound image data; generating a real-time ultrasound image based on a current image frame from the real-time ultrasound data; deriving a motion model based only on the real-time ultrasound image data; resynchronizing the motion model with movement of anatomical features of the subject; identifying a motion-corrected image plane in the imaging volume based on position information from the position tracking sensor and the motion model to generate a motion-compensated pre-operative image; and The real-time ultrasound image is fused with the motion-compensated pre-operative image to produce a combined image.

2. The ultrasound imaging system according to claim 1, wherein: The motion model is defined at least in part based on a periodic frequency, phase, and amplitude of motion of an anatomical feature in the imaging volume, and wherein the processor is configured to determine the periodic frequency, the phase, the amplitude, or a combination thereof based on real-time ultrasound image data received over a time period greater than one motion cycle.

3. The ultrasonic imaging system according to claim 1 or 2, wherein: The processor is configured to: generating long term correlation (LTC) data for each of a plurality of incoming frames of real-time ultrasound image data; and The periodic frequency and the phase associated with each incoming frame are determined by identifying at least two local minima of an LTC curve corresponding to the LTC data associated with each incoming frame.

4. The ultrasound imaging system according to claim 3, wherein: The processor is further configured to perform error minimization on the LTC data to dynamically resynchronize the motion model.

5. The ultrasound imaging system according to any one of claims 1, 2 and 4, wherein: The processor is configured to determine whether the ultrasound probe is stationary based on the position information from the position tracking sensor, and dynamically update the motion model only when the ultrasound probe is determined to be stationary.

6. The ultrasound imaging system according to any one of claims 1, 2 and 4, wherein: The processor is configured to derive the motion model by performing motion estimation on successive image frames received in real time over a period equal to or greater than at least two motion cycles.

7. The ultrasound imaging system according to any one of claims 1, 2 and 4, wherein: The processor is further configured to receive user input indicating a start and an end of a synchronization period, and wherein the processor is configured to perform motion estimation on image frames received within the synchronization period.

8. The ultrasound imaging system according to any one of claims 1, 2 and 4, wherein: The motion model is configured to estimate, as a function of time, a respiratory-induced motion of an anatomical feature along a craniocaudal direction of the subject based on a respiratory rate, a phase, and an amplitude of the subject's respiratory motion.

9. The ultrasound imaging system according to any one of claims 1, 2 and 4, wherein: The probe is a 3D probe, wherein the processor is configured to automatically determine a craniocaudal direction of the subject and cause a beam of the 3D probe to be steered such that the beam is aligned with the craniocaudal direction.

10. The ultrasound imaging system according to any one of claims 1, 2 and 4, wherein: The previously acquired image data comprises a 3D data set acquired using magnetic resonance (MR), computed tomography (CT) or ultrasound (US) imaging.

11. The ultrasound imaging system according to any one of claims 1, 2 and 4, further comprising a display unit, and wherein: The processor is configured to automatically register each incoming frame of real-time ultrasound image data with the previously acquired image data based on the position information and the motion model, and cause the combined image associated with each incoming frame to be displayed on the display unit in real time.

12. An imaging method comprising: receiving real-time ultrasound image data in response to ultrasound waves transmitted toward a subject by a probe, wherein the probe is associated with a position tracking sensor; generating a real-time ultrasound image based on a current image frame from the real-time ultrasound data; deriving a motion model based only on the real-time ultrasound image data; resynchronizing the motion model with movement of anatomical features of the subject; accessing a previously acquired image dataset corresponding to an imaging volume of the object, identifying a motion-corrected image plane of the imaging volume based on position information from the position tracking sensor and the motion model to generate a motion-compensated pre-operative image; and The real-time ultrasound image is fused with the motion-compensated pre-operative image to produce a combined image.

13. The method according to claim 12, wherein: The motion model is defined based at least in part on a periodic frequency, phase, and amplitude of motion of an anatomical feature in the imaging volume, wherein deriving the motion model based on the real-time ultrasound image data comprises determining the cycle frequency, the phase, the amplitude, or a combination thereof based on the real-time ultrasound image data received within a time period greater than one motion cycle, and Wherein, the method further comprises: generating long term correlation (LTC) data for each of a plurality of incoming frames of real-time ultrasound image data; The cycle frequency and the phase associated with each incoming frame are determined based on a local minimum of an LTC curve corresponding to the LTC data associated with each incoming frame.

14. The method according to claim 13, in, The method further comprises: performing error minimization on the LTC data to dynamically resynchronize the motion model; and Movement of the probe is detected and resynchronization of the motion model is suspended in response to the detected movement of the probe.

15. The method of any one of claims 12 to 14, further comprising maintaining registration between the real-time image frames and the previously acquired image data during a period associated with movement of the probe based on a version of the motion model prior to the detected movement of the probe.

16. A non-transitory computer readable medium comprising processor-executable instructions which, when executed by one or more processors of a medical imaging system, cause the one or more processors to perform the method of any one of claims 12 to 15.

Images