Method for providing a secondary medical imaging source

Abstract

A method provides a source of secondary imaging of an anatomical region of interest for use during a medical procedure. The method is based on a set of reference images in a secondary imaging modality of an anatomical region being acquired in advance of movement of a subject or body within the anatomical region. Different movement positions of the subject within the reference images are indexed via measurement of displacement of a predefined reference point fixed relative to the anatomical structure of interest. This is measured using an ultrasound imaging probe which simultaneously images the anatomical region with the secondary imaging modality. The detected reference point position of each reference image is recorded with each image frame. During the intervention, the ultrasound image probe is again used, preferably set in the same position relative to the anatomical structure, and images of the same anatomical structure are captured during the procedure. The reference images can be called up in real time based on measuring the reference point displacement using the ultrasound images and querying the associated data set of images. This can be displayed with each ultrasound image frame, or in place of each ultrasound image frame.

Description

Technical Field

[0001] The present invention relates to a method for providing a secondary medical imaging source of anatomical structures, particularly a method for use during medical procedures. Background Technology

[0002] Medical imaging can be performed during medical procedures to provide visual guidance to clinicians. For example, it can be performed during interventional procedures to provide visualization of the anatomical structures of the object being intervened upon (e.g., areas containing lesions or tumors (or other objects of interest)).

[0003] Medical imaging can be provided in real time. Ideally, the imaging offers high spatial and temporal resolution. This is particularly important when imaging anatomical regions that are moving within the body due to natural movements or cycles (such as the movement of the lungs, liver, or heart). Here, the anatomical structures undergoing intervention may move continuously, and therefore high temporal resolution imaging is desirable.

[0004] High spatial resolution provides a higher level of spatial detail. This is especially important for interventions that require precise interaction with anatomical features.

[0005] Sufficiently high spatial resolution can be achieved using imaging modalities such as MRI, CT, and C-arm fluoroscopy. However, MRI and CT scanners present challenges for real-time imaging due to their size and limited accessibility for patient intervention procedures. C-arm fluoroscopy suffers from the drawback that it emits harmful radiation and is therefore undesirable as a choice for continuous imaging during relatively long procedures.

[0006] In contrast, ultrasound imaging is compact, non-ionizing, and has higher temporal resolution. However, the spatial resolution of ultrasound images is rather low.

[0007] Therefore, it would be desirable to have an imaging method that can provide both high spatial and temporal resolution imaging of the region of interest, while also being compact and non-ionizing.

[0008] Multimodal imaging is another example of a method that provides live imaging during the procedure. This involves a combination of imaging modalities. Examples include a combination of PET and CT scans for accurate tumor localization. However, in terms of large device size, this still suffers from the same drawbacks as CT alone.

[0009] Another example involves combining live fluoroscopy (C-arm device) with annotated previous CT scans to locate and treat regions of interest.

[0010] However, registration of live images with previous CT or MRI scans is complex and computationally demanding. It is often inaccurate due to the application of sophisticated image processing algorithms, which can lead to incorrect registrations. It is also typically based on rigid structures, such as vertebral columns, to provide landmarks for registration. Therefore, it is less useful for purely flexible body structures. Furthermore, live registration of moving structures based on pre-recorded image data and live data is even more complex.

[0011] Therefore, there is still a need for an imaging method that can provide both high spatial and temporal resolution imaging of the region of interest, while also being compact and non-ionizing, and without requiring a complex real-time image registration process. Summary of the Invention

[0012] This invention is defined by the claims.

[0013] According to an example of one aspect of the invention, a method is provided for providing a source of secondary imaging of a patient's moving anatomical target of interest, the method comprising:

[0014] Receive a series of ultrasound images representing a region containing the anatomical target, the images corresponding to different time points within an imaging period;

[0015] For each ultrasound image, the displacement of a predefined reference point within a reference frame fixed with respect to the ultrasound imaging field of view is determined, the predefined reference point being fixed relative to the anatomical target;

[0016] Access includes a dataset comprising a set of reference images representing the anatomical target at different moving positions, the reference images corresponding to imaging modalities different from the ultrasound images, and wherein each reference image in the dataset is associated with the displacement of a corresponding record of the reference point on the object within a reference frame fixed with respect to the reference imaging field of view;

[0017] For each received ultrasound image, at least one reference image is selected based on the displacement of the reference point in the determined received image; and

[0018] For each received ultrasound image, an output representing a selected reference image is generated, which provides a source for secondary imaging.

[0019] The reference image preferably has a higher spatial resolution than the received ultrasound image.

[0020] An embodiment of the present invention involves pre-acquiring a set of images of the anatomical target within a time window during the object's movement using a secondary (preferably higher spatial resolution) imaging modality. Simultaneously with high-quality imaging acquisition, ultrasound images of the same anatomical region are also acquired, and the displacement of a reference point with a fixed position relative to the imaging anatomical structure is tracked within the ultrasound images. For each acquired high-resolution image, the displacement of the reference point in the ultrasound image is recorded, thereby forming a dataset comprising high-resolution images of the anatomical target at different movement positions, with each image associated with the displacement of the reference point in the ultrasound image.

[0021] The above content refers to the setup, calibration, or data acquisition phase.

[0022] Subsequently, during the retrieval phase (e.g., during the interventional procedure), the high-resolution imaging modality is operated discontinuously. Alternatively, ultrasound images are acquired continuously or cyclically, and for each acquired image, the displacement of the reference point is identified, and the associated high-resolution reference image is retrieved from the dataset and displayed to the clinician.

[0023] This method avoids the need for real-time registration between real-time ultrasound images and stored reference images. Instead, registration is achieved indirectly using the displacement of a reference point. Spatially registered secondary images can be identified based on a single measurement variable of the reference point displacement.

[0024] During the retrieval phase, the ultrasound probe can be configured to have the same physical orientation (position and orientation) relative to the patient's anatomy as it does during the acquisition phase, or a known mapping can exist between the probe's field-of-view reference frame during the retrieval phase and the field-of-view reference frame during the setup phase. In this way, the measured displacement of the reference point in the ultrasound image acquired during the recall phase will be registered with the recorded displacement relative to the reference image.

[0025] The "reference frame fixed relative to the reference imaging field of view" mentioned above can be the same as the "reference frame fixed relative to the ultrasound imaging field of view". It can be defined by a coordinate system that is the same as the coordinate system of the ultrasound imaging field of view, or it can be defined by a coordinate system that is different from the ultrasound imaging coordinate system but has a known mapping or registration with the ultrasound imaging coordinate system.

[0026] It should be noted that although a reference point is mentioned, this can refer to a single point or line (e.g., the edge of an anatomical structure, such as an organ), or a region or volume. Therefore, references to reference points throughout this disclosure should be understood as references to any of the points, lines, regions, or volumes.

[0027] Ultrasound imaging can be 2D or 3D. Secondary imaging modalities can be 2D or 3D.

[0028] The method may further include displaying each selected reference image on a display device. Optionally, the method may include displaying each selected reference image simultaneously with the corresponding received ultrasound image.

[0029] If the method is performed in real time during the interventional procedure, it allows clinicians to derive the benefits of both real-time live imaging of the anatomical region (based on ultrasound images) and images of the region acquired in advance (preferably with higher spatial resolution).

[0030] According to one or more embodiments, ultrasound images can be received in real time during the interventional procedure. For example, images can be received from an ultrasound imaging transducer unit (e.g., a probe) or an ultrasound imaging device coupled to the ultrasound transducer unit.

[0031] The anatomical target may be a periodically moving anatomical target. The imaging period may correspond to one or more movement cycles of the anatomical target.

[0032] The stored dataset may include a set of reference images corresponding to a series of time points across a complete movement cycle of the anatomical target.

[0033] Therefore, in this set of embodiments, the reference dataset effectively forms a calibrated reference image sequence of the complete movement cycle of the anatomical target.

[0034] As discussed, the dataset of reference images may be images previously acquired for the same anatomical target during the calibration or acquisition phase, and wherein the associated displacement of the reference point for each reference image is the displacement of a reference frame fixed relative to the ultrasound imaging field of view, recorded in simultaneously acquired ultrasound images.

[0035] The reference frame used to record displacement during the calibration phase can be the same as the reference frame for the ultrasound imaging field of view used during the retrieval phase. The ultrasound imaging field of view can have an associated coordinate system. The determined displacement of the reference point in each received ultrasound image can be represented with respect to this coordinate system. The recorded reference point displacement for each reference image in the dataset can also be recorded according to the same coordinate system, thus making it easy to find the corresponding reference image for a given measured displacement of the reference point during the retrieval phase.

[0036] Alternatively, mapping or transformation can be applied between the ultrasonic field coordinate systems during the calibration and retrieval phases.

[0037] The ultrasound image may be received from an ultrasound imaging device that includes an ultrasound transducer unit (e.g., a probe) in a fixed orientation relative to the patient's anatomy. The ultrasound transducer unit may be held by a support frame that allows for adjustment of the probe's orientation and releasable fixation.

[0038] In some embodiments, the support frame may be adapted to generate data output including an attitude indicator that indicates the current attitude position of the frame (or the probe held within the frame). This allows, for example, the coordinate system of an ultrasound imaging probe to be precisely tracked and controlled, thereby allowing registration (if necessary) to be performed between the displacement of a reference point recorded in a reference dataset and the displacement measured during the retrieval phase. Alternatively, it can be used to ensure ultrasound field-of-view matching in each phase of the calibration and retrieval phases. In some examples, it can also be used to register the coordinate systems of ultrasound imaging and secondary imaging modes.

[0039] According to one or more embodiments, the method may further include generating an interpolated reference image in response to detecting that the displacement of the determined reference point in the received image does not match the recorded reference point displacement of any of the reference images in the dataset and is between the recorded reference point displacements of a first reference image and a second reference image in the reference images. In this case, the interpolated reference image may be generated based on interpolation between the first reference image and the second reference image in the reference images.

[0040] This provides the technical benefit of further increasing the temporal resolution of the source for secondary imaging, as secondary images can be output at every possible location of movement of the anatomical structure, even if the dataset lacks pre-stored images corresponding to a specific movement location.

[0041] Another aspect of the present invention provides a method for performing the above-described calibration or acquisition phase.

[0042] In particular, another aspect of the present invention provides a method for generating a dataset of reference images of a moving anatomical target as a source for providing secondary imaging. The method includes acquiring a series of ultrasound images of an anatomical region containing the same anatomical target during the imaging period using a first imaging modality. The method further includes acquiring a series of ultrasound images of the region containing the same anatomical target simultaneously with the acquisition of the series of first images, wherein the first imaging modality differs from the imaging modality of the ultrasound images. The method further includes determining, for each ultrasound image, the displacement of a predefined reference point fixed relative to the anatomical target within a reference frame fixed with respect to the ultrasound imaging field of view. The method further includes storing, in the dataset, a representation of the first image and a record associated with the determined displacement of the reference point within an ultrasound image acquired simultaneously with the first image, thereby generating a reference imaging dataset for providing a source for secondary imaging.

[0043] Therefore, this method defines a calibration or setup phase for the dataset in which reference images are acquired. The "first image" in this method corresponds to the reference image in the previously defined retrieval phase.

[0044] The method may further include the step of registering the coordinate system of the ultrasound imaging reference frame with the coordinate system of the secondary imaging modality reference frame. Registration in this context means that the ultrasound image and the secondary image are placed in the same geometric reference frame (coordinate system). This can be useful for visualization purposes, such that the ultrasound image and the reference image correspond to the same region of an anatomical structure. For example, this means that, according to certain embodiments, two image sets can be viewed together at the same size, orientation, and spatial location, and can even be visually superimposed (fused).

[0045] According to one or more embodiments, the anatomical target being imaged may be a periodically moving anatomical target, wherein the imaging period corresponds to one or more movement cycles of the anatomical target. For example, a dataset of stored first images (reference images) may include a set of the first images corresponding to a series of time points spanning a complete movement cycle of the anatomical target.

[0046] This method may include acquiring image data over multiple movement cycles of the anatomical target and generating an average or aggregated dataset for a single movement cycle based on the image data from the multiple movement cycles. For example, within each movement cycle, a secondary imaging modality may capture image frames corresponding to slightly different time points or stages within the cycle compared to previous imaging cycles. Therefore, a more comprehensive secondary image dataset (with more time points) can be compiled over a set of multiple cycles.

[0047] According to one or more embodiments, the method of the present invention may include performing a calibration or setup phase according to any example or embodiment outlined above or described below, or according to any claim of this application, and may further include performing a retrieval phase after the calibration phase. The retrieval phase may be performed according to any example or embodiment outlined above or described below, or according to any claim of this application. The reference imaging dataset generated in the calibration or setup phase is used as the dataset of the reference images in the retrieval phase.

[0048] For example, in a favorable case, two phases are performed during the same single intervention process.

[0049] An example of another aspect of the invention provides a processing apparatus arranged to provide a source of secondary imaging of a patient's moving anatomical target of interest. The processing apparatus is adapted to receive a series of ultrasound images representing a region containing the anatomical target, the images corresponding to different time points within an imaging period. The processing apparatus is further adapted to determine, for each ultrasound image, the displacement of a predefined reference point fixed with respect to the anatomical target within a reference frame fixed with respect to the ultrasound imaging field of view. The processing apparatus is further adapted to access a dataset comprising a set of reference images representing the anatomical target at different moving positions, the reference images having different imaging modalities than the ultrasound images, and wherein each reference image is associated in the dataset with a corresponding recorded displacement of the reference point on the object within a reference frame fixed with respect to the reference imaging field of view. The processing apparatus is further adapted to, for each received ultrasound image, select at least one reference image from the reference images based on the determined displacement of the reference point in the received image. The processing apparatus is further adapted to generate, for each received ultrasound image, an output representing the selected reference image, the output providing a source of secondary imaging.

[0050] In other words, the processing apparatus is adapted to perform the retrieval phase of a secondary imaging method according to any example or embodiment outlined above or described below, or according to any claim of this application.

[0051] Optionally, the processing apparatus may further include a data storage device for storing the dataset containing reference images. Alternatively, the data storage device may be external to the processing apparatus. The processing apparatus may include one or more processing modules that perform processing functions, and wherein the one or more processing modules include input / output ports for communicating with the data storage device to access the dataset.

[0052] Another aspect of the present invention may provide a processing apparatus adapted to perform a setup or calibration phase of a method according to any example or embodiment outlined above or described below, or according to any claim of this application.

[0053] A processing apparatus suitable for performing the method in two phases can be provided: a setup phase followed by a retrieval phase. The processing apparatus can be adapted to perform the setup phase in response to receiving an input trigger signal, which may be received, for example, from a user interface (e.g., a clinician triggering the execution or re-execution of the setup phase at a selected time during the interventional procedure).

[0054] An example of another aspect of the invention provides a system. The system includes processing means according to any example or embodiment outlined above or described below, or according to any claim of this application. The system also includes an ultrasound imaging device comprising an ultrasound transducer unit, the imaging device being configured to acquire ultrasound images of the anatomical body and to transmit the acquired ultrasound images to the control unit. The ultrasound transducer unit may be, for example, an ultrasound probe.

[0055] The system may further include a support frame arranged to releasably secure the ultrasound transducer unit relative to the patient's anatomy in a modulated posture. The support frame may be adapted to generate data output including a posture indicator indicating the current posture position of the frame.

[0056] This allows the coordinate system of, for example, an ultrasound imaging probe to be precisely tracked and controlled, thereby allowing registration to be performed between the anatomical displacements recorded in the reference dataset and the displacements measured during the recall phase.

[0057] These and other aspects of the invention will become apparent with reference to one or more embodiments described below, and will be illustrated with reference to one or more embodiments described below. Attached Figure Description

[0058] To better understand the invention and to more clearly illustrate how it can be implemented, reference will now be made to the accompanying drawings by way of example only, wherein:

[0059] Figure 1 The steps of an example method according to one or more embodiments of the first aspect of the present invention are shown;

[0060] Figure 2 An example workflow is illustrated according to one or more embodiments of the present invention;

[0061] Figure 3 An example reference imaging dataset is illustrated schematically;

[0062] Figure 4 The steps of an example method according to one or more embodiments of the second aspect of the present invention are shown;

[0063] Figure 5 An example workflow is illustrated according to one or more embodiments of the second aspect of the present invention;

[0064] Figure 6 An example reference imaging dataset is illustrated schematically for an imaging period covering the complete movement cycle of an object with periodic movement.

[0065] Figure 7 The formation of the assembly dataset is illustrated schematically, which includes reference images captured at different phase points in each of two different movement cycles of a periodically moving object;

[0066] Figure 8 The diagram schematically illustrates the interpolation of a secondary image for the position of the object being imaged, which is the midpoint between two recorded reference images.

[0067] Figure 9 The steps within an example method for detecting and tracking displacement of a reference line within a received ultrasound image are illustrated schematically.

[0068] Figure 10 The illustration schematically depicts the use of a reference line for detecting and tracking displacement within a received ultrasound image. Figure 9 Another step within the example method; and

[0069] Figure 11 The illustration shows the effects of different stages within a step of an example method for detecting and tracking the displacement of a reference line within a received ultrasound image. Detailed Implementation

[0070] The invention will be described with reference to the accompanying drawings.

[0071] It should be understood that the detailed descriptions and specific examples, while indicating exemplary embodiments of the apparatus, system, and method, are intended for illustrative purposes only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, system, and method of the invention will be better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are merely schematic and not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to denote the same or similar parts.

[0072] This invention provides a method for providing a source of secondary imaging of an anatomical region of interest for use during a medical procedure. The method is based on a set of reference images acquired in advance in a secondary imaging modality of the anatomical region during movement of an object or body within the region. Different positions of movement of the object within the reference images are indexed by measuring the displacement of a predefined reference point fixed relative to the anatomical structure of interest. This is measured using an ultrasound imaging probe that simultaneously images the anatomical region using the secondary imaging modality. The detected reference point position is recorded with each image frame. During the intervention, the ultrasound imaging probe is used again, preferably positioned at the same location relative to the anatomical structure, and images of the same anatomical structure are captured during the procedure. The reference images can be recalled in real time based on measuring the reference point displacement using ultrasound images and querying a dataset of associated images. This can be displayed with each ultrasound image frame or instead of each ultrasound image frame.

[0073] The overall approach can be divided into two main phases: a setup or calibration phase and a recall or retrieval phase. In the setup or calibration phase, a reference image dataset is acquired, and in the recall or retrieval phase, reference images are retrieved based on real-time ultrasound measurements of the location of anatomical structures.

[0074] Figure 1 The basic steps of a method for performing a retrieval phase according to a first aspect of the present invention are summarized in block diagram form. (Refer to...) Figure 2 Before describing each step in more detail in the exemplary workflow shown, these steps will first be outlined in general terms.

[0075] This method is used to provide a source for secondary imaging of a moving anatomical target of interest 22 in patient 8.

[0076] The method includes receiving 12 a series of ultrasound images 44 representing a region containing an anatomical target, the images corresponding to different time points within an imaging period.

[0077] The method also includes determining, for each ultrasound image, the displacement 54 of a predefined reference point 24 on or at a fixed position relative to the anatomical target 22 within a reference frame fixed relative to the ultrasound imaging field of view.

[0078] The method also includes accessing a dataset 60 comprising a set of reference images 62 representing anatomical targets at different movement positions, the reference images corresponding to images captured using an imaging modality different from ultrasound images. Each reference image in the dataset is associated with a corresponding recorded displacement of the reference point on the object within a reference frame fixed relative to the reference imaging field of view.

[0079] The method further includes, for each received ultrasound image 44, selecting at least one of 18 reference images 62 based on a determined displacement of a reference point in the received image. The method also includes, for each received ultrasound image, generating 20 an output 64 representing the selected reference image, the output providing a source for secondary imaging.

[0080] Figure 2 An example workflow of a method according to one or more embodiments is illustrated in more detail. Exemplary components of a system that can be used to implement the method are also illustrated schematically.

[0081] Method 10 can be advantageously performed in real time during medical procedures (e.g., interventional medical procedures). Non-limiting examples include minimally invasive image-guided interventional procedures, such as, for example, biopsies of organs in the abdomen. Figure 2 The illustration depicts a patient 8 during an interventional procedure, in which surgical instruments 40 are inserted into the patient's body. The procedure involves an anatomical target 22 of interest. Non-limiting examples include the liver, lungs, kidneys, stomach, heart, or any other organ or internal structure.

[0082] During this procedure, an ultrasound imaging apparatus is used to perform real-time ultrasound imaging, the apparatus comprising an ultrasound transducer unit, in this example, in the form of an ultrasound probe. The ultrasound transducer unit 32 is held in a fixed posture relative to the patient's anatomy. The posture defines, for example, the position and orientation of the probe relative to the patient's anatomy. The ultrasound transducer unit 32 has an imaging field of view 34. The field of view covers an anatomical region within the patient 8, which includes the object of interest 22. The field of view also covers a fixed reference point 24 on the object of interest, or at an anatomical point fixed relative to the object of interest. The reference point should be physically coupled to the object in the sense that it follows or reflects any movement of the object of interest, so that it can be used as a marker to track the movement of the object.

[0083] Optionally, the ultrasound transducer unit is held by a support frame 36, thereby allowing for adjustment of the orientation of the ultrasound transducer unit 32 and releasable fixation. This orientation partially defines the field of view 34 of the transducer unit 32 relative to the patient's anatomy.

[0084] The ultrasonic transducer unit 32 can be operatively connected to the ultrasonic processing module or unit 42, which also forms part of the ultrasonic imaging apparatus. This allows control of the drive of the ultrasonic transducer in the transducer unit 32. For example, it can control the acquisition and reception settings of the probe. It can define or configure the ultrasonic imaging field of view 34. The ultrasonic processing unit 42 can additionally receive and process the ultrasonic data 38 generated by the ultrasonic probe. This can be received, for example, at the ultrasonic processing unit 42 as echo data (RF data), or in some examples as partial beamforming data. In either case, the ultrasonic processing unit performs ultrasonic processing to derive ultrasonic imaging data, such as B-mode ultrasonic image data of the area covered by the field of view 34. This can take the form of data representing a series of ultrasonic images (image frames), each representing a different time point in the process of any imaging period. These can be generated sequentially, one after another, and in real time with the acquisition of ultrasonic echo data by the transducer unit 32.

[0085] Ultrasonic imaging devices 32, 42 may be included as part of a system provided in one aspect of the invention, or may be external or auxiliary devices, wherein components of the system or device provided by the invention are adapted to communicate with the external or auxiliary device during use.

[0086] The processing unit 50 is arranged to receive a series of ultrasound images (ultrasound image data) 44 output by the ultrasound imaging device. For example, the processing unit may include an input / output terminal or a communication module (not shown) for receiving input data.

[0087] The processing device 50 is adapted to process each ultrasound image in a series of ultrasound images in real time, for example, processing each ultrasound image sequentially, in order to determine the displacement of a predefined reference point 24 within a reference frame fixed relative to the ultrasound imaging field of view 34. An image processing module 52 may be included for this function.

[0088] Although a reference point is used, it can refer to a single point or line, such as the edge of an anatomical structure (like an organ). It can also, or alternatively, represent a volume or area. The reference point, line, volume, or region (whose location will be detected) can be user-defined, for example, in a configuration step prior to the acquisition or retrieval phase. This can be based on using a user interface to identify points, lines, volumes, or regions on sample ultrasound images of anatomical areas. This can then be used to indicate which point, line, region, or volume should be tracked. Alternatively, the points, lines, regions, or volumes to be tracked can be preset in advance. For brevity, the term "reference point" should be referenced elsewhere in this disclosure. This should be understood to encompass any of the points, lines, regions, or volumes as explained above.

[0089] "Displacement" refers to a position within a reference frame, for example, fixed relative to the ultrasound imaging field of view 34. For example, a fixed coordinate system can be defined (e.g., in advance) relative to this reference frame, and the determined displacement is the position within that coordinate system. This can be a one-dimensional, two-dimensional, or three-dimensional coordinate position, depending in part on the dimension of the ultrasound imaging. More than one probe 32 can be used to achieve multidimensional reference point tracking and / or volumetric ultrasound imaging can be performed. Since the ultrasound transducer unit 32 is fixed relative to the surface where the patient is placed (e.g., relative to the operating room), the reference frame is also fixed relative to that surface. As an example, this step may include determining the position coordinate identifier (e.g., vector) P of the reference point 24 for each image frame within a predefined coordinate system fixed relative to the ultrasound imaging field of view.

[0090] One or more image processing algorithms can be used to determine the location of reference points within each image frame. These algorithms can employ techniques such as shape recognition or edge detection to identify relevant reference points or lines. This can be based on segmentation applied to each image frame to aid in the identification of relevant anatomical points or lines.

[0091] Once relevant points have been identified in the ultrasound image frame, displacement indications, such as a set of position coordinates or a vector, can be generated. This can be based on mapping the ultrasound imaging field of view 34 to a predefined reference point coordinate system (discussed above). The coordinate system can be defined relative to the field of view such that, based on the definition of the coordinate system, the mapping is inherently known. Alternatively, the mapping can be determined or defined based on the position or orientation of the probe relative to the coordinate system. Those skilled in the art will recognize the standard mathematical techniques used to achieve this.

[0092] A particularly advantageous method for performing the detection and location tracking of reference points is based on an initial user input indicating the area within the ultrasound field of view where one or more internal reference points or lines (e.g., the edges of organs) lie in an ultrasound image frame. This simplifies the image processing algorithms required to identify and track reference points (or lines, regions, or volumes) within an ultrasound image stream. This method will be described in detail later in this disclosure.

[0093] Once the displacement 54 of the reference point has been determined for a given ultrasound image frame, it is used to query the reference image data storage device 60, which stores a dataset of reference images 62 representing the anatomical target 22 at different moving positions, wherein each image is associated or linked with the displacement of the corresponding record of the reference point 24 within a reference frame fixed relative to the reference imaging field of view.

[0094] The process of query 58 then involves selecting a reference image associated with a reference point displacement that is the same as or nearly the same as the reference point displacement in the ultrasound image. For this purpose, it is important that there is a correspondence between the metrics (e.g., coordinate systems) representing the newly measured reference point displacement 54 and the recorded reference point displacement (in data storage 60). For example, the coordinate systems representing the displacement should match.

[0095] For this purpose, optionally, a measurement or coordinate system registration 56 or transformation step may be included. As an example, the reference image data storage device 60 may store coordinate system identifiers and a dataset of reference images, thereby allowing the detection of the coordinate system of the location of reference point 24 in the recorded dataset. Coordinate registration can be performed based on this. In some examples, the attitude information of the ultrasound probe can be obtained and used for coordinate system transformation, for example, based on the reference probe attitude information stored in the data storage device. Alternatively or additionally, coordinate system identifier information from the image processing unit 52 may be used.

[0096] Alternatively, for example, if the reference dataset 60 was acquired only in an acquisition phase performed recently prior to the retrieval phase, the coordinate system can be inherently matched with the ultrasound probe, wherein the ultrasound probe is configured in the same pose relative to the anatomical structure as in the current retrieval phase, and wherein the image processing unit 52 uses the same coordinate system definition to determine the reference point displacement.

[0097] Based on the determined displacement 54 of the reference point 24 in the received ultrasound image frame 24, at least one of the reference images 62 in the data storage device is selected. An output representing or based on the selected image is generated.

[0098] Selecting reference image 62 may include identifying reference images with an exact match to the associated reference point displacement 24. Alternatively, if no exact match to the measured reference point displacement can be found, one of the reference images with the closest matching associated reference point displacement may be selected. Alternatively, and as will be discussed in more detail later, in some cases, a new reference image may be generated based on interpolation between two reference images corresponding to the reference point displacement on either side of the measured reference point displacement 54.

[0099] In each case, an output 64 representing the selected or generated reference image 62 is generated. This can be coupled or transmitted to a display device 72 for display to the user. The reference image can be displayed simultaneously with the display of the received ultrasound image 44 corresponding to the reference image. This can be transmitted to the display unit 72 via processing device 50. A display output module may be included, which receives the ultrasound image 44 and the reference image 62 and generates a combined display output for coupling to the display. By way of non-limiting example, this may include only the reference image, or the reference image and the ultrasound image side by side, or the ultrasound image and the reference image overlapping or superimposed (fused). Any other display format may also be used.

[0100] Viewing images from both modalities together can help clinicians notice any differences between the live ultrasound images and the secondary images, which may occur during interventional procedures, for example, when the organ of interest begins to swell due to the procedure. Optionally, when such a change is detected, the calibration or setup phase (described later) can be repeated to reacquire the reference imaging dataset.

[0101] In some examples, coordinate system registration can be performed between the ultrasound imaging field of view or reference frame and the secondary imaging modality before generating the Display 72 output. This can be useful for visualization purposes, ensuring that the ultrasound image and the reference image correspond to the same region of the anatomical structure. For example, this means that two image sets can be viewed together, as described above, and shown at the same size, orientation, spatial location, and optionally superimposed (fused). Optionally, an image fusion step can be performed after coordinate system registration to check the accuracy of the registration. Registration may only need to be performed once per imaging period or session, rather than for each received ultrasound image.

[0102] In some examples, the support frame 36 may be adapted to generate a data output 76 including an attitude indicator that indicates the current attitude position of the frame. This may be based on the use of one or more electromagnetic sensors incorporated in the frame unit, adapted to sense the (3D) attitude of the probe held in the frame. The frame may include a bracket or gripper portion that releasably holds the probe, and wherein the bracket or gripper portion is pivotally movable relative to a support frame portion of the frame, which is attached to a surface fixed relative to the surface of the object being held. The pivoting position of the bracket or gripper portion relative to the support frame portion may be sensed using electromagnetic sensors or any other suitable sensors, such as optical sensors.

[0103] In some examples, the frame can be electronically actuated to move the held ultrasound probe to a defined posture, wherein the posture can be precisely or uniquely identified by an electronic identifier signal or code. This can be used to ensure that the probe is configured in the same posture position as that used to measure the displacement of reference point 24 when acquiring reference images (as will be explained further later).

[0104] The output signal indicating attitude can be additionally or alternatively used to perform coordinate system transformation or registration between the reference frame of the ultrasonic probe and the reference frame of the secondary imaging mode (as described above), or between the probe coordinate system and the coordinate system used when measuring the displacement of the reference point during the probe setup phase.

[0105] As an example, WO2019 / 110115 describes an example frame unit whose position is electronically trackable. The frame is described with reference to an application in a hair-cutting device. However, the same principles can be applied to provide attitude-trackable holding units for ultrasound probes. For example, using electromagnetic sensing to track the position and orientation of the unit in space (i.e., attitude) is a well-known technique, and those skilled in the art will recognize the means used to implement this. Off-the-shelf solutions for electromagnetic attitude tracking are known.

[0106] In some examples, more than one ultrasonic transducer unit 32 can be used to track the reference point. This makes it possible to determine the displacement of the reference point in multiple dimensions.

[0107] It should be noted that the use of an ultrasound probe is not required. Other examples of ultrasound transducer units include the use of an ultrasound monitoring patch that can be reliably adhered to the skin surface and remain in place throughout the procedure.

[0108] Figure 3 A reference imaging dataset stored by a reference data storage device 60 is schematically illustrated. The reference dataset includes multiple reference images 62a-62e. Figure 3 Only five reference images are shown, but typically more than five reference images can be included. Each reference image is associated with a time point in the dataset during the imaging period and with the displacement indication P of reference point 24 at the time the reference image was captured. This is shown in the form of a three-dimensional vector, but it can take any form, including (as a non-limiting example) 1D or 2D vectors, or 1D or 2D coordinate sets.

[0109] As will be explained later, preferably, the imaging dataset corresponds to a series of images covering the periodically moving anatomical region or the movement cycle of the body, and thus the timestamps of the images can correspond to points in time within the movement cycle.

[0110] As discussed above, the reference image 62 of dataset 60 is an image previously acquired for the same anatomical target 22 during the setup or calibration phase, and wherein the associated displacement 54 of the reference point 24 for each reference image is the displacement relative to a reference frame fixed with respect to the ultrasound imaging field of view, recorded in the simultaneously acquired ultrasound image 44.

[0111] Now refer to Figure 4 Describe this setup or calibration phase. Figure 4 The steps of an example method for performing a setup or calibration phase, according to one or more embodiments, are illustrated in block diagram form.

[0112] In particular, one aspect of the invention provides a dataset 60 for generating reference images 62 of a moving anatomical target 22 to provide a source for secondary imaging, a method 100.

[0113] According to one or more embodiments, method 100 includes acquiring a series of first images 62 of an anatomical region containing the anatomical target during an imaging period using a first imaging modality 82.

[0114] Method 100 further includes acquiring a series of ultrasound images 44 of a region containing the same anatomical target during the imaging period, while acquiring a series of first images, wherein the first imaging modality 82 is different from the imaging modality of the ultrasound images.

[0115] Method 100 further includes, for each ultrasound image in the ultrasound images, determining the displacement of a predefined reference point 24 fixed relative to the anatomical target within a reference frame fixed relative to the ultrasound imaging field of view.

[0116] Method 100 further includes, for each image in the first image, storing in a dataset 108 an associated record of the determined displacement of a reference point within an ultrasound image acquired simultaneously with the first image, thereby generating a reference imaging dataset 60 for providing a source for secondary imaging.

[0117] Figure 5 An example workflow for performing the calibration or acquisition phase is outlined. Most of the steps performed in this phase are the same as those in the recall or retrieval phase discussed above, except that secondary imaging is performed simultaneously with the acquisition of ultrasound images using secondary imaging modality 82, and the reference image is not retrieved from the reference image dataset 60, but is stored in the dataset along with the determined displacement of the anatomical reference point 24. Therefore, similar steps or components will not be described in detail here, and the reader should refer to the previous description provided regarding the retrieval phase.

[0118] The ultrasound transducer unit 32 can be positioned relative to the patient's anatomy in the same orientation as in the subsequent retrieval phase. As in the retrieval phase, ultrasound images of the anatomical region containing the object of interest 22 and including the defined reference point 24 are acquired continuously or periodically. The ultrasound data from the probe is processed 42 to derive ultrasound images 44. As previously discussed, for example, an image processing unit 52 is used to determine the displacement of the reference point 24 in each ultrasound image frame. Simultaneously with ultrasound imaging, a secondary imaging modality 82, such as MRI, CT, X-ray fluoroscopy, or any other imaging modality, acquires a set of first images 62 (corresponding to the reference images retrieved in the retrieval phase). For each secondary image frame (2D or 3D), this is stored in a data storage device 60 in association with the measured displacement of the anatomical reference point 24 within the ultrasound imaging reference frame, which is used to capture ultrasound image frames simultaneously with the associated secondary image frames. In this way, a dataset of reference images is formed, each associated with the measured displacement of the reference point 24 within a reference frame fixed relative to both the ultrasound imaging field of view and the secondary imaging field of view. Therefore, this effectively forms an indexed dataset of reference images, in which indexing is accomplished by measuring displacement via 1D, 2D, or 3D reference points.

[0119] As described above, the displacement of reference point 24 can be represented or recorded according to one or more sets of coordinates or vectors. The coordinate system or metric used to represent the displacement of the reference point should be consistent with the coordinate system or metric used to measure the displacement in the subsequent retrieval phase. Therefore, it can be recorded in dataset 60 to indicate what metric or coordinate system is being used. Optionally, a registration or transformation 56 process for the coordinate system or metric used for displacement measurement can be performed before it is recorded in the dataset, for example, to transform the displacement to a common coordinate system.

[0120] The acquired ultrasound images 44 can be additionally streamed to the display unit 72 in real time. In a preferred embodiment, secondary images can also be output to the display unit in real time. This allows, for example, a user to check in real time the correspondence between the acquired ultrasound images and the secondary images in terms of the imaged anatomical region.

[0121] Optionally, the retrieval phase 10 discussed above can be performed immediately after the acquisition or setup phase 100. For example, the patient can be moved out of the high-resolution image modality (e.g., out of an MRI or CT scanner) while the ultrasound probe(s) are held in the same position (e.g., using a configurable frame 36).

[0122] Optionally, the reference imaging dataset can be reacquired at multiple points during the medical process (through repeated acquisition phases) to keep the imaging up-to-date.

[0123] According to a set of advantageous embodiments, the anatomical target 22 is a periodically moving anatomical target, and the imaging period (within which a reference image is captured and subsequently retrieved) corresponds to one or more cycles of movement of the anatomical target. For example, the anatomical target may be all or part of the heart or lungs. Furthermore, many organs or anatomical regions of the body exhibit secondary periodic movements caused by the respiratory cycle. For example, the liver moves periodically due to the pressure exerted from the diaphragm between the lungs and the liver.

[0124] The stored dataset 60 may include a collection of reference images 62 corresponding to a series of time points across the complete movement cycle of the anatomical target. In this way, high-resolution reference images can be used for the entire cycle of the object's movement.

[0125] Figure 6 An example reference imaging dataset 60 is schematically illustrated, comprising a set of reference images 62a-62e at different time points throughout the entire movement cycle of the object of interest 22 and the associated reference point 24. Although only five images are shown in the illustrated example, a much larger number of reference images would typically be stored. As schematically depicted, each reference image represents the object of interest 22 and the reference point 24 at different corresponding spatial locations within the imaging reference frame.

[0126] Since secondary imaging mode 82 typically has a lower temporal resolution than ultrasound imaging, a greater number of ultrasound image frames are usually acquired during a single movement cycle of the object of interest 22 compared to the number of image frames acquired in secondary (high-resolution) imaging mode 82. Therefore, within a single movement cycle, no higher-resolution reference image is recorded for each detected location of the reference point 24 within the ultrasound image. Therefore, to establish a more complete imaging dataset for the movement cycle, imaging can be performed over multiple movement cycles of the object of interest during the acquisition or setup phase 100, and the secondary images 62 acquired over the multiple cycles are compiled into an aggregate dataset. Since there is typically no perfect temporal alignment between the imaging times of the secondary imaging mode in each cycle, the aggregate dataset will include a set of secondary images covering a greater number of time points throughout the entire movement cycle compared to images acquired during any single cycle.

[0127] This is Figure 7 The diagram is shown schematically. Figure 7 (a) Schematic representation of the detected displacement P of reference point 24 in secondary image frames acquired at a series of time points during the first movement cycle. Figure 7(b) schematically representing the second motion cycle of the second motion cycle of the displacement P detected at reference point 24 in a series of secondary image frames acquired at a series of time points in the process. Figure 7 (c) shows an assembled (aggregated) secondary imaging dataset, in which secondary images from both the first and second periods are included together. As shown, this includes images corresponding to a greater number of time points (or phase points) within the motion period than either period alone. In practice, imaging of more than two periods can be performed. (Assembled or aggregated imaging dataset) Figure 7 (c) can be inherently formed by appending each generated reference image to the reference image dataset 60 during an imaging period spanning several movement cycles. Since the images are indexed in the dataset according to the location of the reference points, an aggregated dataset over multiple movement cycles of the reference points is automatically constructed.

[0128] If the acquisition or setup phase 100 is performed over a sufficiently large number of movement cycles (e.g., 5-10), this should be sufficient to construct a reference imaging dataset 60 comprising a corresponding secondary (higher resolution) image for each of the ultrasound image frames (i.e., for each detected location of reference point 24). However, in some cases this may not be possible (e.g., if the acquisition phase runs for only a limited time period).

[0129] According to one or more embodiments, a function may be included to generate an interpolated secondary image frame if the displacement of reference point 24 is measured to not perfectly match the displacement recorded for any reference image 62 in the dataset (during retrieval phase 10).

[0130] This is Figure 8 The diagram is schematically illustrated. This schematically illustrates the measured displacement P of a reference point for a dataset of reference images acquired during acquisition phase 100. During the retrieval phase, a displacement 90 of reference point 24 is detected, which falls between (and within) the recorded reference point displacements of examples 62d and 62e of the reference images. In response to this determination, an interpolated reference image 92 is generated based on the interpolation between the first and second reference images in the reference images.

[0131] This can be generated, for example, simply by taking the average (median or mean) of the pixel intensity values ​​of the first reference image 62D and the second reference image 62E, with the detected reference point displacement 90 falling between the pixel intensity values ​​of the first reference image 62D and the second reference image 62E. This will result in an image frame located between the two frames. However, this method may lead to a loss of resolution.

[0132] Another more complex way to determine the interpolated image frame is to use motion estimation techniques to estimate the motion of the moving anatomical target between the first reference image 62D and the second reference image 62E, and based on this, determine the estimated position of the anatomical target at the intermediate reference point 90. A similar approach is used in the field of television technology and is known as scan rate conversion. An example method is outlined in the following paper: G. De Haan and PWACBiezen, "An efficient true-motion estimator using candidate vectors from a parametric motion model" (IEEE Transactions on Circuits and Systems for Video Technology, vol. 8, no. 1, pp. 85-91, February 1998). This method is based on motion detection and prediction.

[0133] As an alternative to interpolation, in response to the detection that the determined displacement 54 of a reference point in the received image does not match the recorded reference point displacement of any reference image 62 in the dataset 60 and falls between the recorded reference point displacements of the first and second reference images, the method may include selecting one of the first or second reference images and generating an output 64 indicating the selected one of these images. Therefore, here, a reference image closest to the detected reference point location is selected instead of the reference image of the object of interest where the reference point is precisely at the detected reference point location.

[0134] The method may include determining which of the first and second reference images (reference points) the determined displacement 54 of the reference point in the received image is closest to, and selecting the closest identified image for generating output 64.

[0135] As described above, there are different possible methods for detecting the displacement of reference point 24 within a series of received ultrasound images 44. An advantageous method will now be outlined in more detail. This method is only outlined by way of one example, and other methods, as described above, may be used alternatively.

[0136] The method discussed below aims to provide a simplified algorithm for tracking the displacement of a reference point 44 or line on an object of interest. It offers the advantage of reduced computational complexity compared to other algorithms, such as those based on so-called "optical flow motion tracking." The method is based on receiving input from a user interface indicating the area or region in which the specific reference point or line to be tracked is located. Thus, the user assists the detection algorithm. This requires very little human cognitive time but saves significant computational resources. The method is preferably based on tracking one or more edges of the object of interest.

[0137] This method includes providing user input based on the user's observation of a live ultrasound image stream 44 received from ultrasound devices 32, 42 and displayed on a display 72. For example, user input can be provided via a graphical user interface.

[0138] Specifically, users perform the following actions based on their observations. First, the user delineates one or more regions of interest in the live ultrasonic flow. A user interface is provided, having a user input device (e.g., a pointer, mouse, touchscreen, or electronic pen), and is adapted to receive user input instructing the delineation. The graphical user interface for drawing edges preferably supports dragging, resizing, and deforming the boundaries of one or more regions. The interface may include forcing the delineated regions to align into a parallelogram shape during (or after) each user interaction. Figure 9 The diagram illustrates the selection of the region of interest for the example.

[0139] Select one or more selected regions 108 to surround a portion of the edge 106 of the object of interest, which is a reference point 24 or line whose displacement will be detected and tracked in all received ultrasound image frames. Figure 9 Edge 106 is schematically illustrated in the diagram. As an example, Figure 9 The edges representing the boundaries of the liver are shown. The liver may undergo periodic movements due to pressure exerted by the diaphragm during respiration.

[0140] For moving edges, the two boundaries of the depicted region 108 should be as parallel as possible to the direction of movement 110. This ensures that the moving structure of interest remains within the region of interest.

[0141] The other two boundaries of region 108 should be as parallel as possible to the direction of the edge 106 to be tracked. This ensures maximum enhancement of the edge of interest, as the algorithm uses this to set the direction of pixel averaging (explained later).

[0142] As an additional optional feature, other edges unrelated to the object of interest can also be tracked. For example, Figure 9The user-selected region 104 is shown, which includes an edge 102 formed by the contact plane between the body of the object 8 and the support surface on which the object 8 and transducer unit 32 rest. Monitoring changes in displacement of this edge can be used to provide, for example, an indication that the patient is moving, which can be used to notify the registration between ultrasound and a secondary imaging field of view or a reference frame, or to adjust the registration 56 of the ultrasound coordinate system.

[0143] Based on some examples, the boundaries of one or more regions of interest (ROIs) can be automatically modified or updated during the reference point tracking process (on multiple ultrasound image frames 44). For example, optimization algorithms can be employed that are adapted to evaluate the user-drawn boundaries of one or more ROIs and modify them in terms of length or orientation to ensure that the reference lines or edges remain within the regions throughout their movement.

[0144] Once a user-input depiction of region 108 containing reference points or edges 106 has been received, the method includes calculating the positions of the edges. This can be accomplished by an algorithm that includes the steps described below. Figure 10 The diagram illustrates some of the steps of the algorithm. Figure 11 The ultrasound image frames for the three different stages of the algorithm are shown. Figure 11 (a) shows a depiction of one or more regions of interest 108, 104 as described above.

[0145] The steps of the algorithm are as follows.

[0146] First, within an image frame in which the region of interest 108 has been depicted, the region of interest 108 is conceptually divided into elongated strips. These strips, constructed or defined, extend elongatedly along the boundary of the region 108, which runs parallel to the tracked edge 106. The strips may be continuous, such that they cover the entire region of interest. The strips may have equal widths, or at least most of the strips may have equal widths.

[0147] exist Figure 10 The diagram illustrates the division of each region into strips. Figure 10 The process is shown for a region 108 containing the edge 106 of an organ and an optional region 104 containing the boundary edge 102 of an object surface.

[0148] The algorithm also includes summing the pixel intensity values ​​of pixels within each strip and calculating the average pixel intensity value for the entire strip. The average value is represented, for example, by an arithmetic mean. The summation for each strip is performed by... Figure 10 Arrow 114 is shown schematically.

[0149] The algorithm also includes replacing the pixel intensity value of each pixel with the average intensity value determined for the corresponding strip for each strip. Note that by creating strips along the direction of the tracked edge 106 and summing the pixel values ​​in each strip, typically blurred organ regions are enhanced, while typically noisy, rippled "empty" spaces receive a more uniform gray level with lower intensity. This is in Figure 11 The diagram in (b) shows... Figure 11 (b) shows the regions of interest 108, 104 after pixel averaging for the strip has been performed.

[0150] To detect edges, the algorithm includes the following additional steps.

[0151] Local averaging and pixelation are applied to increase image smoothness and reduce pixel resolution.

[0152] Apply an appropriate threshold to create a sharp edge between the object of interest and the surrounding space. This is in Figure 11 The diagram in (c) is shown.

[0153] The middle of the edge is calculated in two steps. First, a line is defined extending from the midpoint of the bottom or leftmost strip to the midpoint of the top or rightmost strip, passing through all the strips in each of the depicted regions of interest 108, 104. Figure 10 The diagram illustrates a boundary line 112 for depicting region 108 and a boundary line 118 for depicting region 104. The middle of the edge is then found by comparing the pixel intensity values ​​of adjacent stripes along this line. It can be assumed that a pair of adjacent stripes with the largest difference in their respective pixel intensity values, or a difference exceeding a predefined threshold, spatially coincide with the boundary line.

[0154] The edge detection process is performed for each of the received ultrasound images in the series of ultrasound images. This allows for precise determination of the displacement of edges within each image frame.

[0155] The specific steps in the above process will now be described in more detail.

[0156] To understand the explanation, we will first refer to... Figure 10 Describe the mathematical description of the stripe relative to one or more regions of interest 104, 108.

[0157] make Indicates in Figure 10 The optional region of interest 104 shown has a right-hand coordinate (x) d ,y d A thin band. For this band, the following holds true.

[0158]

[0159]

[0160] By, for example, x d from Increment to Iteration was implemented over all stripes in this example region 104. Example region 104 shows two stripes below the edge and an undetermined number of stripes above the edge—see [link to example region 104]. Figure 10 However, the total number of bands can usually be greater than... Figure 10 The examples shown are much more numerous. A single strip can be only a few pixels wide, for example, 1-5 pixels.

[0161] If (x) s ,y s () represents the coordinates of pixels within the strip. The complete set of pixel coordinates for this strip can then be described by the following formula:

[0162]

[0163]

[0164] This holds true if the opposite edges of the defined regions 108 and 104 are parallel to each other (and of equal size), i.e., the region is a parallelogram.

[0165] In the case of regions 108 and 106 having (near) horizontal and / or vertical edges, iteration and summation should be performed on appropriate axes to achieve sufficient resolution in terms of the number of stripes and the number of pixels per stripe. For example, for region 108 containing reference edge 106, the right-hand and left-hand edges of region 108 are (approximately) perpendicular to and parallel to the stripe direction in that region (see...). Figure 10 This will lead to inaccuracies in the pixel count within each strip. Specifically, note that Equations 3 and 4 above require that P5 and P7 in region 108 be replaced by P1 and P2 respectively (and P6 by P3). Because... Very close This will result in a very small enumeration range. Following Formula 3, this would be...

[0166]

[0167] Following formula 4, this may also be due to the following formula: y s Inaccurate results.

[0168]

[0169] However, it should be noted that iterations can be performed on different strips within region 108 on either the x-axis or y-axis, since the x and y coordinates of P1 and P3 are sufficiently different (see [link to relevant documentation]). Figure 10 ).

[0170] If both the iteration on the strip and the pixel enumeration within the strip are performed on the y-axis in region 108, this leads to changes in Equation 1 and the formula...

[0171] The following modifications are made to Equations 2, 3, and 4:

[0172]

[0173]

[0174]

[0175]

[0176] Depending on the direction of the edge of region 108 and the stripe direction within that region, the most accurate method for stripe iteration must be selected according to Equation 1 or 2, or Equation 1b or 2b. The same applies to pixel enumeration within a stripe, which can be performed according to Equation 3 or 4, or Equation 3b or 4b. In this paper, pixel enumeration refers to the process of iterating through all pixels within a single stripe and performing the steps discussed above: summing the pixel values, calculating the average pixel value for each stripe, updating the pixel values ​​with the calculated average, and performing the thresholding step.

[0177] We will now discuss the details related to the feature of summing the pixel values ​​within the strip.

[0178] Strong edge enhancement is achieved by summing the intensity values ​​of all pixels within a strip and replacing each pixel in the strip with the average pixel value of the strip. This can be done in... Figure 11 As seen in (b), Figure 11 (b) shows an example ultrasound image frame after the procedure has been performed.

[0179] The average pixel value is represented, for example, as an arithmetic mean, which is calculated as the sum of the pixel intensity values ​​of all pixels in a given strip divided by the number of pixels in the strip.

[0180] Figure 11 (a) shows the regions of interest 104 and 108 as a graphical overlay in the ultrasound image frame. The summation of pixel intensities within the strips is shown in... Figure 11 As shown in (b).

[0181] The process of pixelation and thresholding will now be discussed.

[0182] The modified stripes are converted into a black and white image by applying a threshold T, which changes each pixel value to zero or the maximum allowed pixel value I depending on whether the original pixel value is below or above the threshold T. max .

[0183] if Indicates the strip after thresholding coordinates (x) s ,y s The pixel value at ) and If we represent the average pixel value at the same coordinates (see above), then the following holds true:

[0184] for

[0185] for

[0186] Ultrasound images typically have 8-bit pixel values, therefore in this case I max =255.

[0187] When on Figure 11 When example regions 108 and 104 in (b) are both pixelated and thresholded, the following is achieved: Figure 11 The result shown in (c) is as follows.

[0188] The detection process in the middle of tracking edges 106 and 102 will now be discussed.

[0189] After applying the threshold, as described above, the stripes in the relevant regions of interest 108, 104 are compared with those obtained by means of... Figure 10 The movement depicted by arrows 112 or 118 intersects the line in the middle of the strip.

[0190] For example, if (x l ,y l If ) represents a point on such a line 118 for an optional region of interest 104, then the following holds true:

[0191]

[0192]

[0193] By comparing pixel values ​​along line 118, if the intensity values ​​of adjacent pixels differ significantly, for example, from a predefined threshold, then at coordinate (x... i ,y i The intersection point with the tracked edge 102 of interest is found at (). Sharp contrast in pixel intensity values ​​at the edge boundary is guaranteed by a previously applied thresholding procedure. For example, at the edge location, the following holds:

[0194] or Where, x i+1 =x i +1, and where, (x i ,y i ) and (x i+1 ,y i+1 It satisfies formulas 5 and 6 above.

[0195] Then we can assume that the calculated edge E i The strip extends along its elongated direction through regions of interest 104 and 108 (see...). Figure 10 ): arrive

[0196] The above process (algorithm) is performed on each of the received ultrasound images 44 to determine reference lines in each image. Only the user is required to provide a delineation of the region of interest (e.g., Figure 10 The user input (104 and 108 in the original text) is used once. By default, the algorithm then applies the same delineation to each ultrasound image in the received set of ultrasound images. As mentioned above, in some examples, the delineation of the region of interest can be automatically modified or adjusted based on an optimization function.

[0197] Although in the example above, the user selects the region of interest using a user interface device, in a variation of this method, the user can alternatively use the user interface to draw the reference line or edge to be traced itself. For example, the user interface could allow the user to draw along the line or edge 106, 102 to be traced. The algorithm can be adapted to define a set of strips to follow the shape or curvature of the drawn edge or line. Therefore, the strips in this example can be curved rather than straight.

[0198] The algorithm described above represents only one example unit for determining the displacement of a reference point in each of the received ultrasound images 44, where the reference point to be tracked is a reference line corresponding to the edge 112 of the object of interest.

[0199] Other methods can be used for the detection and displacement determination of reference points, lines, areas, or volumes in each frame. For example, model-based segmentation algorithms can be used to identify the desired edges or points of anatomical targets in each frame, or standard shape matching or edge detection algorithms can be used.

[0200] An illustrative example is described in the following paper: "Ultrasound Sub-pixel Motion-tracking Method with Out-of-plane Motion Detection for Precise Vascular Imaging" by Hideki Yoshikawa et al.

[0201] (Ultrasound in Medicine & Biology, Vol. 46, No. 3, 2020, pp. 782-795).

[0202] Another illustrative example is described in the following paper: Cornel Zachiu et al., “An improved optical flow tracking technique for real-time MR-guided beamtherapies in moving organs” (IEEE International Symposium on Biomedical Imaging (ISBI 2016), April 2016).

[0203] One aspect of the invention also provides the processing apparatus 50 discussed above, and is configured to perform any of the methods outlined above, including one or both of the setup (acquisition) 10 and retrieval 100 stages. Preferably, the same processing apparatus may selectively execute each stage in response to receiving control command signals, for example, from a user interface or from another control module coordinating the entire imaging system.

[0204] although Figure 2 and Figure 5 A processing device 50 including a reference image dataset 60 is shown, but this is optional. In another example, the dataset may be external to the processing device, and the processing device may include an input / output port or communication module for communicatively connecting to the dataset.

[0205] The specific embodiments of the invention described above employ a processing device. A processing device typically includes a single processor or multiple processors. It may be located in a single comprising device, structure, or unit, or it may be distributed among multiple different devices, structures, or units. Therefore, references to a processing device suitable for or configured to perform a particular step or task may correspond to that step or task being performed individually or in combination by any one or more of the multiple processing components. Those skilled in the art will understand how such a distributed processing device can be implemented.

[0206] One or more processors in a processing device can be implemented in various ways using software and / or hardware to perform a variety of required functions. A processor typically employs one or more microprocessors, which can be programmed using software (e.g., microcode) to perform the required functions. A processor can be implemented as a combination of dedicated hardware performing some functions and one or more programmed microprocessors and associated circuitry performing other functions.

[0207] Examples of circuits that may be employed in various embodiments of this disclosure include, but are not limited to, conventional microprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0208] In various implementations, the processor may be associated with one or more storage media, such as volatile and non-volatile computer memories, such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that, when run on one or more processors and / or controllers, perform the desired functions. The various storage media may be fixed within the processor or controller, or may be transportable, allowing one or more programs stored thereon to be loaded into the processor.

[0209] Those skilled in the art, through studying the accompanying drawings, the disclosure, and the claims, will be able to understand and implement variations of the disclosed embodiments when practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude a plurality.

[0210] A single processor or other unit can implement the functions of several items as described in the claims.

[0211] Although specific measures are described in different dependent claims, this does not imply that combinations of these measures cannot be used advantageously.

[0212] Computer programs can be stored / distributed on suitable media, such as optical storage media or solid-state media supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0213] In cases where the term “suitable” is used in the claims or description, it should be noted that the term “suitable” is intended to be equivalent to the term “configured as”.

[0214] No reference numerals in the claims should be construed as limiting the scope.

Claims

1. A method for providing a source of secondary imaging of a moving anatomical target (22) of interest in a patient (8), the method comprising: Receive (12) a series of ultrasound images (44) representing the region containing the anatomical target, the images corresponding to different time points within the imaging period; For each ultrasound image, the displacement (54) of a predefined reference point within a reference frame fixed with respect to the ultrasound imaging field of view is determined, the predefined reference point being fixed with respect to the anatomical target (22); Access a dataset (60) comprising a set of reference images (62) representing the anatomical target at different moving positions, the reference images corresponding to imaging modalities different from the ultrasound images, and wherein each reference image in the dataset is associated with the displacement of the corresponding record of the reference point within a reference frame fixed with respect to the reference imaging field of view; For each received ultrasound image (44), at least one reference image from the reference images (62) is selected based on the displacement of the determined reference point in the received image, and For each received ultrasound image, an output (64) representing the selected reference image is generated, which provides a source for secondary imaging.

2. The method of claim 1, further comprising displaying each selected reference image (62) on a display device (72), and optionally including displaying each selected reference image simultaneously with the corresponding received ultrasound image (44).

3. The method of claim 1 or 2, wherein, The ultrasound images (44) were received in real time during the interventional procedure.

4. The method according to claim 1 or 2, wherein, The anatomical target (22) is a periodically moving anatomical target, and the imaging period corresponds to one or more moving cycles of the anatomical target.

5. The method according to claim 4, wherein, The stored dataset (60) includes a collection of reference images (62) corresponding to a series of time points across a complete movement cycle of the anatomical target.

6. The method according to claim 1 or 2, wherein, The dataset (60) of reference images is images previously acquired during the calibration phase for the same anatomical target (22), and wherein the associated displacement (54) of the reference point for each reference image is the displacement relative to a reference frame fixed with respect to the ultrasound imaging field of view, recorded in the simultaneously acquired ultrasound images (44).

7. The method according to claim 1 or 2, wherein, The ultrasound image (44) is received from an ultrasound imaging device including an ultrasound transducer unit (32) in a fixed posture relative to the patient's anatomy, wherein the ultrasound transducer unit is held by a support frame (36) that allows for adjustment of the posture of the ultrasound transducer unit and releasable fixation.

8. The method according to any one of claims 1 or 2, wherein, The method includes: in response to detecting that the displacement (54) of the determined reference point in the received image does not match the recorded reference point displacement of any reference image in the reference image (62) of the dataset (60) and lies between the recorded reference point displacements of a first reference image and a second reference image in the reference image, generating an interpolated reference image (92) based on interpolation between the first reference image and the second reference image in the reference image.

9. A method for generating a dataset (60) of reference images (62) of a moving anatomical target (22) for providing secondary imaging of a source, the method comprising: A series of first images (62) containing the anatomical region of the anatomical target are acquired (102) during the imaging period using the first imaging modality (82). While acquiring the series of first images (104), a series of ultrasound images (44) of a region containing the same anatomical target are acquired during the imaging period, wherein the first imaging modality (82) is different from the imaging modality of the ultrasound images; For each ultrasound image in the ultrasound image, determine (106) the displacement of a predefined fixed reference point on the anatomical target within a reference frame fixed with respect to the ultrasound imaging field of view; For each first image in the first image, a representation of the first image and a record associated with the displacement of the determined reference point in one of the ultrasound images acquired simultaneously with the first image are stored in the dataset (108) to generate a reference imaging dataset (60) for providing a source for secondary imaging.

10. The method according to claim 9, wherein, The method further includes the step of registering the coordinate system of the ultrasound imaging reference frame with the coordinate system of the secondary imaging modal reference frame.

11. The method according to claim 9 or 10, wherein, The anatomical target (22) is a periodically moving anatomical target, and the imaging period corresponds to multiple moving cycles, wherein the method includes generating an average dataset or aggregated dataset for a single moving cycle based on image data from the multiple moving cycles.

12. A method comprising: The calibration phase includes performing the method according to any one of claims 9-11; as well as In the retrieval phase following the calibration phase, the retrieval phase includes performing the method according to any one of claims 1-8, wherein the reference imaging dataset generated in the calibration phase is used as the dataset of the reference image in the retrieval phase.

13. A processing device (50) arranged to provide a source of secondary imaging of a patient’s moving anatomical target of interest (22), said processing device being adapted to: Receive a series of ultrasound images (44) representing the region containing the anatomical target, the images corresponding to different time points within the imaging period; For each ultrasound image, the displacement of a predefined fixed reference point on the anatomical target within a reference frame fixed with respect to the ultrasound imaging field of view is determined (54). Access a dataset (60) comprising a set of reference images (62) representing the anatomical target at different positions of movement, the reference images having an imaging modality different from the ultrasound images, and wherein, Each reference image in the dataset is associated with the displacement of the corresponding record of the reference point on the object within a reference frame with a fixed field of view of the reference imaging. For each received ultrasound image, at least one reference image is selected from the reference images (62) based on the displacement of the determined reference point in the received image; For each received ultrasound image, an output (64) representing the selected reference image is generated, which provides a source for secondary imaging.

14. A system comprising: The processing apparatus (50) according to claim 13; as well as An ultrasound imaging device includes an ultrasound transducer unit (32) for acquiring ultrasound images of an anatomical body and is arranged to transmit the acquired ultrasound images to the processing device.

15. The system of claim 14, further comprising a support frame (36) arranged to releasably hold the ultrasound transducer unit relative to the patient's anatomy in an adjustable posture, wherein, The support frame is adapted to generate data output (76) including an attitude indicator that indicates the current attitude position of the frame.

Images