Method and system for cross-referencing two-dimensional (2D) ultrasound scans of tissue volumes - Patents.com

Abstract

A method for cross-referencing 2D ultrasound scans of a tissue volume includes generating first and second 2D representations of a target anatomical structure from a first and second series of 2D ultrasound images of the generated tissue volume, respectively. The method includes generating first and second simulated 2D representations of the target anatomical structure from the second and first series of 2D ultrasound images, respectively. The 2D and simulated 2D representations are processed to at least substantially align pixel locations in the 2D and simulated 2D representations associated with the target anatomical structure. The method includes determining first and second correspondence transformation matrices from the processed 2D and simulated 2D representations, and using the first and second correspondence transformation matrices to determine corresponding locations in the first series of 2D ultrasound images for locations in the second series of 2D ultrasound images associated with the target anatomical structure.

Description

[Technical field]

[0001] Related Applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 323,515, entitled "Method and System for Cross-Referencing of Two-Dimensional (2D) Ultrasound Scans of a Tissue Volume," filed March 25, 2022, which is incorporated by reference in its entirety.

[0002]

[0002] The present invention relates generally to methods and systems for cross-referencing two-dimensional (2D) ultrasound scans of a tissue volume, and more particularly to methods and systems for cross-referencing two sweeps of 2D ultrasound images of a tissue volume obtained from orthogonal or substantially orthogonal fields of view or planes. [Background technology]

[0003]

[0003] Two-dimensional (2D) ultrasound (US) imaging is safe, inexpensive, and widely used in medical practice, as well as having real-time and high-resolution capabilities. Conventional 2D ultrasound imaging techniques can be configured to extract 2D ultrasound images (sometimes called cross-sectional images, image planes / frames, or B-mode / B-scan images) of an anatomical structure or tissue volume scanned by an ultrasound probe.

[0004]

[0004] However, conventional 2D ultrasound imaging techniques have an inherent limitation in that they rely on 2D images to depict three-dimensional (3D) anatomical structures. The transducer may often be manually manipulated (moved) by an operator to obtain a 2D image (or a series of 2D images, also known as a 2D sweep or freehand sweep) of a body organ that covers the 3D anatomical structures.

[0005]

[0005] In 3D imaging diagnostics (e.g., magnetic resonance imaging (MRI) or computed tomography (CT)), images can be obtained with known 3D pixel spacing, which allows multiplanar reformatted imaging, allowing the scan data to be visualized in the transverse (or axial), coronal, or sagittal planes depending on the diagnostic task. However, in freehand ultrasound, only pixel spacing in the imaging plane (xy) is available, and pixel spacing in the out-of-plane direction (z direction) may not be available. Thus, if the ultrasound scan is performed in the transverse plane, a simple projection of the image in the sagittal field of view may lead to a sagittal scan of insufficient quality and low resolution, which may not be usable for diagnostic imaging.

[0006]

[0006] The inherent limitation of ultrasound imaging is that it can have high resolution in the imaging plane (high axial and lateral resolution), but cannot have high resolution in the out-of-plane direction. Therefore, in standard clinical evaluation, it is common to perform two sweeps of 2D ultrasound images from orthogonal directions to better capture the 3D anatomical structures with high resolution. Although this can be helpful in acquiring images, aligning these two views to find corresponding points in another view is a difficult problem because the freehand sweep is performed by hand and there is no reference point to align the images. This problem is also called superimposition of two views in the fields of computer vision and medical imaging.

[0007]

[0007] In clinical practice, images from the two views are typically cross-referenced (or superimposed) in the mind of an expert radiologist / sonographer to form a subjective impression of the 3D anatomy and / or pathology. This practice is time-consuming and inefficient, places a high cognitive load on the radiologist, and often leads to variability and inconsistency in measurements and diagnoses.

[0008]

[0008] This problem is further illustrated herein below, without loss of generality, in the context of thyroid ultrasound examination, with reference to FIG.

[0009] FIG. 4 shows in (a) and (b) a schematic diagram of the (human) thyroid gland (labeled "T"). The arrows in (a) and (b) indicate the scanning direction of the ultrasound probe in the transverse (TRX) and sagittal (SAG) planes, respectively. (c) and (d) are sample TRX and SAG thyroid ultrasound sweeps with manually labeled thyroid lobe boundary overlays (contours labeled "Cl" and "C2").

[0009]

[0010] As part of a typical thyroid ultrasound workflow, a clinician may sequentially scan both the left and right lobes of the thyroid gland. A scan of each side (e.g., the left lobe shown in FIG. 4) may include a scan in the transverse (TRX) plane (shown in FIG. 4(a)), and a scan in the sagittal (SAG) plane (shown in FIG. 4(b)). For example, a scan of each side may be performed first in the transverse (TRX) plane, followed by a scan in the sagittal (SAG) plane.

[0010]

[0011] The clinician may then view the patient's scan, which includes a transverse (TRX) sweep and a sagittal (SAG) sweep of the thyroid gland. The transverse images (such as the image shown in FIG. 4(c)) may often be inspected first for the appearance of nodules or suspicious lesions. When nodules are found, it may often be necessary to manually locate each nodule in the corresponding image obtained along the sagittal view (such as the image shown in FIG. 4(d)) and inspect it in both views. This may require the clinician to scroll through all the frames of the SAG scan to locate the same nodule. Regions of interest (ROIs) or landmarks may then be annotated and measured in both views and recorded for later inspection. This task is cognitively demanding and time consuming. Summary of the Invention [Problem to be solved by the invention]

[0011]

[0012] There is therefore a need to provide a method and system for cross-referencing two-dimensional (2D) ultrasound scans of tissue volumes, aimed at overcoming, or at least ameliorating, one or more of the deficiencies associated with conventional methods and systems, particularly with respect to freehand ultrasound scanning of tissue volumes.It is against this background that the present invention has been developed. [Means for solving the problem]

[0012]

[0013] According to a first aspect of the present invention there is provided a method for cross-referencing of two-dimensional (2D) ultrasound scans of a tissue volume, comprising the steps of: generating a first series of 2D ultrasound images of a tissue volume associated with a first plurality of locations, each image being along a first scan direction of the tissue volume, the tissue volume including the target anatomical structure; generating a second series of 2D ultrasound images of the tissue volume associated with the second plurality of locations, respectively, along a second scan direction of the tissue volume, the second scan direction being at least substantially orthogonal to the first scan direction; generating a first 2D representation of the target anatomical structure from a first series of 2D ultrasound images, the first 2D representation being associated with one of the images in the first series of 2D ultrasound images; generating a first simulated 2D representation of the target anatomy from the second series of 2D ultrasound images, the first simulated 2D representation of the target anatomy and the first 2D representation corresponding to each other in terms of a viewing direction relative to the target anatomy; processing the first 2D representation of the target anatomical structure and the first simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the first simulated 2D representation with pixel locations associated with the target anatomical structure in the first 2D representation; determining a first correspondence transformation matrix from the processed first 2D representation and the processed first simulated 2D representation of the target anatomical structure; generating a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, the second 2D representation being associated with one of the images in the second series of 2D ultrasound images; generating a second simulated 2D representation of the target anatomy from the first series of 2D ultrasound images, the second simulated 2D representation of the target anatomy and the second 2D representation corresponding to each other in terms of a viewing direction relative to the target anatomy; processing the second 2D representation of the target anatomical structure and the second simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the second simulated 2D representation with pixel locations associated with the target anatomical structure in the second 2D representation; determining a second correspondence transformation matrix from the processed second 2D representation and the processed second simulated 2D representation of the target anatomical structure; and determining, for a first location associated with the target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images using the first and second correspondence transformation matrices.

[0013]

[0014] In various embodiments, the step of determining the second location using the first corresponding transformation matrix and the second corresponding transformation matrix includes: Applying a first function of the first corresponding transformation matrix and the second corresponding transformation matrix to pixel coordinates corresponding to the first location to obtain pixel coordinates corresponding to the second location.

[0014]

[0015] In various embodiments, the method comprises: The method further includes determining, for a third location associated with the target anatomical structure in the first series of 2D ultrasound images, a corresponding fourth location associated with the target anatomical structure in the second series of 2D ultrasound images using the first and second correspondence transformation matrices.

[0015]

[0016] In various embodiments, the step of determining the fourth location using the first corresponding transformation matrix and the second corresponding transformation matrix includes: Applying a second function of the first corresponding transformation matrix and the second corresponding transformation matrix to pixel coordinates corresponding to the third location to obtain pixel coordinates corresponding to a fourth location.

[0016]

[0017] In various embodiments, the method comprises: generating a first plurality of binary segmentation masks from the first series of 2D ultrasound images, each binary segmentation mask in the first plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images in the first series of 2D ultrasound images; generating a second plurality of binary segmentation masks from the second series of 2D ultrasound images, each binary segmentation mask of the second plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images of the second series of 2D ultrasound images; a first 2D representation of the target anatomical structure is generated from one of the binary segmentation masks of the first plurality of binary segmentation masks; A second 2D representation of the target anatomical structure is generated from one of the binary segmentation masks of the second plurality of binary segmentation masks.

[0017]

[0018] In various embodiments, a first 2D representation of the target anatomical structure corresponds to a largest binary segmentation mask among the first plurality of binary segmentation masks, and a second 2D representation of the target anatomical structure corresponds to a largest binary segmentation mask among the second plurality of binary segmentation masks.

[0018]

[0019] In various embodiments, generating a first simulated 2D representation of the target anatomical structure from the second series of 2D ultrasound images includes generating the first simulated 2D representation of the target anatomical structure from the second plurality of binary segmentation masks, and generating a second simulated 2D representation of the target anatomical structure from the first series of 2D ultrasound images includes generating the second simulated 2D representation of the target anatomical structure from the first plurality of binary segmentation masks.

[0019]

[0020] In various embodiments, processing the first 2D representation includes generating a first signed distance map from the first 2D representation, and processing the second 2D representation includes generating a second signed distance map from the second 2D representation.

[0020]

[0021] In various embodiments, processing the first simulated 2D representation includes scaling a width of the first simulated 2D representation to match a width of the first 2D representation, and processing the second simulated 2D representation includes scaling a width of the second simulated 2D representation to match a width of the second 2D representation.

[0021]

[0022] In various embodiments, the step of processing the first simulated 2D representation further includes determining and sampling contour pixels associated with a contour of the target anatomical structure from the scaled first simulated 2D representation, and the step of processing the second simulated 2D representation further includes determining and sampling contour pixels associated with a contour of the target anatomical structure from the scaled second simulated 2D representation.

[0022]

[0023] In various embodiments, processing the first simulated 2D representation further includes sampling a first subset of pixels from the scaled first simulated 2D representation, and processing the second simulated 2D representation further includes sampling a second subset of pixels from the scaled second simulated 2D representation.

[0023]

[0024] In various embodiments, processing the first simulated 2D representation further includes estimating a coarse shift between the scaled first simulated 2D representation and the first 2D representation using the first signed distance map and the first sampled subset of pixels, and processing the second simulated 2D representation further includes estimating a coarse shift between the scaled second simulated 2D representation and the second 2D representation using the second signed distance map and the second sampled subset of pixels.

[0024]

[0025] In various embodiments, determining the first correspondence transformation matrix includes estimating a scale and translation between the scaled first simulated 2D representation and the first 2D representation by minimizing a first alignment cost defined as a first cost function for the first correspondence transformation matrix, and determining the second correspondence transformation matrix includes estimating a scale and translation between the scaled second simulated 2D representation and the second 2D representation by minimizing a second alignment cost defined as a second cost function for the second correspondence transformation matrix.

[0025]

[0026] In various embodiments, the first scan direction corresponds to a transverse scan of the tissue volume and the second scan direction corresponds to a sagittal scan of the tissue volume.

[0027] According to a second aspect of the present invention there is provided a system for cross-referencing two-dimensional ultrasound scans of a tissue volume, comprising: Memory, and at least one processor communicatively coupled to the memory and to the ultrasonic transducer, the at least one processor comprising: generating a first series of 2D ultrasound images of a tissue volume associated with a first plurality of locations, each along a first scan direction of the tissue volume, the tissue volume including the target anatomical structure; generating a second series of 2D ultrasound images of the tissue volume associated with the second plurality of locations, respectively, along a second scan direction of the tissue volume, the second scan direction being at least substantially orthogonal to the first scan direction; generating a first 2D representation of the target anatomical structure from the first series of 2D ultrasound images, the first 2D representation being associated with one of the images in the first series of 2D ultrasound images; generating a first simulated 2D representation of the target anatomy from the second series of 2D ultrasound images, the first simulated 2D representation of the target anatomy and the first 2D representation corresponding to each other in terms of a viewing direction relative to the target anatomy; processing the first 2D representation of the target anatomical structure and the first simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the first simulated 2D representation with pixel locations associated with the target anatomical structure in the first 2D representation; determining a first correspondence transformation matrix from the processed first 2D representation and the processed first simulated 2D representation of the target anatomical structure; generating a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, the second 2D representation being associated with one of the images in the second series of 2D ultrasound images; generating a second simulated 2D representation of the target anatomical structure from the first series of 2D ultrasound images, wherein the second simulated 2D representation of the target anatomical structure and the second 2D representation correspond to each other in terms of a viewing direction relative to the target anatomical structure; processing the second 2D representation of the target anatomical structure and the second simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the second simulated 2D representation with pixel locations associated with the target anatomical structure in the second 2D representation; determining a second correspondence transformation matrix from the processed second 2D representation and the processed second simulated 2D representation of the target anatomical structure; and A system is provided that is configured to determine, for a first location associated with a target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images using a first correspondence transformation matrix and a second correspondence transformation matrix.

[0026]

[0028] In various embodiments, when the at least one processor determines the second location using the first corresponding transformation matrix and the second corresponding transformation matrix, the at least one processor further comprises: A first function of the first corresponding transformation matrix and the second corresponding transformation matrix is ​​configured to apply to pixel coordinates corresponding to the first location to obtain pixel coordinates corresponding to the second location.

[0027]

[0029] In various embodiments, the at least one processor comprises: The system is further configured to determine, for a third location associated with the target anatomical structure in the first series of 2D ultrasound images, a corresponding fourth location associated with the target anatomical structure in the second series of 2D ultrasound images using the first and second correspondence transformation matrices.

[0028]

[0030] In various embodiments, when the at least one processor determines the fourth location using the first corresponding transformation matrix and the second corresponding transformation matrix, the at least one processor further comprises: A second function of the first corresponding transformation matrix and the second corresponding transformation matrix is ​​configured to apply to pixel coordinates corresponding to the third location to obtain pixel coordinates corresponding to the fourth location.

[0029]

[0031] In various embodiments, the at least one processor comprises: generating a first plurality of binary segmentation masks from the first series of 2D ultrasound images, each binary segmentation mask of the first plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images of the first series of 2D ultrasound images; generating a second plurality of binary segmentation masks from the second series of 2D ultrasound images, each binary segmentation mask of the second plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images of the second series of 2D ultrasound images; a first 2D representation of the target anatomical structure is generated from one of the binary segmentation masks of the first plurality of binary segmentation masks; A second 2D representation of the target anatomical structure is generated from one of the binary segmentation masks of the second plurality of binary segmentation masks.

[0030]

[0032] In various embodiments, a first 2D representation of the target anatomical structure corresponds to a largest binary segmentation mask among the first plurality of binary segmentation masks, and a second 2D representation of the target anatomical structure corresponds to a largest binary segmentation mask among the second plurality of binary segmentation masks.

[0031]

[0033] In various embodiments, the first scan direction corresponds to a transverse scan of the tissue volume and the second scan direction corresponds to a sagittal scan of the tissue volume.

[0034] In various embodiments, the system further comprises an ultrasound transducer communicatively coupled to the memory and the at least one processor, the at least one processor configured to generate a first series of 2D ultrasound images based on a first series of ultrasound waves acquired by the ultrasound transducer positioned at a first plurality of positions for a first plurality of time instances, and generate a second series of 2D ultrasound images based on a second series of ultrasound waves acquired by the ultrasound transducer positioned at a second plurality of positions for a second plurality of time instances.

[0032]

[0035] According to a third aspect of the present invention there is provided a computer program product embodied in one or more non-transitory computer readable storage media and including instructions executable by at least one processor to perform a method for cross-referencing two-dimensional ultrasound scans of a tissue volume, the method comprising: generating a first series of 2D ultrasound images of a tissue volume associated with a first plurality of locations, each image being along a first scan direction of the tissue volume, the tissue volume including the target anatomical structure; generating a second series of 2D ultrasound images of the tissue volume associated with the second plurality of locations, respectively, along a second scan direction of the tissue volume, the second scan direction being at least substantially orthogonal to the first scan direction; generating a first 2D representation of the target anatomical structure from a first series of 2D ultrasound images, the first 2D representation being associated with one of the images in the first series of 2D ultrasound images; generating a first simulated 2D representation of the target anatomy from the second series of 2D ultrasound images, the first simulated 2D representation of the target anatomy and the first 2D representation corresponding to each other in terms of a viewing direction relative to the target anatomy; processing the first 2D representation of the target anatomical structure and the first simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the first simulated 2D representation with pixel locations associated with the target anatomical structure in the first 2D representation; determining a first correspondence transformation matrix from the processed first 2D representation and the processed first simulated 2D representation of the target anatomical structure; generating a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, the second 2D representation being associated with one of the images in the second series of 2D ultrasound images; generating a second simulated 2D representation of the target anatomy from the first series of 2D ultrasound images, the second simulated 2D representation of the target anatomy and the second 2D representation corresponding to each other in terms of a viewing direction relative to the target anatomy; processing the second 2D representation of the target anatomical structure and the second simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the second simulated 2D representation with pixel locations associated with the target anatomical structure in the second 2D representation; determining a second correspondence transformation matrix from the processed second 2D representation and the processed second simulated 2D representation of the target anatomical structure; and determining, for a first location associated with the target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images using the first correspondence transformation matrix and the second correspondence transformation matrix.

[0033]

[0036] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described herein are not all-inclusive, and many additional features and advantages will be apparent to those skilled in the art, especially when considering the drawings, the specification, and the claims. Furthermore, it should be noted that the language used in this specification has been selected primarily for ease of reading and instruction, and may not be selected to delineate or limit the subject matter of the present invention.

[0034]

[0037] Embodiments of the present invention will be better understood and readily apparent to those skilled in the art from the following description, given by way of example only and in conjunction with the drawings in which: [Brief description of the drawings]

[0035] [Figure 1]

[0038] 1 is a schematic flow diagram of a method for cross-referencing 2D ultrasound scans of tissue volumes according to various embodiments of the present invention. [Diagram 2]

[0039] 2 is a schematic block diagram of a system for cross-referencing 2D ultrasound scans of tissue volumes according to various embodiments of the present invention, such as corresponding to the method shown in FIG. 1. [Diagram 3]

[0040] FIG. 3 is a schematic block diagram of an exemplary computer system that can be used to realize or implement a system for cross-referencing 2D ultrasound scans of tissue volumes according to various embodiments of the present invention, such as the system shown in FIG. [Figure 4]

[0041] FIG. 1 is a schematic diagram of a human thyroid gland and sample transverse and sagittal thyroid ultrasound sweep images. [Diagram 5]

[0042] 1 is a flow chart illustrating aspects of a method for cross-referencing 2D ultrasound scans of tissue volumes according to various embodiments of the present invention. [Figure 6]

[0043] 4 is another flow diagram illustrating aspects of a method for cross-referencing 2D ultrasound scans of tissue volumes according to various embodiments of the present invention. [Figure 7]

[0044] FIG. 7 illustrates a portion of the flow chart shown in FIG. 6 in more detail. [Figure 8]

[0045] FIG. 7 illustrates another portion of the flow chart shown in FIG. 6 in more detail. [Figure 9]

[0046] FIG. 13 illustrates a screenshot of a first view to second view cross-referencing tool or system according to various embodiments of the present invention. [Figure 10]

[0047] FIG. 13 illustrates a screenshot of a second view to first view cross-referencing system according to various embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036]

[0048] Various embodiments of the present invention provide methods (computer-implemented methods) and systems (including a memory and at least one processor communicatively coupled to the memory) for cross-referencing of two-dimensional (2D) ultrasound scans of a tissue volume, and more particularly, methods and systems for cross-referencing of two sweeps of 2D ultrasound images of a tissue volume obtained from orthogonal or substantially orthogonal fields of view or planes.

[0037]

[0049] As described in the Background, two-dimensional (2D) ultrasound imaging is safe, inexpensive, and widely used in medical practice. However, in traditional ultrasound imaging, superimposition (or cross-referencing) of images acquired from two different fields of view is performed in the mind of an expert radiologist / sonographer to form a subjective impression of the 3D anatomy and / or lesions, a time-consuming and inefficient process that places a high cognitive load on the radiologist, often resulting in variability and incorrect diagnosis.

[0038]

[0050] Thus, various embodiments of the present invention provide methods and systems that provide more efficient and / or reliable cross-referencing (registration) of two 2D ultrasound scans of a tissue volume obtained from two different views. Cross-referencing can find corresponding points between one plane and the other. In other words, cross-referencing can map a given point on one plane or view (e.g., a transverse (TRX) view) to a corresponding point on the other plane or view (e.g., a sagittal (SAG) view).

[0039]

[0051] Thus, various embodiments of the present invention can bridge the gap between 2D ultrasound imaging and 3D imaging (e.g., computed tomography (CT) or magnetic resonance imaging (MRI)) by providing correspondence between points in different fields of view. Example clinical applications include, but are not limited to, cardiac, vascular imaging, prostate, bladder, and thyroid imaging.

[0040]

[0052] Various embodiments of the present invention can benefit clinical workflow by simplifying the measurement, screening, and / or monitoring of organs over time to align anatomical structures faster and more reliably.

[0041]

[0053] FIG. 1 illustrates a schematic flow diagram of a method 100 (a computer-implemented method) for cross-referencing two-dimensional (2D) ultrasound scans of a tissue volume (including a target anatomical structure) using at least one processor, in accordance with various embodiments of the present invention.

[0042]

[0054] The method 100 includes generating 102 a first series (or sequence or set) of 2D ultrasound images (which may also be referred to interchangeably as cross-sectional images, image planes, image frames / slices, or B-mode / B-scan images) of a tissue volume associated with a first plurality of locations, each along a first scanning direction of the tissue volume, the tissue volume including the target anatomical structure.

[0043]

[0055] The method 100 further includes generating 104 a second series (or sequence or set) of 2D ultrasound images of the tissue volume associated with the second plurality of locations, respectively, along a second scanning direction of the tissue volume, the second scanning direction being at least substantially orthogonal to the first scanning direction. In various embodiments, as used herein, the term "at least substantially orthogonal" in relation to two directions (e.g., the "first scanning direction" and the "second scanning direction") may be understood to mean that the two directions are orthogonal to each other, or nearly orthogonal to each other, e.g., deviating from orthogonality by 10 degrees or less, e.g., 5 degrees or less, e.g., 2 degrees or less, e.g., 1 degree or less.

[0044]

[0056] In connection with step 102, each of the first series of 2D ultrasound images may be generated based on a first series of ultrasound waves acquired by an ultrasound transducer disposed at a first plurality of positions for a first plurality of time instances. Alternatively, each of the first series of 2D ultrasound images may be generated based on first image data stored in memory related to the first series of ultrasound waves acquired by an ultrasound transducer disposed at a first plurality of positions for a first plurality of time instances.

[0045]

[0057] Similarly, in connection with step 104, each of the second series of 2D ultrasound images may be generated based on the first series of ultrasound waves acquired by an ultrasound transducer disposed at a second plurality of positions for the second plurality of time instances. Alternatively, each of the second series of 2D ultrasound images may be generated based on second image data stored in memory related to the second series of ultrasound waves acquired by an ultrasound transducer disposed at a second plurality of positions for the first plurality of time instances.

[0046]

[0058] In various embodiments, in connection with steps 102 and 104, a first and second series of ultrasound waves may be acquired (in time sequence) at a first plurality of positions along a first scanning direction and a second plurality of positions along a second scanning direction of the tissue volume using an ultrasound transducer configured to emit ultrasound waves into a plane (e.g., a cross section perpendicular to a scanning direction) of the tissue volume and acquire ultrasound waves reflected from such plane of the tissue volume. Such an ultrasound transducer may be referred to as a 2D ultrasound transducer.

[0047]

[0059] Various embodiments of the present invention are directed to cross-referencing 2D ultrasound scans or sweeps acquired by freehand ultrasound scanning of a tissue volume. In this regard, an operator can move a 2D ultrasound transducer (or a portable handheld ultrasound probe equipped with a 2D ultrasound transducer) along a scanning direction of the tissue volume (e.g., along an axis across the length of the tissue volume) to perform an ultrasound scan of the tissue volume, whereby a series of ultrasound waves are acquired by the 2D ultrasound transducer at multiple positions along the scanning direction for multiple time instances, respectively. The ultrasound waves received at each time instance (corresponding position) can then be processed to generate a 2D ultrasound image associated with the corresponding position, as is known in the art, and thus need not be described in detail herein. Thus, a series of 2D ultrasound images of the tissue volume can be acquired, each 2D ultrasound image having an associated position (e.g., tagged or labeled with associated position information), which for example corresponds to the position of the 2D ultrasound transducer where the ultrasound waves (based on which the 2D ultrasound image is generated) were acquired, or corresponds to the position / location along the tissue volume where the ultrasound waves acquired by the 2D ultrasound transducer were reflected.

[0048]

[0060] The 2D ultrasound transducer may be any conventional 2D ultrasound transducer configured to emit and acquire ultrasound waves in a plane of a tissue volume, and therefore need not be described in detail herein. For example, and without limitation, a conventional 2D ultrasound transducer may comprise an array of transducer elements configured to emit and acquire ultrasound waves in a plane of a tissue volume. Thus, it will be understood by those skilled in the art that the present invention is not limited to any particular type of 2D ultrasound transducer.

[0049]

[0061] The method 100 further includes a step 106 of generating a first 2D representation of the target anatomical structure from the first series of 2D ultrasound images, the first 2D representation being associated with one of the images in the first series of 2D ultrasound images.

[0050]

[0062] The method 100 further includes a step 108 of generating a first simulated 2D representation of the target anatomical structure from the second series of 2D ultrasound images, where the first simulated 2D representation of the target anatomical structure and the first 2D representation correspond to each other in terms of a viewing direction relative to the target anatomical structure.

[0051]

[0063] The method 100 further includes a step 110 of processing the first 2D representation of the target anatomical structure and the first simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the first simulated 2D representation with pixel locations associated with the target anatomical structure in the first 2D representation. In various embodiments, the term "at least substantially matched" in the present specification in relation to pixel positions of two different 2D entities (e.g., the first 2D representation after processing and the first simulated 2D representation) may be understood to mean that the position (e.g., 2D coordinate) of a pixel (or a number of pixels, e.g., all the pixels) in one of the two 2D entities (e.g., the first simulated 2D representation after processing) matches or approximately matches the position (e.g., 2D coordinate) of a corresponding pixel (or a number of corresponding pixels) in the other of the two 2D entities (e.g., the first 2D representation after processing), e.g. the deviation in position (e.g., between x coordinates and / or between y coordinates) is no more than a given number of pixel intervals, e.g. no more than 10 pixel intervals, e.g. no more than 5 pixel intervals, e.g. no more than 2 pixel intervals, e.g. no more than 1 pixel interval.

[0052]

[0064] The method 100 further includes determining 112 a first corresponding transformation matrix from the processed first 2D representation and the processed first simulated 2D representation of the target anatomical structure.

[0065] The method 100 further includes a step 114 of generating a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, the second 2D representation being associated with one of the images in the second series of 2D ultrasound images.

[0053]

[0066] The method 100 further includes a step 116 of generating a second simulated 2D representation of the target anatomical structure from the first series of 2D ultrasound images, where the second simulated 2D representation of the target anatomical structure and the second 2D representation correspond to each other in terms of a viewing direction relative to the target anatomical structure.

[0054]

[0067] The method 100 further includes a step 118 of processing the second 2D representation of the target anatomical structure and the second simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the second simulated 2D representation with pixel locations associated with the target anatomical structure in the second 2D representation.

[0055]

[0068] The method 100 further includes determining 120 a second corresponding transformation matrix from the processed second 2D representation and the processed second simulated 2D representation of the target anatomical structure.

[0069] The method 100 further includes step 122 of determining, for a first location associated with the target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images using the first and second correspondence transformation matrices. In various embodiments, the first location can be a point selected by a user in one of the images (image frames) of the second series of 2D ultrasound images, and the second location can be a point obtained (cross-referenced) with the aid of the first and second transformation matrices in one of the images (image frames) of the first series of 2D ultrasound images.

[0056]

[0070] In various embodiments, in connection with step 122, the step of determining the second location using the first and second corresponding transformation matrices includes applying a first function of the first and second corresponding transformation matrices to pixel coordinates corresponding to the first location to obtain pixel coordinates corresponding to the second location.

[0057]

[0071] In various embodiments, the method 100 may further include determining, for a third location associated with the target anatomical structure in the first series of 2D ultrasound images, a corresponding fourth location associated with the target anatomical structure in the second series of 2D ultrasound images using the first and second corresponding transformation matrices. In various embodiments, the third location may be a point selected by a user in one of the images (image frames) of the first series of 2D ultrasound images, and the fourth location may be a point obtained (cross-referenced) with the aid of the first and second transformation matrices in one of the images (image frames) of the second series of 2D ultrasound images.

[0058]

[0072] In various embodiments, the step of determining the fourth location using the first corresponding transformation matrix and the second corresponding transformation matrix includes applying a second function of the first corresponding transformation matrix and the second corresponding transformation matrix to pixel coordinates corresponding to the third location to obtain pixel coordinates corresponding to the fourth location.

[0059]

[0073] In various embodiments, the method 100 may further include generating a first plurality of binary segmentation masks from the first series of 2D ultrasound images, each binary segmentation mask of the first plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images of the first series of 2D ultrasound images. In various embodiments, the first 2D representation of the target anatomical structure may be generated from one of the binary segmentation masks of the first plurality of binary segmentation masks.

[0060]

[0074] In various embodiments, the method 100 may further include generating a second plurality of binary segmentation masks from the second series of 2D ultrasound images, each binary segmentation mask of the second plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images of the second series of 2D ultrasound images. In various embodiments, the second 2D representation of the target anatomical structure may be generated from one of the binary segmentation masks of the second plurality of binary segmentation masks.

[0061]

[0075] In various embodiments, the first plurality of binary segmentation masks and / or the second plurality of binary segmentation masks can be obtained by using segmentation methods or processes known in the field of digital image processing. One example of a suitable segmentation method that can be used by various embodiments is the so-called U-Net, a convolutional neural network developed for biomedical image segmentation. In digital image processing and computer vision, image segmentation refers to the process of dividing a digital image into multiple segments (pixel sets, also known as image objects). More specifically, image segmentation refers to the process of assigning a label (or value) to all pixels in an image such that pixels with the same label (or value) share certain characteristics.

[0062]

[0076] In various embodiments, in each binary segmentation mask, the values ​​of pixels associated with the target anatomical structure may exhibit a first value (e.g., "1"), and the values ​​of pixels not associated with the target anatomical structure may exhibit a second value different from the first value (e.g., "0").

[0063]

[0077] In various embodiments, the first 2D representation of the target anatomical structure may correspond to a largest binary segmentation mask among the first plurality of binary segmentation masks.

[0064]

[0078] In various embodiments, the second 2D representation of the target anatomical structure may correspond to a largest binary segmentation mask among the second plurality of binary segmentation masks.

[0065]

[0079] In various embodiments, the term "largest binary segmentation mask" as used herein may be understood to mean a binary segmentation mask having a maximum number of pixels associated with the target anatomical structure among the plurality of binary segmentation masks. For example, the largest binary segmentation mask may be a binary segmentation mask having a maximum number of pixels having a first value (e.g., "1") among the plurality of binary segmentation masks.

[0066]

[0080] In various embodiments, in connection with step 106, generating the first 2D representation includes removing a region of a largest binary segmentation mask of the first plurality of binary segmentation masks that was erroneously marked as corresponding to the target anatomical structure. For example, a maximum connected component search algorithm may be applied to determine a maximum connected component (largest cluster) in the largest binary segmentation mask, and regions that are not connected to the maximum connected component may be removed from the mask.

[0067]

[0081] In various embodiments, in connection with step 114, generating the second 2D representation includes removing a region of a largest binary segmentation mask of the second plurality of binary segmentation masks that was erroneously marked as corresponding to the target anatomical structure. For example, a maximum connected component search algorithm may be applied to determine a maximum connected component (largest cluster) in the largest binary segmentation mask, and regions that are not connected to the maximum connected component may be removed from the mask.

[0068]

[0082] In various embodiments, in connection with step 108, generating a first simulated 2D representation of the target anatomical structure from the second series of 2D ultrasound images includes generating the first simulated 2D representation of the target anatomical structure from the second plurality of binary segmentation masks. In various embodiments, regions of the first simulated 2D representation that are erroneously marked as corresponding to the target anatomical structure can be removed. For example, a maximum connected component search algorithm can be applied to determine the maximum connected components (largest clusters) and regions that are not connected to the maximum connected components can be removed.

[0069]

[0083] In various embodiments, in connection with step 116, generating a second simulated 2D representation of the target anatomical structure from the first series of 2D ultrasound images includes generating a second simulated 2D representation of the target anatomical structure from the first plurality of binary segmentation masks. In various embodiments, regions of the second simulated 2D representation that are erroneously marked as corresponding to the target anatomical structure can be removed. For example, a maximum connected component search algorithm can be applied to determine the maximum connected components (largest clusters) and regions that are not connected to the maximum connected components can be removed.

[0070]

[0084] In various embodiments, processing the first 2D representation, in connection with step 110, includes generating a first signed distance map from the first 2D representation. In various embodiments, generating the first signed distance map includes determining a boundary contour of the first 2D representation (e.g., a largest binary segmentation mask among the first plurality of binary segmentation masks), generating a first distance map in which the value of each pixel is the distance from that pixel to its nearest contour pixel, converting the first 2D representation (e.g., the largest binary segmentation mask) to the first signed map, and generating the first signed distance map by element-wise multiplication of the first signed map and the first distance map.

[0071]

[0085] In various embodiments, processing the second 2D representation, in connection with step 118, includes generating a second signed distance map from the second 2D representation. In various embodiments, generating the second signed distance map includes determining a boundary contour of the 2D representation (e.g., a largest binary segmentation mask among the second plurality of binary segmentation masks), generating a second distance map in which the value of each pixel is the distance from that pixel to its nearest contour pixel, converting the 2D representation (e.g., the largest binary segmentation mask) to the second signed map, and generating the second signed distance map by element-wise multiplication of the second signed map and the second distance map.

[0072]

[0086] In various embodiments, in connection with step 110, processing the first simulated 2D representation includes scaling a width of the first simulated 2D representation to match a width of the first 2D representation.

[0073]

[0087] In various embodiments, in connection with step 118, processing the second simulated 2D representation includes scaling a width of the second simulated 2D representation to match a width of the second 2D representation.

[0074]

[0088] In various embodiments, in connection with step 110, processing the first simulated 2D representation further includes determining and sampling contour pixels associated with a contour of the target anatomical structure from the scaled first simulated 2D representation.

[0075]

[0089] In various embodiments, in connection with step 118, processing the second simulated 2D representation further includes determining and sampling contour pixels associated with the contour of the target anatomical structure from the scaled second simulated 2D representation.

[0076]

[0090] In various embodiments, in connection with step 110, processing the first simulated 2D representation further includes sampling a first subset of pixels from the scaled first simulated 2D representation.

[0077]

[0091] In various embodiments, in connection with step 118, processing the second simulated 2D representation further includes sampling a second subset of pixels from the scaled second simulated 2D representation.

[0078]

[0092] In various embodiments, in connection with step 110, processing the first simulated 2D representation further includes estimating a coarse shift between the scaled first simulated 2D representation and the first 2D representation using the first signed distance map and the pixels of the first sampled subset.

[0079]

[0093] In various embodiments, in connection with step 118, processing the second simulated 2D representation further includes estimating a coarse shift between the scaled second simulated 2D representation and the second 2D representation using the second signed distance map and the sampled second subset of pixels.

[0080]

[0094] In various embodiments, in connection with step 112, determining the first correspondence transformation matrix includes estimating the scale and transformation between the scaled first simulated 2D representation and the first 2D representation by minimizing a first registration cost defined as a first cost function for the first correspondence transformation matrix. In various embodiments, the first registration cost is defined as a sum of squared distances for the first correspondence transformation matrix.

[0081]

[0095] In various embodiments, in connection with step 120, determining the second correspondence transformation matrix includes estimating the scale and transformation between the scaled second simulated 2D representation and the second 2D representation by minimizing a second registration cost defined as a second cost function for the second correspondence transformation matrix. In various embodiments, the second registration cost is defined as a sum of squared distances for the second correspondence transformation matrix.

[0082]

[0096] In various embodiments, minimizing the first registration cost and / or minimizing the second registration cost includes applying an optimization algorithm, for example a Gauss-Newton algorithm.

[0083]

[0097] In various embodiments, the first scan direction corresponds to a transverse scan of the tissue volume and the second scan direction corresponds to a sagittal scan of the tissue volume.

[0098] In various embodiments, the target anatomical structure includes at least one of a body organ, a portion of a body organ (eg, a thyroid lobe), and a portion of the vasculature.

[0084]

[0099] In various embodiments, the body organ comprises at least one of the thyroid, prostate, bladder, heart, lung, stomach, liver, and kidney.

[0100] FIG. 2 shows a schematic block diagram of a system 200 for cross-referencing two-dimensional (2D) ultrasound scans of a tissue volume according to various embodiments of the present invention, which corresponds to the method 100 for cross-referencing two-dimensional (2D) ultrasound scans of a tissue volume using at least one processor described above according to various embodiments of the present invention.

[0085]

[0101] The system 200 comprises a memory 204 and at least one processor 206 communicatively coupled to the memory 204, the at least one processor 206 being adapted to generate a first series of 2D ultrasound images of a tissue volume associated with a first plurality of locations, each of the 2D images being along a first scanning direction of the tissue volume, the tissue volume including the target anatomical structure, and to generate a second series of 2D ultrasound images of a tissue volume associated with a second plurality of locations, each of the 2D images being along a second scanning direction of the tissue volume, the second scanning direction being at least substantially the same as the first scanning direction. generating a first 2D representation of the target anatomical structure from a first series of 2D ultrasound images, the first 2D representation being associated with one of the images in the first series of 2D ultrasound images; generating a first simulated 2D representation of the target anatomical structure from a second series of 2D ultrasound images, the first simulated 2D representation of the target anatomical structure and the first 2D representation corresponding to each other in terms of a viewing direction relative to the target anatomical structure; generating a pixel location associated with the target anatomical structure in the first simulated 2D representation, the pixel location associated with the target anatomical structure in the first simulated 2D representation being associated with one of the images in the first series of 2D ultrasound images; the first 2D representation and the first simulated 2D representation of the target anatomical structure to at least substantially align with pixel locations associated with the target anatomical structure; determining a first correspondence transformation matrix from the processed first 2D representation and the processed first simulated 2D representation of the target anatomical structure; generating a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, the second 2D representation being associated with one of the images in the second series of 2D ultrasound images; generating representations, wherein the second simulated 2D representation of the target anatomical structure and the second simulated 2D representation correspond to each other in terms of a viewing direction relative to the target anatomical structure; processing the second 2D representation of the target anatomical structure and the second simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the second simulated 2D representation with pixel locations associated with the target anatomical structure in the second 2D representation; determining a second correspondence transformation matrix from the processed second 2D representation of the target anatomical structure and the processed second simulated 2D representation;and determining, for a first location associated with the target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images using the first and second correspondence transformation matrices.

[0086]

[0102] Those skilled in the art will appreciate that the at least one processor 206 may be configured to perform the required functions or operations via a set of instructions (e.g., software modules) executable by the at least one processor 206 to perform the required functions or operations.

[0087]

[0103] Thus, as shown in FIG. 2 , system 200 includes a 2D ultrasound image generator 208 configured to generate a first series of 2D ultrasound images and a second series of 2D ultrasound images, a 2D representation generator 210 configured to generate a first 2D representation of the target anatomical structure from the first series of 2D ultrasound images and generate a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, a simulated 2D representation generator 212 configured to process the first 2D representation of the target anatomical structure and the first simulated 2D representation and generate the second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, and a 2D representation generator 214 configured to generate a first 2D representation of the target anatomical structure from the first series of 2D ultrasound images and generate a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images. The ultrasound system may further include a 2D and simulated 2D representation processor 214 configured to process the second 2D representation and the second simulated 2D representation of the anatomical structure, a correspondence transformation matrix determiner 216 configured to determine a first correspondence transformation matrix from the processed first 2D and first simulated 2D representations and to determine a second correspondence transformation matrix from the processed second 2D and second simulated 2D representations, and a correspondence location determiner 218 configured to determine a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images for a first location associated with the target anatomical structure in the second series of 2D ultrasound images by using the first correspondence transformation matrix and the second correspondence transformation matrix.

[0088]

[0104] In some embodiments, the system 200 may optionally further comprise an ultrasound transducer 202 communicatively coupled to the memory 204 and the at least one processor 206. In some embodiments, the ultrasound transducer 202 may be located within a freehand ultrasound probe.

[0089]

[0105] In some embodiments, the ultrasound image generator 208 may be configured to generate a first series of 2D ultrasound images based on a first series of ultrasound waves acquired by the ultrasound transducer 202 positioned at a first plurality of positions for a first plurality of time instances, and to generate a second series of 2D ultrasound images based on a second series of ultrasound waves acquired by the ultrasound transducer 202 positioned at a second plurality of positions for a second plurality of time instances.

[0090]

[0106] Alternatively or additionally, in some embodiments, the memory 204 may be configured to store first image data related to a first series of ultrasound waves acquired by an ultrasound transducer (e.g., ultrasound transducer 202) positioned at a first plurality of positions for a first plurality of time instances and to store second image data related to a second series of ultrasound waves acquired by an ultrasound transducer (e.g., ultrasound transducer 202) positioned at a second plurality of positions for a second plurality of time instances, and the at least one processor 206 (e.g., 2D ultrasound image generator 208) may be configured to generate a first series of 2D ultrasound images based on the first image data stored in the memory 204 and to generate a second series of 2D ultrasound images based on the second image data stored in the memory 204.

[0091]

[0107] Those skilled in the art will understand that the above-mentioned modules are not necessarily separate modules, and that one or more modules can be realized by or implemented as one functional module (e.g., a circuit or software program) if desired or appropriate, without departing from the scope of the present invention. For example, the 2D ultrasound image generator 208, the 2D representation generator 210, the simulated 2D representation generator 212, the 2D and simulated 2D representation processor 214, the corresponding transformation matrix determiner 216, and / or the corresponding location determiner 218 can be realized (e.g., compiled together) as one executable software program (e.g., referred to as a software application, or simply "app"), and such software program can be stored, for example, in the memory 204 and can be executable by at least one processor 206 to perform the functions / operations described herein according to various embodiments.

[0092]

[0108] 1, and thus various functions or operations configured to be performed by at least one processor 206 may correspond to various steps of the aforementioned method 100 according to various embodiments, and thus need not be repeated for system 200 for the sake of clarity and brevity. In other words, various embodiments described herein in the context of a method are equally valid for the respective system or device, and vice versa.

[0093]

[0109] For example, in various embodiments, the memory 204 may store a 2D ultrasound image generator 208, a 2D representation generator 210, a simulated 2D representation generator 212, a 2D and simulated 2D representation processor 214, a correspondence transformation matrix determiner 216, and / or a correspondence location determiner 218, each of which corresponds to various steps of the aforementioned method 100 executable by at least one processor 206 to perform corresponding functions / operations described herein.

[0094]

[0110] In the present disclosure, various embodiments may provide a computing system, controller, microcontroller, or any other system that provides processing capabilities. Such a system may include one or more processors and one or more computer-readable storage media. For example, the above-mentioned system 200 may include a processor (or controller) 206 and a computer-readable storage medium (or memory) 204, which are used in various processes performed by the system 200, for example, as described herein. The memory or computer-readable storage medium used in various embodiments may be a volatile memory, such as a DRAM (dynamic random access memory), or a non-volatile memory, such as a PROM (programmable read-only memory), an EPROM (erasable PROM), an EEPROM (electrically erasable PROM), or a flash memory, such as a floating gate memory, a charge trap memory, an MRAM (magnetoresistive random access memory), or a PCRAM (phase change random access memory).

[0095]

[0111] In various embodiments, a "circuit" can be understood as any kind of logic implementation entity, and can be a dedicated circuit or a processor that executes software, firmware, or any combination thereof stored in memory. Thus, in one embodiment, a "circuit" can be a hardwired logic circuit or a programmable logic circuit, such as a programmable processor, for example a microprocessor (for example, a complex instruction set computer (CISC) processor or a reduced instruction set computer (RISC) processor). A "circuit" can also be a processor that executes software, for example, any kind of computer program, for example, a computer program using virtual machine code, for example, Java. According to various alternative embodiments, any other kind of implementation of the respective functions described in more detail below can also be understood as a "circuit". Similarly, a "module" can be a part of a system according to various embodiments of the present invention, and can include the above-mentioned "circuit", or can be understood as any kind of logic implementation entity therefrom.

[0096]

[0112] Some portions of this disclosure are explicitly or implicitly presented in terms of algorithms and functions or symbolic representations of operations on data within a computer memory. These algorithmic descriptions and functions or symbolic representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is generally conceived herein to be a self-consistent sequence of steps leading to a desired result. These steps are those requiring physical manipulations of physical quantities, such as electrical, magnetic, or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated.

[0097]

[0113] Unless specifically stated otherwise, as will be apparent hereinafter, throughout this specification discussions utilizing terms such as "generate," "estimate," "modify," "render," and the like will be understood to refer to operations and processes of a computer system or similar electronic device that manipulates and transforms data represented as physical quantities within a computer system into other data similarly represented as physical quantities within a computer system or other information storage, transmission, or display device.

[0098]

[0114] This specification also discloses a system, device, or apparatus for performing the operations / functions of the methods described herein. Such a system, device, or apparatus may be specially constructed for the required purposes, or may comprise a general-purpose computer or other device selectively activated or reconfigured by a computer program stored in the computer. The algorithms presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose machines may be used with computer programs in accordance with the teachings of this specification. Alternatively, the construction of more specialized apparatus to perform the required method steps may be applicable.

[0099]

[0115] In addition, this specification also at least implicitly discloses computer programs or software / functional modules, since it should be clear to those skilled in the art that individual steps of the methods described herein can be implemented by computer code. The computer program is not intended to be limited to any particular programming language and its implementation. It will be understood that various programming languages ​​and their coding can be used to implement the teachings of the disclosure contained herein. Furthermore, the computer program is not intended to be limited to any particular control flow. Different control flows can be used in many other variations of the computer program without departing from the scope of the present invention. It will be understood by those skilled in the art that the various modules described herein (e.g., 2D ultrasound image generator 208, 2D representation generator 210, simulated 2D representation generator 212, 2D and simulated 2D representation processor 214, corresponding transformation matrix determiner 216, and / or corresponding location determiner 218) can be software modules realized by computer programs or instruction sets executable by a computer processor to perform required functions, or can be hardware modules that are functional hardware units designed to perform required functions. It will also be appreciated that a combination of hardware and software modules may be implemented.

[0100]

[0116] Furthermore, one or more of the steps of the computer programs / modules or methods described herein can be executed in parallel rather than sequentially. Such computer programs can be stored in any computer-readable medium. The computer-readable medium can include storage devices such as magnetic or optical disks, memory chips, or other storage devices suitable for interfacing with a general-purpose computer. The computer programs, when loaded and executed in such a general-purpose computer, substantially produce an apparatus that implements the steps of the methods described herein.

[0101]

[0117] In various embodiments, a computer program product is provided that includes instructions embodied in one or more computer-readable storage media (non-transitory computer-readable storage media) and executable by one or more computer processors (e.g., 2D ultrasound image generator 208, 2D representation generator 210, simulated 2D representation generator 212, 2D and simulated 2D representation processor 214, correspondence transformation matrix determiner 216, and / or correspondence location determiner 218) to perform the method 100 for cross-referencing two-dimensional (2D) ultrasound scans of tissue volumes described above with reference to Fig. 1. Thus, various computer programs or modules described herein can be stored in a computer program product that can be received by a system (e.g., a computer system or electronic device), such as the system 200 shown in Fig. 2, for execution by at least one processor 206 of the system 200 to perform the required or desired functions.

[0102]

[0118] The software or functional modules described herein can also be implemented as hardware modules. More specifically, in the case of hardware, a module is a functional hardware unit designed to be used with other components or modules. For example, a module can be implemented using individual electronic components or can form part of an overall electronic circuit, such as an application specific integrated circuit (ASIC). Many other possibilities exist. Those skilled in the art will understand that the software or functional modules described herein can also be implemented as a combination of hardware and software modules.

[0103]

[0119] Those skilled in the art will understand that the system 200 can be composed of separate units or can be composed as one integrated unit. For example, in various embodiments, the system 200 can include a computer system including at least one processor 206, memory 204, 2D ultrasound image generator 208, 2D representation generator 210, simulated 2D representation generator 212, 2D and simulated 2D representation processor 214, correspondence transformation matrix determiner 216, and correspondence location determiner 218, and a separate ultrasound probe including an ultrasound transducer 202 communicatively coupled to the computer system. In other words, the separate ultrasound probe can acquire a series of ultrasound waves for a tissue volume, and then transmit the series of ultrasound waves to a different computer system (e.g., based on wireless or wired communication) to execute the method for cross-referencing two-dimensional (2D) ultrasound scans of a tissue volume described above with reference to FIG. 1. In various other embodiments, the system 200 may correspond to or be embodied as an ultrasound probe including an ultrasound transducer 202, at least one processor 206, a memory 204, a 2D ultrasound image generator 208, a 2D representation generator 210, a simulated 2D representation generator 212, a 2D and simulated 2D representation processor 214, a correspondence transformation matrix determiner 216, and a correspondence location determiner 218.

[0104]

[0120] In various embodiments, the computer system described above can be realized by any computer system (e.g., a portable or desktop computer system), such as the computer system 300 shown generally in FIG. 3 by way of example only and not limitation. Various methods / steps or functional modules (e.g., the 2D ultrasound image generator 208, the 2D representation generator 210, the simulated 2D representation generator 212, the 2D and simulated 2D representation processor 214, the corresponding transformation matrix determiner 216, and / or the corresponding location determiner 218) can be implemented as software, such as a computer program, which executes in the computer system 300 and instructs the computer system 300 (particularly one or more processors) to perform the methods / functions of the various embodiments described herein. The computer system 300 can include a computer module 302, input modules such as a keyboard 304 and a mouse 306, and multiple output devices such as a display 308 and a printer 310. The computer module 302 can be connected to a computer network 312 via a suitable transceiver device 314 to enable access to, for example, the Internet or other network systems such as a local area network (LAN) or a wide area network (WAN). The computer module 302 in this example can include a processor 318 for executing various instructions, a random access memory (RAM) 320, and a read-only memory (ROM) 322. The computer module 302 can also include a number of input / output (I / O) interfaces, for example an I / O interface 324 to the display 308 and an I / O interface 326 to the keyboard 304. The components of the computer module 302 typically communicate via an interconnected bus 328, as known to those skilled in the relevant art.

[0105]

[0121] Those skilled in the art will appreciate that the terminology used herein is for the purpose of describing various embodiments only and is not intended to limit the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising" as used herein specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0106]

[0122] 4 illustrates an exemplary sweep (or series or sequence or plurality) 402 of 2D ultrasound images (e.g., 404a, 404b, 404c, and 404d) of a tissue volume associated with multiple locations along a scan direction 408 of the tissue volume. In this regard, the series of 2D ultrasound images 402 may be generated based on a series of ultrasound waves acquired along the scan direction 408 of an ultrasound probe by an ultrasound transducer (e.g., located within an ultrasound probe 410) at multiple locations (e.g., 406a, 406b, 406c, 406d, 406e).

[0107]

[0123] In order that the present invention may be readily understood and practiced, various exemplary embodiments of the present invention are described below for purposes of illustration only and not limitation. However, those skilled in the art will appreciate that the present invention may be embodied in many different forms or configurations and should not be construed as being limited to the exemplary embodiments described below. On the contrary, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0108]

[0124] Various embodiments of the present invention disclose methods and systems that can assist a user by automatically cross-referencing points in two different views (e.g., TRX and SAG views) using information common to two orthogonal ultrasound sweeps. More specifically, various embodiments of the present invention can assist a user when he or she selects a region of interest on a first view (e.g., transverse view) to locate the corresponding region of interest on a second view (e.g., sagittal view).

[0109]

[0125] According to various embodiments, two 2D sweep scans of the 3D anatomical structure from two (nearly) orthogonal views can be provided for cross-referencing. In what follows, for simplicity and without loss of generality, it is mainly assumed that the first sweep is obtained from a transverse (TRX) view and the second sweep from a sagittal (SAG) view.

[0110]

[0126] According to various embodiments, a segmentation method may be applied to segment the boundaries of a target anatomical structure (e.g., the thyroid gland) in each 2D frame in both fields of view. In various embodiments, as used herein, the term "target anatomical structure" may refer to an anatomical structure of interest (or an internal anatomical structure), such as, for example, a body organ (e.g., the thyroid gland, prostate, bladder, heart, lungs, stomach, liver, kidneys, or other body organs) or a portion of the vasculature (e.g., the carotid artery).

[0111]

[0127] Since both views contain the same 3D anatomical structures, there is common information in these two orthogonal ultrasound sweeps. According to various embodiments, this common information can be used to find the location of any given point [idx trx ,y trx ,x trx ] to the corresponding point [idx sag ,y sag ,x sag], where idx is the frame index and x and y are the coordinates of each point. The frame index indicates the particular image frame in which each point is located in a series of 2D ultrasound images (sweep). For example, [idx trx ,y trx ,x trx ] is the index idx trx is located in the TRX frame (the frame of the TRX sweep) with coordinate y trx ,x trx , and a point having

[0112]

[0128] According to various embodiments of the present invention, a (computer-implemented) method or algorithm for calculation of a transformation for a cross-referencing tool or system can be provided that can assist a user in locating an approximate region of interest (ROI) or landmark (e.g., nodule) on a second image plane (e.g., sagittal plane) when the region of interest or landmark (e.g., nodule) has already been identified (e.g., by a user) on a first image plane (e.g., transverse plane).

[0113]

[0129] According to various embodiments, a cross-referencing tool or system can use a linear transformation (matrix) to select a particular frame index and coordinate in the sagittal (longitudinal) image based on a user-provided frame index and coordinate on the transverse image. The mapping obtained from the transformation can be used to put the transverse and sagittal image pairs into the same spatial coordinate system, allowing points in the transverse field of view to be mapped to points in the sagittal field of view.

[0114]

[0130] When invoked by a user, the method or algorithm may take as input images of two 2D ultrasound scans (e.g., a series of 2D thyroid images) performed along orthogonal scan directions, such as a transverse (TRX) sweep (comprising multiple TRX images (image frames)) and a sagittal (SAG) sweep (comprising multiple SAG images (image frames)), and generate a pair of transformation matrices (also referred to herein as corresponding transformation matrices) T trx and T sag can be output.

[0115]

[0131] Below, a pair of transformation matrices T trx and T sag Aspects of the calculation of are illustratively described in the context of 2D ultrasound imaging of a thyroid lobe, including transverse (TRX) and sagittal (SAG) sweeps. The thyroid lobe may be one example of a target anatomical structure.

[0116]

[0132] According to some embodiments, the calculation of the transformation matrix may include the following steps. 1. Segment the target anatomical structure (thyroid lobe) in all frames for both TRX and SAG sweeps to obtain a binary mask (also called binary segmentation mask) for the image frames of TRX and SAG sweeps. In the binary mask, pixel values ​​assume a first value (e.g., "1") indicating a thyroid region, or a second value (e.g., "0") indicating a non-thyroid region. The segmentation can be performed by any general-purpose segmentation method (e.g., a segmentation method based on a conventional network or a neural network) that may be known in the art.

[0117] 2. Find an optimal binary mask from the TRX sweep and find an optimal binary mask from the SAG sweep. The optimal binary mask for the TRX sweep may be the maximum binary mask among the binary masks obtained from the TRX sweep and the optimal binary mask for the SAG sweep may be the maximum binary mask among the binary masks obtained from the SAG sweep. The optimal (or maximum) binary mask for the TRX sweep may be referred to as a first 2D representation of the thyroid lobe (target anatomy) and the optimal (or maximum) binary mask for the SAG sweep may be referred to as a second 2D representation of the thyroid lobe (target anatomy).

[0118] 3. The shape of the entire stack of masks for the SAG sweep is calculated as (H sag ×W sag ×N sag ) to (H sag ×N sag ×W sag ), where N sag is the number of image frames in the SAG sweep, and H sag and W sag are the number of rows and columns in each SAG image frame, respectively. This creates a simulated stack of TRX masks from the original SAG stack. Then, similar to step 2, select the largest simulated TRX mask.

[0119] 4. Find an initial (coarse) width-to-height ratio for the simulated TRX frame. This ratio can be optimized to align the simulated TRX image with the real TRX image and later refine the ratio. In various embodiments, the width-to-height ratio can be used to scale the width of the simulated TRX mask to obtain a scaled version of the simulated TRX mask, which can be referred to as a scaled first simulated 2D representation of the thyroid lobe (target anatomy).

[0120] 5. Create a distance map for the selected TRX mask (i.e., the optimal binary mask for the TRX sweep obtained in step 2). In the distance map (also called the TRX distance map), the value of a pixel is the distance from that pixel to the nearest pixel on the edge of the thyroid lobe (target anatomy) in the TRX mask. Since using all pixels in the thyroid lobe (target anatomy) can be computationally intensive, points from within the thyroid lobe from the mock TRX mask can be sampled to be used in the next step.

[0121] 6. Solve the optimization problem that should align the previously generated images (sampled points from the TRX distance map and the simulated TRX mask) using a vectorized heuristic search method.

[0122] 7. Sample points from the edge of the thyroid lobe (target anatomical structure) mask in the simulated TRX mask and use the alignment parameters found in the previous step and an optimization algorithm such as the Gauss-Newton algorithm to minimize the alignment cost and obtain the transformation matrix T trx Find.

[0123] 8. Repeat steps (3) to (7) above from the mask of the TRX sweep to obtain the transformation matrix T sag Calculate. 9. Optional: Find the in-plane shift for all frames, which is done similarly to step 6, resulting in the matrix RC trx and thereby compensates for hand movements (of the user, e.g., clinician) that occur during a scanning procedure (using a freehand ultrasound probe).

[0124] 10.T trx and T sag (and optionally RC trx ), and any point (frame index idx trx and coordinates (x trx ,y trx ) to the corresponding point on the SAG sweep (with frame index idx sag and coordinates (xsag ,y sag ) can be mapped to the image of the TRX and SAG sweeps. Thus, cross-referencing between the images of the TRX and SAG sweeps can be realized. In other words, any point in the transverse field of view, i.e., the frame index idx trx In (x trx ,y trx ) for the cross reference point in the sagittal view, i.e. the frame index idx sag In (x sag ,y sag ) may be acquired. The frame index indicates a particular frame (2D ultrasound image frame) in a series of 2D ultrasound images (image frames) from a TRX sweep or a SAG sweep.

[0125]

[0133] A flow chart 500 illustrating aspects of a method for cross-referencing 2D ultrasound scans (eg, method 100 shown in FIG. 1) according to various embodiments will now be described with reference to FIG.

[0134] In Fig. 5, a pair of TRX and SAG view freehand ultrasound sweeps can be provided as input. See 501 and 502 on the left side of Fig. 5, (A) ("2D swept TRX", i.e. transverse scan) and (B) ("2D swept SAG", i.e. sagittal scan). This can be included in steps 102 and 104 of the method of Fig. 1. In an example, thyroid images can be captured by an ultrasound imaging device once from a first view (TRX plane) and then from a second view (SAG plane), for example, similar to that shown and described in Fig. 4(a) and (b).

[0126]

[0135] In addition, two transformation matrices T sag and T trx can be provided as an output (see 503 and 504 on the right side of Figure 5), which can be used to find the location of any point (x trx ,y trx ) and frame index idx trx , the corresponding point (x sag ,ysag ) and frame index idx sag and vice versa. This may be included in steps 112, 120, and 122 of the method of FIG.

[0127]

[0136] The mapping (cross-reference) can be expressed by the following equations (1) and (2).

[0128]

number

[0129]

[0137] Here, (idx trx ,y trx ,x trx ) denotes a point in the transverse (TRX) field of view, and (idx sag ,y sag ,x sag ) indicates the corresponding (i.e., cross-referenced) point in the sagittal (SAG) field, and T sag and T trx is the corresponding transformation matrix, and f sag and f trx is the transformation matrix T that maps positions from TRX to the SAG field of view. sag and T trx and the scaling factor ws trx ,ws sag is a mapping function for , and vice versa. methodology

[0138] The task underlying various embodiments of the present invention can be seen in aligning points of a 3D anatomical structure into two orthogonal (or substantially orthogonal) views, specifically the TRX and SAG views, as illustrated in diagram 500 shown in Figure 5. According to various embodiments, the 3D pixel-to-pixel cross-referencing problem can be decomposed into two parallel simultaneous registration problems, which can be solved simultaneously. TRX to SAG: Cross-reference from a point on the TRX field sweep to the corresponding point on the SAG field sweep (top half of Figure 500)

[0139] To use common (shared) information in the two fields of view, a "simulated version" of the target anatomical structure in the first field of view (Field of View 1, hereafter TRX) can be reconstructed from a scan of the second field of view (Field of View 2, hereafter SAG), and vice versa. The simulated field of view is obtained by projecting the first field of view (Field of View 1) into the second field of view (Field of View 2).

[0130]

[0140] After some post-processing on the simulated field of view, the simulated TRX and the original TRX are simultaneously superimposed to obtain the corresponding transformation matrix T trx can be calculated.

[0141] Similarly, a simulated SAG can be generated from a 2D sweep in TRX, and the corresponding transformation matrix T sag can be calculated.

[0131]

[0142] Referring again to the flow chart 500 shown in FIG. 5, aspects of the method will now be described in more detail. In the flow chart 500, individual steps are indicated by arrows and associated numbers, such as 1, 2, 3, 4, .... It should be understood that the numbering does not necessarily imply a chronological order, e.g., some steps (e.g., steps 1 and 4) may be performed sequentially or simultaneously, and / or some steps labeled with a "higher" number (e.g., step 11) may also be performed before or simultaneously with a step labeled with a "lower" number (e.g., step 2). Additionally, the arrows (steps) of the flow chart 500 as a whole may indicate transitions between different stages of a process flow, with each stage represented by an image labeled with a capital letter, such as "A," "B," "C," etc.

[0132]

[0143] The individual steps illustrated in flow chart 500 of FIG. 5 can be described as follows.

[0144] Step 1: Segmentation of the target anatomical structure in the first field of view (TRX field of view): Image size (height H trx , width W trx ) and the number of TRX frames N trxEach frame of the TRX field of view of a 2D sweep of the target anatomical structure having (see (A)) is fed to a segmentation method (e.g., a U-Net segmentation network) to obtain a corresponding binary segmentation mask (C). The value of pixels in the segmentation mask is 1 (corresponding to the thyroid region) or 0 (corresponding to the non-thyroid region). By way of example, pixels in (C) corresponding to the target anatomical structure (thyroid region) are white, and pixels corresponding to the non-thyroid region are black. Step 1 can be included in step 106 of the method 100 shown in FIG. 1.

[0133]

[0145] Step 2: Find the maximum (optimal) TRX segmentation mask (G) among the generated masks (C) representing the target anatomical structure. Here, the term "optimal" refers to the segmentation mask that has the maximum segmentation mask of the target anatomical structure. Step 2 can be included in step 106 of the method 100 shown in FIG. 1.

[0134]

[0146] Step 3: Generate a signed distance map (K) from the segmentation mask (G). To obtain the signed distance map (K), first the boundary contour of the segmentation mask (G) (i.e., the boundary contour of the target anatomical structure (thyroid region) in the segmentation mask (G)) can be determined. The value of each pixel in the distance map is the distance from that pixel to its closest contour pixel. The value of the pixel in the distance map corresponding to the contour pixel is 0. To obtain the signed distance map, the thyroid segmentation mask (G) can be converted to a signed map (hereinafter referred to as "signed (G)"), where thyroid pixels and non-thyroid pixels have values ​​of -1 and 1 instead of 0 and 1. The signed distance map (K) is then generated by element-wise multiplication of the signed map (G) and the distance map. Step 3 can be included in step 110 of the method 100 shown in FIG. 1.

[0135]

[0147] Step 4: Segmentation of the target anatomical structure in the second view (SAG view): (Hsag ,W sag ,N sag A 2D sweep of the SAG field (B) having a shape of a thyroid region is fed into the SAG segmentation method to obtain a corresponding binary segmentation mask (D) in the SAG field. As in (C), pixels in (D) that correspond to the target anatomical structure (thyroid region) are white, and pixels that correspond to non-thyroid regions are black. Step 4 may be included in step 114 of the method 100 shown in FIG. 1.

[0136]

[0148] Step 5: Generate a simulated TRX segmentation and from the generated SAG mask (D), (H sag ,N sag ) to extract a maximum simulated TRX segmentation mask (H). Step 5 may be included in step 108 of the method 100 shown in FIG.

[0137]

[0149] Step 6: Width (N) of the simulated TRX mask (H) sag ) coarse scale factor ws trx The rough estimation is based on the fact that the width-to-height ratio (WoH) of the thyroid gland in the real TRX frame (G) and the simulated TRX frame (I) should be the same.

[0138]

[0150] In an ultrasound 2D sweep, the in-plane pixel spacing (width and height) in the scanned frame is known, but the out-of-plane pixel spacing between two consecutive frames is unknown, which mainly depends on the movement direction and speed of the ultrasound probe. It is the height (H sag ) is probably approximately correct, and the scale factor (ws trx ) by width (N sag ) may need to be scaled. The goal of this step is to scale the width (N sag ) coarse scale factor ws trxThe coarse estimation is based on the fact that the width-to-height ratio (WoH) of the thyroid gland in the real TRX frame (G) and the simulated TRX frame (I) should be the same. Step 6 can be included in step 110 of the method 100 shown in FIG.

[0139]

[0151] Step 7: Extract and sample contour pixels (M) from the simulated TRX segmentation mask (I). Given the mask (I), first its contours are extracted, and then the longest contour is kept. The obtained longest contour is a chain of one pixel width, and its pixel coordinates are arranged and stored in a matrix having a shape of (2, num_pixels), where num_pixels indicates the number of pixels in the longest contour. Then, a subset (e.g., 50) of the obtained num_pixels contour pixels can be sampled for further edge-based alignment in step 10. In (M), the sampled contour pixels are shown with white points. Step 7 can be included in step 110 of the method 100 shown in FIG. 1.

[0140]

[0152] Step 8: Sample region anchor pixels (L) from (I). Due to the large number of foreground pixels in (I), a subset of them is sampled. For example, foreground pixels whose both row and column indices can be divided by a given number n (e.g., but not limited to, n=25) can be kept. In (L), the sampled pixels are shown with white dots (L). Step 8 can be included in step 110 of the method 100 shown in FIG. 1.

[0141]

[0153] Step 9: Calculate the coarse shift (δR) between the thyroid mask in (I) and the thyroid mask in (G) by maximizing the alignment response calculated from (L) and (K) using the following equation (3): trx , δC trx ) is estimated.

[0142]

number

[0143] where (r,c) are the row and column numbers of foreground pixels in (L). Step 9 may be included in step 110 of the method 100 shown in FIG.

[0154] Step 10: Using the following equation (4), the transformation matrix T trx Estimate the scale and translation between the segmentation masks in (I) and (G) by minimizing the registration cost, which is defined as the sum of squared distances over

[0144]

number

[0145] where J is the distance map (see Figure 6) and W is the transformation T trx This is a warping (transformation) of points on the target anatomical structure boundary (i.e., the white points on image (M) in Figure 5) by

[0146]

[0155] Therefore, the transformation matrix T trx Step 10 may be included in step 112 of the method 100 shown in FIG.

[0156] The optimization problem (i.e., minimizing the registration cost) can be solved, for example, by the Gauss-Newton algorithm, in the same or similar manner as shown, for example, by the pseudocode below ("Algorithm 1: Optimization by the Gauss-Newton Method").

[0147]

[0157] In pseudocode, deltaR trx is the above δR trx Corresponding to deltaC trx is the above δC trx corresponds to the transformation T trx 4 shows warping of boundary points by

[0148]

number

[0149] SAG to TRX: (bottom half of Figure 500)

[0158] These steps are similar to those for the top half of diagram 500, but from SAG to TRX.

[0150]

[0159] Step 11: (H sag ,W sag ,N sag From a stack of SAG segmentation masks (D) with the (overall) shape of (H sag ,W sag ) to extract an optimal real SAG field segmentation mask (Q) having the shape of . This step is similar to step 2. Step 11 may be included in step 114 of the method 100 shown in FIG.

[0151]

[0160] Step 12: Generate a signed distance map (T) based on the mask boundary of (Q). The distance map generation algorithm is the same as that used in 3. Step 12 can be included in step 118 of the method 100 shown in FIG.

[0152]

[0161] Step 13: (H trx ,W trx ,N trx From the stack of TRX segmentation masks (C) with the (overall) shape of (H trx ,N trx 1. Extract an optimal simulated SAG field segmentation mask (P) having the shape of (P=1, P=1, P=2, P=1). The optimal frame extraction algorithm is the same as that in step 5. Step 13 can be included in step 116 of the method 100 shown in FIG. 1.

[0153]

[0162] Step 14: Rough scale factor ws of the simulated SAG mask (P) along its width direction sag Then, ws sagThe coarse estimation is based on the fact that the width-to-height ratio (WoH) of the thyroid gland in the real SAG frame (Q) and the simulated SAG frame (P) should be the same. Step 14 can be included in step 118 of the method 100 shown in FIG. 1.

[0154]

[0163] Step 15: Similar to step 7, extract and sample boundary pixels (contour pixels) (V) from the selected simulated SAG thyroid segmentation mask (R). In (V), the sampled pixels are shown with white points. Step 15 can be included in step 118 of the method 100 shown in FIG. 1.

[0155]

[0164] Step 16: Sample region anchor pixel (U) for (R). The sampling strategy is the same as that used in step 8. In (U), the sampled pixel is shown with a white point. Step 16 can be included in step 118 of the method 100 shown in FIG.

[0156]

[0165] Step 17: Calculate the coarse shift (δR) between the thyroid mask in (T) and the thyroid mask in (Q) by maximizing the alignment response calculated in (U) and (T). sag , δC sag ) This step is the same as that used in step 9. Step 17 may be included in step 118 of the method 100 shown in FIG.

[0157]

[0166] Step 18: Using the following equation (5), the transformation matrix T sag We estimate the scale and translation between the segmentation masks of (R) and (Q) by minimizing the registration cost, which is defined as the sum of squared distances over

[0158]

number

[0159] where S is the distance map (see Figure 6) and W is the transformation T sag The warping of points on the target anatomical structure boundary (i.e., the white points in image (V) of Figure 5) by

[0167] The optimization problem is solved by the same algorithm used in step 10. Thus, the transformation matrix T sag Step 18 may be included in step 120 of the method 100 shown in FIG. Final Mapping Equation

[0168] Calculated T trx and T sag Therefore, any point (frame index idx) from the TRX sweep can be calculated using the following equations (6) to (7). trx and coordinates (x trx ,y trx )) to the corresponding point on the SAG sweep (frame index idx sag and coordinates (x sag ,y sag )).

[0160]

number

[0161]

[0169] This may be included in step 122 of the method 100 shown in Figure 1. Thus, cross-referencing between the two views may be achieved. By way of example, Equation (6) corresponds to one embodiment of Equation (1) above, and Equation (7) corresponds to one embodiment of Equation (2) above.

[0162]

[0170] FIG. 6 shows another flow chart 600 illustrating aspects of a method for cross-referencing 2D ultrasound scans (e.g., method 100 shown in FIG. 1) according to various embodiments, and FIGS. 7 and 8 show portions of the flow chart shown in FIG. 6 in more detail.

[0163]

[0171] Flowchart 600 is somewhat similar to flow chart 500 shown in Figure 5, in particular the steps and stages of flow chart 600 that have the same reference numbers or labels as those shown in flow chart 500 of Figure 5 are the same as those in flow chart 500 and will not be repeated in detail here. Instead, the differences compared to flow chart 500 shown in Figure 5 are primarily described below.

[0164]

[0172] 6 and 7, in the flow chart 600, step 5 (of the flow chart 500) may include steps (5a) and (5b), such that the transition from stage (D) to stage (H) passes through an intermediate stage (F). FIG. 7 shows that the intermediate stage (F), obtained in step (5a) and corresponding to the maximum simulated TRX segmentation mask from the sagittal (SAG) segmentation sweep, contains a (small) false positive region (indicated by reference number 701), i.e. a region of the segmentation mask (F) that is erroneously marked as corresponding to the target anatomical structure (thyroid lobe). This false positive region 701 can be removed by post-processing.

[0165]

[0173] Due to the complex structure of the thyroid region and the relatively low signal-to-noise ratio of ultrasound images, false positive thyroid segmentation regions (such as region 701 in FIG. 7) may sometimes be present in the selected frame. In various embodiments, as shown in step (5b), a maximum connected component search algorithm can be used to filter out these small regions that are not connected to the main thyroid segmentation (region 702 in FIG. 7) in the selected thyroid segmentation mask. That is, in step (5b), the maximum connected component (also called maximum adjacent component, maximum connected component, or maximum cluster) of the mask can be obtained, and thus noise can be removed. Maximum connected component search algorithms are known as such in the field of image processing and therefore will not be described in detail here. As a non-limiting example of such an algorithm, the so-called OpenCV cv2.connectedComponentsWithStats algorithm can be used.

[0166]

[0174] Similar to the method described above for step 5, step 13 may include steps (13a) and (13b) in flowchart 600, where step (13a) results in a maximum simulated SAG segmentation mask from a transverse (TRX) segmentation sweep (stage (N) of flowchart 600) and step (13b) results in the maximum component of the mask (stage (P) of flowchart 600).

[0167]

[0175] Again, similar to the method described above for step 5, step 2 may include steps (2a) and (2b) in flowchart 600, where step (2a) results in a maximum transverse (TRX) segmentation mask from the TRX segmentation sweep (step (E) of flowchart 600) and step (2b) results in the maximum component of the mask (step (G) of flowchart 600).

[0168]

[0176] Finally, similar to the method described above for step 5, step 11 can include steps (11a) and (11b) in flowchart 600, where step (11a) results in a maximum sagittal (SAG) segmentation mask from the SAG segmentation sweep (stage (O) of flowchart 600) and step (11b) results in the maximum component of the mask (stage (Q) of flowchart 600).

[0169]

[0177] Therefore, the method illustrated in flowchart 600 may include steps of further refining the binary segmentation masks (E), (F), (N), and (O) obtained from the TRX and SAG sweeps.

[0170]

[0178] 6 and 8, in flowchart 600, step 3 (of flowchart 500) may include steps (3a) and (3b), such that the transition from stage (G) to stage (K) passes through intermediate stage (J).

[0171]

[0179] In step (3a), a distance map (J) may be generated based on the mask boundary of the binary segmentation mask (G). Given the binary segmentation mask (G), the contour of the mask may be searched by a suitable contour search algorithm that may be known in the art. The longest contour may then be kept. Based on this longest contour, a distance map (J) may be generated. The value of each pixel in the distance map (J) is the distance from that pixel to its closest contour pixel. Points on the contour boundary have a value of 0 in the distance map.

[0172]

[0180] In step (3b), the distance map (J) may be converted to a signed distance map (K). To obtain the signed distance map (K), first the binary segmentation mask (G) may be converted to a signed map, where thyroid and non-thyroid pixels may have values ​​of -1 and 1 instead of 0 and 1. The signed distance map (K) may then be generated by element-wise multiplication of the signed map (G) and the distance map (J).

[0173]

[0181] Similar to the method described above for step 3, step 12 may include steps (12a) and (12b) in the flowchart 600, where step (12a) generates a distance map (S) based on a mask boundary of the binary segmentation mask (Q), and step (12b) converts the distance map (S) into a signed distance map (T).

[0174]

[0182] According to various embodiments, optionally, one or more of the following steps may be included in the flow diagram 500 or the flow diagram 600 to further enhance cross-referencing.

[0175]

[0183] In optional step 19 (not shown), an initial transformation matrix T sag and T trx can be used to further refine the in-plane shift between the real and simulated images.

[0184] The step is the frame-by-frame in-plane shift ((2,N trx ) RC with the shape trx ) in theory. The scanning trajectories of both the TRX sweep and the SAG sweep are assumed to be straight lines. However, it may be difficult for sonographers and radiologists to strictly follow a straight line when scanning the TRX sweep. Therefore, a module for estimating the frame-by-frame in-plane shift of the TRX frames can be added to improve the superposition of each pair of real and simulated TRX frames.

[0176]

[0185] In particular, for each TRX segmentation mask (H trx ,W trx ) shape (E')), the corresponding simulated TRX mask ((H sag ,N sag (F')) having the shape of

[0177]

[0186] Then, (G') and (L') can be obtained according to steps (2b) and 8, respectively. Then, the shift of the TRX frame (F') relative to its corresponding real TRX frame (E') can be obtained by performing a heuristic search algorithm based alignment of (G') and (L'), which is similar to the algorithm used in step 9 (where the signed distance map (K) is replaced by a binary mask (G')).

[0178]

[0187] Then, the in-plane shift (RC trx ) based on the RC of the real SAG frame to its corresponding simulated SAG frame. sag can be obtained for the inverse transformation.

[0188] In another optional step 20 (not shown), the T obtained in step 18 is calculated by performing an intensity-based Lucas-Kanade algorithm. sagThis step is to reduce the aperture problem that may be caused by the sweep due to incomplete target anatomy scanning (e.g., the thyroid gland is too long to be covered by one frame in the SAG field of view in the vertical direction). To reduce the effect of this problem, first, corresponding images ((N') and (O')) of the simulated SAG mask (N) and the real SAG mask (O) can be extracted from (A) and (B). Then, the scale factor ws sag We can resize (N') along its width to (R') by optimizing the sum of squared differences of the two images (N') and (O') using the following equation: sag can be improved.

[0179]

number

[0180]

[0189] If optional steps 19 and 20 have been performed, the improved cross-referencing equation is: Final mapping equation (with additional steps 19 and 20)

[0190] Steps 19 and 20 result in (ws trx ,T trx ,RC trx ) and (ws sag ,T sag ,RC sag ) is then obtained. Then, for any point (frame index idx trx and coordinates (x trx ,y trx )) to the corresponding point on the SAG sweep (frame index idx sag and (x sag ,y sag )).

[0181]

number

[0182]

[0191] For example, Equation (9) corresponds to another embodiment of Equation (1) above, and Equation (10) corresponds to another embodiment of Equation (2) above. Thus, cross-referencing between the two fields of view (TRX and SAG) can be realized.

[0183]

[0192] FIG. 9 shows a screenshot of a first to second view cross-referencing tool or system according to various embodiments of the present invention, where by selecting any point on the transverse scan (e.g., cursor indicator (point P1) on a thyroid nodule in the inferior lobe of the thyroid gland in the transverse view in the left panel), the cross-referencing tool or system locates the corresponding nodule in the sagittal scan (i.e., point P2 with corresponding x,y location and frame number in the sagittal sweep in the right panel).

[0184]

[0193] FIG. 10 shows a screenshot of a cross-referencing system from a second view to a first view according to various embodiments of the present invention, where by selecting another thyroid nodule in the upper lobe of the thyroid gland on the sagittal sweep (point P3 on the right panel), the cross-referencing tool locates that nodule (corresponding x,y position and frame number) on the transverse sweep (point P4 on the left panel).

[0185]

[0194] According to one aspect of the present invention, a method and system are provided for cross-referencing ultrasound sweeps from two orthogonal planes based on a segmentation mask.

[0195] According to another aspect of the present invention, a non-transitory computer-readable storage medium is provided having instructions executed by a processor to receive a pair of ultrasound scans, apply a series of transformations and post-processing to the pair of ultrasound scans, and create a transformation mapping from the two paired scans to form a final probability segmentation.

[0186]

[0196] According to various embodiments, a (computer-implemented) method or algorithm can be provided that can compute a transformation that outputs a particular frame index and coordinate in a second view (e.g., sagittal view) based on a frame index and coordinate on a first view (e.g., transverse view) provided by a user. The algorithm can use a linear transformation to put the transverse and sagittal image pairs into the same spatial coordinate system. This can allow a user to locate a region of interest (e.g., a nodule) in the sagittal image using a region of interest defined in the transverse image.

[0187]

[0197] According to various embodiments, the redundancy of two orthogonal freehand ultrasound sweep acquisitions can be exploited, and since they are considered to be in-plane, out-of-plane distances in one sweep can be accurately retrieved from another sweep. To do this, various embodiments provide a method for overlaying the two acquisitions.

[0188]

[0198] According to various embodiments, a cross-referencing module can be provided that calculates two transformation matrices and then maps frame indices and coordinates on the transverse image provided by the user to frame indices and coordinates in the sagittal (longitudinal) image.

[0189]

[0199] The mapping is based on two transformation matrices, which can be used to put transverse and sagittal image pairs into the same spatial coordinate system, thereby allowing transverse field points to be mapped to sagittal field points and vice versa.

[0190]

[0200] Although embodiments of the present invention have been particularly shown and described with reference to specific embodiments, it will be understood by those skilled in the art that various changes in form and detail can be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the present invention is therefore indicated by the appended claims, and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced.

Claims

1. 1. A method for cross-referencing two-dimensional (2D) ultrasound scans of a tissue volume, comprising: generating a first series of 2D ultrasound images of a tissue volume associated with a first plurality of locations, each along a first scan direction of the tissue volume, the tissue volume including a target anatomical structure; generating a second series of 2D ultrasound images of the tissue volume associated with a second plurality of locations, respectively, along a second scanning direction of the tissue volume, the second scanning direction being at least substantially orthogonal to the first scanning direction; generating a first 2D representation of the target anatomical structure from the first series of 2D ultrasound images, the first 2D representation being associated with one of the images in the first series of 2D ultrasound images; generating a first simulated 2D representation of the target anatomy from the second series of 2D ultrasound images, the first simulated 2D representation of the target anatomy and the first 2D representation corresponding to each other in terms of a view direction relative to the target anatomy; processing the first 2D representation of the target anatomical structure and the first simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the first simulated 2D representation with pixel locations associated with the target anatomical structure in the first 2D representation; determining a first correspondence transformation matrix from the processed first 2D representation and the processed first simulated 2D representation of the target anatomical structure; generating a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, the second 2D representation being associated with one of the images in the second series of 2D ultrasound images; generating a second simulated 2D representation of the target anatomy from the first series of 2D ultrasound images, the second simulated 2D representation of the target anatomy and the second 2D representation corresponding to each other in terms of a view direction relative to the target anatomy; processing the second 2D representation of the target anatomical structure and the second simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the second simulated 2D representation with pixel locations associated with the target anatomical structure in the second 2D representation; determining a second correspondence transformation matrix from the processed second 2D representation and the processed second simulated 2D representation of the target anatomical structure; determining, for a first location associated with the target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images using the first correspondence transformation matrix and the second correspondence transformation matrix; The method includes:

2. determining the second location using the first corresponding transformation matrix and the second corresponding transformation matrix, 2. The method of claim 1, comprising applying a first function of the first corresponding transformation matrix and the second corresponding transformation matrix to pixel coordinates corresponding to the first location to obtain pixel coordinates corresponding to the second location.

3. 3. The method of claim 1 or 2, further comprising: determining, for a third location associated with the target anatomical structure in the first series of 2D ultrasound images, a corresponding fourth location associated with the target anatomical structure in the second series of 2D ultrasound images using the first and second correspondence transformation matrices.

4. determining the fourth location using the first corresponding transformation matrix and the second corresponding transformation matrix, applying a second function of the first corresponding transformation matrix and the second corresponding transformation matrix to pixel coordinates corresponding to the third location to obtain pixel coordinates corresponding to the fourth location. The method according to claim 3.

5. generating a first plurality of binary segmentation masks from the first series of 2D ultrasound images, each binary segmentation mask in the first plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images in the first series of 2D ultrasound images; generating a second plurality of binary segmentation masks from the second series of 2D ultrasound images, each binary segmentation mask in the second plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images in the second series of 2D ultrasound images; Further comprising: the first 2D representation of the target anatomical structure is generated from one of the binary segmentation masks of the first plurality of binary segmentation masks; the second 2D representation of the target anatomical structure is generated from one of the binary segmentation masks of the second plurality of binary segmentation masks. The method according to any one of claims 1 to 4.

6. the first 2D representation of the target anatomical structure corresponds to a largest binary segmentation mask among the first plurality of binary segmentation masks; the second 2D representation of the target anatomical structure corresponds to a largest binary segmentation mask among the second plurality of binary segmentation masks. The method according to claim 5.

7. generating the first simulated 2D representation of the target anatomical structure from the second series of 2D ultrasound images includes generating the first simulated 2D representation of the target anatomical structure from the second plurality of binary segmentation masks; generating the second simulated 2D representation of the target anatomical structure from the first series of 2D ultrasound images includes generating the second simulated 2D representation of the target anatomical structure from the first plurality of binary segmentation masks. The method according to claim 5 or 6.

8. processing the first 2D representation includes generating a first signed distance map from the first 2D representation; and processing the second 2D representation includes generating a second signed distance map from the second 2D representation. The method according to any one of claims 5 to 7.

9. processing the first simulated 2D representation includes scaling a width of the first simulated 2D representation to match a width of the first 2D representation; processing the second simulated 2D representation includes scaling a width of the second simulated 2D representation to match a width of the second 2D representation. The method according to claim 8.

10. processing the first simulated 2D representation further comprises determining and sampling contour pixels associated with a contour of the target anatomical structure from the scaled first simulated 2D representation; and processing the second simulated 2D representation further comprises determining and sampling contour pixels associated with a contour of the target anatomical structure from the scaled second simulated 2D representation.

10. The method of claim 9.

11. the step of processing the first simulated 2D representation further comprises sampling a first subset of pixels from the scaled first simulated 2D representation; and processing the second simulated 2D representation further comprises sampling a second subset of pixels from the scaled second simulated 2D representation. The method of claim 10.

12. processing the first simulated 2D representation further comprises estimating a coarse shift between the scaled first simulated 2D representation and the first 2D representation using the first signed distance map and the first sampled subset of pixels; and processing the second simulated 2D representation further comprises using the second signed distance map and the sampled second subset of pixels to estimate a coarse shift between the scaled second simulated 2D representation and the second 2D representation. The method of claim 11.

13. the step of determining the first correspondence transformation matrix comprises estimating a scale and a translation between the scaled first simulated 2D representation and the first 2D representation by minimizing a first alignment cost defined as a first cost function for the first correspondence transformation matrix; and determining the second correspondence transformation matrix includes estimating a scale and a translation between the scaled second simulated 2D representation and the second 2D representation by minimizing a second alignment cost defined as a second cost function for the second correspondence transformation matrix. The method of claim 12.

14. The method according to any one of claims 1 to 13, wherein the first scanning direction corresponds to a transverse scan of the tissue volume and the second scanning direction corresponds to a sagittal scan of the tissue volume.

15. 1. A system for cross-referencing two-dimensional (2D) ultrasound scans of a tissue volume, comprising: Memory, at least one processor communicatively coupled to the memory; wherein the at least one processor: generating a first series of 2D ultrasound images of a tissue volume associated with a first plurality of locations, each along a first scan direction of the tissue volume, the tissue volume including the target anatomical structure; generating a second series of 2D ultrasound images of the tissue volume associated with a second plurality of locations, respectively, along a second scanning direction of the tissue volume, the second scanning direction being at least substantially orthogonal to the first scanning direction; generating a first 2D representation of the target anatomical structure from the first series of 2D ultrasound images, the first 2D representation being associated with one of the images of the first series of 2D ultrasound images; generating a first simulated 2D representation of the target anatomy from the second series of 2D ultrasound images, wherein the first simulated 2D representation of the target anatomy and the first 2D representation correspond to each other in terms of a view direction relative to the target anatomy; processing the first 2D representation of the target anatomical structure and the first simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the first simulated 2D representation with pixel locations associated with the target anatomical structure in the first 2D representation; determining a first correspondence transformation matrix from the processed first 2D representation and the processed first simulated 2D representation of the target anatomical structure; generating a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, the second 2D representation being associated with one of the images in the second series of 2D ultrasound images; generating a second simulated 2D representation of the target anatomy from the first series of 2D ultrasound images, the second simulated 2D representation of the target anatomy and the second 2D representation corresponding to each other in terms of a view direction relative to the target anatomy; processing the second 2D representation of the target anatomical structure and the second simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the second simulated 2D representation with pixel locations associated with the target anatomical structure in the second 2D representation; determining a second correspondence transformation matrix from the processed second 2D representation and the processed second simulated 2D representation of the target anatomical structure; and and determining, for a first location associated with the target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images using the first correspondence transformation matrix and the second correspondence transformation matrix. system.

16. When the at least one processor determines the second location using the first corresponding transformation matrix and the second corresponding transformation matrix, the at least one processor: configured to apply a first function of the first corresponding transformation matrix and the second corresponding transformation matrix to pixel coordinates corresponding to the first location to obtain pixel coordinates corresponding to the second location. The system of claim 15.

17. The at least one processor: and further configured to determine, for a third location associated with the target anatomical structure in the first series of 2D ultrasound images, a corresponding fourth location associated with the target anatomical structure in the second series of 2D ultrasound images using the first and second correspondence transformation matrices.

17. A system according to claim 15 or 16.

18. When the at least one processor determines the fourth location using the first corresponding transformation matrix and the second corresponding transformation matrix, the at least one processor: configured to apply a second function of the first corresponding transformation matrix and the second corresponding transformation matrix to pixel coordinates corresponding to the third location to obtain pixel coordinates corresponding to the fourth location.

20. The system of claim 17.

19. The at least one processor: generating a first plurality of binary segmentation masks from the first series of 2D ultrasound images, each binary segmentation mask in the first plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images in the first series of 2D ultrasound images; generating a second plurality of binary segmentation masks from the second series of 2D ultrasound images, each binary segmentation mask in the second plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images in the second series of 2D ultrasound images; [0023] 20. The method according to claim 1, further comprising: the first 2D representation of the target anatomical structure is generated from one of the binary segmentation masks of the first plurality of binary segmentation masks; a second 2D representation of the target anatomical structure is generated from one of the binary segmentation masks of the second plurality of binary segmentation masks. A system according to any one of claims 15 to 18.

20. 20. The system of claim 19, wherein the first 2D representation of the target anatomical structure corresponds to a largest binary segmentation mask among the first plurality of binary segmentation masks and the second 2D representation of the target anatomical structure corresponds to a largest binary segmentation mask among the second plurality of binary segmentation masks.

21. The system of any one of claims 15 to 20, wherein the first scanning direction corresponds to a transverse scan of the tissue volume and the second scanning direction corresponds to a sagittal scan of the tissue volume.

22. further comprising an ultrasound transducer communicatively coupled to the memory and the at least one processor; The at least one processor: generating the first series of 2D ultrasound images based on a first series of ultrasound waves acquired by the ultrasound transducers positioned at the first multiple locations for a first multiple time instances; generating the second series of 2D ultrasound images based on a second series of ultrasound waves acquired by the ultrasound transducers positioned at the second multiple locations for a second multiple time instances; configured to: A system according to any one of claims 15 to 21.

23. 1. A computer program product embodied in one or more non-transitory computer-readable storage media and including instructions executable by at least one processor to perform a method for cross-referencing two-dimensional (2D) ultrasound scans of a tissue volume, the method comprising: generating a first series of 2D ultrasound images of a tissue volume associated with a first plurality of locations, each along a first scan direction of the tissue volume, the tissue volume including a target anatomical structure; generating a second series of 2D ultrasound images of the tissue volume associated with a second plurality of locations, respectively, along a second scanning direction of the tissue volume, the second scanning direction being at least substantially orthogonal to the first scanning direction; generating a first 2D representation of the target anatomical structure from the first series of 2D ultrasound images, the first 2D representation being associated with one of the images in the first series of 2D ultrasound images; generating a first simulated 2D representation of the target anatomy from the second series of 2D ultrasound images, the first simulated 2D representation of the target anatomy and the first 2D representation corresponding to each other in terms of a view direction relative to the target anatomy; processing the first 2D representation of the target anatomical structure and the first simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the first simulated 2D representation with pixel locations associated with the target anatomical structure in the first 2D representation; determining a first correspondence transformation matrix from the processed first 2D representation and the processed first simulated 2D representation of the target anatomical structure; generating a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, the second 2D representation being associated with one of the images in the second series of 2D ultrasound images; generating a second simulated 2D representation of the target anatomy from the first series of 2D ultrasound images, the second simulated 2D representation of the target anatomy and the second 2D representation corresponding to each other in terms of a view direction relative to the target anatomy; processing the second 2D representation of the target anatomical structure and the second simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the second simulated 2D representation with pixel locations associated with the target anatomical structure in the second 2D representation; determining a second correspondence transformation matrix from the processed second 2D representation and the processed second simulated 2D representation of the target anatomical structure; determining, for a first location associated with the target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images using the first correspondence transformation matrix and the second correspondence transformation matrix; 2. A computer program product comprising:

Images