Method and system for cross-referencing two-dimensional (2D) ultrasound scans of tissue volumes

Abstract

A method for cross-referencing 2D ultrasound scans of a tissue volume includes generating first and second 2D representations of a target anatomical structure from a first and second series of 2D ultrasound images of the generated tissue volume, respectively. The method includes generating first and second simulated 2D representations of the target anatomical structure from the second and first series of 2D ultrasound images, respectively. The 2D and simulated 2D representations are processed to at least substantially align pixel locations in the 2D and simulated 2D representations associated with the target anatomical structure. The method includes determining first and second correspondence transformation matrices from the processed 2D and simulated 2D representations, and using the first and second correspondence transformation matrices to determine corresponding locations in the first series of 2D ultrasound images for locations in the second series of 2D ultrasound images associated with the target anatomical structure.

Description

[Technical Field]

[0001] Related Applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 323,515, filed March 25, 2022, entitled "Method and System for Cross-Referencing of Two-Dimensional (2D) Ultrasound Scans of a Tissue Volume," which is incorporated herein by reference in its entirety.

[0002]

[0002] The present invention relates generally to methods and systems for cross-referencing two-dimensional (2D) ultrasound scans of a tissue volume, and more particularly to methods and systems for cross-referencing two sweeps of 2D ultrasound images of a tissue volume obtained from orthogonal or substantially orthogonal fields of view or planes. [Background technology]

[0003]

[0003] Two-dimensional (2D) ultrasound (US) imaging is safe, inexpensive, and widely used in medical practice, as well as having real-time, high-resolution capabilities. Conventional 2D ultrasound imaging techniques can be configured to extract 2D ultrasound images (sometimes called cross-sectional images, image planes / frames, or B-mode / B-scan images) of an anatomical structure or tissue volume scanned by an ultrasound probe.

[0004] However, conventional 2D ultrasound imaging techniques have the inherent limitation of relying on 2D images to depict three-dimensional (3D) anatomical structures. The transducer may often be manually manipulated (moved) by an operator to acquire a 2D image (or series of 2D images, also known as a 2D sweep or freehand sweep) of a body organ that covers the 3D anatomical structure.

[0005]

[0005] In 3D imaging diagnostics (e.g., magnetic resonance imaging (MRI) or computed tomography (CT)), images can be acquired with known 3D pixel spacing, allowing multiplanar reformatting imaging, allowing the scan data to be visualized in the transverse (or axial), coronal, or sagittal planes, depending on the diagnostic task. However, in freehand ultrasound, only pixel spacing within the imaging plane (xy) is available, and pixel spacing in the out-of-plane direction (z direction) may not be available. Therefore, if the ultrasound scan is performed in the transverse plane, simply projecting the image in the sagittal field of view can lead to a sagittal scan of poor quality and low resolution, which may not be usable for diagnostic imaging.

[0006]

[0006] An essential limitation of ultrasound imaging is that it can have high resolution in the imaging plane (high axial and lateral resolution), but not high resolution in the out-of-plane direction. Therefore, in standard clinical evaluations, it is common to perform two sweeps of 2D ultrasound images from orthogonal directions to better capture 3D anatomical structures with high resolution. While this can be useful for acquiring images, aligning these two views and finding corresponding points in another view is a difficult problem because freehand sweeps are performed by hand and there are no reference points to align the images. This problem is also known as superimposing two views in the fields of computer vision and medical imaging.

[0007]

[0007] In clinical practice, images from two views are typically cross-referenced (or overlaid) in the mind of an expert radiologist / sonographer to form a subjective impression of 3D anatomy and / or pathology. This practice is time-consuming and inefficient, places a high cognitive load on the radiologist, and often leads to variability and inconsistency in measurements and diagnoses.

[0008]

[0008] This problem is further illustrated herein below in the context of thyroid ultrasound without loss of generality, with reference to FIG.

[0009] Figure 4 shows, in (a) and (b), a schematic diagram of the (human) thyroid gland (labeled "T"). The arrows in (a) and (b) indicate the scanning direction of the ultrasound probe in the transverse (TRX) and sagittal (SAG) planes, respectively. (c) and (d) are sample TRX and SAG thyroid ultrasound sweeps with manually labeled thyroid lobe boundary overlays (outlines labeled "Cl" and "C2").

[0009]

[0010] As part of a typical thyroid ultrasound workflow, a clinician can sequentially scan both the left and right lobes of the thyroid gland. Scanning of each side (e.g., the left lobe shown in FIG. 4) can include scanning in the transverse (TRX) plane (shown in FIG. 4(a)) and the sagittal (SAG) plane (shown in FIG. 4(b)). For example, scanning of each side can be performed first in the transverse (TRX) plane, followed by a scan in the sagittal (SAG) plane.

[0010]

[0011] The clinician can then view the patient's scan, which includes a transverse (TRX) sweep and a sagittal (SAG) sweep of the thyroid gland. The transverse image (such as the image shown in Figure 4(c)) may often be inspected first for the appearance of nodules or suspicious lesions. When nodules are found, it may be necessary to manually locate each nodule in the corresponding image acquired along the sagittal view (such as the image shown in Figure 4(d)) and inspect it in both views. This may require the clinician to scroll through all frames of the SAG scan to locate the same nodule. Regions of interest (ROIs) or landmarks may then be annotated and measured in both views and recorded for later examination. This task is cognitively demanding and time-consuming. Summary of the Invention [Problem to be solved by the invention]

[0011]

[0012] Accordingly, there is a need to provide a method and system for cross-referencing two-dimensional (2D) ultrasound scans of tissue volumes that overcomes, or at least ameliorates, one or more of the deficiencies associated with conventional methods and systems, particularly with respect to freehand ultrasound scanning of tissue volumes. It is against this background that the present invention has been developed. [Means for solving the problem]

[0012]

[0013] According to a first aspect of the present invention, there is provided a method for cross-referencing two-dimensional (2D) ultrasound scans of tissue volumes, comprising: generating a first series of 2D ultrasound images of a tissue volume associated with a first plurality of locations, each image along a first scan direction of the tissue volume, the tissue volume including the target anatomical structure; generating a second series of 2D ultrasound images of the tissue volume associated with the second plurality of locations, each image being along a second scanning direction of the tissue volume, the second scanning direction being at least substantially orthogonal to the first scanning direction; generating a first 2D representation of the target anatomical structure from a first series of 2D ultrasound images, the first 2D representation being associated with one of the images in the first series of 2D ultrasound images; generating a first simulated 2D representation of the target anatomy from the second series of 2D ultrasound images, wherein the first simulated 2D representation of the target anatomy and the first 2D representation correspond to each other in terms of a view direction relative to the target anatomy; processing the first 2D representation of the target anatomy and the first simulated 2D representation to at least substantially align pixel locations associated with the target anatomy in the first simulated 2D representation with pixel locations associated with the target anatomy in the first 2D representation; determining a first correspondence transformation matrix from the processed first 2D representation and the processed first simulated 2D representation of the target anatomical structure; generating a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, the second 2D representation being associated with one of the images in the second series of 2D ultrasound images; generating a second simulated 2D representation of the target anatomy from the first series of 2D ultrasound images, wherein the second simulated 2D representation of the target anatomy and the second 2D representation correspond to each other in terms of a view direction relative to the target anatomy; processing the second 2D representation of the target anatomy and the second simulated 2D representation to at least substantially align pixel locations associated with the target anatomy in the second simulated 2D representation with pixel locations associated with the target anatomy in the second 2D representation; determining a second correspondence transformation matrix from the processed second 2D representation and the processed second simulated 2D representation of the target anatomical structure; and determining, for a first location associated with the target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images using the first correspondence transformation matrix and the second correspondence transformation matrix.

[0013]

[0014] In various embodiments, determining the second location using the first corresponding transformation matrix and the second corresponding transformation matrix includes: Applying a first function of the first corresponding transformation matrix and the second corresponding transformation matrix to pixel coordinates corresponding to the first location to obtain pixel coordinates corresponding to the second location.

[0014]

[0015] In various embodiments, the method comprises: The method further includes determining, for a third location associated with the target anatomical structure in the first series of 2D ultrasound images, a corresponding fourth location associated with the target anatomical structure in the second series of 2D ultrasound images using the first correspondence transformation matrix and the second correspondence transformation matrix.

[0015]

[0016] In various embodiments, determining the fourth location using the first corresponding transformation matrix and the second corresponding transformation matrix includes: Applying a second function of the first corresponding transformation matrix and the second corresponding transformation matrix to pixel coordinates corresponding to the third location to obtain pixel coordinates corresponding to a fourth location.

[0016]

[0017] In various embodiments, the method comprises: generating a first plurality of binary segmentation masks from the first series of 2D ultrasound images, each binary segmentation mask in the first plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images in the first series of 2D ultrasound images; generating a second plurality of binary segmentation masks from the second series of 2D ultrasound images, each binary segmentation mask of the second plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images of the second series of 2D ultrasound images; a first 2D representation of the target anatomical structure is generated from one of the binary segmentation masks of the first plurality of binary segmentation masks; A second 2D representation of the target anatomical structure is generated from one of the binary segmentation masks of the second plurality of binary segmentation masks.

[0017]

[0018] In various embodiments, the first 2D representation of the target anatomical structure corresponds to a largest binary segmentation mask among the first plurality of binary segmentation masks, and the second 2D representation of the target anatomical structure corresponds to a largest binary segmentation mask among the second plurality of binary segmentation masks.

[0018]

[0019] In various embodiments, generating a first simulated 2D representation of the target anatomical structure from the second series of 2D ultrasound images includes generating the first simulated 2D representation of the target anatomical structure from a second plurality of binary segmentation masks, and generating a second simulated 2D representation of the target anatomical structure from the first series of 2D ultrasound images includes generating the second simulated 2D representation of the target anatomical structure from the first plurality of binary segmentation masks.

[0019]

[0020] In various embodiments, processing the first 2D representation includes generating a first signed distance map from the first 2D representation, and processing the second 2D representation includes generating a second signed distance map from the second 2D representation.

[0020]

[0021] In various embodiments, processing the first simulated 2D representation includes scaling a width of the first simulated 2D representation to match a width of the first 2D representation, and processing the second simulated 2D representation includes scaling a width of the second simulated 2D representation to match a width of the second 2D representation.

[0021]

[0022] In various embodiments, processing the first simulated 2D representation further includes determining and sampling contour pixels associated with the contour of the target anatomical structure from the scaled first simulated 2D representation, and processing the second simulated 2D representation further includes determining and sampling contour pixels associated with the contour of the target anatomical structure from the scaled second simulated 2D representation.

[0022]

[0023] In various embodiments, processing the first simulated 2D representation further includes sampling a first subset of pixels from the scaled first simulated 2D representation, and processing the second simulated 2D representation further includes sampling a second subset of pixels from the scaled second simulated 2D representation.

[0023]

[0024] In various embodiments, processing the first simulated 2D representation further includes estimating a coarse shift between the scaled first simulated 2D representation and the first 2D representation using the first signed distance map and the pixels of the first sampled subset, and processing the second simulated 2D representation further includes estimating a coarse shift between the scaled second simulated 2D representation and the second 2D representation using the second signed distance map and the pixels of the second sampled subset.

[0024]

[0025] In various embodiments, determining the first correspondence transformation matrix includes estimating a scale and translation between the scaled first simulated 2D representation and the first 2D representation by minimizing a first alignment cost defined as a first cost function for the first correspondence transformation matrix, and determining the second correspondence transformation matrix includes estimating a scale and translation between the scaled second simulated 2D representation and the second 2D representation by minimizing a second alignment cost defined as a second cost function for the second correspondence transformation matrix.

[0025]

[0026] In various embodiments, the first scanning direction corresponds to a transverse scan of the tissue volume and the second scanning direction corresponds to a sagittal scan of the tissue volume.

[0027] According to a second aspect of the present invention, there is provided a system for cross-referencing two-dimensional ultrasound scans of tissue volumes, comprising: Memory and and at least one processor communicatively coupled to the memory and the ultrasound transducer, wherein the at least one processor: generating a first series of 2D ultrasound images of a tissue volume associated with a first plurality of locations, each along a first scan direction of the tissue volume, the tissue volume including the target anatomical structure; generating a second series of 2D ultrasound images of the tissue volume associated with the second plurality of locations, each image being along a second scanning direction of the tissue volume, the second scanning direction being at least substantially orthogonal to the first scanning direction; generating a first 2D representation of the target anatomical structure from the first series of 2D ultrasound images, the first 2D representation being associated with one of the images in the first series of 2D ultrasound images; generating a first simulated 2D representation of the target anatomy from the second series of 2D ultrasound images, wherein the first simulated 2D representation of the target anatomy and the first 2D representation correspond to each other in terms of a viewing direction relative to the target anatomy; processing the first 2D representation of the target anatomy and the first simulated 2D representation to at least substantially align pixel locations associated with the target anatomy in the first simulated 2D representation with pixel locations associated with the target anatomy in the first 2D representation; determining a first correspondence transformation matrix from the processed first 2D representation and the processed first simulated 2D representation of the target anatomical structure; generating a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, the second 2D representation being associated with one of the images in the second series of 2D ultrasound images; generating a second simulated 2D representation of the target anatomy from the first series of 2D ultrasound images, wherein the second simulated 2D representation of the target anatomy and the second 2D representation correspond to each other in terms of a viewing direction relative to the target anatomy; processing the second 2D representation of the target anatomical structure and the second simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the second simulated 2D representation with pixel locations associated with the target anatomical structure in the second 2D representation; determining a second correspondence transformation matrix from the processed second 2D representation and the processed second simulated 2D representation of the target anatomical structure; and A system is provided that is configured to determine, for a first location associated with a target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images using a first correspondence transformation matrix and a second correspondence transformation matrix.

[0026]

[0028] In various embodiments, when the at least one processor determines the second location using the first corresponding transformation matrix and the second corresponding transformation matrix, the at least one processor: A first function of the first corresponding transformation matrix and the second corresponding transformation matrix is ​​configured to apply to pixel coordinates corresponding to the first location to obtain pixel coordinates corresponding to the second location.

[0027]

[0029] In various embodiments, at least one processor: The system is further configured to determine, for a third location associated with the target anatomical structure in the first series of 2D ultrasound images, a corresponding fourth location associated with the target anatomical structure in the second series of 2D ultrasound images using the first and second correspondence transformation matrices.

[0028]

[0030] In various embodiments, when the at least one processor determines the fourth location using the first corresponding transformation matrix and the second corresponding transformation matrix, the at least one processor: A second function of the first corresponding transformation matrix and the second corresponding transformation matrix is ​​configured to apply to pixel coordinates corresponding to the third location to obtain pixel coordinates corresponding to the fourth location.

[0029]

[0031] In various embodiments, at least one processor: generating a first plurality of binary segmentation masks from the first series of 2D ultrasound images, each binary segmentation mask of the first plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images of the first series of 2D ultrasound images; generating a second plurality of binary segmentation masks from the second series of 2D ultrasound images, each binary segmentation mask of the second plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images of the second series of 2D ultrasound images; a first 2D representation of the target anatomical structure is generated from one of the binary segmentation masks of the first plurality of binary segmentation masks; A second 2D representation of the target anatomical structure is generated from one of the binary segmentation masks of the second plurality of binary segmentation masks.

[0030]

[0032] In various embodiments, the first 2D representation of the target anatomical structure corresponds to a largest binary segmentation mask among the first plurality of binary segmentation masks, and the second 2D representation of the target anatomical structure corresponds to a largest binary segmentation mask among the second plurality of binary segmentation masks.

[0031]

[0033] In various embodiments, the first scanning direction corresponds to a transverse scan of the tissue volume and the second scanning direction corresponds to a sagittal scan of the tissue volume.

[0034] In various embodiments, the system further comprises an ultrasound transducer communicatively coupled to the memory and the at least one processor, the at least one processor configured to generate a first series of 2D ultrasound images based on a first series of ultrasound waves acquired by the ultrasound transducer positioned at a first plurality of positions for a first plurality of time instances, and to generate a second series of 2D ultrasound images based on a second series of ultrasound waves acquired by the ultrasound transducer positioned at a second plurality of positions for a second plurality of time instances.

[0032]

[0035] According to a third aspect of the present invention there is provided a computer program product embodied in one or more non-transitory computer readable storage media and comprising instructions executable by at least one processor to perform a method for cross-referencing two-dimensional ultrasound scans of tissue volumes, the method comprising: generating a first series of 2D ultrasound images of a tissue volume associated with a first plurality of locations, each image along a first scan direction of the tissue volume, the tissue volume including the target anatomical structure; generating a second series of 2D ultrasound images of the tissue volume associated with the second plurality of locations, each image being along a second scanning direction of the tissue volume, the second scanning direction being at least substantially orthogonal to the first scanning direction; generating a first 2D representation of the target anatomical structure from a first series of 2D ultrasound images, the first 2D representation being associated with one of the images in the first series of 2D ultrasound images; generating a first simulated 2D representation of the target anatomy from the second series of 2D ultrasound images, wherein the first simulated 2D representation of the target anatomy and the first 2D representation correspond to each other in terms of a view direction relative to the target anatomy; processing the first 2D representation of the target anatomy and the first simulated 2D representation to at least substantially align pixel locations associated with the target anatomy in the first simulated 2D representation with pixel locations associated with the target anatomy in the first 2D representation; determining a first correspondence transformation matrix from the processed first 2D representation and the processed first simulated 2D representation of the target anatomical structure; generating a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, the second 2D representation being associated with one of the images in the second series of 2D ultrasound images; generating a second simulated 2D representation of the target anatomy from the first series of 2D ultrasound images, wherein the second simulated 2D representation of the target anatomy and the second 2D representation correspond to each other in terms of a view direction relative to the target anatomy; processing the second 2D representation of the target anatomy and the second simulated 2D representation to at least substantially align pixel locations associated with the target anatomy in the second simulated 2D representation with pixel locations associated with the target anatomy in the second 2D representation; determining a second correspondence transformation matrix from the processed second 2D representation and the processed second simulated 2D representation of the target anatomical structure; and determining, for a first location associated with the target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images using the first correspondence transformation matrix and the second correspondence transformation matrix.

[0033]

[0036] It should be noted that the various embodiments described above can be combined with any other embodiment described herein. The features and advantages described herein are not all-inclusive, and many additional features and advantages will be apparent to those skilled in the art, especially upon consideration of the drawings, specification, and claims. Furthermore, it should be noted that the language used herein has been chosen primarily for purposes of readability and instruction, and may not be chosen to delineate or limit the subject matter of the invention.

[0034]

[0037] Embodiments of the present invention will be better understood and readily apparent to those skilled in the art from the following description, given by way of example only, in conjunction with the drawings in which: [Brief explanation of the drawings]

[0035] [Figure 1]

[0038] 1 is a schematic flow diagram of a method for cross-referencing 2D ultrasound scans of tissue volumes according to various embodiments of the present invention. [Figure 2]

[0039] 2 is a schematic block diagram of a system for cross-referencing 2D ultrasound scans of tissue volumes according to various embodiments of the present invention, such as corresponding to the method shown in FIG. 1; [Figure 3]

[0040] FIG. 3 is a schematic block diagram of an exemplary computer system that can be used to realize or implement a system for cross-referencing 2D ultrasound scans of tissue volumes according to various embodiments of the present invention, such as the system shown in FIG. [Figure 4]

[0041] FIG. 1 is a schematic diagram of a human thyroid gland and sample transverse and sagittal thyroid ultrasound sweep images. [Figure 5]

[0042] 1 is a flow chart illustrating aspects of a method for cross-referencing 2D ultrasound scans of tissue volumes according to various embodiments of the present invention. [Figure 6]

[0043] 10 is another flow diagram illustrating aspects of a method for cross-referencing 2D ultrasound scans of tissue volumes according to various embodiments of the present invention. [Figure 7]

[0044] FIG. 7 shows a portion of the flow chart shown in FIG. 6 in more detail. [Figure 8]

[0045] FIG. 7 illustrates another portion of the flow chart shown in FIG. 6 in more detail. [Figure 9]

[0046] 1A-1C illustrate screenshots of a first view to second view cross-referencing tool or system according to various embodiments of the present invention. [Figure 10]

[0047] 1A-1C are screenshots of a second view to first view cross-referencing system according to various embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0036]

[0048] Various embodiments of the present invention provide methods (computer-implemented methods) and systems (including a memory and at least one processor communicatively coupled to the memory) for cross-referencing two-dimensional (2D) ultrasound scans of a tissue volume, and more particularly, methods and systems for cross-referencing two sweeps of 2D ultrasound images of a tissue volume obtained from orthogonal or substantially orthogonal fields of view or planes.

[0037]

[0049] As described in the Background Art, two-dimensional (2D) ultrasound imaging is safe, inexpensive, and widely used in medical practice. However, traditional ultrasound imaging involves the mental superposition (or cross-referencing) of images acquired from two different fields of view by an expert radiologist / sonographer to form a subjective impression of 3D anatomy and / or lesions, a process that is time-consuming and inefficient, places a high cognitive load on the radiologist, and often leads to variability and incorrect diagnosis.

[0038]

[0050] Thus, various embodiments of the present invention provide methods and systems that enable more efficient and / or reliable cross-referencing (registration) of two 2D ultrasound scans of a tissue volume obtained from two different views. Cross-referencing can find corresponding points between one plane and another. In other words, cross-referencing can map a given point on one plane or view (e.g., a transverse (TRX) view) to a corresponding point on another plane or view (e.g., a sagittal (SAG) view).

[0039]

[0051] Thus, various embodiments of the present invention can bridge the gap between 2D ultrasound imaging and 3D imaging (e.g., computed tomography (CT) or magnetic resonance imaging (MRI)) by providing correspondence between points in different fields of view. Example clinical applications include, but are not limited to, cardiac, vascular imaging, prostate, bladder, and thyroid imaging.

[0040]

[0052] Various embodiments of the present invention can benefit clinical workflow by simplifying the measurement, screening, and / or monitoring of organs over time, due to faster and more reliable alignment of anatomical structures.

[0041]

[0053] FIG. 1 illustrates a schematic flow diagram of a method 100 (a computer-implemented method) for cross-referencing two-dimensional (2D) ultrasound scans of a tissue volume (including a target anatomical structure) using at least one processor, according to various embodiments of the present invention.

[0042]

[0054] The method 100 includes generating 102 a first series (or sequence or set) of 2D ultrasound images (sometimes referred to interchangeably as cross-sectional images, image planes, image frames / slices, or B-mode / B-scan images) of a tissue volume associated with a first plurality of locations, each along a first scanning direction of the tissue volume, the tissue volume including a target anatomical structure.

[0043]

[0055] The method 100 further includes generating 104 a second series (or sequence or set) of 2D ultrasound images of the tissue volume associated with the second plurality of locations, respectively, along a second scanning direction of the tissue volume, the second scanning direction being at least substantially orthogonal to the first scanning direction. In various embodiments, as used herein, the term "at least substantially orthogonal" in relation to two directions (e.g., the "first scanning direction" and the "second scanning direction") may be understood to mean that the two directions are orthogonal to each other, or nearly orthogonal to each other, e.g., deviating from orthogonality by 10 degrees or less, e.g., 5 degrees or less, e.g., 2 degrees or less, e.g., 1 degree or less.

[0044]

[0056] In connection with step 102, each of the first series of 2D ultrasound images may be generated based on a first series of ultrasound waves acquired by an ultrasound transducer positioned at a first plurality of positions for a first plurality of time instances. Alternatively, each of the first series of 2D ultrasound images may be generated based on first image data stored in a memory related to the first series of ultrasound waves acquired by an ultrasound transducer positioned at a first plurality of positions for a first plurality of time instances.

[0045]

[0057] Similarly, in connection with step 104, each of the second series of 2D ultrasound images may be generated based on the first series of ultrasound waves acquired by an ultrasound transducer positioned at a second plurality of positions for a second plurality of time instances. Alternatively, each of the second series of 2D ultrasound images may be generated based on second image data stored in memory related to the second series of ultrasound waves acquired by an ultrasound transducer positioned at a second plurality of positions for the first plurality of time instances.

[0046]

[0058] In various embodiments, in connection with steps 102 and 104, first and second series of ultrasound waves can be acquired (in time sequence) at a first plurality of positions along a first scanning direction and a second plurality of positions along a second scanning direction in the tissue volume using an ultrasound transducer configured to emit ultrasound waves into a plane of the tissue volume (e.g., a cross section perpendicular to the scanning direction) and acquire ultrasound waves reflected from such plane of the tissue volume. Such an ultrasound transducer can be referred to as a 2D ultrasound transducer.

[0047]

[0059] Various embodiments of the present invention are directed to cross-referencing 2D ultrasound scans or sweeps acquired by freehand ultrasound scanning of a tissue volume. In this regard, an operator can perform an ultrasound scan of a tissue volume by moving a 2D ultrasound transducer (or a portable, handheld ultrasound probe equipped with a 2D ultrasound transducer) along a scanning direction of the tissue volume (e.g., along an axis across the length of the tissue volume), whereby the 2D ultrasound transducer acquires a series of ultrasound waves at multiple positions along the scanning direction for multiple time instances. The ultrasound waves received at each time instance (corresponding position) can then be processed to generate a 2D ultrasound image associated with the corresponding position, as is known in the art and need not be described in detail herein. Thus, a series of 2D ultrasound images of the tissue volume can be acquired, each 2D ultrasound image having an associated position (e.g., tagged or labeled with associated position information), which corresponds, for example, to the position of the 2D ultrasound transducer at which the ultrasound waves (based on which the 2D ultrasound image is generated) were acquired, or to the position / location along the tissue volume from which the ultrasound waves acquired by the 2D ultrasound transducer were reflected.

[0048]

[0060] The 2D ultrasound transducer can be any conventional 2D ultrasound transducer configured to emit and acquire ultrasound waves relative to a plane of a tissue volume, and therefore need not be described in detail herein. For example, without limitation, a conventional 2D ultrasound transducer can include an array of transducer elements configured to emit and acquire ultrasound waves relative to a plane of a tissue volume. Therefore, those skilled in the art will understand that the present invention is not limited to any particular type of 2D ultrasound transducer.

[0049]

[0061] The method 100 further includes a step 106 of generating a first 2D representation of the target anatomical structure from the first series of 2D ultrasound images, the first 2D representation being associated with one of the images in the first series of 2D ultrasound images.

[0050]

[0062] The method 100 further includes a step 108 of generating a first simulated 2D representation of the target anatomy from the second series of 2D ultrasound images, wherein the first simulated 2D representation of the target anatomy and the first 2D representation correspond to each other in terms of a viewing direction relative to the target anatomy.

[0051]

[0063] The method 100 further includes a step 110 of processing the first 2D representation of the target anatomical structure and the first simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the first simulated 2D representation with pixel locations associated with the target anatomical structure in the first 2D representation. In various embodiments, the term "at least substantially match" as used herein in relation to pixel positions of two different 2D entities (e.g., the processed first 2D representation and the first simulated 2D representation) may be understood to mean that the position (e.g., 2D coordinate) of a pixel (or a plurality of pixels, e.g., all of the pixels) in one of the two 2D entities (e.g., the processed first simulated 2D representation) matches or nearly matches the position (e.g., 2D coordinate) of a corresponding pixel (or a plurality of corresponding pixels) in the other of the two 2D entities (e.g., the processed first 2D representation), e.g., the position deviation (e.g., between x coordinates and / or y coordinates) is not more than a given number of pixel spacings, e.g., not more than 10 pixel spacings, e.g., not more than 5 pixel spacings, e.g., not more than 2 pixel spacings, e.g., not more than 1 pixel spacing.

[0052]

[0064] The method 100 further includes determining 112 a first corresponding transformation matrix from the processed first 2D representation and the processed first simulated 2D representation of the target anatomy.

[0065] The method 100 further includes a step 114 of generating a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, the second 2D representation being associated with one of the images in the second series of 2D ultrasound images.

[0053]

[0066] The method 100 further includes a step 116 of generating a second simulated 2D representation of the target anatomy from the first series of 2D ultrasound images, wherein the second simulated 2D representation of the target anatomy and the second 2D representation correspond to each other in terms of a viewing direction relative to the target anatomy.

[0054]

[0067] The method 100 further includes a step 118 of processing the second 2D representation of the target anatomical structure and the second simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the second simulated 2D representation with pixel locations associated with the target anatomical structure in the second 2D representation.

[0055]

[0068] The method 100 further includes determining 120 a second corresponding transformation matrix from the processed second 2D representation and the processed second simulated 2D representation of the target anatomy.

[0069] The method 100 further includes step 122 of determining, for a first location associated with the target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images using the first and second correspondence transformation matrices. In various embodiments, the first location can be a point selected by a user in one of the images (image frames) of the second series of 2D ultrasound images, and the second location can be a point obtained (cross-referenced) with the aid of the first and second transformation matrices in one of the images (image frames) of the first series of 2D ultrasound images.

[0056]

[0070] In various embodiments, in connection with step 122, the step of determining the second location using the first corresponding transformation matrix and the second corresponding transformation matrix includes applying a first function of the first corresponding transformation matrix and the second corresponding transformation matrix to pixel coordinates corresponding to the first location to obtain pixel coordinates corresponding to the second location.

[0057]

[0071] In various embodiments, method 100 may further include determining, for a third location associated with the target anatomical structure in the first series of 2D ultrasound images, a corresponding fourth location associated with the target anatomical structure in the second series of 2D ultrasound images using the first and second correspondence transformation matrices. In various embodiments, the third location may be a point selected by a user in one of the images (image frames) of the first series of 2D ultrasound images, and the fourth location may be a point obtained (cross-referenced) with the aid of the first and second transformation matrices in one of the images (image frames) of the second series of 2D ultrasound images.

[0058]

[0072] In various embodiments, the step of determining the fourth location using the first corresponding transformation matrix and the second corresponding transformation matrix includes applying a second function of the first corresponding transformation matrix and the second corresponding transformation matrix to pixel coordinates corresponding to the third location to obtain pixel coordinates corresponding to the fourth location.

[0059]

[0073] In various embodiments, the method 100 may further include generating a first plurality of binary segmentation masks from the first series of 2D ultrasound images, each binary segmentation mask of the first plurality of binary segmentation masks corresponding to a respective 2D ultrasound image of the first series of 2D ultrasound images. In various embodiments, the first 2D representation of the target anatomical structure may be generated from one of the binary segmentation masks of the first plurality of binary segmentation masks.

[0060]

[0074] In various embodiments, the method 100 may further include generating a second plurality of binary segmentation masks from the second series of 2D ultrasound images, each binary segmentation mask of the second plurality of binary segmentation masks corresponding to a respective 2D ultrasound image of the second series of 2D ultrasound images. In various embodiments, the second 2D representation of the target anatomical structure may be generated from one of the binary segmentation masks of the second plurality of binary segmentation masks.

[0061]

[0075] In various embodiments, the first plurality of binary segmentation masks and / or the second plurality of binary segmentation masks can be obtained by using segmentation methods or processes known in the field of digital image processing. One example of a suitable segmentation method that can be used by various embodiments is the so-called U-Net, a convolutional neural network developed for biomedical image segmentation. In digital image processing and computer vision, image segmentation refers to the process of dividing a digital image into multiple segments (pixel sets, also known as image objects). More specifically, image segmentation refers to the process of assigning a label (or value) to every pixel in an image so that pixels with the same label (or value) share certain characteristics.

[0062]

[0076] In various embodiments, in each binary segmentation mask, the values ​​of pixels associated with the target anatomical structure may exhibit a first value (e.g., "1"), and the values ​​of pixels not associated with the target anatomical structure may exhibit a second value (e.g., "0") different from the first value.

[0063]

[0077] In various embodiments, the first 2D representation of the target anatomical structure may correspond to a largest binary segmentation mask of the first plurality of binary segmentation masks.

[0064]

[0078] In various embodiments, the second 2D representation of the target anatomical structure may correspond to a largest binary segmentation mask of the second plurality of binary segmentation masks.

[0065]

[0079] In various embodiments, as used herein, the term "largest binary segmentation mask" may be understood to mean a binary segmentation mask having the largest number of pixels associated with the target anatomical structure among the plurality of binary segmentation masks. For example, the largest binary segmentation mask may be a binary segmentation mask having the largest number of pixels having a first value (e.g., "1") among the plurality of binary segmentation masks.

[0066]

[0080] In various embodiments, in connection with step 106, generating the first 2D representation includes removing regions of a largest binary segmentation mask in the first plurality of binary segmentation masks that were erroneously marked as corresponding to the target anatomical structure. For example, a maximum connected component search algorithm may be applied to determine the largest connected component (largest cluster) in the largest binary segmentation mask, and regions that are not connected to the largest connected component may be removed from the mask.

[0067]

[0081] In various embodiments, generating the second 2D representation, in connection with step 114, includes removing regions of a largest binary segmentation mask from the second plurality of binary segmentation masks that were erroneously marked as corresponding to the target anatomical structure. For example, a maximum connected component search algorithm may be applied to determine the largest connected component (largest cluster) in the largest binary segmentation mask, and regions that are not connected to the largest connected component may be removed from the mask.

[0068]

[0082] In various embodiments, in connection with step 108, generating a first simulated 2D representation of the target anatomical structure from the second series of 2D ultrasound images includes generating the first simulated 2D representation of the target anatomical structure from a second plurality of binary segmentation masks. In various embodiments, regions of the first simulated 2D representation that are erroneously marked as corresponding to the target anatomical structure can be removed. For example, a maximum connected component search algorithm can be applied to determine the maximum connected components (largest clusters), and regions that are not connected to the maximum connected components can be removed.

[0069]

[0083] In various embodiments, in connection with step 116, generating a second simulated 2D representation of the target anatomical structure from the first series of 2D ultrasound images includes generating the second simulated 2D representation of the target anatomical structure from the first plurality of binary segmentation masks. In various embodiments, regions of the second simulated 2D representation that are erroneously marked as corresponding to the target anatomical structure can be removed. For example, a maximum connected component search algorithm can be applied to determine the maximum connected components (largest clusters), and regions that are not connected to the maximum connected components can be removed.

[0070]

[0084] In various embodiments, processing the first 2D representation, in connection with step 110, includes generating a first signed distance map from the first 2D representation. In various embodiments, generating the first signed distance map includes determining a boundary contour of the first 2D representation (e.g., a largest binary segmentation mask among the first plurality of binary segmentation masks), generating a first distance map in which the value of each pixel is the distance from that pixel to its nearest contour pixel, converting the first 2D representation (e.g., the largest binary segmentation mask) to the first signed map, and generating the first signed distance map by element-wise multiplication of the first signed map and the first distance map.

[0071]

[0085] In various embodiments, processing the second 2D representation, in connection with step 118, includes generating a second signed distance map from the second 2D representation. In various embodiments, generating the second signed distance map includes determining a boundary contour of the 2D representation (e.g., a largest binary segmentation mask among the second plurality of binary segmentation masks), generating a second distance map in which the value of each pixel is the distance from that pixel to its nearest contour pixel, converting the 2D representation (e.g., the largest binary segmentation mask) to the second signed map, and generating the second signed distance map by element-wise multiplication of the second signed map and the second distance map.

[0072]

[0086] In various embodiments, in connection with step 110, processing the first simulated 2D representation includes scaling a width of the first simulated 2D representation to match a width of the first 2D representation.

[0073]

[0087] In various embodiments, in connection with step 118, processing the second simulated 2D representation includes scaling a width of the second simulated 2D representation to match a width of the second 2D representation.

[0074]

[0088] In various embodiments, in connection with step 110, processing the first simulated 2D representation further includes determining and sampling contour pixels associated with the contour of the target anatomical structure from the scaled first simulated 2D representation.

[0075]

[0089] In various embodiments, in connection with step 118, processing the second simulated 2D representation further includes determining and sampling contour pixels associated with the contour of the target anatomical structure from the scaled second simulated 2D representation.

[0076]

[0090] In various embodiments, in connection with step 110, processing the first simulated 2D representation further includes sampling a first subset of pixels from the scaled first simulated 2D representation.

[0077]

[0091] In various embodiments, in connection with step 118, processing the second simulated 2D representation further includes sampling a second subset of pixels from the scaled second simulated 2D representation.

[0078]

[0092] In various embodiments, in connection with step 110, processing the first simulated 2D representation further includes estimating a coarse shift between the scaled first simulated 2D representation and the first 2D representation using the first signed distance map and the pixels of the first sampled subset.

[0079]

[0093] In various embodiments, in connection with step 118, processing the second simulated 2D representation further includes estimating a coarse shift between the scaled second simulated 2D representation and the second 2D representation using the second signed distance map and the sampled second subset of pixels.

[0080]

[0094] In various embodiments, relating to step 112, determining the first correspondence transformation matrix includes estimating the scale and transformation between the scaled first simulated 2D representation and the first 2D representation by minimizing a first registration cost defined as a first cost function for the first correspondence transformation matrix. In various embodiments, the first registration cost is defined as a sum of squared distances for the first correspondence transformation matrix.

[0081]

[0095] In various embodiments, relating to step 120, determining the second correspondence transformation matrix includes estimating the scale and transformation between the scaled second simulated 2D representation and the second 2D representation by minimizing a second registration cost defined as a second cost function for the second correspondence transformation matrix. In various embodiments, the second registration cost is defined as a sum of squared distances for the second correspondence transformation matrix.

[0082]

[0096] In various embodiments, minimizing the first registration cost and / or minimizing the second registration cost includes applying an optimization algorithm, for example, a Gauss-Newton algorithm.

[0083]

[0097] In various embodiments, the first scanning direction corresponds to a transverse scan of the tissue volume and the second scanning direction corresponds to a sagittal scan of the tissue volume.

[0098] In various embodiments, the target anatomical structure includes at least one of a body organ, a portion of a body organ (eg, a thyroid lobe), and a portion of the vasculature.

[0084]

[0099] In various embodiments, the body organ comprises at least one of the thyroid, prostate, bladder, heart, lung, stomach, liver, and kidney.

[0100] FIG. 2 shows a schematic block diagram of a system 200 for cross-referencing two-dimensional (2D) ultrasound scans of tissue volumes according to various embodiments of the present invention, corresponding to the method 100 for cross-referencing two-dimensional (2D) ultrasound scans of tissue volumes using at least one processor described above according to various embodiments of the present invention.

[0085]

[0101] The system 200 comprises a memory 204 and at least one processor 206 communicatively coupled to the memory 204, the at least one processor 206 being configured to generate a first series of 2D ultrasound images of a tissue volume associated with a first plurality of locations, each along a first scanning direction of the tissue volume, the tissue volume including the target anatomical structure; and generate a second series of 2D ultrasound images of a tissue volume associated with a second plurality of locations, each along a second scanning direction of the tissue volume, the second scanning direction being at least substantially the same as the first scanning direction. generating a first 2D representation of the target anatomical structure from a first series of 2D ultrasound images, the first 2D representation being associated with one of the images in the first series of 2D ultrasound images; generating a first simulated 2D representation of the target anatomical structure from a second series of 2D ultrasound images, the first simulated 2D representation of the target anatomical structure and the first 2D representation corresponding to each other in terms of a viewing direction relative to the target anatomical structure; and generating a pixel location associated with the target anatomical structure in the first simulated 2D representation, the pixel location associated with the target anatomical structure in the first simulated 2D representation being associated with a target anatomical structure in the first 2D representation. processing the first 2D representation and the first simulated 2D representation of the target anatomy to at least substantially align pixel locations associated with the target anatomy; determining a first correspondence transformation matrix from the processed first 2D representation and the processed first simulated 2D representation of the target anatomy; generating a second 2D representation of the target anatomy from the second series of 2D ultrasound images, the second 2D representation being associated with one of the images in the second series of 2D ultrasound images; generating a representation, wherein the second simulated 2D representation of the target anatomy and the second 2D representation correspond to each other in terms of a viewing direction relative to the target anatomy; processing the second 2D representation of the target anatomy and the second simulated 2D representation to at least substantially align pixel locations associated with the target anatomy in the second simulated 2D representation with pixel locations associated with the target anatomy in the second 2D representation; determining a second correspondence transformation matrix from the processed second 2D representation of the target anatomy and the processed second simulated 2D representation;and determining, for a first location associated with the target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images using the first correspondence transformation matrix and the second correspondence transformation matrix.

[0086]

[0102] Those skilled in the art will appreciate that the at least one processor 206 may be configured to perform the required functions or operations via a set of instructions (e.g., software modules) executable by the at least one processor 206 to perform the required functions or operations.

[0087]

[0103] Thus, as shown in FIG. 2 , system 200 includes a 2D ultrasound image generator 208 configured to generate a first series of 2D ultrasound images and a second series of 2D ultrasound images; a 2D representation generator 210 configured to generate a first 2D representation of the target anatomy from the first series of 2D ultrasound images and generate a second 2D representation of the target anatomy from the second series of 2D ultrasound images; a simulated 2D representation generator 212 configured to generate a first simulated 2D representation of the target anatomy from the second series of 2D ultrasound images and generate a second 2D representation of the target anatomy from the second series of 2D ultrasound images; and a simulated 2D representation generator 213 configured to process the first 2D representation of the target anatomy and the first simulated 2D representation and generate the first 2D representation of the target anatomy. The ultrasound system may further include a 2D and simulated 2D representation processor 214 configured to process the second 2D representation and the second simulated 2D representation of the anatomical structure; a correspondence transformation matrix determiner 216 configured to determine a first correspondence transformation matrix from the processed first 2D and first simulated 2D representations and to determine a second correspondence transformation matrix from the processed second 2D and second simulated 2D representations; and a correspondence location determiner 218 configured to determine, for a first location associated with the target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images by using the first correspondence transformation matrix and the second correspondence transformation matrix.

[0088]

[0104] In some embodiments, the system 200 may optionally further comprise an ultrasound transducer 202 communicatively coupled to the memory 204 and the at least one processor 206. In some embodiments, the ultrasound transducer 202 may be located within a freehand ultrasound probe.

[0089]

[0105] In some embodiments, the ultrasound image generator 208 can be configured to generate a first series of 2D ultrasound images based on a first series of ultrasound waves acquired by the ultrasound transducer 202 positioned at a first plurality of positions for a first plurality of time instances, and to generate a second series of 2D ultrasound images based on a second series of ultrasound waves acquired by the ultrasound transducer 202 positioned at a second plurality of positions for a second plurality of time instances.

[0090]

[0106] Alternatively or additionally, in some embodiments, the memory 204 may be configured to store first image data relating to a first series of ultrasound waves acquired by an ultrasound transducer (e.g., ultrasound transducer 202) positioned at a first plurality of positions for a first plurality of time instances, and to store second image data relating to a second series of ultrasound waves acquired by an ultrasound transducer (e.g., ultrasound transducer 202) positioned at a second plurality of positions for a second plurality of time instances, and the at least one processor 206 (e.g., 2D ultrasound image generator 208) may be configured to generate a first series of 2D ultrasound images based on the first image data stored in the memory 204, and to generate a second series of 2D ultrasound images based on the second image data stored in the memory 204.

[0091]

[0107] Those skilled in the art will appreciate that the above-described modules are not necessarily separate modules, and that, where desired or appropriate, one or more modules can be realized by or implemented as a single functional module (e.g., a circuit or software program) without departing from the scope of the present invention. For example, the 2D ultrasound image generator 208, the 2D representation generator 210, the simulated 2D representation generator 212, the 2D and simulated 2D representation processor 214, the correspondence transformation matrix determiner 216, and / or the correspondence location determiner 218 can be realized (e.g., compiled together) as a single executable software program (e.g., referred to as a software application, or simply an "app"), which can be stored, for example, in the memory 204 and executable by at least one processor 206 to perform the functions / operations described herein according to various embodiments.

[0092]

[0108] 1, and thus various functions or operations configured to be performed by at least one processor 206 may correspond, according to various embodiments, to various steps of the aforementioned method 100, and thus need not be repeated for system 200 for the sake of clarity and brevity. In other words, various embodiments described herein in the context of a method are equally valid for the respective system or device, and vice versa.

[0093]

[0109] For example, in various embodiments, the memory 204 may store a 2D ultrasound image generator 208, a 2D representation generator 210, a simulated 2D representation generator 212, a 2D and simulated 2D representation processor 214, a correspondence transformation matrix determiner 216, and / or a correspondence location determiner 218, each of which corresponds to various steps of the aforementioned method 100 executable by at least one processor 206 to perform the corresponding functions / operations described herein.

[0094]

[0110] Various embodiments of the present disclosure may provide a computing system, controller, microcontroller, or any other system that provides processing capabilities. Such a system may include one or more processors and one or more computer-readable storage media. For example, the system 200 described above may include a processor (or controller) 206 and a computer-readable storage medium (or memory) 204, which are used in various processes performed by the system 200, such as those described herein. The memory or computer-readable storage medium used in various embodiments may be volatile memory, such as DRAM (dynamic random access memory), or non-volatile memory, such as PROM (programmable read-only memory), EPROM (erasable PROM), EEPROM (electrically erasable PROM), or flash memory, such as floating gate memory, charge trap memory, MRAM (magnetoresistive random access memory), or PCRAM (phase change random access memory).

[0095]

[0111] In various embodiments, a “circuit” can be understood as any type of logic implementation entity, such as a dedicated circuit or a processor that executes software, firmware, or any combination thereof stored in memory. Thus, in one embodiment, a “circuit” can be a hardwired or programmable logic circuit, such as a programmable processor, e.g., a microprocessor (e.g., a complex instruction set computer (CISC) processor or a reduced instruction set computer (RISC) processor). A “circuit” can also be a processor that executes software, e.g., any type of computer program, e.g., a computer program using virtual machine code, e.g., Java. According to various alternative embodiments, any other type of implementation of the respective functions described in more detail below can also be understood as a “circuit.” Similarly, a “module” can be part of a system according to various embodiments of the present invention and can encompass or be understood as any type of logic implementation entity derived from the “circuit” described above.

[0096]

[0112] Some portions of this disclosure are explicitly or implicitly presented in terms of algorithms and functions or symbolic representations of operations on data within a computer memory. These algorithmic descriptions and functions or symbolic representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. These steps are those requiring physical manipulations of physical quantities, such as electrical, magnetic, or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated.

[0097]

[0113] Unless otherwise specifically stated, as will become apparent hereinafter, throughout this specification, discussions utilizing terms such as "generate," "estimate," "modify," "render," and the like will be understood to refer to the operations and processes of a computer system or similar electronic device that manipulates and transforms data represented as physical quantities within a computer system into other data similarly represented as physical quantities within a computer system or other information storage, transmission, or display device.

[0098]

[0114] This specification also discloses systems, devices, or apparatus for performing the acts / functions of the methods described herein. Such systems, devices, or apparatus may be specially constructed for the required purposes, or may comprise a general-purpose computer or other device selectively activated or reconfigured by a computer program stored in the computer. The algorithms presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose machines may be used with computer programs in accordance with the teachings herein. Alternatively, the construction of more specialized apparatus to perform the required method steps may be applicable.

[0099]

[0115] Additionally, since it should be apparent to one skilled in the art that the individual steps of the methods described herein can be implemented by computer code, this specification also at least implicitly discloses computer programs or software / functional modules. The computer program is not intended to be limited to any particular programming language and its implementation. It will be understood that various programming languages ​​and their coding can be used to implement the teachings of the disclosure contained herein. Furthermore, the computer program is not intended to be limited to any particular control flow. Different control flows can be used in many other variations of the computer program without departing from the scope of the present invention. It will be understood by one skilled in the art that the various modules described herein (e.g., the 2D ultrasound image generator 208, the 2D representation generator 210, the simulated 2D representation generator 212, the 2D and simulated 2D representation processor 214, the correspondence transformation matrix determiner 216, and / or the correspondence location determiner 218) can be software modules implemented by a computer program or instruction set executable by a computer processor to perform the required functions, or can be hardware modules, which are functional hardware units designed to perform the required functions. It will also be appreciated that a combination of hardware and software modules may be implemented.

[0100]

[0116] Furthermore, one or more of the steps of the computer programs / modules or methods described herein may be executed in parallel rather than sequentially. Such computer programs may be stored on any computer-readable medium. The computer-readable medium may include storage devices such as magnetic or optical disks, memory chips, or other storage devices suitable for interfacing with a general-purpose computer. When the computer program is loaded and executed on such a general-purpose computer, it substantially produces an apparatus that implements the steps of the methods described herein.

[0101]

[0117] In various embodiments, a computer program product is provided that includes instructions embodied in one or more computer-readable storage media (non-transitory computer-readable storage media) and executable by one or more computer processors (e.g., 2D ultrasound image generator 208, 2D representation generator 210, simulated 2D representation generator 212, 2D and simulated 2D representation processor 214, correspondence transformation matrix determiner 216, and / or correspondence location determiner 218) to perform the method 100 for cross-referencing two-dimensional (2D) ultrasound scans of tissue volumes described above with reference to Figure 1. Accordingly, the various computer programs or modules described herein can be stored in a computer program product that can be received by a system (e.g., a computer system or electronic device), such as system 200 shown in Figure 2, for execution by at least one processor 206 of system 200 to perform the needed or desired functions.

[0102]

[0118] The software or functional modules described herein can also be implemented as hardware modules. More specifically, in the case of hardware, a module is a functional hardware unit designed to be used in conjunction with other components or modules. For example, a module can be implemented using discrete electronic components or can form part of an overall electronic circuit, such as an application-specific integrated circuit (ASIC). Many other possibilities exist. Those skilled in the art will understand that the software or functional modules described herein can also be implemented as a combination of hardware and software modules.

[0103]

[0119] Those skilled in the art will appreciate that the system 200 can be configured with separate units or as a single integrated unit. For example, in various embodiments, the system 200 can include a computer system including at least one processor 206, memory 204, a 2D ultrasound image generator 208, a 2D representation generator 210, a simulated 2D representation generator 212, a 2D and simulated 2D representation processor 214, a correspondence transformation matrix determiner 216, and a correspondence location determiner 218, and a separate ultrasound probe including an ultrasound transducer 202 communicatively coupled to the computer system. In other words, the separate ultrasound probe can acquire a series of ultrasound waves for a tissue volume and then transmit the series of ultrasound waves to a different computer system (e.g., based on wireless or wired communication) to perform the method for cross-referencing two-dimensional (2D) ultrasound scans of a tissue volume described above with reference to FIG. 1 . In various other embodiments, the system 200 may correspond to or be implemented as an ultrasound probe including an ultrasound transducer 202, at least one processor 206, a memory 204, a 2D ultrasound image generator 208, a 2D representation generator 210, a simulated 2D representation generator 212, a 2D and simulated 2D representation processor 214, a correspondence transformation matrix determiner 216, and a correspondence location determiner 218.

[0104]

[0120] In various embodiments, the computer system described above can be realized by any computer system (e.g., a portable or desktop computer system), such as, by way of example only and not limitation, the computer system 300 shown generally in FIG. 3. The various methods / steps or functional modules (e.g., the 2D ultrasound image generator 208, the 2D representation generator 210, the simulated 2D representation generator 212, the 2D and simulated 2D representation processor 214, the correspondence transformation matrix determiner 216, and / or the correspondence location determiner 218) can be implemented as software, such as a computer program, that executes within the computer system 300 and instructs the computer system 300 (particularly one or more processors) to perform the methods / functions of the various embodiments described herein. The computer system 300 can include a computer module 302, input modules such as a keyboard 304 and a mouse 306, and multiple output devices such as a display 308 and a printer 310. The computer module 302 can be connected to a computer network 312 via a suitable transceiver device 314 to enable access to, for example, the Internet or other network systems, such as a local area network (LAN) or a wide area network (WAN). The computer module 302 in this example can include a processor 318 for executing various instructions, a random access memory (RAM) 320, and a read-only memory (ROM) 322. The computer module 302 can also include a number of input / output (I / O) interfaces, for example an I / O interface 324 to the display 308 and an I / O interface 326 to the keyboard 304. The components of the computer module 302 typically communicate via an interconnection bus 328, as known to those skilled in the relevant art.

[0105]

[0121] Those skilled in the art will understand that the terminology used herein is for the purpose of describing various embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will be further understood that as used herein, the terms "comprises" and / or "comprising" specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0106]

[0122] 4 illustrates an exemplary sweep (or series or sequence or plurality) 402 of 2D ultrasound images of a tissue volume associated with multiple locations (e.g., 404a, 404b, 404c, and 404d) along a scan direction 408 of the tissue volume. In this regard, the series of 2D ultrasound images 402 may be generated based on a series of ultrasound waves acquired along the scan direction 408 of the ultrasound probe by an ultrasound transducer (e.g., located within an ultrasound probe 410) at multiple locations (e.g., 406a, 406b, 406c, 406d, 406e).

[0107]

[0123] In order that the present invention may be readily understood and practiced, various exemplary embodiments of the present invention are described below for purposes of illustration only and not limitation. However, those skilled in the art will recognize that the present invention may be embodied in many different forms or configurations and should not be construed as limited to the exemplary embodiments described below. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0108]

[0124] Various embodiments of the present invention disclose methods and systems that can assist a user by automatically cross-referencing points in two different views (e.g., TRX and SAG views) using information common to two orthogonal ultrasound sweeps. More specifically, various embodiments of the present invention can assist a user in locating a corresponding region of interest on a second view (e.g., sagittal view) when the user selects a region of interest on a first view (e.g., transverse view).

[0109]

[0125] According to various embodiments, two 2D sweep scans of the 3D anatomical structure from two (nearly) orthogonal views can be provided for cross-referencing. Hereinafter, for simplicity and without loss of generality, it is mainly assumed that the first sweep is obtained from a transverse (TRX) view and the second sweep from a sagittal (SAG) view.

[0110]

[0126] According to various embodiments, a segmentation method may be applied to segment the boundaries of a target anatomical structure (e.g., the thyroid gland) in each 2D frame in both fields of view. In various embodiments, as used herein, the term "target anatomical structure" may refer to an anatomical structure of interest (or an internal anatomical structure), such as a body organ (e.g., the thyroid gland, prostate gland, bladder, heart, lungs, stomach, liver, kidney, or other body organ) or a portion of the vasculature (e.g., the carotid artery).

[0111]

[0127] Since both views contain the same 3D anatomical structure, there is common information in these two orthogonal ultrasound sweeps. According to various embodiments, this common information can be used to find the location of any given point [idx trx ,y trx ,x trx ] to the corresponding point [idx sag ,y sag ,x sag], where idx is the frame index and x and y are the coordinates of each point. The frame index indicates the particular image frame in which each point is located in a series of 2D ultrasound images (sweep). For example, [idx trx ,y trx ,x trx ] is the index idx trx is located in the TRX frame (the frame of the TRX sweep) with coordinate y trx ,x trx , and the points having

[0112]

[0128] According to various embodiments of the present invention, a (computer-implemented) method or algorithm for calculating a transformation can be provided for a cross-referencing tool or system that can assist a user in locating an approximate region of interest (ROI) or landmark (e.g., nodule) on a second image plane (e.g., sagittal plane) when the region of interest or landmark (e.g., nodule) has already been identified (e.g., by a user) on a first image plane (e.g., transverse plane).

[0113]

[0129] According to various embodiments, a cross-referencing tool or system can use a linear transformation (matrix) to select a specific frame index and coordinate in the sagittal (longitudinal) image based on a user-provided frame index and coordinate on the transverse image. The mapping obtained from the transformation can be used to put the transverse and sagittal image pair into the same spatial coordinate system, allowing points in the transverse field of view to be mapped to points in the sagittal field of view.

[0114]

[0130] When invoked by a user, the method or algorithm may take as input images of two 2D ultrasound scans (e.g., a series of 2D thyroid images) performed along orthogonal scan directions, such as a transverse (TRX) sweep (comprising multiple TRX images (image frames)) and a sagittal (SAG) sweep (comprising multiple SAG images (image frames)), and generate a pair of transformation matrices (also referred to herein as corresponding transformation matrices) T trx and T sag can be output.

[0115]

[0131] Below, a pair of transformation matrices T trx and T sag Aspects of the calculation of σ are illustratively described in the context of 2D ultrasound imaging of a thyroid lobe, including transverse (TRX) and sagittal (SAG) sweeps. The thyroid lobe may be an example of a target anatomical structure.

[0116]

[0132] According to some embodiments, the calculation of the transformation matrix may include the following steps. 1. Segment the target anatomical structure (thyroid lobe) in all frames for both the TRX sweep and the SAG sweep to obtain a binary mask (also called a binary segmentation mask) for the image frames of the TRX sweep and the SAG sweep. In the binary mask, pixel values ​​assume a first value (e.g., "1") indicating a thyroid region or a second value (e.g., "0") indicating a non-thyroid region. The segmentation can be performed by any general-purpose segmentation method known in the art (e.g., a segmentation method based on a conventional network or a neural network).

[0117] 2. Find an optimal binary mask from the TRX sweep and find an optimal binary mask from the SAG sweep. The optimal binary mask for the TRX sweep can be the largest binary mask among the binary masks obtained from the TRX sweep, and the optimal binary mask for the SAG sweep can be the largest binary mask among the binary masks obtained from the SAG sweep. The optimal (or largest) binary mask for the TRX sweep can be referred to as a first 2D representation of the thyroid lobe (target anatomical structure), and the optimal (or largest) binary mask for the SAG sweep can be referred to as a second 2D representation of the thyroid lobe (target anatomical structure).

[0118] 3. The shape of the entire stack of masks for the SAG sweep (H sag ×W sag ×N sag ) to (H sag ×N sag ×W sag ), where N sag is the number of image frames in the SAG sweep, and H sag and W sag are the number of rows and columns in each SAG image frame, respectively. This creates a simulated stack of TRX masks from the original SAG stack. Then, similar to step 2, select the largest simulated TRX mask.

[0119] 4. Find an initial (coarse) width-to-height ratio for the simulated TRX frame. This ratio can be optimized to align the simulated TRX image with the actual TRX image and later refine the ratio. In various embodiments, the width-to-height ratio can be used to scale the width of the simulated TRX mask to obtain a scaled version of the simulated TRX mask, which can be referred to as a scaled first simulated 2D representation of the thyroid lobe (target anatomical structure).

[0120] 5. Create a distance map for the selected TRX mask (i.e., the optimal binary mask for the TRX sweep obtained in step 2). In the distance map (also called the TRX distance map), the value of a pixel is the distance from that pixel to the nearest pixel on the edge of the thyroid lobe (target anatomy) in the TRX mask. Since using all pixels within the thyroid lobe (target anatomy) can be computationally intensive, points from within the thyroid lobe from the mock TRX mask can be sampled to be used in the next step.

[0121] 6. A vectorized heuristic search method is used to solve the optimization problem that should align the previously generated images (points sampled from the TRX distance map and the simulated TRX mask).

[0122] 7. Sample points from the edge of the thyroid lobe (target anatomical structure) mask in the simulated TRX mask and use the registration parameters found in the previous step and an optimization algorithm such as the Gauss-Newton algorithm to minimize the registration cost and obtain the transformation matrix T. trx Find.

[0123] 8. Repeat steps (3) to (7) above from the TRX sweep mask to obtain the transformation matrix T sag Calculate. 9. Optional: Find the in-plane shift for all frames, which is done similarly to step 6, resulting in the matrix RC trx , thereby compensating for hand movements (of the user, e.g., clinician) that occur during the scanning procedure (using a freehand ultrasound probe).

[0124] 10.T trx and T sag (and optionally RC trx ) and any point (frame index idx trx and coordinates (x trx ,y trx ) to the corresponding point on the SAG sweep (frame index idx sag and coordinates (xsag ,y sag ) can be mapped to the image of the TRX and SAG sweeps. Therefore, cross-referencing between the images of the TRX and SAG sweeps can be realized. In other words, any point in the transverse field of view, i.e., the frame index idx trx In (x trx ,y trx ) for the cross-reference point in the sagittal view, i.e., the frame index idx sag In (x sag ,y sag ) can be acquired. The frame index indicates a particular frame (2D ultrasound image frame) within a series of 2D ultrasound images (image frames) from a TRX sweep or a SAG sweep.

[0125]

[0133] A flow chart 500 illustrating aspects of a method for cross-referencing 2D ultrasound scans (eg, method 100 shown in FIG. 1) according to various embodiments will now be described with reference to FIG.

[0134] In Figure 5, a pair of TRX and SAG view freehand ultrasound sweeps can be provided as input. See 501 and 502 on the left side of Figure 5, (A) ("2D swept TRX", i.e., transverse scan) and (B) ("2D swept SAG", i.e., sagittal scan). This can be included in steps 102 and 104 of the method of Figure 1. Illustratively, thyroid images can be captured by an ultrasound imaging device once from a first view (TRX plane) and then from a second view (SAG plane), similar to, for example, that shown and described in Figures 4(a) and (b).

[0126]

[0135] Furthermore, two transformation matrices T sag and T trx can be provided as an output (see 503 and 504 on the right side of Figure 5), which can be used to calculate the trx ,y trx ) and frame index idx trx to the corresponding point (x sag ,ysag ) and frame index idx sag and vice versa, which can be included in steps 112, 120, and 122 of the method of FIG.

[0127]

[0136] The mapping (cross-reference) can be expressed by the following equations (1) and (2).

[0128]

number

[0129]

[0137] Here, (idx trx ,y trx ,x trx ) indicates a point in the transverse (TRX) field of view, and (idx sag ,y sag ,x sag ) indicates the corresponding (i.e., cross-referenced) point in the sagittal (SAG) view, and T sag and T trx is the corresponding transformation matrix, and f sag and f trx is the transformation matrix T that maps positions from TRX to the SAG field of view sag and T trx and the scaling factor ws trx ,ws sag is a mapping function for , and vice versa. methodology

[0138] The task underlying various embodiments of the present invention can be seen in aligning points of a 3D anatomical structure into two orthogonal (or substantially orthogonal) views, specifically the TRX and SAG views, as shown in diagram 500 shown in Figure 5. According to various embodiments, the 3D pixel-to-pixel cross-referencing problem can be decomposed into two parallel simultaneous registration problems, which can be solved simultaneously. TRX to SAG: Cross-reference from a point on the TRX field sweep to the corresponding point on the SAG field sweep (top half of Figure 500)

[0139] To use common (shared) information within the two fields of view, a "mock version" of the target anatomical structure in the first field of view (Field of View 1, hereafter TRX) can be reconstructed from the scan of the second field of view (Field of View 2, hereafter SAG), and vice versa. The mock field of view is obtained by projecting the first field of view (Field of View 1) into the second field of view (Field of View 2).

[0130]

[0140] After some post-processing on the simulated field of view, the simulated TRX and the original TRX are simultaneously superimposed to obtain the corresponding transformation matrix T trx can be calculated.

[0141] Similarly, a simulated SAG can be generated from a 2D sweep in TRX, and the corresponding transformation matrix T sag can be calculated.

[0131]

[0142] Referring again to flowchart 500 shown in FIG. 5 , aspects of the method will now be described in more detail. In flowchart 500, individual steps are indicated by arrows and associated numbers, such as 1, 2, 3, 4, .... It should be understood that the numbering does not necessarily imply a chronological order; for example, some steps (e.g., steps 1 and 4) may be performed sequentially or simultaneously, and / or some steps labeled with a "higher" number (e.g., step 11) may also be performed before or simultaneously with a step labeled with a "lower" number (e.g., step 2). Furthermore, the arrows (steps) in flowchart 500 as a whole may indicate transitions between different stages of the process flow, with each stage represented by an image labeled with a capital letter, such as "A," "B," "C," etc.

[0132]

[0143] The individual steps shown in flowchart 500 of FIG. 5 can be described as follows.

[0144] Step 1: Segmentation of the target anatomical structure in the first field of view (TRX field of view): Image size (height H trx , width W trx ) and the number of TRX frames N trxEach frame of the TRX field of view of a 2D sweep of the target anatomical structure (see (A)) having a thyroid region is fed to a segmentation method (e.g., a U-Net segmentation network) to obtain a corresponding binary segmentation mask (C). Pixels in the segmentation mask have a value of 1 (corresponding to the thyroid region) or 0 (corresponding to the non-thyroid region). Illustratively, pixels in (C) corresponding to the target anatomical structure (thyroid region) are white, and pixels corresponding to the non-thyroid region are black. Step 1 can be included in step 106 of the method 100 shown in FIG. 1.

[0133]

[0145] Step 2: Find the largest (optimal) TRX segmentation mask (G) among the generated masks (C) representing the target anatomical structure. Here, the term "optimal" refers to the segmentation mask that has the largest segmentation mask of the target anatomical structure. Step 2 can be included in step 106 of method 100 shown in FIG. 1.

[0134]

[0146] Step 3: Generate a signed distance map (K) from the segmentation mask (G). To obtain the signed distance map (K), first, the boundary contour of the segmentation mask (G) (i.e., the boundary contour of the target anatomical structure (thyroid region) in the segmentation mask (G)) can be determined. The value of each pixel in the distance map is the distance from that pixel to its nearest contour pixel. The value of the pixel in the distance map corresponding to the contour pixel is 0. To obtain the signed distance map, the thyroid segmentation mask (G) can be converted to a signed map (hereinafter referred to as "signed(G)"), where thyroid and non-thyroid pixels have values ​​of -1 and 1 instead of 0 and 1. The signed distance map (K) is then generated by element-wise multiplication of the signed map (G) and the distance map. Step 3 can be included in step 110 of the method 100 shown in FIG. 1.

[0135]

[0147] Step 4: Segmentation of target anatomical structures in the second view (SAG view): (Hsag ,W sag ,N sag A 2D sweep of the SAG field of view (B) having the shape of a thyroid region is fed into the SAG segmentation method to obtain a corresponding binary segmentation mask (D) in the SAG field of view. As with (C), pixels in (D) corresponding to the target anatomical structure (thyroid region) are white, and pixels corresponding to non-thyroid regions are black. Step 4 can be included as step 114 of method 100 shown in FIG. 1.

[0136]

[0148] Step 5: Generate a simulated TRX segmentation and extract the resulting SAG mask (D) from it (H sag ,N sag ) and extract a maximum simulated TRX segmentation mask (H) having the shape of (H). Step 5 may be included in step 108 of the method 100 shown in FIG. 1.

[0137]

[0149] Step 6: Width (N) of the simulated TRX mask (H) sag ) coarse scale factor ws trx The rough estimation is based on the fact that the width-to-height ratio (WoH) of the thyroid gland in the real TRX frame (G) and the simulated TRX frame (I) should be the same.

[0138]

[0150] In an ultrasound 2D sweep, the in-plane pixel spacing (width and height) within a scanned frame is known, but the out-of-plane pixel spacing between two consecutive frames is unknown, which mainly depends on the direction and speed of the ultrasound probe movement. It is the height (H) of the simulated TRX mask. sag ) is likely to be approximately correct, and the scale factor (ws trx ) by width (N sag ) may need to be scaled. The goal of this step is to scale the width (N sag ) coarse scale factor ws trxThe coarse estimation is based on the fact that the width-to-height ratio (WoH) of the thyroid gland in the real TRX frame (G) and the simulated TRX frame (I) should be the same. Step 6 can be included in step 110 of the method 100 shown in FIG. 1.

[0139]

[0151] Step 7: Extract and sample contour pixels (M) from the simulated TRX segmentation mask (I). Considering the mask (I), first its contours are extracted, and then the longest contour is kept. The obtained longest contour is a chain of one pixel width, and its pixel coordinates are arranged and stored in a matrix having the shape (2, num_pixels), where num_pixels indicates the number of pixels in the longest contour. Then, a subset (e.g., 50) of the obtained num_pixels contour pixels can be sampled for further edge-based alignment in step 10. In (M), the sampled contour pixels are indicated by white dots. Step 7 can be included in step 110 of the method 100 shown in FIG. 1.

[0140]

[0152] Step 8: Sample region anchor pixels (L) from (I). Due to the large number of foreground pixels in (I), a subset of them is sampled. For example, foreground pixels whose both row and column indices can be divided by a given number n (for example, but not limited to, n=25) can be kept. In (L), the sampled pixels are indicated by white dots (L). Step 8 can be included in step 110 of the method 100 shown in FIG. 1.

[0141]

[0153] Step 9: Determine the coarse shift (δR) between the thyroid mask in (I) and the thyroid mask in (G) by maximizing the alignment response calculated from (L) and (K) using the following equation (3): trx ,δC trx ) is estimated.

[0142]

number

[0143] where (r,c) are the row and column numbers of foreground pixels in (L). Step 9 can be included in step 110 of method 100 shown in FIG.

[0154] Step 10: Using the following equation (4), the transformation matrix T trx Estimate the scale and translation between the segmentation masks in (I) and (G) by minimizing the registration cost, which is defined as the sum of squared distances over (I).

[0144]

number

[0145] where J is the distance map (see Figure 6) and W is the transformation T trx This is a warping (transformation) of points on the boundary of the target anatomical structure (i.e., the white points on image (M) in Figure 5) by

[0146]

[0155] Therefore, the transformation matrix T trx Step 10 may be included in step 112 of the method 100 shown in FIG.

[0156] The optimization problem (i.e., minimizing the registration cost) can be solved, for example, by the Gauss-Newton algorithm in the same or similar manner as shown, for example, by the pseudocode below ("Algorithm 1: Optimization by the Gauss-Newton Method").

[0147]

[0157] In pseudocode, deltaR trx is the above δR trx Corresponding to deltaC trx is the above δC trx corresponds to the transformation T trx 1 shows the warping of boundary points by

[0148]

number

[0149] SAG to TRX: (bottom half of Figure 500)

[0158] These steps are similar to those for the top half of diagram 500, but are performed from SAG to TRX.

[0150]

[0159] Step 11: (H sag ,W sag ,N sag From a stack of SAG segmentation masks (D) with the (overall) shape of (H sag ,W sag ) and extract an optimal real SAG field segmentation mask (Q) having the shape of (Q). This step is similar to step 2. Step 11 can be included in step 114 of method 100 shown in FIG. 1.

[0151]

[0160] Step 12: Generate a signed distance map (T) based on the mask boundary of (Q). The distance map generation algorithm is the same as that used in 3. Step 12 can be included in step 118 of the method 100 shown in FIG.

[0152]

[0161] Step 13: (H trx ,W trx ,N trx From the stack of TRX segmentation masks (C) with the (overall) shape of (H trx ,N trx ) and extract an optimal simulated SAG field segmentation mask (P) with the shape of (P). The optimal frame extraction algorithm is the same as that in step 5. Step 13 can be included in step 116 of method 100 shown in FIG. 1.

[0153]

[0162] Step 14: The coarse scale factor ws of the simulated SAG mask (P) along its width direction sag Then, ws sagThe rough estimation is based on the assumption that the width-to-height ratio (WoH) of the thyroid gland in the real SAG frame (Q) and the simulated SAG frame (P) should be the same. Step 14 can be included in step 118 of the method 100 shown in FIG. 1.

[0154]

[0163] Step 15: Similar to step 7, extract and sample boundary pixels (contour pixels) (V) from the selected simulated SAG thyroid segmentation mask (R). In (V), the sampled pixels are indicated by white dots. Step 15 can be included in step 118 of method 100 shown in FIG. 1.

[0155]

[0164] Step 16: Sample region anchor pixel (U) for (R). The sampling strategy is the same as that used in step 8. In (U), the sampled pixel is shown as a white point. Step 16 can be included in step 118 of method 100 shown in FIG. 1.

[0156]

[0165] Step 17: Determine the coarse shift (δR) between the thyroid mask in (T) and the thyroid mask in (Q) by maximizing the alignment response calculated in (U) and (T). sag ,δC sag ) This step is the same as that used in step 9. Step 17 may be included in step 118 of the method 100 shown in FIG.

[0157]

[0166] Step 18: Using the following equation (5), the transformation matrix T sag Estimate the scale and translation between the segmentation masks in (R) and (Q) by minimizing the registration cost, which is defined as the sum of squared distances over

[0158]

number

[0159] where S is the distance map (see Figure 6) and W is the transformation T sag Warping of points on the target anatomical structure boundary (i.e., white points in image (V) of Figure 5) by

[0167] The optimization problem is solved by the same algorithm as used in step 10. Thus, the transformation matrix T sag Step 18 may be included in step 120 of the method 100 shown in FIG. Final Mapping Equation

[0168] Calculated T trx and T sag Therefore, the following equations (6) to (7) are used to calculate the frame index idx from the TRX sweep. trx and coordinates (x trx ,y trx )) to the corresponding point on the SAG sweep (frame index idx sag and coordinates (x sag ,y sag )) can be mapped to

[0160]

number

[0161]

[0169] This can be included in step 122 of the method 100 shown in Figure 1. Thus, cross-referencing between the two fields of view can be achieved. By way of example, Equation (6) corresponds to an embodiment of Equation (1) above, and Equation (7) corresponds to an embodiment of Equation (2) above.

[0162]

[0170] FIG. 6 shows another flow chart 600 illustrating aspects of a method for cross-referencing 2D ultrasound scans (e.g., method 100 shown in FIG. 1 ) according to various embodiments, and FIGS. 7 and 8 show portions of the flow chart shown in FIG. 6 in more detail.

[0163]

[0171] Flowchart 600 is somewhat similar to flowchart 500 shown in Figure 5, and in particular, steps and stages in flowchart 600 that have the same reference numbers or labels as those shown in flowchart 500 of Figure 5 are the same as those in flowchart 500 and will not be repeated in detail here. Instead, the differences compared to flowchart 500 shown in Figure 5 will be primarily described below.

[0164]

[0172] 6 and 7, in flowchart 600, step 5 (of flowchart 500) can include steps (5a) and (5b), such that the transition from stage (D) to stage (H) passes through intermediate stage (F). FIG. 7 shows that intermediate stage (F), obtained in step (5a) and corresponding to the maximum simulated TRX segmentation mask from the sagittal (SAG) segmentation sweep, contains a (small) false positive region (indicated by reference numeral 701), i.e., a region of segmentation mask (F) that is incorrectly marked as corresponding to the target anatomical structure (thyroid lobe). This false positive region 701 can be removed by post-processing.

[0165]

[0173] Due to the complex structure of the thyroid region and the relatively low signal-to-noise ratio of ultrasound images, false-positive thyroid segmentation regions (such as region 701 in FIG. 7 ) may occasionally exist in the selected frame. In various embodiments, as shown in step (5b), a maximum connected component search algorithm can be used to filter out these small regions that are not connected to the main thyroid segmentation (region 702 in FIG. 7 ) in the selected thyroid segmentation mask. That is, in step (5b), the maximum connected component (also called the maximum adjacent component, maximum connected component, or maximum cluster) of the mask can be obtained, thereby removing noise. Maximum connected component search algorithms are known as such in the field of image processing and therefore will not be described in detail here. As a non-limiting example of such an algorithm, the so-called OpenCV cv2.connectedComponentsWithStats algorithm can be used.

[0166]

[0174] Similar to the method described above for step 5, step 13 may include steps (13a) and (13b) in flowchart 600, where step (13a) results in a maximum simulated SAG segmentation mask from a transverse (TRX) segmentation sweep (stage (N) of flowchart 600) and step (13b) results in the maximum component of the mask (stage (P) of flowchart 600).

[0167]

[0175] Again, similar to the method described above for step 5, step 2 may include steps (2a) and (2b) in flowchart 600, where step (2a) results in a maximum transverse (TRX) segmentation mask from the TRX segmentation sweep (step (E) of flowchart 600) and step (2b) results in the maximum component of the mask (step (G) of flowchart 600).

[0168]

[0176] Finally, similar to the method described above for step 5, step 11 can include steps (11a) and (11b) in flowchart 600, where step (11a) results in a maximum sagittal (SAG) segmentation mask from the SAG segmentation sweep (stage (O) of flowchart 600), and step (11b) results in the maximum component of the mask (stage (Q) of flowchart 600).

[0169]

[0177] Therefore, the method illustrated in flowchart 600 may include steps to further refine the binary segmentation masks (E), (F), (N), and (O) obtained from the TRX and SAG sweeps.

[0170]

[0178] 6 and 8, in flowchart 600, step 3 (of flowchart 500) can include steps (3a) and (3b), such that the transition from stage (G) to stage (K) passes through intermediate stage (J).

[0171]

[0179] In step (3a), a distance map (J) can be generated based on the mask boundary of the binary segmentation mask (G). Given the binary segmentation mask (G), the contour of the mask can be searched by a suitable contour search algorithm that may be known in the art. The longest contour can then be kept. Based on this longest contour, a distance map (J) can be generated. The value of each pixel in the distance map (J) is the distance from that pixel to its nearest contour pixel. Points on the contour boundary have a value of 0 in the distance map.

[0172]

[0180] In step (3b), the distance map (J) can be converted to a signed distance map (K). To obtain the signed distance map (K), the binary segmentation mask (G) can first be converted to a signed map, where thyroid and non-thyroid pixels can have values ​​of -1 and 1 instead of 0 and 1. The signed distance map (K) can then be generated by element-wise multiplication of the signed map (G) and the distance map (J).

[0173]

[0181] Similar to the method described above for step 3, step 12 may include steps (12a) and (12b) in flowchart 600, where step (12a) generates a distance map (S) based on the mask boundary of the binary segmentation mask (Q), and step (12b) converts the distance map (S) into a signed distance map (T).

[0174]

[0182] According to various embodiments, one or more of the following steps may optionally be included in flow diagram 500 or flow diagram 600 to further enhance cross-referencing.

[0175]

[0183] In optional step 19 (not shown), an initial transformation matrix T sag and T trx can be used to further refine the in-plane shift between the real and simulated images.

[0184] The step is the frame-by-frame in-plane shift ((2,N trx ) shaped RC trx ) in the image. In theory, the scanning trajectories of both the TRX sweep and the SAG sweep are assumed to be straight lines. However, it may be difficult for sonographers and radiologists to strictly follow a straight line when scanning the TRX sweep. Therefore, a module for estimating the frame-by-frame in-plane shift of the TRX frames can be added to improve the registration of each pair of real and simulated TRX frames.

[0176]

[0185] In particular, each TRX segmentation mask (H trx ,W trx ) shape (E')), its corresponding simulated TRX mask ((H sag ,N sag (F')) having the shape of (F')) can be recovered.

[0177]

[0186] Then, (G') and (L') can be obtained according to steps (2b) and 8, respectively. Then, the shift of TRX frame (F') relative to its corresponding real TRX frame (E') can be obtained by performing a heuristic search algorithm-based alignment of (G') and (L'), which is similar to the algorithm used in step 9 (where the signed distance map (K) is replaced by the binary mask (G')).

[0178]

[0187] Then, the in-plane shift (RC trx ) based on the RC of the real SAG frame to its corresponding simulated SAG frame. sag can be obtained for the inverse transformation.

[0188] In another optional step 20 (not shown), the T obtained in step 18 is calculated by performing an intensity-based Lucas-Kanade algorithm. sagThis step is to reduce the aperture problem that may be caused by an incomplete target anatomical structure scan sweep (e.g., the thyroid gland is too long to be covered by one frame in the SAG field of view in the vertical direction). To reduce the impact of this problem, we can first extract the corresponding images ((N') and (O')) of the simulated SAG mask (N) and the real SAG mask (O) from (A) and (B). Then, we apply the scale factor ws sag We can resize (N') along its width to (R') by: T sag can be improved.

[0179]

number

[0180]

[0189] If optional steps 19 and 20 are performed, the improved cross-referencing equation is: Final mapping equation (with additional steps 19 and 20)

[0190] Steps 19 and 20 result in (ws trx ,T trx ,RC trx ) and (ws sag ,T sag ,RC sag ) is then obtained. The following formula is used to find the coordinates of any point (frame index idx) from the TRX sweep: trx and coordinates (x trx ,y trx )) to the corresponding point on the SAG sweep (frame index idx sag and (x sag ,y sag )) can be mapped to

[0181]

number

[0182]

[0191] For example, Equation (9) corresponds to another embodiment of Equation (1) above, and Equation (10) corresponds to another embodiment of Equation (2) above. Thus, cross-referencing between the two fields of view (TRX and SAG) can be realized.

[0183]

[0192] FIG. 9 shows a screenshot of a cross-referencing tool or system from a first view to a second view according to various embodiments of the present invention, where by selecting any point on the transverse scan (e.g., the cursor indicator (point P1) on a thyroid nodule in the lower lobe of the thyroid gland in the transverse view in the left panel), the cross-referencing tool or system locates the corresponding nodule in the sagittal scan (i.e., point P2 with the corresponding x,y position and frame number in the sagittal sweep in the right panel).

[0184]

[0193] FIG. 10 shows a screenshot of a cross-referencing system from a second view to a first view according to various embodiments of the present invention, where by selecting another thyroid nodule in the upper lobe of the thyroid gland on the sagittal sweep (point P3 in the right panel), the cross-referencing tool locates that nodule (corresponding x,y position and frame number) in the transverse sweep (point P4 in the left panel).

[0185]

[0194] According to one aspect of the present invention, a method and system are provided for cross-referencing ultrasound sweeps from two orthogonal planes based on a segmentation mask.

[0195] According to another aspect of the present invention, a non-transitory computer-readable storage medium having instructions executed by a processor to receive a pair of ultrasound scans, apply a series of transformations and post-processing to the pair of ultrasound scans, and create a transformation mapping from the two paired scans to form a final probability segmentation is provided.

[0186]

[0196] According to various embodiments, a (computer-implemented) method or algorithm can be provided that can calculate a transform that outputs a particular frame index and coordinate in a second view (e.g., a sagittal view) based on a frame index and coordinates on a first view (e.g., a transverse view) provided by a user. The algorithm can use a linear transform to put the transverse and sagittal image pair into the same spatial coordinate system. This can enable a user to locate a region of interest (e.g., a nodule) in the sagittal image using a region of interest defined in the transverse image.

[0187]

[0197] According to various embodiments, the redundancy of two orthogonal freehand ultrasound sweep acquisitions can be exploited, and because they are considered to be in-plane, out-of-plane distances in one sweep can be accurately retrieved from another sweep. To do this, various embodiments provide a method for overlaying these two acquisitions.

[0188]

[0198] According to various embodiments, a cross-referencing module can be provided that calculates two transformation matrices and then maps the frame index and coordinates on the transverse image provided by the user to the frame index and coordinates in the sagittal (longitudinal) image.

[0189]

[0199] The mapping is based on two transformation matrices, which can be used to put transverse and sagittal image pairs into the same spatial coordinate system, thereby allowing transverse field points to be mapped to sagittal field points and vice versa.

[0190]

[0200] While embodiments of the present invention have been particularly shown and described with reference to specific embodiments, it will be understood by those skilled in the art that various changes in form and detail can be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the present invention is therefore indicated by the appended claims, and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced.

Claims

1. 1. A method for cross-referencing two-dimensional (2D) ultrasound scans of tissue volumes, comprising: generating a first series of 2D ultrasound images of a tissue volume associated with a first plurality of locations, each image along a first scan direction of the tissue volume, the tissue volume including the target anatomical structure; generating a second series of 2D ultrasound images of the tissue volume associated with a second plurality of locations, each image being along a second scanning direction of the tissue volume, the second scanning direction being at least substantially orthogonal to the first scanning direction; generating a first 2D representation of the target anatomical structure from the first series of 2D ultrasound images, the first 2D representation being associated with one of the images in the first series of 2D ultrasound images; generating a first simulated 2D representation of the target anatomy from the second series of 2D ultrasound images, wherein the first simulated 2D representation of the target anatomy and the first 2D representation correspond to each other in terms of a view direction relative to the target anatomy; processing the first 2D representation of the target anatomical structure and the first simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the first simulated 2D representation with pixel locations associated with the target anatomical structure in the first 2D representation; determining a first correspondence transformation matrix from the processed first 2D representation and the processed first simulated 2D representation of the target anatomical structure; generating a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, the second 2D representation being associated with one of the images in the second series of 2D ultrasound images; generating a second simulated 2D representation of the target anatomy from the first series of 2D ultrasound images, wherein the second simulated 2D representation of the target anatomy and the second 2D representation correspond to each other in terms of a view direction relative to the target anatomy; processing the second 2D representation of the target anatomical structure and the second simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the second simulated 2D representation with pixel locations associated with the target anatomical structure in the second 2D representation; determining a second correspondence transformation matrix from the processed second 2D representation and the processed second simulated 2D representation of the target anatomical structure; using the first correspondence transformation matrix and the second correspondence transformation matrix to determine, for a first location associated with the target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images; A method comprising:

2. determining the second location using the first corresponding transformation matrix and the second corresponding transformation matrix, 2. The method of claim 1, comprising applying a first function of the first corresponding transformation matrix and the second corresponding transformation matrix to pixel coordinates corresponding to the first location to obtain pixel coordinates corresponding to the second location.

3. 2. The method of claim 1, further comprising: determining, for a third location associated with the target anatomical structure in the first series of 2D ultrasound images, a corresponding fourth location associated with the target anatomical structure in the second series of 2D ultrasound images using the first and second correspondence transformation matrices.

4. determining the fourth location using the first corresponding transformation matrix and the second corresponding transformation matrix, applying a second function of the first corresponding transformation matrix and the second corresponding transformation matrix to pixel coordinates corresponding to the third location to obtain pixel coordinates corresponding to the fourth location. The method of claim 3.

5. generating a first plurality of binary segmentation masks from the first series of 2D ultrasound images, each binary segmentation mask in the first plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images in the first series of 2D ultrasound images; generating a second plurality of binary segmentation masks from the second series of 2D ultrasound images, each binary segmentation mask in the second plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images in the second series of 2D ultrasound images; further comprising the first 2D representation of the target anatomical structure is generated from one of the binary segmentation masks of the first plurality of binary segmentation masks; the second 2D representation of the target anatomical structure is generated from one of the binary segmentation masks of the second plurality of binary segmentation masks. The method of claim 1.

6. the first 2D representation of the target anatomical structure corresponds to a largest binary segmentation mask among the first plurality of binary segmentation masks; the second 2D representation of the target anatomical structure corresponds to a largest binary segmentation mask among the second plurality of binary segmentation masks; The method of claim 5.

7. generating the first simulated 2D representation of the target anatomical structure from the second series of 2D ultrasound images includes generating the first simulated 2D representation of the target anatomical structure from the second plurality of binary segmentation masks; generating the second simulated 2D representation of the target anatomical structure from the first series of 2D ultrasound images includes generating the second simulated 2D representation of the target anatomical structure from the first plurality of binary segmentation masks. The method of claim 5.

8. processing the first 2D representation includes generating a first signed distance map from the first 2D representation; processing the second 2D representation includes generating a second signed distance map from the second 2D representation; The method of claim 5.

9. processing the first simulated 2D representation includes scaling a width of the first simulated 2D representation to match a width of the first 2D representation; processing the second simulated 2D representation includes scaling a width of the second simulated 2D representation to match a width of the second 2D representation; The method of claim 8.

10. processing the first simulated 2D representation further comprises determining and sampling contour pixels associated with a contour of the target anatomical structure from the scaled first simulated 2D representation; and processing the second simulated 2D representation further comprises determining and sampling contour pixels associated with a contour of the target anatomical structure from the scaled second simulated 2D representation.

10. The method of claim 9.

11. processing the first simulated 2D representation further comprises sampling a first subset of pixels from the scaled first simulated 2D representation; and processing the second simulated 2D representation further comprises sampling a second subset of pixels from the scaled second simulated 2D representation. The method of claim 10.

12. processing the first simulated 2D representation further comprises using the first signed distance map and the first sampled subset of pixels to estimate a coarse shift between the scaled first simulated 2D representation and the first 2D representation; processing the second simulated 2D representation further comprises using the second signed distance map and the sampled second subset of pixels to estimate a coarse shift between the scaled second simulated 2D representation and the second 2D representation. The method of claim 11.

13. determining the first correspondence transformation matrix includes estimating a scale and translation between the scaled first simulated 2D representation and the first 2D representation by minimizing a first registration cost defined as a first cost function for the first correspondence transformation matrix; determining the second correspondence transformation matrix includes estimating a scale and transformation between the scaled second simulated 2D representation and the second 2D representation by minimizing a second registration cost defined as a second cost function for the second correspondence transformation matrix; The method of claim 12.

14. The method of claim 1 , wherein the first scanning direction corresponds to a transverse scan of the tissue volume and the second scanning direction corresponds to a sagittal scan of the tissue volume.

15. 1. A system for cross-referencing two-dimensional (2D) ultrasound scans of tissue volumes, comprising: Memory and at least one processor communicatively coupled to the memory; wherein the at least one processor: generating a first series of 2D ultrasound images of a tissue volume associated with a first plurality of locations, each image along a first scan direction of the tissue volume, the tissue volume including the target anatomical structure; generating a second series of 2D ultrasound images of the tissue volume associated with a second plurality of locations, each image being along a second scanning direction of the tissue volume, the second scanning direction being at least substantially orthogonal to the first scanning direction; generating a first 2D representation of the target anatomical structure from the first series of 2D ultrasound images, the first 2D representation being associated with one of the images in the first series of 2D ultrasound images; generating a first simulated 2D representation of the target anatomy from the second series of 2D ultrasound images, wherein the first simulated 2D representation of the target anatomy and the first 2D representation correspond to each other in terms of a view direction relative to the target anatomy; processing the first 2D representation of the target anatomical structure and the first simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the first simulated 2D representation with pixel locations associated with the target anatomical structure in the first 2D representation; determining a first correspondence transformation matrix from the processed first 2D representation and the processed first simulated 2D representation of the target anatomical structure; generating a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, the second 2D representation being associated with one of the images in the second series of 2D ultrasound images; generating a second simulated 2D representation of the target anatomy from the first series of 2D ultrasound images, wherein the second simulated 2D representation of the target anatomy and the second 2D representation correspond to each other in terms of a view direction relative to the target anatomy; processing the second 2D representation of the target anatomical structure and the second simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the second simulated 2D representation with pixel locations associated with the target anatomical structure in the second 2D representation; determining a second correspondence transformation matrix from the processed second 2D representation and the processed second simulated 2D representation of the target anatomical structure; and and determining, for a first location associated with the target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images using the first correspondence transformation matrix and the second correspondence transformation matrix. system.

16. When the at least one processor determines the second location using the first corresponding transformation matrix and the second corresponding transformation matrix, the at least one processor: configured to apply a first function of the first corresponding transformation matrix and the second corresponding transformation matrix to pixel coordinates corresponding to the first location to obtain pixel coordinates corresponding to the second location; The system of claim 15.

17. the at least one processor: and further configured to determine, for a third location associated with the target anatomical structure in the first series of 2D ultrasound images, a corresponding fourth location associated with the target anatomical structure in the second series of 2D ultrasound images using the first correspondence transformation matrix and the second correspondence transformation matrix. The system of claim 15.

18. When the at least one processor determines the fourth location using the first corresponding transformation matrix and the second corresponding transformation matrix, the at least one processor: configured to apply a second function of the first corresponding transformation matrix and the second corresponding transformation matrix to pixel coordinates corresponding to the third location to obtain pixel coordinates corresponding to the fourth location; 20. The system of claim 17.

19. the at least one processor: generating a first plurality of binary segmentation masks from the first series of 2D ultrasound images, each binary segmentation mask in the first plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images in the first series of 2D ultrasound images; generating a second plurality of binary segmentation masks from the second series of 2D ultrasound images, each binary segmentation mask in the second plurality of binary segmentation masks corresponding to a respective one of the 2D ultrasound images in the second series of 2D ultrasound images; further configured to: the first 2D representation of the target anatomical structure is generated from one of the binary segmentation masks of the first plurality of binary segmentation masks; a second 2D representation of the target anatomical structure is generated from one of the binary segmentation masks of the second plurality of binary segmentation masks; The system of claim 15.

20. 20. The system of claim 19, wherein the first 2D representation of the target anatomical structure corresponds to a largest binary segmentation mask among the first plurality of binary segmentation masks, and the second 2D representation of the target anatomical structure corresponds to a largest binary segmentation mask among the second plurality of binary segmentation masks.

21. The system of claim 15 , wherein the first scanning direction corresponds to a transverse scan of the tissue volume and the second scanning direction corresponds to a sagittal scan of the tissue volume.

22. further comprising an ultrasound transducer communicatively coupled to the memory and the at least one processor; the at least one processor: generating the first series of 2D ultrasound images based on a first series of ultrasound waves acquired by the ultrasound transducers positioned at the first plurality of positions for a first plurality of time instances; generating the second series of 2D ultrasound images based on a second series of ultrasound waves acquired by the ultrasound transducers positioned at the second plurality of positions for a second plurality of time instances; configured to: The system of claim 15.

23. 1. A computer program product embodied in one or more non-transitory computer-readable storage media and including instructions executable by at least one processor to perform a method for cross-referencing two-dimensional (2D) ultrasound scans of tissue volumes, the method comprising: generating a first series of 2D ultrasound images of a tissue volume associated with a first plurality of locations, each image along a first scan direction of the tissue volume, the tissue volume including the target anatomical structure; generating a second series of 2D ultrasound images of the tissue volume associated with a second plurality of locations, each image being along a second scanning direction of the tissue volume, the second scanning direction being at least substantially orthogonal to the first scanning direction; generating a first 2D representation of the target anatomical structure from the first series of 2D ultrasound images, the first 2D representation being associated with one of the images in the first series of 2D ultrasound images; generating a first simulated 2D representation of the target anatomy from the second series of 2D ultrasound images, wherein the first simulated 2D representation of the target anatomy and the first 2D representation correspond to each other in terms of a view direction relative to the target anatomy; processing the first 2D representation of the target anatomical structure and the first simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the first simulated 2D representation with pixel locations associated with the target anatomical structure in the first 2D representation; determining a first correspondence transformation matrix from the processed first 2D representation and the processed first simulated 2D representation of the target anatomical structure; generating a second 2D representation of the target anatomical structure from the second series of 2D ultrasound images, the second 2D representation being associated with one of the images in the second series of 2D ultrasound images; generating a second simulated 2D representation of the target anatomy from the first series of 2D ultrasound images, wherein the second simulated 2D representation of the target anatomy and the second 2D representation correspond to each other in terms of a view direction relative to the target anatomy; processing the second 2D representation of the target anatomical structure and the second simulated 2D representation to at least substantially align pixel locations associated with the target anatomical structure in the second simulated 2D representation with pixel locations associated with the target anatomical structure in the second 2D representation; determining a second correspondence transformation matrix from the processed second 2D representation and the processed second simulated 2D representation of the target anatomical structure; using the first correspondence transformation matrix and the second correspondence transformation matrix to determine, for a first location associated with the target anatomical structure in the second series of 2D ultrasound images, a corresponding second location associated with the target anatomical structure in the first series of 2D ultrasound images; a computer program product,

Images