Ultrasound imaging three-dimensional (3D) visualization from two-dimensional (2D) ultrasound images without 3D reconstruction

Abstract

An ultrasound imaging system (502) includes an ultrasound imaging probe (504) having a transducer array (510) configured to emit an ultrasound pressure field and receive echo signals during a sweep of the probe and a tracking sensor (514) configured to track a spatial position and orientation of the ultrasound imaging probe during the sweep; a beamformer (522) configured to generate a set of 2D ultrasound images for the sweep based on the received echo signals; a visualization module (526) configured to segment a tissue of interest in each 2D ultrasound image of the set of 2D ultrasound images, generate a mask image of the tissue of interest for each segmented image, convert each mask image to a 3D graphical representation of the tissue of interest, and arrange the 3D graphical representations in a 3D space based on the corresponding tracked spatial positions and orientations; and a display monitor (530) configured to present the 3D graphical representations as a 3D visualization.

Description

[0001] The following content deals with ultrasound imaging in general, and is particularly applicable to the realization of three-dimensional (3D) visualization of ultrasound imaging from two-dimensional (2D) ultrasound images without the need for 3D reconstruction. Background Technology

[0002] Ultrasound imaging provides real-time imaging of the interior of an examined subject, such as tissues or organs. An example ultrasound imaging system typically includes an ultrasound imaging probe and a console. The ultrasound imaging probe houses a transducer array, and the console includes a display monitor and a user interface, or is electrically connected to the display monitor and user interface. The transducer array transmits pressure waves and receives echoes generated in response to the pressure waves interacting with structures such as tissues or blood cells. These echoes are converted into analog signals, which are then amplified, digitized, and beamformed to produce scan lines of radio frequency (RF) data. The scan lines are processed (e.g., bandpass filtering, envelope detection, logarithmic compression, etc.), scan conversion is performed, and the images are displayed as two-dimensional (2D) B-mode ultrasound images.

[0003] For three-dimensional (3D) imaging using a one-dimensional (1D) transducer array, in one example, an ultrasound imaging probe is swept over a region and a set of 2D ultrasound images are acquired. While sweeping the ultrasound imaging probe and acquiring the 2D ultrasound images, a tracking mechanism is used to track the ultrasound imaging probe in 3D space. An example of such a tracking mechanism is a tracking sensor, which may be integrated into the ultrasound imaging probe and / or attached to an external location on the ultrasound imaging probe. The tracking sensor is configured to track the spatial position and orientation of the transducer array as the ultrasound imaging probe is swept. Thus, the spatial position and orientation of each 2D ultrasound image in 3D space relative to the other 2D ultrasound images is tracked.

[0004] To generate 3D ultrasound images, in one instance, a 3D reconstruction algorithm is used to reconstruct 2D ultrasound images. For example, an algorithm maps each 2D ultrasound image to a corresponding position in a predefined, equidistant 3D grid based on tracked spatial location and orientation information. Interpolation, filtering, and / or other imaging processing techniques are used to estimate missing data, such as data between 2D ultrasound images, and the 2D ultrasound images are resampled to generate 3D ultrasound images that allow for smooth transitions between them. To form a 3D ultrasound image containing only the tissue of interest, the tissue of interest is first segmented from the 2D ultrasound image. Figure 1 , Figure 2 , Figure 3 and Figure 4 An example of prior art 3D reconstruction of tissue of interest from a set of 2D ultrasound images is illustrated.

[0005] First see Figure 1This schematically illustrates an example acquisition of a set of 2D ultrasound images. In this example, an ultrasound imaging probe 104 is swept over the surface 106 of the subject 108 at a position 110 above the tissue of interest 112 within the subject 108, and the transducer array 114 of the ultrasound imaging probe 102 acquires a set of 2D ultrasound images of the tissue of interest 112 during the sweep. In this example, the tissue of interest 112 is part of a vascular tree. [Go to...] Figure 2 The diagram schematically illustrates a tissue-of-interest (TOO) mask 202 for one of the 2D ultrasound images. To this end, the TOO is segmented in the 2D ultrasound image to form a segmented 2D ultrasound image. Then, the TOO mask 202 is generated based on the segmented 2D ultrasound image by masking out all tissues except the segmented TOO 204. This is performed for a set of 2D ultrasound images.

[0006] Next Figure 3 Here, a portion of an example 3D reconstruction is schematically illustrated. In this example, for illustrative purposes, 2D ultrasound images (rather than the tissue of interest mask generated with them) are shown as positioned within a 3D mesh 302 based on the spatial location and orientation information tracked from each 2D ultrasound image, wherein image processing (e.g., interpolation, filtering, etc.) is performed to reconstruct a 3D stereoscopic space 304 from a set of 2D ultrasound images within the 3D mesh 302. For clarity, in Figure 3 Only a single x,y plane, a single x,z plane, and a single y,z plane of the reconstructed 3D ultrasound stereoscopic space are shown, but it should be understood that a 3D stereoscopic space has been generated. Figure 4 A 3D ultrasound image 402 of the tissue of interest generated by 3D reconstruction is schematically illustrated. In this example, the 3D ultrasound image includes only the tissue of interest 112 ( Figure 1 3D representation of (i.e., part of the vascular tree).

[0007] Typically, a set of 2D ultrasound images is first acquired by sweeping an ultrasound imaging probe over a region. These 2D ultrasound images are then processed using a 3D reconstruction algorithm (e.g., segmenting, converting to a mask, positioning in a 3D mesh, resampling, filtering, etc.) to generate a single 3D ultrasound stereoscopic space. This process introduces a delay between acquiring the set of 2D ultrasound images and generating the 3D ultrasound stereoscopic space; therefore, the 3D ultrasound stereoscopic space is only displayed after the set of 2D ultrasound images has been acquired. Furthermore, 3D reconstruction is time-consuming and computationally burdensome, requiring additional memory and processing resources. In addition, the 3D reconstruction algorithm may introduce errors (e.g., stitching between 2D ultrasound images, numerical rounding, smoothing filtering due to interpolation, etc.) and / or may unintentionally conceal errors / weaknesses in one or more of the 2D ultrasound images (e.g., uneven sampling, etc.).

[0008] Given at least the foregoing, there is a need for improved methods to generate 3D ultrasound stereoscopic space from 2D ultrasound imaging, a need that has not yet been addressed. Summary of the Invention

[0009] This application addresses the aforementioned and other issues. The present invention provides a more detailed description of the concepts described in the specific embodiments. It should not be used to identify the essential features of the claimed subject matter, nor should it be used to limit the scope of the claimed subject matter.

[0010] In one aspect, a system includes an ultrasound imaging system. The ultrasound imaging system includes an ultrasound imaging probe. The ultrasound imaging probe includes a transducer array configured to emit an ultrasound pressure field and receive echo signals during a sweep of the probe. The ultrasound imaging probe also includes a tracking sensor configured to track the spatial position and orientation of the ultrasound imaging probe during sweep. The ultrasound imaging system also includes a beamformer configured to generate a set of 2D ultrasound images for the sweep based on the received echo signals. The ultrasound imaging system further includes a visualization module configured to segment the tissue of interest in each of the set of 2D ultrasound images. The visualization module is also configured to generate a mask image of the tissue of interest for each segmented image in the set of 2D ultrasound images. The visualization module is further configured to convert each mask image into a 3D graphical representation of the tissue of interest. The visualization module is also configured to arrange the 3D graphical representation in 3D space based on the corresponding tracked spatial position and orientation. The ultrasound imaging system also includes a display monitor configured to present the 3D graphical representation as a 3D visualization.

[0011] In one or more instances, the 3D visualization includes gaps between adjacent 3D graphic representations. In one or more instances, additionally or alternatively, the 3D graphic representation is a 3D point cloud. In one or more instances, additionally or alternatively, a monitor displays the 3D visualization after the 3D graphic representation is arranged in 3D space. In one or more instances, additionally or alternatively, a monitor updates the display of the 3D visualization after subsequent 3D graphic representations are added to 3D space. In one or more instances, additionally or alternatively, the visualization module is configured to segment the perimeter of each 2D ultrasound image, include the segmented perimeter in a mask image, include the segmented perimeter together with the 3D graphic representation, and display the segmented perimeter in the 3D visualization. In one or more instances, additionally or alternatively, the sweep of the ultrasound imaging probe is a hands-free sweep of the ultrasound imaging probe. In one or more instances, additionally or alternatively, the ultrasound imaging probe is a laparoscopic ultrasound imaging probe.

[0012] In another aspect, a method includes receiving echo signals using a transducer array of an ultrasound imaging probe during probe sweep. The method also includes tracking the spatial position and orientation of the ultrasound imaging probe during sweep. The method further includes generating a set of 2D ultrasound images for the sweep based on the received echo signals. The method also includes segmenting the tissue of interest (TOI) in each of the 2D ultrasound images in the set of 2D ultrasound images. The method further includes generating a mask image of the TOI for each segmented image in the set of 2D ultrasound images. The method further includes converting each mask image into a 3D graphical representation of the TOI. The method further includes arranging the 3D graphical representation in 3D space based on the corresponding tracked spatial position and orientation. The method further includes displaying the 3D graphical representation as a 3D visualization.

[0013] In another aspect, a computer-readable medium encoded with computer-executable instructions, when executed by a processor, causes the processor to receive echo signals using a transducer array of an ultrasound imaging probe during a scan of the probe. The computer-executable instructions also cause the processor to track the spatial orientation of the ultrasound imaging probe during the scan. The computer-executable instructions further cause the processor to generate a set of 2D ultrasound images for the scan based on the received echo signals. The computer-executable instructions further cause the processor to segment the tissue of interest in each of the 2D ultrasound images in the set of 2D ultrasound images. The computer-executable instructions further cause the processor to generate a mask image of the tissue of interest for each segmented image in the set of 2D ultrasound images. The computer-executable instructions further cause the processor to convert each mask image into a 3D graphical representation of the tissue of interest. The computer-executable instructions further cause the processor to arrange the 3D graphical representation in 3D space based on the corresponding tracked spatial position and orientation. The computer-executable instructions further cause the processor to display the 3D graphical representation as a 3D visualization.

[0014] Other aspects of this application will be recognized by those skilled in the art upon reading and understanding the accompanying specification. Attached Figure Description

[0015] This application is illustrated by way of example and is not limited to the figures in the accompanying drawings, in which the same reference numerals indicate similar elements.

[0016] Figure 1 The illustration schematically illustrates a sweeping ultrasound imaging probe for acquiring a set of 2D images of the tissue of interest.

[0017] Figure 2 An example of an interest map generated by segmenting the interest tissue in one of a set of acquired 2D images is illustrated.

[0018] Figure 3This schematically illustrates a portion of a 3D stereoscopic space generated by 3D reconstruction of 2D images acquired using the group of organizations of interest.

[0019] Figure 4 A reconstructed 3D ultrasound image of the tissue of interest is illustrated schematically.

[0020] Figure 5 A non-limiting example of an imaging system configured for ultrasound imaging and including a 3D visualization module, according to one aspect of an embodiment of the present invention, is illustrated.

[0021] Figure 6 A non-limiting example of a 3D visualization module according to one aspect of the embodiments described herein is illustrated.

[0022] Figure 7 A side view is illustrated graphically as an example of acquiring 2D ultrasound images of tissue of interest over a region of the subject using a handheld sweep ultrasound imaging probe, according to one aspect of the embodiments described herein.

[0023] Figure 8 A perspective view is illustrated graphically of a 2D ultrasound image of a tissue of interest acquired at a specific point using a hands-free sweep ultrasound imaging probe, according to one aspect of the embodiments described herein.

[0024] Figure 9 A mask image of a tissue of interest in one of the 2D ultrasound images according to one aspect of the embodiments described herein is illustrated graphically.

[0025] Figure 10 A 3D visualization of a side view is illustrated graphically using a 3D point cloud generated from acquired 2D ultrasound images, according to one aspect of the implementation scheme described herein.

[0026] Figure 11 The following is a graphical illustration of a perspective view constructed from 3D point clouds generated from acquired 2D ultrasound images, according to one aspect of the implementation scheme described herein.

[0027] Figure 12 A graphical example is illustrated of a 3D visualization constructed using 3D point clouds corresponding to acquired 2D ultrasound images, according to one aspect of the implementation scheme described herein.

[0028] Figure 13 A non-limiting example of a flowchart illustrating a method for forming 3D visualization from 2D ultrasound images acquired by a hands-free scanning ultrasound imaging probe without 3D reconstruction, according to an embodiment of this document. Detailed Implementation

[0029] Embodiments of this disclosure will now be described by way of example with reference to the accompanying drawings, wherein systems, methods, and / or instructions coded in a computer-readable medium generate ultrasound three-dimensional (3D) visualizations based on 2D ultrasound images and their spatial positions and orientations relative to each other in 3D space without 3D reconstruction. As used herein, the term reconstruction, etc., refers to data formed between 2D ultrasound images and / or modifications to portions of 2D ultrasound images to combine the 2D ultrasound images together and form a single 3D ultrasound image.

[0030] As discussed above, such 3D reconstruction increases the latency between the acquisition and generation of a single 3D ultrasound image. This is at least because 2D ultrasound images are acquired and arranged in a 3D mesh before 3D reconstruction is performed, and 3D reconstruction itself is time-consuming. Furthermore, 3D reconstruction increases the computational burden on processing resources, requiring additional memory and processing cycles compared to generating 2D ultrasound images. Moreover, the processing of combined 2D ultrasound images may introduce errors and / or may unintentionally hide errors / weaknesses in the underlying data.

[0031] Using the method described herein, 2D ultrasound images are acquired by sweeping the transducer array of an ultrasound imaging probe across a region above the tissue of interest while tracking the position and / or orientation of the ultrasound imaging probe. During the acquisition of the 2D ultrasound image including the tissue of interest, the tissue of interest is segmented and the 2D ultrasound image is converted into a 3D representation of the tissue of interest. The 3D representation of the tissue of interest is then arranged in 3D space and displayed based on the tracked position and / or orientation. This process is repeated for each newly acquired 2D ultrasound image, wherein the displayed visualization is updated to add the latest 3D representation of the tissue of interest to the displayed visualization, constructing a 3D visualization.

[0032] Since no 3D reconstruction is performed, the method described herein reduces the latency, time consumption, computational burden, etc., associated with 3D reconstruction of 3D ultrasound images from 2D ultrasound images. Furthermore, by taking into account the time for segmentation, transformation, and arrangement of 3D representations, a displayed 3D visualization can be constructed in near real-time. Moreover, since the individual 3D representations of the tissue of interest constitute the displayed 3D visualization, the drawbacks associated with combining 2D ultrasound images to form a single 3D ultrasound image (e.g., introducing errors and / or unintentionally hiding errors / weaknesses) are reduced.

[0033] Go to Figure 5The illustration schematically illustrates a non-limiting example of an imaging system 502 (e.g., an ultrasound imaging system) configured for ultrasound imaging. The ultrasound imaging system 502 includes an ultrasound imaging probe 504 and a console 506. The ultrasound imaging probe 504 and the console 506 communicate with and / or dock with each other via a communication channel 508. In one example, the communication channel 508 includes wired technology, such as a complementary interface and cable between them. In another example, the communication channel 508 includes wireless technology, such as Wi-Fi. ® Bluetooth ® In yet another example, the ultrasound imaging probe 504 and the console 506 are integrated into the same housing, such as part of a handheld ultrasound system.

[0034] Ultrasound imaging probe 504 includes a transducer array 510. The transducer array 510 includes one or more transducer elements 512. Examples of such arrays include 64, 128, 192, 256 elements and / or other numbers of elements, including larger and smaller arrays, one-dimensional (1D) or two-dimensional (2D) arrays, etc. The transducer array 510 can be linear, curved and / or otherwise shaped, fully filled, sparse, and / or combinations thereof, etc. In one example, ultrasound imaging probe 504 includes a laparoscopic ultrasound imaging probe. In another example, ultrasound imaging probe 504 includes another type of ultrasound imaging probe.

[0035] One or more transducer elements 512 are configured to convert an excitation electrical signal into an ultrasonic pressure field and to convert a reflected ultrasonic pressure field into an electrical signal. By way of non-limiting example, one or more transducer elements 512 can be selectively excited by an excitation electrical (pulse) signal, causing at least one subgroup of transducer elements to transmit an ultrasonic pressure field into an examination field of view or a scanning field of view. The ultrasonic pressure field may include a focused ultrasound beam, a defocused (spherical) wave, and / or other ultrasound signals. The one or more transducer elements 512 receive echo signals and generate analog electrical signals indicative of the echo signals. Echo signals are generated in response to the transmitted ultrasonic pressure field interacting with structures such as tissue or blood cells flowing in a portion of a blood vessel.

[0036] As described in more detail below, multiple 2D ultrasound images are acquired and processed to generate a 3D visualization of the tissue of interest within the examined subject. In one example, the transducer array 510 comprises a 1D array, and multiple 2D ultrasound images are acquired by handless sweeping of the transducer array 510 over a region above the tissue of interest, wherein each 2D ultrasound image is acquired piecewise. In another example, the ultrasound imaging probe 504 also includes an electromechanical component configured to rotate the transducer array 510 within the ultrasound imaging probe 504, which is acquired piecewise from the 2D ultrasound images. In yet another example, the transducer array 510 comprises a 2D array or matrix, and multiple 2D ultrasound images can be acquired simultaneously, for example, plane-by-plane.

[0037] The ultrasound imaging probe 504 includes at least one tracking sensor 514. The at least one tracking sensor 514 is configured to track the spatial orientation and position of the transducer array 510. In one example, the at least one tracking sensor 514 is integrated into and is part of the ultrasound imaging probe 504. In another example, the at least one tracking sensor 514 is attached to and / or to the ultrasound imaging probe 504 and is configured to carry at least one tracking sensor 514. Such a tracking sensor may be located within and / or on the ultrasound imaging probe 504 and can track any sweep, rotation, translation, etc., of the transducer array 510. Examples of suitable sensors include single-axis and / or multi-axis electromagnetic sensors, inertial sensors (e.g., gyroscopes, accelerometers, etc.), optical sensors, etc.

[0038] The control console 506 includes transmitting circuitry 516 and receiving circuitry 518. Transmitting circuitry 516 is configured to generate an excitation electrical signal provided to transducer array 510 for transmitting an ultrasonic pressure field. In one example, this includes generating delays for the respective elements of one or more transducer elements 512 for transmitting focusing, beam steering, etc. Receiving circuitry 518 is configured to receive analog electrical signals from at least one element and preprocess the analog electrical signals, such as amplifying, digitizing, focusing, and / or otherwise processing the analog electrical signals.

[0039] For example, in one instance, the receiving circuit 518 includes an amplifier for each element and a corresponding analog-to-digital converter (ADC), wherein each amplifier amplifies the corresponding analog electrical signal from a microvolt level to the voltage range of the ADC. The console 506 also includes a switch (“SW”) 520 configured to switch between the transmitting circuit 516 and the receiving circuit 518, for example, by electrically connecting the transmitting circuit 516 to the transducer array 510 for performing a transmitting operation and electrically connecting the receiving circuit 518 to the transducer array 510 for performing a receiving operation. In an alternative instance, separate switches are used for the transmitting circuit 516 and the receiving circuit 518.

[0040] Console 506 also includes beamformer 522. Beamformer 522 is configured to beamform the signal from receiving circuitry 518, for example, by delay summation (e.g., matched filter beamformer, etc.) and / or other beamforming, and to generate radio frequency (RF) data. Using delay summation beamforming, the digital signal of each element is delayed to align the signal in time, the digital signal is amplified, and then superimposed. In one example, a matched filter that matches the desired received echo pulse shape (bandwidth) operates on the signal. 。

[0041] Console 506 also includes scanline processor 524. Scanline processor 524, when configured for I / Q demodulation, downmixes the RF signal and optionally applies low-pass filtering and / or decimation. This may include employing a Hilbert transform, a combination of complex demodulation bandpass filters and optional decimation, and / or other processing. Scanline processor 524 detects, extracts, and outputs the envelope (i.e., amplitude) of the I / Q signal (or the RF signal without I / Q modulation). In one instance, this is achieved using Hilbert transform and / or other methods.

[0042] Scanline processor 524 compresses the extracted envelope, thereby reducing the dynamic range through logarithmic dynamic range compression and / or other methods, for example, to reduce the dynamic range to a predetermined display precision, and outputs scan lines. Scanline processor 524 outputs the processed scan lines as frames / images (e.g., B-mode images). Scanline processor 524 may apply other processing such as filtering (e.g., via finite impulse response (FIR) filters, infinite impulse response (IIR) filters, etc.), time gain compensation (TGC), noise suppression, and / or other processing. Other processing, such as streaming, is envisioned herein.

[0043] Console 506 also includes a 3D visualization module 526. The 3D visualization module 526 is configured to generate a three-dimensional (3D) visualization of ultrasound based on the acquired 2D ultrasound images and the spatial position and orientation of the acquired 2D ultrasound images tracked by the tracking sensor 514, without requiring 3D reconstruction. Similarly, 3D reconstruction increases the latency between the acquisition and generation of a single 3D ultrasound stereo space, is time-consuming and increases processing resource requirements (e.g., compared to generating only 2D ultrasound images), and may introduce errors and / or blurring in the underlying 2D ultrasound images.

[0044] As described in more detail below, the method presented in this paper constructs 3D visualizations from 2D ultrasound images without any 3D reconstruction, reducing the drawbacks associated with 3D reconstruction (e.g., latency, time consumption, increased computational resources, introduced errors, unintentional introduction of blurring errors in 2D ultrasound images, etc.). Furthermore, the 3D visualizations can be constructed and displayed in near real-time (e.g., as the 2D ultrasound images are acquired and / or become available for processing), providing real-time feedback to the sonographer, who can then adjust the position of the ultrasound imaging probe 504, adjust imaging parameters, and terminate the scan (where diagnostic-quality 3D visualization might not be possible during the scan).

[0045] Console 506 also includes a scan converter 528. Scan converter 528 is configured to convert 2D ultrasound images and / or 3D visualization scans into the coordinate system of display 530. Scan converter 528 can be configured to employ analog scan conversion technology and / or digital scan conversion technology. In the illustrated example, display 530 is integrated with console 506. In another instance, display 530 is a separate and / or remote monitor electrically connected to console 506.

[0046] The console 506 also includes a user interface 532. The user interface 532 includes one or more input devices (such as buttons, knobs, sliders, touchscreens, mice, keyboards, etc.) and / or other input devices, and / or one or more output devices such as visual indicators, audible indicators, etc. The user interface 532 allows a user to control the operation of the ultrasound imaging system 502. The user interface 532 is shown as being integrated with the console 506. In another instance, the user interface 532 is a separate and / or remote keyboard, keypad, touchscreen, etc., electrically connected to the console 506.

[0047] The console 506 also includes a controller 534. The controller 534 includes components such as a microprocessor (…). Processors such as a central processing unit (CPU), a graphics processing unit (GPU), and a computer-readable storage medium. The computer-readable storage medium includes computer-readable instructions, and the processor is configured to execute the instructions stored in the computer-readable storage medium. Controller 534 is configured to control one or more of the following: transmitting circuitry 516, receiving circuitry 518, switch 520, beamformer 522, scanline processor 524, 3D visualization module 526, scan converter 528, display 530, and user interface 532. One or more components of console 506 may be implemented in software and / or hardware.

[0048] Figure 6 A non-limiting example of a 3D visualization module 526 is schematically illustrated. The 3D visualization module 526 includes a segmentation component 602, a mask generator 604, a point cloud determiner 606, and a point cloud aligner 608. The 3D visualization module 526 receives and / or retrieves data as input from a scanline processor 524. Figure 5 2D ultrasound images generated by ) and tracked by sensor 514 ( Figure 5 The generated ultrasound imaging probe tracking information. Based on the ultrasound imaging probe tracking position and / or orientation information, the 3D visualization module 526 outputs a 3D visualization of the tissue of interest from each 2D ultrasound image aligned in 3D space as a series of individual 3D graphical representations of the tissue of interest.

[0049] When 2D ultrasound images become available, segmentation component 602 segments the tissue of interest from each 2D ultrasound image. Segmentation component 602 employs known segmentation algorithms and / or other segmentation algorithms. Example algorithms include thresholding, clustering, artificial intelligence (AI) such as deep learning models (e.g., region-based convolutional neural networks (R-CNN), etc.). In one instance, segmentation is performed automatically by segmentation component 602, i.e., segmentation is performed without user input, such as identifying the periphery of the tissue of interest. In another instance, for at least a portion of the segmentation, segmentation is performed semi-automatically using user input. In one instance, the user identifies the specific type or type of tissue of interest, for example, through input, selection from a menu, etc. In another instance, segmentation component 602 determines the tissue of interest based on the selected scanning protocol.

[0050] Mask generator 604 generates a mask for each segmented 2D ultrasound image. Mask generator 604 employs known segmentation algorithms and / or other segmentation algorithms. In one instance, mask generator 604 initializes a blank mask of the same size as the original image (i.e., all zeros, etc.) and then fills the segmented regions with specific values ​​(e.g., all one-) to form a mask. In another instance, mask generator 604 starts with a 2D ultrasound image or a copy of a 2D ultrasound image and modifies values ​​outside the segmented regions with specific values ​​(e.g., all zeros, etc.) to form a mask. Other segmentation methods are also envisioned herein. The mask for a 2D ultrasound image used for a specific segmentation includes one or more regions corresponding to a tissue of interest. In some instances, the mask may include more than one type of tissue of interest or different types of tissues of interest. In some instances, the mask may also include a contour characterizing the periphery of the original 2D ultrasound image.

[0051] Point cloud determiner 606 converts each mask into a set of one or more 3D data points for one or more types of tissues of interest. In one instance, the set of 3D data points includes a 3D point cloud, i.e., a sphere with coordinates (x, y, z) of spatial position and orientation based on tracking of the ultrasound imaging probe 504. Point cloud determiner 606 employs known point cloud formation algorithms and / or other point cloud formation algorithms. In a non-limiting instance, point cloud determiner 606 loads a mask image, identifies segmented regions, extracts the coordinates of pixels belonging to the segmented regions, assigns depth information to the segmented 2D regions to form a 3D representation, and combines the coordinates and depth information to form a 3D point cloud. In one instance, the data points represent the outer surface of the tissue of interest in the corresponding 2D ultrasound image. In another instance, each point is supplemented with additional information, such as attributes like color or intensity.

[0052] The point cloud aligner 608 aligns each 3D point cloud in 3D space based on the spatial position and orientation tracked by the ultrasound imaging probe 504 to form a 3D visualization. For example, the 3D point cloud of a 2D ultrasound image is placed in the 3D visualization in the same position as the 2D ultrasound image in the stereoscopic space covered by the acquired 2D ultrasound image. Therefore, the 3D visualization will include the 3D point cloud of each 2D ultrasound image, wherein each 3D point cloud is arranged in 3D space based on the spatial position and orientation of the corresponding 2D ultrasound image. The 3D visualization will include no data gaps between each 3D point cloud. These gaps correspond to the spacing between 2D ultrasound images and do not include gaps caused by overlapping adjacent 2D ultrasound images. Therefore, the integrity of the acquired 2D ultrasound images is maintained, and data is not derived to form the data between 3D point clouds, nor is the 3D point cloud manipulated to create a smooth transition between them.

[0053] The 3D visualization module 526 outputs a 3D visualization. As described herein, the 3D visualization is displayed via a display 530. 2D ultrasound images can be displayed simultaneously and / or alternately. As described herein, in one instance, a 3D visualization can be displayed and constructed once the 2D ultrasound images become available for use by the 3D visualization module 526. In another instance, two or more (including all) of the 2D ultrasound images can be converted into 3D point clouds and arranged in 3D space before the 3D visualization is displayed via the display 530. The 3D visualization can be manipulated using known processing techniques and / or other processing techniques such as rotation, translation, annotation, distortion, magnification, smoothing filtering, sharpening, brightening, etc. Where the mask includes the outline representing the periphery of the original image for each 2D ultrasound image, the 3D visualization may also include the periphery of the 2D ultrasound image in a three-dimensional space, which can provide background information such as the position of the ultrasound imaging probe 504 and the sweep direction of the ultrasound imaging probe 504.

[0054] Figure 7 , Figure 8 , Figure 9 , Figure 10 , Figure 11 and Figure 12 The following is a graphical example illustrating the generation of 3D visualizations using the methods described herein. Figure 7 An example of a side view of the tissue of interest combined with an ultrasound imaging probe 504 and an image plane is illustrated graphically. Figure 8 The combination is illustrated graphically. Figure 7 An example of a perspective view of a tissue of interest combining an ultrasound imaging probe 504 and an image plane is described. Figure 9 The combination is illustrated graphically. Figure 7 and Figure 8 The image plane described is an example mask image generated. Figure 10 The 3D point cloud aligned in 3D space from a 2D ultrasound image is illustrated graphically up to the point of combination. Figure 9 An example of a side view of the mask being described. Figure 11 The combination is illustrated graphically. Figure 10 An example of a perspective view of an aligned 3D point cloud. Figure 12 An example of a 3D visualization illustrating the display of the entire scanned tissue of interest is given graphically.

[0055] First see Figure 7The image illustrates, graphically, an example of a side view of a tissue of interest 702 combined with an ultrasound imaging probe 504 and an image plane 704. In this example, the tissue of interest 702 is a tubular tissue of interest, such as a blood vessel and / or other tubular tissues of interest. In this example, the tubular tissue of interest 702 includes a main channel 706 and a bifurcation 708, at which the main channel 706 divides into multiple branches, including branches 710, ... and 712. The main channel 706 includes a major axis 714, which is shown along the centerline of the main channel 706. Branch 708 includes a major axis 716, which is shown along the centerline of branch 708. Branch 710 includes a major axis 718, which is shown along the centerline of branch 710.

[0056] In the illustrated example, an ultrasound imaging probe 504 is activated to begin scanning the tissue of interest 702 at an initial position 720 on the surface 722 of the subject. For the scan, the ultrasound imaging probe 504 is swept along direction 724 by handless scanning, along the major axes 714, 716, and 718 of the main channel 706, branches 710, and 712, respectively. In this example, the ultrasound imaging probe 504 is at its current position 725 on the surface 722 of the subject. An image plane 704 intersects the major axes 714, 716, and 718 of the main channel 706, branches 710, and 712. The image plane 704 extends from the ultrasound imaging probe 504 at the current position 726 and intersects branches 710 and 712.

[0057] At the current position 725, image plane 704 extends through cross-section 726 of branch 710 and cross-section 728 of branch 712. Depending on the position of the ultrasound imaging probe 504, in other instances, image plane 704 may extend only through branch 710, only through main channel 706, etc. The number of 2D ultrasound images acquired from the initial position 720 to the current position 726 and from the current position 725 to the end of the scan, as well as the spacing between 2D ultrasound images, depends on various factors such as frame rate, hands-free sweep speed, hands-free sweep path, elevation resolution / beam height, etc. It should be understood that... Figure 7 (and Figures 8 to 12 The shapes, sizes, and positions of the objects illustrated in the examples are for illustrative purposes and are not restrictive.

[0058] See further Figure 8 The combination is illustrated graphically. Figure 7The illustration depicts a perspective view of the tissue of interest 702, the ultrasound imaging probe 504 (at current position 725), and the image plane 704 (at current position 726). In this example, the image plane 704 is fan-shaped or sector-shaped. In other instances, the image plane can be other shapes, such as squares, rectangles, etc. In this example, the cross-section 726 of branch 710 includes a larger disc-shaped region than the cross-section 728 of branch 712, which includes a smaller disc-shaped region. For clarity and illustration, the remainder of the image plane 704 is illustrated as not intersecting any object; however, it should be understood that the image plane 704 may intersect other tissues, blood cells, etc., along with the object.

[0059] Go to Figure 9 The diagram illustrates the combination of... Figure 7 and Figure 8 The example mask image 902 generated by the image plane 704 is described. Typically, the echo signal received by the ultrasound imaging probe 504 is processed to generate a corresponding 2D ultrasound image, which will include the object at the current position 725 that is transversely intersected by the image plane 704. The segmentation component 602, as combined... Figure 6 The tissue of interest (cross section 726 of branch 710 and cross section 728 of branch 712) may be segmented as described and / or otherwise. For example, segmentation component 602 may segment cross sections 726 and 728 from a 2D ultrasound image using automated and / or semi-automated methods such as thresholding, clustering, deep learning models, etc. Segmentation component 602 may then generate a mask image 902 that includes only a representation 904 of the segmented cross section 726 of branch 710 and a representation 906 of the segmented cross section 728 of branch 712.

[0060] Next Figure 10 The image illustrates the location at position 726 in a graphical manner. Figure 7 and Figure 8 Aligning 2D ultrasound images acquired at ) location in 3D space and based on image masks ( Figure 9 An example of a 3D point cloud up to a side view of mask image 902. In this example, main channel 706, branch 710, and branch 712 are provided respectively. Figure 7 The major axes 714, 716, and 718 of the image serve as a reference frame and for explanation. The point cloud determiner 606 converts each mask image 902 into a set of one or more 3D data points for one or more types of organizations of interest, such as combining... Figure 6 And / or otherwise described. The point cloud aligner 608 aligns each 3D point cloud in 3D space based on the spatial position and orientation tracked by the ultrasonic imaging probe 504 to form a 3D visualization, such as in combination with... Figure 6And / or described in other ways.

[0061] In this example, 3D point cloud 1002 corresponds to branch 710 ( Figure 7 and Figure 8 Characterization of the cross section 726 of 904 ( Figure 9 Furthermore, 3D point cloud 1004 corresponds to branch 712 (). Figure 7 and Figure 8 Characterization of the cross section 728 of 906 ( Figure 9 The 3D point cloud 1006 corresponds to the representation of the cross-section of branch 710 acquired from the initial position 720 to the current position 726. Figure 7 and Figure 8 The 3D point cloud 1008 corresponds to the representation of the cross-section of branch 712 acquired from the initial position 720 to the current position 726. Figure 7 and Figure 8 ).

[0062] As discussed herein, the spacing between 2D ultrasound images depends on various factors, such as frame rate, hands-free sweep speed, hands-free sweep path, elevation resolution / beam height, etc. This spacing between 2D ultrasound images will also exist between 3D point clouds. For example, the spacing 1010 between 3D point cloud 1002 and a prior 3D point cloud corresponds to the spacing between the 2D ultrasound image of 3D point cloud 1002 and the 2D ultrasound image of the prior 3D point cloud. In contrast, prior art 3D reconstruction resamples the 2D ultrasound images and estimates the data for the gaps between the 2D ultrasound images to reconstruct a single 3D stereo space without gaps, even if gaps exist between the original 2D ultrasound images. Resampling and / or data estimation increase memory and processing requirements. Such 3D reconstruction increases the requirements for memory and processing, while the method described herein alleviates this by eliminating the need to perform 3D reconstruction, resampling, and data estimation to fill the gaps between the original 2D ultrasound images.

[0063] See further Figure 11 The diagram illustrates the current position 726 (...). Figure 7 and Figure 8 Aligning 2D ultrasound images acquired at ) location in 3D space and based on image masks ( Figure 9 The 3D point cloud is shown up to the perspective view of the mask image 902. Similarly, the main channel 706, branch 710 and branch 712 are provided respectively. Figure 7 The major axes 714, 716 and 718 are used as a reference frame and for explanation.

[0064] Go to Figure 12An example of a 3D visualization 1202 graphically illustrating the display of the entire scanned tissue of interest 702 is provided. This example includes... Figure 10 and Figure 11 The 3D point cloud described and the position 726 of the ultrasound imaging probe 504. Figure 7 and Figure 8 3D point clouds were generated from 2D ultrasound images acquired during the subsequent sweep. (For example, combined with...) Figure 8 In this example, the image plane 704 is fan-shaped or sector-shaped. In one instance, at least the contour of one of the image planes is also provided with 3D visualization 1202. In the illustrated example, for clarity and explanation, the contours of the first and last image planes are shown and connected to form the contour 1204 of the acquired 3D stereoscopic space. Typically, corresponding contours are provided for one or more 3D point clouds, including the contour of each 3D point cloud. In one instance, the contour is obtained during the segmentation and / or mask generation process. In one instance, the contour visually provides the user with the orientation of the ultrasound imaging probe 504.

[0065] In one instance, 3D visualization 1202 is displayed together with one or more of 2D ultrasound images, segmentation, and mask images. Figure 9 Furthermore, as described herein, a 3D visualization 1202 can be visually constructed during the acquisition of 2D ultrasound images. For example, the initially displayed 3D visualization 1202 may include a 3D point cloud of only the first acquired 2D ultrasound image, wherein, as ultrasound images become available, the displayed 3D visualization 1202 is updated to include a 3D point cloud of subsequently acquired 2D ultrasound images.

[0066] A progressively constructed 3D visualization 1202 is used to provide visual feedback to the sonographer. For example, progressive construction allows the sonographer to adjust the scan, such as the sweep speed, sweep path, imaging parameters, etc. Furthermore, this feedback allows the sonographer to determine early on when a useful 3D visualization cannot be constructed. Similarly, since no 3D reconstruction is performed, the method described herein reduces the latency, time consumption, computational burden, etc., associated with 3D reconstruction of 3D ultrasound images from 2D ultrasound images, and / or reduces the disadvantages associated with combining 2D ultrasound images to form a single 3D ultrasound image because the individual 3D representations of the tissue of interest constitute the displayed 3D visualization.

[0067] Figure 13This document provides a non-limiting example of a flowchart illustrating a method for forming 3D visualizations from 2D ultrasound images acquired hands-free using an ultrasound imaging probe, without reconstructing the 2D ultrasound images. It should be understood that the order of actions in this method is not restrictive. Therefore, other orders are contemplated herein. Furthermore, one or more actions may be omitted, and / or one or more additional actions may be included.

[0068] At 1302, 2D ultrasound images of the tissue of interest are acquired while the ultrasound imaging probe is being swept, as described herein and / or otherwise. The 2D ultrasound images may include B-mode, blood flow, and other 2D ultrasound images. In one example, 2D ultrasound images are acquired during handheld sweeping of the ultrasound imaging probe to obtain stereoscopic spatial scanning images, wherein the ultrasound imaging probe includes a tracking sensor so that the position of the 2D ultrasound image in 3D space relative to other 2D ultrasound images acquired during the sweep can be determined.

[0069] At position 1304, the tissue of interest is segmented in the 2D ultrasound image, as described herein and / or otherwise. As described herein, in one instance, the segmentation is fully automatic, requiring no user input for segmentation, while in another instance, the segmentation is semi-automatic, utilizing user input. Furthermore, segmentation may include thresholding, clustering, AI, and / or other known and / or other methods.

[0070] At 1306, a mask image is generated for the segmented 2D ultrasound image, as described herein and / or otherwise. In one instance, after mask generation, the mask image will include only the representation of each region containing the tissue of interest. In another instance, the mask image will also include the outline of the 2D ultrasound image. For example, if the 2D ultrasound image is a fan-shaped image, the mask image will include the outline of the fan-shaped image along with the representation of the tissue of interest.

[0071] At 1308, the mask image is transformed into a 3D point cloud, as described herein and / or otherwise. For example, in one instance, depth information is assigned to the representation to form a 3D representation by extracting the coordinates of pixels belonging to the representation, and the coordinates and depth information are combined to form a 3D point cloud, thereby transforming each representation of the organization of interest in the mask image into a sphere with coordinates (x, y, z).

[0072] At point 1310, based on ultrasound imaging probe tracking information obtained from the tracking sensor, the 3D point cloud is positioned in 3D space to form a 3D visualization, as described herein and / or otherwise. As discussed herein, in one instance, the 3D visualization is displayed and constructed in real time while 2D ultrasound images are acquired. In another instance, the 3D visualization is constructed from two or more 2D ultrasound images before being displayed.

[0073] At 1312, determine whether another 2D ultrasound image will be acquired. For example, if the user still keeps the ultrasound imaging probe in scanning mode, repeat steps 1302 to 1310 to acquire another 2D ultrasound image during the ultrasound imaging probe's sweep and add another 3D point cloud to the 3D visualization. If the user no longer uses the ultrasound imaging probe for scanning, the scan ends. The 2D ultrasound image, segmented image, mask image, and / or other information can be displayed together with partial and / or final 3D visualization.

[0074] The method described in this paper generates 3D visualizations from 2D ultrasound images without performing 3D reconstruction. Instead, based on the tracking information of the ultrasound imaging probe during the hands-free sweep of the ultrasound imaging probe during acquisition, each 3D point cloud corresponding to a 2D ultrasound image is arranged in 3D space. Since no 3D reconstruction is performed, the method described in this paper reduces the latency, time consumption, and / or computational burden associated with 3D reconstruction of ultrasound images. Furthermore, the displayed 3D visualization can be constructed visually in near real-time, and since the individual 3D point clouds constitute the displayed 3D visualization, the drawbacks associated with combining 2D ultrasound images to form a single 3D ultrasound stereoscopic space are reduced.

[0075] As used herein, elements or steps listed in the singular and beginning with the word "a" or "an" should be understood to not exclude multiple said elements or steps unless such exclusion is explicitly stated. Furthermore, references to "an embodiment" of the invention are not intended to be construed as excluding the existence of additional embodiments that also include the referenced features. Moreover, unless explicitly stated to the contrary, embodiments that "comprise," "include," or "have" one or more elements with a particular attribute may include such additional elements that do not have that attribute. The terms "comprise" and "in" are used as concise linguistic equivalents to the corresponding terms "comprising" and "wherein". Furthermore, the terms "first," "second," and "third," etc., are used merely as notations and are not intended to impose numerical requirements or a particular order of position on their objects.

[0076] Various implementations and / or components (e.g., modules, parts therein, and controllers) may also be implemented as part of one or more computers or processors. A computer or processor may include computing devices, input devices, display units, and interfaces, such as for accessing the Internet. A computer or processor may include a microprocessor. The microprocessor may be connected to a communication bus. A computer or processor may also include memory. Memory may include random access memory (RAM) and read-only memory (ROM). A computer or processor may further include a storage device, which may be a hard disk drive or a removable storage drive, such as a floppy disk drive, an optical disk drive, etc. The storage device may also be other similar means for loading computer programs or other instructions into the computer or processor.

[0077] As used herein, the terms "computer" or "module" can include any processor-based or microprocessor-based system, including systems using microcontrollers, reduced instruction set computers (RISCs), application-specific integrated circuits (ASICs), logic circuits, and any other circuitry or processors capable of performing the functions described herein. The examples above are merely illustrative and are therefore not intended to limit the definition and / or meaning of the term "computer" in any way. A computer or processor executes a set of instructions stored in one or more storage elements to process input data. Storage elements may also store data or other information as desired or required. Storage elements may take the form of an information source within the processor or a physical memory element.

[0078] An instruction set may include various commands that instruct a computer or processor to perform specific operations (such as methods and processes according to various embodiments of the present invention) as a processing machine. The instruction set may be in the form of a software program. Software may take various forms, such as system software or application software. Furthermore, software may take the form of a collection of separate programs or modules, a program module within a larger program, or a portion of a program module. Software may also include modular programming in the form of object-oriented programming. The processor's processing of input data may be in response to operator commands, the results of previous processing, or a request from another processor.

[0079] As used herein, the terms “software” and “firmware” are interchangeable and include any computer program stored in memory for execution by a computer, including RAM memory, ROM memory, EPROM memory, EEPROM memory, and non-volatile RAM (NVRAM) memory. The memory types described above are merely exemplary and therefore do not limit the types of memory that can be used to store computer programs.

[0080] It should be understood that the above description is intended to be illustrative and not restrictive. For example, the above embodiments (and / or aspects thereof) may be used in combination with each other. Furthermore, many modifications may be made to adapt particular situations or materials to the teachings of various embodiments of the invention without departing from the scope of the invention. While the dimensions and types of materials described herein are intended to define parameters of various embodiments of the invention, these embodiments are by no means restrictive but exemplary. Many other embodiments will be apparent to those skilled in the art upon review of the above description.

[0081] This written description uses examples to disclose various embodiments of the invention, including the best mode, and also enables those skilled in the art to practice various embodiments of the invention, including making and using any device or system and performing any included methods. The patent scope of the various embodiments of the invention is defined by the claims, and may include other examples that would occur to those skilled in the art. Such other examples are intended to fall within the scope of the claims if they have structural elements that are not indistinguishable from the literal language of the claims, or if they include equivalent structural elements that differ only slightly from the literal language of the claims.

[0082] The embodiments illustrated in the accompanying drawings and described above are merely illustrative embodiments and are not intended to limit the scope of the appended claims, including any equivalents included within the scope of the claims. Various modifications are possible and will be apparent to those skilled in the art. Any combination of non-exclusive features described herein is intended to be within the scope of this disclosure. That is, features of the described embodiments may be combined with any suitable aspect described above, and optional features of any aspect may be combined with any other suitable aspect. Similarly, features listed in dependent claims may be combined with non-exclusive features of other dependent claims, particularly where the dependent claims are subordinate to the same independent claim. Since some jurisdictions require single-claim dependents, these dependents may already be used in practice, but this should not be construed as meaning that features in dependent claims are mutually exclusive.

Claims

1. An ultrasound imaging system (502), the ultrasound imaging system comprising: Ultrasonic imaging probe (504), the ultrasonic imaging probe comprising: A transducer array (510) configured to emit an ultrasonic pressure field and receive echo signals during sweep of the probe; and A tracking sensor (514) is configured to track the spatial position and orientation of the ultrasound imaging probe during the sweep; A beamformer (522) is configured to generate a set of 2D ultrasound images for the sweep based on the received echo signals; 3D visualization module (526), ​​the 3D visualization module is configured as follows: Segment the tissue of interest in each 2D ultrasound image within the set of 2D ultrasound images; Generate a mask image of the tissue of interest for each segmented image in the set of 2D ultrasound images; Each mask image is converted into a 3D graphical representation of the tissue of interest; and The 3D graphic representation is arranged in 3D space based on the corresponding tracked spatial position and orientation; and The display (530) monitor is configured to present the 3D graphic representation as a 3D visualization.

2. The ultrasound imaging system according to claim 1, wherein, The 3D visualization includes the gaps between adjacent 3D graphic representations.

3. The ultrasound imaging system according to claim 1, wherein, The 3D graphic representation is a 3D point cloud.

4. The ultrasound imaging system according to claim 1, wherein, After the 3D graphic representation is arranged in the 3D space, the display monitor shows the 3D visualization.

5. The ultrasound imaging system according to claim 4, wherein, After subsequent 3D graphical representations are added to the 3D space, the display monitor updates the display of the 3D visualization.

6. The ultrasound imaging system according to claim 1, wherein, The visualization module is configured to segment the periphery of each 2D ultrasound image, include the segmented periphery in the mask image, include the segmented periphery together with the 3D graphic representation, and display the segmented periphery in the 3D visualization.

7. The ultrasound imaging system according to claim 1, wherein, The sweep of the ultrasound imaging probe is a handheld sweep of the ultrasound imaging probe.

8. The ultrasound imaging system according to claim 1, wherein, The ultrasound imaging probe is a laparoscopic ultrasound imaging probe.

9. A method, the method comprising: During the sweep of the probe, the echo signal is received using the transducer array of the ultrasonic imaging probe. The spatial position and orientation of the ultrasound imaging probe are tracked during the sweep; A set of 2D ultrasound images is generated for the sweep based on the received echo signal; Segment the tissue of interest in each 2D ultrasound image within the set of 2D ultrasound images; Generate a mask image of the tissue of interest for each segmented image in the set of 2D ultrasound images; Each mask image is converted into a 3D graphical representation of the organization of interest; The 3D graphic representation is arranged in 3D space based on the corresponding tracking spatial position and orientation; as well as The 3D graphical representation is displayed as a 3D visualization.

10. The method according to claim 9, further comprising: The 3D graphic representation, along with the gaps between adjacent 3D graphic representations, is arranged in the 3D space.

11. The method according to claim 9, wherein, The 3D graphic representation is a 3D point cloud.

12. The method according to claim 9, further comprising: The 3D visualization is displayed after the 3D graphic representation is arranged in the 3D space.

13. The method according to claim 12, further comprising: The display is updated after subsequent 3D graphical representations are added to the 3D space.

14. The method according to claim 9, further comprising: Segment the periphery of each 2D ultrasound image; The perimeter of the segment is included in the mask image; The perimeter of the segment is included together with the 3D graphical representation; and the perimeter of the segment is displayed in the 3D visualization.

15. A computer-readable medium encoded with computer-readable instructions, which, when executed by a processor, cause the processor to: During the sweep of the probe, the echo signal is received using the transducer array of the ultrasonic imaging probe. The spatial orientation of the ultrasound imaging probe is tracked during the sweep; A set of 2D ultrasound images is generated for the sweep based on the received echo signal; Segment the tissue of interest in each 2D ultrasound image within the set of 2D ultrasound images; Generate a mask image of the tissue of interest for each segmented image in the set of 2D ultrasound images; Each mask image is converted into a 3D graphical representation of the organization of interest; The 3D graphic representation is arranged in 3D space based on the corresponding tracking spatial position and orientation; and The 3D graphical representation is displayed as a 3D visualization.

Images