B-mode image rendering based on tissue differentiation

Abstract

In some examples, tissue types in a B-mode image are distinguished by multi-parametric imaging and / or other segmentation techniques. The tissue types are used to assign material properties, such as color, translucency, and reflectance, to voxels in the B-mode image. The material properties are used to generate a rendered image. In some examples, one or more of the material properties assigned to the voxels correspond to values ​​of tissue parameters acquired during multi-parametric imaging. The rendered image includes data based on both the B-mode image and the multi-parametric imaging.

Description

[Technical Field]

[0001] This application relates to generating color images from grayscale images. More particularly, this application relates to generating enhanced renderings, for example, enhanced based on tissue type, from B-mode ultrasound images. [Background technology]

[0002] Since its introduction several decades ago, B-mode ultrasound imaging has been used to generate grayscale images. However, as advances in image processing have been made in other imaging modalities, users have begun to demand more colorful and / or realistic images from B-mode imaging. In some applications, color and / or other visualization improvements make B-mode images easier to interpret. As the use of ultrasound imaging increases among non-professionals (e.g., non-radiologists, non-sonographers, and inexperienced sonographers), the need for easier image interpretation also increases. Summary of the Invention

[0003] As disclosed herein, realistic rendering techniques based on tissue type and / or mechanical properties are applied instead of grayscale mapping of B-mode images. Different tissue types (e.g., muscle, fat, liver, kidney) are represented with different colors and / or other material properties that make the tissue types appear more realistic and / or easier to distinguish and / or interpret. In some examples, the different tissue types are determined based on multiparametric imaging (e.g., attenuation, speed of sound, elastography). In some examples, other or additional techniques, such as neural networks, B-mode textures, and / or anatomical models, are used to determine tissue types. Color and / or other material properties are applied by using different rendering "recipe" for each tissue type. For example, tissue identified as fat is assigned a white color and material properties that make the fat appear matte or matte, while tissue identified as healthy liver is assigned a red color and material properties that make the liver appear shiny.

[0004] According to at least one example disclosed herein, an apparatus includes a processor configured to: receive a B-mode image of an imaged volume, the B-mode image having a plurality of first voxels, each first voxel associated with a different spatial location within the imaged volume, the B-mode image having an intensity value corresponding to an intensity of an echo signal received from the associated spatial location within the imaged volume; receive a tissue parameter map having one or more tissue parameter values ​​for each of the plurality of first voxels; generate a 3D rendering dataset having a plurality of second voxels by assigning one or more material property values ​​to each of a plurality of second voxels, the one or more material property values ​​assigned to each of the plurality of second voxels being based at least in part on the intensity value associated with each first voxel in the B-mode image and further based at least in part on the tissue parameter value associated with each voxel obtained from the tissue parameter map; and generate an enhanced B-mode rendering from the 3D rendering dataset. The apparatus includes a display configured to display the enhanced B-mode rendering.

[0005]

[0005] According to at least one example disclosed herein, a method includes receiving a B-mode image of an imaged volume, the B-mode image having a plurality of first voxels, each associated with a different spatial location within the imaged volume, the first voxel having an intensity value corresponding to an intensity of an echo signal received from the associated spatial location within the imaged volume; receiving a tissue parameter map having one or more tissue parameter values ​​for each of the plurality of first voxels; assigning one or more material property values ​​to each of a plurality of second voxels of a 3D rendering dataset, the one or more material property values ​​assigned to each of the plurality of second voxels being based at least in part on the intensity value associated with each first voxel in the B-mode image and further based at least in part on the tissue parameter value associated with each voxel obtained from the tissue parameter map; and generating an enhanced B-mode rendering from the 3D rendering dataset.

[0006]

[0006] According to at least one example disclosed herein, a method includes segmenting a B-mode image to determine a tissue type of a respective one of a plurality of first voxels of the B-mode image; assigning one or more material property values ​​to each of a plurality of second voxels of a 3D rendering dataset, the one or more material property values ​​being based at least in part on the tissue type; generating an enhanced B-mode rendering from the 3D rendering dataset; and displaying the enhanced B-mode rendering on a display. [Brief explanation of the drawings]

[0007] [Figure 1]

[0007] FIG. 1 is an example of a B-mode image and corresponding tissue parameter color map. [Figure 2]

[0008] FIG. 1 is a block diagram of a collocated ultrasound imaging system according to an example of the present disclosure. [Figure 3]

[0009] FIG. 2 is a block diagram illustrating an example processor, according to an example of the present disclosure. [Figure 4]

[0010] 1 is a graphical representation of a process for generating renderings from B-mode and multi-parametric imaging, according to an example of the present disclosure. [Figure 5]

[0011] FIG. 10 is an example diagram of a tissue parameter table. [Figure 6]

[0012] 1 is a graphical representation of a process for generating a rendering from B-mode imaging, according to an example of the present disclosure. [Figure 7]

[0013] FIG. 1 is a block diagram of a process for model training and deployment, according to an example of the present disclosure. [Figure 8]

[0014] FIG. 10 is a graphical overview of a two-step rendering process, according to an example of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0008]

[0015] The following description of certain illustrative examples is merely exemplary in nature and is not intended to limit the present disclosure or its application or intended uses in any way. The following detailed description of exemplary devices, systems, and methods refers to the accompanying drawings, which form a part of this specification and which show, by way of illustration, specific examples in which the described devices, systems, and methods may be practiced. These examples are described in sufficient detail to enable those skilled in the art to practice the disclosed devices, systems, and methods; however, it should be understood that other examples may be utilized and that structural and logical changes may be made without departing from the spirit and scope of the present disclosure. Moreover, for purposes of clarity, detailed descriptions of specific features will not be discussed so as not to obscure the description of the present disclosure when apparent to those skilled in the art. Therefore, the following detailed description is not to be taken in a limiting sense, and the scope of the present devices, systems, and methods is defined only by the appended claims.

[0009]

[0016] Photorealistic rendering techniques for ultrasound, such as the TrueVue system developed by Koninklijke Philips, have recently become commercially available. However, existing photorealistic rendering techniques are typically applied to three-dimensional (e.g., volumetric) data sets and do not distinguish between tissue types. New ultrasound imaging and data analysis techniques have also been developed that can determine one or more parameters of tissue (e.g., the speed of sound through tissue, the attenuation of ultrasound signals in tissue, and stiffness). However, in existing systems, these parameters are typically presented separately from the B-mode image, or the parameters are color-mapped and simply overlaid on the B-mode image.

[0010]

[0017] FIG. 1 shows an example of a B-mode image and corresponding tissue parameter color map. The B-mode image 100 is of a portion of the liver and surrounding tissue. The tissue parameter color map 105 is a map of the speed of sound. In other words, the speed at which an ultrasound signal travels through different regions of tissue is mapped to different colors. In the B-mode image 100, the subcutaneous fat layer 102, the underlying muscle layer 104, and the liver tissue 106 exhibit different textures and pixel intensities. In the color map 105, the subcutaneous fat layer 102 is bright blue, indicating a slow speed of sound; the muscle layer 104 is deep red, indicating a fast speed of sound; while the liver tissue 106 is light to dark blue, indicating an even slower speed of sound. These distinct variations in color make the different tissue types much easier to distinguish in the tissue parameter color map 105. Nevertheless, when the color map 105 is overlaid on the B-mode image 100, the structure and texture information provided by the B-mode image 100 becomes obscured. Furthermore, because the resolution of the color map 105 is lower than that of the B-mode image 100, tissue types cannot be accurately correlated pixel-to-pixel (or voxel-to-voxel). For example, in principle, the fat layer 102 should have a slower speed of sound than the liver tissue 106, but the fat layer 102 appears to have a faster speed in the color map 105 due to averaging contributions from the adjacent muscle layer 104.

[0011]

[0018] According to examples of the present disclosure, multi-parametric ultrasound images are used to determine different tissue types in a B-mode image. In some examples, other or additional techniques, such as neural networks, B-mode textures, and / or anatomical models, are used to determine tissue types. Pixels or voxels associated with different tissue types (e.g., muscle, fat, liver, kidney) are assigned different color and / or other material property values. Based on the values, a rendering is generated that has the appearance of a color B-mode image. The material properties assigned to the voxels affect how the voxels are rendered (e.g., how the voxel's appearance in the rendering changes with the material properties). The color and / or other material property values ​​enable the generation of B-mode images in which tissue types appear more realistic and / or make the tissue types easier to distinguish and / or interpret. In some examples, multiple parameters are conveyed in the B-mode image rather than requesting a side-by-side view of the B-mode image and the parameter map or overlaying the B-mode image with the parameter map.

[0012]

[0019] FIG. 2 shows a block diagram of an ultrasound imaging system 200 constructed in accordance with the principles of the present disclosure. The ultrasound imaging system 200 according to the present disclosure can include a transducer array 214, which can be included in an ultrasound probe 212, such as an external or internal probe, such as an intravascular ultrasound (IVUS) catheter probe. In another example, the transducer array 214 is in the form of a flexible array configured to be conformally applied to the surface of a subject (e.g., a patient) to be imaged. The transducer array 214 is configured to transmit ultrasound signals (e.g., beams, waves) and receive echoes (e.g., received ultrasound signals) in response to the transmitted ultrasound signals. Various transducer arrays can be used, such as, for example, a linear array, a curvilinear array, or a phased array. The transducer array 214 can have a two-dimensional array (as shown) of transducer elements capable of scanning in both the elevation and azimuth dimensions, for example, for 2D and / or 3D imaging. As is commonly known, the axial direction is the direction perpendicular to the plane of the array (in the case of an axially extending curvilinear array), the azimuth direction is generally defined by the vertical dimension of the array, and the elevation direction is transverse to the azimuth direction.

[0013]

[0020] In some examples, the transducer array 214 is coupled to a microbeamformer 216, which is within the ultrasound probe 212 and controls the transmission and reception of signals by the transducer elements of the array 214. In some examples, the microbeamformer 216 controls the transmission and reception of signals by the active elements of the array 214 (e.g., an active subset of the elements of the array that defines the active aperture at any given time).

[0014]

[0021] In some examples, the microbeamformer 216 is coupled, for example, by a probe cable or wirelessly, to a transmit / receive (T / R) switch 218, which switches between transmit and receive and protects the main beamformer 222 from high-energy transmit signals. In some examples, for example, in portable ultrasound systems, the T / R switch 218 and other elements of the system may reside on the ultrasound probe 212 rather than in an ultrasound system base that houses the image processing electronics. The ultrasound system base typically includes circuitry for signal processing and image data generation, as well as software and hardware components with executable instructions for providing a user interface.

[0015]

[0022] The transmission of ultrasound signals from the transducer array 214 under the control of the microbeamformer 216 is directed by a transmit controller 220, which is coupled to the T / R switch 218 and the main beamformer 222. The transmit controller 220 controls the direction in which the beam is directed. The beam may be directed straight from the transducer array 214 (perpendicular to the transducer array 214) or at a different angle for a wider field of view. The transmit controller 220 is further coupled to a user interface 224 to receive input from a user's operation (e.g., user control) of a user input device. The user interface 224 can include one or more input devices, such as a control panel 252, which can include one or more mechanical controls (e.g., buttons, sliders, etc.), touch-sensitive controls (e.g., a trackpad, touchscreen, or the like), and / or other known input devices.

[0016]

[0023] In some examples, the partially beamformed signals produced by the microbeamformer 216 are coupled to a main beamformer 222, where the partially beamformed signals from individual patches of transducer elements are combined into a fully beamformed signal. In some examples, the microbeamformer 216 is omitted. In these examples, the transducer array 214 is under the control of the main beamformer 222, and the main beamformer 222 performs all beamforming of the signals. In examples, with or without the microbeamformer 216, the beamformed signals of the main beamformer 222 are coupled to a processing circuitry device 250, which may include one or more processors (e.g., a signal processor 226, a B-mode processor 228, a Doppler processor 260, and one or more image generation and processing components 268) configured to produce ultrasound images from the beamformed signals (i.e., the beamformed RF data).

[0017]

[0024] The signal processor 226 is configured to process the received beamformed RF data in various manners, such as bandpass filtering, decimation, I and Q component separation, and harmonic signal separation. The signal processor 226 also performs additional signal enhancements, such as speckle reduction, signal compounding, and electronic noise reduction. The processed signals (sometimes referred to as I and Q components, or IQ signals) are coupled to additional downstream signal processing circuitry for image generation. The I and Q signals are coupled to multiple signal paths within the system, each associated with a unique arrangement of signal processing components suitable for generating different types of image data (e.g., B-mode image data, Doppler image data). For example, the system may include a B-mode signal path 258 that couples signals from the signal processor 226 to a B-mode processor 228 to produce B-mode image data.

[0018]

[0025] The B-mode processor 228 can employ amplitude detection for imaging body structures. The B-mode processor 228 generates signals for tissue and / or contrast images. The signals produced by the B-mode processor 228 are coupled to a scan converter 230 and / or a multiplanar reformatter 232. The scan converter 230 is configured to arrange the echo signals from the spatial relationship in which they were received into a desired image format. For example, the scan converter 230 arranges the echo signals into a two-dimensional (2D) sector-shaped format or a pyramidal or otherwise shaped three-dimensional (3D) format.

[0019]

[0026] In some examples, the system can include a Doppler signal path 262 that couples the output from the signal processor 226 to a Doppler processor 260. The Doppler processor 260 is configured to estimate the Doppler shift and generate Doppler image data. The Doppler image data can include color data, which is then overlaid with B-mode (i.e., grayscale) image data for display. The Doppler processor 260 is configured to remove unwanted signals (i.e., noise or disturbances associated with non-moving tissue), for example, using a wall filter. The Doppler processor 260 is further configured to estimate velocity and power according to known techniques. For example, the Doppler processor can include a Doppler estimator, such as an autocorrelator, where the velocity (Doppler frequency) estimation is based on the magnitude of a lag-one autocorrelation function (e.g., R1) and the Doppler power estimation is based on the magnitude of a lag-zero autocorrelation function (e.g., R0). Velocity estimates are referred to as color Doppler data, and power estimates are referred to as power Doppler data. Motion can also be estimated using known phase-domain (e.g., parametric frequency estimators, such as MUSIC and ESPRIT) or time-domain (e.g., cross-correlation) signal processing techniques. Other estimators of the time or spatial variance of velocity, such as acceleration or derivatives of time and / or space velocity, can be used instead of or in addition to the velocity estimator. In some examples, velocity and power estimates (e.g., color and power Doppler data) undergo segmentation and post-processing, such as further thresholding to further reduce noise, as well as filling and smoothing. The velocity and / or power estimates are then mapped to desired ranges of color and / or intensity for the display according to one or more color and / or intensity maps. The map data, also referred to as Doppler image data, is then coupled to a scan converter 230 where the Doppler image data is converted to a desired image format to form a color Doppler or power Doppler image.

[0020]

[0027] In some embodiments, system 200 can include a parameter imaging signal path 264 that couples signals from signal processor 226 to a parameter processor 266 to generate tissue parameter measurements. Parameter processor 266 generates one or more types of tissue parameter measurements. In some examples, the tissue parameter measurements are mapped to pixel or voxel color and / or intensity values ​​to generate a map (e.g., stiffness map, speed of sound map) that is overlaid on the B-mode and / or Doppler images. In some examples, scan converter 230 matches the tissue parameter measurements with the B-mode and / or Doppler images.

[0021]

[0028] The multiplanar reformatter 232 can convert echoes received from points in a common plane (e.g., slice) in a volumetric region of the body into an ultrasound image (e.g., a B-mode image) of this plane, as described, for example, in U.S. Pat. No. 6,443,896 (Detmer). In some examples, the user interface 224 is coupled to the multiplanar reformatter 232 for selection and control of the display of multiple multiplanar reformatted (MPR) images. In other words, the user selects a desired plane within the volume for which a 2D image will be generated. In some examples, in addition to selecting the location and / or orientation of the plane within the volume, the user also selects the thickness of the plane. In some examples, the plane data of the multiplanar reformatter 232 is provided to a volume renderer 234.

[0022]

[0029] The volume renderer 234 generates an image (also called a projection, a 3D scene, or a rendering) of the 3D dataset as viewed from a given reference point, for example, as described in U.S. Patent No. 6,530,885 (Entrekin et al.). In some examples, the image generated by the volume renderer 234 is based on a volume having voxels (e.g., a 3D dataset), but the final image rendered by the volume renderer 234 is a 2D dataset having pixels (e.g., a 2D image). The 2D image is then displayed on a conventional display (e.g., a liquid crystal display).

[0023]

[0030] According to examples of the present disclosure, the volume renderer 234 receives 2D and / or 3D B-mode images from the scan converter 230 and / or the multiplanar reformatter 232. The 2D images include a 2D dataset having pixels with intensity values. The 3D images include a 3D dataset having voxels with intensity values. In some examples, the volume renderer 234 converts the 2D dataset into a 3D dataset, as described in more detail in European Patent Application No. 20290062.7, filed August 20, 2020, entitled "RENDERING OF TWO DIMENSIONAL DATA SETS." In other examples, the volume renderer 234 converts the 2D dataset into a single layer of voxels with intensity values ​​corresponding to the pixel intensity values. The volume renderer 234 receives tissue parameter measurements corresponding to the 2D or 3D images from the scan converter 230 and / or the multiplanar reformatter 232. In some examples, the volume renderer 234 assigns material properties (e.g., color, absorption, scattering) to each voxel in the 3D dataset based at least in part on the tissue parameter measurements. The volume renderer 234 generates a rendering (e.g., an enhanced B-mode rendering) based at least in part on the material properties assigned to the voxels. The volume renderer 234, in some examples, simulates at least one light source when generating the rendered image.

[0024]

[0031] In some examples, the volume renderer 234 applies other segmentation techniques, such as B-mode texture analysis, anatomical models, and / or artificial intelligence / machine learning (e.g., neural networks), to distinguish tissue types. The volume renderer 234 assigns material properties to voxels based at least in part on the tissue types determined by the other segmentation techniques. In some cases, when other segmentation techniques are used, the volume renderer 234 does not use tissue parameter measurements, and / or the parameter processor 266 is omitted. In some cases, the other segmentation techniques are used to supplement and / or enhance the tissue parameter measurements to assign material properties to voxels. For example, the resolution of the tissue parameter measurements is increased to more closely match the resolution of the B-mode image.

[0025]

[0032] Output from the scan converter 230 (e.g., B-mode image, Doppler image), multiplanar reformatter 232, and / or volume renderer 234 (e.g., rendering, 3D scene) is coupled to an image processor 236 for further enhancement, buffering, and temporary storage before being displayed on an image display 238. In some examples, the Doppler image is overlaid by the scan converter 230 and / or image processor 236 onto a B-mode image of the tissue structure for display.

[0026]

[0033] The graphics processor 240 generates graphic overlays for display on the images, including standard identifying information such as the patient name, the date and time of the image, imaging parameters, and the like, for which the graphics processor 240 is configured to receive input from the user interface 224, such as a typed patient name or other annotations.

[0027]

[0034] System 200 includes local memory 242. Local memory 242 may be provided as any suitable non-transitory computer-readable medium (e.g., flash drive, disk drive) that stores data generated by system 200, including images, anatomical models, rendering "recipes," executable instructions, input provided by a user via user interface 224, or any other information necessary for the operation of system 200.

[0028]

[0035] As described above, system 200 includes a user interface 224. User interface 224 includes a display 238 and a control panel 252. Display 238 includes a display device implemented using various known display technologies, such as LCD, LED, OLED, or plasma display technology. In some examples, display 238 includes multiple displays. Control panel 252 is configured to receive user input (e.g., desired image plane, desired light source, etc.). Control panel 252 includes one or more hard controls (e.g., buttons, knobs, dials, encoders, mouse, trackball, or the like). In some examples, control panel 252 additionally or alternatively includes soft controls (e.g., GUI control elements, or simply GUI controls) implemented on a touch-sensitive display. In some examples, display 238 is a touch-sensitive display that includes one or more soft controls of control panel 252.

[0029]

[0036] In some examples, the various components shown in FIG. 2 are combined. For example, image processor 236 and graphics processor 240 are implemented as a single processor. In another example, Doppler processor 260 and B-mode processor 228 are implemented as a single processor. In some examples, the various components shown in FIG. 2 are implemented as separate components. For example, signal processor 226 is implemented as a separate signal processor for each imaging mode (e.g., B-mode, Doppler). In some examples, one or more of the various processors shown in FIG. 2 are implemented by a general-purpose processor and / or microprocessor configured to perform a specified task. In some examples, one or more of the various processors are implemented as application-specific circuitry. In some examples, one or more of the various processors (e.g., image processor 236) are implemented by one or more graphical processing units (GPUs).

[0030]

[0037] Figure 3 is a block diagram illustrating an example processor 300 in accordance with the principles of the present disclosure. Processor 300 may be used to provide one or more of the processors described herein, such as image processor 236 shown in Figure 2. Processor 300 may be any suitable processor type, including, but not limited to, a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA) programmed to form a processor, a graphical processing unit (GPU), an application specific integrated circuit (ASIC) designed to form a processor, or a combination thereof.

[0031]

[0038] The processor 300 includes one or more cores 302. The cores 302 include one or more arithmetic logic units (ALUs) 304. In some examples, the cores 302 include a floating point logic unit (FPLU) 306 and / or a digital signal processing unit (DSPU) 308 in addition to or as an alternative to the ALUs 304.

[0032]

[0039] The processor 300 includes one or more registers 312 communicatively coupled to the core 302. The registers 312 may be implemented using dedicated logic gate circuits (e.g., flip-flops) and / or any memory technology. In some examples, the registers 312 may be implemented using static memory. The registers provide data, instructions, and addresses to the core 302.

[0033]

[0040] In some examples, processor 300 includes one or more levels of cache memory 310 communicatively coupled to core 302. Cache memory 310 provides computer-readable instructions to core 302 for execution. Cache memory 310 provides data for processing by core 302. In some examples, computer-readable instructions are provided to cache memory 310 by local memory, such as, for example, local memory attached to external bus 316. Cache memory 310 may be provided in any suitable cache memory type, such as, for example, metal-oxide-semiconductor (MOS) memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), and / or any other suitable memory technology.

[0034]

[0041] Processor 300 includes a controller 314 that controls input to processor 300 from other processors and / or components in the system (e.g., control panel 252 and scan converter 230 shown in FIG. 2 ) and / or output from processor 300 to other processors and / or components in the system (e.g., display 238 and volume renderer 234 shown in FIG. 2 ). Controller 314 controls data paths in ALU 304, FPLU 306, and / or DSPU 308. Controller 314 may be implemented as one or more state machines, data paths, and / or dedicated control logic. The gates of controller 314 may be implemented as standalone gates, FPGAs, ASICs, or any other suitable technology.

[0035]

[0042] Registers 312 and cache memory 310 communicate with controller 314 and core 302 via internal connections 320A, 320B, 320C, and 320D, which may be implemented as buses, multiplexers, crossbar switches, and / or any other suitable connection technology.

[0036]

[0043] Input and output to processor 300 are provided via bus 316, which comprises one or more conductive lines. Bus 316 is communicatively coupled to one or more components of processor 300, such as controller 314, cache memory 310, and / or registers 312. Bus 316 is coupled to one or more components of the system, such as display 238 and control panel 252, as previously described.

[0037]

[0044] The bus 316 is coupled to one or more external memories. The external memories include read-only memory (ROM) 332. The ROM 332 may be masked ROM, electronically programmable read-only memory (EPROM), or any other suitable technology. The external memories include random access memory (RAM) 333. The RAM 333 may be static RAM, battery-backed static RAM, dynamic RAM (DRAM), or any other suitable technology. The external memories include electrically erasable programmable read-only memory (EEPROM) 335. The external memories include flash memory 334. The external memories include magnetic storage devices such as disks 336. In some examples, the external memories are included in a system such as the ultrasound imaging system 200 shown in FIG. 2, for example, as local memory 242.

[0038]

[0045] The functionality performed by a volume renderer, such as volume renderer 234, to generate renderings from B-mode imaging and / or multi-parametric imaging according to examples of the present disclosure will now be described in more detail with reference to the following figures.

[0039]

[0046] FIG. 4 shows a graphical representation of a process for generating renderings from B-mode and multi-parametric imaging according to an example of the present disclosure, the process being provided or embodied in one or more processors, such as the volume renderer 234, the image processor 236, or a combination thereof, of the ultrasound imaging system 200. A volumetric B-mode image 402 and a volumetric tissue parameter map 404 are received by a processor (e.g., the volume renderer 234). In the example shown in FIG. 4, both the B-mode image 402 and the tissue parameter map 404 are 3D datasets. In some examples, the datasets were 2D datasets converted to 3D datasets, as described with reference to FIG. 2. In some examples, the datasets were originally 3D datasets. In some examples, the B-mode image 402 comprises a volumetric dataset of intensity values ​​I(x, y, z) assigned to voxels. The intensity values ​​correspond to the intensities of echo signals received from spatial locations within the imaged volume associated with the voxels. Each voxel corresponds to a different spatial location within the imaged volume. The tissue parameter map 404 comprises a volumetric dataset of tissue parameter values ​​P(x,y,z). In some examples, the tissue parameter map 404 comprises values ​​of multiple parameters (e.g., density, speed of sound, attenuation) for each voxel of the dataset. Each voxel corresponds to a different spatial location within the scanned volume. In some examples, there is a one-to-one correspondence between the voxels of the B-mode image 402 and the tissue parameter map 404.

[0040]

[0047] In some examples, the B-mode image 402 and the tissue parameter map 404 are acquired from the same volume of the subject. The B-mode image 402 and the tissue parameter map 404 are acquired during the same acquisition. In other words, the B-mode image 402 and the tissue parameter map 404 are generated from echo-based signals received by a transducer array (e.g., transducer array 214), where the echoes are in response to the same set of transmitted ultrasound signals. In other examples, the B-mode image 402 and the tissue parameter map 404 are acquired during different acquisitions. For example, this occurs when one or more of the tissue parameters of interest require different acquisition settings (e.g., different frequency, pulse width, pulse duration, etc.) than the B-mode signals.

[0041]

[0048] The processor generates a 3D rendering dataset having voxels 406 by assigning one or more material properties to the voxels 406 based on the B-mode image 402 and the tissue parameter map 404. In some examples, there is a voxel 406 corresponding to every voxel in the B-mode image 402.

[0042]

[0049] In some examples, as shown in box 408, the processor assigns values ​​for one or more material properties based on a function of the B-mode image 402 and / or the tissue parameter map 404. For example, the material property is a function of intensity I and / or a tissue parameter value P. In the example shown, two material properties, absorption A and scattering S, are shown, although in other examples, more or fewer material properties may be assigned values. Continuing the example, absorption is based on a function of both intensity from the B-mode image 402 and the tissue parameter value from the tissue parameter map 404, and scattering is based on a function of the tissue parameter value from the tissue parameter map 404. Still, this is provided by way of example only; in other examples, all material properties are a function of both the B-mode image 402 and the tissue parameter map 404, or some material properties are solely a function of the B-mode image 402. In some examples, the material properties are defined by a transfer function based on intensity and / or tissue parameter values.

[0043]

[0050] In some examples, as shown in box 410, the processor analyzes tissue parameter values ​​from the tissue parameter map 404 to determine the tissue type at each voxel 406. In some examples, the tissue parameter values ​​are compared to a table to determine the tissue type. An example of a tissue parameter table 500 accessed by a processor (e.g., volume renderer 234) is shown in FIG. 5. In some examples, the B-mode image 402 is further analyzed to determine the tissue type. Once the tissue type is determined, the processor assigns material property values ​​based on a profile of the particular tissue type. By profile, it is meant that the material properties 406 assigned to the voxel of the tissue type are predefined rather than directly calculated based on the tissue parameters (e.g., as described with reference to box 408). In the example shown in FIG. 4, the profile includes absorption and scattering values. Naturally, fewer or additional material properties may be included in the profile. Furthermore, while only three tissue types are shown in box 410, profiles of more or fewer tissue types may be included.

[0044]

[0051] Applying material properties based on a function of intensity and / or tissue parameters (e.g., as shown in box 408) rather than based on a predetermined profile (e.g., as shown in box 410) is preferred for enhanced rendering and in some applications when one or more tissue parameters (e.g., stiffness, attenuation) are to be conveyed in information about the imaged anatomical structure (e.g., B-mode image data). For example, material properties may be assigned such that fat tissue is rendered a different color than liver tissue, making it easier for a user to distinguish the fat tissue from the liver tissue. In the same example, voxels associated with liver tissue may be further assigned different intensity and / or “brightness” values ​​based on the measured stiffness values. Thus, while voxels of liver tissue have the same color, different parts of the liver may appear dimmer and / or duller based at least in part on the stiffness values. Nevertheless, applying a predetermined profile based on tissue type, as described with reference to box 410, uses less computational power and / or avoids confusing a user with irrelevant information when the actual tissue parameter values ​​are of no or close to interest to the user.

[0045]

[0052] In some examples, values ​​of material properties are assigned based on a combination of a predetermined profile and a function of intensity and / or tissue parameters. For example, the tissue type of voxel 406 is determined based on the intensity and / or tissue parameters. Based on the tissue type, a corresponding profile of material properties is used to assign values ​​of some of the material properties. Nevertheless, one or more of the material properties do not have predetermined values ​​but are assigned based on a function of intensity and / or tissue parameter values. In some examples, the function for use of the material properties is stored in the tissue type profile.

[0046]

[0053] Once material properties have been assigned to the voxels 406, a processor (e.g., volume renderer 243) generates a rendering 412 from the voxels 406. In some examples, the rendering 412 is shown on a display, such as display 238. As described in more detail with reference to FIG. 8, the processor, in some examples, generates the rendering 412 using a two-step rendering process. In the example shown in FIG. 4, the B-mode image 402 is of a portion of the section, and the tissue parameter map 404 shows corresponding speed of sound values ​​in the tissue. In the rendering 412, the upper layer of fat 414 and the lower layer of muscle 416 are more easily distinguishable from the B-mode image 402. Yet, unlike when the tissue parameter map 404 is simply overlaid on the B-mode image 402, the details of the B-mode image 402 are not lost in the rendering 412.

[0047]

[0054] 6 shows a graphical representation of a process for generating a rendering from B-mode imaging, according to an example of the present disclosure. A volumetric B-mode image 602 is received by a processor (e.g., volume renderer 234). In some examples, a volumetric tissue parameter map 604 is also received by the processor. The B-mode image 602 and the tissue parameter map 604 are substantially the same as the B-mode image 402 and the tissue parameter map 404.

[0048]

[0055] In the example shown in FIG. 6 , a processor (e.g., volume renderer 234, image processor, or another or any combination thereof) applies one or more image segmentation techniques to B-mode image 602 and / or tissue parameter map 604 to determine material properties to assign to each voxel. As shown in box 620, the processor provides or communicates with a neural network to determine the material properties. However, other artificial intelligence or machine learning techniques may be used. Training an AI / machine learning model to perform the segmentation is described in more detail with reference to FIG. 7 . Other non-AI / machine learning techniques may also be used, such as edge detection, B-mode texture analysis, and / or anatomical modeling (examples of which can be found in U.S. Pat. No. 7,957,572). These other techniques may be used instead of or in addition to AI / machine learning techniques. Based on the segmentation, the processor then assigns one or more material properties to voxels 606 of the volume data set. In some examples, the volume data set has a voxel corresponding to every voxel in B-mode image 602.

[0049]

[0056] In some examples, as shown in box 608, a processor (e.g., volume renderer 234) assigns values ​​for one or more material properties based on a function of the segmentation technique used, similar to the example described with reference to box 408 shown in FIG. 4. In the example shown, two material properties, absorption A and scattering S, are shown, although in other examples, values ​​may be assigned to more or fewer material properties. For example, the output of a neural network or other segmentation technique performed in box 620 is a set of material property values ​​for each voxel 606. In some examples, as shown in box 610, the segmentation determines a tissue type at each voxel 606. For example, the output of a machine learning model is a tissue type for each voxel 606. Once the tissue type is determined, the processor applies a tissue-type-specific profile of material properties to the voxel 606, similar to the example described with reference to box 410 shown in FIG. 4. In some examples, the value of the material property is assigned based on a combination of the predetermined profile and the output from the segmentation shown in box 620. For example, the tissue type of voxel 606 is determined based on the segmentation. Based on the tissue type, a corresponding profile of material properties is used to assign values ​​for some of the material properties. Nevertheless, one or more of the material properties do not have predetermined values ​​and are assigned based on the separate output of the segmentation shown in box 620. Once material properties have been assigned to voxel 606, the volume renderer generates an enhanced B-mode rendering 612 from voxel 606 in the same manner as described with reference to FIG. 4.

[0050]

[0057] In some examples, such as those described with reference to FIG. 6 , a processor (e.g., volume renderer 234) may include any one or more machine learning models, artificial intelligence algorithms, and / or neural networks to perform image segmentation. In some examples, a processor (e.g., volume renderer 234, image processor 236, or another processor, or a combination thereof) may include a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder neural network, or the like to determine the material properties of a voxel and / or the tissue type of a voxel. The models and / or neural networks may be implemented with hardware (e.g., neurons are represented by physical components) and / or software (e.g., neurons and pathways are implemented in a software application) components. Models and / or neural networks implemented according to the present disclosure may use various topologies and learning algorithms to train the models and / or neural networks to produce desired outputs. For example, a software-based neural network may be implemented using a processor (e.g., a single or multi-core CPU, a single GPU or GPU cluster, or multiple processors arranged for parallel processing) configured to execute instructions that are stored on a computer-readable medium and that, when executed, cause the processor to perform a trained algorithm to determine material properties of voxels and / or to determine tissue types of voxels. In some embodiments, the processor (e.g., a volume renderer) implements the model and / or neural network in combination with other image processing methods (e.g., segmentation, histogram analysis).

[0051]

[0058] In various examples, the model and / or neural network is trained using any of a variety of now-known or later-developed learning techniques to obtain a model and / or neural network (e.g., a trained algorithm, transfer function, or hardware-based system of nodes) configured to analyze input data in the form of ultrasound images, measurements, and / or statistics. In some embodiments, the model and / or neural network is trained statically. That is, the model and / or neural network is trained using a data set and then deployed to an ultrasound system (e.g., system 200), such as a processor, e.g., volume renderer 234. In some embodiments, the model and / or neural network is trained dynamically. In these embodiments, the model and / or neural network is trained using an initial data set and deployed to the ultrasound system. Nevertheless, the model and / or neural network continues to train and be modified based on ultrasound images acquired by the ultrasound system after deployment of the model and / or neural network to the ultrasound system.

[0052]

[0059] Figure 7 shows a block diagram of a process for model training and deployment, according to an example of the present disclosure. The process shown in Figure 7 is used to train a model (e.g., artificial intelligence, machine learning, deep learning) implemented by a processor (e.g., volume renderer 234) of an ultrasound system. For example, the process shown in Figure 7 is used to train a model used to segment images, as shown in block 620 of Figure 6.

[0053]

[0060] Phase 1 on the left side of Figure 7 illustrates model training. To train a model, a training set containing multiple instances of an input array and an output classification is presented to the model's training algorithm (e.g., the AlexNet training algorithm, as described by Krizhevsky, A., Sutskever, I., and Hinton, G.E., "ImageNet Classification with Deep Convolutional Neural Networks," NIPS 2012 or its successors). Training involves selecting a start algorithm and / or network architecture 712 and preparing training data 714. The start architecture 712 can be a blank architecture (e.g., an architecture with a defined layer and node arrangement but no previously trained weights, a defined algorithm with or without a set number of regression coefficients) or a partially trained model, such as an Inception network, that is subsequently further developed for analysis of ultrasound data. The start architecture 712 (e.g., blank weights) and training data 714 are provided to the training engine 710 to train the model. After a sufficient number of iterations (e.g., once the model consistently performs within an acceptable error), the model 720 is considered trained and ready for deployment, which is shown in the middle of FIG. 7 , Phase 2. On the right side of FIG. 7 , i.e., Phase 3, the trained model 720 is applied (via the inference engine 730) to analyze new data 732, data that was not presented to the model during initial training (in Phase 1). For example, the new data 732 includes unknown data, such as live ultrasound images acquired during a patient scan (e.g., cardiac images during an echocardiogram). The trained model 720, implemented via the engine 730, is used to analyze the unknown data according to the training of the model 720 to provide output 734 (e.g., tissue type, material property values). The output 734 is then used by the system for subsequent processes 740 (e.g., assigning material properties to voxels, generating renderings).

[0054]

[0061] In embodiments in which the trained model 720 is used as a model implemented or embodied by a processor (e.g., volume renderer 234) of the ultrasound system, the starting architecture is a convolutional neural network architecture, or in some examples, a deep convolutional neural network architecture trained to provide one or more material property values ​​of voxels and / or tissue types of voxels. The training data 714 includes a plurality (hundreds, often thousands, or more) of annotated / labeled images, also referred to as training images. For example, voxels in the training images are labeled with tissue types and / or material properties. It will be understood that the training images need not include complete images produced by the imaging system (e.g., representing the complete field of view of the ultrasound probe), but may include patches or portions of images.

[0055]

[0062] In some examples, when insufficient ultrasound training data exists, images from other modalities (e.g., computed tomography, magnetic resonance imaging, or even photographs from the National Institutes of Health Visualized Human Image Data Project) are used as training data. In some examples, ultrasound data is simulated from images from other modalities using an ultrasound simulator such as k-wave or FAMUS II. The simulated ultrasound data is labeled and then used as training data.

[0056]

[0063] In some examples, labeled tissue parameter maps are also provided as training data. In these examples, the model is trained to segment images and / or assign material property values ​​to voxels based on the B-mode images and the tissue parameter maps. In some examples, multiple types of tissue parameter maps (e.g., speed of sound, attenuation, and stiffness maps) are provided as training data.

[0057]

[0064] In various embodiments, the trained model is implemented, at least in part, in a computer-readable medium having executable instructions that are executed by one or more ultrasound system processors, such as the volume renderer 234.

[0058]

[0065] FIG. 8 provides a graphical overview of a two-step rendering process according to examples of the present disclosure. In some examples, the rendering process shown in FIG. 8 is used to generate an enhanced B-mode rendering, such as renderings 412 and / or 612. In some examples, a processor of the ultrasound system, such as volume renderer 234, performs an illumination pass and a compositing pass to generate the rendering. In some examples, the illumination pass is performed before the compositing pass. During the illumination pass, a virtual light source 800 is simulated for volume 810. Volume 810 has voxels that have been assigned material properties by a processor (e.g., volume renderer 234) based at least in part on the B-mode image. The material properties, in some examples, are assigned using the methods described with reference to FIGS. 4-6. In some examples, such as the one shown in FIG. 6, light source 800 is a point light source that emits light in all directions with equal intensity. In other examples, light source 800 is a directional light source (e.g., a spotlight, a beam). In some examples, multiple light sources are simulated. The number of light sources, the type of light sources, the position of light sources 800 relative to volume 810, and / or the intensity of light sources 800 are preset or selected by a user via a user interface. In some examples, other characteristics of the light sources, such as the size of light sources 800 and / or the color (e.g., wavelength range) of light sources 800, are also selected by the user. Examples of suitable light sources and user control of light sources can be found in U.S. Patent Application Nos. 16 / 306,951 and 16 / 347,739, the contents of which are incorporated herein by reference for any purpose.

[0059]

[0066] Continuing with the lighting pass, light from light source 800 is propagated in three-dimensional space through volume 810. While only a single light ray 802 is shown in FIG. 8, it is understood that many light rays are propagated through volume 810 to simulate light source 800. The amount of light that reaches each voxel in volume 810 is calculated according to the properties of light source 800 and the material properties assigned to each voxel. For example, certain voxels have wavelengths that are dependent on absorption, which causes the voxels to appear a particular color. For example, voxels determined to correspond to muscle tissue may absorb less red wavelengths, causing these voxels to appear red in the final rendering. The calculated amount of light for each voxel is stored for use in the compositing pass.

[0060]

[0067] During a synthesis pass, parallel light rays 804 propagate from an observation plane 808 of a virtual observer 806 through a volume 810. In some examples, the light rays 804 are perpendicular to the observation plane 808. The distance of the observation plane 808 from the volume 810 and / or the orientation of the observation plane 808 relative to the volume 810 are preset or selected by a user via a user interface. In some examples, a processor (e.g., the volume renderer 234) implements a ray marching numerical scheme that uses front-to-back red-green-blue (RGB) accumulation with tri-linear interpolated volumetric samples 812 along the direction of the light rays 804. Based on the light calculated for each voxel in the illumination path and the progression of ray 804, a processor (e.g., volume renderer 234) calculates the final values ​​of the voxels and / or pixels of the rendering that are shown to the user on a display (e.g., display 238).

[0061]

[0068] As disclosed herein, systems and methods using multi-parametric ultrasound images are used to determine different tissue types and / or assign material properties to voxels in B-mode images. In some examples, other or additional techniques are used to determine tissue types and / or assign material properties to voxels. Voxels are assigned different values ​​of color, density, scatter, and / or other material properties based at least in part on the tissue type. A color B-mode image is generated based on the material property values ​​assigned to the voxels. B-mode images generated by the systems and methods disclosed herein appear more realistic and / or make tissue types easier to distinguish and / or interpret. In some examples, multiple parameters are conveyed in the rendered B-mode image without requiring the user to analyze separate images and parameter maps or obscuring the B-mode image with parameter maps.

[0062]

[0069] In various examples in which components, systems, and / or methods are provided using a computer-based system or a programmable device, such as programmable logic, it should be understood that the systems and methods described above can be implemented using any of a variety of known or later-developed programming languages, such as "C," "C++," "FORTRAN," "Pascal," "VHDL," and the like. Accordingly, various storage media, such as magnetic computer disks, optical disks, electronic memory, and the like, can be provided, which can contain information capable of directing a device, such as a computer, to perform the systems and / or methods described above. When the information and programs contained on the storage media are accessed by an appropriate device, the storage media can provide the information and programs to the device, thus enabling the device to perform the functions of the systems and / or methods described herein. For example, when a computer is provided with a computer disk containing appropriate material, such as source files, object files, executable files, or the like, the computer receives the information and appropriately configures itself to perform the various functions, thereby performing the functions of the various systems and methods outlined in the diagrams and flowcharts above. That is, the computer can receive various portions of information from the disc relating to different elements of the systems and / or methods described above, provide the individual systems and / or methods, and coordinate the functionality of the individual systems and / or methods described above.

[0063]

[0070] In light of the present disclosure, it is noted that the various methods and devices described herein can be implemented in hardware, software, and / or firmware. Furthermore, various methods and parameters are included by way of example only and not in any limiting sense. In light of the present disclosure, one skilled in the art will be able to implement the present teachings in determining the techniques of the present teachings themselves and the equipment necessary to effect these techniques while remaining within the scope of the present invention. One or more functions of the processor described herein may be implemented using an application-specific integrated circuit (ASIC) or general-purpose processing circuitry incorporated into fewer or a single processing unit (e.g., a CPU) and programmed with executable instructions to perform the functions described herein.

[0064]

[0071] While the present devices, systems, and methods have been described with particular reference to ultrasound imaging systems, it is further envisioned that the present systems are extendable to other medical imaging systems in which one or more images are systematically acquired. Thus, the present systems may be used to acquire and / or record image information relating to, but not limited to, the kidneys, testes, breasts, ovaries, uterus, thyroid, liver, lungs, musculoskeletal, spleen, heart, arteries, and vascular systems, as well as other imaging applications related to ultrasound-guided interventions. Furthermore, the present systems may also incorporate one or more programs used in conventional imaging systems that provide the features and advantages of the present systems. Certain additional advantages and features of the present disclosure will be apparent to those skilled in the art upon studying the present disclosure or will be experienced by those employing the novel systems and methods of the present disclosure. Another advantage of the present systems and methods is that conventional medical imaging systems can be readily upgraded to incorporate the features and advantages of the present systems, devices, and methods.

[0065]

[0072] Of course, it should be recognized that any one of the examples, examples, or processes described herein may be combined with one or more other examples, examples, and / or processes, or may be separated and / or implemented in separate devices or device parts in accordance with the present apparatus, systems, devices, and methods.

[0066]

[0073] Finally, the foregoing discussion is intended to be merely illustrative of the present system and method, and should not be construed as limiting the appended claims to any particular example or group of examples. Accordingly, while the present system has been described in particular detail with reference to illustrative examples, it will be further recognized that numerous modifications and alternative examples may be devised by those skilled in the art without departing from the broader intended spirit and scope of the present system and method, as set forth in the following claims. Accordingly, the specification and drawings are to be considered illustrative, and are not intended to limit the scope of the appended claims.

Claims

1. receiving a B-mode image of an imaged volume, the B-mode image having a plurality of first voxels, each first voxel associated with a different spatial location within the imaged volume, each first voxel having an intensity value corresponding to an intensity of an echo signal received from the associated spatial location within the imaged volume; receiving a tissue parameter map having one or more tissue parameter values ​​for each of the first plurality of voxels; generating a 3D rendering data set having a plurality of second voxels by assigning one or more material property values ​​to each of the plurality of second voxels, wherein the one or more material property values ​​assigned to each of the plurality of second voxels are based at least in part on the intensity value associated with each of the first voxels in the B-mode image and further based at least in part on tissue parameter values ​​associated with each of the voxels obtained from the tissue parameter map; and generating an enhanced B-mode rendering from said 3D rendering dataset; a processor configured to: a display configured to display the enhanced B-mode rendering; and An apparatus comprising: the processor: determining a tissue type of a respective one of the plurality of first voxels based at least in part on the tissue parameter value; assigning the one or more material property values ​​based on the tissue type-based material property value profile; and Further, the device.

2. The apparatus of claim 1 , wherein the tissue type is determined by comparing the tissue parameter value to a table of tissue parameter values.

3. the processor: segmenting the B-mode image to determine a tissue type of a respective one of the first plurality of voxels; assigning the one or more material property values ​​based on the tissue type-based material property value profile; and The apparatus of claim 1 , further comprising:

4. The apparatus of claim 3 , wherein the B-mode image is segmented using a trained machine learning model.

5. The apparatus of claim 1 , wherein the tissue parameter values ​​include at least one of a speed of sound value, an attenuation value, a stiffness value, or a density value.

6. The apparatus of claim 1 , wherein the enhanced B-mode rendering is generated using a two-pass process having an illumination pass and a compositing pass.

7. receiving a B-mode image of an imaged volume, the B-mode image having a plurality of first voxels, each first voxel associated with a different spatial location within the imaged volume, each first voxel having an intensity value corresponding to an intensity of an echo signal received from the associated spatial location within the imaged volume; receiving a tissue parameter map having one or more tissue parameter values ​​for each of the first plurality of voxels; assigning one or more material characteristic values ​​to each of a plurality of second voxels of the 3D rendering data set, the one or more material characteristic values ​​assigned to each of the plurality of second voxels being based at least in part on the intensity value associated with each of the first voxels of the B-mode image and further based at least in part on tissue parameter values ​​associated with each of the voxels obtained from the tissue parameter map; generating an enhanced B-mode rendering from the 3D rendering dataset; 1. A method comprising: determining a tissue type of a respective one of the plurality of first voxels based at least in part on the tissue parameter value; assigning the one or more material property values ​​based on the tissue type-based material property value profile; The method further comprising:

8. 8. The method of claim 7, further comprising segmenting the B-mode image to determine a tissue type for a respective one of the first plurality of voxels, wherein assigning the one or more material characteristic values ​​is based on a profile of material characteristic values, the profile being based on tissue type.

9. The method of claim 8 , wherein the segmenting step is performed by a machine learning model.

10. generating a simulated ultrasound image from images acquired from another imaging modality; training the machine learning model to segment the B-mode image with the simulated ultrasound image; The method of claim 9 further comprising:

11. The method of claim 8 , wherein the segmenting step comprises analyzing the texture of the B-mode image.

12. The method of claim 8 , wherein the segmenting step comprises applying an anatomical model to the B-mode image.

13. The method of claim 7 , wherein determining the tissue type comprises comparing the tissue parameter value to a table of tissue parameter values.

14. generating the enhanced B-mode rendering comprises: propagating a ray of light from a virtual light source through the second plurality of voxels; propagating light through the plurality of second voxels normal to a viewing plane of a virtual observer; 8. The method of claim 7, comprising:

Images