Method and system for coloring a volume rendered image

Abstract

The present invention is entitled "Methods and Systems for Coloring Volume Rendered Images". The present invention provides various methods and systems for medical imaging. In one embodiment, the method includes displaying a volume rendered image from a 3D medical imaging dataset; positioning a first virtual marker within a rendering volume of the volume rendered image, the rendering volume being defined by the 3D medical imaging dataset; and illuminating the rendering volume by projecting simulated light from the first virtual marker. In this manner, the illumination of the rendering volume by the first virtual marker visually indicates the position and depth of the first virtual marker within the volume rendered image.

Description

Technical Field

[0001] Embodiments of the subject matter disclosed herein relate to medical imaging. Background Art

[0002] Some non-invasive medical imaging modalities (such as ultrasound) can acquire three-dimensional (3D) data sets. The 3D data set can be visualized with a volume rendered image, which is typically a 2D representation of the 3D medical imaging data set. There are currently many different techniques for generating volume rendered images. One such technique, ray casting, includes projecting multiple rays through the 3D medical imaging data set. Each sample (e.g., voxel) in the 3D medical imaging data set is mapped to a color and transparency. Data is accumulated along each ray. According to a common technique, the data accumulated along each ray is displayed as a pixel in the volume rendered image. In addition, to facilitate the visualization of target anatomical features, particularly on different volume rendered images across different views showing the 3D data set and / or on different 2D slices across the 3D data set, a user can position one or more annotations (referred to as virtual markers) within the 3D data set. When drawing an image from the 3D data set, these virtual markers can be included at one or more appropriate locations in the image. However, in some views, it may be difficult to judge the depth of the virtual marker. Summary of the invention

[0003] In one embodiment, a method includes: displaying a volume rendered image rendered from a 3D medical imaging dataset; positioning a first virtual marker within a rendered volume of the volume rendered image, the rendered volume being defined by the 3D medical imaging dataset; and illuminating the rendered volume by projecting simulated light from the first virtual marker. In this manner, illumination of the rendered volume by the first virtual marker visually indicates a position and depth of the first virtual marker within the volume rendered image.

[0004] It should be understood that the above brief description is provided to introduce in a simplified form selected concepts that are further described in the detailed description. It is not meant to identify key or essential features of the claimed subject matter, the scope of which is solely defined by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any disadvantages noted above or in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The present disclosure will be better understood by reading the following description of non-limiting embodiments with reference to the accompanying drawings, in which:

[0006] Figure 1 An exemplary ultrasound imaging system according to one embodiment is shown;

[0007] Figure 2is a schematic diagram of a geometric shape that may be used to generate a volume rendered image according to one embodiment;

[0008] Figure 3 is a flow chart illustrating a method for generating a volume rendered image from a 3D data set;

[0009] Figure 4 is a schematic diagram of multiple light sources and orientation of a 3D medical imaging data set according to one embodiment;

[0010] Figure 5 is an exemplary volume rendered image including three virtual markers; and

[0011] Figure 6 An exemplary volume rendered image is shown with three virtual markers and with corresponding illumination from simulated light projected from each virtual marker. DETAILED DESCRIPTION

[0012] The following description relates to a method for utilizing a medical imaging system such as Figure 1 Various embodiments of non-invasive volumetric medical imaging (such as volumetric ultrasound imaging) performed by an ultrasound imaging system using a medical imaging system. In particular, the following description relates to coloring a volume rendered image generated from a volumetric data set acquired from a medical imaging system. Figure 2 The volume rendered image may be colored with a light source associated with the virtual marker to provide depth cues to enhance the determination of the location of the virtual marker, such as by Figure 3 As shown in the method. In order to obtain an additional sense of depth and perspective, the volume rendered image is usually colored with one or more external light sources based on the direction of the light. Coloring can be used to convey the relative positioning of structures or surfaces in the volume rendered image. Coloring helps the viewer to more easily visualize the three-dimensional shape of the object represented by the volume rendered image. Virtual markers may be present in the volume rendered image to mark target anatomical features. However, despite the coloring from the external light source, it may be difficult for the user of the medical imaging system or other clinicians to judge the depth of the virtual marker in the volume rendered image. Therefore, according to the embodiments disclosed herein, the virtual marker itself can act as a light source for the purpose of coloring the volume rendered image. The virtual marker (or the light source associated with the virtual marker) can simulate light together with one or more external light sources (such as Figure 4) is projected onto structures surrounding a virtual marker in a volume rendered image. The projected light may have an intensity that decreases as a function of distance from the light source, and may cast shadows on structures in the volume rendered image similar to real light. In at least some examples, the virtual marker may be positioned in accordance with a user request, and may be moved in accordance with a user request. The light source associated with the virtual marker may also move in tandem with the virtual marker, and the shading of the volume rendered image may be updated as the virtual marker (and therefore the light source) moves. In addition, a user of the medical imaging system (or other end user, such as a clinician viewing the volume rendered image on an external display device) may adjust the intensity of light projected from one or more virtual marker light sources. When multiple virtual markers are present in the same 3D data set, each virtual marker may be assigned a different color, and the light source may also project light having the assigned color to improve visual clarity between the virtual markers, such as Figure 5 and Figure 6 In doing so, a viewer of the volume rendered image can more easily and quickly determine the depth of each virtual marker.

[0013] Figure 1 1 is a schematic diagram of an ultrasound imaging system 100 according to one embodiment. The ultrasound imaging system 100 includes a transmit beamformer 101 and a transmitter 102, which drive elements 104 within a transducer array or ultrasound probe 106 to transmit pulsed ultrasound signals into the body (not shown). The ultrasound probe 106 may, for example, include a linear array probe, a curved array probe, a sector probe, or any other type of ultrasound probe. Therefore, the elements 104 of the ultrasound probe 106 may be arranged in a one-dimensional (1D) or 2D array. Still referring to Figure 1 , the ultrasound signal is backscattered from the body structure to produce an echo that returns to the element 104. The echo is converted into an electrical signal or ultrasound data by the element 104, and the electrical signal is received by the receiver 108. The electrical signal representing the received echo passes through the receive beamformer 110 that outputs the ultrasound data. According to some embodiments, the probe 106 may include electronic circuits to perform all or part of the transmit beamforming and / or receive beamforming. For example, all or part of the transmit beamformer 101, the transmitter 102, the receiver 108, and the receive beamformer 110 may be located within the ultrasound probe 106. In the present disclosure, the terms "scanning" or "scanning" may also be used to refer to the acquisition of data by the process of transmitting and receiving ultrasound signals. In the present disclosure, the terms "data" and "ultrasound data" may be used to refer to one or more data sets acquired with an ultrasound imaging system.

[0014] The user interface 115 may be used to control the operation of the ultrasound imaging system 100, including input for controlling patient data, for changing scanning or display parameters, for selecting various modes, operations and parameters, etc. The user interface 115 may include one or more of the following: a rotary member, a mouse, a keyboard, a trackball, hard keys linked to specific actions, soft keys that can be configured to control different functions, a graphical user interface displayed on the display device 118 (in an embodiment where the display device 118 includes a touch-sensitive display device or a touch screen), etc. In some examples, the user interface 115 may include a proximity sensor configured to detect an object or gesture within a few centimeters of the proximity sensor. The proximity sensor may be located on the display device 118 or as part of a touch screen. The user interface 115 may include, for example, a touch screen positioned in front of the display device 118, or the touch screen may be separate from the display device 118. The user interface 115 may also include one or more physical controls, such as buttons, sliders, knobs, keyboards, mice, trackballs, etc., either alone or in combination with a graphical user interface icon displayed on the display device 118. The display device 118 may be configured to display a graphical user interface (GUI) according to instructions stored in the memory 120. The GUI may include user interface icons representing commands and instructions. The user interface icons of the GUI are configured so that a user can select commands associated with each specific user interface icon in order to start various functions controlled by the GUI. For example, various user interface icons may be used to represent windows, menus, buttons, cursors, scroll bars, etc. According to an embodiment in which the user interface 115 includes a touch screen, the touch screen may be configured to interact with the GUI displayed on the display device 118. The touch screen may be a single-touch touch screen configured to detect a single contact point at a time, or the touch screen may be a multi-touch touch screen configured to detect multiple contact points at a time. For embodiments in which the touch screen is a multi-touch touch screen, the touch screen may be configured to detect a multi-touch gesture involving contact from two or more fingers of a user at a time. The touch screen may be a resistive touch screen, a capacitive touch screen, or any other type of touch screen configured to receive input from a stylus or one or more fingers of a user. According to other embodiments, the touch screen may include an optical touch screen that uses technology such as infrared light or other frequencies of light to detect one or more contact points initiated by a user.

[0015] According to various embodiments, the user interface 115 may include an off-the-shelf consumer electronic device, such as a smart phone, a tablet computer, a laptop computer, etc. For the purposes of this disclosure, the term "off-the-shelf consumer electronic device" is defined as an electronic device that is designed and developed for general consumer use and is not specifically designed for use in a medical environment. According to some embodiments, the consumer electronic device may be physically separate from the rest of the ultrasound imaging system 100. The consumer electronic device may communicate with the processor 116 via a wireless protocol such as Wi-Fi, Bluetooth, a wireless local area network (WLAN), near field communication, etc. According to one embodiment, the consumer electronic device may communicate with the processor 116 via an open application programming interface (API).

[0016] The ultrasound imaging system 100 also includes a processor 116 for controlling the transmit beamformer 101, the transmitter 102, the receiver 108, and the receive beamformer 110. The processor 116 is configured to receive input from the user interface 115. The receive beamformer 110 may include a conventional hardware beamformer or a software beamformer according to various embodiments. If the receive beamformer 110 is a software beamformer, the receive beamformer 110 may include one or more of the following: a graphics processing unit (GPU), a microprocessor, a central processing unit (CPU), a digital signal processor (DSP), or any other type of processor capable of performing logical operations. The receive beamformer 110 may be configured to perform conventional beamforming techniques as well as techniques such as retrospective transmit beamforming (RTB). If the receive beamformer 110 is a software beamformer, the processor 116 may be configured to perform some or all functions associated with the receive beamformer 110.

[0017] The processor 116 is in electronic communication with the ultrasound probe 106. For the purposes of this disclosure, the term "electronic communication" may be defined as including both wired communication and wireless communication. The processor 116 may control the ultrasound probe 106 to acquire data. The processor 116 controls which elements of the elements 104 are active and the shape of the beam emitted from the ultrasound probe 106. The processor 116 also communicates electronically with the display device 118, and the processor 116 may process the data into an image to be displayed on the display device 118. According to one embodiment, the processor 116 may include a CPU. According to other embodiments, the processor 116 may include other electronic components capable of performing processing functions, such as a GPU, a microprocessor, a DSP, a field programmable gate array (FPGA), or any other type of processor capable of performing logical operations. According to other embodiments, the processor 116 may include multiple electronic components capable of performing processing functions. For example, the processor 116 may include two or more electronic components selected from a list of electronic components, which include: a CPU, a DSP, an FPGA, and a GPU. According to another embodiment, the processor 116 may also include a composite demodulator (not shown) that demodulates the RF data and generates raw data. In another embodiment, demodulation can be performed earlier in the processing chain. The processor 116 is adapted to perform one or more processing operations according to a plurality of optional ultrasound modalities on the data. As the echo signals are received, the data can be processed in real time during the scanning session. For the purposes of this disclosure, the term "real time" is defined as including a process performed without any intentional delay. For example, an embodiment can acquire images at a real-time rate of 7 to 20 volumes / second. The ultrasound imaging system 100 can acquire 2D data of one or more planes at a significantly faster rate. However, it should be understood that the real-time volume rate may depend on the length of time it takes to acquire each volume of data for display. Therefore, when acquiring relatively large volumes of data, the real-time volume rate may be slower. Therefore, some embodiments may have a real-time volume rate significantly faster than 20 volumes / second, while other embodiments may have a real-time volume rate slower than 7 volumes / second. Data may be temporarily stored in a buffer (not shown) during a scanning session and processed in a less real-time manner in real-time or offline operation. Some embodiments of the present disclosure may include multiple processors (not shown) to handle the processing tasks handled by the processor 116 according to the exemplary embodiments described above. It should be appreciated that other embodiments may use different processor arrangements.

[0018] The ultrasound imaging system 100 can continuously acquire data at a volume rate of, for example, 10 Hz to 30 Hz. The image generated from the data can be refreshed at a similar frame rate. Other embodiments can acquire and display data at different rates. For example, depending on the size of the volume and the intended application, some embodiments can acquire data at a volume rate less than 10 Hz or greater than 30 Hz. A memory 120 is included to store the volume of processed acquired data. In an exemplary embodiment, the memory 120 has sufficient capacity to store a volume of ultrasound data equivalent to at least several seconds. The data volume is stored in a manner that facilitates retrieval based on the order or time of its acquisition. The memory 120 may include any known data storage medium.

[0019] Optionally, contrast agents can be used to implement the embodiments of the present disclosure. When using ultrasound contrast agents including microbubbles, contrast imaging generates enhanced images of anatomical structures and blood flow in the body. After collecting data using contrast agents, image analysis includes separating harmonic components and linear components, enhancing harmonic components, and generating ultrasound images by utilizing the enhanced harmonic components. Separation of harmonic components from the received signal is performed using appropriate filters. Using contrast agents for ultrasound imaging is well known to those skilled in the art and will not be described in detail.

[0020] In various embodiments of the present disclosure, the processor 116 may process data through other or different mode-related modules (e.g., B-mode, color Doppler, M-mode, color M-mode, spectral Doppler, elastic imaging, TVI, strain, strain rate, etc.) to form 2D or 3D data. For example, one or more modules may generate B-mode, color Doppler, M-mode, color M-mode, spectral Doppler, elastic imaging, TVI, strain, strain rate, and combinations thereof, etc. Image lines and / or volumes are stored, and timing information indicating the time when data was acquired in the memory may be recorded. These modules may include, for example, a scan conversion module for performing a scan conversion operation to convert the image volume from beam space coordinates to display space coordinates. A video processor module may be provided that reads the image volume from the memory and displays the image in real time while the patient is being operated on. The video processor module may store the image in an image memory, read and display the image from the image memory.

[0021] As mentioned above, the ultrasound probe 106 may include a linear probe or a curved array probe. Figure 1 Further depicted is a longitudinal axis 188 of the ultrasound probe 106. The longitudinal axis 188 of the ultrasound probe 106 extends through and parallel to the handle of the ultrasound probe 106. Furthermore, the longitudinal axis 188 of the ultrasound probe 106 is perpendicular to the array plane of the element 104.

[0022] Although an ultrasound system is described by way of example, it should be understood that the present techniques may also be useful when applied to images acquired using other imaging modalities, such as magnetic resonance imaging (MRI), CT, tomosynthesis, PET, C-arm angiography, etc. For example, a volumetric imaging dataset may be acquired using another suitable modality, such as MRI, and the virtual markers and light sources discussed herein may be applied to a volume rendered image generated from the volumetric magnetic resonance dataset. The present discussion of the ultrasound imaging modality is provided merely as an example of one suitable imaging modality.

[0023] Figure 2 is a schematic diagram of geometric shapes that may be used to generate a volume rendered image, according to one embodiment. Figure 2 The 3D medical imaging dataset 150 and the viewing plane 154 are included. The 3D medical imaging dataset 150 may be acquired using a suitable imaging modality. For example, an ultrasound probe of an ultrasound imaging system (e.g., Figure 1 The ultrasound imaging system 100 may be configured to acquire a 3D imaging dataset 150 using a probe 106 of the ultrasound imaging system 100. For example, the ultrasound probe may be scanned across a physical non-virtual volume (e.g., an abdomen or a torso of a patient) to generate the 3D medical imaging dataset 150, wherein the 3D medical imaging dataset 150 includes data (e.g., voxels) describing the physical non-virtual volume (e.g., in a configuration corresponding to the configuration of the physical non-virtual volume). The 3D medical imaging dataset 150 may be stored in a memory of the computing device (e.g., Figure 1 As described below, the processor (such as Figure 1 A processor 116) generates a volume rendered image from a 3D medical imaging dataset.

[0024] refer to Figure 1 and Figure 2 In addition, the processor 116 can generate the volume-rendered image according to a variety of different techniques. According to one embodiment, the processor 116 can generate the volume-rendered image from the view plane 154 through a ray casting technique. The processor 116 can cast a plurality of parallel rays from the view plane 154 to or through the 3D medical imaging dataset 150. Figure 2A first ray 156, a second ray 158, a third ray 160, and a fourth ray 162 are shown that define a viewing plane 154. It should be understood that additional rays may be cast to assign values ​​to all pixels 163 within the viewing plane 154. The 3D medical imaging data set 150 may include voxel data, where a voxel or volume element is assigned a value or intensity. In addition, an opacity may also be assigned to each voxel. The value or intensity may be mapped to a color according to some embodiments. The processor 116 may use a "front to back" or "back to front" technique for volume composition to assign a value to each pixel in the viewing plane 154 that is intersected by a ray. For example, starting from the front, i.e., from the direction in which the image is viewed, the intensities of all voxels along the corresponding ray may be summed. Then, optionally, the intensity may be multiplied by the opacity corresponding to the opacity of the voxel along the ray to generate an opacity weighted value. These opacity weighted values ​​are then accumulated along each ray in a front to back or back to front direction. The process of accumulating values ​​is repeated for each pixel 163 in the viewing plane 154 to generate a volume rendering image. According to one embodiment, the pixel values ​​from the viewing plane 154 can be displayed as a volume rendering image. The volume rendering algorithm can be additionally configured to use an opacity function that provides a gradual transition of opacity from zero (completely transparent) to 1.0 (completely opaque). When assigning values ​​to each of the pixels 163 in the viewing plane 154, the volume rendering algorithm can take into account the opacity of the voxels along each ray. For example, voxels with an opacity close to 1.0 will block most of the contribution of voxels further along the ray, while voxels with an opacity closer to zero will allow most of the contribution of voxels further along the ray. In addition, when visualizing the surface, a thresholding operation can be performed in which the opacity of the voxels is redistributed based on the value. According to an exemplary thresholding operation, the opacity of voxels with values ​​above the threshold can be set to 1.0, while the opacity of voxels with values ​​below the threshold can be set to zero. Other types of thresholding schemes can also be used. The opacity function may be used to assign an opacity other than zero and 1.0 to voxels whose values ​​are close to a threshold in a transition zone. The transition zone may be used to reduce artifacts that may occur when using a simple binary thresholding algorithm. For example, a linear function that maps opacity to a value may be used to assign opacity to voxels having a value in the transition zone. Other types of functions that progress from zero to 1.0 may also be used. Volume rendering techniques other than those described above may also be used to generate a volume rendered image from a 3D medical imaging dataset.

[0025] The volume rendered image may be colored to present a better depth perception to the user. This may be performed in several different ways according to various embodiments. For example, multiple surfaces may be defined based on volume rendering of a 3D medical imaging dataset. According to one embodiment, a gradient may be calculated at each pixel. Processor 116 ( Figure 1) can calculate the amount of light at the location corresponding to each pixel and apply one or more shading methods based on the gradient and the specific light direction. The viewing direction can correspond to Figure 2 . The processor 116 may also use multiple light sources as input when generating a volume rendered image. For example, as rays are cast, the processor 116 may calculate the amount of light reflected, scattered, or transmitted from each voxel in a particular viewing direction along each ray. This may involve summing the contributions from multiple light sources. The processor 116 may calculate the contributions from all voxels in the volume. The processor 116 may then synthesize the values ​​from all voxels or interpolate values ​​from neighboring voxels to calculate the final value of a pixel displayed on the image. Although the above examples describe an embodiment of integrating voxel values ​​along rays, the volume rendered image may also be calculated according to other techniques, such as using the highest value along each ray, using the average value along each ray, or using any other volume rendering technique.

[0026] Although the volume rendered image is a 2D rendering of image data included by the 3D medical imaging dataset 150 as viewed from the viewing plane 154, the volume rendered image has the appearance of depth (e.g., structures shown in the volume rendered image may be illuminated differently depending on the distance of a voxel in the 3D medical imaging dataset 150 from the viewing plane 154). The volume rendered image may be described herein as having a rendered volume, where the rendered volume is defined by the voxel data of the 3D medical imaging dataset and refers to the appearance of depth of the volume rendered image (e.g., as viewed from the viewing plane 154). Figures 5 and 6 Describes an example of drawing a volume.

[0027] Figure 3 is a flow chart illustrating a method 300 for generating a volume rendered image. Figure 1 , but it should be understood that the method 300 may be implemented with other systems and components without departing from the scope of the present disclosure. In some embodiments, the method 300 may be implemented as executable instructions in any suitable combination of the ultrasound imaging system 100, an edge device (e.g., an external computing device) connected to the ultrasound imaging system 100, a cloud in communication with the imaging system, etc. As an example, the method 300 may be implemented in a non-transitory memory of a computing device, such as Figure 1 A controller (eg, processor 116 and memory 120) of the ultrasound imaging system 100.

[0028] At 302, a 3D medical imaging dataset of a 3D volume is obtained. A suitable imaging modality (such as Figure 1The 3D data set is acquired by using an ultrasound probe 106 (e.g., an ultrasound probe 106), and the 3D volume may be a portion or the entirety of an imaged subject, such as a patient's heart. Thus, in some examples, a 3D data set may be generated from ultrasound data acquired via an ultrasound probe. The 3D medical imaging data set may include voxel data, where each voxel is assigned a value and an opacity. The value and opacity may correspond to the intensity of the voxel.

[0029] At 304, method 300 includes determining whether a request to include a virtual marker on and / or within the 3D data set is received. The virtual marker may be included in the 3D data set in response to a request from a user. For example, a user may select a menu item or control button displayed on a graphical user interface that indicates that the virtual marker will be positioned within the 3D data set. The virtual marker may indicate an anatomical feature of interest or otherwise mark an area of ​​interest of an imaged 3D volume, and may be displayed in an image acquired with an ultrasound system and displayed on a display device and / or saved for later viewing, as will be described in more detail below. If a request to include a virtual marker is received, method 300 proceeds to 312 to position the virtual marker at an indicated position in the 3D data set. In some examples, the position may be indicated by a user. For example, as an example, a user may indicate the position via movement of a cursor and subsequent mouse, keyboard, or other input that indicates that the position of the cursor is the position of the virtual marker. When a user is viewing a 3D dataset or a portion of a 3D dataset (e.g., as a volume rendering image), a virtual marker may be positioned within the 3D dataset, and the user may indicate a desired location within the 3D dataset where the virtual marker is to be placed via cursor movement / input input or input touch input. In other examples, the virtual marker may be positioned with respect to a 2D slice of the displayed 3D dataset according to a similar mechanism (e.g., via a mouse-controlled cursor or via touch input). In other examples, the user may input an input indicating that the virtual marker should be positioned at a target anatomical structure, and the ultrasound system may automatically determine where to position the virtual marker. When an aspect of the 3D dataset including the virtual marker is displayed (such as a 2D slice or a volume rendering image, as explained below), the virtual marker is displayed at the indicated location. The virtual marker may be associated with one or more voxels of the 3D dataset, and / or the virtual marker may be associated with an anatomical feature of the 3D volume, and when one or more voxels and / or anatomical features are displayed, the virtual marker may be displayed as an annotation on the displayed image. The virtual marker may take on a suitable visual appearance, such as a solid circle, rectangle or other shape, letters or words, or other desired appearance.

[0030] At 314, a volume rendered image is generated from the 3D data set. Figure 2The volume rendered image may be generated in response to a user request, or the volume rendered image may be automatically generated, for example, in response to a scan protocol or workflow that specifies that a volume rendered image is to be generated. The volume rendered image may be a two-dimensional image of one or more desired planes of a 3D volume (e.g., a 2D representation of a rendered volume defined by data of a 3D data set), or the volume rendered image may be a two-dimensional image of a surface of a 3D volume or other suitable volume rendered image.

[0031] As previously explained, the virtual marker may be positioned on a surface of a 3D dataset or within the 3D dataset. When a volume rendered image is generated from a 3D dataset, it may be difficult for a user of the ultrasound system (e.g., a clinician) to judge the depth of the virtual marker. For example, it may be challenging for a user to determine whether the virtual marker is intended to be positioned within a cavity formed by an imaging structure, or whether the virtual marker is intended to be positioned on a surface defining the cavity. Therefore, as will be explained in more detail below, the virtual marker may be associated with a first light source linked to the virtual marker such that the first light source is positioned at the same location as the virtual marker. The volume rendered image is illuminated / colored using the first light source to add depth cues to the image and allow the user to more easily determine the location of the virtual marker.

[0032] Thus, generating the volume rendered image includes coloring the volume rendered image from a first light source positioned at the virtual marker, as shown at 316. Additionally, generating the volume rendered image includes coloring the volume rendered image from a second light source positioned away from the 3D dataset, as shown at 318. The second light source may be one or more external light sources that are not positioned within the 3D dataset. The first light source is linked to the virtual marker and is therefore positioned within the 3D dataset (in image space). For example, the first light source may be positioned at one or more voxels of the 3D dataset.

[0033] As part of the generation of the volume rendered image, a shading for the volume rendered image is determined. Figure 2 As described, the coloring of the volume rendered image may include calculating how light from two or more different light sources (e.g., a first light source and a second light source) will interact with the structure represented in the volume rendered image. The algorithm controlling the coloring may calculate how the light will be reflected, refracted, and diffused based on the intensity, opacity, and gradient in the 3D dataset. The intensity, opacity, and gradient in the 3D dataset may correspond to tissues, organs, and structures in the volume of interest from which the 3D dataset was acquired. Light from multiple light sources is used to calculate the amount of light along each ray used to generate the volume rendered image. Therefore, the position, orientation, and other parameters associated with the multiple light sources will directly affect the appearance of the volume rendered image. In addition, the light source can be used to calculate the coloring of the surface represented in the volume rendered image.

[0034] As explained above, shading from a first light source and one or more second light sources may be performed, wherein light from the first light source and the one or more second light sources is used to calculate shading and / or to calculate the amount of light along each ray used to generate a volume rendered image. In some examples, the shading produced by the first light source may be determined by estimating the normal of each surface of the volume rendered image and applying a shading model having a diffuse component and a specular component. The intensity of the simulated light projected by the first light source in the 3D dataset may be a function of the distance from the first light source / virtual marker within the 3D dataset (e.g., inversely proportional to the squared distance from the first light source / virtual marker within the 3D dataset). Shading from the first light source may include superimposing one or more shadows, each cast by one or more corresponding structures in the 3D volume, onto one or more surfaces of the 3D volume. In some examples, shading from the second light source may be determined in a similar manner (e.g., using the same shading model) compared to the determination of shading from the first light source (e.g., the shading produced by the second light source may be determined by estimating the normal of each surface of the volume rendered image and applying the same shading model (the model having a diffuse component and a specular component) used to calculate shading of the first light source). However, because the first light source is located within the 3D dataset (e.g., the first light source is positioned within the 3D dataset and the second light source is positioned outside or external to the 3D dataset), the light emitted by the first light source is visually distinguishable from the light emitted by the second light source. As an example, the light emitted by the first light source may have a different color relative to the light emitted by the second light source. As another example, because the first light source is located within the 3D dataset, the light emitted by the first light source may have increased apparent intensity and / or brightness (e.g., under the condition that the first light source and the second light source have the same light intensity, the light emitted by the first light source may appear brighter and / or stronger than the light emitted by the second light source because the first light source is positioned within the 3D dataset and the second light source is positioned outside the 3D dataset). The first light source being located within the 3D dataset may result in the first light source being positioned closer to a structure described by the 3D dataset (e.g., represented by a voxel of the 3D dataset), and because the first light source is positioned closer to the structure, the structure may be illuminated by the first light source in a greater amount relative to the amount of illumination of the structure by the second light source.

[0035] In some examples, contributions from a first light source and a second light source (e.g., light emitted by the first light source and the second light source) may be summed to determine an amount of illumination of a portion of a volume rendered image. For example, a surface of a volume rendered image that receives light from each of the first light source and the second light source may be rendered with increased brightness relative to a condition where the same surface receives light from only the second light source. In some examples, the second light source may emit white light, and the first light source may emit light of a different color (e.g., red light). A surface that receives light from each of the first light source and the second light source may be illuminated according to a combination of white light from the second light source and colored light from the first light source (e.g., a surface illuminated by both the first light source and the second light source may appear colored the color of the first light source, where the amount of saturation of the color is a function of the distance of the first light source).

[0036] In some examples, illumination caused by the first light source and / or the second light source can be determined using a Phong illumination model modulated by occlusion to account for occlusion. In this example, determining the illumination of the voxel during ray casting can include summing diffuse and specular contributions modulated by occlusion of the first light source and / or the second light source. In some examples, the occlusion value can be determined by tracing shadow rays from each light source to each voxel to determine the degree of occlusion.

[0037] As mentioned above about Figure 2 As explained, the volume rendered image may be colored from a second light source, and in some examples from one or more additional light sources located away from the 3D dataset in imaging space, to provide illumination and / or shading on the volume rendered image that helps distinguish and identify structures in the volume rendered image, provide depth cues, and simulate how the imaged structures will appear when viewed using visible light. The one or more second light sources may be based on the above description of Figure 2 The one or more second light sources may be positioned in a manner similar to the examples provided (e.g., a main light, a fill light, and / or a backlight) or other suitable configurations. The one or more second light sources may be fixed in position, or the position, angle, light characteristics, etc. may be adjusted by a user or by the ultrasound system. The one or more second light sources may be spaced apart from the 3D dataset by one or more suitable distances, which may be in the range of millimeters, centimeters, or meters, or may be spaced apart from the 3D dataset by a suitable number of voxels. The 3D dataset may include a plurality of voxels and be defined by a boundary, and the one or more second light sources may be positioned outside the boundary of the 3D dataset. In this manner, the one or more second light sources may provide surface shading for the volume rendered image.

[0038] At 320, the colored volume rendered image is displayed on a display device associated with the ultrasound system, such as the display device 118. The colored volume rendered image may additionally or alternatively be stored in a memory, such as the memory 120, and / or as part of an electronic medical record of the imaged subject for later review. The displayed volume rendered image includes a visual depiction of a virtual marker (e.g., as explained above) at the indicated location, and structures surrounding the virtual marker in the volume rendered image are illuminated with simulated light projected from the first light source. In addition, surfaces of the structures depicted in the volume rendered image are illuminated with simulated light projected from one or more second light sources.

[0039] At 322, the intensity of the simulated light projected from the first light source may be updated in response to a user request. For example, the user may enter a suitable input (e.g., to a menu or control button displayed on a display device) requesting an adjustment (e.g., increase or decrease) in the intensity of the light projected from the first light source. When the intensity of the light is adjusted, the coloring of the illuminated structure around the virtual marker is also adjusted, and thus an adjusted volume rendered image with adjusted coloring may be displayed. In some examples, the user may request that light not be projected from the first light source, and thus the volume rendered image may include only coloring from one or more second light sources in such examples. At 324, if requested, the position of the virtual marker is updated, and as the position of the virtual marker changes, the position of the first light source is updated accordingly, and thus the coloring of the volume rendered image is updated. For example, the user may enter an input indicating that the virtual marker should be repositioned. When the position of the virtual marker changes, the position of the first light source also changes because the first light source is linked to the virtual marker. When the position of the first light source changes, the illumination / coloring of the structure in the volume rendered image also changes, and thus the coloring may be adjusted in the volume rendered image, or the updated volume rendered image may be displayed with the updated coloring. Then, method 300 returns.

[0040] Returning to 304, if a request to position a virtual marker on or within the 3D dataset has not been received, the method 300 proceeds to 306 to generate a volume rendered image without the virtual marker from the 3D dataset. Figure 2 Generating the volume rendered image, for example, generates an image from a specified view plane using ray casting. Generating the volume rendered image without the virtual marker may include rendering the volume rendered image from one or more second light sources positioned away from the 3D volume, and not rendering the volume rendered image using any light source associated with any virtual marker.

[0041] At 310, the colored volume rendered image is displayed on a display device associated with the ultrasound system, such as the display device 118. The colored volume rendered image may additionally or alternatively be stored in a memory, such as the memory 120, and / or as part of an electronic medical record of the imaged subject for later review. The colored volume rendered image generated and displayed when the virtual marker is not present does not include the virtual marker or the light source associated with the virtual marker. The method 300 then returns.

[0042] Figure 4 is a schematic diagram of a 3D dataset 402 and an orientation 400 of multiple light sources that may be used to apply colorization to a volume rendered image of the 3D dataset 402, according to one embodiment. Figure 4 402. The orientation 400 includes a first light source 404, a second light source 406, and an optional third light source 408. The first light source 404, the second light source 406, and the optional third light source 408 can be used to calculate the shading for the volume rendering image. However, as previously described, the light sources can also be used during ray casting while generating the volume rendering. The orientation 400 also includes a viewing direction 410 representing the location from which the 3D dataset 402 is viewed.

[0043] Figure 4 A side view is shown, and it should be understood that each light source may be positioned at a different height relative to the 3D dataset 402 and the viewing direction 410 .

[0044] The first light source 404 is a virtual marker light source that is positioned at a location that corresponds to (e.g., is the same as) a location of a virtual marker placed by a user of the ultrasound system. Figure 4 In the example shown in , the first light source 404 is a point light that projects light in all directions, but other configurations are possible, such as the first light source 404 is a spotlight. In examples where the first light source 404 is not a point light, the directionality of the light projected from the first light source can be adjusted by a user. The first light source 404 is positioned at a location that overlaps with the 3D dataset. For example, the first light source 404 can be positioned at one or more voxels of the 3D dataset.

[0045] The second light source 406 can be positioned at a location spaced apart from the 3D dataset 402. For example, as shown, the second light source 406 can be positioned to illuminate the front surface of the 3D dataset 402, and thus can be placed away from the front surface of the 3D dataset (with respect to the viewing direction). The second light source 406 can be a suitable light source, such as a key light (e.g., it can be the strongest light source used to illuminate the volume rendering). The second light source 406 can illuminate the volume rendering image from the left or right side of the reference of the viewing direction 410. When included, the third light source 408 can be a fill light positioned on the opposite side of the volume rendering as the key light with respect to the viewing direction 410 to reduce the harshness of shadows from the key light.

[0046] Figure 4 The light sources shown in are exemplary, and other configurations are possible. For example, there may be a fourth light source, where the fourth light source is positioned behind the 3D dataset 402 to act as a backlight. The backlight may be used to help highlight the volume imaged in the 3D dataset 402 and separate it from the background. In addition, the second light source 406 and the third light source 408 (when included) may be positioned in other suitable locations and / or have other suitable intensities, light shapes, etc.

[0047] Figure 4 A coordinate system 412 is included. As shown, the 3D data set is arranged along an x-axis and a z-axis (and a y-axis, but in Figure 4 The data set is not shown extending along the y-axis). Figure 4 Also shown in FIG. 4 is an exemplary view plane 414. The view plane 414 can extend along the x-axis and the y-axis and can be the view plane in which the volume rendered image is rendered. For example, when generating a volume rendered image with respect to the view plane 414, all data in the view 3D data set in front of the plane 414 (with respect to the z-axis) can be discarded, and the volume rendered image can be generated such that the view plane 414 acts as the front surface of the volume rendered image.

[0048] Figure 5 shows the use of an imaging system such as Figure 1An exemplary volume rendered image 500 generated from a 3D dataset of medical imaging data acquired by an ultrasound imaging system 100 (using an ultrasound imaging system 100). In at least some examples, the volume rendered image 500 can be generated from the 3D dataset 402 along a viewing plane 414. The volume rendered image 500 depicts the structure of a heart 502, for example, the imaging volume is the heart. Segments of an internal tissue structure 512 at the viewing plane are shown, as well as surfaces of the heart behind the viewing plane that are not blocked by tissue in the viewing plane, such as cavity 514 and cavity 516. The structures shown in the volume rendered image 500 form a rendered volume for the volume rendered image 500. For example, the internal tissue structure 512 is shown at a different depth relative to the viewing plane compared to cavity 514 and cavity 516. The difference in depth of the various structures relative to the viewing plane provides a three-dimensional appearance or rendered volume to the 2D volume rendered image 500. The coordinate system 510 is in Figure 5 , where the viewing plane extends along the x-axis and the y-axis. The surface behind the viewing plane is located behind the viewing plane along the z-axis.

[0049] One or more external light sources (such as Figure 4 Thus, internal tissue structures 512 in front of the volume rendered image (e.g., along the viewing plane) have a relatively large amount of illumination, while structures further away (e.g., Figure 5 The back surface of the cavity shown has little or no illumination, as understood by cavity 514. In addition, shadows are cast by one or more external light sources and structures between the surface positioned behind the viewing plane along the z-axis. For example, shadows are cast into cavity 516.

[0050] Image 500 includes three virtual markers, namely a first virtual marker 504, a second virtual marker 506, and a third virtual marker 508. Figure 3 As explained, each virtual marker can be positioned according to user input to mark the target anatomical structure. Each virtual marker is depicted in a different color, for example, the first virtual marker 504 is shown in yellow, the second virtual marker 506 is shown in red, and the third virtual marker 508 is shown in green, so as to enhance the visualization and distinction of the virtual markers.

[0051] If through Figure 5As appreciated, it may be difficult to judge the location of a virtual marker along the z-axis (e.g., along the depth of the 3D volume) in the volume rendered image 500. For example, it may be difficult to determine whether the first virtual marker 504 is intended to be positioned along the back surface of the cavity behind the first virtual marker 504 (e.g., at a first distance from the xy viewing plane along the positive z-direction), or whether the first virtual marker 504 is intended to be positioned closer to the viewing plane (e.g., at a second, shorter distance from the xy viewing plane along the positive z-direction).

[0052] Therefore, according to the embodiments disclosed herein, each virtual marker may be associated / linked to a corresponding light source, and each light source may be used to illuminate structures surrounding the corresponding virtual marker to provide depth cues to help the user judge the depth of each virtual marker (e.g., to illuminate structures of the rendering volume that forms the volume rendering image 500). Figure 6 A second volume rendered image 600 showing a heart 502 is shown, similar to the volume rendered image 500. In the second volume rendered image 600, each virtual marker includes a light source that projects simulated light to illuminate structures surrounding each virtual marker. For example, the first virtual marker 504 can be associated with a first virtual marker light source, the second virtual marker 506 can be associated with a second virtual marker light source, and the third virtual marker 508 can be associated with a third virtual marker light source. Each virtual marker light source can project simulated light of a different color, such that the first virtual marker light source projects yellow light, the second virtual marker light source projects red light, and the third virtual marker light source projects green light.

[0053] By including virtual marker light sources, the depth of each virtual marker can be more easily determined by a user of the ultrasound system. Figure 6 As understood, the first virtual marker 504 is positioned relatively closer to the viewing plane than the back surface of the cavity on which the first virtual marker 504 is placed. Likewise, the second virtual marker 506 is positioned closer to the viewing plane than the surface behind the second virtual marker 506.

[0054] When multiple virtual markers are positioned in the 3D data set, the light source associated with each virtual marker may project light to one or more of the same voxels. For example, a first virtual marker light source associated with the first virtual marker 504 may project light to the region 518 of the imaging volume, and a second virtual marker light source associated with the second virtual marker 506 may also project light to the region 518. The contributions from the two light sources may be summed and used to illuminate / color the voxels of the region 518. In other examples, a cone structure or other simulated structure may be placed around each virtual marker light source to limit the projection of each light source to a threshold range around the corresponding associated virtual marker, which may reduce the overlap of illumination from the virtual marker light sources. In addition, in an example where the volume rendered image includes a virtual marker that is blocked (in the view of the volume rendered image) by tissue or other anatomical structures, the virtual marker light source may appear to glow to signal to the viewer that the virtual marker is positioned within the imaged tissue, although not visible. In other examples, when the volume rendered image includes a blocked virtual marker, the light projected from the virtual marker light source may not be displayed.

[0055] The technical effect of associating a light source with a virtual marker positioned within a volumetric medical imaging dataset and coloring a volume rendered image (rendered from the volumetric medical imaging dataset) according to simulated light projected from the light source is to increase a viewer's depth perception of the virtual marker.

[0056] As used herein, the elements or steps listed in the singular and beginning with the word "one" or "a kind of" should be understood as not excluding a plurality of the elements or steps, unless such exclusion is explicitly stated. In addition, the reference to "an embodiment" of the present invention is not intended to be interpreted as excluding the existence of additional embodiments that also include the cited features. In addition, unless explicitly stated to the contrary, the embodiment of "including", "comprising" or "having" an element or multiple elements with a specific characteristic may include additional such elements without the characteristic. The terms "including" and "in..." are used as the concise language equivalents of the corresponding terms "including" and "wherein". In addition, the terms "first", "second" and "third" etc. are only used as marks, and are not intended to impose numerical requirements or specific positional order on their objects.

[0057] This written description uses examples to disclose the invention, including the best mode, and also to enable a person skilled in the relevant art to practice the invention, including making and using any devices or systems and performing any included methods. The patentable scope of the invention is defined by the claims, and may include other examples that occur to a person skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insignificant differences from the literal language of the claims.

Claims

1. A method comprising: Display volume rendered images drawn from 3D medical imaging datasets; positioning a first virtual marker within a rendered volume of the volume rendered image, the rendered volume being defined by the 3D medical imaging dataset; and The rendering volume is illuminated by projecting simulated light from the first virtual marker, wherein shading produced by the simulated light projected by the first virtual marker is determined by estimating normals for each surface of the volumetric rendering image and applying a shading model having a diffuse component and a specular component, and wherein the shading produced by the simulated light projected by the first virtual marker includes superimposing shadows cast by a first structure within the rendering volume onto surfaces of a second structure within the rendering volume. 2 . The method of claim 1 , wherein an intensity of the simulated light projected by the first virtual marker decreases as a function of a distance from the first virtual marker within the rendering volume. 3 . The method of claim 1 , further comprising positioning a second virtual marker within the rendering volume, and wherein illuminating the rendering volume comprises projecting simulated light from the second virtual marker.

4. The method of claim 3, wherein a color of the simulated light projected by the first virtual marker is different from a color of the simulated light projected by the second virtual marker. 5 . The method of claim 1 , wherein the first virtual marker projects the simulated light in a spherical manner so as to illuminate the rendering volume in all directions from the first virtual marker.

6. The method of claim 1, wherein positioning the first virtual marker comprises positioning the first virtual marker in response to user input.

7. The method according to claim 1 further comprises acquiring the 3D medical imaging dataset via an ultrasound probe, the 3D medical imaging dataset comprising a plurality of voxels representing a physical non-virtual volume scanned by the ultrasound probe and associated intensity and / or opacity values. 8 . The method of claim 7 , wherein illuminating the rendering volume comprises determining a contribution of the simulated light to each voxel of the plurality of voxels, and applying the contribution to each voxel.

9. A method for volume rendering 3D medical imaging data, comprising: generating a volume rendered image from a 3D dataset acquired using a medical imaging device, the volume rendered image comprising a virtual marker positioned at a first location in the 3D dataset; shading the volume rendered image from a first light source positioned at the first location and from a second light source positioned at a second location spaced apart from the 3D dataset, wherein the shading produced by the first light source and the second light source is determined by estimating a normal to each surface of the volume rendered image and applying a shading model having a diffuse component and a specular component, and wherein the shading produced by the first light source and the second light source includes superimposing shadows cast by first structures within the volume rendered image onto surfaces of second structures within the volume rendered image; as well as The rendered volume rendered image is displayed on a display device.

10. The method of claim 9, further comprising receiving user input requesting display of the virtual marker at the first location, and in response, positioning the virtual marker at the first location in the 3D dataset.

11. The method of claim 9, wherein coloring the volume rendered image comprises determining, for each voxel of the 3D data set, a first contribution of the first light source to the voxel and a second contribution of the second light source to the voxel, and coloring the voxel according to the first contribution and the second contribution. 12 . The method of claim 9 , wherein coloring the volume rendered image from the first light source comprises coloring the volume rendered image from the first light source according to a surface shading algorithm.

13. The method of claim 9, wherein generating the volume rendered image comprises generating the volume rendered image from a plurality of voxels of the 3D dataset using ray casting, and wherein shading the volume rendered image from the first light source comprises including a contribution of the first light source to each of the plurality of voxels in the ray casting.

14. The method of claim 9, wherein the medical imaging device is an ultrasound probe.

15. A system comprising: Ultrasound probe; monitor; and A processor configured with instructions stored in a non-transitory memory, the instructions, when executed, causing the processor to: generating a volume rendered image from a 3D dataset acquired using the ultrasound probe, the volume rendered image comprising a virtual marker positioned at a first location in the 3D dataset; shading the volume rendered image from a first light source positioned at the first location and from a second light source positioned at a second location spaced apart from the 3D dataset, wherein the shading produced by the first light source and the second light source is determined by estimating a normal to each surface of the volume rendered image and applying a shading model having a diffuse component and a specular component, and wherein the shading produced by the first light source and the second light source includes superimposing shadows cast by first structures within the volume rendered image onto surfaces of second structures within the volume rendered image; as well as The rendered volume rendered image is displayed on the display.

16. The system of claim 15, wherein the first light source has a first light intensity and the second light source has a second, different light intensity.

17. The system of claim 15, further comprising instructions stored in the non-transitory memory that, when executed, cause the processor to: The position of the first light source is adjusted from the first position to a third position in response to user input requesting adjustment of the virtual marker from the first position to a third position.

18. The system of claim 15, wherein the volume rendered image is a first volume rendered image having a first view plane; and Also included are instructions stored in the non-transitory memory that, when executed, cause the processor to: generating a second volume rendered image from the 3D dataset acquired using the ultrasound probe, the second volume rendered image including the virtual marker maintained at the first position of the 3D dataset, the second volume rendered image having a second different view plane; rendering the second volume rendered image from the first light source positioned at the first location; as well as The rendered second volume rendered image is displayed on the display.

19. The system of claim 15, further comprising instructions stored in the non-transitory memory that, when executed, cause the processor to: adjusting the intensity or color of the first light source in response to user input; and The rendered volume rendered image on the display is updated based on the adjusted intensity or color of the first light source.

Images