System and method for automated light placement for medical visualization

Abstract

An automatic light arrangement for medical visualization comprises: providing a medical 3D image (D), providing spatial information about a region of interest (R) in the 3D image (D) and spatial information about a virtual camera (C), determining a plurality of possible arrangements for light sources (L) by using depth information (DI) based on the 3D image together with the spatial information about the region of interest (R) in the image and the spatial information about the virtual camera (C), wherein valid arrangements are those arrangements for which shadows (S) on the region of interest (R) are below a predefined threshold, and / or wherein the determination or arrangement is based on a plurality of predefined perceptual metrics specifically applied to the region of interest, prioritizing the determined arrangements, and selecting the arrangement with the best priority ranking.

Description

[0001] Related applications

[0002] This application claims priority to 21161885.5, filed on March 10, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Embodiments relate to systems and methods for automatic light arrangement of medical images, and more particularly for automatic light placement for region-of-interest based medical visualization. Background Art

[0004] Light design is an important research area that focuses on the importance of complex lighting setups in visualization. Proper lighting is essential in medical visualization, where accurate rendering of patient anatomy is paramount. However, finding the optimal setup is not trivial. Despite significant advances in the field of light design, finding the optimal parameters for lighting a scene remains a manual and time-consuming task.

[0005] Cinematic rendering (“CR”) is a 3D visualization technique that can produce photorealistic images from conventional computed tomography (CT) or magnetic resonance (MR) volumes (see, for example, Comaniciu D et al., “Shaping the future through innovations: From medical imaging to precision medicine,” Med Image Anal. 2016;33:19-26). CR uses a global illumination model based on Monte Carlo path tracing to create realistic shading that enhances the depth and shape perception of volumetric datasets.

[0006] Typically, CR methods use high dynamic range (HDR) light maps to illuminate scenes in a natural way. These maps can be combined with synthetic light to highlight specific structures or illuminate internal organs. The orientation of these light maps and the arrangement of the synthetic light can provide valuable perceptual information about the depth, shape, and surface properties of the data.

[0007] The traditional approach to lighting design for volume rendering is based on manually setting lighting parameters and modifying them until the desired image is produced. Since this process is time-consuming and often requires some technical and aesthetic knowledge of lighting, several methods have been proposed to facilitate this task.

[0008] Poulin et al. ("Sketching shadows and highlights to position lights", Proceedings Computer Graphics International, Hasselt and Diepenbeek, Belgium, 1997, pp. 56-63) let the user specify the desired properties of shadows and highlights on the final image. An inverse shading method is then used to determine the light source positions that satisfy these constraints.

[0009] Zhou, J et al. (“Topology-aware illumination design for volume rendering”. BMC Bioinformatics 17, 309; 2016) discloses a method for volume rendering illumination design based on data topology. The purpose of this method is not to determine the optimal light position and direction, but to adjust the illumination coefficient based on the topological properties, topological distance, and topological saliency of the data.

[0010] Zhang and Ma ("Lighting design for globally illuminated volume rendering," IEEE Transactions on Visualization and Computer Graphics, Vol. 19, No. 12, pp. 2946-2955, 2013) applied a three-point lighting setup commonly used in photography to volume rendering. A key light was set based on an analysis of the statistical distribution of surface normals across the entire volume dataset. A fill light was used to illuminate shadowed areas to make details visible, while a backlight was used to improve depth perception. This approach produced a well-balanced image, but did not take into account which areas were in shadow and which were in light. This could result in images with strong shadows that obscured important details in the data. Summary of the Invention

[0011] The object is to improve known systems, devices and methods to facilitate automatic light arrangement for medical visualization, in particular when controlling a medical imaging system to set an optimal light arrangement.

[0012] This object is achieved by a method, a system, a control device and a medical imaging system according to the present disclosure.

[0013] According to one embodiment, a method for light arrangement for medical visualization, in particular for controlling a medical imaging system to set an optimal light arrangement, comprises the following actions:

[0014] - providing a medical 3D image, which can be, for example, a complete 3D image with voxels or also a sequence of 2D slice images describing a 3D volume, wherein the 3D image is a volumetric image including depth information or the possibility of calculating depth information with a given camera view,

[0015] - providing spatial information about the region of interest in the image and the virtual camera (e.g., manually defined or automatically detected),

[0016] - optionally: providing information about a plurality of predefined artificial light sources (ie information on what type of light source and in particular how many of this / these type of light source should be used),

[0017] - determining a plurality of possible arrangements for the light sources by using depth information based on the 3D image together with spatial information about the region of interest in the image and spatial information about a virtual camera (or virtual observer), wherein valid arrangements are those in which shadows on the region of interest are below a predefined threshold,

[0018] - prioritizing the determined arrangements, in particular by using criteria derived from photography,

[0019] -Select the arrangement with the best priority sorting.

[0020] These actions can be followed by the following actions:

[0021] - Rendering an image with shadows using the selected light source arrangement. However, once the lighting arrangement is set, the lighting calculations can be calculated via well-known equations.

[0022] First, the 3D image data (3D volume), the region of interest, the position of the virtual camera, and the type of light source must be defined or at least known for further determination. This can be done automatically (e.g., by using preferences) or by manual selection by the user. Typically, these actions are prior art, as selecting images and ROIs and (manually) selecting and arranging light sources are common practices in the art. However, the embodiments add specific features described below. With respect to the 3D image, it is preferably defined as a sequence of 2D image slices from a medical scan, which is commonly referred to as a 3D volume. Alternatively, the 3D image can be constructed from a mathematical / functional description, which can be the result of a computational simulation (e.g., molecular dynamics).

[0023] Provision of information about artificial light sources, the positions of these light sources being not essential information (since the arrangement is calculated in the course of the method). Preferred information is about the types of light sources forming the lighting arrangement. The action may be a simple predefinition that a single type of light source should be used (e.g. a main light with a predefined intensity). The action may also be the provision of a lighting map with defined properties read from a memory device. However, the action may also include user input about the light sources to be used (e.g. main light and auxiliary light). Information may also be hardwired into the method, e.g. "if there is no user input and no memory device is available, use the main light for the following action". It should be noted that the number and type of light sources may be enhanced later, e.g. if additional light sources are added.

[0024] A virtual camera means knowing the camera angle, position and orientation relative to the 3D volume and ROI data.

[0025] A "region of interest" (ROI) is also not a physical object, but a data sample (3D area) within a data set (volume) of a 3D image, and in particular not the entire 3D image. It can be defined by its position and the voxel information at that position. In particular, an ROI can be viewed as a polygonal selection within a 3D image. For example, it can define the boundaries of an object under consideration (such as an organ). In the field of medical technology, an ROI is a specific region (of image data) that has been identified to provide important medical information. The concept of ROI is often used in the literature of image processing and radiology. An ROI can represent a suspected tumor, an aneurysm, etc.

[0026] Compared to techniques that generate images with well-balanced highlight, mid-tone and shadow areas, which may result in the disadvantage that these images can be well balanced while obscuring important structures, this embodiment has the advantage that it uses the ROI rather than just the entire image to calculate the lighting.

[0027] The term "providing" here means that certain information is available to the method. How the data is provided is not important. Thus, "providing" can particularly mean "reading an input", "reading a predefined parameter", "calculating" or "measuring".

[0028] Spatial information about the region of interest is information about the position, orientation and in particular also the shape of the region of interest within the 3D image.

[0029] The spatial information about the virtual camera is the information about the position and viewing direction of the virtual camera (the viewing direction should be pointing to the region of interest). Regarding the calculation, preferably, only the ROIs within the camera's frustum are considered when calculating the lighting. In this case, there may be other ROIs, but they are ignored until the camera is turned to include them in the view.

[0030] Each spatial information can be provided as input or as a predefined parameter.

[0031] Preferably, spatial information about the ROI is identified based on user selection of a surgical instrument, automatic detection, automatic segmentation, data analysis, user gaze tracking, or 3D tracking. For example, the spatial information can be defined as regions of high curvature. Regions of high curvature are generally structures that carry important information compared to uniform regions (or regions with low curvature), and therefore they can be used as ROIs.

[0032] The term "light source" includes any synthetic lighting that can be applied to artificial illumination for reconstructing medical images. Such a light source is preferably a key light or a light map (also called "ambient map"), but can also be in particular a spot light or an auxiliary light. The light source can comprise only one single light source, or two or more light sources (e.g. "three-point lighting"), in particular a key light or a light map in combination with at least one spot light and / or an auxiliary light. The key light is preferably modeled as a directional light that is infinitely far away from the scene. It is particularly defined as a direction vector that does not have an exact position and tends to illuminate the entire scene from infinity. The intensity of the preferred key light is set to 1 (relative to the units of the other lights). A light map can be used instead of or in addition to the key light. Preferably, the light map is oriented so that the highest intensity (usually the sky) illuminates in a direction that is also the direction of the key light.

[0033] In particular, one or more light sources are selected from the group consisting of a key light, a light map, an auxiliary light (to illuminate shadows), a backlight, and a spot light. It should be noted that since the method involves the reconstruction of medical images, the light source may not be a real light source, but a synthetic light source commonly used for 3D rendering. Such light sources themselves are well known in the art.

[0034] In order to apply the method according to one embodiment, multiple light sources must be "provided", that is, it must be clear what type of light source to use initially and how many of each type. This can be achieved by preferences made by predefinition or user input. However, the light sources can also be provided by automatically selecting or calculating them. For example, it can be predefined that a main light with given characteristics (direction and shape of the light cone, intensity) is always used. When the method detects the presence of shadows that are darker than a predefined threshold and cannot be softened by modifying the position of the main light, auxiliary lights or point lights can be automatically added.

[0035] The term "arrange" (or "place") means determining the orientation and / or position of a plurality of light sources. This means, at least in the case of non-omnidirectional light sources, providing the position of the light sources in the environment of the region of interest (which may also include an infinite distance) and the orientation of the light sources in this environment, i.e., the vectors of the main direction(s) of the light beam(s).

[0036] For example, a light map (a pre-rendered texture that stores information about the simulated lighting environment) can be positioned so that the alignment of the brightest areas of the light map corresponds to the calculated direction of light. This arrangement is then the definition of the orientation of the light map around the volume, i.e. the orientation of the light rays.

[0037] Regarding the main light, we can assume it is a directional or frontal light. We also assume that the directional light is infinitely far away, so again, only the orientation of the parallel light rays needs to be calculated. For frontal lights, position is important because the light intensity is inversely proportional to the square of the distance.

[0038] Point lights simulate light radiating from a single, infinitesimal point in space with a specific orientation. When there's no configuration where all ROIs can be illuminated by the main light, a specific ROI is specifically chosen to be illuminated. The point light is then preferably placed near a given ROI and aimed in its direction.

[0039] For a uniform omnidirectional light source, there is no need to define an orientation, as the light is emitted in all directions. However, its position must be defined.

[0040] Regarding the auxiliary lights (and spot lights), preferably different intensities and / or lighting angles are tested for different orientations in determining a plurality of possible arrangements, for example considering different intensities and / or lighting angles in nested rings.

[0041] Auxiliary lights are often used to illuminate parts of a scene that would otherwise remain dark. These lights are preferably modeled as point lights with a position and direction and a specific opening angle, so that only objects within a certain radius can be illuminated. For these lights, the intensity is inversely proportional to the square of the distance to the illuminated object (the inverse square law) as the surface area covered increases. Knowing the size of the area that needs to be illuminated (in the ROI) and the distance to the auxiliary light source, the appropriate intensity can be calculated. If the auxiliary light (as a point light) is placed, for example, at a distance d from the ROI, it will illuminate an area of ​​size 4πd with an intensity I0 / A. 2 , where I0 is the initial intensity of the auxiliary light, and it is set to a value I0<1 (relative to the intensity of the main light).

[0042] Regarding the important act of determining multiple possible arrangements of light sources, two or more possible arrangements are determined. The arrangements should be different. This is preferably done using backlighting techniques known in the art.

[0043] To perform this determination, depth information based on the 3D image (provided or calculated in the 3D image) and spatial information about the region of interest in the image (clarifying the position of the ROI in the 3D image) are used to determine the position, orientation, and shape of surrounding structures of the region of interest and shadows. The spatial information about the virtual camera is used to determine how the virtual camera sees the region of interest and, for example, which shadows are relevant for the virtual camera, since shadows hidden by structures in front cannot be seen by the virtual camera and are therefore less relevant.

[0044] The quality of the visual information of the image around each ROI can be determined based on some perceptual metric specifically applied to these regions, including how dark these regions are compared to the rest of the image. Some of these metrics can include the average illuminance around the ROI. Other metrics are described in more detail in Shacked et al., "Automatic Lighting Design Using a Perceptual Quality Metric."

[0045] The "quality" of the shadows in the ROI can be quantified by radiance or brightness and / or the area being shadowed and / or whether a particular point in the ROI is covered by the shadow. To determine a valid placement, the quantified "quality" of the shadows in the ROI(s) can be compared to a predefined threshold (which can be a constant value or a map of values, each value for a different location in the ROI). In this case, if the shadow is below the threshold, e.g., is brighter or smaller than the defined threshold and does not cover a particular point in the ROI, the placement is a valid placement. Furthermore, the "quality" of the shadows in the ROI(s) can be compared to the brightness of other areas of the image, or to the darkest / brightest / median values ​​of the image.

[0046] Preferably, there are certain boundary conditions for the possible positions of light sources, and the entire space around the region of interest is not used. For example, a preferred boundary condition for the position of the key light (or the sky for a lightmap) is the frustum (i.e., a pyramid) defined by the ROI, in particular plus a predetermined space around the ROI (e.g., a bounding box), which is used as the vertices of the pyramid, and an orientation or direction towards the virtual camera (or virtual observer). Point lights and fill lights can be placed anywhere, even within this volume.

[0047] Regarding prioritization, there are many possibilities depending on the medical scene shown in the image and the examination for which the rendered image is intended. Prioritization can be employed in particular in photography. For example, it should be avoided that the main light illuminates the ROI from the upper right or that light in the direction of the camera view should be avoided to avoid the requirement of a flat image. Prioritization of auxiliary lights can be achieved by selecting auxiliary lights that are placed closest to the camera to object axis (to avoid creating visible shadows). However, the prioritization can also depend on the size or coloring of the shadows created with such an arrangement.

[0048] In practice, the prioritization can be performed by using a frustum of the ROI in the camera direction. Since the intersection of the frustums is usually not a single point, the vertex Vi of the resulting intersection object can be picked in such a way that the direction of the light is not parallel to the view direction to avoid a flat image. For each vertex v i , we can calculate v i The average vector associated with each ROI is calculated, and the dot product of these vectors with the view direction can also be calculated. The minimum dot product can be selected as the highest priority. Directional lights can be used for key lights, so the distance to the ROI is not important.

[0049] The selected arrangement may be output to a device for rendering a 3D image with shading or used for directly rendering a 3D image, wherein the selected arrangement may also be considered as an output when used by a rendering algorithm.

[0050] According to one embodiment, a system for light arrangement for medical visualization, in particular a system for controlling a medical imaging system to set an optimal light arrangement, includes the following components:

[0051] a data interface designed to receive a medical 3D image, spatial information about a region of interest in the image and a virtual camera, and preferably information about a plurality of predefined artificial light sources,

[0052] a determination unit (processor) designed to determine a plurality of possible arrangements for the light sources by using depth information based on the 3D image together with spatial information about the region of interest in the image and spatial information about the virtual camera, wherein valid arrangements are those in which shadows on the region of interest are below a predefined threshold,

[0053] a prioritization unit (processor) designed to prioritize the determined arrangements,

[0054] An output unit (user interface) designed to select the arrangement with the best priority ranking and to output the selected arrangement.

[0055] The system may preferably further comprise a rendering unit (renderer) designed for rendering the image with shadows using the selected arrangement of light sources.

[0056] According to one embodiment, a control device (controller) for controlling a medical imaging system comprises the above system. Alternatively or additionally, the control device is designed to perform the above method. The control device may include additional units or devices for controlling components of the medical imaging system.

[0057] A medical imaging system according to an embodiment, such as a magnetic resonance imaging system (MRI) or a computed tomography system (CT), includes the control device described above.

[0058] In summary, the placement of light sources controls the location of shadows and highlights. However, shadows are also important because they provide essential visual clues about the shape, relative position, and surface properties of objects. Shadows create a sense of depth, a key clue for capturing realistic images. However, if the lighting is not set up correctly, shadows can also hide important details in the data. This can be overcome by selecting the optimal placement of light sources implemented by the embodiments.

[0059] Some units or modules of the above-mentioned system or control device can be fully or partially implemented as software modules running on the processor of the system or control device. Implementation mainly in the form of software modules can have the advantage that applications already installed in existing systems can be updated with relatively little effort to install and run these units of the present application. The purpose is also achieved by a computer program product having a computer program that can be directly loaded into the memory of the device of the system or the control device of the medical imaging system and includes a program unit to perform the actions of the method when the control device or system executes the program. In addition to the computer program, such a computer program product may also include other parts such as documents and / or additional components, as well as hardware components such as hardware keys (software dogs, etc.) to facilitate access to the software.

[0060] A non-transitory computer-readable medium such as a memory stick, hard disk, or other transportable or permanently mounted carrier may be used to transport and / or store the executable portions of a computer program product so that these portions can be read from a processor unit (processor) that controls a device or system. The processor unit may include one or more microprocessors or their equivalents.

[0061] As disclosed in the following description, the present disclosure provides particularly advantageous embodiments and features. Features of the present disclosure may be appropriately combined to provide other embodiments not described herein.

[0062] Preferably, spatial information of the region of interest in the 3D image is provided by automatic detection. Regarding the ROI, spatial information can be provided, in particular, via a segmentation algorithm or using an eye tracking system that provides information about the focus of attention. Preferably, the ROI is identified based on user selection of a surgical instrument, automatic segmentation, data analysis, user gaze tracking, or 3D tracking.

[0063] Preferably, after the first determination of the plurality of possible arrangements of the light sources, a plurality of further light sources are additionally provided, and a second determination of the plurality of possible arrangements for the additional light sources is performed. This preferably means that the arrangements of all light sources (the light sources provided at the beginning and the newly added light sources) are determined, which is advantageous when no good arrangement can be found. Preferably, point lights are added if no position can be found for a particular ROI (for example, if there is ambient occlusion around the ROI). Point lights should always be directed to the area with the most objectionable shadows.

[0064] A "good" arrangement is one in which all relevant ROIs (the ROIs considered by the method) are illuminated while providing good overall illumination. A "good" light arrangement should, for example, place the primary light within the intersection of the frustums (each frustum being defined by the ROI in the direction of the camera). If such an arrangement does not exist, auxiliary light sources are added for such ROIs that are not well illuminated. The quality can be quantified by determining the intensity and / or area of ​​shadows in the ROI and comparing this value to a predefined threshold.

[0065] However, it may also be advantageous for the second determination to be based on multiple arrangements determined by the first determination. This means that the arrangements found for the initially provided light sources remain unchanged, and only the newly added light sources are included as possible new arrangements in these initial arrangements. The total number of arrangements can be greater than the total number of arrangements after the first determination, because for any "first arrangement," several arrangements of the newly added light sources may be found.

[0066] Preferably, the prioritization action is performed based on criteria from photography. Preferably, a high priority is assigned to the sky or key light of the light map for locations above and towards the right side of the scene that illuminate most of the region of interest (in this arrangement). Alternatively or additionally, a low priority is assigned to the key light in the direction of the virtual camera (e.g., this may be valid for all light sources). This should be avoided to avoid flat images. Alternatively or additionally, a high priority is given to the auxiliary light when it is placed closest to the camera-to-object axis to avoid creating visible shadows. Alternatively or additionally, a low priority is given to a light source coming from the front unless it is the only way to illuminate all ROIs.

[0067] Preferably, at least one light source is a high dynamic range (HDR) light probe. This is advantageous for enhancing realism by illuminating the scene and producing natural-looking images. Alternatively or additionally, at least one light source is omnidirectional. This is advantageous for illuminating shadows.

[0068] Preferably, backlighting techniques are used to automatically arrange light sources in the scene, in particular directional light maps or placement of primary light sources. For the region of interest, the ROI centroid is preferably used as the viewpoint, with a frustum oriented towards the camera position. Thus, for the region of interest, the centroid of the region of interest is calculated as the viewpoint, and a frustum oriented towards the position of the virtual camera is calculated. The calculated depth information is used to detect which light positions will generate shadows on the ROI.

[0069] This process is repeated for multiple ROIs, tracking which light positions illuminate the ROI most. For larger ROIs, a bounding box can be placed around the ROI and the method repeated using the centroid or visible corners of the bounding box. Thus, the method is preferably performed for multiple regions of interest in a 3D image and / or for multiple disconnected objects forming a single region of interest. Thus, the light arrangement for multiple regions is determined. This is preferably performed by placing a bounding box around the region of interest and / or object and repeating the determination using the centroid or visible corners of the bounding box.

[0070] Preferably, a light map is used as the light source. This particular embodiment of a light map can be considered as a non-changing arrangement or fixed light distribution of an "internal" light source. Preferably, by determining the internal arrangement of light or lighting of the light map, the internal arrangement of light or lighting of the light map is not changed, but different orientations of the light map are determined when searching for an effective (or "good") arrangement. Thus, in practice, the light map can be rotated so that the brighter part (usually the sky) points in the right direction. This can be done similarly to calculating a suitable main light direction.

[0071] Preferably, the surgical instrument is the region of interest. This is particularly advantageous when 3D tracking and positioning of the device is required, especially for virtual reality, medical intervention and / or surgical planning simulations.

[0072] The method may also include elements of "cloud computing." In the field of cloud computing, IT infrastructure, such as storage space, processing power, and / or application software, is provided via a data network. Communication between users and the "cloud" occurs via data interfaces and / or data transmission protocols.

[0073] In the context of "cloud computing," in a preferred embodiment of this method, data is provided to the "cloud" via a data channel (e.g., a data network). The "cloud" comprises a (remote) computing system, such as a computer cluster that typically does not include the user's local machine. The cloud can be provided, in particular, by a medical facility that also provides a medical imaging system. In particular, image acquisition data is sent to the (remote) computer system (the "cloud") via a RIS (Radiology Information System) or a PACS (Picture Archiving and Communication System).

[0074] Within the scope of a preferred embodiment of the system, the aforementioned units are present on the "cloud" side. The preferred system also includes a local computing unit (computer) connected to the system via a data channel (e.g., a data network, particularly configured as a RIS or PACS). The local computing unit includes at least one data receiving interface for receiving data. Furthermore, the local computer preferably also has a transmission interface for sending data to the system.

[0075] Obscuring important areas is avoided. In medical visualization, many structures, such as bones, often cast shadows on other important structures. Proper setting of lighting parameters is essential in generating images that provide clear illustrations of complex anatomical structures.

[0076] Instead of generating balanced images with good contrast, the method of the current embodiment is based on regions of interest within the volume data.One of the main objectives is to provide optimal illumination of important anatomical structures in order to produce images useful for medical diagnosis or treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Other objects and features of the present invention will become apparent from the following detailed description considered in conjunction with the accompanying drawings. It should be understood, however, that the drawings are designed only for purposes of illustration and not as a definition of the limits of the present invention.

[0078] Figure 1 A simplified MRI system is shown having a system according to one embodiment.

[0079] Figure 2 Block diagram showing the process flow of the preferred method.

[0080] Figure 3 An example of lighting is shown.

[0081] Figure 4 An exemplary backlighting based on ROI location is shown.

[0082] Figure 5 Exemplary frustums of light designs for two given ROIs are shown.

[0083] Figure 6Shown is an example rendered image illuminated by a light map.

[0084] Figure 7 Shown is an example rendered image illuminated by front lighting.

[0085] Figure 8 An exemplary rendered image is shown illuminated by upper right lighting.

[0086] Figure 9 Shows an example rendered image illuminated by a light map and a key light.

[0087] Figure 10 Shows an example rendered image lit by a light map, key light, and fill light.

[0088] Figure 11 Shows an example rendered image illuminated by a light map, key light, and point lights.

[0089] Figure 12 It is shown that in one embodiment two ROIs with frustums are used during backlighting.

[0090] In the drawings, like reference numerals refer to like objects throughout. Objects in the drawings are not necessarily rendered to scale. DETAILED DESCRIPTION

[0091] Figure 1 A schematic representation of a magnetic resonance imaging system 1 ("MRI system") is shown. The MRI system 1 comprises an actual magnetic resonance scanner (data acquisition unit) 2 having an examination space 3 or patient tube cabin, wherein a patient or test person is located on a drive couch 8, at the body of which the actual examination object is located.

[0092] The magnetic resonance scanner 2 is typically equipped with a basic field magnet system 4, a gradient system 6, and an RF transmit antenna system 5 and an RF receive antenna system 7. In the exemplary embodiment shown, the RF transmit antenna system 5 is a whole-body coil permanently installed in the magnetic resonance scanner 2, whereas the RF receive antenna system 7 is designed as a plurality of local coils (here represented by only a single local coil) for parallel imaging arranged on a patient or test object.

[0093] The basic field magnet system 4 is typically designed such that it generates a basic magnetic field running in the z-direction in the longitudinal direction of the patient, i.e., along the longitudinal axis of the magnetic resonance scanner 2. The gradient system 6 generally includes individually controllable gradient coils so that gradients in the x-, y-, or z-direction can be switched (activated) independently of one another.

[0094] The MRI system 1 shown here is a full-body system with a patient tube cabin in which the patient can completely enter. However, in principle, other MRI systems can be used, such as a C-shaped housing with lateral openings and smaller magnetic resonance scanners in which only one body part can be placed.

[0095] Below, we use an MRI system as an example of a medical imaging system. Obviously, in addition to MRI data, images from CT, ultrasound, or other medical imaging systems can also be used.

[0096] In addition, the MRI system 1 has a central control device (controller) 13 for controlling the MRI system 1. The central control device 13 includes a sequence control unit (sequencer) 14 for controlling the measurement sequence. The sequence control unit 14 can be used to control a series of radiofrequency pulses (RF pulses) and gradient pulses according to a selected pulse sequence or, respectively, according to a series of multi-pulse sequences, to acquire magnetic resonance images of the patient O within a measurement session. For example, this series of pulse sequences can be predetermined within a measurement or control protocol. Different control protocols for different measurements or measurement sessions are typically stored in a memory 19 and can be selected (and possibly modified as needed) by the operator and then used to perform the measurement.

[0097] To output the individual RF pulses of the pulse sequence, the central control device 13 has a radio frequency transmission device (transmitter) 15, which generates and amplifies the RF pulses and feeds them to the RF transmission antenna system 5 via a suitable interface (not shown in detail). To control the gradient coils of the gradient system 6, the control device 13 has a gradient system interface 16. The sequence control unit 14 communicates with the radio frequency transmission device 15 and the gradient system interface 16 in a suitable manner to transmit the pulse sequence.

[0098] Furthermore, the control device 13 has a radio frequency receiving device (receiver) 17 (which likewise communicates in a suitable manner with the sequence control unit 14 ) in order to acquire the magnetic resonance signals (i.e., raw data) for the individual measurements, which are received in a coordinated manner from the RF receiving antenna system 7 within the scope of the pulse sequence.

[0099] A reconstruction unit (computer) 18 receives the acquired raw data and reconstructs magnetic resonance image data from it for use in the measurement. This reconstruction is typically performed based on parameters specified in a corresponding measurement or control protocol. The image data can then be stored in a memory 19, for example.

[0100] The central control device 13 can be operated via a terminal 10 having an input unit and a display unit 9, whereby the entire MRI system 1 can also be operated by an operator via this terminal. MR images can also be displayed on the display unit (screen) 9, and measurements can be planned and started via the input unit (possibly in combination with the display unit 9), and in particular, suitable control protocols can be selected (and possibly modified) using suitable pulse sequence series as described above.

[0101] The control device 13 comprises a system 12 for automatic light arrangement for medical visualization, which is designed to carry out the method according to the invention. The system 12 comprises the following components:

[0102] A data interface 20 is designed to receive a medical 3D image D, here with depth information DI, spatial information about a region of interest R in the image, and spatial information about a virtual camera C, and in particular information about a plurality of predefined artificial light sources L (e.g., from a memory or user input; see the following figures explaining the method according to the invention). The plurality of lights can also be calculated (on the fly) based on the ROI and the extent to which they are illuminated by the initial key light. The data interface can also be used to transmit data output from the system 12. The data interface 20 can be designed to receive data from a user interface, a PACS, or other data lines, and can include different components designed to receive data via different data channels.

[0103] A determination unit 21 is designed to determine a plurality of possible arrangements of the light source L by using depth information DI based on the 3D image D and spatial information about the region of interest R in the image and spatial information about the virtual camera C, wherein valid arrangements are those arrangements for which the shadow S on the region of interest R is below a predefined threshold.

[0104] A prioritization unit 22 is designed to prioritize the determined arrangements.

[0105] The output unit 23 is designed to select the arrangement with the best priority ranking and output the selected arrangement. The output unit can output data via the data interface 20.

[0106] A rendering unit is designed to render an image illuminated by the selected arrangement of light sources and having shadows resulting from the illumination, for example using a known rendering process.

[0107] The components of the system are preferably represented as software modules.

[0108] The MRI system 1 according to one embodiment, and in particular the control device 13, can have a number of additional components, which are not shown in detail but are typically present in such systems, such as a network interface in order to connect the entire system to a network and to be able to exchange raw data and / or image data or, respectively, parameter maps, but also additional data (e.g. patient-related data or control protocols).

[0109] The manner in which appropriate raw data is acquired by irradiating RF pulses and generating gradient fields and an MR image is reconstructed based on the raw data is known to those skilled in the art and therefore does not need to be explained in detail here.

[0110] Figure 2 A block diagram illustrating the process flow of a preferred method for automatic light arrangement for medical visualization according to the present invention.

[0111] In action I, a medical 3D image D is provided, here with depth information DI, spatial information about a region of interest R in the 3D image D, and spatial information about a virtual camera C. This can be done by user input, accessing image data via PACS, or measurements in the 3D image. If of interest, more than one region can be selected, such as an organ and a surgical instrument.

[0112] It should be noted that the region of interest R as well as the virtual camera C are only data sets related to this 3D image D, since the rendering of the (digital) image is the goal.

[0113] Spatial information of the region of interest R in the 3D image D can in particular be provided by automatic detection, preferably by detecting the region based on automatic segmentation of the surgical instrument, data analysis, user gaze tracking or 3D tracking, in particular where regions of high curvature are preferably detected via a segmentation algorithm or using an eye tracking system and defined as regions of interest.

[0114] Furthermore, information about a plurality of predefined artificial light sources L is provided, i.e., what type of light source should be used and how many light sources of a given type should be used. This provision can be accomplished by reading data from a memory (e.g., RAM, EEPROM, hard disk, MMC), in particular by preferences, calculated data, or user input. For example, at least one light source L can be a high dynamic range light probe and / or an omnidirectional light source. Furthermore, a light map can be used as the light source L. Three-point illumination is preferred.

[0115] In action II, multiple possible arrangements of the light source L are determined by using depth information DI based on the 3D image and spatial information about the region of interest R in the image and about the virtual camera C. Valid arrangements are those where the shadow S on the region of interest R is below a predefined threshold.

[0116] In this example, there is a possibility (see the box below action II with two dashed arrows) that, after first determining a plurality of possible arrangements of the light sources L, a plurality of additional light sources L, in particular point lights, may be additionally provided, and a second determination of a plurality of possible arrangements may be performed for the additionally provided plurality of light sources L. The second determination may be based on the plurality of arrangements determined by the first determination.

[0117] Preferably, a backlighting technique can be applied to automatically arrange the light sources L, preferably a directional light map or placement of a primary light source. Such backlighting techniques are well known in the art and are commonly used to infer information from images. For a region of interest R, the centroid of the region of interest can be calculated as the viewpoint, and a frustum oriented toward the position of the virtual camera C can be calculated.

[0118] In action III, the determined arrangements are prioritized. The action of prioritizing is performed based on criteria from the photography.

[0119] Possible criteria for high priority are:

[0120] a) (for the key light or lightmap sky) to have a position above and towards the right side of the scene that illuminates most of the area of ​​interest, or

[0121] b) (For auxiliary light source) Place it closest to the camera-to-object axis.

[0122] Possible criteria for low priority are:

[0123] a) a light shining in the direction of the virtual camera, or

[0124] b) The light source shines from the front of the ROI, unless it is the only means of illuminating the entire region of interest.

[0125] In action IV, the arrangement with the best priority is selected from the determined arrangements.

[0126] In act V, the image is rendered with shadows based on the selected arrangement of light sources.

[0127] The determination can be performed for a plurality of regions of interest R, R1 in the 3D image D (e.g., see Figure 5) to execute the method.

[0128] Figure 3 An example of lighting is shown. A region of interest comprising two different objects (dashed boxes) is illuminated by a light source L and viewed by a virtual camera C. As can be seen by the thick black lines, shadows S are generated when an object completely or partially obscures the light source L.

[0129] Figure 4 Backlighting based on ROI position is shown. This backlighting technique is used to automatically orient a light map or place a primary light source in a scene (ie, arrange the light source).

[0130] Figure 5 Two regions of interest R (circles) are shown. During the initial operation, the ROI centroid is used as the viewpoint, where the frustum looks towards the virtual camera C. The calculated depth information is used to detect which light positions will generate shadows on the ROI. For the second ROI, the process is repeated, tracking which light positions illuminate the most of both ROIs. For larger ROIs, a bounding box B (as shown here) can be placed around the ROI, and the method can be performed using the centroid or visible corners of the bounding box.

[0131] Once the potential placement of light sources has been determined, they can be prioritized following criteria from photography. For example, frontal lighting should be avoided unless it is the only way to illuminate all ROIs. Using these criteria, the highest priority can be given to locations that illuminate the right side of the scene most of a given ROI, above and to the right.

[0132] Figures 6 to 11 Results are shown for different arrangements of the light sources L. In all these figures, the results of the same image of the blood vessels around the neck as ROI being rendered by different arrangements of light are shown.

[0133] Figure 6 Shown is an image rendered using the default light map to illuminate the scene with the sky at the top. Although the light map illuminates the scene in almost all directions, the vasculature around the neck still appears quite dark for diagnostic purposes.

[0134] Thus, it can be seen that even with these omnidirectional light sources, some areas of the volume may still be hidden by shadows. For example, the vasculature around the neck may be indistinguishable due to the shadow cast by the jawbone. Although this image has a good balance of highlights, midtones, and shadow areas, it is not effective for accurate diagnosis of patients with vascular conditions.

[0135] Figure 7 Shown is a rendered image using frontal lighting. This configuration produces unobstructed shadows, but the resulting image is rather flat.

[0136] Figure 8 Shown is a rendered image illuminated by upper right lighting, which produces a fairly good 3D appearance, but severe shadows.

[0137] Figure 9 Shown is a rendered image generated by placing a lightmap with an overhead sky and a key light to illuminate the vasculature, avoiding hard shadows on the ROI.

[0138] Figure 10 Shows a rendered image lit by a light map, key light, and fill light. Figure 9 The same setup, with a fill light to make the shadows look softer.

[0139] Figure 11 Shows a rendered image lit by a lightmap, key light, and point light. This image uses the same Figure 9 The same configuration, with the addition of a point light placed inside the skull to illuminate the backside of a given ROI, creates a glowing effect.

[0140] Occasionally, ambient occlusion can be found around a ROI, since there is no guarantee that there is a configuration where all (entire) ROIs can be illuminated by the key light. Figure 7 A configuration where the key light is behind the camera provides fewer shadows, but should be avoided because it produces a flat image. Figure 8 ) or Figure 6 The lighting in the scene provides a good 3D impression, but it obscures severe shadows in important areas. To brighten these obscured areas, you can use a point light or fill light.

[0141] The ambient occlusion of the ROI can be approximated based on the depth map from the camera view. When the ambient occlusion is found to be significant, a point light can be placed behind the ROI (see Figure 11 ).

[0142] exist Figure 12 In Figure 1, two ROIs are shown together with their frustums. For a better overview, the scene is shown from above. Prioritization can be performed by using the frustums of the ROIs in the camera direction. Since the intersection of the frustums is usually not a single point, the vertex V of the resulting intersection object can be picked in such a way that the direction of the light is not parallel to the view direction. i , to avoid flattening the image. For each vertex v i , you can calculate v iThe average vector associated with each ROI is calculated, and the dot product of these vectors with the view direction can also be calculated. A dot product near 0.707 can be selected as the highest priority. This corresponds to an angle between the light and the camera closer to 45°. A directional light can be used as the main light, so the distance to the ROI is not important.

[0143] Although the present invention has been disclosed in the form of preferred embodiments and variations thereon, it will be appreciated that numerous additional modifications and variations may be made thereto without departing from the scope of the invention. For the sake of clarity, it will be appreciated that the use of "a" or "an" throughout this application does not exclude a plurality, and "comprising" does not exclude other actions or elements. Reference to "a unit" or "a device" does not exclude the use of more than one unit or device.

Claims

1. A method for automatic light placement for medical visualization, the method comprising: Provide medical 3D images, providing spatial information about a region of interest in the 3D image and spatial information about a virtual camera, determining a plurality of arrangements for light sources using depth information based on the 3D image together with spatial information about the region of interest in the medical 3D image and spatial information about the virtual camera, wherein valid arrangements of the plurality of arrangements are those arrangements for which shading on the region of interest is below a predefined threshold, prioritizing the determined plurality of arrangements, and The placement with the greatest priority ranking is selected.

2. The method according to claim 1, wherein The spatial information of the region of interest in the 3D image is provided by automatic detection.

3. The method according to claim 2, wherein: The automatic detection includes detecting regions based on automatic segmentation of the surgical instrument, data analysis, user gaze tracking, or 3D tracking, wherein regions of high curvature are detected via data analysis and defined as regions of interest.

4. The method according to claim 1, wherein After the first determination of a plurality of arrangements for the light sources, a further plurality of light sources is further provided, and a second determination of a plurality of arrangements is performed for the further provided plurality of light sources.

5. The method according to claim 4, wherein One of the further light sources comprises a point light, wherein the second determination is based on a plurality of arrangements determined by the first determination.

6. The method according to claim 1, wherein The prioritized actions are performed based on criteria from the photography.

7. The method according to claim 6, wherein: Guidelines from photography include: Give high priority to the sky or key light of the lightmap located above and towards the right side of the scene that illuminates most of the area of ​​interest. Give low priority to lights in the direction of the virtual camera, Give high priority to fill lights placed closest to the camera-to-object axis, and Low priority is given to light sources coming from the front, unless the light source coming from the front is the only way to illuminate all areas of interest.

8. The method according to claim 1, wherein At least one light source is a high dynamic range light probe and / or an omnidirectional light source.

9. The method according to claim 1, wherein The method is performed for multiple regions of interest in the 3D image and / or for multiple disconnected objects forming one region of interest by placing a bounding box around the region of interest and / or object and repeating the determining step by using the centroid or visible corners of the bounding box.

10. The method according to claim 1, wherein A light map is used as one of the light sources, the light map having a fixed light distribution, wherein the internal arrangement of the light or illumination of the light map is not changed by determining its arrangement, but rather different orientations of the light map are determined in the search for an arrangement.

11. The method according to claim 1, wherein Apply backlighting techniques to automatically orient light maps or place key light sources.

12. The method according to claim 1, wherein For the region of interest, the centroid of the region of interest is calculated as the viewpoint, and a frustum is calculated towards the position of the virtual camera.

13. The method according to claim 1, wherein The surgical instrument is the region of interest. The method of claim 1 , further comprising providing information about a plurality of predefined artificial light sources.

15. A system for automated light placement for medical visualization, the system comprising: A data interface is configured to receive a medical 3D image, spatial information about a region of interest in the medical 3D image, spatial information about a virtual camera, and information about a plurality of predefined artificial light sources, and a processor is configured to: use depth information based on the medical 3D image together with the spatial information about the region of interest and the spatial information about the virtual camera to determine a plurality of arrangements for the light sources, wherein valid arrangements among the plurality of arrangements are those arrangements in which shadows on the region of interest are below a predefined threshold, the processor is configured to prioritize the plurality of arrangements, and the processor is configured to select an arrangement with the best priority ranking and output the selected arrangement.

16. The system according to claim 15, wherein: The processor is a controller of the medical imaging system.

17. A non-transitory computer-readable medium having stored thereon instructions readable and executable by a processor, the instructions comprising: determining a plurality of arrangements for light sources using depth information based on the medical 3D image together with spatial information about a region of interest in the medical 3D image and spatial information about a virtual camera, wherein valid arrangements of the plurality of arrangements are those arrangements for which shadows on the region of interest are below a predefined threshold, prioritizing the determined plurality of arrangements, and The placement with the greatest priority ranking is selected.

Images