Image Rendering Device

Abstract

To reduce a calculation load on global illumination and reduce unnaturalness of appearance during display change.SOLUTION: An image processing apparatus according to an embodiment comprises a processing circuit. The processing circuit is configured to render an image from volumetric image data based on illumination from at least one simulated light source. When the position or another property of the at least one simulated light source is changed, the illumination is determined from a current portion of light intensity and at least one trailing portion of light intensity.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The embodiments disclosed in this specification and the drawings relate to an image rendering device. Place Regarding. [Background technology]

[0002] Various image rendering techniques are known for generating images representing a scene and / or features of interest from volumetric image data, e.g., a set of voxels. Such image rendering techniques have applications in many fields, including applications in medical imaging, where images are rendered from volumetric image data acquired using any suitable imaging modality, such as computed tomography (CT), magnetic resonance (MR), ultrasound, positron-emission tomography (PET), single photon emission computed tomography (SPECT) imaging, etc.

[0003] Many image rendering techniques are computationally intensive, especially when a user interacts with the image, e.g., by navigating the image or by changing viewpoint, lighting, or other conditions. Such interaction is common, for example, in medical imaging applications where a user may wish to view anatomical features, pathologies, or other features from different positions or under different conditions, e.g., to diagnose or evaluate the condition of a patient or other subject.

[0004] Global illumination and other advanced rendering techniques often require more computational power than is available to perform at maximum levels. This becomes evident when re-rendering or re-calculation is required due to user interaction, such as moving the virtual camera or light source or other conditions. For example, lowering the resolution of the irradiance volume is known to result in interactive rendering with lower image quality and / or lower accuracy or other quality of lighting conditions when the virtual light source moves or the virtual camera moves with the light. When the interaction stops, e.g., the virtual camera or light stops moving, the rendering can be performed at the highest quality with the now fixed position. However, this can result in unnatural transitions after the camera and / or light source movement ends, where the user may notice subtle changes across the entire image after there is no apparent movement of the camera or light. Such effects can have significant quality consequences on systems with low or moderate levels of computational power, especially in the case of workstations or other computing devices used in clinical settings. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] US Patent Publication No. 2016063758 [Patent Document 2] US Patent Publication No. 2016217563 [Patent Document 3] US Patent Publication No. 2018227568 Summary of the Invention [Problem to be solved by the invention]

[0006] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to reduce the calculation load of global illumination and to reduce the unnatural appearance when changing the display. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0007] In a first aspect, an image processing apparatus is provided that includes a processing circuit configured to render an image based on volumetric image data based on illumination from at least one simulated light source, the illumination being determined from a current portion of light intensity and at least one trail portion of light intensity when a position or other property of the at least one simulated light source changes. The sum of the light intensities of the current portion of the light intensity and the trail portion of the light intensity is equal to the sum of the light intensities before the position of the at least one light source changes.

[0008] A sum of the light intensities of the light intensity current portion and the light intensity trail portion may be equal to a sum of the light intensities before the position of the at least one light source is changed.

[0009] The current portion of light intensity and the trail portion of light intensity may change based on changes in a position or other property of the at least one light source.

[0010] A plurality of lighting states may be generated, each for a different time point, due to changes in a position or other property of the at least one light source. The processing circuitry may be configured to re-render the image for each of the different time points in a series using a combination of a current lighting state based on the current intensity for a current time point and at least one trail lighting state based on a trail portion of at least one of the light intensities for at least one previous time point.

[0011] The lighting conditions may comprise an irradiance volume or other data set representing irradiance as a function of position.

[0012] The at least one trail lighting state may comprise a transformed and / or faded version of what was a current lighting state when the at least one past time point was a current time point.

[0013] To obtain the trail lighting conditions for the at least one past time point, a time-dependent function or other function may be applied to what was the current lighting condition when the at least one past time point was the current time point.

[0014] The time-dependent or other function may include an exponential function.

[0015] The number of past time points used for the re-rendering may be determined depending on measurements of the performance of the imaging processor and / or on available computational resources.

[0016] The variation of irradiance as a function of position in the historical lighting conditions may be modified to provide blurring or other desired effects, or the variation may be modified by applying a bidirectional reflectance distribution function or other function.

[0017] In response to fixation of a virtual light source, or cessation of other changes in at least one property of said light source, the number of trail lighting states used for re-rendering may be tapered down to zero.

[0018] The lighting conditions for the current time point may be calculated at a lower resolution or other quality measure than the lighting conditions for at least one of the previous time points.

[0019] The lighting conditions for the current time point may then be recalculated at a higher resolution or other quality measure than the original lower resolution or other quality measure.

[0020] The processing circuitry may be configured to identify a movement of the simulated light source representing a jump, and in response to the identified jump, render the image without using the current portion of the light intensity and / or the trail portion of the light intensity.

[0021] The processing circuitry may be configured to create a plurality of frames, each corresponding to a respective one of the time points and representing a rendered image at that time point.

[0022] The processing circuitry may be configured to accept user input and to generate a change in a position or other property of the at least one simulated light source based on the user input.

[0023] The other properties may comprise at least one of orientation, beam direction or size, or light color.

[0024] The volumetric image data may comprise at least one of medical imaging data, CT, MR, ultrasound, PET, or SPECT imaging data.

[0025] In a further aspect that may be independently provided, a method of image processing is provided comprising rendering an image from volumetric image data based on illumination from at least one simulated light source, the illumination being determined from a current portion of light intensity and at least one trail portion of light intensity when a position or other property of the at least one simulated light source changes.

[0026] Features according to one aspect may be provided as features according to any other aspect, for example an apparatus feature may be provided as a method feature and vice versa. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 is a schematic diagram of an example of an apparatus according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram illustrating an example of the relative positions of a virtual camera and a virtual light source during an interactive process in which the position of the virtual light source moves relative to the scene as seen by the camera. [Diagram 3] FIG. 3 is a schematic flow chart according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] An embodiment provides an image processing apparatus comprising a processing circuit configured to render an image from volumetric image data based on illumination from at least one simulated light source, the illumination being determined from a current portion of light intensity and at least one trail portion of light intensity when a position or other property of the at least one simulated light source changes.

[0029] An embodiment provides an image processing method comprising rendering an image from volumetric image data based on illumination from at least one simulated light source, the illumination being determined from a current portion of light intensity and at least one trail portion of light intensity when a position or other property of the at least one simulated light source changes.

[0030] An example of a data processing device 20 according to an embodiment is shown diagrammatically in Fig. 1. In this embodiment, the data processing device 20 is configured to process medical imaging data. In other embodiments, the data processing device 20 may be configured to process any suitable image data.

[0031] The data processing device 20 comprises a computing device 22, in this example a personal computer (PC) or workstation, connected to a display screen 26 or other display device, and one or more input devices 28, such as a computer keyboard and / or mouse.

[0032] The computing device 22 is configured to acquire an image data set from a data store 30. The image data set has been generated by processing data acquired by the scanner 24 and stored in the data store 30.

[0033] The scanner 24 is configured to generate medical imaging data, which may comprise two-dimensional, three-dimensional, or four-dimensional data of any imaging modality. For example, the scanner 24 may comprise a magnetic resonance (MR or MRI) scanner, a CT (computed tomography) scanner, a cone-beam CT scanner, an X-ray scanner, an ultrasound scanner, a PET (positron emission tomography) scanner, or a SPECT (single photon emission computed tomography) scanner. The medical imaging data may comprise or be associated with additional conditioning data, which may comprise, for example, non-imaging data.

[0034] Computing device 22 may receive medical image data and / or further conditioning data from one or more further data stores (not shown) instead of or in addition to data store 30. For example, computing device 22 may receive medical image data from one or more remote data stores (not shown) that may form part of a Picture Archiving and Communication System (PACS) or other information system.

[0035] The computing device 22 provides processing resources for automatically or semi-automatically processing the image data. The computing device 22 includes a processing unit 32. The processing unit 32 includes lighting circuitry 34 for simulating lighting, e.g., providing at least one virtual light source and generating irradiance volumes or other illumination states using the at least one virtual light source. The processing unit 32 also includes rendering circuitry 36 for rendering an image based on the image data and the simulated lighting, and interface circuitry 38 for obtaining user or other input and / or outputting rendered image frames or rendered data from the rendering circuitry to, e.g., the display screen 26 for display.

[0036] In this embodiment, the circuits 34, 36, 38 are each implemented in the computing device 22 by a computer program having computer readable instructions executable to perform the method of the embodiment. However, in other embodiments, the various circuits may be implemented as one or more Application Specific Integrated Circuits (ASICs) or Field Programmable Gate Arrays (FPGAs). The processing device 32 is an example of a processing circuit in this embodiment. The processing device 32 may also be referred to as a processing unit. The lighting circuit 34, the rendering circuit 36, and the interface circuit 38 may also be examples of processing circuits in this embodiment. The lighting circuit 34 may also be referred to as a lighting unit, the rendering circuit 36 ​​may also be referred to as a rendering unit, and the interface circuit 38 may also be referred to as an interface unit.

[0037] Computing device 22 also has a hard drive and other components of a PC, including RAM, ROM, a data bus, an operating system including various device drivers, and hardware devices including a graphics card, although such components are not shown in FIG.

[0038] The data processing apparatus 20 of FIG. 1 is configured to carry out the methods illustrated and / or described below.

[0039] An embodiment is characterized in that instead of just downsizing / reducing the resolution of the lighting calculation during the relative movement of one or more light sources, the processing unit 32 is configured to amortize the light movement over several frames by splitting the simulated light into at least two parts, namely a current part and a trail part. The trail part can be faded / decreased each time energy is added to the current part, thereby keeping the intensity stable. Depending on the relative movement of one or more light sources, the current part of the light intensity can be added to the trail part of the light intensity before starting the lighting from the new position. When the relative movement stops, the current part can be continuously refined / increased until it is at full intensity. Amortizing the current light update based on the relative movement over several frames allows maintaining a high image quality and the image is clear. At a rendering rate of 20 frames per second (fps), amortization over 5 frames amounts to a maximum light delay of 250 ms. This can also result in a kind of motion blur effect. The current part of the light intensity is the light intensity at the current light source position. The trail portion of the light intensity is the light intensity at the past light source position.

[0040] The amortization processes according to one embodiment are described in more detail with respect to FIGS.

[0041] 2 shows an example of a virtual camera 50 that provides a perspective into a volume represented by an image dataset. The image dataset comprises a representation of a feature of interest 52 that is present within that volume. In the case of a medical application, the feature of interest may comprise, for example, an anatomical feature or a pathology.

[0042] The volumetric image dataset of this embodiment comprises an array of voxels. The array of voxels represents a three-dimensional volume corresponding to some or all of a three-dimensional region of a patient. Each voxel has a location in the coordinate space of the volumetric imaging dataset and an associated signal intensity. In alternative embodiments, the volumetric imaging dataset may comprise multiple channels of signal intensity. For example, more than one signal intensity channel may be used if the volumetric imaging dataset comprises dual-energy CT data, if the volumetric imaging dataset comprises both pre-contrast and post-contrast data, or if the volumetric imaging dataset comprises data acquired in a multi-volume fusion scenario. In further embodiments, the volumetric imaging dataset may comprise non-voxel-based data.

[0043] FIG. 2 also illustrates a virtual light source 54 provided by lighting circuitry 34. The virtual light source is illustrated in FIG. 2 at a series of positions 54a, 54b, 54c, 54d, each position corresponding to a different time. In this embodiment, movement of the virtual light source from position 54a, through 54b and 54c, to 54d is provided in response to user input determined by the interface circuitry. User input may be obtained via any suitable user input device, such as, for example, a touch screen, a trackball, a rotary switch, a mouse, or a keyboard. If the virtual camera is fixed relative to the volume in this example, movement of the virtual light source illustrated in FIG. 2 results in relative movement of light source 54 and the viewpoint of virtual camera 50.

[0044] In the embodiment of Figure 2, the virtual light source is fixed at position 54a for an extended period of time before moving to 54d, during which no amortization occurs or such amortization has little or no effect. We first generally describe the rendering process that produces an image rendered with the virtual light source fixed at position 54a, and then we describe the lighting and rendering process while the virtual light source moves from 54a to 54d.

[0045] When the virtual light source 54 is at position 54a, the lighting circuitry 34 simulates light from the light source 54 using a lighting simulation process. The lighting simulation process comprises casting a ray from the light source 54 through the volume.

[0046] Each simulated ray from a virtual light source distributes virtual light energy into the volume along the ray's path. For each ray, the lighting circuitry 34 calculates the irradiance due to that ray at each of a series of sample points along the ray. Any suitable spacing of the sample points may be used. The irradiance at each sample point is the product of the virtual light energy of the ray when it reaches that sample point and an absorption function at that sample point.

[0047] In this embodiment, an absorption function is assigned to each sample point depending on the intensity of voxels adjacent to the sample point. For example, the intensity at the sample point may be determined by interpolating intensities from adjacent voxels, and an absorption function may be assigned depending on the determined intensity. High voxel intensity may be associated with high absorption. In another embodiment, an absorption function may be assigned to each voxel depending on the voxel intensity, and the absorption function for the sample point may be determined by interpolating absorption functions from neighboring voxels.

[0048] In some embodiments, the absorption function is a function having absorption characteristics that are color dependent and may be described as a spectral absorption. The absorption function may be defined, for example, such that blue and green light is absorbed more strongly than red light to represent tissue absorption. The irradiance calculated for each sample point is a spectral irradiance, which may be described as a combination of red, green, and blue components. In other embodiments, the irradiance may not be a spectral irradiance.

[0049] In this embodiment, the calculated irradiance is not directional irradiance. In other embodiments, an associated irradiance direction may be determined at each sample point for which irradiance is calculated. The irradiance direction may be the direction of incidence of light at each sample point. Any suitable irradiance model may be used.

[0050] For each of the sample points on a ray for which irradiance is calculated, the lighting circuitry 34 may distribute the irradiance to multiple neighboring voxels using any suitable method. For example, in some embodiments, the irradiance may be distributed to the nearest eight neighboring voxels. In other embodiments, the irradiance at each sample point may be distributed to any suitable number of voxels.

[0051] In this embodiment, as each ray passes through the volume, a portion of the virtual light energy is absorbed at each sample point, causing the ray to become weaker and redder. When a sufficient portion of the virtual light energy of a ray is absorbed, no further calculations may be performed on the ray for further positions along the ray. In this embodiment, if 99% of the virtual light energy of a ray is absorbed, no further calculations may be performed on the ray for further positions along the ray. In other embodiments, the portion may be different, such as 90% or 95%, for example.

[0052] In some embodiments, the lighting simulation process may simulate scattering as well as direct illumination. At some sample points along a ray, scattering may occur as well as or instead of absorption. In this embodiment, only single scattering is considered. In some embodiments, multiple scattering may be considered. In some embodiments, reflection may also be considered.

[0053] In this embodiment, the result of the lighting simulation process performed by the lighting circuitry 34 for the virtual light source 54 at location 54a is an irradiance volume with at least one irradiance value for each voxel location in the volumetric imaging data set. In an irradiance volume, each voxel has an associated irradiance, which may be a combination of irradiance contributions from multiple light rays that have accumulated virtual light energy at sample points near the voxel. The irradiance volume represents the irradiance within the irradiance volume as a function of position by providing an individual irradiance for each voxel location within the irradiance volume.

[0054] In this embodiment, a global illumination lighting model is used to calculate the irradiance volume, although in other embodiments, any suitable lighting model may be used.

[0055] The irradiance volume (eg, the irradiance values ​​for each voxel within the volume) is sometimes referred to as a lighting state, such as the current lighting state when the virtual light source is at position 54a.

[0056] In other embodiments, the irradiance volume may be defined as a grid of reference points. The reference points may or may not coincide with the voxels. In some embodiments, the irradiance volume may be defined as a grid of reduced sample voxels. In further embodiments, the irradiance volume may be defined as a set of points that do not form a grid, for example a point cloud, a tree structure, or a tree structured point cloud. Any suitable method may be used to represent the irradiance as a function of position.

[0057] Once the lighting conditions (e.g., irradiance volumes) have been calculated, a rendering circuit performs a rendering process based on the image data and the calculated lighting conditions of a virtual light source 54 at a position 54 a. The rendering process is performed at a viewpoint provided by a virtual camera 50, which has a viewpoint position and viewpoint orientation.

[0058] The virtual camera may be positioned according to input from a user (e.g., a clinician or radiologist). Alternatively, the rendering circuitry 36 may position the virtual camera 50 automatically.

[0059] The rendering circuitry 36 then renders an image based on the irradiance volume. In rendering, the rendering circuitry 36 uses both the irradiance volume and the volumetric imaging data set.

[0060] The image is rendered as seen by a virtual camera 50. The rendering circuitry 36 casts rays from the camera into the volume represented by the volumetric imaging dataset. Each ray may correspond to a pixel of the two-dimensional image dataset resulting from the rendering. Each ray from the camera steps through the volume represented by the volumetric imaging dataset by a voxel spacing. In other embodiments, a different spacing may be used.

[0061] For a given ray cast by the camera, the rendering circuitry 36 determines, from the calculated lighting conditions (e.g., irradiance volume), the irradiance value at each of a series of incremental points along the ray. If the incremental point represents a void, the ray skips to the next incremental point.

[0062] The lighting circuitry 34 can determine the irradiance at each incremental point by interpolating the irradiance from neighboring voxels of the irradiance volume. In this embodiment, the nearest eight neighboring voxels are considered in the interpolation of each point. In other embodiments, a different number of voxels may be considered. The rendering circuitry 36 integrates the irradiance at each incremental point along the ray from the camera to obtain a pixel color value for the ray. The rendering circuitry 36 thereby determines a color value for each pixel of the two-dimensional image dataset.

[0063] Once the rendered image data has been calculated by rendering circuitry 36, interface circuitry 38 outputs the rendered image data to display screen (display) 26 which then displays the rendered image. Alternatively, the interface circuitry may transmit the rendered image data to a remote device, for example over a network, and / or send it to data store 30 or other storage device for storage.

[0064] The lighting and rendering process performed when the virtual light source 54 is fixed at the position 54a and there is no relative movement between the virtual light source 54a and the volume viewed by the camera has been described above. A feature of the embodiment is to change the lighting and rendering process during a period when there is a relative movement of the virtual light source 54. An example of the flow of the lighting and rendering process during a period when there is a relative movement of the virtual light source will be described below with reference to the flowchart of FIG.

[0065] In a first stage 100 of processing, the processing unit 32 determines whether a new frame is needed or requested. In this embodiment, this determination is based on a desired or predetermined frame refresh rate at which to recreate and / or redisplay the frames. The frame refresh rate may be, for example, 25 frames per second or any other suitable frame rate.

[0066] The next stage 102 determines whether there has been a change in the relative position of the virtual light source 54 and the volume seen by the camera.

[0067] In this embodiment, the virtual light source 54 has just moved from position 54a to position 54b, so there is relative movement and the process moves to stage 104 where it is determined whether a light trail is currently present in the lighting and / or rendering process.

[0068] Since the movement has just begun and no light-trail currently exists, a light-trail is created at stage 106. Creating a light-trail in this embodiment involves creating and / or storing data representing virtual light illumination conditions, in this embodiment irradiance volumes, at past positions corresponding to past times.

[0069] In this example, the light-trail data is added to the irradiance volume data of what was the current irradiance volume before the movement (eg, the irradiance volume created by the light at the past position 54a).

[0070] At stage 110, a new current lighting state is created, in this embodiment an irradiance volume is created for the virtual light source at the new current position 54b. The irradiance volume is calculated using the process described above based on casting rays from the virtual light source, and calculating the irradiance of each ray at a series of sample points along the ray based on the intensity of the light source and an absorption function and / or scattering or other function to make up the irradiance volume.

[0071] The embodiment of FIG. 3 is characterized in that, in order to reduce the computational load, during relative movement the current light conditions (e.g., irradiance volume) are calculated with a lower resolution and / or accuracy and / or other quality parameters than when the camera and light source are fixed.

[0072] Thus, in this example, the irradiance volume calculated for the virtual light source at position 54b (current light state) is calculated with a step size between sample points that is larger than the step size between sample points used to calculate the irradiance volume when the virtual light source was fixed at position 54a for a long period of time (this irradiance volume from position 54a at this point in the process is now used as the trail light portion). Although a larger step size may be used to reduce the computational burden, interpolation or other processing can be used to ensure that the resolution of the irradiance volume data set for the current light portion (e.g., the irradiance volume for the light at position 54b) is the same (e.g., irradiance values ​​for each voxel) as the resolution of the irradiance volume for the trail light portion (e.g., the irradiance volume for the light at position 54a at this point in the process).

[0073] The embodiment is characterized in that the total intensity of the contributions from the current light and the trail light remains constant. For example, in a variant of the embodiment, the intensity of the virtual light source used in calculating the current irradiance volume (light at position 54b) is reduced by 50% from its normal intensity when fixed, and the irradiance value in the irradiance volume of the current light state (calculated when the virtual light source 54 was fixed at position 54a in this example) is also reduced by 50% to keep the overall lighting intensity constant despite the movement and the contributions of the virtual light source from two or more positions. In some variants or alternative embodiments, any suitable time-dependent or other function can be used to determine the relative weights to give to the current and trail portions, for example, in some embodiments, an exponential function may be applied to the intensity and / or irradiance values, such that the contributions from one or more trail light portions decay exponentially with time. In some embodiments, the function used or its parameters and / or the number of past time points for which the trail light state is used can vary, for example, depending on available computational resources and / or performance measurements.

[0074] At least one trail lighting condition may comprise a transformed and / or faded version of what was the current lighting condition when the at least one past time point was the current time point.

[0075] In a next stage 112, which is an optional stage in this variation of the embodiment, a function, e.g. a blurring function, is applied to the trail light state (e.g. the irradiance volume of the past position 54a of the virtual light source 54). The function may, for example, be one that represents or takes into account the directional uncertainty due to light tracking interactions. The function may provide any suitable desired effect.

[0076] In a next stage 114, the rendering circuitry 36 re-renders an image of the volume based on the irradiance and volumetric imaging data set using the rendering process described above in relation to stage 114. However, the irradiance values ​​used in performing the rendering calculations are now obtained from a combination of the irradiance values ​​of the irradiance volumes obtained for both the trail light position and the current light position (e.g. 54a and 54b in this stage of processing).

[0077] Once the rendered image data has been calculated by rendering circuitry 36, interface circuitry 38 outputs the rendered image data to display screen 26, which displays the rendered image data.

[0078] Processing returns to stage 100. If it is time to display a new frame, for example based on the current frame rate, processing proceeds to stage 102 to determine whether there has been further relative movement of virtual light source 54. In this example operation of the embodiment, virtual light source 54 has just been moved to position 54c, so there has been further relative movement.

[0079] Processing then proceeds to stage 104 where it is determined that a light trail currently exists (eg, an irradiance volume resulting from a past virtual light position 54).

[0080] Processing then proceeds to stage 116 where a fall-off function is applied to the existing trail light conditions. The fall-off function is a time-dependent function. Any suitable function may be used, for example, such that the importance of the trail light contribution decreases over time. In this example operation of the embodiment, an exponential function is applied, whereby the magnitude of the irradiance values ​​in the irradiance volume obtained at the virtual light position 54a is further reduced by a scaling factor, for example, obtained by application of an exponential or other function.

[0081] Processing then proceeds to stage 108 where the light-trail data is again modified to include irradiance volume data for what was the current irradiance volume until the most recent movement (e.g., now the irradiance volume created by the light at past position 54b). In this way, the light-trail data now comprises contributions from irradiance data obtained at both past virtual light positions 54a, 54b, but with a greater weighting being given to the irradiance data for the more recent virtual light position 54b.

[0082] As mentioned above, the irradiance volume calculated for the virtual light source at position 54b is calculated with a step size between sample points that is larger than the step size between sample points used to calculate the irradiance volume when the virtual light source was fixed at position 54a for a long period of time, thereby reducing the computational load. In a modified or alternative embodiment, the time between the previous iteration of stage 110 and the return to stage 108 is used by the lighting circuitry 34 to refine the irradiance volume calculated for the virtual light source 54b using a smaller step size, for example the same step size used to calculate the irradiance volume for the fixed position 54a. Alternatively, the irradiance volume for position 54b with the original, larger step size may still be used.

[0083] In the next stage 110, a new current lighting condition is again created, in this embodiment an irradiance volume is again created for the virtual light source at the new current position 54c. The irradiance volume is calculated using the process described above based on casting rays from the virtual light source 54, calculating the irradiance of each ray at a series of sample points along the ray based on the intensity of the light source and an absorption function and / or scattering or other function to build up the irradiance volume. To reduce the computational load, a larger step size is again used.

[0084] Processing again proceeds to stages 112 and 114 where blurring or other processing is applied to the light trail data, rendering is performed based on the current light data (e.g., irradiance volume data for virtual light position 54c) and the trail light data (e.g., irradiance volume data for virtual light positions 54a, 54b with different weights to keep the overall effective light intensity constant), and the rendered image is displayed.

[0085] In this example, processing then proceeds again to stages 100, 102, 104, 116, 108, 110, 112, 114, where the current light portion contribution is the irradiance volume calculated at the new virtual light position 54d, and the irradiance volumes obtained at the previous virtual light positions 54a, 54b, 54c contribute to the trail light portion, with decaying weighting over time such that the irradiance resulting from virtual light position 54c contributes more than those from virtual light positions 54b and 54a.

[0086] 3 embodiment then remains stationary. Thus, when processing next proceeds to stage 102, it is determined that there has been no further relative movement, and at stage 118 it is determined that there is light-trail data (e.g., irradiance volume contributions for virtual light positions 54a, 54b, 54c), and processing proceeds to stage 120.

[0087] In stage 120, a fall-off function, such as the exponential function described above, is applied to the trail light data, which reduces the trail light contribution (e.g., the irradiance volume contribution of virtual light positions 54a, 54b, 54c) relative to the current light contribution (e.g., the irradiance volume contribution of virtual light position 54d). This stage also applies a cut-off so that if the trail light contribution of a virtual light position (e.g., virtual light positions 54a, 54b, or 54c) falls below a threshold, it is set to zero.

[0088] In the next stage 122, the current light contribution (e.g. the irradiance volume data for the virtual light position 54d) is updated. This may comprise recalculating or refining the irradiance volume data with a reduced step size or otherwise increasing the accuracy (e.g. in some embodiments taking into account additional effects such as scattering or reflection that may have been omitted from the calculation while the light source was moving).

[0089] In a next stage, not shown in Fig. 3, the rendering circuitry 36 renders an image based on the irradiance and volumetric imaging data set as described above. The irradiance value used in performing the rendering calculation is obtained from a combination of the irradiance values ​​from the irradiance volume obtained at the current light position (e.g., position 54d) and the remaining irradiance contributions for the trail light positions (e.g., positions 54a, 54b, 54c) that have not yet fallen to zero by application of an exponential or other falloff function.

[0090] In the next stage 124, it is determined whether all trail light contributions have been reduced to zero. If not, the process returns to stage 100 and stages 102, 118, 120, 122 and 124 are repeated, resulting in a series of rendered images with reduced contributions from past positions of the virtual light and more refined and / or accurate lighting calculations for the current position.

[0091] Once all trail light contributions have fallen to zero, processing passes from stage 124 to stage 126, from which it returns to stage 100. The same rendered image continues to be displayed on the display screen 26 until there is further movement of the light source or other change in lighting or rendering.

[0092] The operation of the embodiment of FIG. 3 above will be described in relation to the case where the virtual light source moves relative to the volume it illuminates while the virtual camera position is fixed. In some embodiments, one or more virtual light sources are linked to the position of the virtual camera and changes in the camera position are monitored, for example, at stage 102. In other embodiments, trail light states are used to re-render in response to changes in light properties as well as or instead of the position of the virtual light source, for example, at least one of the light source orientation, beam direction, size or shape of the light beam produced by the virtual light source, or color of the light.

[0093] In some embodiments, there may be multiple virtual light sources, and the trail light state may be used in response to a change in the position or other property of any one or more of the light sources.

[0094] In some embodiments, the processing circuitry determines whether a change in position or other property indicates a jump (e.g., a change that is greater than a threshold amount and less than a threshold time) and, in response to an identified jump, renders the image using only the current portion of the light intensity and / or without the trail portion of the light intensity. In this manner, for example, a normal rendering process can be used when a jump occurs.

[0095] Although a particular lighting calculation and rendering process has been described in connection with a particular embodiment based on ray casting, any suitable lighting calculation and rendering process may be used in alternative embodiments.

[0096] Any suitable virtual light source may be used, such as, for example, directional or non-directional light sources, monochromatic light sources, and / or combinations of light sources.

[0097] One embodiment provides a global illumination rendering apparatus that amortizes continuous light movement or camera and light movement by combining partial updates of a new light configuration with transformed and / or faded portions of a previous light state.

[0098] The light trail portion and the latest portion may be stored separately and then combined when additional changes in the light configuration cause the latest portion to become stale.

[0099] The amortization may occur over several frames of updates.

[0100] A falloff function may be used for the trail light and the total remaining intensity may be used to determine if additional frames are needed. An exponential falloff may be a suitable falloff function.

[0101] The amortized frame count may be calculated based on the current or previous interactive performance of the running system. Multiple moving lights may contribute to the same trail state while maintaining separate current lighting states.

[0102] A bidirectional reflectance distribution function (BRDF) may be applied to or modified depending on the trail light conditions to model the directional uncertainty caused by light tracking interactions.

[0103] The device may identify interactions that should be considered jumps rather than continuous movements, for which the system will rely on regular image quality control metrics.

[0104] In some embodiments, the lighting conditions have been described in the form of an irradiance volume. Any other suitable lighting conditions may be used. References to irradiance volumes in relation to some embodiments may be replaced with references to radiance volumes.

[0105] Although particular circuits are described herein, in alternative embodiments, the functionality of one or more of those circuits may be provided by a single processing resource or other component, or the functionality provided by a single circuit may be provided by a combination of two or more processing resources or other components. Reference to a single circuit encompasses multiple components that provide the functionality of that circuit, whether or not such components are separate from one another. Reference to multiple circuits encompasses single components that provide the functionality of those circuits.

[0106] While certain embodiments have been described, these embodiments are presented for illustrative purposes only and are not intended to limit the scope of the invention. In fact, the novel methods and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions, and changes in the form of the methods and systems described herein may be made without departing from the spirit of the invention. The appended claims and their equivalents are intended to cover such forms and modifications as fall within the scope of the invention. [Explanation of symbols]

[0107] 20 Data Processing Device 22 Computing Equipment 24 Scanner 26 Display Screen 28 Input Devices 30 Data storage unit 32 Processing equipment 34 Lighting Circuit 36 Rendering Circuit 38 Interface Circuit 50 Virtual Camera 52 Interest Features 54, 54a to 54d Virtual light source, virtual light source position, virtual light position

Claims

1. An image processing device including a processing circuit, the processing circuitry is configured to render an image based on the volumetric image data based on illumination from at least one simulated light source; When a position or other property of the at least one simulated light source changes, the illumination is determined from a current portion of light intensity and at least one trail portion of light intensity; a sum of the light intensities of the current portion of the light intensity and the trail portion of the light intensity is equal to a sum of the light intensities before the position of the at least one light source is changed; Image processing device.

2. the current portion of light intensity and the trail portion of light intensity vary based on changes in a position or other property of the at least one light source. The image processing device according to claim 1 .

3. An image processing device comprising a processing circuit, the processing circuitry is configured to render an image based on the volumetric image data based on illumination from at least one simulated light source; When a position or other property of the at least one simulated light source changes, the illumination is determined from a current portion of light intensity and at least one trail portion of light intensity; varying a position or other property of the at least one light source to generate a plurality of lighting states, each lighting state for a different point in time; the processing circuitry is configured to re-render the image for each of the series of different time points using a combination of a current lighting state based on the current intensity for a current time point and at least one trail lighting state based on at least one trail portion of the light intensity for at least one past time point. Image processing device.

4. the lighting conditions comprise an irradiance volume or other data set representing irradiance as a function of position; The image processing device according to claim 3 .

5. the at least one trail lighting state comprises a transformed and / or faded version of what was a current lighting state when the at least one past time point was a current time point.

5. The image processing device according to claim 3.

6. a time-dependent or other function is applied to what was the current lighting condition when the at least one past time point was the current time point to obtain the trail lighting condition for the at least one past time point; The image processing device according to claim 3 .

7. The time-dependent or other function includes an exponential function. The image processing device according to claim 6.

8. the number of past time points used for the re-rendering is determined depending on one or both of a measurement of the performance of the image processing device and available computational resources; The image processing device according to claim 3 .

9. the variation of irradiance as a function of position in the historical lighting conditions is modified to provide blurring or other desired effects; The image processing device according to any one of claims 3 to 8.

10. Modifying the variation by applying a bidirectional reflectance distribution function or other function; The image processing device according to claim 9 .

11. gradually reducing to zero the number of trail lighting states used for re-rendering in response to fixation of a virtual light source or cessation of other changes in at least one property of said light source; The image processing device according to claim 3 .

12. the lighting conditions for the current time point are calculated at a lower resolution or other quality measure than the lighting conditions for at least one of the past time points; The image processing device according to claim 3 .

13. The lighting conditions for the current time point are then recalculated at a higher resolution or other quality measure than the original lower resolution or other quality measure. The image processing device according to claim 12.

14. the processing circuitry is configured to generate a plurality of frames, each corresponding to a respective one of the time points and representing a rendered image at that time point. The image processing device according to claim 3 .

15. An image processing device comprising a processing circuit, the processing circuitry is configured to render an image based on the volumetric image data based on illumination from at least one simulated light source; When a position or other property of the at least one simulated light source changes, the illumination is determined from a current portion of light intensity and at least one trail portion of light intensity; the processing circuitry is configured to identify a movement of the simulated light source that represents a jump, and in response to the identified jump, render the image without using the current portion of the light intensity and / or the trail portion of the light intensity. Image processing device.

16. the processing circuitry is configured to accept user input and generate a change in a position or other property of the at least one simulated light source based on the user input. The image processing device according to any one of claims 1 to 15.

17. The other properties comprise at least one of orientation, beam direction or size, or light color. The image processing device according to any one of claims 1 to 16.

18. the volumetric image data comprises at least one of medical imaging data, computed tomography (CT), magnetic resonance (MR), ultrasound, positron-emission tomography (PET), or single photon emission computed tomography (SPECT) imaging data; The image processing device according to any one of claims 1 to 17.

Images