Simulation of a mosaic filter sensor by converting the original color space of the virtual environment to the pixel brightness of the sensor.

Abstract

Synthetic image data is generated by deriving the pixel brightness of the sensor pixels, which depends on the brightness and color values ​​of the light mapped onto the sensor pixels, using a simulation of a mosaic filter sensor. [Solution] For each filter color generated within the mosaic filter, a mapping table is created to assign to the filter color (24). In each mapping table, a pixel brightness scale is assigned to a number of color values ​​from the raster scan of the original color space (28). For the sensor pixels of the simulated mosaic filter sensor, the filter color of the color filter covering the sensor pixels is examined (30), and the mapping table assigned to the filter colors is referenced (32). Using the readout of the pixel brightness scale assigned to the color values ​​(34), the pixel brightness of the sensor pixels is derived depending on the color values ​​of the light mapped onto the sensor pixels (36).

Description

Technical Field

[0001] The present invention relates to the simulation of a mosaic filter sensor by converting the original color space of a virtual environment into the pixel luminance of a sensor.

Background Art

[0002] In the development of computer-based control systems, it has long been known and customary to operate a control system in a virtual working environment in order to test the reaction of the control system to external stimuli. For this purpose, data recorded by a control system (test subject) placed under test using a sensor system is synthetically generated by a simulator computer and input into the test subject at an appropriate interface. The complexity of the simulation can here handle a wide spectrum, depending on the requirements for the test conditions, from the simple input of pre-calculated data to a complex virtual environment that also simulates the electromechanical working environment of the control system and external actors acting on the simulated working environment. This simulation can further be designed as a closed control loop that captures and takes into account the control data generated based on the synthetic data input by the control system. The virtual test environment offers several advantages compared to field tests in the physical world. These virtual test environments are inherently safe, accurately reproducible, and extreme test scenarios that rarely occur in the real world can also be created in a simple manner. However, it takes effort to provide a sufficiently realistic virtual test environment, and this is even more so the more sensor data the system under test processes and the more sensitive it is to changes in the sensor data.

[0003] In recent years, significant progress has been made in the development of autonomous robotic systems, moving from highly automated robotic systems capable of safely and independently navigating, at least temporarily, in changing environments such as road traffic, crowds, unfamiliar terrain, farmland, or combat environments. Such autonomous systems inevitably possess at least one imaging sensor, from which detailed images of the surrounding environment, including the environmental topology, object type, location, and movement state, can be derived from the supplied sensor data. In parallel with this development, sensor simulations have become available on the market, enabling the testing of such robotic systems in the aforementioned virtual environments. These simulations generate synthetic image data from virtual imaging sensors placed in the virtual environment, which can stimulate control systems using image data and compare their responses to target preset values.

[0004] Imaging sensors include cameras, radar, and lidar. Cameras, as passive sensors, are relatively inexpensive and have the advantage of providing sufficiently detailed, high-resolution images from which a lot of information can be extracted. Different technologies exist even within camera-type image sensors. The most popular, especially due to its low cost, is the mosaic filter sensor. In this technology, the original image sensor is not color-sensitive. Instead, a mosaic filter is placed in the camera's optical path upstream of the array of sensor pixels on the image sensor. The mosaic filter is a periodic mosaic of color filters arranged so that each sensor pixel is covered by exactly one color filter. Mosaic filters are often Bayer filters, and their mosaic consists of 50% green and 25% each of red and blue color filters, although other types of mosaic filters also exist. Mosaic filter sensors covered by Bayer filters are often also called Bayer sensors. Different color filters result in a strong discrepancy in the exposure of adjacent sensor pixels, and the color information of the image is reconstructed from these discrepancies in post-processing.

[0005] Basically, the synthesized image data output by sensor simulation must be suitable for the interface into which it is input to the object being tested. In this case, as a rule of thumb, the simulation of image data becomes easier the higher the level of image data processing at the interface. A list of objects in the vicinity of a virtual sensor, their types, and their movement states can be easily provided because this information is typically already stored in the virtual environment. In contrast, if the synthesized image data is intended to be input into the image preprocessing of a camera system, it is necessary to simulate the raw data generated by the image sensor of each camera. Conventional techniques require the simulation of the camera optical system as well as the technical and physical processes inside the image sensor.

[0006] This type of camera simulation is described, for example, in "Full Spectrum Camera Simulation for Reliable Virtual Developments and Validation of ADAS and Automated Driving Applications" in the publication "2015 IEEE Intelligent Vehicles Symposium (IV)" by Renae Molenaar et al. Further examples can be found in "Digital camera simulation" in the publication "Applied Optics Vol. 51 No. 4 (2012)" by Joyce Farrel, Peter B. Catrysse, and Brean Wandell et al.

[0007] In particular, simulating image sensors is complex and often yields unsatisfactory results. For simulation to work, one must first know key characteristic values, which in itself is considered a drawback, as sensor suppliers may strictly control these values. However, even if a realistic technical and physical software simulation of an image sensor based on these characteristic values ​​is successfully implemented, the resulting synthetic raw data often falls short of the raw data produced by a physical image sensor, because this physical sensor is designed for exposure using physical light in real-world environments. Dynamic 3D virtual environments are typically modeled using graphics engines such as Epic Games' Unreal Engine, Crytek's CryEngine, and Unity Technologies' Unity Engine. These engines, based on their respective color spaces, model colored light that produces a believable and realistic display to the naked eye, but does not correspond to physical reality. Typically, the original color space of a virtual environment is the RGB color space, which is based on the additive mixing of discrete red, green, and blue light components. [Overview of the project] [Problems that the invention aims to solve]

[0008] Given this background, the objective of the present invention is to simplify and improve the simulation of a mosaic filter sensor for generating raw data from a composite camera. [Means for solving the problem]

[0009] The present invention relates to a method for generating synthetic image data suitable for mimicking camera raw data generated by a mosaic filter sensor, such as a Bayer sensor. This means that the synthetic image data is suitable for being fed into a subject designed to process raw data, instead of the camera raw data generated by a physical mosaic filter sensor, and for deriving an available evaluation of whether the subject's response to the raw data from the physical mosaic filter sensor corresponds to a desired response, based on the subject's response to the synthetic raw data.

[0010] This method includes the step of positioning and spatially oriented a virtual camera in a three-dimensional virtual environment on a computer system. The virtual environment is preferably a 3D environment that is dynamically calculated in real time by a graphics engine and, in particular, rendered. The virtual camera is preferably suspended in a virtual representation of a robot system, in which case, in particular, the instantaneous positioning and / or instantaneous spatial orientation of the camera is preset by its internal state and / or motion state and / or the position of the robot system in the virtual environment. For example, the virtual camera may be positioned in a virtual autonomous vehicle moving within an environment simulated by the virtual environment, for example, within a simulated road traffic environment.

[0011] This method includes the step of simulating the mapping of colored light onto sensor pixels of a mosaic filter sensor installed in a virtual camera using a camera optical system. For the light mapped onto the sensor pixels, one luminance and one color value are defined here from the original color space of the virtual environment, for example, the sRGB color space. Both the color value and luminance of the light for each pixel depend on the proportion of light in the spatial region of the virtual environment that the simulated optical system of the virtual camera maps onto the simulated mosaic filter sensor, and thus both the color value and luminance of the light depend on the position and spatial orientation of the virtual camera.

[0012] The method further includes the step of generating composite image data using a simulation of a mosaic filter sensor installed in a virtual camera. This simulation of the mosaic filter sensor includes deriving the pixel brightness of the sensor pixels depending on the brightness and color values ​​of the light mapped onto each sensor pixel. Pixel brightness is preferably understood to mean a measured variable output from the sensor pixel depending on the amount of light mapped onto the sensor pixel during the exposure time, and which is interpreted and displayed as pixel brightness by downstream image processing of the sensor pixel. This pixel brightness can particularly preferably represent the voltage and / or current and / or charge generated and simulated by the sensor pixel, and the composite image data is generated from the total of the simulated charge and / or current and / or charge.

[0013] To simulate a mosaic filter sensor, the present invention assumes the creation of a look-up table assigned to each filter color occurring within the mosaic filter, so that each filter color has at least one look-up table assigned to it, and each look-up table has exactly one filter color assigned to it. For example, a Bayer filter, i.e., a Bayer sensor mosaic filter, includes blue, green, and red filter colors. For the simulation of a Bayer sensor, the present invention assumes the creation of at least three look-up tables accordingly, of which (without generality limitations) the first look-up table is uniquely assigned to the blue filter color, the second look-up table is uniquely assigned to the green filter color, and the third look-up table is uniquely assigned to the red filter color. Some Bayer sensors also include two different green color filters whose transmission spectra are slightly different. These different green hues can be interpreted as different filter colors. In this case, the method according to the present invention may also include the step of creating a fourth look-up table to store one unique look-up table for each of the two green hues occurring within the mosaic filter.

[0014] The mapping tables are designed so that, in each mapping table, one pixel luminance scale is assigned to each of the multiple color values ​​from the raster scan of the virtual environment's original color space. The pixel luminance scale should be understood as a variable from which the pixel luminance of a sensor pixel, covered by a color filter of the assigned filter color and exposed using light of each color value, can be derived. In this context, the derivation of pixel luminance can be understood as the direct reading of pixel luminance from each mapping table, or it can be understood as the calculation of pixel luminance considering the pixel luminance scale. For this calculation, the amount of light mapped onto each sensor pixel, particularly favorably, the light intensity and exposure time, can also be considered.

[0015] Of course, it is not necessary to repeat the step of creating a mapping table for each generation of composite image data using the simulation of the mosaic filter sensor according to the present invention. Advantageously, a mapping table suitable for a given series of mosaic filter sensors is created once, then stored, and possibly stored multiple times on spatially distributed storage media, distributed for recall and repeated use during subsequent simulations of each mosaic filter sensor.

[0016] For the generation of composite image data, it is assumed that the filter color of the color filter covering each sensor pixel of the simulated mosaic filter sensor is examined, and after this examination, the mapping table assigned to each filter color is referenced. Then, from this referenced mapping table, the color value of the light mapped onto each sensor pixel is obtained for the numerous sensor pixels of the simulated mosaic filter sensor, especially all sensor pixels, and the pixel brightness of each pixel is derived from the pixel brightness scale. In order to generate composite image data from the total pixel brightness derived from the sensor pixels, the pixel brightness scale assigned to each color value is read from the referenced mapping table for each sensor pixel.

[0017] Mapping tables for color conversion are well-known from conventional color correction techniques called color grading, and are used particularly for aesthetic reasons in movies and video games. Here, mapping tables are used to convert, for example, a color space that may be a computer-generated virtual environment to other color values ​​in the same color space in order to change the color atmosphere of the displayed scene.

[0018] In contrast, the mapping table of the method according to the present invention converts the color space of the virtual environment to the “color space” of the mosaic filter sensor. This conversion avoids technical and physical processes within the sensor, enabling a purely empirical but realistic simulation of the sensor, thus greatly facilitating the generation of realistic composite image data. Furthermore, camera simulations using the mapping table according to the present invention can be performed very quickly and with relatively little computational resources because they eliminate the need for physical simulation of the camera sensor. This makes it particularly suitable for real-time camera simulations (i.e., hard real-time camera simulations at the same working speed as a physical camera would work in a real environment) that are required for stimulating a physical object under test ("hardware in the loop").

[0019] A raster based on raster scanning in the original color space is advantageously designed as a regular grid of color values ​​in the original color space. The spacing of the grid points can correspond to the resolution limit of the original color space. If the spacing of the grid points is greater than the resolution limit of the original color space, it may be assumed that when reading out the pixel luminance scale assigned to a single color value, the pixel luminance scale is read from a grid point adjacent to that color value, or interpolation is performed between the pixel luminance scales assigned to at least two adjacent grid points in order to read the pixel luminance scale.

[0020] In one embodiment of the present invention, a mapping table is created based on measurements. The measurement process includes the steps of generating a single physical representation of each of several color values ​​in the original color space in the form of electromagnetic waves that can be captured using a camera, and further capturing each physical representation from the several physical representations using a physical camera equipped with a mosaic filter sensor. In other words, light for each color value (i.e., the electromagnetic spectrum corresponding to the color value) is generated for different color values ​​and used for exposure of at least one sensor pixel of the mosaic filter sensor. Here, for each color value, raw data generated by each color channel of the mosaic filter sensor is measured (each color channel corresponds to a filter color generated on the mosaic filter). These raw data, in particular, correspond to the electrical response of the sensor pixel, i.e., the voltage, current, or charge generated by the exposure of the sensor pixel.

[0021] Subsequently, a mapping table can be created based on the measurements obtained during the measurement process. To do this, it is necessary to convert the spectral colors of the measured electromagnetic waves to their corresponding colors in the virtual environment's original color space. The procedure for this will be revealed in the following description of the embodiment.

[0022] The pixel luminance scales stored in the mapping tables represent, in this embodiment, the actually measured sensor pixel response of the mosaic filter sensor installed in a physical camera. Thus, these mapping tables are designed to strictly simulate the electrical sensor pixel response of this mosaic filter sensor during the process of the method according to the present invention. This makes the mapping tables particularly suitable for testing subjects that utilize image data from cameras belonging to the same series as the physical camera in which the mosaic filter sensor used for measurement is installed.

[0023] To generate electromagnetic waves, different means exist, and in this case, the capture of the physical representation of different color values can basically be done serially or in parallel. For example, the electromagnetic waves of each color value to be measured can be done serially using a specific light source designed to emit light of the desired variable electromagnetic spectrum in the visible region or near-visible region of sufficiently high quality.

[0024] However, preferably, the generation of electromagnetic waves is done by illuminating at least one color card that can be obtained, for example, for camera calibration. An important feature of the color cards for the present invention is that they show a plurality of color values on one plane, either distributed on a single color card or on multiple color cards. In this case, for each color value, the reflection spectrum characteristic of the color value on the color card is clearly defined and known. The color card is irradiated with light of a clearly defined electromagnetic spectrum, and the color card irradiated in this way is mapped using a physical camera to measure the electrical response of the sensor pixels to the color values displayed on the color card.

[0025] Advantageously, the color values displayed on at least one color card already form a raster scan of the original color space. Therefore, the color values represented on at least one color card are converted into the original color space of the virtual environment and form a raster that covers the entire range of the original color space. The number of color values displayed on the color card is so large that the grid points of the raster are sufficiently dense, and the mapping table created based on the color card can be directly used for the implementation of the method according to the present invention without increasing in advance the number of pixel luminance scales stored in the mapping table, for example, by interpolation of the pixel luminance scales. This is particularly advantageous.

[0026] However, generally commercially available color cards do not meet the latter condition. Therefore, this method can include the step of complementing the measured values or pixel luminances obtained based on the colors mapped on the color card by an estimation method, particularly by interpolation or extrapolation of the measured values or pixel luminance scales. This is applied regardless of whether the color values mapped on the color card already form a raster scan of the original color space of the virtual environment. Most color cards map rather a small number of arbitrarily distributed color values, at least from the perspective of the present invention. In such cases, it is advantageous to perform subsequent complementation of the measured values in order to achieve a grid-like and sufficiently dense raster scan.

[0027] As already stated above, the electromagnetic spectrum of the light used for illuminating the color card must be clearly defined and known. The specific radiation distribution curve (electromagnetic spectrum) of the illumination is referred to as an illuminant in the industry, and several specific illuminants have been declared as so-called standard illuminants. For example, standard illuminant A (incandescent lamp), D65 (standard daylight), F1 to F6 (fluorescent tubes, different types), LED-RGB1 (white LED light), etc.

[0028] It is clear that the raw data generated by a digital camera depends on the illuminants of the camera environment, and it is useful to consider this dependency for a realistic simulation of the camera. Accordingly, in an advantageous embodiment of the present invention, it is assumed that illuminants defined by the radiant distribution curve of the illumination of a color card are captured, and a mapping table created using this illumination is assigned to the illuminants. Accordingly, the step of generating synthetic image data to mimic camera raw data from a physical mosaic filter sensor can be carried out under the assumption that the physical mosaic filter sensor is installed and that a physical camera, from which the generation of its image data is to be simulated, is located in an environment illuminated according to the illuminants. Preferably, the illuminants are designed to simulate typical illuminants that occur in the physical world, and in particular, standard illuminants may be used.

[0029] Many virtual environments store a large number of original, different (standard) illuminants, and are designed so that the rendering of the virtual environment dynamically adapts to the illuminant that is just selected. When using a virtual environment designed in this way, it is particularly advantageous to create and store multiple sets of mapping tables that adapt to the different illuminants stored by the virtual environment. These mapping tables can be created, for example, by performing the method steps for creating the mapping tables multiple times using color cards as described above, where a different illuminant is selected for the illumination of the color card in each run, and the created mapping table is assigned in each run to the illuminant selected for illumination in the most recent run. Then, to generate the composite image data, the mapping table to be assigned to the illuminant displayed by the virtual environment can be flexibly selected. During the generation of the composite image data, it is also possible to dynamically swap mapping tables, particularly depending on the situation and location, and flexibly use each mapping table to be used for rendering the virtual environment in the most recent virtual environment. For example, if an automated virtual vehicle moving within a virtual environment has a virtual camera, the mapping table can be flexibly changed depending on the situation, for example, what time of day and / or what weather conditions are displayed within the virtual environment, or whether the virtual vehicle is just in a shaded area, in direct sunlight, or in an underpass with artificial lighting.

[0030] In an alternative embodiment of the present invention, the step of creating a mapping table is performed by simulation instead of measurement. In the course of this simulation, a pixel luminance scale is calculated for each color value, where, for the calculation of the pixel luminance scale, at least one transmission spectrum of each color filter to which the mapping table to be calculated is assigned, and the quantum efficiency of the sensor pixel are taken into consideration, respectively. Advantageously, an illuminant is further considered for the calculation of the pixel luminance scale. This illuminant, for example, a standard illuminant, is characterized and defined by the radiation distribution curve of light that can illuminate the environment of the physical camera to be simulated. As described above in relation to the creation of a mapping table based on measurement, under the assumption that the physical camera is placed in an environment illuminated according to the illuminant, the mapping table thus calculated can also be assigned to the illuminant in order to carry out the generation of synthetic image data to mimic the raw camera data from the physical mosaic filter sensor as described.

[0031] In a further alternative embodiment of the present invention, the creation of the mapping table is based on a hybrid procedure that includes both the measurements described above and the simulation, or calculation, of the pixel brightness scale described above, in which case both the measurement results and the simulation results are taken into consideration when creating the mapping table.

[0032] The present invention also relates to a computer program product that includes means for generating a mapping table as described above and a composite image data as described above.

[0033] The drawings and their following descriptions illustrate exemplary embodiments of the present invention. [Brief explanation of the drawing]

[0034] [Figure 1] This diagram shows the functional configuration of a mosaic filter sensor. [Figure 2]This diagram shows the measurement structure and flow for creating a mapping table. [Figure 3] This is a flowchart for generating composite image data after creating a mapping table. [Modes for carrying out the invention]

[0035] Figure 1 schematically illustrates the operating principle of the mosaic filter sensor 2. This mosaic filter sensor 2 includes a matrix arrangement of sensor pixels 2a and a mosaic filter 2b designed as a periodic matrix mosaic of color filters 6. The mosaic filter 2b is positioned in front of the arrangement of sensor pixels 2a on the optical axis of the camera on which the mosaic filter 2b is installed, and covers the arrangement of sensor pixels 2a. The color filters 6 are arranged such that each sensor pixel is strictly covered by one color filter 6 from the arrangement of sensor pixels 2a.

[0036] For better visibility, the array of sensor pixels 2a shows only one subdivision consisting of 2x2 sensor pixels, and accordingly, the mosaic filter 2b also shows only one submosaic consisting of 2x2 color filters 6. An actual mosaic filter sensor 2 typically contains millions of sensor pixels and a corresponding number of color filters 6 on the mosaic filter 2b. The mosaic filter 2b is shown exemplarily as a Bayer filter, i.e., the submosaic consists of one blue color filter 6a, two green color filters 6b, and one red color filter 6c. This submosaic is arranged in multiple parallel rows along both axes on the mosaic filter 2b, so as a whole, a periodically repeating, i.e., identically repeating, mosaic of green, red, and blue color filters 6 is produced.

[0037] The camera optical system 8 illuminates the mosaic filter 2b with ambient light. An illustrative example shows how electromagnetic waves 4 in the green spectrum strike the displayed sub-mosaic on the mosaic filter sensor 2b. The color filters 6 of the mosaic filter 2b do not change the wavelength or spectral path of the electromagnetic waves 4, but reduce the amplitude of the electromagnetic waves 4 to different degrees depending on the transmission spectrum of the different color filters 6. While the electromagnetic waves 4 pass through the green color filter 6b with minimal loss, the blue color filter 6a and red color filter 6c absorb most of the energy of the green light passing through them, and the amplitude of the electromagnetic waves 4 is significantly reduced after passing through the blue color filter 6a and red color filter 6c.

[0038] Light passing through individual color filters 6 is projected onto sensor pixels by microlenses (not shown). Each individual sensor pixel is a photodetector that converts photons into electrons, thereby generating a current whose current intensity is proportional to the time photon density of the light projected onto each sensor pixel. Each sensor pixel includes an electrical capacitor (not shown), which is charged by the current and builds a voltage proportional to the current within a defined time (e.g., the camera's exposure time). These voltages are captured by the sensor system and transferred to the camera's image preprocessing unit 14, which reconstructs the image's color information from the voltage differences read from adjacent sensor pixels.

[0039] In this mapping, voltage U1 is read exemplarily from the blue color filter 6a, voltage U2 from the adjacent green color filter 6b, and voltage U3 from the adjacent red color filter 6c. Since the electromagnetic wave 4 is in the spectral green range, and only the green color filter is optimized for green light transmission, voltages U1 and U3 are significantly smaller than voltage U2. Under the reasonable assumption that the entire displayed submosaic is illuminated almost uniformly, the image preprocessor 14 can infer from the measured voltage difference that the electromagnetic wave 4 exposing the submosaic is green, and store the color information corresponding to that pixel.

[0040] The total voltages read during each exposure process—that is, voltages U1, U2, U3 and all further voltages read from individual sensor pixels—constitute camera raw data. From this data, the camera, in a further processing step, interprets the voltages as pixel brightness to generate an image displayable on a display device, for example, in JPG or Raw format. In other words, this camera raw data, in the illustrated example, is a matrix of voltages equal in dimensions to a matrix of sensor pixels. However, this raw data does not necessarily have to represent voltages. Depending on the camera model, this raw data can also represent currents or charges. Furthermore, the raw data does not have to directly represent voltages, currents, or charges, but rather can represent variables derived from them, especially variables derived by scaling.

[0041] In other words, for each sensor pixel, there is a linear dependence between the intensity or quantity of light (number of photons) projected onto the color filter during the exposure process and the voltages U1, U2, U3… read from the sensor pixels covered by each color filter 6 during the same exposure process. The relationship between the quantity of light and the resulting voltage is essentially determined by two variables: the wavelength-dependent transmittance of the color filter 6, the so-called transmission spectrum, and the similarly wavelength-dependent quantum efficiency of the sensor pixels. Transmittance is the ratio of the light component passing through the color filter, i.e., the light intensity before and after passing through the color filter, and quantum efficiency is the degree of efficiency of the sensor pixel in converting light into electric current. The pixel brightness scale stored in the mapping table represents the linear dependence of the voltages U1, U2, U3… on the light intensity or quantity of light mapped to each sensor pixel for each clearly defined color of the electromagnetic wave 4.

[0042] The upper part of the mapping in Figure 2 shows a structure for creating a mapping table based on measurements. This measurement structure includes a color card 12 and a light source 10 designed and positioned to illuminate the color card 12 with an illuminant that uniformly and clearly defines the color card 12. The color card 12 is provided and designed for camera calibration. This color card 12 includes, exemplarily, 16 differently colored color fields, each of which has a clearly defined color value with a clearly defined reflectance spectrum. This means that each individual color field emits electromagnetic waves 4 having a clearly defined electromagnetic spectrum that represents the physical representation of the color value of the respective color field. The camera 16 is positioned for mapping the color card 12, so that the lens 8 maps the electromagnetic waves 4 emitted from each color field onto a portion of the mosaic filter sensor 2. As described in the description of Figure 1, the mosaic filter sensor 2 generates raw data U as a matrix of pixel luminances (in this case, voltages).

[0043] As an introductory step to creating the mapping table 20, six three-dimensional arrays 18,20 are assigned to the computer system. Three of these arrays represent the CIE color space. These CIE color spaces are a class of tristimulus color spaces optimized to reflect the color perception of the human eye based on the excitation of three types of biological photoreceptors (cones). Three further arrays represent the intrinsic color space of the virtual environment, exemplaryly the sRGB color space. This sRGB color space belongs to the RGB color space, a class of tristimulus color spaces, which are optimized to stimulate the cones of the human eye to produce the desired color perception for the observer through the driving control of red, green, and blue light elements within the pixels of the display device. Common graphics engines for generating virtual environments typically render the virtual environment based on the RGB color space.

[0044] In general, the original color space of a virtual environment should be understood to mean, in particular, the color space on which the virtual environment is rendered, or a color space on which the virtual environment is rendered, and which is at least sufficiently similar. Advantageously, both the color space of the mapping table 20 selected as the "original color space" and the color space used for rendering the virtual environment belong to at least the same class of color spaces. Exemplarily, in one embodiment of the present invention, it is possible to create a mapping table 20 for the sRGB color space (i.e., consider the sRGB color space as the "original color space"), and although the virtual environment is rendered based on a specific RGB color space, this color space is very similar to the sRGB color space, so the composite image data still mimics the raw camera data quite well.

[0045] The mapping shows how the electromagnetic wave 4, already shown in Figure 1, is radiated from the color field of the color card 12, and thus represents the physical representation of the color values ​​shown within the same color field. Raw data U is read out to create the mapping table 20. As already explained above, raw data U is a matrix of voltages, where each matrix entry represents the single, clearly defined sensor pixel of the mosaic filter sensor 2. Here, four sensor pixels are determined that are located within the partial surface of the mosaic filter sensor 2, which is covered by the sub-mosaic of the mosaic filter 2b and exposed by the electromagnetic wave 4. Voltages U1, U2, and U3 are read out directly from the matrix entries of raw data U representing the determined sensor pixels.

[0046] The first array 18a, the second array 18b, and the third array 18c each represent a CIE color space. Rather than representing different CIE color spaces, the first array 18a, the second array 18b, and the third array 18c are three independent instances of the same CIE color space. The color values ​​of electromagnetic wave 4, i.e., the electromagnetic spectrum, are assigned unique CIE coordinates in the CIE color space, which are marked by a cross within the mapping. The first array 18a stores the voltage U1 at this coordinate. This first array 18a is assigned to the blue color filter 6a of the mosaic filter 2b. The second array 18b stores the voltage U2 at this coordinate. This second array 18b is assigned to the green color filter 6b of the mosaic filter 2b. The third array stores the voltage U3 at this coordinate. This third array 18c is assigned to the red color filter 6c of the mosaic filter 2b.

[0047] Three previously undescribed arrays representing the sRGB color space (or more generally: the original color space) are mapping tables 20. The first mapping table 20a is assigned to the blue color filter 6a of the mosaic filter sensor 2b. CIE coordinates from the CIE color space are assigned unique sRGB coordinates in the sRGB color space, and these coordinates are also marked by a cross. The assigned sRGB coordinates represent sRGB color values ​​that produce a similar color impression to the previously determined CIE coordinate color value or electromagnetic wave 4 in the eyes of a standardized observer, and can be obtained from the CIE coordinates using a known, generally accessible conversion formula. This first mapping table 20a stores a first pixel luminance scale P1 in sRGB coordinates. The first pixel luminance scale is obtained by multiplying the voltage U1 by a scaling factor s to normalize the voltage U1 to the unit luminance and unit exposure time of the electromagnetic wave 4.

[0048] The second mapping table 20b is assigned to the green color filter 6b of the mosaic filter sensor 2b. This second mapping table 20b stores the second pixel brightness scale P2 in sRGB coordinates. This second pixel brightness scale P2 is obtained by multiplying the voltage U2 by the scaling factor s.

[0049] The third mapping table 20c is assigned to the red color filter 6c of the mosaic filter sensor 2b. This third mapping table 20c stores the third pixel brightness scale P3 in sRGB coordinates. This third pixel brightness scale P3 is obtained by multiplying the voltage U3 by the scaling factor s.

[0050] The process described is repeated similarly for each color value displayed on the color card 12. As a result, in the example of the color card 12 shown, 16 pixel luminance scales will be stored in each mapping table 20 in 16 different sRGB coordinates, corresponding to the 16 mapped color values.

[0051] If necessary, the pixel luminance scale stored in the mapping table 20 can be supplemented by mathematical estimation methods, such as interpolation and / or extrapolation, if it does not yet represent a raster scan of the sRGB color space or if the distance between grid points in the raster is too large. Since these types of methods, especially extrapolation, can only produce an approximate pixel luminance scale at best, the color card 12 is advantageously designed so that the pixel luminance scale stored in the mapping table 20 by measurement already forms a scan, particularly a raster scan, that almost completely covers the original color space. This color card is particularly advantageous in that the pixel luminance scale stored in the mapping table 20 by measurement, particularly through the formation of a raster scan, already densely covers the original color space to the extent that supplementation of the pixel luminance scale by interpolation is unnecessary.

[0052] The completed mapping table 20 is obtained using the function R3 →R, and by specifying the pixel brightness for each sRGB color value, the electrical response of the sensor pixel can be determined by a simple calculation using several dynamic variables, particularly the brightness or intensity of electromagnetic wave 4 and the exposure time. This corresponds to the electrical response of the corresponding sensor pixel in the physical camera 16 when the sensor pixel is exposed with light of the electromagnetic spectrum that produces the same color impression to a human observer as each sRGB color value.

[0053] The three mapping tables 20 relate to illuminants emitted from the light source 10. Other illuminants generate other spectra of electromagnetic waves emitted from the color field of the color card 12. The process described above in relation to Figure 2 can be repeated multiple times using different light sources 10 emitting different illuminants to create and store mapping tables 20 for different illuminants.

[0054] Since the processes responsible for converting light into voltage (transmission by the color filter 6 and light detection by the sensor pixels) are known and quantifiable, the pixel brightness scale in the mapping table 20 can also be calculated by computer simulation of the aforementioned processes.

[0055] The flowchart in Figure 3 provides an overview of how the mapping table 20 is used in the simulation of the mosaic filter sensor 2.

[0056] In the first simulation step 22, the exposure of the mosaic filter sensor 2 is calculated based on the simulation of the camera optical system 8, the latest state of the virtual environment including the latest position and spatial orientation of the virtual camera, and the latest state of a number of static and dynamic graphic objects in the virtual environment. After this exposure calculation is complete, for each sensor pixel of the virtual mosaic filter sensor 2, the exposure of each sensor pixel is calculated, which includes the sRGB color value and the luminance or light quantity of the electromagnetic wave 4 that exposes each sensor pixel. In order to store the composite image data, in the second simulation step 24, a matrix is ​​allocated for each sensor pixel of the virtual mosaic filter sensor, containing entries for storing the electrical response of each sensor pixel.

[0057] The electrical responses of individual sensor pixels of the virtual mosaic filter sensor 2 are subsequently calculated serially. In the third simulation step 26, the process jumps to the first sensor pixel, i.e., the first sensor pixel is selected for the calculation of its electrical response. In the fourth simulation step 28, the sRGB color value and luminance of the light exposing the first sensor pixel are read. The two values, luminance and sRGB color value, have already been calculated during the sum calculation in the first simulation step 22 and are prepared for reading in the fourth simulation step.

[0058] In the fifth simulation step 30, the filter color of the sensor pixels is determined. In the virtual camera, one filter color is assigned to each sensor pixel of the virtual mosaic filter sensor 2. The filter color assigned to the sensor pixels corresponds to the filter color of the color filter within the mosaic of the physical mosaic filter 2b that covers the sensor pixels of the physical mosaic filter sensor 2.

[0059] In the sixth simulation step 32, the mapping table 20 assigned to the filter color obtained in the fifth simulation step 30 is referenced, or rather, called. In the seventh simulation step 34, the pixel brightness scale assigned to the sRGB color values ​​read in the fourth simulation step 28 is read from the mapping table 20 referenced in the sixth simulation step 32.

[0060] In the eighth simulation step 36, the electrical response of the sensor pixel is calculated, taking into account the pixel brightness scale read in the seventh simulation step 34, the exposure time, and the brightness read in the fourth simulation step 28. In a specific example, this electrical response might be, for example, a voltage. Assuming that the filter color assigned to the sensor pixel is blue and the read pixel brightness scale is P1 (see mapping in Figure 2), the electrical response is, for example, given by the following equation: U1=(I / I U )·(E / E U )P1 It can be obtained from, where I is luminance, I U E is the unit brightness, E is the exposure time, E U is the unit exposure time. The calculated electrical response is written in the ninth simulation step 38 to an entry in the matrix allocated in the second simulation step 24, which is provided for storing the electrical response of the sensor pixel.

[0061] The electrical responses stored in the matrix constitute the composite image data. After the completion of the ninth simulation step 38, a check is performed to determine whether all sensor pixels have been processed, i.e., whether an electrical response has been calculated for each sensor pixel and stored in the matrix. If this check is positive, the matrix containing the composite image data is released for further processing. This release should be understood as, in most cases, the composite image data stored in the matrix being transferred logically or physically to a downstream instance, which then reads the composite image data and processes it further. The downstream instance may, in particular, be a test subject designed to process the composite image data.

[0062] If the test is negative, that is, if not all sensor pixels have been processed yet, the process skips to the next sensor pixel and then feeds back to the fourth processing step 28 to calculate the electrical response for the currently selected sensor pixel in the same way and store it in a matrix.

[0063] In particular, if the subject expects a continuous stream of image data, the simulation process schematically shown in Figure 3 can be repeatedly performed in synchronization with, for example, the sequence of exposure processes in a virtual camera, where each execution of the illustrated simulation step generates a burst of composite image data, and the stream of image data is designed as a continuous time sequence of bursts.

Claims

1. A computer-implemented method for generating composite image data to mimic camera raw data (U) from, for example, a Bayer sensor, a mosaic filter sensor (2), The mosaic filter sensor (2) includes an array of sensor pixels (2a), and further includes a mosaic filter (2b), such as a Bayer filter, designed as a periodic mosaic of color filters (6) of different colors, arranged such that each sensor pixel consisting of the array of sensor pixels (2a) is strictly covered by one color filter (6). The above method comprises the following steps, namely, The steps include positioning and spatially oriented a virtual camera in a three-dimensional virtual environment on a computer system, A camera optical system (8) simulates the mapping of colored light onto the sensor pixels of a mosaic filter sensor (2) installed in the virtual camera, comprising the steps of defining luminance and color values ​​from the original color space of the virtual environment for the light mapped onto each sensor pixel, depending on the position and spatial orientation of the virtual camera; A method comprising the steps of generating composite image data by using a simulation of a mosaic filter sensor (2) installed in the virtual camera to derive the pixel brightness of the sensor pixels depending on the brightness and color value of the light (4) mapped onto each sensor pixel, The aforementioned method, The steps include creating a mapping table (20) to be assigned to each filter color that occurs within the mosaic filter (2b), wherein at least one mapping table is assigned to each filter color, and exactly one filter color is assigned to each mapping table. The steps include configuring the mapping table (20) such that, in each mapping table (20), one pixel brightness scale (P1, P2, P3) is assigned to each of the many color values ​​from the raster scan of the original color space of the virtual environment, and the pixel brightness of the sensor pixels covered by the assigned filter color color filter (6) and exposed using the light of each color value can be derived from the pixel brightness scale (P1, P2, P3), For the sensor pixels of the simulated mosaic filter sensor (2b), the steps include: checking the filter color of the color filter (6) covering each sensor pixel and referring to the mapping table (20) assigned to each filter color; The steps include: using the reading of the pixel brightness scales (P1, P2, P3) assigned to each color value from the respective mapping table (20) to be referenced, to derive the pixel brightness of each sensor pixel depending on the color value of the light mapped onto each sensor pixel; A method characterized by including the following.

2. The raster that forms the basis of the aforementioned raster scan is designed as a regular grid of color values ​​in the original color space. The method according to claim 1.

3. In deriving the pixel brightness, the amount of light mapped onto each sensor pixel, particularly the light intensity and exposure time, are taken into consideration. The method according to claim 1 or 2.

4. The step of creating the aforementioned mapping table (20) is the following method step, namely, A step of generating one physical representation of each of the multiple color values ​​of the original color space in the form of electromagnetic waves (4) that can be captured using a camera, The steps include capturing each physical representation (4) from a plurality of physical representations (4) using a physical camera (16) on which the mosaic filter sensor (2) is installed, A step of measuring raw data (U) generated by each color channel of the mosaic filter sensor (2) by capturing the respective physical representation (4) of each color value, wherein each color channel corresponds to the filter color generated on the mosaic filter (2b), The steps include creating a mapping table (20) based on the measured values ​​(U1, U2, U3) obtained during the measurement, thereby determining that the pixel brightness scales (P1, P2, P3) stored in the mapping table (20) represent the sensor pixel response of a mosaic filter sensor installed in a physical camera, including, The method according to any one of claims 1 to 3.

5. To generate the electromagnetic waves (4), at least one color card (12) designed in particular for camera calibration is illuminated. The method according to claim 4.

6. The color values ​​displayed on the at least one color card (12) form a raster scan of the original color space. The method according to claim 5.

7. The step of creating the mapping table (20) includes interpolation or extrapolation, in particular, by estimation methods of measured values ​​(U1, U2, U3) or pixel brightness scales (P1, P2, P3) obtained based on the color values ​​mapped on the color card (12). The method according to claim 5 or 6.

8. The above method comprises the following steps, namely, A step of capturing the illuminant defined by the radiation distribution curve of the illumination on the color card (12), The steps include assigning the mapping table (20) to the illuminant, The steps include generating synthetic image data to mimic the raw camera data (U) from the physical mosaic filter sensor (2), under the assumption that the physical camera (16) is located in an environment illuminated according to the illuminant, Includes, The illuminant is specifically a standard illuminant and / or is designed to simulate a typical illuminant occurring in the physical world. The method according to any one of claims 5 to 7.

9. To create each mapping table (20), the pixel brightness scales (P1, P2, P3) are calculated for each color value, taking into account the transmission spectrum of the respective color filter (6) and the quantum efficiency of the sensor pixel. The method according to any one of claims 1 to 3.

10. The pixel brightness (U) is calculated considering the illuminant defined by the radiation distribution curve of the light illuminating the environment of the physical camera (16). The above method comprises the following steps, namely, The steps include assigning the mapping table (20) to the illuminant, The steps include generating synthetic image data to mimic the raw camera data (U) from the physical mosaic filter sensor (2), under the assumption that the physical camera (16) is located in an environment illuminated according to the illuminant, Includes, The illuminant is specifically a standard illuminant and / or is designed to simulate a typical illuminant occurring in the physical world. The method according to claim 9.

11. The pixel brightness (U1, U2, U3) is generated by the simulated sensor pixels and represents the simulated voltage and / or simulated current and / or simulated charge. The composite image data is generated from the total of the simulated voltage and / or current and / or charge. The method according to any one of claims 1 to 10.

12. A computer program product for generating composite image data to mimic camera raw data (U) from a mosaic filter sensor (2), for example, a Bayer sensor, The mosaic filter sensor (2) includes an array of sensor pixels (2a), and further includes a mosaic filter (2b), such as a Bayer filter, designed as a periodic mosaic of color filters (6) of different colors, arranged such that each sensor pixel consisting of the array of sensor pixels (2a) is strictly covered by one color filter (6). The aforementioned computer program product utilizes the following means, namely, A means for positioning and spatially oriented a virtual camera in a three-dimensional virtual environment on a computer system, A means for simulating the mapping of colored light (4) onto sensor pixels of a mosaic filter sensor (2) installed in a virtual camera using a camera optical system (8) installed in the virtual camera, wherein the means defines the luminance and color values ​​from the original color space of the virtual environment for the light (4) mapped onto each sensor pixel, depending on the position and spatial orientation of the virtual camera. A means for generating composite image data by deriving the pixel brightness of a sensor pixel based on the brightness and color value of the light mapped onto each sensor pixel, using a simulation of a mosaic filter sensor (2) installed in the virtual camera, In computer program products including, A mapping table (20) is stored for each filter color generated within the mosaic filter (2b), thereby assigning at least one mapping table to each filter color, and ensuring that exactly one filter color is assigned to each mapping table. Each of the mapping tables (20) is designed such that, in each mapping table (20), one pixel brightness scale (P1, P2, P3) is assigned to each of the many color values ​​from the raster scan of the original color space of the virtual environment, and the pixel brightness (U1, U2, U3) of the sensor pixels that are covered by the color filter (6) of the assigned filter color and exposed using the light (4) of each color value can be derived from the pixel brightness scale (P1, P2, P3). The computer program product generates composite image data for the sensor pixels of the simulated mosaic filter sensor (2), Each of the filter colors of the color filter (6) covering each sensor pixel is inspected. Refer to the mapping table (20) assigned to each filter color, Using the readout of the pixel brightness scales (P1, P2, P3) assigned to each color value, the pixel brightness of each sensor pixel is derived depending on the color value of the light mapped onto the sensor pixel. A computer program product characterized by being configured in such a way.

13. The computer program product is configured to take into account the amount of light, particularly the light intensity and exposure time, mapped onto each sensor pixel when deriving the pixel brightness. The computer program product according to claim 12.

14. The mapping table (20) is designed to generate synthetic image data to mimic the raw camera data from the physical mosaic filter sensor, with each illuminant defined by the radiation distribution curve assigned to a standard illuminant and / or simulation of a typical illuminant occurring in the physical world, and further assuming that the physical camera is placed in an environment illuminated according to the illuminants. The computer program product according to claim 12 or 13.

15. The mapping table is designed to take into account the transmission spectrum of the color filter assigned to each pixel based on the pixel brightness scale, and the quantum efficiency of the sensor pixel. A computer program product according to any one of claims 12 to 14.

Images