METHOD FOR SIMULATING AN OPTICAL IMAGE

Description

[0001] The invention relates to a method for generating an image and a computer program product.

[0002] For photographic image capture, especially the recording of image sequences, image capture systems are used that contain a camera module and a lens. The camera module includes an image sensor, which can be an electronic image sensor or a light-sensitive chemical film. The lens typically contains a variety of optical elements such as optical lenses, apertures, and / or mirrors. Such lenses often include a variable aperture, also known as a diaphragm. Optical imaging occurs when light from the object being photographed passes through these elements and onto the image sensor or film. Such optical imaging is usually subject to optical aberrations, such as defocus, coma, spherical aberration, geometric distortion, or radial dimming due to vignetting.These so-called geometric aberrations are often dependent on the focus setting, the aperture of the lens, and / or other parameters, such as the focal length. Taken together, these aberrations result in a characteristic geometric imaging behavior for each given lens due to its specific optical design. The effect is particularly noticeable when imaging out-of-focus features, as these appear blurred. The further the feature is from the ideally focused area, and the wider the aperture of the lens, the greater the blurriness.

[0003] The characteristic with which such out-of-focus object points are rendered is commonly called bokeh. For example, lenses are known in which out-of-focus object points that lie significantly off-axis are rendered as horizontally stretched and slightly curved blurred areas. This can be the case, for example, with lenses of the so-called Biotar type, as described, for instance, in US 1,786,916.

[0004] Mirror-type lenses, one example of which is described in DD 263604 A1, exhibit a completely different characteristic. Such lenses typically project defocused object points onto ring-shaped areas of blur. The bokeh produced with these lenses, especially with objects containing pronounced highlights, often shows a distinctly busy ring pattern.

[0005] In the production of feature films and similar media, moving photographic footage is often combined with virtual images and / or virtual image sequences. This process is frequently called rendering. The result of such rendering is a digitally generated image or image sequence. This image or image sequence is generated by calculation from a mathematical model of the scene to be depicted. This scene model contains, among other things, information about properties such as the shape, color, and optical surface finish of the objects to be modeled within the scene. This scene model is then combined with a mathematical lighting model, which includes the location, type, color, and direction of each light source.To actually calculate the image, at least the position in the scene, the shooting direction, and the field of view of a virtual camera are required. The computational generation of the image or image sequence of the modeled scene with the virtual camera, and the individual steps of this computational generation, are referred to here as simulation. Often, a simple pinhole camera principle (camera obscura) is used for this calculation, which is based on a simple geometric projection according to the mathematical principle of the intercept theorem. Although very simple, this forms the basis for a simulation of an ideal photographic image, as no geometric image errors occur.

[0006] The process of blending rendered sequences of virtual images with real-world footage is a common practice in real-time 3D computer graphics and visual effects in cinematography. The resulting effects are known as VFX (visual effects). This blending of real and virtual images is also known as augmentation. Augmentation is often part of compositing, a step in the post-production of a film where images from different sources are combined.Especially when combining photographically acquired images or image sequences of real scenes with virtually generated content, it is desirable that this virtually generated content, in a final image or image sequence, exhibits no or only weakly perceptible differences from photographically acquired images or image sequences of real scenes, thus conveying the most photorealistic impression possible. This is achieved by generating the digitally created content with boundary conditions that closely approximate those of real-world image capture. A correct simulation of the real imaging chain includes, on the one hand, the boundary conditions for the mathematical description of the scene and the lighting, and on the other hand, the boundary conditions for the capture process, i.e., in particular, the optical imaging onto the image sensor. These are not provided by a simple geometric camera model, i.e., a simulation of a pinhole camera.

[0007] Another important application for rendered images or image sequences is fully animated images or sequences, meaning those where the entire information content originates from a virtual scene and is not mixed with any real-world images or sequences. This includes the field of animated film production and virtual reality, which is used, for example, in computer games, but also in training simulators such as flight simulators for pilot training or train simulators for driver training. Especially in animated films and computer games, it is desirable to create a certain photorealistic impression of the scene. Furthermore, this method can be used, for example, in medical imaging to combine real medical images with computer-generated images.

[0008] Typically, rendered images are generated using techniques such as ray tracing. Ray tracing calculates the optical ray paths from the light sources to the object and from the object to the image sensor. In practice, this calculation is often performed by tracing the rays backward from the image sensor to the object, i.e., in the opposite direction to the propagation of the physical rays within the optical device itself. Of course, calculating along the direction of light propagation is equally possible.

[0009] The aforementioned simple pinhole camera model is frequently used to simulate the optical imaging onto the image sensor in such ray-tracing calculations. One of its advantages is that it produces so-called ideal images, which contain no geometric aberrations. In this model, all rays of the ray-tracing calculation are calculated by passing through a point in front of the simulated image sensor. The distance of this point from the sensor is chosen precisely so that the geometrically achieved field of view corresponds to the field of view of the camera system being simulated. This point in the simulation corresponds to the center of a very small pinhole in a real pinhole camera. Besides the simplicity of the mathematical model, the advantage of simulating a pinhole camera lies in the high speed at which such calculations can be performed.In particular, dedicated graphics processing units (GPUs) can perform such rendering simulations very quickly and in parallel for many pixels. This parallelization results in a significant speed increase, even enabling the calculation of entire complex virtual scenes in real time.

[0010] However, by its very nature, this process does not account for the image errors or characteristics described above, which occur when taking a real photograph with a real camera and lens. As a result, the simulated image often differs significantly from the impression one would get when viewing comparable, real images. This can be particularly problematic when rendering and combining real images or image sequences. To compensate for this, two-dimensional image processing techniques and artistic methods are frequently employed. For example, blurring is added to the idealized images obtained through pinhole camera simulation, aiming to approximate the blurring produced by the real camera as closely as possible.Correcting or introducing geometric distortions and brightness falloff in the form of vignetting is another effect that can be easily factored in. Such corrections are generally adjusted phenomenologically so that the visual impression of the simulated images comes as close as possible to that of images obtained from real photographs.

[0011] It is therefore desirable to replicate the properties of a real camera lens used to capture real images in the simulation, including as many of its characteristics as possible. This is important both to achieve the most accurate possible match of characteristics such as bokeh and to enable a straightforward process that requires no empirical experimentation with parameters or even artistic input, but is deterministically comprehensible and thus predictable. Of course, it is fundamentally possible to apply ray tracing to the entire optical part of the image acquisition system. In fact, such ray tracing has been widely used in the design and optimization of optical systems for many years. However, its use in VFX or 3D computer graphics is uncommon, as these techniques involve a large number of calculations and are therefore very resource- and time-intensive, and thus expensive.The GPUs mentioned, which are usually used for rendering, are not designed for such complex rendering simulations.

[0012] The significant computational effort in a ray-tracing simulation using a realistic model of the simulated lens arises from the need to calculate numerous ray deflections within the lens according to the laws of refraction in geometric optics. Such deflections occur at every interface between optical material and air, as well as at interfaces between different optical materials, for example, in cemented elements. These ray calculations must be performed multiple times for each pixel, using different angles of incidence on the image sensor, to ensure that the transparent area of ​​the aperture diaphragm of the simulated lens is accurately scanned. Generating high-quality images may require approximately 1000 or more calculated rays per pixel.Ray tracing typically requires such calculations to be performed for different wavelengths of light, since optical materials exhibit wavelength-dependent refractive indices and thus wavelength-dependent beam deviations due to their dispersion properties. Furthermore, there is beam absorption within the lens, known as vignetting, for example at the aperture diaphragm, but also at lens mounts. These beam absorptions must also be captured by the simulation, as they contribute significantly to the bokeh and vignetting of a lens.

[0013] Ray tracing calculations must be performed multiple times for each pixel of the image sensor in such a way that the light rays incident on each sensor pixel penetrate the exit pupil of the lens being simulated as a sufficiently dense bundle. Only in this way can the imaging properties of a lens being simulated, caused by geometric effects, be realistically simulated.

[0014] Besides the high computational effort, a disadvantage of ray-traced lens simulation, particularly in the field of VFX, is that it requires precise knowledge of the lens's optical design. This includes parameters such as lens materials, lens radii, radii of curvature, aspheric parameters, and / or distances between the lens's optical elements, especially if these distances are variable. This means that a user of the ray-tracing method needs knowledge of all these parameters. Conversely, without knowledge of these parameters, simulating the image of a lens using ray tracing is impossible. To avoid having to disclose details about the lens's design, it is desirable to have a way for third parties to perform realistic simulations of the lens's image without requiring any detailed information about its construction.

[0015] A well-known method for reducing computational effort is the one described by Schrade et al. in E. Schrade, J. Hanika, C. Dachsbacher, Sparse high-degree polynomials for wide-angle lenses, Eurographics Symposium on Rendering 2016, Vol. 35, No. 4. This method uses an abstract lens model in which the coefficients of several polynomials of a transformation formula are chosen such that, given the position and direction of a ray incident on the lens to be simulated, the approximate position and direction of the corresponding ray exiting the lens is output. Such a lens model or simulated lens is abstract in that it contains no information about the physical structure of the modeled or simulated lens, which is why its structure cannot be deduced or calculated from the model's coefficients.It only requires the rule for how rays are deflected during their propagation through the lens. In this way, Schrade et al. describe an abstracted simulation of the imaging process of a lens in an optical imaging system.

[0016] The simulation of the image formed by the simulated lens is performed by applying the transformation formula with suitable coefficients to the simulated location and direction of a simulated light ray emitted towards the image sensor. The result of the simulation is the simulated location and direction of a simulated light ray incident from the object. The simulation method can also be applied without limitation to a ray calculation in the reverse direction, yielding the location and direction of a simulated emitted light ray. It goes without saying that a correspondingly inverted transformation formula must be used in this case.

[0017] The simulation method by Schrade et al. assumes a fixed focus position and a fixed focal length for the simulated lens. If either of these parameters is changed, the entire abstract lens model, represented by the coefficients of the transformation formula used, must be recalculated. This process is computationally intensive and therefore time-consuming.

[0018] In the simulation method by Schrade et al., it is further assumed that the beam incident on the lens strikes the first lens surface, and that the lens's effect on the beam begins there. Thus, the simulation method described there reveals information about the simulated lens to the user, namely the shape of the front surface of the simulated lens. A complete abstraction of the lens is therefore not achieved.

[0019] Regarding the focusing of the simulated lens, the method by Schrade et al. uses the displacement of the simulated image sensor's plane relative to the simulated lens. This type of focusing is only implemented in simple lenses, such as the aforementioned Biotar. Modern lenses often use the displacement of one or more lenses relative to other optical elements of the lens for focusing. Lenses can be grouped together and shifted as a single unit. A solution for the rapid simulation of such complex lenses cannot be derived from the work of Schrade et al.

[0020] A variable focal length lens, also called a pancratic lens, varifocal lens, or zoom lens, also utilizes optical lenses and groups of optical lenses that can be moved relative to each other to achieve changes in focal length. If the theory of Schrade et al. is to be used to simulate such zoom lenses, a separate transformation formula with suitable coefficients must be provided for each simulated, movable assembly. Furthermore, the displacement of the assemblies must be specified. Therefore, a complete abstraction of the effect of the simulated lens is not possible for zoom lenses. Moreover, a simulation according to the theory of Schrade et al. allows conclusions to be drawn about the size of the aperture diaphragm and its penetration of rays depending on position and direction in image space, from which sensitive properties of the lens can be derived.Furthermore, it is not shown how to deal with the fact that rays are shadowed, i.e. absorbed, at different positions in a lens, for example apertures or lens mount parts, and therefore do not contribute to image formation.

[0021] It is therefore an object of the invention to further develop the simulation method taught in the prior art in such a way that the set of coefficients describing the lens is completely abstracted from the lens's structure and therefore does not allow any conclusions to be drawn about the structure of the simulated lens. It is also an object of the invention to further develop the simulation method taught in the prior art in such a way that changes to the focus, focal length, or one or more continuous parameters of the simulated lens do not require computationally intensive recalculations of the coefficients of the transformation formula.

[0022] The first problem is solved by the features of claim 1. The second problem is solved by the features of claim 2. Further advantageous embodiments are described by the features of the dependent claims. Further features of the invention will become apparent from the following description, the following claims, and / or the accompanying figures.

[0023] The invention and its embodiments are described with reference to the following figures: Fig. 1 schematically illustrates the principle of image capture. Fig. 2 schematically illustrates the principle of the simulation according to the invention. Fig. 3 schematically shows the effect of the optical imaging of a lens. Fig. 4 schematically shows the structure of the first data set. Fig. 5 schematically shows the structure of the second data set. Fig. 6 schematically illustrates the principle of simulating a beam path. Fig. 7schematically shows the optical structure of an example lens to be simulated. Fig. 8 schematically shows the positioning of pinhole camera images relative to the entrance pupil. Fig. 9 schematically shows an arrangement of pinhole camera images relative to the entrance pupil in the form of a Fibonacci spiral. Fig. 10 schematically shows a random arrangement of pinhole camera images relative to the entrance pupil. Fig. 11 schematically shows the exemplary setup of an image simulation using an abstract staircase object and a corresponding pinhole camera image of this staircase object. Fig. 12 schematically shows images of an abstract staircase object simulated according to the invention with one, a few and many pinhole camera images used for this purpose (5700, 5800, 5900), as well as positioning of the pinhole cameras relative to the entrance pupil. Description of the solution according to the invention

[0024] The principle of image capture is schematically represented in Figure 1As shown. A given lens 1, to be simulated for photographic image acquisition, has lenses 2 and often an adjustable aperture 3. Each lens has a specific radius and is held by an opaque mount 4. The image sensor 100, also referred to as the sensor in the following, is mounted at a position dependent on the specific optical design of the lens 1 to be simulated. The sensor 100 to be simulated can consist of a large number of light detectors, usually arranged in a grid, called pixels 5. Each of these pixels captures a picture point during the image acquisition. The image is optically formed at the location of the sensor 100, and in a real camera system, the incident light intensity is electronically or chemically detected at each pixel or picture point: the capture of a picture point.The sensor can be equipped with a grid-shaped color filter 101, for example arranged in a so-called Bayer pattern, so that each pixel 5 can detect only one specific color. It is also possible for each pixel 5 to detect all three primary colors, for example, red, green, and blue. In the simulation, this corresponds to the registration and storage in the processing computer system of at least the calculated intensity, and if necessary, also the color of the simulated incident light beam. Storage preferably takes place in so-called random access memory (RAM), but can also occur, for example, in so-called flash memory or on a magnetic disk. Light rays emitted from a point in a scene to be imaged, and which potentially contribute to the image, pass through the lens and are refracted at the lens interfaces.Their direction is changed according to the law of refraction in optics. A portion of these light rays falls on parts of the lens mount 4 of the lens 1 and is absorbed, thus not contributing to the image formation.

[0025] The absorption of light rays by parts of the lens is called shadowing and causes vignetting. Another portion of the light rays may fall upon the aperture diaphragm 3, where they are absorbed and do not contribute to the image formation. A further portion of the light rays passes through the lens and strikes the image sensor 100. These light rays do contribute to the image formation.

[0026] The lens 1 to be simulated can have one or more adjustable parameters. These can include one or more of the following: Aperture diaphragm: Lenses often incorporate at least one approximately circular aperture whose optically transparent diameter can be changed by the user. Iris diaphragms are frequently used. For focusing, a lens can have one or more lenses that are movable along the optical axis relative to the image plane and preferably also relative to other optical elements. For variable focal length adjustment, a lens can have one or more lenses that are movable along the optical axis relative to the image plane and preferably also relative to other optical elements. Instead of the focal length, the lens magnification can also be used as a continuous input variable.Magnification is a scaling factor derived from the quotient of the size of an image of an object on the sensor or film plane and the size of the imaged, focused object. It has been shown that choosing magnification as an input variable is particularly well-suited for the method according to the invention. For variable adjustment of the field curvature, a lens can have one or more lenses that are movable along the optical axis relative to the image acquisition plane and preferably also relative to other optical elements. For variable manipulation of a wavefront of the light passing through the lens, a lens can have movable freeform elements. These can be, for example, one or two so-called Alvarez elements that are movable perpendicular to the optical axis.

[0027] Such adjustable parameters on the lens 1 to be simulated correspond in a lens simulation to preferably continuously adjustable input variables. The known teaching for simulating optical imaging by lenses by Schrade et al. serves as the starting point for the method according to the invention. There, the continuously adjustable input variables are defined before the calculation of the coefficients modeling the lens and determine their values. If these values ​​are to be changed, the coefficients modeling the lens must be recalculated in this simulation method.

[0028] The simulation of the optical imaging of a lens to be simulated according to the invention is based on a transformation rule 900. This is shown schematically in Figure 2shown. This transformation rule 900 processes data from one or more input data sets 1000 in a calculation 4000 performed by a computer 300 and transforms them into an output data set 2000. The transformation rule 900 can include, among other things, a polynomial. The transformation rule 900 can include, among other things, a spline function. The in Figure 3The effect of the optical imaging shown is manifested by a simulated beam 800, emitted by the simulated object and passing through a surface different from the simulated sensor 100 at a certain position as an incident beam 2010 with a certain direction. After passing through the lens, it strikes a point on the simulated sensor 100 in a specific direction as an outgoing beam 2020, which typically differs from the original beam direction. The resulting beam displacement and change in direction of the passing light beam 800 is the effect of the lens 1. The beams designated as incident beam 2010 and outgoing beam 2020 are partial beams of the beam 800 passing through the lens.Some of the information required for a complete description of the light beam 800 may also include information on polarization, one or more wavelengths or colors, each associated with an intensity.

[0029] The effects caused by a lens 1 on a passing beam 800 can include beam displacement, change in direction, change in polarization, color change, attenuation of beam intensity, splitting of the beam into partial beams, or other changes. Several of these effects can also occur simultaneously on a beam 800. Splitting into partial beams can be caused, for example, by partial transmission and reflection at optically refracting surfaces or coating surfaces, by splitting into ordinary and extraordinary beams in optically anisotropic crystals, or by diffraction at diffractive structures or holograms. All these effects are effects of a lens 1 as described in the invention.

[0030] For practical reasons, optical simulations are often performed by calculating the rays in the opposite direction, i.e., starting from the sensor. The description of the invention here follows this approach; however, it is explicitly pointed out again that, according to the invention, a ray simulation in the direction of light is also captured.

[0031] According to the invention, the simulated sensor 100 is located on a first surface, the sensor surface 110. Light rays 800 to be simulated strike this surface at an impact point 111.

[0032] According to the invention, a first data set 1010, the input data set, contains, among other things, data suitable for describing, in conjunction with the transformation rule 900, the effect of the simulated lens on simulated light rays 800 passing through the simulated lens. The characteristics of the simulated lens are contained exclusively in the first data set 1010 and not in the transformation rule 900. Neither the data set 1010 nor the transformation rule 900, however, contain data about light rays to be simulated or parameters adjustable on the simulated lens. The transformation 900 and the data set 1010 can together be referred to as the virtual lens 901, since the provision of only this information is sufficient for a user to create renderings of virtual scenes with their own data. The first data set 1010 can be used as described in Figure 4As shown schematically, the data sets contain one or more subsets 1010r, 1010g, 1010b, each assigned to a specific wavelength of light or a specific wavelength band. This accounts for chromatic aberrations that occur in real lenses for physical reasons. Preferably, three such subsets 1010r, 1010g, 1010b are provided, namely for red, green, and blue. However, more or fewer such subsets can also be used. According to the invention, color effects can also be calculated using a difference approach from only one data set 1010, particularly when the color aberrations are small.

[0033] The first data set 1010 further contains data according to the invention, which includes information about a first surface or virtual input surface or virtual front surface 1011 that is penetrated by simulated rays of the imaging beam path or their extension at a first point of penetration 1013. Preferably, this surface is located upstream or downstream of the simulated sensor. Preferably, this first surface or virtual front surface 1011 is a rotationally symmetric surface whose axis of symmetry corresponds to the optical axis OA of the simulated lens and which is preferably located upstream of the actual surfaces of the lens to be simulated. This includes, in particular but not limited to, spherical surfaces and other surfaces formed by rotating conic sections, as well as planes. The first surface or virtual front surface 1011 may coincide with a lens surface of the simulated lens, but need not.Preferably, the virtual front surface 1011 does not coincide with a lens surface of the simulated lens. Another preferred position of the virtual front surface is the entrance pupil of the lens. Particularly preferably, the virtual front surface 1011 is positioned further away from the sensor 100 than the lens surface furthest from the sensor. Preferably, the virtual front surface is chosen such that its position and shape remain constant when the focal length, focus, or other adjustable lens parameters are changed. This reduces the complexity of the simulation and increases confidentiality, since the virtual front surface 1011 thus contains no information about the actual structure of the lens being simulated.

[0034] The dataset can consist of a polynomial. Depending on the desired accuracy of the beam position or beam direction, as well as the information on shadowing or vignetting surfaces, the polynomial can contain at least 5, or at least 10, or at least 20, or at least 30 coefficients. For example, a sparse polynomial with 20 coefficients of maximum degree 6 for beam position and beam direction, and with 5 coefficients of maximum degree 3 for shadowing information, can be used.

[0035] A second one, in Figure 5The schematically shown data set 1020 contains, among other things, information 1021 about a simulated light beam 800, consisting of information about an impact point 111 of this simulated light beam 800 on the sensor surface 110, as well as directional information 112 of this beam. The information about the impact point 111 can be given by a three-dimensional vector in a coordinate system connected to the simulated lens, but also by two-dimensional parameters that describe the position on the sensor surface 110 in a parameterized manner, as shown in Figure 6shown. This can be, for example, but not exclusively, an azimuth angle and a radius, or a normalized radius, or an elevation angle. The direction information 112 of the simulated beam can, for example, and not necessarily exclusively, be given by specifying a three-dimensional vector, but also by specifying two angles with respect to a coordinate system associated with the simulated lens. Another particularly preferred representation of the direction according to the invention can also be achieved by the parameterized specification of a Figure 6The second point of penetration 201 through a second surface 200, shown, consists of a second point of penetration 201, which, in conjunction with the point of impact 111, establishes a one-to-one relationship to the directional information 112. When specifying the second point of penetration 201 on the second surface 200 in a parameterized manner, it has been shown that specifying a radius rn normalized to 1 is particularly advantageous, because then the directional information 112 consists only of an angle between 0 and 360 degrees and a radius rn in the range 0 < rn ≤ 1, and no further specification of the axial position of the second surface 200 is required. For further calculations, only a set of two parameters, for example, angle and normalized radius, is needed, while the axial position can be disregarded.It has proven advantageous to choose the position of an aperture diaphragm for the second surface 200, as this is where most rays are frequently blocked, especially when the lens is stopped down. When simulating the lens at its widest aperture, the maximum normalized radius rn = 1 is chosen; when simulating a stopped-down lens, a correspondingly smaller value for rn is used. The second data set 1020 can include this information 1022 about the stopping down of the lens being simulated. Since, due to the parameterization, the actual axial position of the second surface 200 and the actual size of the aperture diaphragm are not known, or at least do not need to be known, while the effect of the aperture diaphragm is modeled by appropriately choosing the radius rn, maximum confidentiality regarding the actual position of the aperture diaphragm can be ensured.

[0036] In addition to the preferred information about the radius of an approximately circular aperture opening, data describing changes in the shape of the aperture diaphragm can also be included. For example, the iris diaphragm of a Carl Zeiss Sonnar 2.8 / 135 with a C / Y mount exhibits a pronounced deviation from rotational symmetry at an aperture setting of f / 4. The transparent area of ​​the diaphragm or aperture diaphragm is described by at least one dimension, such as a radius, but other dimensions can also be used to describe more complex shapes of the diaphragm.

[0037] The second data set 1020 can further include information 1023 about one or more additional imaging parameters adjustable on the lens that influence the imaging. According to the invention, this includes, for example, but not exclusively, the focal length and / or the focus distance of the simulated lens and / or a parameter for variably adjusting the image field curvature. Alternatively to the focal length of the lens, the image scale can also be included, relative to the image scale of objects that are sharply imaged onto the sensor plane. Furthermore, the data set 1020 can include information about beam deflections by variable mirrors or other optical elements, such as diffractive elements, polarizers, optical filters such as graduated neutral density filters, frequency filters, variable beam splitters or Alvarez plates, or other movable freeform surfaces.

[0038] The second data set 1020 can further include information 1024 about one or more wavelengths, wavelength ranges or colors of the light rays to be simulated.

[0039] The second data set 1020 thus contains all quantities selected or effected by the simulation user, such as adjustable parameters on the simulated lens and information about the light rays to be simulated 1021, 1024. The first data set 1010, on the other hand, contains only the data relating to the lens's imaging behavior for all conceivable combinations of the adjustable parameters 1022, 1023. It goes without saying that during the training and optimization phase for creating the first data set 1010, the entire range of parameters 1022, 1023 that are to be permitted is taken into account.

[0040] This range is usually determined by the capabilities of the real-world model of the lens being simulated.

[0041] For the simulation of a given lens according to the invention, the first data set 1010 is required before the actual simulation can be carried out. This data set describes the imaging behavior of the lens to be simulated and the virtual front surface 1011 of the simulated lens 1 using the transformation rule 900. The data set 1010 can be obtained by a calculation as follows.

[0042] First, a second surface 200 is selected between the sensor position and the virtual front surface. Data set 1010 is pre-calculated taking into account the possible adjustable parameters 1022, 1023, and the second surface 200. The following steps are performed for one or more wavelengths or wavelength ranges: For a multitude of rays and a variety of adjustable parameter settings, such as focus distance or focal length, ray tracing is performed using the exact optical setup of the lens being simulated. Data set 1010 is then calculated using an optimization algorithm. Known iterative optimization algorithms or state-of-the-art fitting algorithms can be used for this purpose. One particularly suitable algorithm is known as the "Orthogonal Matching Pursuit" algorithm. The result of this fitting or optimization process is data set 1010, which, for a given input ray and given adjustable lens parameters, provides the position and direction of the output ray, taking into account the second data set 1020.In addition to this optimization process, one or more locations within the lens 1 to be simulated can be identified where rays are shadowed and thus absorbed, for example, because they enter the lens housing or an aperture. A particularly suitable and preferred aperture or shadowing surface 210 is the aperture diaphragm 3, which can be selected in diameter and / or shape. Such an aperture is present in many lenses. Information about all considered shadowing surfaces 210 is incorporated into the data set by creating several sub-datasets 1030. Each of these sub-datasets 1030 contains information about the coefficients suitable for partially simulating the lens to be simulated from the sensor surface to the respective shadowing surface 210. The actual axial positions of these shadowing surfaces 210 are not relevant.A beam position can be specified for each shadowing surface 210 by a parameterized approach, where the parameters consist, for example, of an azimuth angle and a radius, or an elevation angle, or, particularly preferably, a normalized radius. The data set 1010 must at least include information about the coefficients suitable for simulating the beam paths from the sensor surface to the virtual front surface 1011, or to the lens front surface, or to the entrance pupil of the lens 1. The virtual front surface 1011 can be predefined. However, it is also possible to optimize the virtual front surface 1011 as part of the optimization process for the data set 1010.

[0043] As an alternative to the optimizing calculation method, dataset 1010 can also be obtained by measuring at least one real-world embodiment of the lens to be simulated. For this purpose, sample objects can be imaged with the lens, and the resulting images can be recorded. This can be done for a variety of lens parameters, such as focus, focal length / magnification, and / or aperture. The required set of coefficients, which constitutes dataset 1010, can then be calculated from the resulting images. Another metrological method for obtaining the dataset is to illuminate a real-world embodiment of the lens to be simulated with a single light beam of a suitable wavelength, for example, a laser beam. The effect of the lens on the beam can be measured at the other end of the lens.This can be done for other wavelengths and different points and directions of the light beam. The required set of coefficients that constitutes data set 1010 can then be calculated from the measured values ​​obtained in this way.

[0044] It goes without saying that the computationally and / or metrologically obtained values ​​of the data set 1010 can subsequently be empirically modified in order to, for example, replicate manufacturing tolerances or imperfections of real embodiments of the lenses.

[0045] For the simulation of the imaging of a given lens 1 according to the invention, a computer program is loaded onto a computer 300, which enables the computer 300 to perform a calculation according to the transformation rule 900. The previously pre-calculated first data set 1010, which contains information about the lens to be simulated and the virtual front surface 1011, is also loaded onto the computer 300.

[0046] A data set 1020 is generated, consisting of information 1021 about a ray 2010 to be simulated as falling on the sensor, and information about one or more parameters 1022, 1023 adjustable on the lens. The information 1021 about the ray 2020 to be simulated includes, in addition to the color or wavelength, an impact point 111 of the ray on the sensor 100 and directional information 111, which may also be derived from information about the second point of penetration 201 through a second surface 200, which may contain the aperture diaphragm. The point of impact 111 preferably corresponds to the position of a simulated pixel of the simulated sensor 100. Preferably, only those rays are considered that strike the second surface 200 only within the area transparent via the set aperture according to the information 1022.For a circular aperture diaphragm with a normalized radius rb, this can be achieved by generating and calculating only the rays to be simulated that have points of impact with a radius r < rb. For more complex geometries of the aperture diaphragm 3 or shading surface 210, rays can be generated that strike the second surface 200 outside the area transparent to the set aperture according to information 1022, but these are discarded because only rays originating from the transparent area of ​​an aperture can reach the sensor. The procedure described here for the aperture diaphragm can be carried out analogously for all other shading surfaces 210 that may be present in the lens.

[0047] The processing unit of computer 300 then performs a calculation according to transformation rule 900 with input data sets 1010 and 1020 to simulate a single light ray 800, which contributes to the image to be simulated at a selected pixel. This results in an output data set 2000, which contains which incident ray 2010 is converted by the simulated lens into the ray 2020 of input data set 1021 incident on the sensor. If one or more shading surfaces, for example an aperture diaphragm, are present in the lens to be simulated, the simulation can proceed in several sub-steps, with each calculation proceeding from the sensor surface 110 to one of the shading surfaces 210. A separate sub-step is calculated for each of the shading surfaces 210, and the order in which these sub-steps are executed is irrelevant.Areas that cause significant shadowing are given preference over those that cause little shadowing. The individual steps can also be performed in parallel. Each step is followed by a check to determine whether the beam is absorbed or transmitted at a third point of penetration 211 through the shadowing surface. Absorbed beams are discarded, while transmitted beams are tracked further. A calculation is also performed from the sensor surface to the virtual front surface 1011, the objective lens surface, or the entrance pupil. The resulting output data set 2000 contains information about the beam 2010 incident on the virtual front surface 1011, the objective lens surface, or the entrance pupil.These include the first point of penetration 1013 of the beam 2010 through the virtual front surface 1011 and directional information, which is described, for example, by a three-dimensional vector, or by a parameterized representation of the point of penetration through another surface, or by two angles related to a coordinate system connected to the simulated lens.

[0048] To calculate color images, the simulation step for simulating a single light beam can optionally be performed multiple times, preferably three times, with a different wavelength or wavelength band of the simulated light being assumed for each of these calculations, and a different sub-dataset 1010r, 1010g, 1010b being used for the simulation. For simulated lenses with small chromatic aberrations, calculating only one dataset 1010 may suffice, whereby a differential correction of the resulting beam directions and / or beam positions is performed for at least one of the considered wavelengths. The simulation step consisting of several single-beam simulations for different wavelengths, or consisting of a single-beam simulation followed by a differential correction step for additional colors, will be referred to below as the polychromatic beam simulation step.

[0049] Based on the initial data set for a given beam 2020, the intensity of the incident light 2010 can be deduced from the model of the scene to be depicted, from which the intensity contribution or equivalently the brightness contribution of this simulated light beam to the signal of the pixel under consideration in the respective wavelength or color is obtained.

[0050] The described monochromatic or polychromatic beam simulation step is performed for the same target pixel for a plurality of beams to be simulated. These are chosen such that they originate from different positions in the second surface 200, preferably the aperture diaphragm 3.

[0051] For each of these simulated rays, a brightness contribution, the intensity of the incident light ray 2010 at a given wavelength, is determined from information about the light emanating from the object feature of the modeled scene observed in that direction, and all brightness contributions for a pixel are summed. The brightness contributions of shadowed rays are discarded or set to zero so that they contribute nothing. The resulting pixel intensities or brightness values ​​are stored in computer memory, preferably in random access memory (RAM), flash memory, or on a magnetic disk.

[0052] The described simulation steps are repeated for the remaining pixels of the sensor until a first image is fully constructed.

[0053] To generate an image sequence for a cinematic image sequence, the described steps are repeated for further images. The scene to be modeled, the position of the camera to be simulated, and / or the adjustable lens parameters, such as focus, focal length, and / or aperture, can be changed. Such changes are particularly necessary for special cinematic effects, where, for example, a focus shift directs the viewer's attention from one object feature to another. As described, such changes do not require any recalculation of data set 1010. Changes in adjustable lens parameters are simply incorporated into data set 1020, which offers a significant speed advantage over prior art methods and makes the rendering of such scenes with the inventive method particularly efficient.

[0054] As described, a compositing process step can be added, in which the simulated image or image sequence is merged with real recorded images.

[0055] In Figure 7 This is a schematic lens section of an exemplary lens to be simulated, shown for a focus setting of "infinity". As shown in the illustration in Figure 7 As can be seen, the objective 1 has a first, a second, a third, and a fourth lens 7, 8, 9, 10, which are arranged one behind the other in this order starting from the object side along an optical axis OA of the objective 1. The first and third lenses 7, 9 each have positive refractive power, and the second and fourth lenses 8, 10 each have negative refractive power.

[0056] The inventive method for simulating the effect of the lens made of Figure 7This can therefore comprise the following steps: First, a virtual front surface 1011 is defined. The virtual front surface 1011 is located at a predefined distance in front of the image sensor 100 and has a predefined and / or optimized radius of curvature. In the exemplary embodiment, the distance from the sensor is 60.0 mm and the radius of curvature is 13.365 mm; however, other values ​​could be selected depending on the application and / or requirements. In the second step, the lens model is created through a training phase. For this purpose, a large number of training or validation beams, each with a defined wavelength, are generated. For example, there can be more than 5000 training beams, preferably more than 10000 training beams, and / or more than 2000 validation beams, preferably more than 4000 validation beams, per wavelength and / or focus and / or considered value of one or more other variable lens parameters.In this embodiment, the focus settings are generated. In this embodiment, 9 approximately evenly distributed focus settings and a wavelength of 440 nm for the simulated light were assumed.

[0057] In the next step, the shading surfaces are identified. In the exemplary embodiment, these are the radius of the front surface of the first lens 7 of 8.7 mm and another shading surface 11 with a radius of 6.0 mm.

[0058] In the next step, the parameter set for the abstract mathematical model given by the transformation rule is created using an optimization procedure. In the exemplary implementation, this is a sparse polynomial with 20 coefficients of maximum degree 6 for the beam position and direction, and with 5 coefficients of maximum degree 3 for the shadowing information.

[0059] This exemplary model is optimized using the orthogonal matching pursuit method, and the aforementioned coefficients are calculated.

[0060] The resulting output data includes at least one of the following values: radius of curvature of the virtual front surface, distance of the virtual front surface to the sensor, minimum aperture (F min), supported focus range (d), focal length (f)

[0061] The user of the example lens model receives the following metadata: Radius of curvature of the virtual front surface: 13.365 mm Distance of the virtual front surface to the sensor: 60.0 mm Minimum aperture: F min = 2.87 Supported focus range: d = 507 mm to infinity Focal length: f = 50.0 mm

[0062] Input variables of the exemplary lens model: xs, ys: Beam position at the sensor, definition ranges -18.0 mm ≤ xs ≤ 18.0 mm and -12 mm ≤ ys ≤ 12 mm. xa, ya: Beam position in the virtual aperture, definition range: xa 2< + ya 2< < (F min / F) 2< at aperture value F β = f / (f - d)

[0063] Output variables of the exemplary objective model: xf, yf: Ray position projected onto a tangent plane at the vertex of the virtual front surface in mm. uf, vf: Ray direction projected onto a tangent plane of the virtual front surfaces at the ray point. xv1, yv1, xv2, yv2: Ray position on shadowing surfaces. The ray is shadowed if x2 < + y2 < > 1 at one of the shadowing, vignetting surfaces.

[0064] In addition, the fit data of the parameterized optics of the lens to be simulated in the exemplary embodiment are included: Xf coefficient Exponent xs Exponent ys Exponent xa Exponent ya Exponent β Order -0,459663284 1 0 0 0 0 1 8,768720761 0 0 1 0 0 1 -1,304539732 1 0 0 0 1 2 0,000110631 3 0 0 0 0 3 -0,00496582 2 0 1 0 0 3 0,000109726 1 2 0 0 0 3 -0,005078146 1 1 0 1 0 3 0,091802222 1 0 2 0 0 3 0,067526222 1 0 0 2 0 3 0,025031505 0 1 1 1 0 3 -0,07933075 0 0 1 2 0 3 0,000671128 3 0 0 0 1 4 0,000725495 1 2 0 0 1 4 1,134.481.527 0 0 3 0 1 4 -5,36E-05 2 1 1 1 0 5 0,137138593 2 0 1 0 2 5 0,158010585 1 1 0 1 2 5 -0,784407072 1 0 0 2 2 5 3,288014989 1 0 0 0 4 5 0,000741927 3 0 2 0 1 6 u f coefficient Exponent xs Exponent ys Exponent xa Exponent ya Exponent β Order 0,014551901 1 0 0 0 0 1 -0,666745323 0 0 1 0 0 1 0,06893195 1 0 0 0 1 2 0,000145798 2 0 1 0 0 3 7,79E-06 1 2 0 0 0 3 -0,000358482 1 1 0 1 0 3 -0,00282992 1 0 2 0 0 3 0,002553763 1 0 0 2 0 3 -0,000240285 0 2 1 0 0 3 0,013381898 0 1 1 1 0 3 0,021614836 0 0 3 0 0 3 -0,159397768 0 0 1 2 0 3 -1,127168311 0 0 1 0 2 3 4,83E-05 1 2 0 0 1 4 -0,002460254 1 1 0 1 1 4 -0,00172234 0 2 1 0 1 4 -7,99E-06 1 2 2 0 0 5 0,000348182 1 1 2 1 0 5 -0,176833532 1 0 0 2 2 5 -0,47254192 0 1 1 1 2 5 x v1 coefficient Exponent xs Exponent ys Exponent xa Exponent ya Exponent β Order -0,027987126 1 0 0 0 0 1 1,011147804 0 0 1 0 0 1 -0,000368083 2 0 1 0 0 3 -0,000342116 1 1 0 1 0 3 0,005565995 1 0 0 2 0 3 Xv2 coefficient Exponent xs Exponent ys Exponent xa Exponent ya Exponent β Order 0,031357313 1 0 0 0 0 1 0,926341281 0 0 1 0 0 1 0,043996073 1 0 0 0 1 2 -0,257282737 0 0 1 0 1 2 3,06E-06 3 0 0 0 0 3

[0065] The coefficients for yf, vf, yv1, and yv2 are not shown, as they can be directly derived from those for xf, uf, xv1, and xv2, respectively, due to symmetry. x f = ∑ i , j , k , l , m c ijklm x s i y s j x a k y a l β m with the coefficient c, the exponent i of xs, the exponent j of ys, the exponent k of xa, the exponent l of ya, and the exponent m of β. The same applies to uf in the second table. One could therefore also label Xf. c ijklm i j k l m Order -0,459663284 1 0 0 0 0 1 8,768720761 0 0 1 0 0 1 -1,304539732 1 0 0 0 1 2

[0066] It is understood that information about the scene to be depicted is required to calculate the intensity or brightness and color of a pixel. If the directional information about the first point of intersection 1013 of the simulated light ray striking the virtual front surface 1011 is available, it is possible to calculate from which point on the modeled object the incident light originates. The information about the direction and the point of impact or first point of intersection 1013 of the simulated light ray is obtained by the simulation method according to the invention.

[0067] One way to determine the object point from which a light ray entering the lens 2010 originates is conventional ray tracing. This has the advantage that the resulting representations are physically accurate and therefore photorealistic. A disadvantage is that such calculations are very computationally intensive and therefore require a lot of processing time. It is desirable to have a method that allows significantly faster access to information than ray tracing regarding the color, intensity, or brightness of a light ray emanating from a modeled object and striking the simulated lens 1 or a virtual front surface 1011 in front of the lens.

[0068] It is therefore also an object of the invention to provide a method which allows one to obtain physically approximately correct information much faster than with ray tracing, regarding the color, intensity, or brightness of a light beam that emanates from a modeled object and falls on the simulated lens or a virtual front surface in front of the lens.

[0069] This problem according to the invention is solved by the features of claim 1 and the dependent claims relating thereto.

[0070] It is desirable that this method according to the invention can benefit from the special properties of GPUs with regard to speed and parallelization. Knowledge of the beam directions, intensities, and colors of the light rays 800 incident on an entrance surface 3000 is equivalent to knowledge of the so-called light field at this entrance surface. The entrance surface 3000 is such that it can, for example, include the entrance pupil, the front surface of the front lens, or the virtual front surface 1011. In principle, another surface can also be chosen, provided that the rays 800 contributing to image formation pass through this surface. It is advantageous if the entrance surface 3000 is chosen to be larger than, for example, the entrance pupil, the front surface of the front lens, or the virtual front surface 1011.If a simulation of image formation through a lens is to be carried out using the simulation method described in the invention, it is particularly advantageous to select the entrance surface 3000 such that it comprises the entrance pupil of the lens to be simulated or the front surface of the simulated lens 1 or the virtual front surface 1011, because then the light field approximated in the entrance surface 3000 can be used to simulate image formation without a further transformation step.

[0071] If the light field is known, the image formation through the simulated lens 1 can be accurately modeled using this information. Knowledge of the light field, or of the parts of the light field relevant for imaging, can be obtained through ray tracing. However, this is computationally intensive and therefore slow. To calculate a realistic image, however, complete knowledge of the light field is not absolutely necessary; a sufficiently good approximation is adequate.

[0072] A method according to the invention to achieve such a sufficiently good approximation is the method described below.

[0073] A multiple set of ideal images are generated at various positions on the entrance surface 3000. This can be achieved by calculating these images using a pinhole camera model for pinhole cameras positioned at the respective locations. In addition to the ideal images, creating a depth map of the scene to be modeled is also useful. The pinhole camera images can be advantageously arranged so that they are each located at the position where an incident simulated light beam strikes the entrance surface 3000. In this case, the intensity and color of the light beam can be directly derived from the pinhole camera image, given the direction of this beam.This can be achieved, on the one hand, by generating pinhole camera images at the respective positions and then performing a beam calculation through the lens from the position of the pinhole camera to a specific pixel, or by first performing a beam calculation to a specific position on the entrance surface 3000 and then generating the corresponding pinhole camera image.

[0074] To increase the number of available pinhole camera images, new pinhole camera images can be calculated from existing images and at least one associated depth map by interpolation or extrapolation. The use of at least one depth map is necessary to generate the interpolated pinhole camera images with correct perspective. The generation of these interpolated images can be performed, for example, using a method known as "screen space ray tracing."

[0075] In principle, it is possible to determine such interpolations from a single pinhole camera image with an associated depth map. However, at least two, and particularly preferably three or more pinhole camera images are preferred. Figure 8 The situation is shown as an example for four, two and one rendered pinhole camera image 5100 and the resulting interpolated or extrapolated pinhole camera images 5200 in relation to the entrance pupil 5000.

[0076] New pinhole camera images can also be obtained using artificial intelligence (AI) methods, with or without a prior depth map, for example with the help of a neural network.

[0077] The positions of the pinhole cameras can be arranged particularly advantageously in one of the following constellations: A fixed spiral grid of constant density, for example in the form of a so-called Fibonacci spiral as in Figure 9shown. The positions of such pinhole cameras are preferably arranged within the transparent area of ​​the entrance surface 3000. Purely random arrangement, as shown in Figure 10 shown. At least three pinhole cameras at positions outside the transparent area of ​​the entrance surface 3000, such that the entrance pupil or the frontal surface or the virtual frontal surface 1011 lies within the polygon described by the positions of the pinhole cameras.

[0078] The positions of the pinhole camera images used can be adjusted to the scene being modeled in order to provide a sufficient number of perspective views and avoid artifacts. For example, to view through a thin tube, perspectives looking into the tube and other perspectives looking at the tube from the outside may be needed.

[0079] Naturally, any other arrangement of the pinhole cameras is also included according to the invention. Fixed positions with an approximately constant density improve the parallelizability of the method, particularly on GPU-based computing systems, and reduce noise in the resulting image. The quality of the resulting image increases with the number of ideal images used. This is especially true in regions with high blur, because the images of defocused object points that form the bokeh are particularly complex. It has proven advantageous to heuristically adjust the position density by using information from the depth map and comparing it with the focus setting of the lens being simulated. The position of the pinhole camera grid can also be randomly rotated or disturbed, which can lead to higher image quality.It is also advantageous to adjust the density of the pinhole cameras statically or dynamically to a heuristic quality measure of the resulting pixel.

[0080] Information about the light field is obtained by interpolating the information from the individual ideal images. This can be done individually for each required wavelength, or it can be done exactly for only one wavelength, while the information for the other required wavelengths is approximated differentially from it.

[0081] The image generation is simulated by simulating the contributions of all image points or pixels of the sensor 100 to be simulated. For this purpose, the light contributions of a pixel are integrated by first determining the direction of incidence of a multitude of rays incident on this pixel by simulating the lens 1 onto the entrance surface 3000. Rays that are blocked by the lens 1 are discarded. The multitude of rays can preferably be chosen such that the entrance surface 3000 is sufficiently uniformly penetrated by the rays. Another preferred multitude of rays can be chosen such that they intersect the entrance surface 3000 precisely at the positions where ideal images are present. When converting interpolated ideal images into ray color and intensity, the inclusion of weighting factors can be advantageous, since each image pixel corresponds to a light cone of a different size.For example, pixels at the very edge cover a smaller angular range than those in the center of the image sensor.

[0082] According to the invention, this simulation is carried out using the described, parameterized method according to the invention.

[0083] For each simulated ray, the brightness contribution of that ray at the corresponding pixel is deduced from the knowledge of the ray direction, its position on the entrance surface 3000, and the light field or the approximate light field. This is done by evaluating the associated pinhole camera image. The brightness contributions of all simulated rays are summed at their respective pixels, thus creating the image of the scene to be modeled.

[0084] The beam direction at a given point of incidence on the image sensor and on the entrance surface 3000 depends on the wavelength. To determine the luminance contribution of a light beam from a light field or an approximate light field, a separate calculation can be performed for each required wavelength or color. If the changes in direction are small, it may be sufficient to start with a calculation rule for only one principal wavelength and make only small modifications to this rule for other wavelengths. For this purpose, the position in the light field is first determined for a principal wavelength W1, and the luminance contribution or intensity contribution is calculated from this. Then, for the same points of incidence of the simulated light beam on the image sensor and on the entrance surface 3000, the position used for W1 is used as the starting point for finding the correct position for the other wavelengths.Then, the brightness contribution(s) for one or more wavelengths can be calculated. This can be done, for example, by interpolation, preferably linear interpolation, for the selected wavelengths. By applying this procedure to a large number of pixels of the image sensor, an image of the scene to be modeled, including the polychromatic aberrations of the lens, can be obtained.

[0085] The accuracy with which the beam direction is determined by parametric optics is crucial for the quality of the resulting simulated image. This follows from the laws of geometric optics, particularly the intercept theorem. To a first approximation, the directional differences of the beam for different wavelengths or colors at the same point of incidence on the image sensor and the entrance surface 3000 are linear. Depending on the chosen parametric representation, these differences can be fitted with a reduced set of parameters while maintaining the same overall accuracy, thus requiring less computation time during the evaluation of the parametric function.

[0086] Figure 11Figure 1 schematically illustrates the setup of an image simulation using an abstract staircase object and a corresponding pinhole camera image of this staircase object. The simulation simulates the recording of a staircase-like set of objects 5600, which are captured with a camera 5500 such that the individual steps are at different distances from the camera 5500. One of the steps 5500 is in focus, while the other steps are out of focus. An image of the staircase object 5700 captured with an arbitrarily small aperture corresponds to a pinhole camera image, which has infinite depth of field. Here, all the individual steps are rendered with the same sharpness. Figure 12 shows a comparison of three simulations using the method according to the invention.

[0087] The image generated by the simulation can be combined with real images through compositing. According to the invention, the simulated image is generated using a lens simulation that replicates the lens used in a real photograph, resulting in a particularly harmonious overall impression of the combined image. Reference symbol list

[0088] 1 Lens 2 Lenses 3 Aperture 4 Mount 5 Pixel 7 First lens 8 Second lens 9 Third lens 10 Fourth lens 100 Image sensor 110 Sensor area 111 Point of incidence of simulated light rays on the sensor 112 Directional information of the ray hitting the sensor 200 Second area 201 Second point of penetration 210 Shadow area 211 Third point of penetration 300 Computer 800 Simulated light ray 900 Transformation rule 901 Virtual lens 1000 Input data sets 1010 First data set ("Virtual lens") 1010r, 1010g,1010b Subsets for different colors 1011 Virtual front surface 1012 Axis of symmetry 1013 First point of penetration 1020 Second dataset 1021 Information about a simulated light beam 1022 Information about the set aperture 1023 Information about other imaging parameters adjustable on the lens 1024 Information about the wavelength or color of the light to be simulated 1030 Subsets 2000 Output dataset 2010 Incident beam 2020 Outgoing beam 3000 Entrance area 5000 Entrance pupil 5100 Rendered pinhole camera image 5200 Interpolated or extrapolated pinhole camera images 5500 Camera 5600 Staircase-arranged set of objects 5700 Staircase object taken with an arbitrarily small aperture, corresponds to a pinhole camera image 5800 Staircase object with few 5900 staircase objects captured with many pinhole camera images,

Claims

1. Method for generating an image by simulating an image representation of a scene using an optical imaging system which comprises an image recorder (100) located on a first surface (110) and a lens (1), comprising the following steps: providing a first data record (1010) comprising data, which describe the effect on light rays (800) of the lens (1) to be simulated, about a second surface (200) and about a virtual front surface (1011), providing a transformation rule (900), selecting a multiplicity of picture elements on the image recorder (100), calculating a respective brightness contribution for light rays (800) incident on each of the picture elements, comprising the following steps: - selecting a point of incidence (111) for light rays (800) on the image recorder (100), - selecting a multiplicity of different second points of intersection (201) for light rays (800) on the second surface (200), - carrying out the following steps for each of the second points of intersection (201): providing a second data record (1020) comprising data about the point of incidence (111) and about the respective second point of intersection (201), calculating a first point of intersection (1013) of the light ray (800) with the virtual front surface (1011) and the direction of the light ray (800) at the first point of intersection (1013) by way of applying the transformation rule (900) to the first data record (1010) and the second data record (1020), determining the brightness contribution of the light ray (800), storing an information item regarding the brightness contribution of the light ray (800), summing the brightness contributions arising, and storing the result of the summation, characterized in that the brightness contribution of each of the light rays (800) simulated to this end, which intersect the virtual front surface (1011) at the first point of intersection (1013), is determined with the aid of a pinhole camera image in each case, the nature of the image being such that it corresponds to the image generated by a pinhole camera placed at the respective first point of intersection (1013).

2. Method according to Claim 1, characterized in that - the lens (1) has at least one adjustable imaging parameter (1022, 1023) and - the second data record (1020) contains information items about the at least one adjustable imaging parameter (1022, 1023) of the lens (1).

3. Method according to Claim 2, characterized in that the adjustable imaging parameter (1023) comprises the focus setting and / or the focal length and / or the magnification and / or the field curvature of the lens (1).

4. Method according to any of Claims 1 to 3, characterized in that the lens (1) comprises a stop, preferably an aperture stop (3), and the second surface (200) coincides with the stop, preferably the aperture stop (3).

5. Method according to Claim 4, characterized in that one of the adjustable imaging parameters (1022) describes at least one of the dimensions of the aperture stop (3).

6. Method according to Claim 5, characterized in that the aperture stop (3) is at least approximately circular and the information items relating to the second point of intersection (201) of the light ray (800) with the second surface (200) contain a normalized radius.

7. Method according to any of Claims 1 to 6, characterized in that the first data record (1010) comprises data - in relation to at least one shading area (210) in the lens (1) and - in relation to the effect of a part of the lens (1) on at least one light ray (800), which extends between at least one shading area (210) and the image recorder (100), and comprises the following steps before storing the brightness component of the at least one light ray (800): - calculating a third point of intersection (211) with the at least one shading area (210), - checking whether the at least one light ray (800) is absorbed by or transmitted through the third point of intersection (211), - discarding the light ray (800) or setting the brightness component to zero if the at least one light ray (800) is absorbed.

8. Method according to any of Claims 1 to 7, characterized in that the virtual front surface (1011) and the entrance pupil of the lens to be simulated coincide.

9. Method according to Claim 8, characterized in that the second surface (200) coincides with the virtual front surface.

10. Method according to any of Claims 1 to 9, characterized in that at least one of the pinhole camera images was calculated by interpolation or extrapolation from other pinhole camera images.

11. Method for generating an image, comprising the following steps: - providing a real image recorded by a real camera, - generating a virtual image according to any of Claims 1 to 10, - fusing or overlaying at least a portion of the real image and at least a portion of the virtual image, - storing the image created, wherein the adjustable lens parameters used for the simulation correspond at least approximately to those used during the real recording.

12. Method for generating an image sequence consisting of individual images, comprising the following steps: - providing a virtual scene, - providing a camera position in relation to the virtual scene, - calculating the individual images of the image sequence in accordance with one of the methods according to Claims 1 to 11, - storing the image sequence.

13. Computer program product suitable for carrying out a method according to any of Claims 1 to 12 after being loaded onto a computer.