Method for simulating an optical image representation

Abstract

A method for generating a luminance contribution of a picture element of an image by using an optical imaging system to simulate an image representation of a scene, the method comprising the steps of: - providing a first data record comprising data describing an influence of a light ray on a lens to be simulated, - providing a second data record comprising data on an entry point of the light ray on an image recorder and on a virtual front surface, - providing a conversion rule, - calculating a first intersection point of the light ray with the virtual front surface and a direction of the light ray at the first intersection point by means of applying the conversion rule to the first data record and the second data record, - determining a luminance contribution of the light ray, - storing information on the calculated luminance contribution of the light ray, wherein - the first data record comprises data on a second surface, and - the second data record comprises data on a second intersection point of the light beam with the second surface.

Description

Technical Field

[0001] This invention relates to a method for generating luminance contributions, a method for generating image elements, a method for generating images, a method for generating image sequences, and a computer program product. Background Technology

[0002] Image recording systems, comprising camera modules and lenses, are used for the photographic recording of images, particularly for recording image sequences. The camera module contains an image recorder, which can be an electronic image recording sensor or a photosensitive chemical film. Lenses typically contain multiple optical elements, such as optical lens elements, aperture stops, and / or mirrors. Such lenses often include aperture stops (so-called aperture stops) whose size can be changed. Optical image representation is achieved by allowing light from the object to be imaged to pass through these elements and fall onto the image recording sensor or film material. This optical image representation is typically affected by optical imaging aberrations, such as defocus, coma, spherical aberration, geometric distortion, or radial brightness reduction due to so-called vignetting. These so-called geometric imaging aberrations often depend on the focus setting, lens settings, aperture stops, and / or other parameters, such as the lens's focal length. In general, the imaging aberrations of each given lens cause characteristic geometric imaging behavior due to its specific optical lens design. This effect is particularly noticeable in the image representation of out-of-focus object features, because such features are blurred; more precisely, the farther the object feature is from the ideal focus area, the more blurred it becomes, and the larger the aperture stop of the lens is set, the more blurred it becomes.

[0003] This characteristic of imaging a defocused point is commonly referred to as bokeh. As an example, lenses are known in which a defocused point significantly off-axis is imaged onto a stretched, slightly curved out-of-focus area. As an example, this can be applied to so-called biotar-type lenses, such as those described in US 1,786,916.

[0004] As an example, reflective lenses (one of which is described exemplarily in DD 263604 A1) have entirely different characteristics. Such lenses typically image a defocused point onto a ring-shaped area of ​​out-of-focus. When using such lenses for image representation, the bokeh often exhibits a noticeably unstable ring pattern, especially with objects having significant highlights.

[0005] During the production of films, etc., moving photographic recordings are often mixed with virtual images and / or sequences of virtual images. This method is often referred to as rendering. The result of such rendering is a digitally generated image or a sequence of digitally generated images. This image or sequence of images is generated through calculations from a mathematical model of the scene to be presented. This scene model contains, in particular, information about the characteristics (e.g., shape, color, and optical surface properties) of the objects to be modeled in the scene. A mathematical model of lighting (consisting, in particular, of the position, type, color, and orientation of the various light sources to be modeled) is applied to this scene model. For the actual calculation of the images, at least the position of the virtual camera in the scene, its recording orientation, and its field of view are required. Here, the generation of images or sequences of images of the scene modeled by using a virtual camera, and the partial steps of generation through calculation, are referred to as simulation. This calculation often uses the simple principle of a pinhole camera (camera obscura), which is based on a simple geometric projection according to the mathematical principle of the intercept theorem. Although very simple, this forms the basis of the simulation of an ideal, typical photographic image representation because geometric image aberrations do not occur in this case.

[0006] The process of blending rendered virtual image sequences with real-world recorded image sequences is a common procedure in real-time 3D computer graphics and cinematography visual effects. Effects generated in this way are called VFX (from "visual effects"). Blending real images with virtual images is also known as enhancement. Enhancement is often part of so-called compositing, a step in film post-production or post-processing where images from different sources are brought together. Especially when blending images or sequences of real scenes obtained through photography with virtually generated content, the desired outcome is that this virtually generated content in the final image or sequence is indistinguishable from, or has only barely discernible differences from, the real scene obtained through photography, thus giving the most realistic impression possible. This is achieved by generating digitally generated content with boundary conditions that are as close as possible to the boundary conditions of the real image recording. A correct simulation of the real imaging chain includes, firstly, boundary conditions for the mathematical description of the scene and lighting, and secondly, boundary conditions related to the recording (i.e.,, in particular, the recording of an optical image representation on an image recorder or sensor). These are not reproduced by a simple geometric camera model (i.e., a simulation of a pinhole camera).

[0007] Another important application of rendered images or sequences of rendered images is fully animated images or sequences of images, meaning those whose entire information content originates from a virtual scene and is not mixed with real images or sequences of real images. This includes the fields of animated film production and virtual reality, such as computer games, and also training simulators, such as flight simulators for training pilots or train simulators for training locomotive engineers. Particularly in the fields of animated films and computer games, there is a desire to generate a certain level of realistic scene impression. Furthermore, this method can be used, for example, in the field of medical imaging to blend real medical image recordings with computer-generated image representations.

[0008] Rendering images typically uses techniques such as ray tracing. In the case of ray tracing, the paths of light rays from the light source to the object and from the object to the image recorder are calculated. In practice, this calculation is often performed such that light rays are traced backward from the image recorder to the object, that is, in the opposite direction to the physical direction of light propagation within the optical device itself. Naturally, tracing along the direction of light is equivalent and equally possible.

[0009] Often, the simple pinhole camera model described above is used for this ray tracing calculation to simulate an optical image representation on an image recorder. One advantage here is the generation of a so-called ideal image, free from geometric imaging aberrations. In this model, all rays in the ray tracing calculation are computed through a point located upstream of the simulated image recorder, the distance of which is chosen just large enough that the field of view for geometric calculation corresponds to the field of view of the camera system being simulated. This simulated point corresponds to the center of the very small hole in a real pinhole camera. The advantage of simulating a pinhole camera lies not only in the simplicity of the mathematical model but also in the high speed at which this calculation can be performed. In particular, a dedicated graphics processing unit (GPU) can perform this imaging simulation very quickly and in parallel for many image elements. This parallelization results in a significant speedup, enabling the calculation of entire complex virtual scenes in real time.

[0010] However, in principle, the aforementioned image aberrations or image representations that occur when recording with a real camera using real lenses are not taken into account, and the image impression produced by simulation often differs significantly from the impression produced when observing a reference real image. This can be problematic, especially when the aim is to blend rendered image content with real-recorded images or image sequences. To compensate for this, two-dimensional image processing techniques and artistic methods are often used. As an example, a blur is subsequently introduced, calculated, into one or more idealized images obtained through simulation using a pinhole camera, aiming to be as similar as possible to the blur produced by a real camera. Correction or introduction of geometric distortion, as well as a decrease in brightness in the form of vignetting, are other effects that can be easily added computationally. Typically, this correction is phenomenologically modulated so that the image impression of the image obtained through simulation is as close as possible to the image impression of the image obtained through real recording.

[0011] Therefore, the goal is to reproduce the characteristics of the real recording lenses used to record realistic images, and to reproduce these characteristics as accurately as possible in the simulation. First, it's crucial to obtain the most accurate correspondences possible to characteristics such as bokeh. Second, it should be possible to work using a simple process that doesn't require empirical experimentation with parameters, or even artistic skill, but is deterministically understandable and therefore predictable. Naturally, ray tracing techniques can, in principle, be applied to the entire optical component of an image recording system. In fact, ray tracing has been widely used in the design and optimization of optical systems for many years. However, its use in VFX or 3D computer graphics is unusual because these techniques involve a great deal of computation, and are therefore very resource- and time-consuming, and thus expensive. The GPUs commonly used for rendering, as mentioned earlier, are not designed for such complex imaging simulations.

[0012] During ray tracing simulations, the realistic model of the lens to be simulated incurs significant computational costs because the multiple ray deflections within the lens must be calculated according to the laws of geometrical optics refraction. These deflections occur at every interface between the optical material and air, and also at every interface between different optical materials, such as in the case of so-called cemented elements. For each image element on an image recorder with a different angle of incidence, this ray tracing must be performed multiple times to ensure proper rasterization of the transmission region of the aperture stop of the lens to be simulated. In this process, each image element may require approximately 1000 or more rays to be calculated to generate a high-quality image. Moreover, within the scope of ray tracing, this calculation must typically be performed for different wavelengths of light because optical materials have wavelength-dependent refractive indices and therefore wavelength-dependent ray deviations due to dispersion characteristics. Additionally, there is light absorption within the lens, known as shading, for example at the aperture stop and also at the lens element mount. This light absorption must also be simulated and captured, as it significantly contributes to bokeh and vignetting within the lens.

[0013] For each pixel of the image sensor, ray tracing needs to be performed multiple times in such a way that the rays incident on each sensor pixel pass through the exit pupil of the lens to be simulated, forming a sufficiently dense beam. This is the only procedure that allows for a realistic simulation of the imaging characteristics of the lens to be simulated due to geometric effects.

[0014] Besides the high computational cost, a drawback of ray tracing simulations of lenses (especially in the VFX field) is that it requires precise knowledge of the optical setup of the lens to be simulated. For example, this includes variables such as lens element material, lens element radius, radius of curvature, aspherical parameters, and / or distances between the optical elements of the lens (especially if these distances are variable). This means that the operator or user of the ray tracing method needs to know all these parameters. Conversely, it is impossible to simulate the imaging characteristics of a lens via ray tracing without knowing these parameters. To avoid having to share details about the lens structure to be simulated, an option is desirable that allows a third party to perform a realistic simulation of the imaging characteristics of the lens to be simulated, without requiring any details about the structure at all.

[0015] One known method for reducing computational costs is described in Schrader et al., “Sparse high-degree polynomials for wide-angle lenses,” *Eurographics Symposium on Rendering*, 2016, Vol. 35, No. 4. Here, an abstract lens model is used, where the coefficients of multiple polynomials of the transformation rules are chosen such that, given the position and direction of light incident on the lens to be simulated, the output is the position and direction of the corresponding light rays exiting the lens. This lens model or simulated lens is abstract because it does not contain information about the physical structure of the lens being modeled or simulated; this is why its structure cannot be derived or obtained from the model's coefficients through reverse computation. All that is needed is the rule governing how light rays deflect as they propagate through the lens. In this way, Schrader et al. describe an abstract simulation of the imaging properties of lenses in optical imaging systems.

[0016] The simulation of an image representation using a simulated lens is performed by applying a transformation rule with appropriate coefficients to the simulated position and direction of simulated light rays emitted toward the image recorder. The simulation result is the simulated position and direction of the incident light rays from the object. This simulation method can also be applied unrestricted to reverse ray tracing, and can provide the simulated position and direction of the outgoing light rays as a result. Naturally, in this case, an appropriate inversion transformation rule should be used.

[0017] Schrade et al.'s simulation method assumes that the focal position and focal length of the simulated lens are fixed. If one of these parameters is changed, the entire abstract lens model, represented by the coefficients of the transformation rule used, needs to be recalculated. This procedure is computationally intensive and therefore very time-consuming.

[0018] Another assumption made by Schrade et al. in their simulation method is that the light rays incident on the lens are incident on the first lens surface, and the lens's influence on the light rays begins from there. As a result, the simulation method described therein discloses information about the simulated lens to the user of the method, particularly the shape of the simulated lens's front surface. Therefore, it is not a complete abstraction of the lens.

[0019] Regarding focus setting for analog lenses, Schrader et al.'s method uses the displacement of the plane of the analog image recorder relative to the analog lens. This type of focus setting is only achievable with simple lenses, such as the aforementioned double Gaussian lens. In modern lenses, focus is often adjusted by displacing one or more lens elements relative to the other optical elements of the lens. In this case, the lens elements can be combined into a group of lens elements that are displaced as a whole unit. A solution for rapidly simulating such complex lenses cannot be found from the teachings of Schrader et al.

[0020] Lenses with variable focal lengths (also known as panoramic lenses, zoom lenses, or zoom lenses) also use optical lens elements and groups of optical lens elements that are displaceable relative to other lenses in order to change the focal length. If the teachings of Schrader et al. are used to simulate such zoom lenses, individual conversion rules with appropriate coefficients must be provided for each individually simulated, displaceable component. Furthermore, the displacement of the components must be disclosed. Therefore, the effect of simulating a zoom lens is not entirely abstract. Moreover, simulations following the teachings of Schrader et al. allow conclusions about the mounting size of the aperture stop and the light penetration based on its position and orientation in image space, from which the lens's sensitive characteristics can be derived. Furthermore, it is not shown how to handle light rays that are blocked at different locations within the lens (e.g., at the aperture stop or mounting portion), that is, light rays that are absorbed and therefore do not contribute to image generation. Summary of the Invention

[0021] Therefore, one object of the present invention is to develop the simulation method taught in the prior art such that the set of coefficients describing the lens is completely abstracted from the structure of the lens, and thus does not allow conclusions to be drawn about the structure of the simulated lens. Another object of the present invention is to develop the simulation method taught in the prior art such that if the focal point, focal length, or one or more continuous parameters of the simulated lens are changed, it does not require a computationally intensive recalculation of the coefficients of the transformation rule.

[0022] The primary objective is achieved through the features described below.

[0023] This invention relates to a method for generating the brightness contribution of image elements in an image by simulating an image representation of a scene using an optical imaging system, the optical imaging system including an image recorder and a lens located on a first surface.

[0024] The method includes the following steps:

[0025] - Provide a first data record, which includes data describing the effect of light on the lens to be simulated.

[0026] - Provide a second data record, which includes data about the point of incidence of light on the image recorder and about the virtual front surface.

[0027] - Provide conversion rules,

[0028] - By applying the transformation rule to the first data record and the second data record, the first intersection point of the ray with the virtual front plane and the direction of the ray at the first intersection point are calculated.

[0029] - Determine the brightness contribution of this ray.

[0030] - Stores information about the calculated brightness contribution of the ray.

[0031] in,

[0032] - This first data record includes data about the second surface, and

[0033] - The second data record includes data about the second intersection point of the beam and the second surface.

[0034] The second objective is achieved through the features described below.

[0035] In a preferred embodiment, the lens has at least one adjustable imaging parameter, and the second data record contains information about at least one adjustable imaging parameter of the lens.

[0036] Another advantageous embodiment is described by the features described below.

[0037] In a preferred embodiment, the adjustable imaging parameters include the lens focal length setting and / or focal length and / or magnification and / or field curvature.

[0038] In a preferred embodiment, the lens includes an aperture stop, preferably an aperture stop, and a second surface that coincides with the aperture stop, preferably with the aperture stop.

[0039] In a preferred embodiment, one of these adjustable imaging parameters describes at least one dimension of the aperture stop.

[0040] In a preferred embodiment, the aperture stop is at least approximately circular, and the information associated with the second intersection point of the ray and the second surface includes a normalized radius.

[0041] In a preferred embodiment, the first data record includes:

[0042] - Data related to at least one shielding area in the lens, and

[0043] - Data relating to the effect of a portion of the lens on at least one beam of light that extends between the at least one shaded area and the image recorder.

[0044] And before storing the brightness component of the at least one ray, the following steps are included:

[0045] - Calculate the third intersection point with the at least one shielding area.

[0046] - Check whether the at least one ray of light is absorbed by the third intersection point or is transmitted through the third intersection point.

[0047] - If at least one ray is absorbed, discard the ray or set the brightness component to zero.

[0048] The present invention also relates to a method for generating image elements, the method comprising the following steps:

[0049] - Select the point of incidence of the light on the image recorder.

[0050] - Select multiple different second intersection points on the second surface.

[0051] - Perform ray tracing as described in any of the preceding claims for each of these second intersection points.

[0052] - Sum the resulting brightness contributions, and

[0053] - Store the result of the summation.

[0054] The present invention also relates to a method for generating an image, characterized by the following steps:

[0055] - Select multiple image elements on this image recorder

[0056] - Calculate the brightness contribution of the light incident on each of these image elements using the method described above, and

[0057] - Store the results.

[0058] In a preferred embodiment, the brightness contribution of each of these simulated rays is determined by means of a pinhole camera image, which intersects the virtual front surface at the first intersection point, and the nature of the image is such that the image corresponds to an image generated by a pinhole camera placed at the respective first intersection point.

[0059] In a preferred embodiment, the virtual front surface coincides with the incident pupil of the lens to be simulated.

[0060] In a preferred embodiment, the second surface coincides with the virtual front surface.

[0061] In a preferred embodiment, at least one of these pinhole camera images is calculated by interpolation or extrapolation from other pinhole camera images.

[0062] The present invention also relates to a method for generating an image, the method comprising the following steps:

[0063] - Provides real images recorded by real cameras.

[0064] - Provides the virtual image generated as described above.

[0065] - Fuse or overlay at least a portion of the real image and at least a portion of the virtual image.

[0066] - Store the created image.

[0067] The adjustable lens parameters used for simulation at least approximately correspond to the lens parameters used during actual recording.

[0068] The present invention also relates to a method for generating an image sequence composed of individual images, the method comprising the following steps:

[0069] -Provide virtual scenes,

[0070] - Provide the camera positions associated with this virtual scene.

[0071] - Calculate the individual images of the image sequence according to one of the methods described above.

[0072] - Store the image sequence.

[0073] The present invention also relates to a computer program product that is adapted to perform the methods described above after being loaded into a computer.

[0074] Further features of the invention will become clear from the following description, the following claims and / or the accompanying drawings. Attached Figure Description

[0075] The present invention and its embodiments are described with reference to the following figures:

[0076] Figure 1 The principle of image recording is illustrated schematically.

[0077] Figure 2 The principle of the simulation according to the present invention is illustrated schematically.

[0078] Figure 3 The effect of optical imaging by the lens is illustrated schematically.

[0079] Figure 4 The structure of the first data record is illustrated schematically.

[0080] Figure 5 The construction of the second data record is illustrated schematically.

[0081] Figure 6 This schematically illustrates the principle of ray tracing simulation.

[0082] Figure 7 The optical structure of an exemplary lens to be simulated is shown schematically.

[0083] Figure 8 The positioning of the pinhole camera image relative to the incident pupil is illustrated schematically.

[0084] Figure 9 The diagram illustrates the arrangement of the pinhole camera image relative to the entrance pupil, in the form of a Fibonacci spiral.

[0085] Figure 10 The random arrangement of pinhole camera images relative to the entrance pupil is illustrated schematically.

[0086] Figure 11 An exemplary construction of an image simulation based on an abstract staircase object and an associated pinhole camera image of this staircase object is schematically shown.

[0087] Figure 12 An image of an abstract stepped object simulated according to the present invention is schematically shown, wherein one, several, and many pinhole camera images are used for this purpose, as well as the positioning of the pinhole camera relative to the incident pupil. Detailed Implementation

[0088] Description of the solution according to the invention

[0089] Figure 1The image recording principle is schematically illustrated. A given lens 1 to be simulated for photographic image recording includes a lens element 2 and an aperture stop 3, often of adjustable size. Each lens has a certain radius and is held by a non-transparent mounting base 4. An image recorder 100 (hereinafter also referred to as a sensor) is attached to a location depending on the specific design of the optics of the lens 1 to be simulated. The sensor 100 to be simulated can consist of multiple photodetectors (so-called pixels 5), which are typically arranged in a grid. Each of these pixels records an image element during recording. An image is optically generated at the location of the sensor 100, and in a real camera system, there is electronic or chemical detection of the corresponding incident light intensity at each pixel or image element: the recording of the image element. The sensor can be equipped with a grid-like color filter 101, for example, arranged in a so-called Bayer pattern, such that each pixel 5 can detect only one color. Another possibility is that each pixel 5 can, for example, detect all three primary colors, such as red, green, and blue. In the simulation, this corresponds to the registration and storage of at least calculated intensities in the processing computer system, and, if necessary, also to the simulation of the color of the incident light. Storage is preferably implemented in so-called random access memory (RAM), but can also be implemented on, for example, so-called flash memory or a hard disk. Light rays emanating from a point in the scene to be imaged and potentially contributing to image representation pass through the lens and are refracted at the interfaces of the lens elements in the process, that is, their directions change according to the laws of optical refraction in their respective cases. A portion of these rays are incident on the partial mounting 4 of lens 1 and are absorbed without contributing to image representation. Absorption of light at the partial lens is called shading and results in so-called vignetting. Another portion of the light rays may be incident on the aperture stop 3, where they are absorbed without contributing to image representation. Yet another portion of the light rays passes through the lens and is incident on the image recorder 100. These rays contribute to image representation.

[0090] The lens 1 to be simulated may include one or more adjustable parameters on the lens 1. In particular, these parameters may be one or more of the following:

[0091] - Aperture stop: At least one approximately circular aperture stop is mounted in the lens, and the optical transmission diameter of the aperture stop can be changed by the user. This is often made by a so-called variable aperture stop.

[0092] - To set the focus, the lens may be equipped with one or more lens elements that can be displaced along the optical axis relative to the image detection plane, and preferably also relative to other optical elements.

[0093] - To variably adjust the focal length, the lens can be equipped with one or more lens elements that can be shifted along the optical axis relative to the image detection plane, and preferably also relative to other optical elements. Instead of focal length, the magnification of the lens can also be used as a continuous input variable. Magnification is a scaling factor derived from the ratio of the image size of the object on the sensor or film plane to the size of the object being focused on in this imaging. It has been found that choosing magnification as an input variable is particularly well suited to the method according to the invention.

[0094] - To variably adjust the field curvature, the lens may be equipped with one or more lens elements that can be shifted along the optical axis relative to the image detection plane, and preferably also relative to other optical elements.

[0095] - To allow for variable manipulation of the wavefront of light passing through the lens, the lens can be equipped with displaceable free-form elements. As an example, these elements can be one or two so-called alpha elements, which can be displaced perpendicular to the optical axis.

[0096] This adjustable parameter on the lens 1 to be simulated corresponds to a continuously adjustable input variable, preferably in lens simulation. Known teachings from Schrader et al. regarding the simulation of optical imaging through a lens serve as the starting point for the method according to the invention. In these known teachings, continuously adjustable input variables are defined before calculating the coefficients that model the lens and determine its values. If these values ​​change, the coefficients for modeling the lens need to be recalculated in this simulation method.

[0097] The simulation of optical imaging of the lens to be simulated according to the present invention is based on transformation rule 900. This is in Figure 2 The diagram is schematically shown. In the calculation 4000 performed by the computer 300, this transformation rule 900 processes data from one or more input data records 1000 and transforms these data into output data records 2000. The transformation rule 900 may, in particular, include polynomials. The transformation rule 900 may, in particular, include spline functions. Figure 3 The optical imaging effect shown is expressed as follows: a simulated ray 800 (emitted by a simulated object and intersecting a surface different from the simulated sensor 100 at a specific location and in a specific direction as incident ray 2010) passes through a lens and, as an outgoing ray 2020, is incident on a point of the simulated sensor 100 in a specific direction, which is typically deviated from the original ray direction. The resulting beam shift and the resulting change in the direction traversing ray 800 are effects of lens 1. The rays referred to as incident ray 2010 and outgoing ray 2020 are portions of ray 800 passing through the lens. Some information required to fully describe ray 800 may also be information about polarization, one or more wavelengths, or color, each of which is related to intensity.

[0098] The effects on the transverse ray 800 caused by lens 1 can include ray deflection, change of direction, change of polarization, change of color, attenuation of ray intensity, splitting of ray into partial rays, or other changes. Multiple such effects can also occur simultaneously with respect to a single ray 800. As an example, splitting of ray into partial rays can be caused by partial transmission and reflection at optical refractive surfaces or coating surfaces, by splitting into normal and anomalous rays in an optically anisotropic crystal, or by diffraction at diffraction structures or holograms. All of these effects are effects of lens 1 detected according to the present invention.

[0099] For practical reasons, optical simulation is often performed by tracing light rays in the opposite direction, that is, starting from the sensor. This description of the invention follows this method, but explicitly reiterates the fact that, according to the invention, the simulation of light rays in the direction of light is captured by the invention.

[0100] According to the present invention, the analog sensor 100 is located on a first surface (sensor surface 110). The light 800 to be simulated is incident at the incident point 111 on this surface.

[0101] According to the present invention, the first data record 1010 (input data record) specifically contains data suitable for describing the effect of the simulated lens on the simulated light ray 800 passing through the simulated lens in conjunction with the transformation rule 900. In this case, the characterization of the simulated lens is contained only in the first data record 1010, and not in the transformation rule 900. In contrast, the data record 1010 of the transformation rule 900 contains neither data about the light ray to be simulated nor adjustable parameters on the simulated lens. The transformation rule 900 and the data record 1010 together can be referred to as the virtual lens 901, because sharing only this information is sufficient to enable the user to create a rendering of a virtual scene using their own data. Figure 4 As illustrated, the first data record 1010 may comprise one or more partial data records 1010r, 1010g, 1010b, each assigned to a specific wavelength of light or a specific wavelength band. This takes into account imaging chromatic aberration, which occurs in a real lens due to physical reasons. Preferably, three such partial data records 1010r, 1010g, 1010b remain available, particularly for red, green, and blue. However, more or fewer of these partial data records may also be used. According to the invention, color effects can also be calculated using a differential method from only one data record 1010, especially when chromatic aberration is small.

[0102] According to the invention, the first data record 1010 further includes data including information about a first surface or virtual incident surface or virtual front surface 1011, which intersects with a simulated ray of the imaging beam path or its continuation at a first intersection point 1013. This surface is preferably located upstream or downstream of the simulated sensor. Preferably, this first surface or virtual front surface 1011 is a rotationally symmetric surface whose axis of symmetry corresponds to the optical axis OA of the simulated lens and is preferably located upstream of the actual surface of the lens to be simulated. In particular, but not exhaustively, this list includes spherical surfaces and other surfaces formed by a rotating conical section and planes. The first surface or virtual front surface 1011 may coincide with the lens element surface of the simulated lens, but this is not necessarily the case. Preferably, the virtual front surface 1011 does not precisely coincide with the lens element surface of the simulated lens. Another preferred location for the virtual front surface is the entrance pupil of the lens. Particularly preferably, the virtual front surface 1011 is positioned such that it is further away from the sensor 100 than the lens element surface furthest from the sensor. Preferably, the virtual front surface is selected such that its position and shape remain unchanged as the focal length, focus, or other adjustable lens parameters change. This reduces the complexity of the simulation and increases confidentiality, because the virtual front surface 1011 therefore does not contain any information about the actual construction of the lens to be simulated.

[0103] Data recordings may include polynomials. Depending on the desired fitting accuracy for ray position or direction and information about the occlusion surface from which the vignetting is generated, the polynomial may include at least 5, at least 10, at least 20, or at least 30 coefficients. As an example, for ray position and direction, a sparse polynomial with 20 coefficients of no more than order 6 may be used, and for occlusion information, a sparse polynomial with 5 coefficients of no more than order 3 may be used.

[0104] Figure 5 The schematically illustrated second data record 1020 specifically includes information 1021 about the simulated ray 800, consisting of information about the incident point 111 of this simulated ray 800 on the sensor surface 110 and the direction information 112 of this ray. The information about the incident point 111 can be provided by a three-dimensional vector in a coordinate system associated with the simulated lens, or by two-dimensional parameters that parametrically describe the position on the sensor surface 110, such as... Figure 6 As shown in the figure. This is an example, but not exhaustive; it could be azimuth and radius, or normalized radius, or elevation. The direction information 112 of the simulated light ray can be given exemplarily, but not necessarily exhaustively, by specifying a three-dimensional vector or by specifying two angles associated with the coordinate system related to the simulated lens. Another particularly preferred direction representation according to the invention can also be constituted by a parameterized specification of the second incident point 201 of the second surface 200, such as... Figure 6 As shown, it, combined with the incident point 111, generates a bijective relation for direction information 112. Regarding the parameterization specification of the second incident point 201 with respect to the second surface 200, it is found that the radius r, normalized to 1... n The specification is particularly advantageous because, in that case, the directional information 112 is determined solely by the angles between 0 and 360 degrees and the angles between 0 and r. n radius r within the range ≤1 n The configuration is complete, and the specification of the axial position of the second surface 200 is no longer required. Further calculations only require two sets of parameters, such as the angle and the normalized radius, while the axial position can be disregarded. It was found advantageous to choose the position of the aperture stop for the second surface 200, as most light is often blocked there, especially when the aperture stop of the lens is reduced. Then, the maximum normalized radius r... n =1 is selected to simulate a lens with an open aperture, while r is selected when simulating a lens with a narrowed aperture. n The corresponding smaller value. The second data record 1020 may include information 1022 about the aperture reduction of the lens to be simulated. Since the actual axial position of the second surface 200 and the actual size of the aperture stop are unknown, or at least do not need to be known, due to parameterization, while by appropriately selecting the radius r n This allows for the reproduction of the effect of the aperture stop, thus ensuring maximum confidentiality regarding the actual location of the aperture stop.

[0105] In addition to preferred information about the radius of the approximately circular aperture, data describing variations in the aperture stop shape may also be included. As an example, the variable aperture of the Carl Zeiss Sonnar 2.8 / 135 (with a C / Y mount and a focal ratio of 4) exhibits a significant deviation from rotational symmetry. The transmission area of ​​an aperture stop or stop is described by at least one dimension, such as the radius; however, other dimensions may be used to describe more complex aperture forms.

[0106] Furthermore, the second data record 1020 may include information 1023 relating to one or more other imaging parameters that are adjustable on the lens and affect imaging. According to the invention, this includes (by way of example, but this list is not exhaustive) parameters simulating the focal length and / or focusing distance of the lens and / or parameters for variable adjustment of the field curvature. As an alternative to the focal length of the lens, it may also include an imaging scale relating to the imaging scale of the object focused onto the sensor plane. Additionally, the data record 1020 may include information about beam deflection caused by variable mirrors or other optical elements such as diffraction elements, polarizers, optical filters (e.g., neutral density filters), frequency filters, variable beam splitters, or Alvarez plates or other movable freeform surfaces.

[0107] In addition, the second data record 1020 may include information 1024 related to one or more wavelengths, wavelength ranges or colors of the light to be simulated.

[0108] Therefore, the second data record 1020 contains all variables selected or influenced by the user of the simulation, such as adjustable parameters on the simulated lens and information 1021, 1024 about the light rays to be simulated. In contrast, the first data record 1010 contains only data on the lens imaging behavior for all conceivable combinations of adjustable parameters 1022, 1023. It goes without saying that during the training or optimization phase for creating the first data record 1010, the full range of acceptable parameters 1022, 1023 is considered. Typically, this range is specified by the likelihood of a realistic model of the lens to be simulated.

[0109] In order to perform a simulation according to the invention on a given lens, a first data record 1010 is required before performing the actual simulation. This first data record describes the imaging behavior of the lens to be simulated and the virtual front surface 1011 of the lens 1 to be simulated by applying transformation rule 900. The data record 1010 can be obtained by calculation, as described below.

[0110] Initially, a second surface 200 is selected between the sensor's position and the position of the virtual front surface. While considering possible adjustable parameters 1022, 1023 and the second surface 200, data recording 1010 is pre-calculated. For this purpose, the following steps are performed for one or more wavelengths or wavelength ranges:

[0111] - Ray tracing is performed using the precise optical construction of the lens to be simulated, with multiple settings for multiple rays and one or more adjustable parameters (such as focal length or focus distance).

[0112] - Data record 1010 is calculated using an optimization algorithm. For this purpose, known iterative optimization or fitting algorithms from the prior art can be used. A particularly suitable algorithm is called the "orthogonal matching tracking" algorithm. The result of this fitting or optimization process is data record 1010, which provides the position and direction of the output ray for a given input ray and given adjustable lens parameters, while also taking into account the second data record 1020.

[0113] In addition to this optimization process, one or more locations can be identified in the lens 1 to be simulated where light is blocked and thus absorbed, for example, upon entering the lens housing or the aperture stop. The aperture stop 3 (whose diameter and / or shape can be selected) is a separately labeled and preferred stop or shielding surface 210. Many lenses contain such stops. Information about all the shielding surfaces 210 considered is included as part of a plurality of partial data records 1030 are formed. Each of these partial data records 1030 contains information about coefficients applicable to the partial simulation of the lens to be simulated from the sensor surface to the corresponding shielding surface 210. The actual axial positions of these shielding surfaces 210 are irrelevant. The light position can be specified for each shielding surface 210 by means of a parametric method, consisting, for example, of azimuth and radius or elevation angle, or particularly preferably a normalized radius.

[0114] - Data record 1010 must include at least information about coefficients that are suitable for simulating the light path from the sensor surface to the virtual front surface 1011 or to the lens front surface or to the entrance pupil of lens 1.

[0115] - The virtual front surface 1011 can be predefined. However, the virtual front surface 1011 can also be optimized as part of the optimization process of the data record 1010.

[0116] As an alternative to this optimized calculation method, data recording 1010 can also be obtained by measuring at least one physical embodiment of the lens to be simulated. For this purpose, an exemplary object can be imaged using the lens, and the resulting image can be recorded. This can be performed for multiple settings of parameters adjustable on the lens, such as focus and / or focal length / magnification and / or aperture stop opening. The desired set of coefficients forming data recording 1010 can then be established by reverse calculation from the obtained image. Another option for obtaining data recording by measurement is to illuminate a single beam of light (e.g., a laser beam) of suitable wavelength through the physical embodiment of the lens to be simulated. The effect of the lens on the beam can be detected by measurement at the other end of the lens. This can be performed for other wavelengths of the beam and different incident points and directions. The desired set of coefficients forming data recording 1010 can then be established by reverse calculation from the measurements thus obtained.

[0117] It goes without saying that the value of data record 1010 obtained through calculation and / or measurement can still be changed subsequently based on experience, for example, to reproduce manufacturing tolerances or defects in the physical embodiment of the lens.

[0118] In order to perform the simulation according to the invention for imaging a given lens 1, a computer program product is loaded onto computer 300, which positions computer 300 in a position where it can perform calculations according to conversion rule 900. A previously pre-calculated first data record 1010 (which contains information about the lens to be simulated and the virtual front surface 1011) is also loaded onto computer 300.

[0119] A data record 1020 is generated, which consists of information 1021 about the light ray 2010 to be simulated incident on the sensor and information about one or more adjustable parameters 1022, 1023 on the lens. In addition to color or wavelength, the information 1021 about the light ray 2020 to be simulated includes the incident point 111 and direction information 111 of the light ray on the sensor 100, which can also be provided from information about a second intersection point 201 with the second surface 200 (which may contain an aperture stop). The incident point 111 preferably corresponds to the position of a simulated pixel of the simulated sensor 100. Preferably, only those rays incident on the second surface 200 are considered within the area transmitted according to the information 1022 regarding the setting of the aperture stop. In the case of a circular aperture stop with a normalized radius rb, this can be achieved by generating and tracking only the light rays to be simulated with an incident point having a radius r < rb. In the case of complex geometry of the aperture stop 3 or the shielding surface 210, light rays incident on the second surface 200 outside the area transmitted according to the information 1022 regarding the setting of the aperture stop can be generated, but these rays are discarded because only light rays from the transmission area of ​​the aperture stop can reach the sensor. The procedure described herein for the aperture stop can be similarly performed for all other shielding surfaces 210 that may exist in the lens.

[0120] Then, to simulate a single ray 800 that contributes to the image representation to be simulated at the selected pixel, the computing unit of computer 300 performs calculations according to conversion rule 900 using input data records 1010, 1020. This produces output data record 2000, which contains the incident ray 2010 being converted by the simulated lens into ray 2020 incident on the sensor as in input data record 1021. If one or more shading surfaces (e.g., aperture stops) exist in the lens to be simulated, the simulation can be run in multiple partial steps, where calculations are performed from one of the sensor surface 110 to the shading surface 210 in each case. Separate partial steps are calculated for each shading surface 210, and the order in which these partial steps are executed is irrelevant. Preferably, surfaces causing significant shading are considered before those causing minor shading. The partial steps can also be run in parallel. A test step is then performed in each case to determine whether the ray is absorbed by the shading surface at the third intersection point 211 or transmitted through the shading surface. Absorbed ray is discarded, and transmitted ray is further traced. Ray tracing from the sensor surface to the virtual front surface 1011, the lens front surface, or the entrance pupil is also achieved. The output data record 2000 obtained in this way contains information about the rays 2010 incident on the virtual front surface 1011, the lens front surface, or the entrance pupil. This includes the first intersection point 1013 of the ray 2010 with the virtual front surface 1011 and direction information, such as by a three-dimensional vector or by a parameterized representation of the intersection point with another surface or by two angles associated with the coordinate system of the simulated lens.

[0121] To calculate a color image, the simulation steps for simulating individual rays can optionally be performed multiple times, preferably three times, wherein in each of these ray tracing calculations, it is assumed that different wavelengths or different wavelength bands of the simulated light and different portions of data records 1010r, 1010g, 1010b can be used for simulation. For simulated lenses with small chromatic aberration, ray tracing for only one data record 1010 may be sufficient, wherein differential correction of the resulting ray direction and / or the resulting ray position is performed for at least one considered wavelength. A simulation step consisting of multiple single-ray simulations for different wavelengths or of single-ray simulations followed by differential correction steps for other colors will hereinafter be referred to as a multicolor ray simulation step.

[0122] Based on the output data record for a given ray 2020, the intensity of the incident light 2010 can be derived from a model of the scene to be imaged, from which the intensity contribution or equivalent brightness contribution of this simulated ray to the signal of the considered pixel is generated at the wavelength or color respectively considered.

[0123] For the same target pixel, the monochromatic or multicolor ray simulation step is performed for multiple rays to be simulated. These rays to be simulated are selected such that they are emitted from different positions on the second surface 200, preferably from the aperture stop 3.

[0124] For each of these simulated rays, the brightness contribution for a given wavelength (the intensity of the incident ray 2010) is determined from information about the light from object features of the modeled scene observed in this direction, and all brightness contributions of the image elements are summed. Brightness contributions from occluding rays are discarded or set to zero so that they make no contribution. The intensity or brightness level of the image elements thus obtained is stored in computer memory, preferably in random access memory (RAM), or in flash memory or on a hard disk.

[0125] The described simulation steps are repeated for the other pixels of the sensor until the first image is fully constructed.

[0126] To generate an image sequence for a cinematic image sequence, the described steps are repeated for other images. In this case, the scene to be modeled, the position of the camera to be simulated, and / or adjustable lens parameters (e.g., focus and / or focal length and / or aperture stop) may change. Such changes are particularly necessary for cinematic effects, where, for example, the observer's attention shifts from one object feature to another via focus shift. As described, there is no recalculation of data record 1010 in the case of such changes. The changes in adjustable lens parameters are only included in data record 1020, which provides a significant speed advantage compared to prior art methods and makes the rendering of such scenes by means of the method according to the invention particularly efficient.

[0127] As described, this can still be followed by a synthesis method step, in which a simulated image or a sequence of simulated images is fused with the actual recorded image.

[0128] Figure 7 The illustration shows a schematic lens element portion of an exemplary lens to be simulated, with a focal point set at "infinity". (As shown from...) Figure 7 As shown in the diagram, lens 1 includes first, second, third, and fourth lens elements 7, 8, 9, and 10, which are arranged sequentially along the optical axis OA of lens 1, starting from the object side. The first and third lens elements 7 and 9 each have positive refractive power, and the second and fourth lens elements 8 and 10 each have negative refractive power.

[0129] Therefore, according to the invention, it is used for simulation Figure 7 The method for achieving the effect of a lens may include the following steps:

[0130] First, a virtual front surface 1011 is defined. The virtual front surface 1011 is located at a predetermined distance in front of the image sensor 100 and has a predetermined and / or optimized radius of curvature. In an exemplary embodiment, the distance from the sensor is 60.0 mm, and the radius of curvature is 13.365 mm; however, other values ​​may be chosen for this purpose, depending on the application and / or requirements. In the second step, a lens model is created through a training phase. For this purpose, multiple training or validation rays are generated, each ray having a defined wavelength. As an example, these may be more than 5000 training rays, but preferably more than 10000, and / or more than 2000 validation rays, but preferably more than 4000, each wavelength and / or focal point and / or one or more other variable lens parameters, taking into account. In an exemplary embodiment, focal settings are generated. The exemplary embodiment assumes nine approximately uniformly distributed focal settings and simulated light with a wavelength of 440 nm.

[0131] The next step is to determine the shielding surfaces. In an exemplary embodiment, these are the radius of the front surface of the first lens element 7 (8.7 mm) and another shielding surface 11 (6.0 mm).

[0132] In the next step, an optimization method is used to create a parameter set for the abstract mathematical model given by the transformation rules. In an exemplary embodiment, this is a sparse polynomial with 20 coefficients of no more than order 6 for ray position and ray direction, and a sparse polynomial with 5 coefficients of no more than order 3 for occlusion information.

[0133] This exemplary model is optimized using the orthogonal matching pursuit method, and the specified coefficients are calculated in the process.

[0134] The resulting output data includes at least one value from the following: the radius of curvature of the virtual front surface, the distance between the virtual front surface and the sensor, and the minimum focal ratio (F). min ), Supported focal range (d), focal length (f).

[0135] Users of the exemplary lens model obtain the following metadata:

[0136] -Radius of curvature of the virtual front surface: 13.365mm

[0137] - Distance between the virtual front surface and the sensor: 60.0mm

[0138] -Minimum focal ratio: F min =2.87

[0139] - Supported focal range: d = 507mm to infinity

[0140] Focal length: f = 50.0mm

[0141] Input variables for an exemplary lens model:

[0142] -x s y s The sensor's light position is defined within the range of -18.0mm ≤ x. s ≤18.0mm and -12mm≤y s ≤12mm

[0143] -x a y a The position of the light ray in the virtual aperture, defined range: for focal ratio F, x a 2 +y a 2 <(F) min / F) 2

[0144] -β=f / (fd)

[0145] Output variables of an exemplary lens model:

[0146] -x f y f : The position of the ray projected onto the tangent plane at the vertex of the virtual front surface, in mm.

[0147] -u f v f The position of the ray projected onto the tangent plane of the virtual front surface at the ray point.

[0148] -x v1 y v1 x v2 y v2 : The position of the light source on the shading surface. If x is at one of the shading surfaces where the vignetting is generated. 2 +y 2 If the value is greater than 1, then the light is blocked.

[0149] In addition, there is fitting data for the parametric optics of the lens to be simulated in the exemplary embodiment:

[0150] x f

[0151] -0.459663284 1 0 0 0 0 1 8.768720761 0 0 1 0 0 1 -1.304539732 1 0 0 0 1 2 0.000110631 3 0 0 0 0 3 -0.00496582 2 0 1 0 0 3 0.000109726 1 2 0 0 0 3 -0.005078146 1 1 0 1 0 3 0.091802222 1 0 2 0 0 3 0.067526222 1 0 0 2 0 3 0.025031505 0 1 1 1 0 3 -0.07933075 0 0 1 2 0 3 0.000671128 3 0 0 0 1 4 0.000725495 1 2 0 0 1 4 1.134481527 0 0 3 0 1 4 -5.36E-05 2 1 1 1 0 5 0.137138593 2 0 1 0 2 5 0.158010585 1 1 0 1 2 5 -0.784407072 1 0 0 2 2 5 3.288014989 1 0 0 0 4 5 0.000741927 3 0 2 0 1 6

[0152] u f

[0153] 0.014551901 1 0 0 0 0 1 -0.666745323 0 0 1 0 0 1 0.06893195 1 0 0 0 1 2 0.000145798 2 0 1 0 0 3 7.79E-06 1 2 0 0 0 3 -0.000358482 1 1 0 1 0 3 -0.00282992 1 0 2 0 0 3 0.002553763 1 0 0 2 0 3 -0.000240285 0 2 1 0 0 3 0.013381898 0 1 1 1 0 3 0.021614836 0 0 3 0 0 3 -0.159397768 0 0 1 2 0 3 -1.127168311 0 0 1 0 2 3 4.83E-05 1 2 0 0 1 4 -0.002460254 1 1 0 1 1 4 -0.00172234 0 2 1 0 1 4 -7.99E-06 1 2 2 0 0 5 0.000348182 1 1 2 1 0 5 -0.176833532 1 0 0 2 2 5 -0.47254192 0 1 1 1 2 5

[0154] x v1

[0155] -0.027987126 1 0 0 0 0 1 1.011147804 0 0 1 0 0 1 -0.000368083 2 0 1 0 0 3 -0.000342116 1 1 0 1 0 3 0.005565995 1 0 0 2 0 3

[0156] x v 2

[0157] 0.031357313 1 0 0 0 0 1 0.926341281 0 0 1 0 0 1 0.043996073 1 0 0 0 1 2 -0.257282737 0 0 1 0 1 2 3.06E-06 3 0 0 0 0 3

[0158] No figure shows y f v f y v1 and y v2 The coefficients, due to symmetry, are derived directly from x. f u f x v1 and x v2 The coefficients are derived from this. Here,

[0159]

[0160] Among them, coefficients c and x s The exponents i and y s The exponents j and x a The exponents k and y a The exponent 1 of β and the exponent m of β. This also applies to u in the second table. f It can also be written like this:

[0161] x f

[0162] -0.459663284 1 0 0 0 0 1 8.768720761 0 0 1 0 0 1 -1.304539732 1 0 0 0 1 2

[0163] It should be understood that information about the scene to be imaged is needed to calculate the intensity or brightness of pixels and their color. If information about the direction of the light rays to be simulated incident on the first intersection point 1013 on the virtual front surface 1011 is available, the point from which the light rays incident on the modeled object are emitted can be calculated in reverse. Information about the direction of the simulated light rays and the point of incidence or the first intersection point 1013 can be obtained, for example, by the simulation method according to the invention, but can also be obtained by other methods that provide equivalent results. Conventional ray tracing is an example. Instead of the first intersection point 1013 with the virtual front surface 1011, the point of incidence on the first optically effective surface of the simulated lens, the point of incidence in the entrance pupil of the simulated lens, or the point of incidence on a surface located further in front of the lens and / or closer to the scene to be modeled can also be used.

[0164] One option for determining the object point from which the ray 2010 incident on the lens originates is conventional ray tracing. Its advantage is that the resulting representation is physically correct and therefore realistic. The disadvantage is that this calculation is very complex and therefore requires a lot of computation time. It is desirable to have a method that allows for much faster access to information about the color and intensity or brightness of the light rays emitted from the modeled object and incident on the virtual front surface 1011 in front of the simulated lens 1 or the lens itself, compared to when using ray tracing.

[0165] Therefore, another object of the present invention is to provide a method that makes available, physically nearly accurate information about the color and intensity or brightness of light emitted from a modeled object and incident on a virtual front surface in front of a simulated lens or lens much faster than when using ray tracing.

[0166] This objective according to the invention is achieved by combining the described method for simulating a lens with the features of claim 10 and the features of the dependent claims that reference this claim.

[0167] In this context, it is expected that the method according to the invention can benefit from the specific characteristics of GPUs related to speed and parallelization.

[0168] Understanding the direction, intensity, and color of the ray 800 incident on the incident surface 3000 is equivalent to understanding the so-called light field at this incident surface. The properties of the incident surface 3000 allow it to include, for example, an incident pupil, the front surface of the front lens element, or a virtual front surface 1011. In principle, any other surface can be chosen, provided that the ray 800 facilitating image creation passes through this surface. Advantageously, the incident surface 3000 is chosen to be larger than, for example, the incident pupil, the front surface of the front lens element, or the virtual front surface 1011. It is particularly advantageous to choose the incident surface 3000 such that it includes the incident pupil of the lens to be simulated or the front surface of the lens 1 to be simulated, or the virtual front surface 1011, because in this case, an approximate light field in the incident surface 3000 can be used without another conversion step for the simulation of image creation.

[0169] If the light field is known, it can be correctly modeled using an image of lens 1 to be simulated. Understanding the light field, or part of the light field, associated with the image representation can be obtained through ray tracing. However, this is computationally intensive and therefore slow. However, calculating a realistic image does not necessarily require a complete understanding of the light field; instead, a sufficiently good approximation is sufficient.

[0170] The method according to the invention for obtaining such a sufficiently good approximation is described below.

[0171] Multiple ideal images are generated at different locations on the incident surface 3000. This can be achieved by calculating these images using pinhole camera models for attaching pinhole cameras to the corresponding locations. In addition to the ideal images, it is also useful to create depth maps corresponding to the scene to be modeled. Advantageously, the pinhole camera images can be arranged such that they are each located at the position where the incident simulated light rays are incident on the incident surface 3000. In this case, information about the intensity and color of the light rays can be directly collected from the pinhole camera images based on the knowledge of the direction of this beam. This can be achieved either by generating pinhole camera images at the corresponding locations and then performing ray tracing from the pinhole camera's position to a specific pixel via a lens, or by initially performing ray tracing to a specific location on the incident surface 3000 and then generating the corresponding pinhole camera image.

[0172] To increase the number of available pinhole camera images, new pinhole camera images can be calculated by interpolation or extrapolation from existing pinhole camera images and at least one associated depth map. Here, at least one depth map is needed to generate perspective-corrected interpolated pinhole camera images. For example, these interpolated images can be generated using a method known as "screen-space ray tracing".

[0173] In principle, this interpolation can already be performed from a single pinhole camera image with an associated depth map. However, at least two, particularly preferably three or more, pinhole camera images are preferred. Figure 8 The following examples illustrate the cases of four, two, and one rendered pinhole camera images 5100 and the resulting interpolated or extrapolated pinhole camera images 5200 associated with the incident pupil 5000.

[0174] New pinhole camera images can also be obtained using artificial intelligence (AI) methods, such as with the help of neural networks, with or without available depth maps.

[0175] The pinhole camera can be positioned in one of the following particularly advantageous configurations:

[0176] - A fixed spiral mesh with constant density, for example, in the form of... Figure 9 The so-called Fibonacci spiral shape is shown. This pinhole camera is preferably positioned within the transmission region of the incident surface at a depth of 3000°.

[0177] - Purely random arrangement, such as Figure 10 As shown.

[0178] - At least three pinhole cameras are located outside the transmission area of ​​the incident surface 3000, such that the incident pupil or front surface or virtual front surface 1011 is located within the polygon described by the positions of the pinhole cameras.

[0179] The location of the pinhole camera images used can be adapted to the scene being modeled to obtain a sufficient number of perspective views and to avoid artifacts. As an example, viewing through a thin tube might require seeing both a perspective view of the inside of the tube and other perspective views of the tube from the outside.

[0180] Naturally, the invention also includes any other arrangement of pinhole cameras. Fixed positions with an approximately constant density improve the parallelism of the method, particularly on GPU-assisted computing systems, and reduce noise in the resulting images. The quality of the resulting images increases with the number of ideal images used. This is particularly applicable to regions with significant blur, as the images of defocused points forming the bokeh are particularly complex. It has been found advantageous to heuristically adjust the position density by using information from the depth map and comparing this with the focal point setting to be simulated for the lens to be simulated. The positions of the pinhole camera grid can also be randomly rotated or perturbed, which may produce even higher image quality. It is also advantageous to statically or dynamically adjust the density of the pinhole cameras to adapt to heuristic quality measurements of the elements in the resulting image.

[0181] Information about the light field is obtained by interpolating information from various ideal images. This can be done individually for each desired wavelength or just for a single wavelength, with information for other desired wavelengths approximated differentially.

[0182] The simulation of image generation is achieved by simulating the contribution of all image elements or pixels of the sensor 100 to be simulated. To this end, the light contribution of a pixel is integrated by initially determining the incident directions of multiple rays incident on the incident surface 3000 for each ray. Rays blocked in lens 1 are discarded. The multiple rays can preferably be selected such that the incident surface 3000 is sufficiently uniformly penetrated by the light. Another preferred selection of multiple rays can be such that they intersect the incident surface 3000 precisely at the location where the ideal image exists. Including weighting factors may be advantageous when converting the interpolated ideal image into light color and intensity, since each image pixel corresponds to a light cone of a different size. As an example, pixels at the edges of the image recorder cover a smaller angular range than pixels at the center.

[0183] This simulation is preferably performed using the parametric method described in the invention, but other simulations, such as ray tracing, may also be used.

[0184] For each simulated ray, its brightness contribution at the corresponding pixel is derived from knowledge of the ray's direction, its position on the incident surface 3000, and the light field or approximate light field. This is achieved by evaluating the corresponding associated pinhole camera image. The brightness contributions of all simulated rays are summed at their respective pixels, and the result is an image of the scene to be modeled.

[0185] The beam direction at a given incident point on the imaging system and incident surface 3000 depends on the wavelength. To determine the brightness contribution of light from the light field or approximate light field, separate calculations can be performed for each desired wavelength or color. If the direction variation is small, it may be sufficient to assume only one calculation rule for the dominant wavelength and make only minor changes to this rule for other wavelengths. First, the relative position of the dominant wavelength W1 in the light field is determined, and thus the brightness contribution or intensity contribution is determined. Then, for the same incident point of the simulated light at the imaging system and incident surface 3000, the relative position used by W1 is used as a starting point for finding the correct relative positions for other wavelengths. Then, one or more brightness contributions can be calculated for one or more wavelengths. As an example, this can be achieved by interpolation, preferably linear interpolation, for selected wavelengths. By applying this procedure to multiple image elements of the imaging system, an image of the scene to be modeled, including the polychromatic aberrations of the lenses, can be obtained.

[0186] The accuracy of beam direction determination by parametric optics is crucial to the quality of the resulting simulated image. This stems from the laws of geometric optics, particularly the intercept theorem. Given the same imaging system and the same point of incidence on the incident surface 3000, the directional differences of light rays of different wavelengths or colors are linear in the first approximation. Depending on the chosen parameter representation, these differences can be fitted using a reduced set of parameters while maintaining overall accuracy, resulting in less computation time required during the evaluation of the parametric function.

[0187] Figure 11 The construction of an image simulation based on an abstract stepped object and its associated pinhole camera image is illustrated schematically by way of example. A set of objects 5600 arranged in a stepped manner, recorded by a camera 5500, is simulated such that each step is at a different distance from the camera 5500. One step is in the camera's focus, while the others are out of focus. An image of the stepped object 5700 recorded with an arbitrarily small aperture corresponds to a pinhole camera image with infinity depth of field. In this case, each step is imaged with the same sharpness. Figure 12 A comparison of three simulations using the method according to the invention is shown.

[0188] The simulated image can be blended with the actual recorded image. According to the invention, the simulated image is generated by lens simulation that simulates the lenses used in the actual recording, and as a result, the resulting blended image has a particularly harmonious image impression.

[0189] Overview of the solution according to the invention

[0190] A) A method for generating the brightness contribution of picture elements of an image by simulating an image representation of a scene using an optical imaging system, the optical imaging system including an image recorder (100) and a lens (1) located on a first surface (110).

[0191] The method includes the following steps:

[0192] - Provide a first data record (1010) which includes data describing the effect of light rays (800) on the lens (1) to be simulated.

[0193] - Provides a second data record (1020) including data about the incident point (111) of the light ray (800) on the image recorder (100) and data about the virtual front surface (1011).

[0194] - Provide conversion rules (900),

[0195] - By applying the transformation rule (900) to the first data record (1010) and the second data record (1020), the first intersection point (1013) of the ray (800) with the virtual front plane (1011) and the direction of the ray (800) at the first intersection point (1013) are calculated.

[0196] - Determine the brightness contribution of this ray (800).

[0197] - Store information about the calculated brightness contribution of this ray (800).

[0198] in,

[0199] - The first data record (1010) includes data about the second surface (200), and

[0200] - The second data record includes data about the second intersection point (201) between the beam (800) and the second surface (200).

[0201] B) According to the method described in A),

[0202] in,

[0203] - The lens (1) has at least one adjustable imaging parameter (1022, 1023), and

[0204] - The second data record (1020) contains information about at least one adjustable imaging parameter (1022, 1023) of the lens (1).

[0205] C) According to the method described in B),

[0206] in,

[0207] The adjustable imaging parameters (1023) include the focal length setting and / or focal length and / or magnification and / or field curvature of the lens (1).

[0208] D) Based on the methods described in A), B), or C).

[0209] in,

[0210] The lens (1) includes an aperture stop, preferably an aperture stop (3), and

[0211] The second surface (200) coincides with the aperture stop, preferably with the aperture stop (3).

[0212] E) According to the method described in D),

[0213] in,

[0214] One of these adjustable imaging parameters (1022) describes at least one dimension of the aperture stop (3).

[0215] F) According to the method described in E),

[0216] in,

[0217] The aperture stop (3) is at least approximately circular, and the information associated with the second intersection point (201) of the ray (800) and the second surface (200) includes the normalized radius.

[0218] G) The method according to any one of A) to F).

[0219] in,

[0220] The first data record (1010) includes:

[0221] - Data related to at least one shielding area (210) in the lens (1), and

[0222] - Data relating to the effect of a portion of the lens (1) on at least one beam (800),

[0223] The beam extends between the at least one shielded area (210) and the image recorder (100).

[0224] And before storing the brightness component of the at least one ray (800), the following steps are included:

[0225] - Calculate the third intersection point (211) with the at least one shielding area (210),

[0226] - Check whether the at least one ray (800) is absorbed by the third phase intersection (211) or is transmitted through the third phase intersection.

[0227] - If at least one ray (800) is absorbed, then discard the ray (800) or set the brightness component to zero.

[0228] H) A method for generating picture elements for an image.

[0229] The method includes the following steps:

[0230] - Select the incident point (111) of the light ray (800) on the image recorder (100),

[0231] - Select multiple different second intersection points (201) on the second surface (200),

[0232] - Perform ray tracing according to A) to G) for each of these second intersection points (201).

[0233] - Sum the resulting brightness contributions, and

[0234] - Store the result of the summation.

[0235] J) A method for generating images.

[0236] Its characteristics are defined by the following steps:

[0237] - Select multiple image elements on this image recorder

[0238] - Calculate the luminance contribution of the light rays (800) incident on each of these image elements using the method described according to H), and

[0239] - Store the results.

[0240] K) According to the method described in J),

[0241] in,

[0242] The brightness contribution of each of these simulated rays (800) is determined using pinhole camera images, which intersect the virtual front surface (1011) at the first intersection point (1013).

[0243] The properties of this image make it correspond to an image generated by a pinhole camera placed at the corresponding first intersection point (1013).

[0244] L) According to the method described in K),

[0245] in,

[0246] The virtual front surface (1011) coincides with the incident pupil of the lens to be simulated.

[0247] M) According to the method described in L),

[0248] in,

[0249] The second surface (200) coincides with the virtual front surface.

[0250] P) according to any one of K), L) or M)

[0251] in,

[0252] At least one of these pinhole camera images was calculated by interpolation or extrapolation from other pinhole camera images.

[0253] Q) A method for generating an image, the method comprising the following steps:

[0254] - Provides real images recorded by real cameras.

[0255] - Provide a virtual image generated according to any one of K), L), M), or P).

[0256] - Fuse or overlay at least a portion of the real image and at least a portion of the virtual image.

[0257] - Store the created image.

[0258] The adjustable lens parameters used for simulation at least approximately correspond to the lens parameters used during actual recording.

[0259] R) A method for generating an image sequence composed of individual images, the method comprising the following steps:

[0260] -Provide virtual scenes,

[0261] - Provide the camera positions associated with this virtual scene.

[0262] - Calculate the individual images of the image sequence according to one of the methods described in any one of K), L), M), P), or Q).

[0263] - Store the image sequence.

[0264] S) A computer program product adapted to perform the method according to any one of A) to R) after being loaded into a computer.

[0265] **********

[0266] List of reference numerals

[0267] 1 Lens

[0268] 2 Lens element

[0269] 3-aperture stop

[0270] 4 mounting bases

[0271] 5 pixels

[0272] 100 Image Recorders / Sensors

[0273] 110 Sensor Surface

[0274] 111 Simulate the incident point of light on the sensor

[0275] 112 Information about the direction of light incident on the sensor

[0276] 200 Second Surface

[0277] 201 Second Intersection Point

[0278] 210 Shielding Surface

[0279] 211 Third intersection point

[0280] 300 computers

[0281] 800 simulated rays

[0282] 900 Conversion Rules

[0283] 901 Virtual Lens

[0284] 1000 Input Data Records

[0285] 1010 First data record (“virtual lens”)

[0286] 1010r, 1010g, and 1010b record partial data for different colors.

[0287] 1011 Virtual Front Surface

[0288] 1012 axis of symmetry

[0289] 1013 First intersection point

[0290] 1020 Second Data Record

[0291] 1021 Information about the light rays to be simulated

[0292] 1022 Information about setting the aperture stop

[0293] 1023 Information regarding other imaging parameters that can be adjusted on the lens.

[0294] 1024 Information about the wavelength or color of the light to be simulated

[0295] 1030 Partial Data Records

[0296] 2000 Output Data Records

[0297] 2010 Incident Ray

[0298] 3000 Incident Surface

Claims

1. A method for generating the brightness contribution of picture elements of an image by simulating an image representation of a scene using an optical imaging system, the optical imaging system including an image recorder (100) and a lens (1) located on a first surface (110). The method includes the following steps: - Provide a first data record (1010) which includes data describing the effect of light rays (800) on the lens (1) to be simulated. - Provides a second data record (1020) including data about the incident point (111) of the light ray (800) on the image recorder (100) and data about the virtual front surface (1011). - Provide conversion rules (900). - By applying the transformation rule (900) to the first data record (1010) and the second data record (1020), the first intersection point (1013) of the ray (800) with the virtual front surface (1011) and the direction of the ray (800) at the first intersection point (1013) are calculated. -The brightness contribution of the ray (800) is determined based on the first intersection point (1013) between the ray (800) and the virtual front surface (1011) and the direction of the ray (800) at the first intersection point (1013). - Store information about the calculated brightness contribution of this ray (800). in, - The first data record (1010) includes data about the second surface (200), and - The second data record includes data about the second intersection point (201) of the ray (800) and the second surface (200).

2. The method as described in claim 1, in, - The lens (1) has at least one adjustable imaging parameter (1022, 1023), and - The second data record (1020) contains information about at least one adjustable imaging parameter (1022, 1023) of the lens (1).

3. The method as described in claim 2, in, The adjustable imaging parameters (1023) include the focal length and / or magnification and / or field curvature of the lens (1).

4. The method as described in claim 2, in, The lens (1) includes an aperture stop, and The second surface (200) coincides with the aperture.

5. The method as described in claim 4, in, The aperture stop is an aperture stop (3), and The second surface (200) coincides with the aperture stop (3).

6. The method as described in claim 5, in, One of these adjustable imaging parameters (1022) describes at least one dimension of the aperture stop (3).

7. The method as described in claim 6, in, The aperture stop (3) is at least approximately circular, and the information associated with the second intersection point (201) of the ray (800) and the second surface (200) includes the normalized radius.

8. The method as described in any of the preceding claims, in, The first data record (1010) includes: - Data related to at least one shielding area (210) in the lens (1), and - Data relating to the effect of a portion of the lens (1) on at least one ray (800) extending between the at least one shaded area (210) and the image recorder (100), And before storing the brightness component of the at least one ray (800), the following steps are included: - Calculate the third intersection point (211) with the at least one shading area (210). - Check whether the at least one ray (800) is absorbed by the third phase intersection (211) or transmitted through the third phase intersection. - If at least one ray (800) is absorbed, discard the ray (800) or set the brightness component to zero.

9. A method for generating picture elements for an image. The method includes the following steps: - Select the incident point (111) of the light ray (800) on the image recorder (100). - Select multiple different second intersection points (201) on the second surface (200). - Perform the method as described in any of the preceding claims for each ray passing through each of these second intersection points (201). - Sum the resulting brightness contributions, and - Store the result of the summation.

10. A method for generating images, Its characteristics are defined by the following steps: - Select multiple image elements in the image recorder. - Calculate the luminance contribution of the light rays (800) incident on each of these image elements using the method described in claim 9, and - Store the results.

11. The method as described in claim 10, in, The brightness contribution of each of these simulated rays (800) is determined using pinhole camera images in their respective cases, and these rays intersect the virtual front surface (1011) at the first intersection point (1013). The properties of this image make it correspond to an image generated by a pinhole camera placed at the corresponding first intersection point (1013).

12. The method as described in claim 11, in, The virtual front surface (1011) coincides with the incident pupil of the lens to be simulated.

13. The method as described in claim 12, in, The second surface (200) coincides with the virtual front surface.

14. The method as described in any one of claims 11 to 13, in, At least one of these pinhole camera images was calculated by interpolation or extrapolation from other pinhole camera images.

15. A method for generating an image, the method comprising the following steps: - Provides real images recorded by real cameras. - Provide the image generated as described in any one of claims 11 to 14, - Fuse or overlay at least a portion of the real image with at least a portion of the image. - Store the created image. The adjustable lens parameters used for simulation at least approximately correspond to the lens parameters used during actual recording.

16. A method for generating an image sequence composed of individual images, the method comprising the steps of: -Provide virtual scenes, - Provide the camera positions associated with this virtual scene. - Generate the individual images of the image sequence according to any one of the methods described in claims 11 to 15. - Store the image sequence.

17. A computer program product adapted to perform the method as described in any of the preceding claims after being loaded onto a computer.

Images