Method and data processing system for synthesizing images

Abstract

The invention relates to a method for creating a second sequence of single images with the aid of a first sequence of single images, wherein the single images in the first sequence of single images or in the second sequence of single images have been taken with the aid of a lens, the method comprising the following steps: determining the entrance pupil and the field of view of the lens for the single images in the first sequence, and creating or adjusting the single images in the second sequence taking into account the entrance pupil and the field of view of the lens of the corresponding single images in the first sequence.

Description

Background of the Invention 1. Technical Field

[0002] The invention relates to a method for creating a second sequence of single images by means of a first sequence of single images, wherein the single images of the first sequence of single images have been taken by means of a lens. In particular, the invention relates to a software-based method for modifying and / or creating a moving image based on predefined parameters. 2. Background technology

[0004] Nowadays, animations are generated by means of image synthesis (also called rendering) for example for feature films, computer games, medical imaging, security applications for face recognition, in computer-based simulations, in virtual reality applications or also in industrial measurement technology. For example, the technology used here is to calculate the virtual beam path from the virtual sensor point through the virtual pinhole camera aperture to the virtual three-dimensional scene and to associate the corresponding color and intensity with the sensor point depending on the collision point in the virtual scene.

[0005] Often, there is a desire to superimpose or combine film sequences generated in this form of image synthesis with scenes from real filming. Thus, for example, an image sequence with image content generated as described, which for example shows a studio situation with real actors, should be embedded in a real filming image sequence, which has image content generated as described, which for example shows virtual, computer-animated creatures or backgrounds.

[0006] Here, there is a desire to integrate the image content to be embedded in the real image sequence as seamlessly as possible. Depending on the image content, it is not necessarily necessary to suggest to the viewer that the image content is actually real content, but rather an impression should be generated that the entire image content was originally created without post-processing. For this impression, great efforts are currently made, which mainly include image-by-image manipulation in post-production. Here, with the help of image processing programs, the individual image contents are changed so that the individual image contents are adapted as well as possible to the real scene. These image post-processings must mostly be performed manually and based on empirical values. For a convincing viewing experience, it is crucial that the modeled special effects merge with the real image content to a certain extent and that the viewer does not perceive the interruption. The settings suitable for the corresponding situation (FOV, aperture, entrance pupil, orientation, focal length, etc.) are simulated with the help of physical models. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a method for creating a second sequence of individual images by means of a first sequence of individual images, which method meets the above-mentioned requirements, provides a higher degree of seamless integration and at the same time requires less labor costs.

[0008] This technical problem is solved by a method according to independent claim 1. The method according to the invention for creating a second sequence of single images by means of a first sequence of single images, wherein the single images in the first or second sequence of single images have been taken by means of a lens, comprises the following steps: determining a beam characteristic in the lens for the single images in the first sequence; and creating or adjusting the single images in the second sequence taking into account the beam characteristic in the lens of the corresponding single images in the first sequence.

[0009] The adjustment of the image content of the individual images in the second sequence can be a subsequent conversion of an already calculated or otherwise generated sequence of individual images. In this case, for example, the individual images in the first sequence can be precisely combined with the images in the second sequence. The creation of the individual images in the second sequence can be the creation of a sequence of individual images by means of a lens, i.e. the recording of a sequence of images in the traditional sense, or the calculation of a sequence of individual images based on a virtual scene in the sense of image synthesis (rendering). In this case, for example, individual elements such as persons or objects can be extracted from the images of the second sequence and integrated into the images of the first sequence.

[0010] In a preferred embodiment, the step of determining the characteristics of the light beam within the lens comprises determining an entrance pupil and a field of view of the lens for a single image in the first sequence.

[0011] The step of determining the incident pupil and field of view (also called Field of View (FoV)) of the lens can be achieved by taking into account the exact line of sight direction, the field of view actually captured by the lens and the incident pupil when adjusting or creating the image content of a single image in the second sequence for calculating or converting the image content of the second sequence.

[0012] For example, during the focusing of a lens for adapting to a different lens-object distance, the entrance pupil and / or the field of view may change slightly, but nevertheless noticeably. This effect is also noticeable as a change in the image segment and is also referred to as "focus breathing" (English: "focus breathing"). Although this effect is hardly significant in single-shot images, it is noticeable in movie sequences and, in particular, presents a significant challenge for conventional processing methods when inserting simulated or virtual (i.e. calculated) image content afterwards. If, on the other hand, the entrance pupil and the field of view are already taken into account when calculating the image content to be added, the image content can be combined in a significantly simpler manner and, in particular, as far as possible without manual intervention.

[0013] A preferred embodiment of the invention comprises a step of combining the image content of a first sequence of single images with the image content of a second sequence of single images, wherein the combination comprises matching the image content of the single images in the second sequence with the image content of the single images in the first sequence or creating the single images in the second sequence taking into account the entrance pupil and the field of view of the lens of the corresponding single images in the first sequence, respectively. After the generation or adjustment of the second sequence of single images has been carried out with information about the orientation of the entrance pupil and the field of view of the lens, the two image sequences can be combined without major adjustments, for example by superimposing the two image sequences or inserting the image content of the second sequence of single images into the image content of the first sequence of images.

[0014] In a further embodiment of the invention, it is provided that in the step of adjusting or creating a single image, in addition to the entrance pupil and the field of view, the position of the exit pupil, the depth of field, the vignetting, distortion, bokeh, color errors, focus, the position of the lens element in the lens, the aperture, the exposure time, the color rendering and / or the reflections inside the camera are also taken into account. The mentioned parameters can be taken into account individually or in any combination. In addition, other relevant parameters such as the position of the light source, the type of light source, the distribution and intensity of shadows, motion blur, etc. can also be collected. In the case of the light source, for example, the spectral distribution of the light, the emission characteristics, the radiation intensity, etc. can be collected. The more characteristics of the lens or scene that are available and taken into account in the step of adjusting or creating, the more likely it is that the image content of the second image sequence will be better matched to the image content of the first image sequence.

[0015] If a sufficiently large number of the mentioned parameters are taken into account in the step of adjusting or in particular creating the individual images, the viewer can be given the impression of a combined image sequence which was not subsequently combined.

[0016] In one embodiment of the invention, it is provided that the step of creating the single images in the second sequence comprises creating the single images by rendering the virtual scene. This represents a common requirement and can be solved in a simple manner by acquiring the position of the entrance pupil in the scene and the field of view of the lens for the single images in the first sequence.

[0017] Alternatively, in one embodiment of the invention, the individual images can be created with the aid of a lens for the camera before the step of adjusting the individual images in the second sequence. The lens for the camera does not have to be the same lens with which the first sequence of individual images was taken. Rather, based on the determination of the entrance pupil and the field of view, the image content of the second sequence of individual images can be converted so that the combination of the first and second sequence of individual images can be carried out without problems.

[0018] In an extension of the invention, it is provided that the step of determining includes recording one or more imaging parameters of the lens when recording the first sequence of individual images. When creating or adjusting the second sequence of individual images, imaging parameters (such as entrance pupil, field of view, etc.) that were acquired for each individual image or for a plurality of individual images can be used, for example. This reduces the effort for combining the first and second individual image sequences. For example, in so-called blue screen or green screen shots (in which actors or objects are shot against a neutral (for example blue or green) background), the parameters used in the shooting can be acquired, in particular also acquired in a time-resolved manner, and these parameters can be used when creating the virtual scene based on the shooting. This offers the advantage that no subsequent calculations have to be performed with regard to the settings that may be used, the image contents of the two individual image sequences can be combined in a simple manner, and the end result is very realistic / credible.

[0019] In a further development of this embodiment, it is provided that the imaging parameters of the lens are used in real time to create the individual images in the second sequence. This can be used, for example, to forward the green / blue screen scene in real time during the formation (i.e., the shooting) to a simulation computer, which performs the specified combination with the prepared image content or with the currently calculated image content in real time and displays it on a display (such as a monitor or an eyepiece). As a result, the corresponding blue screen shooting / green screen shooting can be evaluated during the formation and, if necessary, corrected or repeated.

[0020] In one embodiment of the invention, it is provided that the step of determining the beam characteristics is performed for the first lens, and the step of creating the individual images in the second sequence is additionally performed taking into account the beam characteristics in the second lens. Thus, for example, a first sequence of individual images that has been captured using the first lens (e.g. a wide-angle lens) can be converted into a second sequence of individual images, thereby creating the impression that the second sequence of individual images has been captured using another lens (e.g. a telephoto lens).

[0021] In a further development of the invention, it is provided that the step of determining includes recording a temporal sequence of imaging parameters. Thus, in particular in the case of film shootings which include recording a series of individual images in a strict temporal sequence, the parameters required for the combination can be acquired or taken into account for all individual images or for a specific subset of the individual images. For example, before actually starting the shooting, the photographer can prepare the scene to be shot by shooting in advance with a virtually working camera in a virtual scene. This can be referred to as previsualizing.

[0022] In this way, for example, different camera settings or different lens types can be tried and checked with regard to artistic and / or possible technical feasibility. Possible time-resolved acquired parameters can be, for example, the position of the lens, the orientation of the lens, the current settings with regard to focus, aperture and / or zoom. These records can be used in subsequent work steps to plan the set.

[0023] Since filming time for movies is usually expensive and often limited, pre-visualization is applied with the help of rendering to minimize the total cost. The advantage of pre-visualization (English: "Previsualisation") is that the director, cinematographer or VFX (visual effects) supervisor can experiment with different staging and art direction application options, such as lighting, camera placement, camera movement and optical settings of the lens (such as focus segment, focal length or depth of field), director orientation and processing without incurring significant overhead for production. For projects with larger budgets, the director works with the actors in the visual effects department or in a special room.

[0024] Previsualization can add music, sound effects, and dialog to simulate the look of a fully produced and edited sequence of scenes. They are most often used for scenes with stunts and special effects such as chroma key. During previsualization, digital video, photography, hand-drawn art, clip art, and 3D animation are combined. In addition to the film and photography industries, renderings can also be used for previsualization in architectural planning, cosmetic surgery, or medical implant positioning.

[0025] Here, as described above, the method according to the invention can additionally include planning the recording of the first sequence of individual images as a function of the recording of the time series of imaging parameters and / or recording the first sequence of individual images using the recording of the time series of imaging parameters. As already explained, this enables the creation of image sequences with a high degree of pre-planning by means of very realistic recording simulations.

[0026] Furthermore, the invention relates to a data processing system having means for carrying out the method according to the invention and to a computer program.

[0027] With the invention, the optical design, i.e. the size, properties and orientation of the lenses within the lens, can be used in the simulation of the scene. As already explained, modern animations for feature films are produced with the aid of rendering. In this case, virtual rays are traced from virtual sensor pixels through a virtual pinhole camera aperture into a virtual three-dimensional scene. Depending on where the ray occurs in the scene, the pixel then receives a corresponding color and intensity. With the invention, the simple pinhole aperture model is replaced by ray tracing of the optical design program. This results in an extremely realistic simulation of the lens. In order to further increase the realism, wave optical effects can also be simulated in addition to or in addition to the ray tracing method. In this case, for example, the point spread function can be calculated with the aid of a diffraction integral.

[0028] One possible embodiment of the invention can be to plan the scene (pre-visualization) using a three-dimensional computer model of the film set. In such a model, for example, the scene to be filmed can be modeled and / or animated only roughly, and the requirements for actors, cameras (and their settings and positions), required equipment, scenery and / or special effects can be derived therefrom (according to the desired scene). Before the start of the filming, the photographer plans the filming in the virtual scene using a virtual camera. By realistically simulating the camera, different camera settings can be tested here, for example before the real filming. As a result, costs can be saved, undesirable situations can be identified and the scene can be better planned. For example, the focus setting, which is usually performed manually, can be practiced, especially in fast-moving subjects, or the suitability of certain types of lenses (real or not yet realized prototypes) can be tested. Settings related to artistic aspects and technical feasibility can also be tried. For example, the position, direction, focus, aperture, zoom settings can be stored in a time-resolved manner in the simulation. This information is then used to plan the set. Optionally, this data can then be transferred to an electronically controlled real camera and the setting sequence can be played back on the real camera.

[0029] Further embodiments may include the partial or complete photo-realistic creation of movie scenes based on 3D models. Real movie scenes may be superimposed with computer-animated scenes, for example, virtual creatures in the real scene and / or actors in the virtual scene or a combination of both. For this purpose, a so-called green screen is often used. For the quality of the combination of real scenes and computer animations, it is important how well the lighting conditions, motion blur, color rendering, noise, etc. in the model match the filming of the real scene. These modeled special effects that are not integrated with the scene will immediately appear unprofessional to the audience. Modeling the lens using the pinhole camera model only takes into account the (constant) focal length of the lens in a very simplified way. The stereoscopic playback of the scene depends only on the orientation of the entrance and exit pupils and the focal length variation of the lens. Here, these pupil orientations are functions that depend solely on the lens, the focus distance and the zoom setting. In addition, the aperture number is important, which determines the depth of field of the scene. In addition, there are a series of lens parameters, such as distortion, edge light falloff, color deviation, chromatic aberration, reflection dependence, etc. Specifically, in a green screen scene, for example, the position, orientation, focus setting, aperture setting, and zoom setting in the image capture can be stored in a time-resolved manner. These settings are used when creating the virtual scene. The advantage is a realistic and credible appearance and simple superposition of images.

[0030] Alternatively or additionally, in the case of a green screen scene, for example, the position, orientation, focus setting, aperture setting and zoom setting can be forwarded directly to the real-time simulation when the image is captured and the two images superimposed on the eyepiece or display.

[0031] In general, depending on the implementation, a number of advantages can be achieved with the present invention: By pre-creating settings within a virtual scene, camera setup times can be shortened. Entire camera shoots can be planned in advance, and camera settings and their changes can be recorded in a virtual camera and played back on a real camera. The quality of filming of certain movie shoots, especially in scenes that cannot be repeated arbitrarily often, can be improved by training the cameraman beforehand. When filming a virtual scene, the same impression can be generated by an exact simulation of a real lens as when filming the real scene. Scenes simulated with a real camera can be animated / generated. Virtual prototyping can be performed on lenses / cameras that still exist in reality.

[0032] One embodiment of the invention provides that a 3D image simulation is performed within the scope of a computer program using stored and interpolated point spread function data. The method comprises the following steps:

[0033] The size, shape and orientation of the exit pupil of the lens are determined for the selected set of image points; for example, a 2D point grid with a grid spacing of approximately 5.4 mm may be selected for this purpose.

[0034] For additional image locations, interpolating the size, shape and orientation of the image points between the selected image points;

[0035] Input object space function; the object space function can be understood as a three-dimensional function of intensity that depends on the distance relative to the entrance pupil of the lens; the object space function includes, for example, spectral characteristics such as RGB data or grayscale data, and / or the distance from the object to the entrance pupil (depth map);

[0036] Creating a point spread function by associating an object space function with a lens function, which lens function relates the object space to the image space; in this way, the point spread function includes the optical design data of the lens, including, if necessary, the coating data of individual or all optical elements in the lens;

[0037] integrating the point spread function with respect to the object coordinates;

[0038] integrating the point spread function about the shape of the exit pupil in order to account for wavefront deformation through the exit pupil;

[0039] Generates the object space to be rendered.

[0040] Additionally, in this embodiment, digital aberrations (distortion and / or shading, etc.) may be compensated.

[0041] Another alternative embodiment of the present invention provides that a 3D image simulation is performed using the data of the optical transfer function. The method has the following steps:

[0042] The size, shape and orientation of the exit pupil of the lens are determined for the selected set of image points; for example, a 2D point grid with a grid spacing of approximately 5.4 mm may be selected for this purpose.

[0043] for additional image positions, interpolating the size, shape and orientation of the image points between the selected image points to obtain an exit pupil function;

[0044] Input object space function; the object space function can be understood as a three-dimensional function of intensity that depends on the distance relative to the entrance pupil of the lens; the object space function can be obtained, for example, from a combination of wavelength and optical transfer function; to obtain the optical transfer function, a convolution (autocorrelation) is performed on the exit pupil function.

[0045] Compute continuous Fourier transforms from object space functions;

[0046] integrating the product of the optical transfer function and the Fourier transformed object space function with respect to the exit pupil coordinates for a fixed wavelength;

[0047] The calculations are repeated for multiple wavelengths;

[0048] Integrate with respect to wavelength over each color channel (spectral response function) and the spectral distribution of the light source;

[0049] Generate images for each color channel (RGB);

[0050] Overlay images for each color channel (RGB) to obtain realistic images.

[0051] In the mentioned implementation, the problem arises of performing a Fourier transform on strongly defocused object space points. This requires a very high grid sampling of the exit pupil, for example 10000 x 10000 instead of 30 x 30. This is time consuming and has a long run time.

[0052] Correspondingly, an alternative embodiment of the present invention provides for performing ray tracing-based rendering for 3D image simulation. The method is based on the fact that a light beam entering the lens can be traced back to a light source.

[0053] The method has the following steps:

[0054] The size, shape and orientation of the exit pupil of the lens are determined for the selected set of image points; for example, a 2D point grid with a grid spacing of approximately 5.4 mm may be selected for this purpose.

[0055] For additional image locations, interpolating the size, shape and orientation of the image points between the selected image points;

[0056] Calculate the beam direction; for example place it within the boundaries defined by the exit pupil;

[0057] A beam function is created; the beam function may represent, for example, a probability function and describes a ray distribution that depends on the direction and, for example, relates input coordinates to output coordinates; each input beam may form hundreds to millions of output beams, depending on the rendering quality that needs to be achieved.

[0058] Simulate all beams up to the light source;

[0059] For all discrete output beams, integrate with respect to the angular space of the source and the distance to the source;

[0060] Repeating the method for each input beam directed from the lens to the object, for example 10,000 beams;

[0061] For all input beams, integrate all output beams reaching the source;

[0062] Generate an image. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The embodiments of the present invention are described in more detail below with reference to the accompanying drawings. In the accompanying drawings:

[0064] Figure 1A , 1B A flow chart and a schematic diagram are shown according to a first method of the present invention;

[0065] Figure 2A , 2B A flow chart and a schematic diagram are shown according to a second method of the present invention;

[0066] Figure 3A , 3B A flowchart and a schematic diagram showing a third method according to the present invention; and

[0067] Figure 4 A schematic diagram of a fourth method according to the present invention is shown. DETAILED DESCRIPTION

[0068] Figure 1A , Figure 1B A first method according to the invention for creating a second sequence of single images by means of a first sequence of single images is shown. In a first step S11, the first sequence of single images is generated by means of a real camera 10. For the first method, it is not important whether the first sequence of single images is actually stored. The sequence of single images can be stored, but it is also possible to only generate and not record the sequence of single images.

[0069] In a second step (S12), camera settings and camera positions of the first image sequence are acquired. Preferably, this occurs during the generation of the image sequence, i.e., the recording of the first image sequence by means of the real camera 10. If this is performed during the recording, the recording of the first image sequence can be omitted. The camera settings and camera positions can be at least the position of the entrance pupil and the field of view of the lens performing the recording, but further parameters such as the aperture, exposure time, zoom setting, focus, etc. can also be acquired in addition.

[0070] In a further step (S13), the camera settings and the camera position are transmitted as data 11 to the virtual camera 12. This step can also be carried out after the first two steps S11, S12 or simultaneously therewith. Thus, the recording of the first image sequence (S11), the acquisition of the camera settings and the camera position (S12) and their transmission (S13) can be carried out simultaneously or also successively with a time offset with appropriate intermediate storage. Correspondingly, the camera settings and the camera position can be transmitted.

[0071] The virtual camera 12 can be, for example, a parameter set for the settings of an image synthesis program 18, with which the virtual image sequence 14 formed by the virtual scene 16 can generate a second image sequence by means of camera settings and camera positions in accordance with a further step S14. The image synthesis program can be, for example, a calculation program using a ray tracing algorithm. With the help of data obtained from the real camera 10 and possibly known data in advance, a model of the camera 10 can be set up in the image synthesis program. On the one hand, with the help of this model, more precisely the virtual camera 12, effects that are specific to the lens and are therefore relevant for optical imaging can be calculated. This enables a realistic simulation of the lens used or the entire camera 10. On the other hand, based on the time-resolved acquisition of the entrance pupil and the field of view, if necessary, the movement of the real camera 10 and, if necessary, also the effects related thereto, such as the influence of lighting conditions at different positions, or effects such as motion blur, color rendering or noise can have an impact on the generation of the virtual image.

[0072] In a specific embodiment of the method, the real camera 10 can be used, for example, as a controller for a virtual camera 12 for generating a virtual image sequence 14 in a virtual scene 16. In this way, a computer animation scene can be generated in a simple manner.

[0073] Figure 2A , Figure 2B A second embodiment of the method according to the invention is shown. For the following embodiments, the same reference numerals are used for the same or comparable features and are not described again to avoid repetition. As in the previously described method, Figure 2A , 2B The method also provides for recording a first image sequence (S21) by means of a real camera 10. In this case, the method provides for recording the first image sequence in a real, existing real scene 20 and for generating a real image sequence 22. As already explained above, in this method, camera settings and camera positions are also acquired (S22) and transmitted to the virtual camera 12 (S23).

[0074] The information obtained in these steps S21-S23 is now used in a further step (S24) to generate a second image sequence 24 in the virtual scene 16 with the virtual camera 12 by means of an image synthesis program 18, taking into account the camera settings and the camera position 11. When generating the second virtual image sequence 24, for example, information 11 about the position of the entrance pupil of the real image sequence 22 and the field of view of the camera 10 is already available. In a specific application, the first real image sequence 22 can be, for example, a green screen scene in a film shooting. In this case, for example, the position and orientation of the camera 10, the focus, the aperture, the zoom setting can be acquired in a time-resolved manner during the image shooting. This information can then be used accordingly when creating the virtual scene 16 and ultimately when creating the virtual image sequence 24.

[0075] In a further step S24, the first real image sequence 22 and the second virtual image sequence 24 can be combined to form a combined image sequence 26. In this case, in the example of the green screen technique mentioned, the virtual image content of the virtual image sequence 24 can be integrated into the image content of the real image sequence 22. Due to the largely seamless insertion possibility, a realistic and believable appearance of the combined image sequence 26 is obtained. Furthermore, the superposition / insertion can be carried out in a very simple manner.

[0076] Figure 3A , Figure 3B A further embodiment of the method according to the invention is shown in FIG. In a first step (S31), a virtual scene 16 is created taking into account the spatial data and other data 13 of the real scene 20. In a second step (S32), a first image sequence 24 is captured with the aid of a virtual camera 12. Since in this embodiment, it is not important to store this first image sequence 24. More precisely, the purpose of this embodiment is to give the photographer the opportunity to test and plan different camera settings before actually capturing the real image sequence 22. With the realistic simulation of the real camera 10 by the virtual camera 12, it is also possible in this way, for example, to plan and try out different camera settings and camera positions in terms of artistic and technical feasibility.

[0077] Correspondingly, in a further step (S33), the camera settings and camera position of the virtual camera 12 are acquired. This is preferably done in a time-resolved manner, i.e. each acquired camera setting and camera position is associated with a time stamp. Preferably, the settings and position are acquired for each single image in the sequence of single images. Depending on the design, the acquisition frequency can also be matched to the actual situation. For example, a lower frequency can be set than for each image. Here, the frequency can be adjusted to a fixed frequency (every n images) or can be adjusted depending on the changes that occur. The acquired settings can, for example, include position, direction, focus, aperture or zoom settings. The information acquired in this way can then be used to plan the set and the camera shooting.

[0078] Optionally, the camera settings and camera positions thus acquired can be transmitted to the real camera 10 in a further step (S34). These settings can then be taken into account in the real capture of the second single image sequence. For example, the camera settings can be replayed in the form of a setting sequence during the capture in the real camera 20, so that the photographer is relieved of the burden and only needs to be concerned with fewer capture details during the capture.

[0079] Figure 4 Another embodiment of the method according to the present invention is shown in FIG. Figure 4 As described in , the method is provided with two method steps, step S41 and step S42, and these two method steps can be performed independently of each other in terms of time.

[0080] With step S41 , a starting basis for subsequent rendering is generated, which is referred to herein as VFX generation. The starting basis may be, for example, data of a virtual scene containing the required three-dimensional information.

[0081] Any other basis can also be used which enables rendering to be performed based on three-dimensional information, thereby creating a sequence of individual images. For example, this can also be a pre-processed two-dimensional image with additional depth information or a two-dimensional image in which each pixel is associated with a 3D position in the three-dimensional scene. In this case, for example, the 3D information can be integrated into a two-dimensional image predefined in terms of the displayed scene, simulated for the displayed scene or generated for the displayed scene, based on three-dimensional data that has been obtained by acquiring the displayed real scene, for example with the aid of a 3D sensor.

[0082] In step S42, lens data / camera data which can be associated with a sequence of individual images (frames of a movie sequence) are recorded. The recording can involve, for example, a lens of a camera and can be performed, for example, with the aid of the lens.

[0083] Preferably, the acquisition of the movie sequence (S43) can be performed simultaneously with the acquisition of the lens data / camera data (S42).

[0084] The lens data / camera data may comprise a lens data sequence, wherein a sequence for example has a set of lens data / camera data for a single image (frame) respectively. Alternatively or additionally, only lens data / camera data that can be associated with an entire sequence of single images (movie sequence) may be acquired.

[0085] The lens data / camera data may include, for example, information about possible properties and / or settings of the lens itself, such as identification of the lens type or individual lenses, information about the lens type, focal length of the lens, calibrated focal length, calibrated aperture value (T-Stop-Wert), depth of field, hyperfocal distance, horizontal viewing angle or position and / or size of the entrance pupil. These lens data / camera data may, for example, be taken by a real camera or a real lens during the recording of a first sequence of individual images (movie sequence). The determination / calculation and / or recording of the lens data / camera data may, for example, be carried out for each focus point and each effective aperture value and / or for each individual image (frame).

[0086] Alternatively or additionally, the lens data / camera data may include specific physical data that characterize one or more, preferably all optical elements, such as lenses, apertures and / or filters in the lens. These physical data may be, for example, the lens shape, the aperture of the element, for example the maximum aperture (minimum aperture number) of the lens, the spectral transmittance, etc. In general, the specific physical data may be data that enable conclusions to be drawn about the optical design of the lens and from this to calculate imaging characteristics and / or to perform methods using ray tracing, ray casting, point spread functions or optical transfer functions.

[0087] Additionally or alternatively, the lens data / camera data may include information about the three-dimensional structure of the real captured scene. The acquisition (S44) of the 3D data may be performed in parallel with the acquisition (S42) of the characteristics and / or settings of the lens, for example.

[0088] The 3D data may be integrated into the lens data / camera data. Alternatively or additionally, the 3D data may also be added (S45) to the movie sequence data.

[0089] When collecting 3D data, for example, one or more 3D sensors can be used, which generate spatial data by optically or acoustically scanning the captured scene, and the spatial data can be associated with the captured single image sequence (movie sequence). One or more sensors can be arranged separately from the lens or camera, for example. Preferably, one or more sensors are located on the camera or lens or integrated into the camera or lens.

[0090] Alternatively or additionally, the lens data / camera data may include position data, orientation data and / or motion data, which are additionally recorded in a time-resolved manner, i.e., may be associated with individual images (frames) in a sequence of individual images (movie sequence), for example. The position information, orientation information and / or motion information make it possible to determine the position of the camera and / or lens relative to the captured real scene and the orientation of the camera and / or lens relative to the captured scene. By means of time-resolved acquisition, information about the motion of the camera and / or lens may also be acquired or derived directly or indirectly. For example, the motion of the camera and / or lens may be derived from changes in orientation or position. Alternatively or additionally, a sensor that directly acquires the motion of the camera / lens, such as a gyroscope sensor, may be used.

[0091] In summary, the lens data / camera data may include lens-specific data, 3D data of the real scene and / or position data / orientation data / motion data.

[0092] It can be provided that the lens data / camera data are further processed in real time already during recording and made available ( S46 ) for rendering.

[0093] Alternatively or additionally, the lens data / camera data may be recorded and used only in a subsequent further processing step which takes place temporally after the recording ( S47 ).

[0094] In step S48 , rendering is performed, that is, a two-dimensional image is generated based on the three-dimensional data.

[0095] In rendering, lens data / camera data may be taken into account ( S46 , S47 ).

[0096] Rendering may be performed using an algorithm that uses one or more of the following methods: ray tracing (beam function), ray casting, point spread function, or / and optical transfer function.

[0097] Alternatively or additionally, as already explained above, when generating a two-dimensional image, for example, a pre-processed two-dimensional image with depth information or a two-dimensional image with integrated three-dimensional information can be used. Rendering can also only represent a correction of an already existing two-dimensional image based on lens data / camera data.

[0098] By using the lens / camera data, rendered individual images (frames) or sequences of individual images (movie sequences) can be given an appearance that gives the impression that they have been filmed with the lens / camera belonging to the lens / camera data.

[0099] Here, the lens data / camera data can be used for rendering in real time when the lens data / camera data is captured. Here, the lens data / camera data can be used in real time (S46) for the rendering process, for example, by transmitting the lens data / camera data wirelessly or by wire to a rendering computer, and can be used for rendering.

[0100] Alternatively or additionally, provision can be made that previously acquired lens data / camera data are used ( S47 ) for the rendering.

[0101] In this case, it can be provided that the shot data / camera data contain, for example, only shot-specific and / or lens-specific data that are not time-resolved.

[0102] Alternatively or additionally, the lens data / camera data may, for example, only comprise position data and / or orientation data and / or movement data.

[0103] In one embodiment, different lens data / camera data can also be used in combination in the rendering. For example, non-time-resolved lens-specific and / or lens-specific data of a first lens can be combined with position data and / or orientation data and / or motion data of a second lens / second camera. For example, a movie sequence shot with a first lens can provide position data and / or orientation data and / or motion data for rendering together with the lens-specific and / or lens-specific data of a second lens. Thus, one can give the impression that the movie sequence has been shot with different lenses. In this case, the rendering can be performed in real time or in a subsequent step.

[0104] Rendering can generate, for example, a movie sequence with a resolution of 2K in real time at a frame rate of 24 fps. With the increase in computing power, 4K or 8K resolutions with the mentioned frame rates are also possible.

[0105] The rendered movie sequence may be observably displayed (S49) on a rendering monitor during or after generation (S44).

[0106] The first sequence of individual images (first film sequence) rendered taking into account the lens data / camera data can be combined (“Composite”) with the second sequence of individual images (second film sequence) that was actually filmed in a subsequent step S50. If the first film sequence has been rendered or corrected using the lens data / camera data with which the second film sequence was filmed, the two film sequences can be combined in an easy and seamless manner.

[0107] The film sequences assembled into one film sequence may be recorded (S51) and visually displayed (S52) on a composite monitor.

Claims

1. A method for creating a second sequence of individual images using a first sequence of individual images, wherein: A single image in the first single image sequence or in the second single image sequence has been captured by means of a lens (S11), and the method comprises the following steps: - for a single image in the first sequence, determining (S12) the intra-lens beam characteristics; - creating or adjusting (S14) the single images of the second sequence taking into account the intra-lens beam characteristics of the corresponding single images of the first sequence, The step of determining (S12) the characteristics of the light beam in the lens comprises determining an entrance pupil and a field of view of the lens for a single image in the first sequence; or / and Therein, the step of creating or adjusting the single images in the second sequence is performed taking into account the entrance pupil and the field of view of the lens of the corresponding single images in the first sequence.

2. The method according to claim 1, wherein: The step of creating or adjusting comprises using a point spread function or / and an optical transfer function or / and a beam function.

3. The method according to claim 1 or 2, comprising the step of combining (S25) the image content of the first sequence of individual images with the image content of the second sequence of individual images, wherein: The combining comprises matching the image content of the individual images in the second sequence to that of the individual images in the first sequence or creating individual images in the second sequence, respectively taking into account the entrance pupil and the field of view of the lens of the corresponding individual images in the first sequence.

4. The method according to claim 3, wherein: The steps for adjusting or creating a single image may include taking into account the position of the exit pupil, depth of field, vignetting, distortion, bokeh, color errors, focus, the position of the lens elements within the lens, aperture, exposure time, color representation, and / or reflections within the lens, in addition to the entrance pupil and field of view.

5. The method according to claim 1 or 2, wherein: The step of creating a single image in the second sequence comprises creating ( S24 ) the single image by rendering a virtual scene.

6. The method according to claim 1 or 2, wherein: The step of adjusting the single image in the second sequence is preceded by creating the single image with the aid of a lens for the camera (21).

7. The method according to claim 1 or 2, wherein: The step of determining the beam characteristics is performed for the first shot, and the step of creating the individual images of the second sequence is additionally performed taking into account the beam characteristics in the second shot.

8. The method according to claim 1 or 2, wherein: The steps to determine include: - recording (S22) one or more imaging parameters of the lens when taking the first single image sequence; And wherein the step of adjusting or creating includes: - using (S24) said imaging parameters when adjusting or creating a single image in the second sequence.

9. The method according to claim 8, wherein: The imaging parameters of the lens are used in real time to create the individual images in the second sequence.

10. The method according to claim 1 or 2, wherein: The determining step includes recording a time series of imaging parameters.

11. The method according to claim 10, comprising the steps of: a) planning the acquisition of a first sequence of single images based on the recording of the time series of imaging parameters (S34); or / and b) Acquiring a first sequence of individual images using the temporal recording of the imaging parameters ( S34 ).

12. Use of a method according to any of the preceding claims in one or more of the following fields: cinematography, photography, computer games, medical imaging, security applications, computer-based simulations, virtual reality and / or industrial measurement technology.

13. A data processing system having means for carrying out the steps of any one of claims 1 to 12.

14. A volatile or non-volatile computer-readable storage medium having a computer program, the computer program having commands, which, when executed on a data processing system according to claim 13, cause the data processing system to execute the following steps: - for a predetermined set of image points, determining the size, shape and / or orientation of the exit pupil of the lens; - Input object space function; - create a point spread function and / or input an optical transfer function and / or create a beam function; -Generate an image.

Images