Method for creating an output image

Abstract

In a method for creating an output image based on a moving image recorded in a virtual image studio using a camera and a background playback device, an image data set is received representing a real image captured by the camera. A virtual image is then generated by computationally simulating the recording of a substitute representation of the virtual background at the background playback device at a specific camera position during the recording of the real image. The image content of the substitute representation corresponds to the image content of the background representation. The output image is created as a superimposed image in which the foreground of the real image is superimposed on the virtual image.

Description

[0001] The invention relates to a method for creating an output image based on a moving image recording which has been produced in a virtual image recording studio by means of a camera and a background reproduction device arranged behind and / or above a real subject.

[0002] For example, moving image recordings can be carried out in a picture recording studio to create a cinema film. This allows for control over the recording conditions and to be able to record the film independently of environmental influences. However, particularly when recording scenes that take place outdoors, it can often be necessary to replace the background from the picture recording studio during post-production in order to, for example, use a depiction of a landscape in which the scene takes place as the background. To enable such a replacement of the background, blue or green screens are often used in conventional picture recording studios. This allows a foreground of the image, for example several actors playing the scene, to be isolated from an image generated by the camera and the intended background can be added later.

[0003] However, the use of a blue or green screen has the disadvantage that actors cannot directly perceive any events taking place in the background and therefore cannot interact with the background. Another problem during recording is that reflections of the light created by the blue or green screen off real objects in the foreground, such as the windows of a real car used in the image recording studio, are captured by the camera. These reflections can lead to unnatural color impressions in the images created by the camera. Furthermore, illuminating people with a blue or green screen can also lead to their faces being depicted with an unnatural blue or green tone.

[0004] To avoid this phenomenon, known as blue or green spill, and also to facilitate acting, motion picture recordings can be conducted in virtual image recording studios equipped with a background playback device, on which a background representation of a virtual background can be displayed for direct recording by the camera. In such motion picture recordings, the virtual background can thus be visible behind and / or above a real subject, particularly the actors playing the scene, so that, for example, the actors can perceive events occurring in this virtual background and coordinate their acting accordingly.In addition, the actors and other real subjects located in the virtual image recording studio can be illuminated by the background representation of the virtual background displayed on the background display device in a similar way to what would be expected in a real background corresponding to the virtual background, so that the aforementioned blue or green spill in the image generated by the camera can be avoided and the real subjects can be displayed with natural-looking colors.

[0005] However, even when using such background reproduction devices, effects may arise in an image generated by the camera through which the artificially generated virtual background is recognizable in the image and which may lead to an unnatural impression of an image background of the generated image.

[0006] For example, the representation of the virtual background on the background reproduction device can be reproduced by pixel elements, in particular light-emitting diodes, arranged in a grid, whereby this grid can be recognizable in images generated by the camera during a high-resolution recording. Although this can generally be counteracted by taking pictures with a wide-open aperture of a camera lens, thereby blurring the background representation, such an approach undesirably limits the creative design options for moving image recordings made in the virtual image recording studio. Furthermore, all images generated in the virtual image recording studio must be carefully examined to ensure that the artificial creation of the background does not result in recognizable artifacts in the images.In this respect, the use of a background playback device can involve a relatively high level of review effort in post-production, without the time spent on review being able to be used for creative further processing of the recordings from the virtual image recording studio.

[0007] It is therefore an object of the invention to provide a method for creating an output image which is based on such a moving image recording in a virtual image recording studio, wherein the method enables a recording in the virtual image recording studio using a background playback device for reproducing a background representation of a virtual background, without the artificial generation of the background leading to recognizable artifacts in the output images.

[0008] This object is achieved by a method having the features of claim 1.

[0009] The procedure includes the following steps: - Receiving an image data set which represents a real image recorded by means of the camera in the virtual image recording studio, wherein the real image contains a real subject in an image foreground and a background representation of a virtual background reproduced by the background reproduction device is contained in an image background; - Receiving camera position data and background position data, wherein a recording position of the camera relative to the background reproduction device when recording the real image in the virtual image recording studio can be determined on the basis of the camera position data and the background position data; - Determining a virtual image, wherein a recording of a reproduction of a substitute representation of the virtual background is computationally simulated at the background reproduction device at the recording position of the camera, wherein an image content of the substitute representation corresponds to an image content of the background representation; and - Creating the output image as an overlay image in which the foreground of the real image is superimposed on the virtual image.

[0010] In the method, an image data set is first received for a real image generated by the camera in the virtual image recording studio. This real image comprises a real subject in the image foreground, for example, actors playing a scene and any other real subjects used for the scene, and the background representation reproduced by the background reproduction device in an image background, so that the real image can, in principle, be completely captured with the intended background. In this respect, the real image can be based, in particular, on a moving image recording in a virtual image recording studio, which was carried out as explained above, by the background representation being reproduced behind and / or above the real subject by the background reproduction device and being reproduced by the camera together with the real subjects in the foreground.

[0011] As explained, such recordings conducted in virtual image recording studios – which result in real images of the aforementioned type – enable actors, for example, to react to events occurring in the virtual background. In particular, such motion picture recordings ensure that reflections of the light emitted by the background display device by the real subjects do not result in unnatural color impressions of the depicted real subjects in the real image, so that the real subjects can be depicted with natural-looking colors and under lighting conditions that would be expected in a real background corresponding to the virtual background.In particular, the aforementioned blue or green spill resulting from the use of blue or green screens can be avoided, allowing the foreground of the real image to exhibit a natural color impression. However, the real image may potentially exhibit the aforementioned artifacts in the background, in which, for example, a grid of pixel elements of the background display device may be recognizable.

[0012] To address this problem, the method disclosed herein first receives camera position data and background position data, based on which the recording position of the camera relative to the background display device can be determined as the position of the camera relative to the background display device at which the camera was located during the generation of the real image. In particular, it may already be necessary to consider information about the recording position of the camera when creating the background display in order to generate the background display depending on the recording position of the camera and to display it on the background display device.For this purpose, a control device of the background playback device can, for example, access a game engine during moving image recording, which generates the background representation depending on the camera's recording position in a similar way to how, in a video game, a representation of a virtual environment on a screen can be generated depending on a player's position in the virtual environment. In this respect, the recording position can, for example, be encoded in the camera position data and directly assigned to the real image in the form of metadata, so that it can be made available, for example, to post-production together with the image data set.

[0013] The background position data can, in particular, determine a positioning of the background reproduction device in a coordinate system defined in the virtual image recording studio, so that, for example, an extent and arrangement of the background reproduction device in the virtual image recording studio can be determined and / or defined by the background position data. Therefore, the camera position data can, for example, also include coordinates in the defined coordinate system that determine an absolute position of the camera at which the camera was located during the creation of the real image, so that the recording position relative to the background reproduction device can be determined based on the respective absolutely defined coordinates.In addition, the image data set may, for example, also contain information that can be used to determine the camera's recording position during the creation of the real image, as explained in more detail below.

[0014] Based on the camera position data and the background position data, a virtual image can then be determined, whereby a recording of a reproduction of a substitute representation of the virtual background is computationally simulated on the background reproduction device at the camera's recording position. In other words, based on the camera position data and the background position data, a recording in the virtual image recording studio can be simulated by the camera, but instead of the background representation actually reproduced in the virtual image recording studio, the reproduction of the substitute representation is simulated on the background reproduction device.Thus, a situation can be simulated in which the camera located at the recording position in the virtual image recording studio relative to the background display device images the substitute representation displayed on the background display device, thereby generating the virtual image. In particular, the size of the substitute representation in the simulation can be determined by the background position data.

[0015] In principle, the virtual image can thus be understood as an image that is not actually generated, but could theoretically be generated by the camera if, during recording, the background display device had displayed the substitute image instead of the background image. As an output of this step of determining the virtual image, a virtual image data set representing the virtual image can be generated.

[0016] Subsequently, in the method disclosed here, the output image is created as an overlay image in which the foreground of the real image is superimposed on the virtual image. In this respect, the foreground and background of the real image can be separated from each other to a certain extent, with the background being replaced by the virtual image, so that the virtual image of the substitute representation can represent the background of the output image. Conversely, the foreground of the real image can be isolated from the background and then superimposed as the foreground of the virtual image.The output image can therefore be represented in particular by an image data set which comprises, as foreground, a part of the image data set which represents the image foreground of the real image, and which comprises, as background, a part of the virtual image data set which represents the virtual image at those points of the virtual image which are not overlaid by the image foreground.

[0017] In order to facilitate the creation of the output image, it can also be provided in particular that the real subject is not taken into account during the computational simulation of the recording of the substitute representation, and that recording of the substitute representation in the virtual image recording studio without real subjects is simulated. The virtual image can therefore ultimately represent a pure background image in corresponding embodiments. By comparing such a virtual image or a virtual image data set representing the virtual image with the real image or the image data set, the image foreground can therefore be easily isolated, for example by edge detection. After such isolation of the image foreground, the output image can in turn be easily created by overlaying the isolated image foreground on the virtual image.

[0018] In summary, the real subject can be reproduced naturally during the creation of the real image by imaging the real subject in front of and / or below a background representation generated by the background rendering device, so that the real image can have a natural-looking foreground. However, in order to counteract the disadvantages mentioned above, which arise from imaging the background rendering device and can create an unnatural impression of the image background, the image background can be subsequently replaced - essentially similar to a blue screen or green screen recording - by simulating a recording of a substitute representation and using the resulting virtual image as the background of the output image.

[0019] For example, it can be provided to generate the substitute representation with a higher resolution than the background representation, so that when the camera computationally simulates the recording of the substitute representation, the resolution of the substitute representation no longer appears as a raster in the virtual image. In a sense, in the course of determining the virtual image, recording in front of a background reproduction device with a higher resolution than the real background reproduction device can be simulated or computationally simulated. This can be achieved, for example, by simulating the background reproduction device in the simulation with a higher density of pixel elements than the density of pixel elements of the real background reproduction device, in order to simulate the reproduction of the substitute representation on such a background reproduction device with increased resolution.

[0020] While this type of output image creation allows for natural-looking output images for motion picture recordings from a virtual image recording studio, replacing the background of the real image also eliminates the need for measures aimed at achieving the best possible background in the real image. Since the background of the real image is replaced anyway, a camera lens, for example, can be adjusted as desired without requiring a blurred image of the background.Furthermore, such a method for creating output images in the virtual image recording studio also allows for the use of background display devices with comparatively low resolution, since artifacts resulting from a low resolution of the background display can be compensated for by replacing the image background. By using background display devices with lower resolution, the costs of motion picture recordings in a virtual image recording studio can be reduced, while a low-resolution background display can still enable natural lighting of real subjects and the recognition of events occurring in the virtual background during the performance.

[0021] The moving image recording can be represented, in particular, by a sequence or a plurality of consecutive real images, which, when played back, can result in a moving image sequence. The method described can be performed, in particular, for each of the real images in the sequence in order to create output images for the entire moving image recording. In particular, the method can be performed during post-production.

[0022] Further embodiments are explained in the dependent claims, the description and the drawings.

[0023] In some embodiments, an imaging window of the camera can be taken into account when determining the virtual image. For example, based on the camera's recording position and / or a camera lens setting, a frustrum of the camera and a section of the background rendering device can be determined that is located within the camera's imaging window and is therefore imaged by the camera. To reduce the computational effort when simulating the recording of the reproduction of the substitute representation, the reproduction of the substitute representation can, for example, be simulated only on a section of the background rendering device that was actually imaged by the camera.In principle, however, it is also possible to simulate a reproduction of the substitute representation on the entire background reproduction device or on all sections of the background reproduction device on which the background representation was also reproduced.

[0024] In some embodiments, the substitute representation can have a higher resolution than the background representation. This can prevent, in particular, the occurrence of a grid of pixel elements of the background display device that reproduce the background representation and / or a moiré effect by superimposing this grid with a grid of pixels of an image sensor of the camera in the virtual image. For this purpose, the resolution of the substitute representation can, in particular, correspond to a resolution of an image sensor of the camera or be higher than a resolution of the image sensor of the camera.

[0025] In some embodiments, the resolution of the background display can be determined by the background display device. In particular, the resolution of the background display device can be determined by an arrangement and / or density of pixel elements, in particular light-emitting diodes, which display the background display. In particular, the background display device can be an LED wall having a plurality of light-emitting diodes as pixel elements arranged on one or more panels in an at least two-dimensional arrangement. A resolution of the background display can therefore be determined by a distance between the pixel elements, wherein such a distance can be, for example, at least 2 mm and / or at least 3 mm.To a certain extent, in order to determine the virtual image, it can therefore be assumed that a background display device with an increased density of pixel elements compared to the real background display device is arranged in the virtual image recording studio, so that the reproduction of the substitute representation on the background display device can be simulated or computationally reproduced with a higher resolution.

[0026] In some embodiments, the virtual background may represent a three-dimensional scene.

[0027] In particular, it can be provided to project the three-dimensional scene onto the background display device to display the background image. The background display device can, in particular, be designed to be flat and / or curved or arched in sections, for example, to surround and / or arch over real motifs in a semicircular manner.

[0028] Furthermore, the background representation can be determined in particular as a function of the recording position of the camera. In this case, provision can be made for access to a game engine which can be designed to create a projection of the virtual background onto the background reproduction device from a virtual background provided as a three-dimensional model as a function of the recording position of the camera and to output it as a background representation. In particular, the recording position and / or a viewing angle of the camera can be taken into account in that, similar to a video game, the background representation is generated and projected onto the background reproduction device as a function of a player's position in a virtual environment in such a way that the camera would perceive the virtual background in a real background corresponding to the virtual background.

[0029] In some embodiments, in order to determine the virtual image, the recording of the reproduction of the substitute representation can be computationally simulated without taking the real subject into account. As already explained, in some embodiments, recording a reproduction of the substitute representation in the virtual image recording studio can be simulated without real subjects, so that the virtual image can be determined as a pure background image to form the background of the output image. This can, in particular, enable simple isolation of the foreground of the real image by comparing the real image and the virtual image, since in such embodiments the real image and the virtual image can differ significantly from one another in that the virtual image does not have the foreground or the real subject.

[0030] In some embodiments, the method may further comprise the step of determining the substitute representation of the virtual background.

[0031] In some embodiments, the method may include the steps of: - Receiving a background data set representing the background representation, and - Determine the replacement representation by interpolation from the background representation.

[0032] In particular, the method allows the background representation actually displayed on the background reproduction device during generation of the real image to be received as a background data set. Based on the received background representation, the substitute representation can then be determined by interpolation, wherein the interpolation can, in particular, increase the resolution of the substitute representation compared to the resolution of the background representation. By providing the background representation in this way, it is not necessary, in particular, for the substitute representation to be regenerated from a model, for example, during the course of the method, taking into account the recording position of the camera. Instead, the recording position of the camera only needs to be taken into account when computationally simulating the recording in the virtual image recording studio.For example, the output of a game engine used to control the background playback device during motion picture recording, or background data representing this output, can be assigned to the real image and transmitted to post-production. Interpolation can, for example, use the nearest neighbor method, bilinear interpolation, and / or bicubic interpolation.

[0033] In some embodiments, the substitute representation can further be determined from a model of the virtual background, in particular by computationally projecting a three-dimensional model of the virtual background onto the background rendering device. For computationally projecting the virtual background onto the background rendering device, the background position data can be taken into account, in particular.

[0034] In such embodiments, a game engine can therefore be virtually simulated, which generates the background representation in the real virtual image recording studio for playback on the background playback device. In such embodiments, the entire recording can thus be computationally simulated in the virtual image recording studio by first determining the substitute representation based on a model of the virtual background, in particular a three-dimensional model, and then simulating a recording of a playback of this substitute representation from the recording position by the camera. Nevertheless, a recording without taking the real subject into account can be simulated.

[0035] Furthermore, in such embodiments, the substitute representation can be determined from the model depending on the camera's recording position. The substitute representation can thus be determined virtually in the same way as the background representation was determined for the moving image recording.

[0036] Alternatively or additionally, in other embodiments, a replacement data set representing the replacement representation can also be received. For example, in such embodiments, it can be provided that a control device of the background rendering device, in particular by means of a game engine, generates the background representation based on a model of the virtual background, which is matched to a resolution of the background rendering device and enables reproduction by the background rendering device. On the other hand, the control device of the background rendering device, in particular the game engine, can also generate the replacement representation, the image content of which corresponds to the background representation, but can in particular have a higher resolution than the background representation.For example, in such embodiments, the control device or game engine can initially generate a projection onto the background display device with the maximum possible resolution based on a model of the virtual background, depending on the camera's recording position. This projection can then be saved as a substitute display. Based on this substitute display, the background display can then be generated by adjusting the resolution to the resolution of the background display device, and the background display device can be controlled to display it.Since in such a procedure the substitute representation is generated anyway as an intermediate step for generating the background representation, a data set representing the substitute representation can be transmitted to post-production, so that no further calculation steps are required to determine the substitute representation during post-production.

[0037] In some embodiments, the camera position data can include information about the orientation of the camera. In particular, the camera position data can also be used to determine a viewing direction of the camera in order to determine a section of the background display device and / or the background representation that is contained in the real image. The orientation of the camera can therefore be taken into account accordingly when determining the virtual image.

[0038] In some embodiments, information about a lens setting can also be received during the acquisition of the real image, and the virtual image can be determined based on the information about the lens setting. In particular, alternatively or additionally, information about a lens type, for example, a model designation or a serial number, can also be received and taken into account when determining the virtual image.

[0039] In particular, properties of the real image, such as the sharpness of the image background, can be influenced by a setting of a lens connected to the camera, for example a lens permanently connected to the camera or an interchangeable lens, so that providing information about a setting of the lens can, in particular, make it possible to take such settings into account when determining the virtual image. In particular, to determine the virtual image, a recording of the substitute representation by the camera positioned at the recording position can be simulated with settings of a lens connected to the camera that correspond to the settings of the lens connected to the camera during the generation of the real image.Such settings of a camera lens can also be stored, in particular as metadata, during the motion picture recording and assigned to the image data set representing the real image.

[0040] In some embodiments, the information about the camera lens setting may include information about a focus position, a zoom factor, an aperture, and / or a focal length. In principle, in some embodiments, the computational simulation of the recording of the substitute representation may rely on a model and / or a simulation of the camera and / or a lens connected to the camera.

[0041] In some embodiments, information about a camera type can be received, whereby the virtual image can be determined depending on the information about the camera type. For example, a model identification and / or a serial number of the camera can be transmitted to enable a replication of the recording adapted to the actual camera used. For example, models of various cameras can be retrieved in post-production or a software product used for this purpose, so that the correct simulation for reproducing the recording can be selected based on the respective camera type.

[0042] In some embodiments, information about an image sensor and / or a resolution of the camera can be received, wherein the virtual image can be determined depending on the information about the image sensor and / or the resolution of the camera. For example, the information can include information about a type of image sensor of the camera. Furthermore, it can generally also be provided that the above-mentioned information about a camera type includes information about an image sensor of the camera and / or a resolution of the camera. In particular, both a lens connected to the camera and an image sensor of the camera can be taken into account in order to achieve the most complete possible simulation of the recording of the reproduction of the substitute representation.

[0043] In some embodiments, to determine the virtual image, a respective section of the substitute representation can be determined for each pixel of an image sensor of the camera, which is mapped onto the pixel. In particular, during the computational simulation, a beam path through the camera can be simulated in order to ultimately determine the virtual image as a respective piece of image information for each pixel of an image sensor of the camera. For this purpose, in particular, received information about an image sensor and / or a resolution of the camera and / or a setting of a lens can be used.

[0044] In some embodiments, when determining the output image, the foreground of the real image can be isolated by comparing the image data set with a virtual image data set that represents the virtual image. In particular, this can be provided in embodiments in which the virtual image is determined without taking the real subject into account, so that the virtual image contains exclusively the substitute representation and the foreground can form a clear distinction between the real image and the virtual image. For example, it can be provided to isolate the foreground from the real image by edge detection in order to then be able to overlay the foreground on the virtual image.

[0045] In some embodiments, when determining the output image in an environment of the image foreground, a color adjustment may be performed that approximates the virtual image to the environment of the image foreground of the real image.

[0046] By adapting the virtual image to the real image in the area surrounding the foreground of the image, any jumps in brightness or color that could occur at a transition between the foreground of the image contained in the real image and the virtual image can be avoided. For example, the area surrounding the foreground of the image can be defined by a predetermined number of pixels around the foreground of the image, in particular an outer edge of the foreground of the image, for example 500 pixels, 100 pixels, 50 pixels or 10 pixels. In particular, color adaptation can therefore be carried out for pixels of the virtual image located in such a defined area surrounding the foreground of the image. If the foreground of the image comprises several real motifs, such color adaptation can also be carried out, in particular, around each of the real motifs depicted.

[0047] In some embodiments, the output image can alternatively or additionally be determined in the vicinity of the image foreground as a blend between the real image and the virtual image. Such an approach can, in particular, avoid hard edges between the image foreground and the virtual image by designing the transition as a smooth blend.

[0048] In some embodiments, the real image may include position markers displayed on the background display device, wherein the camera position data may form part of the image data set representing the displayed position markers. Furthermore, in such embodiments, the recording position may be determined based on the position of the position markers in the real image.

[0049] Furthermore, in some embodiments, it can be provided that the position markers are reproduced on the background reproduction device during the moving image recording and that the recording position is determined during the performance of the moving image recording based on the position of the position markers in the real image, wherein the determined recording position can be encoded in the camera position data.

[0050] In particular, in such embodiments, it can therefore be provided that, during the performance of the moving image recording, position markers are reproduced on the background reproduction device, which can overlay the reproduced background representations. By identifying such position markers in the real image generated by the camera, the recording position of the camera relative to the background reproduction device can then be deduced, for which purpose, for example, triangulation methods can be applied. Such determination of a recording position of a camera relative to a background reproduction device based on a position of position markers in an image generated by the camera is explained, for example, in DE 10 2021 106 488 A1, to the content of which reference is therefore made with regard to such determination of the recording position of a camera.

[0051] For example, it can be provided that the position markers are imaged by the camera and therefore included in the image data set that represents the real image. This can make it possible to identify the position markers during post-production and to deduce the camera's recording position based on the position of the identified position markers. Alternatively, the position markers can also be used directly during the motion picture recording to determine the recording position, in particular to be able to use the recording position determined in this way to generate the background image. In this case, the recording position determined during the motion picture recording using the position markers can be directly encoded in the camera position data and transmitted to post-production.

[0052] The recording position can be determined, for example, by a position determination device of the virtual image recording studio and / or the camera, which can have an image evaluation device for identifying the position markers.

[0053] In some embodiments, a sequence of background images can be reproduced on the background reproduction device while the moving image recording is being carried out, wherein the position markers in the sequence of background images can be reproduced and not reproduced intermittently, in particular alternately.

[0054] For example, in the case of moving image recordings, it may be provided to output a moving image sequence at a specific frequency and, for example, to play a movie at this frequency. However, it may also be provided that the moving image recording is carried out at a higher, for example, doubled, image recording frequency, so that ultimately more images are available than are actually required for the moving image sequence to be output. This can make it possible to display the position markers only in time windows in which an image is generated by the camera that is not intended for the ultimately output moving image sequence, so that the recording position of the camera can be determined without position markers having to be displayed in the images intended as output.

[0055] In such embodiments, therefore, in particular, two image data sets for real images can be received, which have the background representation in the image background, wherein a first of the two real images can include the position markers and a second of the two real images can not include the position markers. This can make it possible in post-production to determine the recording position of the camera based on the first real image, but to use the second of the two real images, which does not include the position markers, for the method for determining the output image. In particular, the use of real images without position markers can facilitate the isolation of the image foreground, since such real images can be distinguished from the virtual image, in particular exclusively by the image foreground—and not by depicted position markers.

[0056] In some embodiments, the camera's recording position can be determined during recording based on the position of the position markers in real images that include the position markers, wherein the determined position can be encoded in the camera position data. In such embodiments, post-production can also be provided with, for example, only image data sets that represent real images that do not contain any position markers.

[0057] In some embodiments, the position markers can form a pattern superimposed on the background image. For example, such a pattern can comprise a checkerboard-like structure and / or a barcode-like structure, wherein a barcode-like structure can, in particular, comprise several parallel lines. By designing the position markers as such a pattern, the position markers can, in particular, be easily identifiable in the real images, enabling the camera's recording position to be determined.

[0058] In some embodiments, the pattern may comprise various geometric structures, wherein the various geometric structures may be contained in different color channels of the real image. In particular, the background rendering device may be configured to generate and / or render the various geometric structures in different color channels. For example, the position markers may therefore be rendered as both a checkerboard-like and a barcode-like structure, wherein the respective structures may be generated in different color channels.

[0059] In some embodiments, the position markers may form slight brightness deviations compared to a respective section of the background representation surrounding the position marker, wherein a brightness deviation between a respective position marker and a section of the background representation surrounding the position marker may in particular be a maximum of 2 percent.

[0060] By generating such slight brightness deviations, the position markers can be made invisible to the human eye, yet can be resolved by the camera and / or an image analysis device. This allows the position markers to be generated without, for example, disrupting human perception of the background image, such as actors playing the scene.

[0061] The invention further relates to a data processing system comprising means for carrying out a method of the type disclosed herein.

[0062] Furthermore, the invention relates to a computer program product comprising instructions that, when executed by a computer, cause the computer to perform a method of the type disclosed herein. In particular, such a computer program product can be provided for post-production.

[0063] The invention also relates to a method for determining two input images for a method for creating an output image for a moving image recording carried out in a virtual image recording studio using a camera and a background reproduction device. In the method, a background representation of a virtual background is generated behind and / or above a real subject as a function of a recording position of the camera relative to the background reproduction device and is reproduced on the background reproduction device. By means of the camera, a real image of the real subject and the background representation are determined, which comprises the real subject in an image foreground and the background representation in an image background. Furthermore, a substitute representation of the virtual background is determined, wherein an image content of the substitute representation corresponds to an image content of the background representation.A virtual image is determined, whereby and / or by computationally reproducing (in other words, simulating) a recording of a reproduction of the substitute representation of the virtual background at the background reproduction device at the recording position of the camera. The real image and the virtual image are further output and / or stored as input images for creating the output image.

[0064] According to this aspect of the invention, it can thus be provided that the real image and the virtual image are determined in order to be used as input images for creating the output image. For example, it can be provided that the output image is then created as an overlay image in which the foreground of the real image overlays the virtual image. However, according to the method explained here, the virtual image does not necessarily have to be determined during post-production to create the output image, but can be provided to post-production, for example, directly as an input image together with the real image.In particular, it can be provided that a control device of the background reproduction device in the virtual image recording studio is designed to determine the virtual image in order to then be able to make it available for post-production, for example together with the real image generated by the camera.

[0065] In some embodiments, the substitute representation may have a higher resolution than the background representation. In this respect, this method can also determine a virtual image in which any artifacts due to the reproduction of the background representation on the background reproduction device can be avoided.

[0066] In some embodiments, the substitute representation can be determined by interpolation from the background representation. In particular, such interpolation can increase the resolution of the background representation in order to obtain a substitute representation with a higher resolution than the background representation.

[0067] In some embodiments, the background representation and the substitute representation can be determined from a model of the virtual background, in particular by projecting the virtual background onto the background rendering device, and / or in particular depending on the recording position of the camera. In such embodiments, a game engine can be used, in particular, to generate a projection onto the background rendering device from a three-dimensional model of the virtual background depending on the recording position of the camera. The game engine can, in particular, generate two representations at different resolutions in order to take into account the resolution specified by the background rendering device for the background representation, but to generate the substitute representation with an increased resolution.

[0068] In some embodiments, position markers for determining the recording position of the camera can be displayed on the background display device, wherein the recording position of the camera can be determined based on a position of the position markers in the real image.

[0069] In particular, in the method for determining two input images, the further steps explained above in connection with a method for creating an output image for determining a virtual image, a real image, a background representation, a substitute representation and / or a recording position of the camera can also be carried out individually or in combination with one another in order to achieve the effects to be achieved thereby.

[0070] The invention also relates to a method for creating an output image based on a moving image recording which has been created in a virtual image recording studio by means of a camera and a background reproduction device arranged behind and / or above a real subject, in which input images determined according to a method explained above for determining two input images are received and the output image is created as an overlay image in which the image foreground of the real image is superimposed on the virtual image.

[0071] In the method for creating an output image, the two input images can thus be received and used directly to create the output image. For example, this can be done within the framework of post-production, to which the two input images are made available. However, it can also be provided that the stored real image and the stored virtual image are used at a control device of the background playback device and thus in the virtual image recording studio to create the output image and make it available to post-production for further processing.

[0072] The invention is explained below purely by way of example using an embodiment with reference to the drawings.

[0073] They show: Fig. 1 a schematic representation of a virtual image recording studio in which a moving image recording is carried out by means of a camera and a background playback device, Fig. 2 a schematic representation of a motion picture camera used in the virtual image recording studio, Fig. 3 a schematic illustration of a recording in the virtual image recording studio and a real image recorded by the camera, Fig. 4 an illustration of a step for receiving the real image and for receiving camera position data and background position data at a data processing system, Fig. 5 is an illustration of a step for generating a virtual image by recreating a shot in the virtual image recording studio by the data processing system, Fig. 6 is an illustration of a step for creating an output image, wherein the real image and the virtual image are used as input images for creating the output image, and Fig. 7 an illustration of a reproduction of position markers on the background reproduction device, by means of which a recording position of the camera can be determined.

[0074] Fig. Figure 1 schematically shows a virtual image recording studio 13 in which a moving image recording is carried out by means of a camera 23. The camera 23 can therefore be designed as a moving image camera in order to be able to carry out moving image recordings which are a sequence of real images generated by the camera 23. Fig. can be stored. For this purpose, the camera 23 has a lens 59, which can in particular be designed as an interchangeable lens that can be selectively connected to a housing of the camera 23 in order to always be able to use a lens 59 that is optimally adapted to the environment in the image recording studio 13 or the scene to be recorded. In particular, an image sensor with a plurality of light-sensitive sensor elements can be arranged in the housing of the camera 23, onto which light penetrating through an aperture of an aperture can be directed by means of a lens system or at least one lens to generate an image (cf. Fig. 2).

[0075] Furthermore, a background display device 15 is arranged in the image recording studio 13 for carrying out the moving image recording. This device has an active lighting device 31 designed as an LED wall 33 and is designed to display a background representation 19 of a virtual background 21 for recording by the camera 23. For this purpose, the lighting device 31 or the LED wall 33 has a plurality of actively luminous pixel elements 35 arranged next to one another in a two-dimensional arrangement. For example, the pixel elements 35 can be designed as individually controllable light-emitting diodes 44 or as individually controllable light-emitting diode units, wherein each of such a light-emitting diode unit can comprise several, in particular three, light-emitting diodes 44.In particular, it can be provided that the pixel elements 35 are designed as light-emitting diode units with three respective light-emitting diodes 44, wherein one of the three light-emitting diodes 44 can emit red light, one light-emitting diode 44 can emit green light, and one light-emitting diode 44 can emit blue light. The light-emitting diode unit can also comprise a color mixer in order to be able to adjust a respective color and / or brightness emitted by the pixel element 35 by individually controlling the light-emitting diodes 44 of a light-emitting diode unit. The light-emitting diodes 44 can be designed, for example, as LEDs or as organic light-emitting diodes 44 or OLEDs. In principle, background reproduction devices can also be used to reproduce a representation of a virtual background in the image recording studio 13, which generate the representation by means of rear projection.

[0076] The background display device 15 comprises a plurality of panels 41, which can also be referred to as panels. A respective plurality of actively luminous pixel elements 35 are arranged on each panel 41 of the plurality of panels 41, so that a section of the background image 19 can be displayed on each of the panels 41. The panels 41 are, in particular, rectangular and / or square and borderless, so that the background image 19 can be displayed without visible interruptions, even at the transitions between panels 41. The panels 41 are further arranged in a two-dimensional matrix to form the background display device 15. In this respect, the background display device 15 comprises a plurality of panels 41 in the embodiment shown.

[0077] The background representation 19 here reflects, by way of example, a three-dimensional scene 43 with motifs 91, 92, 93, and 94, three trees, and a path, which can be generated by appropriately controlling the pixel elements 35, in particular by appropriately adjusting their respective color and brightness. The three-dimensional scene 43 is projected onto the essentially two-dimensional arrangement of the pixel elements 35 of the illumination device 31, with the motifs 91, 92, and 93, in particular, appearing at different distances from the illumination device 31 or the background display device 11 in order to simulate the three-dimensionality of a real background corresponding to the virtual background 21.

[0078] In particular, the reproduction of the background representation 19 of the virtual background 21 serves to create a background for a recording of a real subject 17, for example an actor, against which a recording can be made or a film scene can be played. This essentially allows any landscapes, rooms, or environments to be created in the image recording studio 13, against or in which a scene, for example for a movie, is to be shot. Furthermore, it is possible to represent movements in the virtual background 21, such as a passing car, by temporally varying control of the pixel elements 35, to which movements the actor 17 can react more easily than when acting in front of a blue or green screen.

[0079] The background rendering device 15 extends essentially vertically, so that the actor 17 can move in front of the virtual background 21. However, in order to be able to display the virtual background 21 more extensively, the background rendering device 15 can also extend around the actor 17 or above him, wherein the background rendering device 15 can in particular have a horizontal orientation above the actor 17. In order to be able to surround the actor 17 or to create a transition from the shown vertical orientation to a horizontal orientation, the background rendering device 15 or the lighting device 31 or the LED wall 33 can also be arched or curved at least in sections.

[0080] In addition to displaying the virtual background 21, the background rendering device 15 can also be used to illuminate the real subject 17 and thereby, for example, support additional studio lighting of the image recording studio 13. Furthermore, by illuminating the real subject 17 using the background rendering device 15, the interaction of the real subject 17 or the actor 17 with light sources present in the virtual background 21, such as lanterns or lamps, can be improved by the real subject 17 casting a shadow that corresponds to the lighting conditions visible in an image generated by the camera 23.

[0081] In order to generate the background representation 19 of the virtual background 21 and to control the pixel elements 35 for reproducing the representation, the background rendering device 15 has a control device 37 connected to a memory 39. In particular, a model of the virtual background 21 can be stored in the memory 39, so that the control device can generate the virtual background 21 based on the model. Furthermore, the control device 37 can be configured to project the virtual background 21 onto the background rendering device 15, and in particular the two-dimensional arrangement of the pixel elements 35.When generating the background representation 19, the control device 37 can, in particular, take into account a recording position A of the camera 23 in order to generate the background representation 19 in the same way as a camera 23 positioned in a real background corresponding to the virtual background 21 would perceive the real background. For this purpose, the control device 37 can, for example, use a game engine.

[0082] A possible embodiment of the camera 23 is shown in Fig. 2 schematically illustrates. The camera 23 has a camera body 53 to which a lens 59 is attached. The lens 59 can in particular be designed as an interchangeable lens, so that differently designed lenses 59 can be optionally connected to the camera body 53 and a lens 59 optimal for a particular recording can always be selected. The lens 59 has three lens rings 81, by means of which respective parameters of the lens 59 can be adjusted. For example, a focus distance, a focal length, a zoom factor and / or an aperture, in particular an opening of an iris diaphragm, can be set or adjusted by rotating a respective one of the lens rings 81.

[0083] To adjust the lens rings 81, a lens ring drive unit 85 is connected to the camera body 53 via a support rod 87, which has a respective lens servo motor 83 for each of the lens rings 81. These lens servo motors 83 can rotate the lens rings 81, thereby making adjustments to the lens 59. In particular, the lens ring drive unit 85 can be remotely controlled, so that the aforementioned lens parameters can be set or changed remotely.

[0084] Also arranged on the camera body 53 is a display device 49, via which information about the settings of the camera 23 can be displayed to a user. The display device 49 can, in particular, be a display. In addition, the camera 23 has an input device 51 arranged on the camera body 53, via which the user can adjust the settings of the camera 23. In particular, an exposure time of the camera 23 can be set on the input device 51, wherein a control device 25 connected to the input device 51 can be designed to control the camera 23 according to the entered exposure time. In particular, the display device 49 and the input device 51 can be formed by a touchscreen, via which information can be displayed to the user and user inputs can be received.

[0085] To image light entering through the lens 59 and to create real Fig. In order to be able to generate these images, the camera 23 has an image sensor 1 arranged within the camera body 53. This image sensor 1 can, for example, be based on CMOS technology or CCD technology and have a plurality of light-sensitive sensor elements that can be arranged in several rows and columns. Furthermore, the camera 23 has a readout circuit 97, which is designed to read out the signals of the respective sensor elements, to process them, to digitize them and to output them to or via a signal output 99. For this purpose, the readout circuit 97 can in particular comprise amplifiers, multiplexers, analog-digital converters, buffer memories and / or microcontrollers. Ultimately, an image data set B can thus be generated by the camera 23, which image data set B represents a real image generated by the camera 23. Fig. or represents an image of a field of view of the camera 23, and the image data set B can be output via the signal output 99. In order to check the field of view of the camera 23 and to align the camera 23 to a respective image section, a viewfinder 79 through which a user can see is also arranged on the camera body 53. Furthermore, the control device 25 is connected to a memory 47 and can, for example, be configured to optionally write the image data set B into the memory 47 or output it via the signal output 99.

[0086] Furthermore, the background display device 15 has an interface 103, and the camera 23 has an interface 101, via which, in particular, information I can be transmitted from the camera 23 to the background display device 15. In particular, the control device 37 of the background display device 15 can be designed to control the active illumination device 31 depending on information I received from the camera 23. In addition, the interface 101 of the camera 23 can also enable the image data set B and / or the information I to be transmitted to a data processing system 38, as will be explained below.

[0087] Fig. 3 illustrates again the execution of the moving image recording in the virtual image recording studio 13, wherein Fig. 3 furthermore a real image generated by the camera 23 Fig. illustrated. The real Fig. has the real motif 17 in a foreground 27 and contains the background image 19 in a background 29, so that basically the entire scene to be recorded is shown. However, in the real Fig. It can be seen that the background image 19 is reproduced on the background display device 15 by the pixel elements 35 and with a resolution determined by the background display device 15, which is ultimately determined by the arrangement of the pixel elements 35 or their density on the background display device. Since the resolution of the image sensor 1 of the camera 23 in the embodiment shown is higher than the resolution of the background image 19, in the real Fig. a grid 89 can be seen, which reflects the arrangement of the pixel elements 35 of the background display device 15.

[0088] In this respect, by using the background reproduction device 15 and reproducing the background representation 19 in the virtual image recording studio 13, it can be achieved that the actor 17 can adapt his acting to events taking place in the virtual background 21 and the lighting of the actor 17 by the background reproduction device 15 does not lead to in the real Fig. recognizable color errors in the form of a blue or green spill in the image foreground 27, but the actor 17 can be depicted in natural-looking colors. However, in the image background 29, due to the recognizable grid 89, it is still noticeable that the real Fig. was not recorded in a real background corresponding to the virtual background, but in front of the artificially generated background representation 19.

[0089] In order to map the grid 89 of pixel elements 35 in the real Fig. To suppress this, the lens 59 of the camera 23 could, for example, be operated with a wide open aperture, so that the background image 19 is only blurred in the real Fig. However, such a requirement for the settings of the lens 59 restricts the design options during the recording in the virtual image recording studio 13 in an undesirable manner and, in addition, a review of all real images generated during the moving image recording remains Fig. necessary to ensure that no artifacts caused by the reproduction of the virtual background 21 on the background reproduction device 15 remain in output images intended, for example, for a cinema film.

[0090] In order to counteract these problems and to obtain a natural-looking background in an output image 11 when using a background display device 15 for displaying a background representation 19 in a virtual image recording studio 13, the present application discloses a method for creating such an output image 11, which is described below by way of example with reference to Fig. 3 to 7 is explained.

[0091] In such a method, following a step 105 in which the real Fig. is generated (cf. Fig. 3), in a step 107 the real Fig. representing image data set B is received at a system 38 for data processing, so that the real Fig. can be provided to the system 38 as an input image 71 for creating the output image 11 (cf. Fig. 4). In particular, the image data set B can be transmitted to a system 38 which, in the course of post-production, processes the real images generated during the motion picture recording. Fig. is provided. However, it is also possible for the system 38 to form part of the control device 37, which is also provided for controlling the background display device 15. The system 38 can, for example, receive the image data set B directly from the camera 23 via the interface 101, or the image data set B can first be transmitted to the control device 37 and from there to the system 38.

[0092] How Fig. 4 also illustrates, the system 38 can further receive, in a step 109, camera position data K and background position data P, by means of which the recording position A of the camera 23 relative to the background reproduction device 15 can be determined. For this purpose, it can also be provided that the recording position A, for example coordinates of the camera 23 in a coordinate system defined with respect to the virtual image recording studio 13, is transmitted directly to the system 38. The background position data P can, in particular, determine a positioning of the background reproduction device 15 in the virtual image recording studio 13, so that an extension and orientation of the background reproduction device 15 in the virtual image recording studio 13 can be determined and / or determined based on the background position data P.

[0093] In addition, information I can be transmitted to the system 38 and received by the system 38, for example, an alignment of the camera 23 during the generation of the real Fig. so that a section of the background display device 15 imaged by the camera 23 can be determined based on the recording position A and the information I about the orientation of the camera 23. In addition, the information I can also contain information about a camera type of the camera 23 and / or a setting of the lens 59 during the generation of the real Fig. , such as a focus setting, an aperture, or a zoom factor. Furthermore, information I about the image sensor 1 of the camera 23 and / or a resolution of the camera 23 or the image sensor 1 can be provided.

[0094] How Fig. 5, the system 38 can then, in a step 111, in particular taking into account the camera position data K and the background position data P, on the basis of which the recording position A of the camera 23 is determined during the generation of the real Fig. can be determined, a virtual Fig. This is done by computationally reproducing - in other words: simulating - a recording of a reproduction of a substitute representation 57 of the virtual background 21 on the background reproduction device 15 by the camera 23 positioned at the recording position A. In this case, an image content of the substitute representation 57 corresponds to an image content of the background representation 19, wherein in the embodiment according to Fig. 5 the replacement representation 57, however, has an increased resolution compared to the background representation 19.

[0095] In order to be able to generate the replacement representation 57 with an increased resolution compared to the background representation 19, the system 38 can, for example, receive a replacement data set E which represents the replacement representation 57 (cf. Fig. 5). For example, it can be provided that the control device 37 of the background rendering device 15 is designed to determine the background representation 19 by projecting it onto the background rendering device 15, based on a model M of the virtual background 21 stored in the memory 39, and in doing so to take into account the resolution of the background rendering device 15, but also to generate the replacement representation 57 with a resolution that is increased compared to the resolution of the background rendering device 15. For this purpose, the control device 27 can, for example, first generate a representation of the virtual background 21 from the model M in a maximum possible resolution, in particular determined by the model M, and temporarily store or save this representation as a replacement data set E.Based on this replacement data set E, the control device 37 can then determine the background display 19 adapted to the background display device 15 and its resolution.

[0096] Alternatively or additionally, it can also be provided that the system 38 receives a background data set D which represents the background representation 19. For example, a background representation 19 generated by the control device 37 or a background data set D representing this background representation 19 can be stored and made available to post-production or to the system 38. The system 38 can then be designed in particular to determine the substitute representation 57 or a data set representing the substitute representation 57 by interpolation from the background data set D based on the background data set D, in order to thereby generate the substitute representation 57 with a resolution increased compared to the background representation 19.

[0097] Furthermore, the model M of the virtual background 21 can also be transmitted to the system 38 or read from a memory by the system 38, wherein the system 38 can be designed to determine the substitute representation 57 based on the model M as a function of the camera position A. In particular, the system 38 can be designed to virtually emulate a game engine which generates the background representation 19 in the virtual image recording studio 13 as a function of the recording position A of the camera 23. In this respect, in such an embodiment, the control device 37, which can generate the background representation 19 from the model M in the same way, can be virtually emulated.

[0098] Based on the substitute representation 57, a recording of the substitute representation 57 can then be computationally reproduced or simulated by the camera 23 during a (simulated) playback on the background playback device 15. In particular, this computational reproduction takes place without taking into account the real motif 17, so that the virtual Fig. can represent a pure background image, in which only the replacement representation 57 is contained. Furthermore, the determination of the virtual Fig. or the computational replication or simulation of the recording of the substitute representation 57, in particular taking into account information I received in step 109, which may, for example, relate to a setting of the lens 59 of the camera 23 or a resolution of the image sensor 1. For example, any sharpness effects, a brightness of the real Fig. and / or zoom effects in the virtual Fig. The virtual Fig. can then - especially in the form of a virtual Fig. representing virtual image data set B* - as a further input image 73 for creating the output image 11.

[0099] Creating the output image 11 based on the input images 71 and 73 - the real Fig. and the virtual Fig. or respective Fig. representing image data sets B and B* - is in Fig. 6, wherein the steps to be performed to create the output image 11 are described summarized as a step 113.

[0100] After determining the virtual Fig. System 38 has the real Fig. as input image 71 and the virtual Fig. as input image 73. By comparing the real Fig. and the virtual Fig. or the respective Fig. representing image data sets B and B*, the image foreground 27, which represents the real motif 17, can first be extracted from the real Fig. isolated. This can be achieved, for example, through edge detection.

[0101] The output image 11 can then be determined as an overlay image 63 in which the image foreground 27 or the real motif 17 the virtual Fig. In this illustration, on the one hand, the virtual motif 17 is in the foreground, which has no color errors due to the illumination by the background display device 15, and on the other hand, the virtual Fig. in which in particular the grid 89 of the background display device 15 is not recognizable due to the higher resolution of the substitute representation 57.

[0102] In addition, it can be provided, for example, that when creating the output image 11 in an environment 115 of the image foreground 27, a color adjustment F of the virtual Fig. through which the virtual Fig. the real Fig. in order to avoid any color jumps in the environment 115 of the image foreground 27 in the output image 11. Alternatively or additionally, the output image 11 in the environment 115 of the image foreground 27 can be designed as a soft blend between the virtual Fig. and the real Fig. designed to avoid the appearance of hard edges in the output image 11.

[0103] In particular, the procedure explained can be applied to all real Fig. which are carried out as a sequence of real Fig. represent the moving image recording, in order to be able to generate, for example, an entire cinema film without any recognizable imaging errors due to the imaging of the background playback device 15. Furthermore, it can also be provided that the generation of the input images 71 and 73 is carried out, for example, by the camera 23 and the control device 37 of the background playback device 15, so that the two input images 71 and 73 can be made available directly to a post-production and in the post-production, for example, the Fig. 6, step 113 for creating the output image based on the input images 71 and 73 can be performed. Furthermore, in some embodiments, the control device 37 of the background rendering device 15 can be configured to determine the output image 11, so that the output image 11 can be output, for example, by the control device 37 of the background rendering device 15 and made available to post-production for further processing.

[0104] Fig. Figure 7 further illustrates that, during the execution of the moving image recording, in addition to the background representation 19, position markers 65 can also be reproduced on the background reproduction device 15, which form a pattern 67 and, to a certain extent, overlay the background representation 19. In particular, individual pixel elements 35 can be controlled not to reproduce the background representation 19, but rather to reproduce a position marker 65.

[0105] By reproducing the position markers 65 on the background reproduction device 15, it can be achieved that the position markers 65 in the real Fig. and therefore form part of the image data set B representing the real image. The position markers 65 indicate within the real Fig. a respective position L, whereby this position L is determined by identifying the position markers 65 in the real Fig. For example, using a triangulation method, the position marker 65 in the real Fig. the recording position A of the camera 23 can be deduced, so that the control device 37 can determine the recording position A, for example by identifying the position markers 65 and their positions L, and can generate the background representation 19 depending on the recording position A thus determined. In addition, the recording position A determined on the basis of the positions L of the position markers 65 can be transmitted to the system 38 in step 109. However, it can also be provided that the system 38 is designed to determine the position markers 65 and their positions L in the real Fig. to determine the recording position A based on these positions L and to use them for the computational simulation of the recording of the reproduction of the substitute representation 57. Therefore, the camera position data K - as the positions L of the position markers 65 to be determined - can also be contained in the image data set B in some embodiments.

[0106] Various possibilities for inferring a recording position A of a camera 23 in a virtual image recording studio 13 on the basis of the position L of detected position markers 65 are described in particular in DE 10 2021 106 488 A1, so that reference is made to the explanations contained therein.

[0107] In particular, the position markers 65 may further form slight brightness deviations compared to a section of the background representation 19 surrounding a respective position marker 65, which are not recognizable to a human observer. For a human observer, the background representation 19 may appear unaffected despite the reproduction of the position markers 65, although the camera 23 may be configured to resolve such slight brightness deviations. The position markers 65 may also be reproduced as a geometric pattern to facilitate easy identification in the real world. Fig. It is also possible to play back the position markers only intermittently during the motion picture recording in order to also capture real Fig. generate and in particular to be able to use it for creating the output image 11, in which no position markers are shown. List of reference symbols 1 image sensor 11 Output image 13 virtual image recording studio 15 Background playback device 17 real motif, actor 19 Background illustration 21 virtual background 23 Camera 25 real illustration 27 Foreground of the image 29 Illustration background 31 Lighting device 33 LED wall 35 pixel elements 37 Control device 38 Data processing system 39 storage 41 panels 43 three-dimensional scene 44 LEDs 47 storage 49 Playback device 51 Input device 53 camera bodies 55 virtual image 57 Replacement representation 59 Camera lens, interchangeable lens 63 Overlay image 65 position markers 67 patterns 71 Entrance image 73 Entrance image 79 viewfinders 81 Lens ring 83 Lens servo motor 85 Lens ring drive unit 87 Grab bar 89 grids 91 first object 92 second object 93 third object 94 fourth object 97 Readout circuit 99 Signal output 101 Interface 103 Interface 105 steps 107 steps 109 steps 111 steps 113 steps 115 Surroundings A recording position B Image dataset B* virtual image dataset D Background dataset E Replacement data set F Color adjustment H Background dataset I Information K Camera position data L Location of the position markers M model P Background position data QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2021 106 488 A1 [0050, 0106]

Claims

[1] Method for creating an output image (11) based on a moving image recording which has been generated in a virtual image recording studio (13) by means of a camera (23) and a background reproduction device (15) arranged behind and / or above a real subject (17), comprising the steps: - Receiving an image data set (B) which represents a real image (25) recorded by means of the camera (23) in the virtual image recording studio (13), wherein the real image (25) contains a real motif (17) in an image foreground (27) and a background representation (19) of a virtual background (21) reproduced by the background reproduction device (15) is contained in an image background (29); - Receiving camera position data (K) and background position data (P), wherein a recording position (A) of the camera (23) relative to that of the background reproduction device (15) when recording the real image (25) in the virtual image recording studio (13) can be determined on the basis of the camera position data (K) and the background position data (P); - Determining a virtual image (55), wherein a recording of a reproduction of a substitute representation (57) of the virtual background (21) on the background reproduction device (15) at the recording position (A) of the camera (23) is computationally simulated, wherein an image content of the substitute representation (57) corresponds to an image content of the background representation (19), and - Creating the output image (11) as an overlay image (63) in which the image foreground (27) of the real image (25) is superimposed on the virtual image (55). [2] Method according to claim 1, wherein the resolution of the background representation (19) is determined by the background display device (15), in particular by an arrangement of pixel elements (35), in particular light-emitting diodes (44), on a panel (41) of an active illumination device (33) of the background display device (15), on which panel the pixel elements (35) are arranged in an at least two-dimensional arrangement. [3] Method according to claim 1 or 2, wherein the substitute representation (57) has a higher resolution than the background representation (19). [4] Method according to one of the preceding claims, wherein the virtual background represents a three-dimensional scene (43). [5] Method according to one of the preceding claims, wherein, in order to determine the virtual image (55), the recording of the reproduction of the substitute representation (57) is simulated computationally without taking into account the real motif (17). [6] Method according to one of the preceding claims, further comprising the steps: - receiving a background data set (D) representing the background representation (19), and - Determining the replacement representation (57) by interpolation from the background representation (19). [7] Method according to one of the preceding claims, wherein the substitute representation (57) is determined from a model (M) of the virtual background (21), in particular by computationally projecting a three-dimensional model (M) of the virtual background (21) onto the background display device (15). [8] Method according to claim 7, wherein the substitute representation (57) is determined from the model (M) as a function of the recording position (A) of the camera (23). [9] Method according to one of the preceding claims, wherein a replacement data set (E) is received which represents the replacement representation (57). [10] Method according to one of the preceding claims, wherein the camera position data (K) comprise information (I) about an orientation of the camera (23). [11] Method according to one of the preceding claims, wherein information (I) about a setting of a lens (59) is received during the recording of the real image (25), wherein the virtual image (55) is determined as a function of the information (I) about the setting of the lens. [12] Method according to claim 11, wherein the information (I) about the setting of the lens of the camera (23) comprises information about a focus position, a zoom factor, an aperture and / or a focal length. [13] Method according to one of the preceding claims, wherein information (I) about a camera type (23) of the camera (23) is received, wherein the virtual image (55) is determined as a function of the information (I) about the camera type (23). [14] Method according to one of the preceding claims, wherein information (I) about an image sensor and / or a resolution of the camera (23) is received, wherein the virtual image (55) is determined as a function of the information (I) about the image sensor and / or the resolution of the camera (23). [15] Method according to one of the preceding claims, wherein, in order to determine the virtual image (55), for each pixel of an image sensor of the camera (23) a respective section of the substitute representation (57) is determined which is imaged on the pixel. [16] Method according to one of the preceding claims, wherein, in determining the output image (11), the image foreground (27) of the real image (25) is isolated by comparing the image data set (B) with a virtual image data set (B*) which represents the virtual image (55). [17] Method according to one of the preceding claims, wherein, when determining the output image (11) in an environment (115) of the image foreground (27), a color adjustment (F) is carried out which approximates the virtual image (55) in the environment of the image foreground (27) to the real image (25). [18] Method according to one of the preceding claims, wherein the output image (11) is determined in an environment (115) of the image foreground (27) as a blend between the real image (25) and the virtual image (55). [19] Method according to one of the preceding claims, wherein the real image (25) comprises position markers (65) reproduced on the background reproduction device (15), wherein the camera position data (K) form part of the image data set (B) which represents the imaged position markers (65), wherein the recording position (A) is determined based on a position (L) of the position markers (65) in the real image (25); or wherein the position markers (65) are reproduced on the background reproduction device (15) during the moving image recording, and wherein the recording position (A) is determined during the moving image recording on the basis of the position (L) of the position markers (65) in the real image (25), wherein the determined recording position (A) is encoded in the camera position data (K). [20] Method according to claim 19, wherein during the execution of the moving image recording a sequence of background representations (19) is reproduced on the background reproduction device (15), wherein the position markers (65) in the sequence of background representations (19) are reproduced and not reproduced intermittently, in particular alternately. [21] Method according to claim 20, wherein the recording position (A) of the camera (23) is determined during the recording based on the position (L) of the position markers (65) in real images (25) comprising the position markers (65), wherein the determined recording position (A) is encoded in the camera position data (K). [22] Method according to one of claims 19 to 21, wherein the position markers (65) form a pattern (67) superimposed on the background representation (19). [23] The method of claim 22, wherein the pattern (67) comprises different geometric structures, the different geometric structures being contained in different color channels of the real image (25). [24] Method according to one of claims 19 to 23, wherein the position markers (65) form slight brightness deviations compared to a respective section of the background representation (19) surrounding the position markers (65), in particular wherein a brightness deviation between a respective position marker (65) and a section of the background representation (19) surrounding the position marker (65) is a maximum of 2%. [25] A data processing system (38) comprising means for carrying out a method according to any one of the preceding claims. [26] A computer program product comprising instructions which, when executed by a computer, cause the computer to carry out a method according to any one of the preceding claims. [27] Method for determining two input images (71, 73) for a method for creating an output image (11) for a moving image recording carried out in a virtual image recording studio (13) by means of a camera (23) and a background playback device (15), in which - a background representation (19) of a virtual background (21) is generated behind and / or above a real subject (17) as a function of a recording position (A) of the camera (23) relative to the background display device (15) and is displayed on the background display device (15), - a real image (25) of the real subject (17) and the background representation (19) is determined by means of the camera (23), which image comprises the real subject (17) in an image foreground (27) and the background representation (19) in an image background (29); - a substitute representation (57) of the virtual background (21) is determined, wherein an image content of the substitute representation (57) corresponds to an image content of the background representation (19); - a virtual image (55) is determined, wherein a recording of a reproduction of the substitute representation (57) of the virtual background (21) on the background reproduction device (15) is simulated computationally at the recording position (A) of the camera (23); and - the real image (25) and the virtual image (55) are output and / or stored as input images (71, 73) for creating the output image (11). [28] Method according to claim 27, wherein the substitute representation (57) has a higher resolution than the background representation (19). [29] Method according to claim 27 or 28, wherein the replacement representation (57) is determined by interpolation from the background representation (19). [30] Method according to one of claims 27 to 29, wherein the background representation (19) and the substitute representation (57) are determined from a model (M) of the virtual background (21), in particular by projecting the virtual background (21) onto the background display device (15). [31] Method according to one of claims 27 to 30, wherein position markers (65) for determining the recording position (A) of the camera (23) are displayed on the background display device (15), wherein the recording position (A) is determined based on a position (L) of the position markers (65) in the real image (25). [32] Method for creating an output image (11) based on a moving image recording which has been created in a virtual image recording studio (13) by means of a camera (23) and a background reproduction device (15) arranged behind and / or above a real motif (17), in which input images (71, 73) determined according to a method according to one of claims 28 to 31 are received and the output image (11) is created as an overlay image (63) in which the image foreground (27) of the real image (25) is superimposed on the virtual image (55).

Images