VIRTUAL THREE-DIMENSIONAL OBJECTS IN A LIVE VIDEO

DE502019012449D1Active Publication Date: 2024-11-28TRACKMEN GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE502019012449
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-09-13
Filing Date
2019-09-03
Publication Date
2024-11-28
Estimated Expiration
2039-09-03

AI Technical Summary

Technical Problem

Existing methods for integrating virtual objects into live video broadcasts struggle to maintain a natural appearance when camera perspectives change, often resulting in distortion or unnatural movements, and require complex data processing and manual post-processing.

Method used

A method involving generating an object video from multiple angles, recording the camera positions, and embedding this video into a live video environment, allowing the object to be displayed naturally from any camera perspective with minimal data and processing requirements.

Benefits of technology

Enables seamless integration of virtual objects into live video with natural appearance and movement, reducing data volume and manual processing, while maintaining a realistic view even when camera angles change.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to a method for displaying a three-dimensional object in a real environment, displayed in a final video displayed on a monitor.

[0002] In television and internet video productions, virtual objects are increasingly being embedded into live broadcasts. At first glance, viewers cannot distinguish between the real live video and the overlaid virtual object. Especially if the camera remains stationary during live production or while the live video is being generated, virtual overlays are almost unrecognizable. However, it becomes problematic when the camera position changes during recording, as the perspective of the overlaid object must then change accordingly.

[0003] For example, images or videos of people are often created and placed as two-dimensional images or videos in the room or studio by overlaying them. The viewer then sees an image displayed on their monitor or television, largely composed of the live video, but also embedded with the two-dimensional person.

[0004] However, if the camera recording the live image is moved so that it is no longer looking directly at the superimposed person, the images appear increasingly narrower. Ultimately, the superimposed person would no longer be visible if the camera were to move 90° around them and position itself to the side.

[0005] To avoid this effect, there is the option of optically rotating the image of the superimposed computer graphic as the camera moves around it. This prevents the image from appearing narrower due to the new angle, but it is then only seen from straight ahead from all perspectives, i.e., two-dimensionally.

[0006] It is also known to initially capture objects, such as people, using a three-dimensional scan. In this process, the depth of the object is measured with each image taken with the camera in order to ultimately create a three-dimensional model of the object. In the next step, the various images of the object from the respective perspectives are superimposed (mapped) to create the three-dimensional body. In essence, the individual images are "stretched" onto the body piece by piece. However, this occurs section by section, and the transitions cannot always be represented accurately. The three-dimensional shape of the object is also imperfect, which can often be seen, for example, in faces captured in such images; so-called artifacts arise.Regardless, the creation of such three-dimensional scans is complex and requires a large volume of data; manual post-processing is also necessary.

[0007] Another problem arises when the object is, for example, a person and is supposed to move in the subsequent live video. The three-dimensional scan then needs to be additionally animated, which requires knowledge or information about joints and possible directions of movement, for example. Even such an animation is rarely perfect; it is usually clearly recognizable that it is an animated video, and the movements often appear unnatural.

[0008] US 2015 / 358508 A1 describes a system for shooting a video film using a camera, a tracking module for determining the position of the camera and a composition module for generating an image composition of the camera recordings and a projection of a virtual animation, wherein the projection is generated depending on the camera position.

[0009] DE 10 2005 005795 A1 discloses a device for producing computer-aided object animation. An object is positioned on a turntable and recorded two-dimensionally with a camera from various angles until a complete panoramic view of the object is created.

[0010] EP 0 772 350 A2 describes a keying system and a method for generating image compositions. The system comprises a live camera for filming an object and a storage unit on which a plurality of background images are stored, each background image corresponding to a viewing point in a three-dimensional system with respect to the angle to the recorded background and the position of the object camera. The system further comprises a control unit for selecting the stored background image whose viewing point corresponds to that of the live camera, and a keying composition unit that combines the live camera image and the selected background image.The disadvantage of this is that the image composition is done using a chroma and / or luminance key, which means that a strong contrast is necessary between the object and an area to be overlaid by the background image, and a background image can only be displayed in a place that stands out from the rest of the surroundings with high contrast.

[0011] The object of the present invention is to integrate a virtual object into a generated live video in the most visually appealing way possible. In particular, perspective changes should be possible by moving the live camera without the superimposed object video appearing unnatural or becoming excessively distorted. It is also essential, however, that the method is as simple to implement as possible. It should require as little data volume as possible and, in particular, require only minimal or no manual post-processing.

[0012] According to the invention, the object is achieved by a method having the method steps of patent claim 1.

[0013] The core idea of ​​the invention is therefore that an object video is first generated that shows the object from different angles. For this purpose, the object is first recorded with a film camera from different angles or filming directions. The respective position of the film camera relative to the object is recorded and stored.

[0014] The term filming direction refers to a central axis of the object, which can be, for example, the vertical axis or the horizontal axis, but also any other central axis passing through the object.

[0015] In a second process step, this object video is loaded into a computer graphic and, according to the invention, displayed on a rectangle (i.e., only two-dimensionally as a video texture) that is positioned in the environment, hereinafter referred to as the studio. The studio is filmed with a live camera during recording, and a live video is generated. In this video, the object appears virtually to the viewer at a specific location in the room. Of course, the environment does not have to be a closed studio; the live video can also be generated at any outdoor location.

[0016] The video perceived by the viewer therefore consists of the live video with embedded object video. For simplicity, this video perceived by the viewer is referred to as the final video. The object video is adjusted to the position of the live camera using a standard data processing device.

[0017] It is essential that when filming the object, the relative angle between the object and the filming camera is known and that the video sequence that was filmed at this relative angle can be determined later.

[0018] In a particularly simple design variant, the object can be placed on a turntable that rotates during the filming process. The then stationary filming camera films the object, for example a person, continuously, preferably through 360°. The filming process is therefore one-dimensional, with the object simply rotated around its vertical or perpendicular axis. The assignment of the perspective or relative angle to a specific video sequence can be achieved, for example, using a corresponding measuring device on the turntable. An encoder that determines the respective degree of rotation is suitable for this. A linear assignment can also be achieved by counting the individual recorded images and then relating them to the corresponding number of degrees.

[0019] In a particularly advantageous, simple design variant, the number of degrees rotated and filmed by the turntable, usually 360 degrees, is divided by the number of recorded video images, resulting in the angle per video image. Although the calculated result is somewhat less accurate than that of an encoder, it simplifies the setup and is sufficient for most desired applications. This method can also be used for other perspectives, for example, when rotating the object around its horizontal axis or when filming a static or motionless object by moving the camera around the object's vertical axis.

[0020] In a particularly simple design variant, the turntable can have a visible scale on its exterior, which is also filmed during the filming process. This allows the recorded object video to show the perspective from which the object was filmed at a specific time, or the angle at which it was set. In this case, the object video is embedded into the live video without the scale, but the stored information resulting from the scale is still used as a video property for the desired display.

[0021] In an alternative design, the object is positioned in a fixed and immobile position. The filming camera then changes its position, for example, orbiting the object 360° and continuously filming it. Similar to the aforementioned design with a turntable, the relative position or angle between the filming camera and the object's central axis is continuously determined and stored.

[0022] Both variants create an object video that shows the object from different directions, preferably over a full 360°.

[0023] It is essential that the position of the recording live camera is also constantly determined. Common tracking systems can be used for this. If the relative angle between the displayed object, i.e. the rectangle or the object video shown within it, changes, the video sequence of the object video that corresponds to the relative angle of the live camera to the virtual object is played. If the live camera moves around the virtual object, the object is tracked by specifically playing the object video and always appears from the correct perspective. The object video is played back or forward as appropriate for this purpose. This means that the object, for example a person, always appears in the final video shown as the viewer would see it in the studio if it were real.

[0024] The method according to the invention can be further improved by recording the object video with the highest possible resolution. This makes it possible to clearly show partial areas of the object during subsequent playback by zooming in.

[0025] Furthermore, unlike state-of-the-art methods, the object can also move during filming, which is particularly advantageous when filming people. For example, a turn of the head or movement of the eyes, but ultimately even more complex movements can be filmed. Although these movements are always the same when reproduced in the final video, the result is a very natural movement, unlike with the familiar scanned and animated figures or methods. If, for example, the person rotates around their own vertical axis while the filming camera moves around it, care should be taken when recording the live video to ensure that the movement of the live camera corresponds to the movement of the filming camera. Otherwise, the movements would be slower or faster and therefore appear unnatural.

[0026] In principle, it's also possible to record multiple objects simultaneously to capture natural occlusions or shadows. Alternatively, different objects can be recorded individually and later combined into an object video.

[0027] According to the invention, it is also possible to tilt the turntable during filming in order to set a different filming angle to the filming camera.

[0028] As an alternative to arranging the object on a turntable that rotates around its vertical axis, the invention also allows for an arrangement on a horizontal shaft that rotates accordingly around the horizontal axis. This can be advantageous, for example, for objects that are also intended to be viewed from above. Examples of this include pieces of furniture, models of buildings, plants, and the like.

[0029] According to the invention, the objects can be attached to a rotating horizontal shaft and filmed by a stationary camera. Alternatively, it is of course also possible to move the filming camera around a stationary object in a vertical plane.

[0030] As an additional dimension, the distance of the recording camera from the object can also be recorded and saved during filming. This makes it possible to change the distance of the live camera from the virtual object even during the creation of the live or final video, as the object video can be adjusted accordingly.

[0031] According to the invention, in a particularly simple variant of the method according to the invention, the detection of the distance of the recording camera can be omitted. The object video or the video texture (the area in the final video over which the video is, so to speak, spanned) is arranged in a fixed location in the surrounding area. When the live camera moves away, this movement is measured by the camera tracking system, and the virtual scene, i.e. the object video or the area over which it is spanned, is shifted accordingly. It then automatically becomes smaller or larger, since the live camera is moving, like a natural object. Since only a section of the image is obtained, the perspective is slightly compressed or stretched, giving the object a somewhat flatter or steeper impression of depth.The perspective on the object may not be the correct one, but since the depth of the object, for example, is very shallow for humans, this is hardly noticeable to the viewer; the eye is very uncritical. This makes it possible to further reduce the effort and the amount of data.

[0032] The same applies to the camera height, which can also be changed and measured when generating the object video. This allows the height of the live camera to be changed accordingly during the generation of the final video.

[0033] Ultimately, the invention makes it possible to film the object from all directions in order to later represent all directions in a final video. However, it has proven particularly advantageous to create a simple, preferably one-dimensional shot, as this minimizes data volume. Most studio recordings are created one-dimensionally anyway, and the live camera usually remains at the same height during recording.

[0034] In a further advantageous embodiment of the invention, it is also possible to generate the object video live and to overlay it as a second live video onto the first live video recorded by the live camera. During the live recording in the studio, the position of the live camera is measured using a suitable camera tracking system. Instead of searching for the right spot in the video, however, the turntable is rotated, or controlled, so to speak, depending on the position of the live camera. The object or person is located simultaneously, i.e. live, on the turntable. A particular advantage is that the person can move live and react to the live events; the turntable always rotates the person in the right direction, which is dictated by the position of the live camera. The object video is therefore not buffered, but also broadcast live as a second live video within the first live video.

[0035] The invention is explained in more detail with reference to the following figures. These are intended merely to illustrate the principle of the invention, but are not intended to limit the invention to the exemplary embodiments shown. They show: Figure 1: a turntable with object and film camera, Figure 2: a device for horizontal rotation of the object and a film camera, Figure 3: an environment for the generation of a final video, Figure 4: a simplified representation of the final video from a first perspective, Figure 5: a simplified representation of the final video from a second perspective.

[0036] Please note that the figures are drawn in a highly simplified manner and, in particular, they are not to scale.

[0037] Figure 1illustrates a first possibility for generating an object video. An object 20 to be filmed is arranged on a turntable 22. A measuring device for detecting the rotation of the turntable 22, i.e., the angle of rotation, is provided on the turntable 22. The measuring device 24 is connected to a data processing device (not shown).

[0038] A film camera 26 is fixedly mounted and films the object 20 while the turntable 22 rotates, and its rotation is detected by the measuring device 24. In this way, an object video 28 is generated that shows the object 20 from various filming directions. In the illustrated embodiment, the vertical axis YY forms a central axis to which the filming direction refers. In principle, the turntable 22 can be rotated by just a few degrees, but also by 360° or even 720°; ultimately, this depends on the requirements of the object video 28 to be generated. The 720° rotation ensures that no connection errors or seams occur when moving completely around the object.

[0039] Figure 2shows an alternative in which the object is fixed to a horizontally aligned shaft 30. The filming camera 26 is also stationary and arranged at a fixed height above the object 20. Of course, the filming camera 26 can also be positioned below or to the side of the object. In this embodiment, the vertical axis XX forms the central axis to which the filming direction refers.

[0040] Figure 3 shows an exemplary environment 32 in which a live video 34 is to be generated. The environment is typically a studio for a television production. A live camera 36 generates the live video 34 and films, for example, a person 38 and an object 40, represented as a table. The live camera 36 can be moved within the environment 32, i.e., change its position.

[0041] The Figures 4 and 5illustrate a generated final video 42, which is formed from the recorded live video 34 and the object video 28. The object video 28 is embedded in the live video 34, so that the object 20 appears to the viewer to be stationary in the environment 32. Such a final video 42 is broadcast, for example, via a television station on television sets.

[0042] The Figures 4 and 5 illustrate the effect that occurs when the live camera 36 changes its position around the surrounding area 32. The perspective of the object 20 changes, which is nevertheless displayed in the correct perspective according to the invention.

[0043] The invention is not limited to the described and illustrated embodiments, but also encompasses further possibilities that can be realized based on the invention. For example, multiple object videos 28 can also be embedded in the final video 42. For example, two or more object videos 28, also recorded live, can also be embedded, for example, multiple individual people. The invention is defined by the independent claim.

Claims

1. A method for displaying a three-dimensional object (20) in real surroundings (32), displayed in a final video (42) played back on a monitor, having the method steps of - creating at least one object video (28) by filming the object (20) by means of a film camera (26) from different filming directions for creating an object video (28), wherein the angle of an optical axis of the film camera (26) with respect to a central axis of the object (20) is in each case determined for the respective filming direction, so that an angle can be assigned to each filming direction, - loading the object video (28) into a computer graphic, displaying the object video (28) on a rectangle and positioning the rectangle in the surroundings (32), - creating a live video (34) by means of at least one live camera (36) and simultaneously fading the object video (28) into the live video (34) at a defined position in the surroundings (32) for creating the final video (42), - changing a relative angle between the rectangle with the object video (28) shown therein and the live camera (36) by changing the position of the live camera (36) in the surroundings (32) and simultaneously displaying the object (20) in the final video (46) in a desired perspective in the surroundings (32), in that - the respective position of the live camera (36) in the surroundings (32) is determined while recording the live video (34), - that video sequence of the object video (28), which corresponds to the relative angle of the live camera (36) to the object (20), is played, so that the object video (28), adapted to the determined position of the live camera (36), shows the object (20) in the final video (46) from that position, which corresponds approximately to the direction of the live camera (36) to the desired position of the object (20) in the surroundings (29).

2. The method according to claim 1, characterized in that the filming of the object (20) comprises - stationarily arranging the film camera (26), - turning the object (20) around its normal axis (Y-Y).

3. The method according to claim 2, characterized in that the filming of the object (20) comprises - positioning the object (20) on a turntable (22), - turning the turntable (22) and simultaneously filming the object (20) by means of the film camera (26), - determining the filming direction by determining the number of degrees of the turning of the turntable (22).

4. The method according to claim 1, characterized in that the filming of the object (20) comprises - stationarily arranging the film camera (26) - turning the object (20) around its horizontal axis (X-X).

5. The method according to claim 1, characterized in that the filming of the object (20) comprises: - stationarily arranging the film camera (26), - simultaneously turning the object (20) around its normal axis (Y-Y) and its vertical axis (x-x).

6. The method according to claim 1, characterized in that the filming of the object (20) comprises - stationarily and statically arranging the object (20), - filming the object (20) while simultaneously moving the film camera (26) in a horizontal plane around the object (20).

7. The method according to claim 1, characterized in that the filming of the object (20) comprises - stationarily and statically arranging the object (20), - filming the object (20) while simultaneously moving the film camara (26) in a vertical plane around the object (20).

8. The method according to one of claims 1 to 7, characterized in that the film camera (26) is moved by at least 360°, preferably by 720°, around the object (20) during the filming.

9. The method according to one of claims 1 to 8, characterized in that the position of the film camera (26) is varied during the filming in such a way that the object (20) is filmed from all directions.

10. The method according to one of claims 1 to 9, characterized in that the filming directions of the film camera (26) to the central axis of the object (20) is stored and the object video (28) is adapted to the determined position of the live camara (36) in the surroundings (32) by playing a video sequence of the object video (28), which shows the object (20) from that direction, which corresponds approximately to the direction of the live camera (36) to the desired position of the object (20) in the surroundings (29).

11. The method according to one of claims 1 to 10, characterized by dividing the turned and filmed number of degrees of the object (20) by the recorded number of video images as basis for finding that video sequence, which shows the object (20) in the final video (34) in the desired perspective.

12. The method according to one of claims 1 to 10, characterized in that the at least one object video (28) is embedded into the first live video as second live video.