DEVICE AND METHOD FOR OVERLATING AT LEAST A PART OF AN OBJECT WITH A VIRTUAL SURFACE

DE502016016787D1Inactive Publication Date: 2024-11-28BITMANAGEMENT SOFTWARE GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE502016016787
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2015-11-20
Filing Date
2016-11-21
Publication Date
2024-11-28
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Current methods for overlaying 3D objects with virtual surfaces suffer from low resolution, noise, and latency, making real-time applications like video conferences difficult due to the limitations of optical reference determination and visible reference points, which restrict the accuracy and extent of the overlay.

Method used

A device using a contactless position determiner to achieve absolute position measurement of reference points, combined with stereo cameras to generate a high-resolution 3D grid structure, allowing for precise alignment and overlay of virtual surfaces with detailed surface information, enabling real-time rendering and seamless transitions.

Benefits of technology

This solution provides high-resolution, real-time overlay of virtual surfaces with minimal latency, allowing for lifelike animations and precise alignment, suitable for applications like video conferencing and virtual reality, without the need for infrared lighting or visible markers.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The present invention relates to a device and a method for superimposing at least part of an object with a virtual surface, wherein the object is a person. Exemplary embodiments show the superimposition of the face of an actor with a mask or the face or the entire body of another person. The orientation and shape of the mask or the face of the other person is determined, for example, using an absolute reference point on the actor's face, the position and orientation of which are determined absolutely and thus not just relative to the recording 3D cameras. The absolute position determination can be carried out, for example, using sonar or another contactless positioning method.

[0002] Known methods for giving objects recorded in 3D different surfaces use optical reference determination. This can, for example, be a regular or irregular arrangement of points on the object, such as the actor's face. The points can be glued on or placed on the actor's face using infrared technology so that they are visible to the 3D cameras. However, this method can only determine the reference points in their position relative to the cameras with an accuracy of approximately ± 1 cm. Furthermore, this method is characterized by comparatively high noise. With a maximum range of five meters, this method cannot depict expansive scenes.In addition to the low resolution resulting from the number of reference points on the actor's face, the facial recognition used to track the reference points is comparatively slow, preventing real-time use, for example, in video conferences, as the resulting latencies make continuous communication and discussion between conference participants difficult. Furthermore, the low resolution makes the reference points recognizable in the overlaid video, as the mask is only adjusted to the actor's face at these positions.

[0003] If conventional dots are glued to the actors, these dots can be searched for in two shots taken from different angles. The 3D depth of the dot is determined from the difference in the viewing direction of the same dot and the positions of the cameras. The position of the glued dots in space controls a computer animation of a 3D mesh. The animation is then controlled and interpolated based on only these few dots. A lifelike model is therefore not possible due to the low resolution of the 3D mesh and the low dot density.

[0004] WO2013167901 (A1) relates to a technique for mixing or compositing computer-generated 3D objects in real time and a video feed from a movie camera, in which the movie camera can be moved in 3D and in which sensors mounted in or on the camera provide real-time position data defining a 3D position and 3D orientation of the camera or allowing the 3D position to be calculated.

[0005] The Wikipedia article XP055338471 "Photogrammetry" describes measurement methods for taking measurements of objects using photographs to determine 3D coordinates.

[0006] US 2010271368 A1 describes a technique for applying a 3D scan of a physical target object to a virtual environment. A 3D scan of a person's head, or part of it, is applied to a virtual body in a virtual environment. This is used to give virtual characters in a video game environment realistic faces and facial expressions. A hybrid approach is used, including surface analysis to generate a 3D scan and a conventional motion capture technique to provide spatial context to associate with the 3D scan.

[0007] The present invention is therefore based on the object of creating an improved concept for overlaying objects in a 3D image with a virtual surface.

[0008] This problem is solved by the subject matter of the independent patent claims. Further developments according to the invention are defined in the subclaims.

[0009] Embodiments show a device for superimposing a virtual surface on at least part of an object. The device comprises a (contactless) position determiner configured to determine an absolute position of a reference point on the object in an environment. Furthermore, the device comprises a computing unit that, based on the absolute position of the reference point, determines depth information from a plurality of voxels or vertices in a 3D image of the object, generates a 3D grid structure of the object, and superimposes the virtual surface on the 3D grid structure.

[0010] The present invention is based on the idea of ​​locating an object being recorded in 3D using a separate position determiner, which makes it possible to determine the object's position absolutely (and not just relative to the recording cameras) in its surroundings. This position determination is advantageous because reference points are used to align the overlay image data, the position of which can be determined by the position determiner in the sub-millimeter range. Furthermore, (stereo) cameras capable of capturing 3D images are capable of determining depth information of pixels relative to the camera. The combination of the absolute position value of the global reference point can then be compared with the relative position value obtained by the cameras, and based on this comparison, absolute depth information can be determined for each additional reference point, e.g., each voxel or each vertex of the recorded 3D image.A 3D grid structure with the highest possible resolution is created for overlaying the virtual surface, over which the virtual surface or a texture of it can be placed. This makes it possible to extract even very detailed surface information from the virtual surface after overlay.

[0011] According to embodiments, the computing unit can calculate a 3D grid structure of the virtual surface in addition to the 3D grid structure of the object. The 3D grid structure of the object can then be adapted to the 3D grid structure of the virtual surface in order to overlay the adapted 3D grid structure with a texture of the virtual surface. To simplify this step for multiple applications, a difference grid structure can be determined from the 3D grid structure of the object and the 3D grid structure of the virtual surface, which is then applied to the 3D grid structure of the object.In embodiments that show a sequence of 3D images in which the object is to be overlaid with the virtual surface, the difference grid structure can be adapted depending on a change in the object's reference points in the 3D grid structure used to calculate the difference grid structure and the object's 3D grid structure from the current image in order to transfer a movement in the object to the virtual surface. This is advantageous because, for an existing difference grid structure, which was obtained, for example, from an initial image of the object and the virtual surface, only the change or difference in the position of reference points (e.g., each voxel or each vertex) compared to the initial image can be determined, and the difference grid structure is adapted or distorted according to the determined deviation.This makes it possible to transfer the person's facial expressions or gestures to the virtual surface with minimal computing effort.

[0012] To capture the 3D image or sequence of 3D images, two identical cameras can be used, each capturing an image of the object from two different viewing angles. The computing unit can generate a 3D image from the captured images and determine the depth information of the plurality of voxels or vertices in the 3D image relative to a position of the two cameras. Comparing the relative depth information of the reference point with the absolute position of the reference point, which is obtained from the position determiner, allows absolute depth information of the voxels or vertices to be calculated based on the relative depth information of the voxels or vertices.This is advantageous because the 3D rendering, the determination of the absolute depth information of the voxels or vertices, and the overlay with the virtual surface can be calculated in a central processing unit, which in embodiments can also be integrated in at least one of the two cameras. In embodiments in which the processing unit is integrated in at least one of the two cameras, the 3D image overlaid with the virtual surface is output directly from the camera. Since the overlay can take place in real time, a director of a 3D film, for example in which the faces of the actors are overlaid with virtual faces, is able to view the acted scene in real time, i.e. without any visible time delay to the actions of the actors and without beforehand recording and editing the scene, directly from the camera on a monitor and to assess the scene as a whole.

[0013] Further embodiments show that the computing unit can form a combined surface in which a part of the object that is excluded from the overlay with the virtual surface comprises a real surface area of ​​the object. Thus, for example, the aforementioned overlaid faces of the actors can show the actor's real mouth or eyes, while the rest of the face is concealed by a mask or the virtual surface. For this purpose, the excluded parts of the actor's face can also be moved to the appropriate position on the mask, for example, if the mouth or eye area of ​​the mask and the actor's face are not congruent.

[0014] According to embodiments, in addition to the natural reference point, which on a person's face could be, for example, the tip of the nose, additional external reference markers can be attached to the object. These external reference markers or reference points enable the position of the object to be determined with greater accuracy, for example, when diffuse light or weak illumination is present. They thus provide an additional orientation point for the cameras or the overlay algorithm.

[0015] According to further embodiments, at least three sound sources can be attached to the object, each emitting a unique audio signal that differs from the other sound sources. The position determiner is further connected to receivers that are arranged at different angles around the object and receive the audio signals. The position determiner evaluates runtime information from the three audio signals to calculate a position and orientation of the reference point in the environment based on the runtime information. For example, radar, sonar, or ultrasound can be used to determine an extremely precise position of the reference point in the environment and also to determine an orientation of the reference point. In other words, starting from the reference point, a (unit) vector can be calculated that determines the position and orientation of the reference point based on an initially defined direction.A maximum deviation of the position of the object can be between (±) 5 mm and 10 µm.

[0016] Preferred embodiments of the present application are explained in more detail below with reference to the accompanying drawings. They show: Fig. 1 shows a schematic block diagram of the device; Fig. 2 shows a schematic representation of the superimposition of a person's mouth with the virtual surface of a computer-generated mouth; Fig. 3 shows a schematic arrangement of the device according to an embodiment; Fig. 4 shows a schematic representation of a relationship between an absolute coordinate system and a relative coordinate system, which is related, for example, to a position of one or a plurality of cameras; Fig. 5 shows a schematic flow diagram of a method for superimposing an object with a virtual surface.

[0017] In the following description of the figures, identical or equivalent elements are provided with the same reference numerals so that their description is interchangeable in the different embodiments.

[0018] Fig. 1 shows a schematic representation of the device 2 for superimposing at least a portion of an object 4 with a virtual surface 6. The device 2 has a position determiner 8 and a computing unit 10. The position determiner 8 is designed to determine an absolute position 12 of a reference point 14 on the object 4 in an environment. Furthermore, the computing unit 10 is designed to determine, based on the absolute position 12 of the reference point 14, depth information from a plurality of voxels or vertices in a 3D image 16 of the object 4 and to generate a 3D grid structure of the object 4 and to superimpose the virtual surface 6 on the 3D grid structure. The 3D image of the object can be recorded, for example, by two cameras, whose individual images of the object are combined to form a 3D image. In the Fig. 1 In the embodiment shown, the object 4, here a hexagon, is partially overlaid with the virtual surface 6, here a square. The computing unit 10 outputs a 2D or 3D image 18 of the object 4, which is overlaid with the virtual surface 6.

[0019] Fig. 2 shows a schematic representation of the object 4, here for example the face of a person, in which the person's mouth is overlaid with the virtual surface 6, here for example a computer-generated mouth. The 3D image 16 of the object 4 is generated, for example, using cameras. In the 3D image 16, the position determiner tracks a (natural) reference point 12 on the object, here the tip of the person's nose. The computing unit then calculates, based on the absolute position of the reference point 12, depth information from a plurality of voxels or vertices 20 in the 3D image 16, here in the area of ​​the person's mouth. From the voxels or vertices 20 and the associated depth information, a 3D grid structure 22 of the object, here only the person's mouth, which is to be overlaid, is generated. The virtual surface 6 is superimposed on the 3D grid structure 22 to obtain the output image 18.The output image 18 can be in 2D or 3D depending on the terminal device on which the output image is to be displayed, or a 2D terminal device can calculate a two-dimensional representation of the 3D output image.

[0020] According to embodiments, the computing unit is configured to calculate a 3D grid structure 24 of the virtual surface 6. The 3D grid structure 22 of the object 4 can then be adapted to the 3D grid structure 24 of the virtual surface 6 in order to obtain an adapted 3D grid structure and to superimpose a texture of the virtual surface 6 onto the adapted 3D grid structure. In other words, the virtual surface can be divided into a 3D grid structure (mesh), which describes the shape of the virtual surface, and the texture of the virtual surface, i.e., the visual appearance. Thus, the virtual surface can be adapted to the object, creating the impression of a seamless transition from the superimposed surface to the person's face.For this purpose, it is advantageous to also adapt adjacent voxels that do not belong to the 3D grid structure 22, for example by means of interpolation, so that a gradual transition between the depth information of the voxels in the person's face and the voxels of the virtual surface is adapted.

[0021] According to a further embodiment, the 3D grid structure can be superimposed on the virtual surface using a difference grid structure. The difference grid structure can be determined using the computing unit from the 3D grid structure of the object and the 3D grid structure of the virtual surface. The difference grid structure can then be applied to the 3D grid structure of the object in order to adapt the 3D grid structure of the object to the 3D grid structure of the virtual surface and to obtain the adapted 3D grid structure. The application of the difference grid structure to the 3D grid structure of the object can be done, for example, by means of point-wise, i.e., voxel-wise or vertex-wise addition, if the difference mask was generated, for example, from the difference between the 3D grid structure of the object and the 3D grid structure of the virtual surface. Advantageously, the calculation of the 3D grid structure can be carried out initially oronce based on a 3D model of the object and the virtual surface for each point, i.e., for example, a voxel or vertex, or each selected reference point of the object and the virtual surface in a 360° panoramic view. In other words, a 3D grid structure can be created for each object and the virtual surface from a 360° panoramic view of the object and the virtual surface, so that the difference grid structure can be created, for example, based on a voxel-wise, vertex-wise, or generally reference point-wise subtraction of the 3D grid structure of the virtual surface and the 3D grid structure of the object.

[0022] If a sequence of 3D images or a video of the object is available, the difference mesh structure can be adjusted to transfer a movement in the object to the virtual surface. For this purpose, the computing unit can calculate an adjustment value based on a relative deviation of the 3D mesh structure of the object in a current 3D image from the 3D mesh structure of the object with which the difference mesh structure was generated. In other words, a difference of a mesh of the object in the current 3D image (from the sequence of 3D images or the current 3D video frame) can be calculated, at least for the visible voxels or vertices, from the corresponding voxels or vertices of the mesh of the initial 3D model.Based on these determined deviations between the current 3D grid structure and the initial 3D grid structure, the difference grid structure can be adjusted, for example, again through vertex-wise or voxel-wise addition, to transfer an (internal) movement of the object, such as a changed facial expression, to the virtual surface. This is done by overlaying the final 3D grid structure with the texture of the virtual surface, whereby the texture is adapted to the final 3D grid structure.

[0023] For example, reference points on a person's face, which describe the position of an eyelid or the angle of the mouth, are tracked in order to transfer, for example, a blink or a speech of the person to the overlaid mask. Other, preferably unchanging reference points, such as the tip of the nose or the earlobes, can be used to determine the person's line of sight so that the virtual surface or mask superimposed on the person's face can also be perspectively adapted to the person's or actor's point of view. In the simplest case, this is achieved by selecting the corresponding visible section of the difference grid structure, provided that this was created from initial 360° 3D models of the object and the virtual surface. As already implicitly mentioned, the 3D grid structure can assume or achieve any resolution.If highly detailed images are required, each voxel or vertex of the object can form a grid element of the 3D grid structure.

[0024] In a preferred embodiment, the computing unit is designed to overlay a person as an object with a virtual person as a virtual surface or a face of a person as an object with a virtual face as a virtual surface, wherein a facial expression or gesture of the person is preserved by the overlay of the 3D grid structure of the virtual surface.

[0025] According to embodiments, the computing unit can also be configured to form a combined surface, wherein a part of the object that is excluded from the overlay with the virtual surface comprises a real surface area of ​​the object. For example, a part of the person's face, e.g., the mouth, eyes, nose, or ears, can be excluded from the overlay with the virtual face or mask, so that the real part of the actor's face can be recognized at this point. However, it may also be necessary, for example, to display the actor's mouth or eyes at the corresponding location on the mask, even though the facial proportions of the person and the mask are not congruent, and thus the mouth of the mask, i.e., the virtual surface, is located at a different location than the mouth of the person or actor.This allows the actor's real mouth area to be cut out of the current 3D image and inserted into the designated place on the mask.

[0026] Fig. 3 shows a schematic representation of the device 4 in a block diagram with some optional features that can be implemented in different embodiments or in combination. Thus, in embodiments, the device 4 has two similar cameras 26a, 26b, each configured to capture an image 28a, 28b of the object 4 from two different viewing angles.The computing unit 10 is further configured to generate the 3D image from the images of the two cameras 26a, 26b, to determine the depth information of the plurality of voxels or vertices in the 3D image relative to a position of the two cameras, and to compare the relative depth information of the reference point 14 with the current (absolute) position (or the absolute depth information derived from the position) of the reference point in order to obtain absolute depth information of the voxels or vertices based on the relative depth information of the voxels or vertices. In other words, the computing unit 10 determines a 3D image of the object 4 from the images of the 2D cameras 26a, 26b. Based on the absolute position 12 of the reference point 14, the computing unit can overlay the object 4 in the 3D image with the virtual surface 6, and generate the output image 18.

[0027] In exemplary embodiments, the virtual surface 6 can be provided by a simulation unit 30, which is designed to generate a 360° 3D model of the virtual surface. The computing unit 10 is further designed to superimpose a section of the virtual surface on the object, depending on a viewing angle of a user viewing the initial image 18 or a viewing angle of the camera 26 on the object. The simulation unit 30 is optionally part of the device 2. The virtual surface can also be calculated or created from another source, for example, an exclusively computer-generated model of the virtual surface without an explicit real template. The division of the virtual surface into the 3D grid structure, which is advantageous for the overlay process, as well as the texture of the virtual surface can be carried out in the computing unit or the simulation unit.In the case of a computer-generated model, the texture and 3D mesh can also be provided separately.

[0028] According to embodiments, at least three sound sources 32a, 32b, 32c can be attached to the object 4, wherein each of the three sound sources 32a, 32b, 32c is configured to emit an audio signal that differs from the audio signal of the other sound sources. The position determiner 8 can further be connected to receivers 34 that are arranged at different angles around the object 4 and receive the audio signals. From the received audio signals 36, the position determiner can determine travel time information of the audio signals from the sound sources to the receivers and, based on the travel time information, calculate a position and orientation of the reference point 14 in the environment. Thus, starting from the reference point 14, the position determiner can determine a vector having a direction that, compared to an initial or origin direction of the vector, describes an orientation of the object or the reference point.Alternatively or in addition to the provisions in . Fig. 3 In addition to the sound sources 32 shown (e.g. loudspeakers), which can use sonar, radar or ultrasound, further signal sources 32 such as GPS transmitters, RFID tags, lasers or reflectors for laser radiation can also be attached to the object. The previous explanations apply accordingly to these further signal sources 32. These are advantageously arranged in such a way, for example at the back of the head of the person being recorded if a person's face is shown predominantly from the front, that they are not visible to the cameras. This reduces the use of image processing algorithms to retouch the sound sources. The receivers 34 are advantageously matched to the type of signal sources 32 so that they can receive the signals and the position determiner determines a travel time based on the received signals.

[0029] If the object is a person (or the person's face), the position determiner is thus able to determine the absolute position of the reference point using signal sources 32 (or reflectors that reflect signals from signal sources) and receivers 34. The signal sources (or reflectors) can be arranged outside the face, e.g. at the back of the head or in the hair of the person. The receivers are further connected to the position determiner 8. The receiver can thus determine a travel time of the signals from the signal sources to the receivers and determine the absolute position of the reference point in space. For this purpose, it is advantageous if the spatial arrangement of the signal sources to the reference point is known, since the position determiner can only directly determine the position of the signal sources or reflectors (arranged outside the person's face) and calculates the position of the reference point based on the spatial arrangement of the signal sources (or reflectors).Reflectors) to the reference point can be determined. The reflectors can, for example, reflect laser beams or light in the visible or invisible frequency spectrum. The signal sources can be arranged at a spatial distance from the object if appropriate reflectors are attached to the object or person.

[0030] It should be noted that the reference point can advantageously be a single fixed point on the object. Fig. 3 The embodiments shown with multiple transmitters and receivers do indeed measure the positions of the individual transmitters or receivers on the object. This allows real-time tracking or real-time position determination of the object, since such a position determination of a point can be carried out faster and more precisely than, for example, using arrangements that use two imaging methods (e.g., two different camera systems) and thus not only (exactly) track or locate a reference point, but also, for example, record the face of a person with both different camera systems and combine these images (or use information from one image to improve the second image). Thus, for example, using the described position determiner (or method for position determination, see, for example, Fig. 3 ) it is possible to display interactions between objects (e.g. if these are people) in a virtual conference room, for example, without any visible delay.

[0031] Additionally or alternatively, the computing unit can determine the depth information based on stereoscopic images from a stereoscopic camera, wherein the position determiner is configured to determine the absolute position of the reference point based on signal sources or reflectors on the object, wherein the signal sources are arranged in a manner that is not visible (or invisible or hidden) to the stereoscopic cameras. However, it may be sufficient that the object from which the 3D grid structure is generated is not obscured by the signal sources or reflectors from the camera's viewing angle. This means that the signal sources or reflectors can be arranged outside the face (e.g., at the back of a person's head) if the 3D grid structure is generated from the person's face.Thus, no signal sources or reflectors obscure the person's face, which is overlaid with the virtual surface.

[0032] In other words, the stereoscopic cameras can be aimed at the object from a specific viewing angle. Based on the stereoscopic images, the 3D grid structure can be generated from the object's viewing angle. The signal sources or reflectors can then be positioned away from or spatially spaced from the part of the object visible from the viewing angle, so that the signal sources or reflectors avoid overlapping the object from the viewing angle of the stereoscopic cameras. To create the 3D grid structure, the signal sources or reflectors then do not need to be subtracted from the (stereoscopic) camera image. Using the face as an example, the signal sources or reflectors can therefore be visible to the cameras next to or above the head, for example, but they do not obscure the face.

[0033] In the described method, the position determiner is capable of determining the position of the reference point 14 with a maximum deviation of between 5 mm and 10 µm, between 1 mm and 1 µm, or advantageously between 0.5 mm and 0.1 mm. Thus, using sound sources, GPS, RFID, and / or laser scanning, it is possible to determine an absolute position of an object or a reference point on the object in an environment surrounding the object for aligning images from a 3D camera in order to obtain depth information for a 3D recording of the object. Depth information is, for example, the absolute distance of the object or the reference point from the cameras. The cameras are only capable of determining depth information from pixels relative to another pixel, but not the absolute distance of the point from the camera.

[0034] According to further embodiments, the position determination can be configured to determine the absolute position of the reference point 14 based on a natural reference point, for example, the tip of a person's nose, which is provided by the object, and based on an external reference point 38, for example, a marker or reflective ball attached to the object, in order to obtain a position determination with improved accuracy. This can be advantageous, for example, in poor lighting conditions.

[0035] Fig. 4 shows a schematic representation of a relationship between an absolute coordinate system 50 and a relative coordinate system 52. The absolute coordinate system is determined by the position determiner 8. The position determiner records movements of the reference point 14 absolutely in the coordinate system 50. A movement v O of the object, or more precisely of the reference point, in the x-direction results in a change in the x-position 12b of the reference point on the x-axis of the absolute coordinate system 50. The position 12 of the reference point is represented here in a simplified manner in two dimensions (x, y), with a position vector 12 being composed of its components 12a and 12b. The two-dimensional representation is further simplified and can be expanded analogously to three dimensions. Based on a position of the camera(s) 26, a relative coordinate system 52 can be defined, wherein a movement v K of the camera(s) in the x-direction results in a displacement of the coordinate system 52.Thus, an absolute position of a reference point on the object cannot be clearly determined using the cameras. However, the position determiner in the absolute coordinate system provides both an absolute position 12 of the reference point 14, which can be compared with a relative position in the relative coordinate system 52, which is obtained by the cameras. By comparing the positions of the reference point in the two coordinate systems, an absolute position of each voxel or vertex in a 3D image can be calculated. Thus, on the one hand, a position in the 3D image to which the superimposed surface is aligned, as well as the depth information of each voxel or vertex in the 3D image, is known, so that the virtual surface can be aligned in its position and the object can be adapted to the virtual surface. This means that, for example, mass images can also be realized in which objects such as, for example,Faces of people in a crowd can be reliably overlaid with virtual surfaces.

[0036] In other words, the 3D camera recording or other 3D scanning creates almost any number of 3D points, e.g. voxels or vertices, in space. In comparison to known techniques, this makes it possible to achieve a significantly higher degree of animation of an object with a virtual superimposed surface, theoretically down to every single pixel, voxel or vertex in the face or object. For this purpose, a mesh 22 is created from the 3D camera image 16, which is an animated mesh in a sequence of 3D camera images or a 3D video image. Differences in the mesh grids can be calculated from a 3D mesh model of an object, e.g. a deceased actor, and the mesh model of the object from the 3D camera image or a live 3D depth image series of a living person. These differences can relate to changed depth information, i.e. a changed distance in space, e.g. B. in relation to the recording cameras.

[0037] Furthermore, reference points can be marked at prominent locations on the object, for example the nose or mouth, on a person's face, so that by comparing the reference points in the mesh model, a different anatomy, such as the distance between the tip of the nose and the mouth, can be transferred from the virtual surface to the person's face. These differences are saved, and the live 3D image 18 is displayed with an associated delta value that represents the differences. A (live) texture of the virtual surface, e.g. the face of a living person, can be changed (at runtime) in such a way that the live texture is distorted in order to obtain the exact dimensions of the mask or the adapted 3D grid structure and to overlay the adapted 3D grid structure with the texture. In a further step, the texture of the eyes and mouth can be distorted or morphed so that it matches the geometry of the object orof the actor. To enable faster location of reference points that are not located via the positioner, for example, they can be made up to achieve greater contrast. This avoids excessive delay in the image processing system that detects the reference points.

[0038] Compared to known methods, the device according to the invention has a number of advantages. For example, only natural light is used for recording. Infrared illumination of the actor is not necessary. Furthermore, the 3D video can be recorded at, for example, 60 frames per second in 4K resolution, which is already achievable with standard cameras. Normal lenses can also be used. Infrared detection of the cameras is not necessary. The accuracy of the method can be more than a tenth of a millimeter and can reach a resolution corresponding to the resolution of the camera image. Furthermore, due to the high resolution and the absolutely determined position of the reference point, it is also possible to overlay actors' faces with different security features of other people in mass shots, as well as distant and therefore smaller objects with virtual surfaces.

[0039] In other words, at least two film cameras can film a person or an actor at a suitable distance from each other. The person can first be measured in three dimensions, from which a 3D model of the head and of the entire person can be created, from which the 3D grid model of the object is determined. In the case of a person, the tip of the person's nose, for example, serves as a reference point for a real-time measuring system 8, from which the 3D model of the person can be determined. The principle can also be applied to other objects, with suitable reference points being selected depending on the object. The real-time measuring system, such as Nexonar, can function with sound waves according to embodiments. The advantage of sound wave measuring orother measuring systems that can determine an absolute position of the reference point is that no color change or irradiation on the face or body of the person is necessary.

[0040] For example, the face is initially measured at prominent points in the 3D model. This can begin at the reference point for which an absolute position is available, such as the tip of the nose. The 3D mesh generated by the person or actor is then measured to the reference point in such a way that the distance or position of each individual vertex to the reference point is determined. At all points that do not change, such as the tip of the nose, the measured vertex is then used as the depth value. This results in a very precise reference model of the person, since the 3D model can have a very large number of points, for example approximately 140,000 or more, up to the number of voxels present in the person's face. A 3D depth value model of the recording is then determined from the position of the two cameras 32a, 32b using geometric calculations.A 3D value is determined from the differences between the two images 28a, 28b using geometric triangulation and the precise position of the cameras 26a, 26b and the reference model. This 3D value is advantageously located between the measured points of the reference 3D model. For each pixel, voxel, or vertex of the two image recordings, a precise depth value can thus be determined, which can be precisely determined from the reference point, the angle of view of the cameras, and the position of the cameras. In the case of occlusions, the value of the reference model is taken and interpolated.

[0041] Thus, using the pattern matching method, each point of the mesh can be precisely determined in relation to the reference point 14 using the two 2D cameras.

[0042] In addition, a computer-generated mask is created with the highest possible accuracy, for example of a deceased person. This mask is then aligned with the generated mask of the actor using computer graphics tools, for example, in order to overlay the mask of the deceased person, i.e. the virtual surface, onto the actor's face. The meshes of the two masks are determined using rigging, for example, whereby the difference in distances between the individual points is also taken into account and added together. The result is that the computer-animated mask correlates exactly with the mask of the living actor's face. This can be done with pinpoint accuracy, ideally at the pixel level, with the fixed distances in the Z direction (depth) being corrected based on the determined depth information. Thus, both masks are superimposed and correlated.

[0043] During filming, the pixel arrangement changes with each frame and each movement, allowing head movements to be corrected using the reference point of object 4. This changes the mask with each frame, causing the correlated mask of the person or the virtual surface to automatically follow. This allows animation to be achieved in the output image 18.

[0044] A computer-generated texture can be applied to the correlated, moving mask, depicting the person or the virtual surface in as high a resolution as possible. This creates a new image that can be 100% lifelike, animating and representing the deceased person or the virtual surface.

[0045] Additionally, it is possible to work with a few (external) markers 38 to make corrections in poor lighting conditions. This is done by using additional (external) reference points.

[0046] Embodiments according to the invention demonstrate that, using the described 3D engine, 3D models can be combined with video recordings in real time, so that the animated head of the human or the virtual surface can be combined with the video recordings of the body of the living actor. This can be realized in a 2D or 3D output image 18. This resulting image or video can show the real body of the living actor with the face of the human or the virtual surface.

[0047] This process of overlaying and blending 3D and video can also be applied to individual areas of the mask. For example, the eyes of the live actor can be blended into the human face or the virtual surface. The same applies to the mouth, teeth, and tongue. These overlays can also be generated in real time directly from the video images of the two cameras. Eye areas can be cut out by the mask position and copied to another location on the mask. The cutout can be created as a 3D mesh or as a video texture. This has the advantage of being able to see the real eye area. The mouth can also be cut out of the mask in the same way. The real mouth from the live image is blended into the mask. The mouth can be distorted in perspective to fit the mask of the deceased actor exactly.Embodiments further show that the output images 18 can be generated or output stereoscopically or in 3D, as already described. Furthermore, it is also possible, if the object is recorded, for example, from three different camera perspectives or camera settings, to generate a 360° view or a 360° model. A combination of the aforementioned output signal types is also possible. The 3D engine according to the invention then renders the 3D and video scenes and the 360° together as a single scene with millimeter precision in real time. However, this method is not limited to the overlay of the face, but can also include the entire body (the entire human being), which can then be generated completely as a 3D model in real time. The overlay can give the 3D model a different appearance, for example, with more muscles or less belly.This can be applied in film recordings or for computer games or interactive scenes, whether with a camera perspective (stereo), in 3D, or in a 360° model. Furthermore, it should be noted that the device according to the invention is not limited to the superimposition of people's bodies or faces, but can be applied to any type of object.

[0048] According to a further embodiment, an entire stage, for example a Beatles concert with stationary furniture, can be digitized and then displayed in real time as a 3D model. Moving actors, e.g. the musicians, and additional moving objects, such as guitars or beer bottles, can then be viewed separately. For example, a viewer can use VR (Virtual Reality) glasses to display an output image sequence 18 in which, on the one hand, the scenery with the stationary furniture and, on the other hand, the actors or musicians are displayed. The actors can be actors whose bodies are superimposed in real time with the bodies of the band members to be represented, e.g., the Beatles, so that the viewer has the impression of attending a real Beatles concert.The result is a live VR concert featuring real and superimposed artists (the Beatles 40 years ago), guitars, and other moving objects, which can be viewed as a complete real-time VR model in 360° panoramic view. This allows viewers to, for example, walk around the stage wearing VR headsets and participate in the virtual concert.

[0049] Furthermore, in a further embodiment, holograms or holographic projections of deceased persons or other objects or people can be created. The holograms can be created by real actors at any location. Before the hologram is created, the virtual surface is superimposed on the actors so that the hologram shows the actor with the superimposed surface. The hologram can then be projected onto the stage. There, the holograms can interact and communicate, for example, with real actors present on stage. The holograms follow the movements and facial expressions of the actors. The projection of the holograms is therefore advantageously carried out in real time, i.e. the viewer does not see any delay between the actions of the real actors and the hologram when the actors react to the actions of the real actors.The stage presentation can be displayed to the performers, for example, on a monitor. This allows the performers to be located at a remote location where the hologram is also generated. The hologram can then be sent, for example, via the internet, to a holographic projector on or next to the stage, which projects the hologram received, for example, via the internet, holographically onto the stage. Furthermore, in this embodiment, the audience does not need VR glasses or other aids to view the stage presentation with the holograms.

[0050] Fig. 5shows a schematic flow diagram of a method for superimposing at least part of an object with a virtual surface. The method 400 includes a step S405 with contactless determination of an absolute position of a reference point on the object in an environment, a step S410 with determining, based on the absolute position of the reference point, depth information from the plurality of voxels or vertices in a 3D image of the object, a step S415 with generating a 3D grid structure of the object, and a step S420 with superimposing the 3D grid structure with the virtual surface.

[0051] Although some aspects have been described in the context of a device, it should be understood that these aspects also represent a description of the corresponding method, so that a block or component of a device can also be understood as a corresponding method step or as a feature of a method step. Analogously, aspects described in the context of or as a method step also represent a description of a corresponding block, detail, or feature of a corresponding device. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, some or more of the key method steps may be performed by such an apparatus.

[0052] Depending on specific implementation requirements, embodiments of the invention may be implemented in hardware or software. The implementation may be performed using a digital storage medium, such as a floppy disk, a DVD, a Blu-ray disc, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH memory, a hard disk, or other magnetic or optical storage device storing electronically readable control signals that can interact or interact with a programmable computer system to perform the respective method. Therefore, the digital storage medium may be computer-readable.

[0053] Some embodiments according to the invention thus comprise a data carrier having electronically readable control signals capable of interacting with a programmable computer system such that one of the methods described herein is carried out.

[0054] In general, embodiments of the present invention may be implemented as a computer program product having a program code, wherein the program code is effective to perform one of the methods when the computer program product is run on a computer.

[0055] The program code can, for example, also be stored on a machine-readable medium.

[0056] Other embodiments include the computer program for performing one of the methods described herein, wherein the computer program is stored on a machine-readable medium. In other words, one embodiment of the method according to the invention is thus a computer program that has program code for performing one of the methods described herein when the computer program is executed on a computer.

[0057] A further embodiment of the method according to the invention is thus a data carrier (or a digital storage medium or a computer-readable medium) on which the computer program for carrying out one of the methods described herein is recorded.

[0058] A further embodiment of the method according to the invention is thus a data stream or a sequence of signals that represents the computer program for carrying out one of the methods described herein. The data stream or the sequence of signals can be configured, for example, to be transferred via a data communication connection, for example, via the Internet.

[0059] A further embodiment comprises a processing device, for example a computer or a programmable logic device, which is configured or adapted to carry out one of the methods described herein.

[0060] A further embodiment comprises a computer on which the computer program for performing one of the methods described herein is installed.

[0061] A further embodiment according to the invention comprises a device or system designed to transmit a computer program for performing at least one of the methods described herein to a recipient. The transmission can be electronic or optical, for example. The recipient can be, for example, a computer, a mobile device, a storage device, or a similar device. The device or system can, for example, comprise a file server for transmitting the computer program to the recipient.

[0062] In some embodiments, a programmable logic device (e.g., a field-programmable gate array, an FPGA) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field-programmable gate array may interact with a microprocessor to perform any of the methods described herein. In general, in some embodiments, the methods are performed by any hardware device. This may be general-purpose hardware such as a computer processor (CPU) or method-specific hardware such as an ASIC.

[0063] The above-described embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. Therefore, it is intended that the invention be limited only by the scope of the following claims and not by the specific details presented in the description and explanation of the embodiments herein.

Claims

1. Apparatus (2) for superimposing at least part of a 3D image (16) of a person with a virtual surface (6), the apparatus (2) comprising: one or several cameras (26) for providing the 3D image (16) of the person (4), wherein a relative coordinate system (52) is set on the basis of a position of the one or several cameras (26), and a computing unit (10), which is configured to determine depth information from a plurality of voxels or vertices (20) in the 3D image (16) of the person (4) and to generate a 3D grid structure (22) of the person (4) and to superimpose the virtual surface (6) of the 3D grid structure (22), characterized by a position determiner (8), which is configured to determine an absolute position (12) of a reference point (14) of the person (4) in an environment, three signal sources (32a, 32b, 32c) for attachment to the person, wherein each of the three signal sources (32) is configured to emit a signal that differs from the signal of the respective other signal sources, three receivers (34a, 34b, 34c) for positioning at differing angles around the person (4), wherein the receivers are connected to the position determiner (8) and receive the signals (36a, 36b, 36c), wherein the position determiner (8) is configured to determine runtime information of the received signals (36a, 36b, 36c) from the signal sources (32a, 32b, 32c) to the receivers (34a, 34b, 34c) and to compute the absolute position (12) and an orientation of the reference point (14) of the person (4) in the environment on the basis of the runtime information, wherein the computing unit (10) is configured to - determine the depth information of the plurality of voxels or vertices (20) in the 3D image (16) of the person (4) on the basis of the absolute position (12) of the reference point (14), and - compare positions of the reference point in the absolute coordinate system (50) and relative coordinate system (52) in order to compute a position of each voxel or vertex (20) in the 3D image, so that a position in the 3D image at which the virtual surface is oriented and the depth information of each voxel or vertex in the 3D image are known.

2. Apparatus (2) according to claim 1, wherein the signal sources (32a, 32b, 32c) comprise reflectors for laser beams, and the position determiner (8) is configured to determine the absolute position on the basis of a runtime of the reflected laser beams.

3. Apparatus (2) according to any one of the preceding claims, wherein the one or several cameras (26) include: two similar cameras (26a, 26b), which are configured to each capture an image (28a, 28b) of the person (4) from two viewing angles on the person (4) that differ from one another, wherein the computing unit (10) is configured to generate the 3D image (16) from the images (28) of the two cameras (26), or a stereoscopic camera.

4. Apparatus (2) according to any one of the preceding claims, wherein the computing unit (10) is configured to compute a 3D grid structure (24) of the virtual surface (6), to adapt the 3D grid structure (22) of the person (4) to the 3D grid structure (24) of the virtual surface (6) in order to obtain an adapted 3D grid structure and to superimpose a texture of the virtual surface (6) on the adapted 3D grid structure.

5. Apparatus (2) according to claim 4, wherein the computing unit (10) is configured to determine a difference grid structure from the 3D grid structure (22) of the person (4) and the 3D grid structure (24) of the virtual surface (6) and to apply the difference grid structure to the 3D grid structure (22) of the person (4) in order to adapt the 3D grid structure (22) of the person (4) to the 3D grid structure of the virtual surface (6) and to obtain the adapted 3D grid structure.

6. Apparatus (2) according to claim 5, wherein the computing unit (10) is configured to adapt the difference grid structure in a sequence of 3D images (16) in order to transfer a movement of the person (4) to the virtual surface (6), wherein the computing unit (10) computes an adaptation value based on a relative deviation of the 3D grid structure (22) of the person (4) in a current 3D image (16) from the 3D grid structure (22) of the person with which the difference grid structure is generated.

7. Apparatus (2) according to any one of the preceding claims, wherein each voxel or vertex (20) of the person (4) forms a grid element of the 3D grid structure (22) of the person.

8. Apparatus (2) according to any one of the preceding claims, wherein the computing unit (10) is configured to form a combined surface, wherein a part of the person (4), which is excluded from the superimposition with the virtual surface (6), comprises a real surface region of the person (4).

9. Apparatus (2) according to any one of the preceding claims, wherein the position determiner (8) determines the absolute position (12) of the reference point (14) by means of sonar, by means of radar, by means of ultrasound, by means of laser scanning and / or by means of RFID.

10. Apparatus (2) according to any one of the preceding claims, comprising a simulation unit (30), which is configured to generate a 360° 3D model of the virtual surface (6), wherein the computing unit (10) is configured to superimpose a section of the virtual surface (6) on the person (4) depending on a viewing angle on the person (4).

11. Method (400) for superimposing at least part of a 3D image (16) of a person with a virtual surface, comprising: providing the 3D image (16) of the person (4) by one or several cameras (26), wherein a relative coordinate system (52) is set on the basis of a position of the one or several cameras (26), determining depth information from a plurality of voxels or vertices (20) in the 3D image (16) of the person (4), generating (S415) a 3D grid structure of the person; and superimposing (S420) the 3D grid structure with the virtual surface, characterized by attaching three signal sources (32a, 32b, 32c) to the person, wherein each of the three signal sources (32) is configured to emit a signal that differs from the signal of the respective other signal sources, positioning three receivers (34a, 34b, 34c) at differing angles around the person (4), wherein the receivers are connected to a position determiner (8) and receive the signals (36a, 36b, 36c), and the position determiner is configured to determine an absolute position (12) of a reference point (14) of the person (4) in an environment, determining runtime information of the received signals (36a, 36b, 36c) from the signal sources (32a, 32b, 32c) to the receivers (34a, 34b, 34c), computing an absolute position (12) and an orientation of the reference point (14) of the person (4) in an environment on the basis of the runtime information, determining the depth information of the plurality of voxels or vertices (20) in the 3D image (16) of the person (4) on the basis of the absolute position (12) of the reference point (14), comparing the positions of the reference point in the absolute coordinate system (50) and relative coordinate system (52) in order to compute a position of each voxel or vertex (20) in the 3D image, so that a position in the 3D image at which the virtual surface is oriented and the depth information of each voxel or vertex in the 3D image are known.