Method and apparatus for immersive video conferencing

Abstract

Methods and apparatus are provided for encoding / decoding semantic description data representing a 3D geometric and photometric face model for immersive remote presentation. In an embodiment, video data including a user's face is encoded by extracting semantic description data representing a 3D geometric and luminosity model of the user's face. In another embodiment, an immersive video is decoded from semantic description data by: determining a head pose of a face of a remote user in the immersive video; determining a parametric model of a lighting environment of the immersive video; synthesizing the face and head postures of the remote user and the parameter model; and generating a modified immersive video comprising an image of the synthetic face of the user in the immersive video. In an embodiment, the generation of the immersive video is caused to be looped by taking the composite face and the immersive video at the previous frame at an input.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of European patent application No. 22306339.7 filed on September 12, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present embodiment generally relates to a method and apparatus for encoding / decoding semantic description data representing a 3D facial model for immersive telepresence. The present embodiment also generally relates to a method and apparatus for encoding or decoding based on a neural network. Background Art

[0004] Telepresence refers to the use of virtual reality technology, such as to enable apparent participation in a distant event. A popular application is found in telepresence video conferencing systems, which immerse participants in a single common environment. Specifically, in a typical use case, such a system is intended to ensure that a user sitting at a table in a conference room gets the impression that the other participant is sitting at the same table in the same conference room and looking directly at him / her when he / she speaks.

[0005] Immersive telepresence systems require some computer vision processing of captures of distant participants to achieve their goals. Typically, these captures are 2D videos obtained by commercial cameras. First, the head pose, i.e., the position and orientation of the distant participant's head in the received image, needs to be changed at the receiver end to establish eye contact with the user in his / her viewing device. Similar operations should be performed on the position of the distant participant's irises to adjust the gaze direction. Second, the facial lighting in the distant participant's image needs to be replaced with the lighting of the virtual immersive environment hosting the video conference.

[0006] Therefore, what is desired is an efficient immersive telepresence system that solves two well-known problems of telepresence, (a) achieving correct pose and eye position of the rendered face to support correct eye contact, and (b) illuminating the rendered face using a lighting model appropriate for the immersive environment. Summary of the invention

[0007] According to various embodiments, methods and apparatus are provided for encoding / decoding semantic description data representing 3D geometric and photometric facial models for immersive telepresence.

[0008] According to one embodiment, a method is provided, wherein the method includes receiving semantic description data of a 3D model representing a user's face; determining a head posture of the user's face in an immersive video; determining a parametric model of the lighting of an environment of the immersive video; synthesizing the user's face with the head posture under the parametric model of the lighting of the environment of the immersive video; and generating a modified immersive video, the modified immersive video including an image of the synthesized face of the user in the immersive video.

[0009] According to another embodiment, a method is provided, wherein the method includes receiving video data including a user's face; determining semantic description data representing a 3D model of the user's face in the video data by applying an encoder to the video data; and providing the semantic description data for rendering the user's face in an immersive video.

[0010] One or more embodiments further provide a device comprising one or more processors, wherein the one or more processors are configured to execute any one embodiment of the above method.

[0011] One or more embodiments further provide a computer program including instructions, which, when executed by one or more processors, cause the one or more processors to perform any one of the methods according to any of the above embodiments. One or more of the present embodiments further provide a computer-readable storage medium having stored thereon instructions for editing video footage, encoding at least one image or video, or decoding at least one image or video according to any of the above embodiments.

[0012] One or more embodiments further provide a bit stream comprising image or video data encoded according to any one embodiment of the above encoding method. One or more of the present embodiments further provide a computer readable storage medium having the above bit stream stored thereon.

[0013] One or more embodiments also provide a method for transmitting a bitstream, the bitstream comprising image or video data encoded according to any one embodiment of the encoding method described herein. One or more embodiments also provide an apparatus for transmitting a bitstream, the bitstream comprising image or video data encoded according to any one embodiment of the encoding method described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A block diagram of a system according to an embodiment is illustrated within which aspects of the present embodiment may be implemented.

[0015] Figure 2 A block diagram of a system according to another embodiment is illustrated within which aspects of the present embodiment may be implemented.

[0016] Figure 3A remote presentation system according to an embodiment is schematically illustrated, and various aspects of the embodiment can be implemented in the remote presentation system.

[0017] Figure 4 A method for processing a 2D facial image for facial reconstruction according to an embodiment is schematically illustrated.

[0018] Figure 5 A telepresence system with its encoder and its decoder implementing the method according to the invention is schematically illustrated.

[0019] Figure 6 Components of a 3D environment model according to an embodiment are schematically illustrated.

[0020] Figure 7 A method for training a NN autoencoder to generate semantic data according to another embodiment is illustrated.

[0021] Figure 8 A method for training a generative adversarial network (GAN) for generating photo-realistic immersive video according to another embodiment is illustrated.

[0022] Fig. 9 A method for generating photo-realistic immersive video refined using a generative adversarial network (GAN) according to another embodiment is illustrated.

[0023] Fig.10 A method for decoding at least one image according to another embodiment is illustrated.

[0024] Fig.11 A method for encoding at least one image according to another embodiment is illustrated.

[0025] Fig.12 A method for decoding at least one image according to another embodiment is illustrated.

[0026] Fig.13 Two remote devices communicating over a communication network are shown according to an example of the present principles.

[0027] Fig.14 The syntax of a signal according to an example of the present principles is shown. DETAILED DESCRIPTION

[0028] The present principles will now be described in the specific context of immersive video conferencing. However, the present principles are not limited to video conferencing, but can be directly and explicitly exported to any telepresence system in which the representation of a user is driven by a distant capture of his or her face by a camera. Such systems include, but are not limited to, gaming frameworks in which users are represented by avatars whose movements and expressions are driven by distant live video capture, or more generally, frameworks that fall within the scope of the metaverse in which participants interact in a virtual environment through their embodiments as avatars, and the avatar appearance, movement, and expression are driven by distant live video capture of the participants' faces. Such frameworks can host business applications such as e-learning, e-tourism, and e-commerce, to name a few.

[0029] Figure 3 A remote presentation system according to an embodiment is schematically illustrated, and various aspects of the embodiment can be implemented in the remote presentation system. Figure 3 The telepresence system of comprises three communication devices D1, D2, D3 connected via a communication network. The communication device D1 comprises a camera for capturing a scene as a series of images forming video data. The captured scene here consists of at least the face of a first user P1, e.g. in front of a table T. The communication device D1 further comprises a display for rendering the video, wherein remote users P2 and P3 are displayed in an immersive environment, e.g. in front of a representation of the same table T and apparently looking directly at user P1. As in Figure 3 As represented on the right of , user P1 is also represented in the immersive environment by a display of a communication device D2 or D3, for example in front of a representation of the same table T, and also obviously looking at user P2 or P3.

[0030] In order to realize the rendering of a common immersive environment in a telepresence system, the communication device D1 includes a transmitter / encoder for processing the captured video data as described below and providing description data for synthesizing the face of the user P1 in the telecommunication identification D2 or D3. The communication device D1 also includes a receiver / decoder for receiving and processing the description data provided by the telecommunication devices D2, D3 and rendering the faces of the users P2 and P3 in the immersive video displayed by the communication device D1.

[0031] Similarly, the communication device D2 is able to reproduce a video with an immersive effect due to the fact that a user P2 viewing the immersive video reproduced on the device D2 will get the impression that the user P1 is sitting next to the same table T and looking at him / her while she / he speaks. This is possible due to the receiver / decoder of the device D2 implementing a method according to the present principles of at least one embodiment by processing the following processing steps:

[0032] receiving semantic description data representing geometric and photometric components of a 3D model of a face of a first user P1;

[0033] Determining a head pose of a face of a first user P1 in the immersive video;

[0034] A parametric model for determining the lighting of an immersive video environment;

[0035] · Synthesize the face of the first user P1 with the head pose and lighting model; and

[0036] Generating a modified immersive video comprising an image of a synthesized face of the first user P1 in the immersive video.

[0037] In a refined embodiment of the decoding step, the generating further takes as input the synthesized face of the first user and the modified immersive video at a previous frame to enhance the displayed immersive video.

[0038] Therefore, according to at least one embodiment, instead of transmitting / encoding 2D video data representing the face of the first user, the transmitter / encoder of device D1 processes the 2D video data to obtain semantic description data of a 3D model - such as a 3D geometric and photometric model - representing the face of the first user in the first video data by applying an encoder to the video data; and provides the semantic description data for rendering the face of the first user in the immersive video. Advantageously, at least one embodiment allows a significant reduction in the amount of data transmitted over the communication network. In addition to the low bit rate, those skilled in the art will appreciate that the semantic data is independent of the resolution of the displayed video data, and therefore compression efficiency is particularly important for high resolutions of the displayed video.

[0039] In order to obtain semantic description data, it is proposed to modify some of the processing used, for example, in the face reproduction scheme. By using semantic description data of the user's face, the present principles specifically solve two well-known problems of remote presentation, (a) achieving correct pose and eye position of the rendered face to support correct eye contact, and (b) illuminating the rendered face with a lighting model suitable for the immersive environment. The disclosed modifications propose instantiations of the face reproduction scheme in at least two remote devices. However, the present principles are not limited to the specific face reproduction embodiments described below, and any method that provides semantic description data that allows reconstruction of the user's face in an immersive environment is compatible with the present principles. Advantageously, such a method can be performed on a user smartphone, a user laptop, or deployed on the cloud of a social network.

[0040] Figure 4A method for processing a 2D facial image for facial reproduction according to an embodiment is schematically illustrated. The goal of facial reproduction is to transfer the expression and head posture of a driving character to a target face while preserving the target identity. To this end, semantic data related to the face of the target character and the face of the driving character are determined. An example of a facial reproduction scheme is disclosed in the paper “Deep Video Portraits” by H. Kim et al., published in ACM Transactions on Graphics (Vol. 37, No. 4, pp. 163:1–163:14, 2018). As shown in Figure 4 As shown in , the rendering pipeline 400 is fed with two facial images, one for the target character 410 whose face is to be rendered, and the other for the driving character 420, which determines the facial pose and expression in the output image. In the Deep Video Portraits paper, in order to improve the temporal consistency of the output, the input is two temporal blocks of facial images instead of two images, but this does not change the principle of the method described below. The front-end module 430 extracts a semantic parameter 3D facial model 440 from each of these images 410, 420. This model only addresses the internal parts of the face, including the eyes, nose and mouth, but does not process the hair. The semantic parameter 3D facial model consists of the following components:

[0041] Identity: A 3D mesh representing the 3D geometry of a face with a neutral expression, i.e. the facial appearance of the character;

[0042] Expression: displacement of vertices of the neutral expression mesh caused by facial expressions, typically as a result of showing emotions and / or speaking words;

[0043] Pose (or head pose, or rigid head pose): the 3D rotation and translation of the face in the image relative to the frontal viewpoint;

[0044] Appearance: skin reflectivity of the 3D geometry mesh surface; and

[0045] Illumination volume: a parametric model of the lighting environment of the face.

[0046] 3D model extraction 430 can be performed in several ways. In the Deep Video Portraits paper cited above, it is obtained by a comprehensive analysis method that minimizes the difference between the facial interior area of ​​the actual image and the facial interior area reconstructed from the calculated model parameters through an optimization scheme on the model parameters. However, the present principles are not limited to optimization-based calculation of semantic facial parameters from images as proposed by Kim et al. For example, according to a variant embodiment, the following method can be used for Figure 7 A NN autoencoder is proposed to generate semantic description data, where the decoder module of the NN autoencoder provides a face synthesis module in a remote device.

[0047] The goal of the reproduction scheme is to generate an output facial image that has the identity and environment of the input target character, but with the pose and expression of the driving face. Therefore, the target character can be regarded as a puppet whose expression and head pose are controlled by the driving character face. To this end, as in Figure 4 As shown above, first 3D facial model parameters 440 are extracted for both the target face and the driving face. Then, a hybrid facial model 450 consisting of the identity, appearance and lighting components of the target face and the pose and expression of the driving face is assembled. Finally, a rendering module 460 synthesizes a facial image from the hybrid model 450. The rendering operation is equivalent to reconstructing the facial internal image 470 using the image formation process implied by the semantic model parameters 450. Module 460 outputs a synthetic facial internal image 470 determined according to the model parameter value set. Finally, a generator module 480 converts the synthetic facial internal image 470 into a full frame of a photo-realistic image 490, in which the target character now mimics the head movement, facial expression and eye gaze of the driving character. The generator module 480 is user-specific. It is a neural network that is trained on a video of a target character with a given background, a given hairstyle and a given outfit. The photo-realistic image 490 it outputs combines the background, hairstyle and clothing represented in the training video with the rendering of the target character's face driven by the hybrid facial model input 450. It is trained to render the target person's face in each image at the same location as in its input facial interior image 470, and with the same orientation and scale.

[0048] Figure 5 A telepresence system with its encoder and its decoder implementing the method according to the invention is schematically illustrated.For the sake of clarity, but without loss of generality, the telepresence scheme will focus on a scenario with only two participants - a sender and a receiver. Figure 5 Shows Figure 4 A novel arrangement of modules of a reproduction scheme, wherein the modules are implemented in remote devices of a telepresence system, namely a sender's communication device and a receiver's communication device.

[0049] exist Figure 5 In the arrangement of , the encoder - also called the transmitter - comprises an encoding module 510, which performs tasks corresponding to the 3D model extraction module. From the captured 2D video of the sender's face, it extracts semantic description data representing a parametric 3D face model of the sender's face. The model only resolves the inner parts of the face, including the eyes, nose and mouth, but not the hair. The sender semantic description data includes at least:

[0050] An identity indicator representing the sender's appearance with a neutral expression, such as a 3D mesh representing the 3D geometry of the sender's face with a neutral expression;

[0051] Expression indicators representing an emotional expression relative to a neutral expression of the sender, typically as a result of showing an emotion and / or speaking an utterance; e.g., multiple displacements of 3D mesh vertices caused by a facial expression; and

[0052] Appearance indicators representing the reflectivity of the user, such as skin reflectivity on a 3D mesh surface.

[0053] The sender semantic description data is then packaged in a bitstream and provided to a remote receiver's device over a communication network for immersive rendering. The receiver's communication device receives the sender semantic description data representing a parametric 3D facial model of the sender's face and computes an expected head pose of the sender's face in an immersive environment and a parametric model of the lighting environment of the sender's face in the immersive environment.

[0054] Figure 6 An example of components of a 3D environment model according to an embodiment is schematically illustrated. An immersive environment representing a participant in a video conference is obtained from a predetermined 3D model of a scene. For example, the scene may represent a room with a floor, walls, and windows, further including a table 610 and chairs 620 around the table 610. In this example, in the case of multiple participants in a video conferencing system, each user will be assigned to a predetermined chair 620, on which she / he will be represented as sitting in an immersive video displayed on a receiver device of the other participants. At each receiver device, an image of the virtual environment is calculated by rendering a projection of the aforementioned 3D environment model on an image plane of a virtual camera 630, the position, posture, and optical parameters (particularly including its focal length) of the virtual camera 630 being predetermined.

[0055] Back to Figure 5 At each receiving device, a pose estimation module 520 computes the head pose of the 3D model of the sender's face, which drives the computation of an image of the sender's face so that the image can be directly overlaid on a rendering of the environment. Figure 6 In the aforementioned example scenario of , the head pose 650 is calculated so that the image generated by the overlay represents the sender sitting in the reserved seat assigned to him or her, with his or her face looking at the virtual camera. Figure 6 As shown above, the 3D head pose model consists of scale, translation and rotation components defined in the 3D coordinate system 640 of the predetermined 3D scene model. The calculations performed by the pose estimation module 520 are equivalent to positioning, aligning and scaling the sender 3D face model inside the virtual environment. In more detail, these calculations involve:

[0056] First, adjust the scale component to match the physical size of the scene, for example, adjust the width of the sender's 3D face model so that it matches the size of the chair in the 3D scene model;

[0057] Second, set the translation vector to the displacement vector from the optical center of the virtual camera to the expected location of the center of the sender’s face in the 3D scene;

[0058] Third, adjust the rotation angle of the head pose so that the sender’s gaze points to the optical center of the virtual camera.

[0059] These processing steps achieve eye contact between the receiver and the sender's representation on the receiver's display. The lighting calculation module 530 provides a lighting model of the virtual immersive environment to the receiver's device at each frame of the video. The model is typically calculated by a 3D authoring tool based on light sources positioned by an artist designing the 3D scene, for example as a set of spherical harmonic coefficients on a spherical mapping of the 3D scene. Then, a hybrid 3D model is generated using the identity, expression, and appearance components of the received sender's face and the calculated head posture and lighting body components of the sender's face in the immersive environment. The rendering module 540 synthesizes the sender's facial internal image in the immersive environment from the hybrid 3D model. Next, the generator module 550 converts the synthesized facial internal image into a full frame of a photo-realistic video of the sender. As will be further discussed below, according to one embodiment, the rendering module 540 and / or the generator at 550 are based on a neural network trained on a video of the sender's face against a uniform background. The frames output by the generator 550 replicate the background. As a final post-processing step 560, a foreground area representing the sender is extracted in each frame output by the generator 550 and overlaid on a rendering of the 3D environment, using a color keying technique known from the prior art. This rendering is obtained by projecting the aforementioned 3D environment model onto the image plane of the aforementioned virtual camera. Due to the adjustment of the head pose performed by the pose calculation module 530, the head pose used to calculate the internal facial image fed to the GAN matches the expected position, orientation and scale of the 3D model of the sender's head in the environment. As a result, the image of the sender's head rendered at the output of the generator 550 in the same position and with the same orientation and scale as the internal facial image at the generator input can be directly overlaid on the aforementioned rendering of the environment to produce an image in which the sender's face and orientation are consistent with the rest of the scene.

[0060] According to a variant embodiment, where the immersive environment is predefined and can be used in the training of the generator at the NN-based rendering module 540 and / or 550, the generator module 550 converts the synthesized facial internal image directly into a full frame of the photo-realistic video of the sender, into the predefined immersive environment. Those skilled in the art will appreciate that in this case, a single sender is rendered into the immersive video. Advantageously, in this variant, the extraction of the foreground sender's head from the background and the final synthesis step are avoided.

[0061] A variant embodiment of the encoding module 510 and the decoding module 540 representing semantic description data of a geometric and photometric 3D face model is now described.

[0062] According to a first variant, the encoding module 510 implements the 3D model extraction of the Deep Video Portraits paper. Therefore, this module performs a comprehensive analysis optimization of the model parameters, the goal of which is to minimize the difference between the facial internal area of ​​the 2D captured image and the reconstruction of the facial internal area calculated based on the model parameters. A semantic parameter 3D facial model is calculated to achieve this goal, and the semantic parameter 3D facial model includes parameters representing the sender's identity, expression, (rigid) head posture, appearance and illuminant. A comprehensive analysis method for reconstructing the sender's facial internal area from the assumed 3D facial model parameters is integrated in the encoder.

[0063] According to a second variant, the encoding module 510 is the neural network encoder part of a neural network NN autoencoder trained on a dataset of facial images.

[0064] Figure 7 The diagram illustrates a method for training a NN autoencoder for generating semantic data according to another embodiment. Such a method was originally proposed by A. Tewari et al. in the paper “MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction” published in the 2017 International Conference on Computer Vision. The network consists of an encoder module 710 and a subsequent decoder module 720. The encoder outputs a 3D semantic parameter model 730 of the input face being sought. The decoder is handcrafted and is used to reconstruct the image inside the face based on the model parameters. The network is trained end-to-end to minimize the reconstruction loss of the image inside the face. Here, the training dataset consists of a large collection of facial images with various appearances, head poses, expressions, and lighting conditions. Once the autoencoder is trained, the NN encoder part is used as the encoding module 510, as in Figure 5As shown above, the NN decoder part of the autoencoder is used as the decoding module 540. Therefore, before starting the immersive video conference, the decoding device (decoder) configures the decoding module 540 with the trained NN decoder.

[0065] Variant embodiments of the generator module 550 are described in the following sections.

[0066] According to a first variant, the generator module 550 implements a generative adversarial network (GAN) described in the Deep Video Portraits paper. Figure 8 A method for training a generative adversarial network (GAN) for generating photo-realistic immersive video according to another embodiment is illustrated. Figure 5 The purpose of the generator module 550 at the back end of the video conferencing system is to remove artifacts in the internal facial area to make it look like a believable face, and to form a complete image by hallucinating the hair area, background, and parts of the face that are not visible in the synthesized internal facial image input to the generator. These parts include the inside of the mouth, which becomes visible due to the driving facial expression. Figure 8 In the figure, GAN is represented by module 810. It consists of Figure 5 The discriminator module 820 and the generator module 830 of the module 550 of FIG. GAN are generally designed as conditional generative adversarial networks, where the generator network 830 and the discriminator network 820 are jointly trained and conditioned on some input driving the generation process, as originally proposed by M. Mirza and S. Osindero in the paper “Conditional Generative Adversarial Nets” at arXiv:1411.1784, available at https: / / arxiv.org / pdf / 1411.1784.pdf. GAN is a generative model that synthesizes new images consistent with the content of the dataset on which it is trained. Here, the training dataset consists of facial images of the sender given a uniform background. Furthermore, the conditional GAN ​​is trained to produce images consistent with the conditioned data provided at its input. Therefore, after convergence, the conditional GAN ​​generator should produce credible facial images of the sender in the same background and with the same appearance and expression as the synthesized facial internal images provided as the conditioned input.

[0067] Since it is conditioned by its internal facial image having the desired pose and expression of the driving face, the facial image reproduced at its output should also have the desired pose and expression.

[0068] However, the task of the generator module is a difficult one as it is expected to both hallucinate the missing output image parts and also make the synthesized facial interior images constructed from the simplified facial model photo-realistic.

[0069] According to a second variation, the generator module 550 is refined to improve the credibility of the images it produces by feeding it additional inputs that are closer to the outputs it needs to produce. Fig. 9 A method for generating photo-realistic immersive video refined using a generative adversarial network 910 (GAN) according to another embodiment is illustrated. Advantageously, the generator 930 is fed with data closer to the image it must synthesize at its output, thereby reducing the difficulty of the task it must perform and improving its performance. In practice, a temporal feedback loop is introduced in the generator, as in Fig. 9 As shown above. In frame n-1 (corresponding to time t of the processed video n-1 ) as the photo-realistic facial image 940 output by the generator and the one generated by the rendering module at time t n The calculated facial internal images 950 are stacked to form a n Input to generator 930. The difference between the photo-realistic images in two consecutive time steps is expected to be small, because for typical values ​​of the video sampling period, changes in the pose, lighting, and to a lesser extent the facial expressions of the participants of the video conference are expected to be small. Therefore, t n-1 The photo-realistic facial image at t provides the generator with information about what it should n A strong hint of what output to generate.

[0070] Fig.10 A method 1000 for decoding semantic description data representing a user's face and generating an immersive video including the user's face according to an embodiment is illustrated. In a first step 1010, semantic description data representing a 3D geometric and photometric model of a remote user's face is received. For example, the semantic description data forms part of a bitstream. In step 1020, as described above with reference to Figure 5 and Figure 6As described, a rigid head pose of the face of a remote user in an immersive video is calculated. In step 1030, a parametric model of the lighting of the virtual environment of the immersive video is obtained. The rigid head pose, the lighting of the virtual environment, together with the received parameters, allow the rendering of the user's face to be synthesized in the immersive video to be synthesized in step 1040. Then, in step 1050, a photo-realistic image of the facial rendering is generated for a uniform background. Finally, in step 1060, an area representing the photo-realistic face of the user is segmented from the uniform background and overlaid on a predetermined image rendering of the virtual environment to generate a frame of the modified immersive video. As mentioned above, according to a variant, in the case where the immersive environment is set to correspond to the background captured in the training video of the sender's face, steps 1050 and 1060 are jointly performed in the generation step. According to a variant, the generation in step 1050 is performed using a GAN. Although the present principles are not limited to the GAN used for the generation step, those skilled in the art will appreciate that GAN is currently the most efficient implementation for generating such a photo-realistic image from a synthetic view. Advantageously, the method 1000 thus reduces the amount of data to be transmitted in the video conferencing system by decoding semantic description data of the 3D face model instead of 2D video data. Furthermore, the method advantageously achieves the correct pose and eye position of the rendered face to support correct eye contact, and illuminates the rendered face with a lighting model suitable for the immersive environment. According to another variant, the generation loop is performed by taking at input the synthesized face of the user and the immersive video at the previous frame, such as Fig. 9 Advantageously, the generator output at the previous frame provides a strong hint about what should be generated at the current frame. Having this data available at the generator input helps it produce better quality video with fewer artifacts.

[0071] Fig.11 A method 1100 for encoding semantic description data representing a user's face according to an embodiment is illustrated. In a first step 1110, video data is received, the video data including the user's face. In an encoding step 1120, corresponding to Figure 4 The task of the 3D model extraction module of the encoding step 1120 is applied to the video data to obtain semantic description data representing a 3D geometric and photometric model of the user's face in the first video data. As mentioned above, this model only resolves the internal parts of the face, including the eyes, nose and mouth, but not the hair. According to a variant, the semantic description data at the output of the encoding step 1120 includes:

[0072] An identity indicator that represents the user's appearance with a neutral expression, such as a 3D mesh that represents the 3D geometry of the sender's face with a neutral expression;

[0073] an indication of an expression representing an emotional expression relative to a neutral expression of the user, such as a number of displacements of vertices of a 3D mesh caused by the facial expression; and

[0074] An indication of the appearance of the user's reflectivity, such as skin reflectivity on a 3D mesh surface;

[0075] An indication of a rigid head pose, which represents the 3D rotation and translation of the face in the input video image relative to a frontal parallel viewpoint;

[0076] and an indication of an illuminant, such as a parametric model of the lighting environment of a face.

[0077] In principle, these components are extracted because Figure 6 The variant autoencoder needs the full model to work, but none of them need to be transmitted because some of them are replaced by parts of the immersive video. Advantageously, even more bitrate is saved.

[0078] Thus, in step 1130, the extracted semantic description data is provided to a remote decoder for rendering the face in the input video into an immersive video. Advantageously, only a portion of the above mentioned components of the semantic description data are provided to the remote decoder, namely the identity, expression and appearance components. The semantic description data is completed on the decoding side by a rigid head pose of the user's face in the immersive video and an illuminant that is a parametric model of the illumination of the immersive video environment.

[0079] In the above-described method for encoding / decoding at least one image, results are provided for facial images, however, the present principles are not limited to this kind of images, and the method provided herein is applicable to any other kind of images as long as a model is available.

[0080] Figure 1 A block diagram of a system according to an embodiment is illustrated within which aspects of the present embodiment may be implemented. Figure 1 Schematically shows a communication device, for example according to an embodiment Figure 3 video conferencing equipment.

[0081] According to an embodiment, the above method is implemented as instructions that cause one or more processors to execute the method steps.

[0082] According to one embodiment, Figure 1A block diagram of an example of a system in which the above-mentioned various aspects and embodiments can be implemented is illustrated. System 100 can be embodied as a device including various components described below, and is configured to perform one or more aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances and servers. The elements of system 100 can be embodied in a single integrated circuit, multiple ICs and / or discrete components, either alone or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, system 100 is coupled to other systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port communication. In various embodiments, system 100 is configured to implement one or more aspects described in this application.

[0083] The system 100 includes at least one processor 110, which is configured to execute instructions loaded therein, for implementing various aspects described in the present application, for example. The processor 110 may include embedded memory, input and output interfaces, and various other circuit systems as known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). The system 100 includes a storage device 140, which may include a non-volatile memory and / or a volatile memory, including but not limited to an EEPROM, a ROM, a PROM, a RAM, a DRAM, an SRAM, a flash memory, a magnetic disk drive, and / or an optical disk drive. As a non-limiting example, the storage device 140 may include an internal storage device, an additional storage device, and / or a network accessible storage device.

[0084] According to an embodiment, the system 100 includes an encoder / decoder module 130, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents a (one or more) module that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. Additionally, the encoder / decoder module 130 may be implemented as a separate element of the system 100, or may be incorporated into the processor 110 as a combination of hardware and software as known to those skilled in the art.

[0085] Program code to be loaded onto the processor 110 to perform various aspects described in this application may be stored in the storage device 140 and subsequently loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of various items during the execution of the processes described in this application. Such stored items may include, but are not limited to, one of a plurality of input video shots, mosaic images, warps, 3D models, color transform information, visibility maps, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0086] In several embodiments, memory internal to the processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing required during the pre-processing steps and / or video editing of the methods described herein. However, in other embodiments, memory external to the processing device (e.g., the processing device can be the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory can be a memory 120 and / or a storage device 140, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations.

[0087] As indicated in block 105, input may be provided to the elements of system 100 through various input devices. Such input devices include, but are not limited to, (i) an RF section that receives an RF signal transmitted over the air, for example, by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0088] In various embodiments, the input device of frame 105 has associated corresponding input processing elements, as known in the art. For example, the RF part can be associated with elements suitable for the following: (i) selecting the desired frequency (also referred to as selecting a signal, or limiting the signal band to a band), (ii) down-converting the selected signal, (iii) again band-limiting the narrower frequency band to select the signal band that can be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired data packet stream. The RF part of various embodiments includes one or more elements that perform these functions, for example, a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF part can include a tuner that performs various functions in these functions, including, for example, down-converting a received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or a baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and filter to the frequency band of expectation again by filtering, down-conversion and perform frequency selection.Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements of performing similar or different functions.Adding element can include inserting element between existing element, for example, inserting amplifier and analog-to-digital converter.In various embodiments, the RF part comprises antenna.

[0089] Additionally, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 100 to other electronic devices across the USB and / or HDMI connections. It will be appreciated that various aspects of input processing, such as Reed-Solomon error correction, may be implemented as desired, for example, within a separate input processing IC or within the processor 110. Similarly, various aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within the processor 110, as desired. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 110 and the encoder / decoder 130, which operate in combination with memory and storage elements to process the data streams as desired for presentation on an output device.

[0090] The various elements of system 100 may be provided within an integrated housing within which the various elements may be interconnected and transmit data therebetween using suitable connections 115, such as an internal bus as known in the art, including an I2C bus, wiring, and printed circuit boards.

[0091] The system 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data through the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or a network card, and the communication channel 190 may be implemented within, for example, a wired and / or wireless medium.

[0092] In various embodiments, a Wi-Fi network such as IEEE 802.11 is used to stream data to the system 100. The Wi-Fi signals of these embodiments are received through a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet to allow streaming applications and other over-the-top communications. Other embodiments provide streaming data to the system 100 using a set-top box that delivers data through the HDMI connection of the input block 105. Still other embodiments provide streaming data to the system 100 using the RF connection of the input block 105.

[0093] The system 100 can provide output signals to various output devices, including a display 165, a speaker 175, and other peripherals 185. In various examples of embodiments, the other peripherals 185 include one or more of a stand-alone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functions based on the output of the system 100. In various embodiments, control signals are transmitted between the system 100 and the display 165, the speaker 175, or other peripherals 185 using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to the system 100 via dedicated connections through the corresponding interfaces 160, 170, and 180. Alternatively, the output devices can be connected to the system 100 using a communication channel 190 via the communication interface 150. The display 165 and the speaker 175 can be integrated into a single unit with other components of the system 100 in an electronic device (e.g., a television). In various embodiments, the display interface 160 includes a display driver, such as a timing controller (T Con) chip.

[0094] For example, if the RF portion of input 105 is part of a separate set-top box, the display 165 and speaker 175 may alternatively be separate from one or more other components. In various embodiments where the display 165 and speaker 175 are external components, the output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0095] Figure 2 A block diagram of a system according to another embodiment is illustrated within which aspects of the present embodiment may be implemented. Figure 2 Schematically shows a communication device, for example according to an embodiment Figure 3 communication equipment. Figure 2 An embodiment of an apparatus using the aforementioned method is shown. The apparatus includes a processor 210 and may be interconnected with a memory 220 through at least one port. Both the processor 210 and the memory 220 may also have one or more additional interconnections to external connections.

[0096] The processor 210 is further configured to receive an image or output an image generated using the aforementioned method and encode at least one image or decode at least one image.

[0097] According to Fig.13 In the example of the present principles illustrated in FIG. 1 , in the context of a transmission between two remote devices A and B on a communication network NET, device A comprises a processor associated with memories RAM and ROM, which are configured to implement as described with respect to Figure 5 , 7 or any one of the embodiments of the method for encoding at least one image as described in claim 11, and the device B comprises a processor associated with a memory RAM and a ROM, which is configured to implement the method as described in relation to Figure 5 , 7 , any embodiment of the method for decoding at least one image described in 8, 9, 10 or 12.

[0098] According to an example, the network is a broadcast network adapted to broadcast / transmit encoded images from device A to decoding devices including device B.

[0099] The signal intended to be transmitted by device A carries at least one bitstream comprising coded data representative of at least one image.

[0100] Fig.14 An example of the syntax of such a signal is shown when transmitting at least one coded picture by a packet-based transport protocol. Each transmitted packet P comprises a header H and a payload PAYLOAD.

[0101] Various methods are described herein, and each method includes one or more steps or actions for realizing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Additionally, terms such as "first", "second", etc. can be used to modify elements, parts, steps, operations, etc. in various embodiments, for example, "first decoding" and "second decoding". Unless specifically required, the use of such terms does not imply the sequencing of the operation to modification. Therefore, in this example, the first decoding does not need to be performed before the second decoding, and can occur in a time period overlapping with the second decoding, for example, before, during, or with the second decoding.

[0102] Unless otherwise indicated or technically excluded, the aspects described in this application can be used alone or in combination.

[0103] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes, and the described aspects are not limited to these specific values.

[0104] The implementations and aspects described herein can be implemented in, for example, methods or processes, devices, software programs, data streams, or signals. Even if only discussed in the context of a single implementation form (e.g., discussed only as a method), the implementation of the features discussed can also be implemented in other forms (e.g., devices or programs). Devices can be implemented in, for example, appropriate hardware, software, and firmware. The method can be implemented in, for example, a device, such as a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0105] References to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0106] Additionally, the application may refer to "determining" information of various segments. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0107] Additionally, the application may refer to "accessing" various segments of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0108] Additionally, the application may refer to "receiving" various pieces of information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from storage). Furthermore, during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is often involved in one way or another.

[0109] It will be appreciated that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," any of the following uses of " / ," "and / or," and "at least one of..." are intended to include selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to many of the listed items, as will be apparent to one of ordinary skill in this and related arts.

[0110] In addition, as used herein, the word "signal" refers to, among other things, indicating something to a corresponding decoder. In this way, in one embodiment, the same parameters are used on both the encoder side and the decoder side. Therefore, for example, the encoder can transmit (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and other parameters, signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select specific parameters. By avoiding the transmission of any actual function, bit saving is achieved in various embodiments. It will be appreciated that signaling can be implemented in a variety of ways. For example, in various embodiments, one or more grammatical elements, signs, etc. are used to signal information to the corresponding decoder. Although the verb form of the word "signal" is related to the above, the word "signal" can also be used as a noun in this article.

[0111] As will be clear to one of ordinary skill in the art, implementations may generate a variety of signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing the method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

Claims

1. A method comprising: Receiving semantic description data of a 3D model representing a user's face; Determining the head pose of a user’s face in an immersive video; A parametric model for determining lighting of an immersive video environment; Synthesizing the user's face with head pose and under a parametric model of the lighting of the environment of the immersive video; and A modified immersive video is generated, the modified immersive video including an image of a synthesized face of a user in the immersive video.

2. The method according to claim 1, wherein: The semantic description data of the 3D model representing the user's face includes: Use neutral expressions to indicate identity of the user's appearance; an indication of the expression of an emotional expression or facial deformation induced by the spoken utterance, relative to the user's neutral expression; and An appearance indicator of the reflectivity of the user's face.

3. The method according to claim 2, wherein: The identity indication is a 3D mesh representing the 3D geometry of a face with a neutral expression.

4. The method according to claim 3, wherein: The expression indication includes a plurality of displacements of vertices of the 3D mesh caused by the emotional expression.

5. The method according to claim 3, wherein: The appearance indication includes reflectivity on the surface of the 3D mesh.

6. The method according to claim 1, wherein: The head pose of a user's face in an immersive video includes an indication of a 3D rotation and translation of the face in an image of the immersive video relative to a fronto-parallel viewpoint.

7. The method according to claim 1, wherein: Generating the modified immersive video further includes: generating an image of a composite face of the user against a uniform background; and The generated image of the user's synthetic face extracted from a uniform background is composited into the immersive video.

8. The method according to claim 1, wherein: The generation takes at input a synthesized face of a user and a modified immersive video at a previous frame.

9. The method according to claim 1, wherein: Generation of modified immersive videos using generative adversarial networks.

10. A method comprising: receiving video data including a user's face; determining semantic description data representing a 3D model of a face of a user in the video data by applying an encoder to the video data; and Provides semantic description data for rendering the user's face in immersive video.

11. The method according to claim 10, wherein: The semantic description data of the 3D model representing the user's face includes at least: Use neutral expressions to indicate identity of the user's appearance; an indication of the expression of an emotional expression or facial deformation induced by the spoken utterance, relative to the user's neutral expression; and An appearance indicator of the reflectivity of the user's face.

12. The method according to claim 11, wherein: The identity indication is a 3D mesh representing the 3D geometry of a face with a neutral expression.

13. The method according to claim 12, wherein: The expression indication includes a plurality of displacements of vertices of the 3D mesh caused by the emotional expression.

14. The method according to claim 12, wherein: The appearance indication includes reflectivity on the surface of the 3D mesh.

15. An apparatus comprising one or more processors configured to: Receiving semantic description data of a 3D facial model representing a user's face; Determining the head pose of a user’s face in an immersive video; A parametric model for determining lighting of an immersive video environment; Synthesizing the user's face with head pose and under a lighting parameter model of the immersive video environment; and A modified immersive video is generated, the modified immersive video including an image of a synthesized face of a user in the immersive video.

16. The device according to claim 15, wherein: The semantic description data of the 3D model representing the user's face includes: Use neutral expressions to indicate identity of the user's appearance; an indication of the expression of an emotional expression or facial deformation induced by the spoken utterance, relative to the user's neutral expression; and An appearance indicator of the reflectivity of the user's face.

17. The device according to claim 16, wherein: The identity indication is a 3D mesh representing the 3D geometry of a face with a neutral expression.

18. The device according to claim 17, wherein: The expression indication includes a plurality of displacements of vertices of the 3D mesh caused by the emotional expression.

19. The device according to claim 17, wherein: The appearance indication includes reflectivity on the surface of the 3D mesh.

20. The device according to claim 15, wherein: The head pose of a user's face in an immersive video includes an indication of a 3D rotation and translation of the face in an image of the immersive video relative to a fronto-parallel viewpoint.

21. The device according to claim 15, wherein: To generate the modified immersive video, the one or more processors are further configured to: generating an image of a composite face of the user against a uniform background; and The generated image of the user's synthetic face extracted from a uniform background is composited into the immersive video.

22. The device according to any one of claims 15 and 21, wherein: To generate the modified immersive video, one or more processors take at input a synthesized face of a user and the modified immersive video at a previous frame.

23. The apparatus of any one of claims 15-22, further comprising a generative adversarial network for generating the modified immersive video.

24. An apparatus comprising one or more processors configured to: receiving video data including a user's face; determining semantic description data representing a 3D model of a face of a user in the video data by applying an encoder to the video data; and Provides semantic description data for rendering the user's face in immersive video.

25. The device according to claim 24, wherein: The semantic description data of the 3D model representing the user's face includes: Use neutral expressions to indicate identity of the user's appearance; an indication of the expression of an emotional expression or facial deformation induced by the spoken utterance, relative to the user's neutral expression; and An appearance indicator of the reflectivity of the user's face.

26. The device according to claim 25, wherein The identity indication is a 3D mesh representing the 3D geometry of a face with a neutral expression.

27. The device according to claim 26, wherein: The expression indication includes a plurality of displacements of vertices of the 3D mesh caused by the emotional expression.

28. The device according to claim 26, wherein The appearance indication includes reflectivity on the surface of the 3D mesh.

29. A device in a video conferencing system, comprising: The device according to any one of claims 24 to 28; and At least one device according to any one of claims 15 to 23.

30. A device in a video conferencing system, comprising: The device according to any one of claims 24 to 28; At least one device according to any one of claims 15 to 23; an antenna configured to receive a bitstream including semantic description data representing a 3D model of a remote user's face; and A display is configured to display a modified immersive video including an image of a synthesized face of a remote user in the immersive video.

31. A bitstream comprising semantic description data for rendering a user's face in an immersive video, wherein the semantic description data representing a 3D model of the user's face comprises: Use neutral expressions to indicate identity of the user's appearance; an indication of the emotional expression or facial deformation induced by the spoken utterance relative to the user's neutral expression; and An appearance indicator of the reflectivity of the user's face.

32. A computer readable medium comprising a bitstream according to claim 31.

33. A computer-readable storage medium having stored thereon instructions for causing one or more processors to execute the method according to any one of claims 1 to 14.

Images