Coding techniques and metadata for video communication using generated facial video.

Abstract

A method, system, and metadata for video communication using generated face video (GFV) are described. In an encoder, a GFV bitstream is generated containing multiplexed coded face video pictures and GFV metadata. Using supplemental enhancement information (SEI), a GFV SEI message includes syntax elements describing face features and at least one of the following: the presence of one or more faces, spatial sampling, temporal sampling, primary coded picture characteristics and driving picture processing, background processing, SEI persistence, and compression parameters for face features. In a decoder, the decoder combines information extracted from the GFV metadata with the decoded face video pictures to generate a reconstructed output video.

Description

[Technical Field]

[0001] [Cross-reference of related applications] This application claims priority rights to U.S. Provisional Patent Application No. 63 / 509,119 filed on 20 June 2023, U.S. Provisional Patent Application No. 63 / 511,827 filed on 3 July 2023, U.S. Provisional Patent Application No. 63 / 587,703 filed on 3 October 2023, and U.S. Provisional Patent Application No. 63 / 572,783 filed on 1 April 2023.

[0002] [Technical field] The present invention generally relates to video communications. More specifically, embodiments of the present invention relate to coding techniques and metadata for video communications using generative video, including generative face video (GFV). [Background technology]

[0003] In recent years, video conferencing has experienced tremendous growth due to the limitations of face-to-face meetings and the increasing demand for remote work. Recent developments in generative face video (GFV) enable more efficient digital communication between parties by replacing traditional pixel-based representations of images and videos with a feature set that can generate facial images at ultra-low bitrates when received by a suitable decoder.

[0004] To improve existing communication technologies using generated facial video, improved coding techniques and metadata have been developed, as recognized by the inventors of this specification.

[0005] The techniques described in this section are techniques that can be pursued, but are not necessarily techniques that have been conceived or pursued previously. Therefore, unless otherwise specified, none of the techniques described in this section should be assumed to qualify as prior art simply by being included in this section. Similarly, any problems identified with respect to one or more techniques should not be assumed to be recognized in any prior art based on this section, unless otherwise specified. [Brief explanation of the drawing]

[0006] Embodiments of the present invention are shown in the accompanying drawings as examples, not as limitations, and similar reference numerals refer to similar elements. [Figure 1] This document describes an exemplary process for generating generative feature video (GFV) using conventional techniques. [Figure 2] This describes an encoding and decoding process for video communication using GFV according to a first exemplary embodiment of the present invention. [Figure 3] A second exemplary embodiment of the present invention shows an encoding and decoding process for video communication using GFV. [Figure 4] This invention illustrates an exemplary process for feature compression in GFV communication according to an exemplary embodiment of the present invention. [Figure 5] This figure shows an exemplary process according to an exemplary embodiment of the present invention, which uses GFV-related messaging to facilitate machine analysis. [Figure 6] This describes an exemplary process using GFV-related messaging to facilitate video improvement or enhancement, according to an exemplary embodiment of the present invention. [Modes for carrying out the invention]

[0007] Methods and metadata for video communications using generated face video technology are described herein. In the following description, for illustrative purposes, numerous specific details are provided to provide a complete understanding of the invention. However, it is evident that the invention can be carried out without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail to avoid unnecessarily obscuring, obscuring, or ambiguizing the invention.

[0008] [overview] The exemplary embodiments described herein relate to methods and metadata for video communication using generated face video (GFV) technology. In one embodiment, a processor receives a sequence of pictures in a video sequence. For the current picture in the sequence of video pictures, the processor generates a GFV bitstream containing a multiplexed coded face video picture and GFV metadata. Using supplemental enhancement information (SEI), a GFV SEI message includes syntax elements describing face features and at least one of the following: the presence of one or more faces, spatial sampling, temporal sampling, primary coded picture characteristics and driving picture processing, background processing, SEI persistence, and compression parameters for face features. In a decoder, the decoder combines the information extracted from the GFV metadata with the decoded face video picture to generate a reconstructed output video.

[0009] [Review of Face Generation Video Methods] The exemplary embodiments described herein focus on a generative face video method based on a first-order motion model (FOMM) for image animation, as described in Reference [1] and its derivatives (References [2-5]). In all these methods, a key issue is how to estimate the temporal motion in video coding using facial features. As shown in Figure 1, the inputs to this process are key frames (reference frames) (102) and driving frames (104). The output is a generated driving frame (target frame) (127). The algorithm can be summarized into four modules, as shown in Figure 1. 1) Feature Extraction (105): The feature extraction module is applied to both keyframes and driving frames. Features can be facial landmarks, 2D or 3D keypoints, region matrices, compact features, facial semantics, etc., which will be discussed in more detail in later sections. 2) Motion estimation (110): The motion estimation module estimates (coarse) motion based on keyframe features and driving frame features and / or keyframes. 3) Input preparation for the generator (115): The estimated motion, combined with keyframes (102), is input to the generator (120) and can be used to predict dense motion fields, warp keyframes or other features, create occlusion masks, etc. 4) Generator (image generation) (120): The generator is used to output a generated driving frame (127), for example, using a Generative Adversarial Network (GAN). Essentially, facial features function as a compact description of facial movement along the time domain.

[0010] The process of FIG. 1 means that the driving frame (104) can be generated by the key frame and the characteristics of the key frame and the driving frame. Therefore, when using the generated face video framework for ultra-low bitrate face video communication, the following need to be transmitted in the bitstream. 1) Key frame 2) Characteristics for the driving frame The characteristics for the key frame can be derived at the decoder. The compactness of the extracted characteristics affects the bitrate for both the metadata and the video key bitstream. Potentially, the characteristics of the driving frame can be compressed to further improve the efficiency. It also affects the decoding complexity of the decoder reconstruction algorithm. Generally, the face characteristics and the reconstruction algorithm are closely coupled and affect the final R-D performance.

[0011] Before considering specific embodiments of the present invention, the following section provides a short overview of the most common generative face characteristics. Commonly used face reconstruction algorithms using these face characteristics are also discussed.

[0012] [Description of face characteristics] Landmarks are used to identify and represent the main parts of the human body. For example, for the face region, there are landmarks for the nose, eyebrows, mouth or the outer corner of the eye. Landmarks are usually the 2D or 3D coordinates of their positions in the image plane.

[0013] Facial landmarks are used to identify and represent key parts of a human face, such as the mouth, right eyebrow, left eyebrow, right eye, left eye, nose, and chin. Standard face datasets provide 68 x and y coordinate annotations indicating the most important points of a person's face (as supported, for example, by the Dlib library at dlib.net). Newer datasets and algorithms leverage high-density "face meshes" with over 468 3D facial landmarks (as supported, for example, by the MediaPipe library at https: / / developers.google.com / mediapipe).

[0014] In some use cases for video conferencing, hand gestures are also important for conveying information, and therefore various hand landmark areas are defined (e.g., wrist, index fingertip, etc.).

[0015] In some use cases for video conferencing, postural landmarks (such as the nose and left elbow) can be used to exchange body language. Without limitation, the exemplary embodiments presented here primarily focus on facial landmarks.

[0016] Facial keypoints are important areas of a person's face from which facial expressions, and therefore emotions, can be evaluated. The term "keypoint" is used to indicate the location of major object parts in an image. For example, it defines prominent spatial locations or points in an image, such as important parts of the face (tip of the nose, eyebrows, lips) or important points of the body (joints, hips, elbows). Furthermore, keypoint detection can be task-dependent and potentially learnable in a self-supervised manner. Self-supervised keypoints may also be present in parts of the face that are not necessarily important (Reference [1]). In recent developments, keypoint detection is often performed by neural networks. Keypoint representation acts as a bottleneck, resulting in a compact motion representation. The location of keypoints can be represented via 2D coordinates (Reference [1]) or 3D coordinates (Reference [3]).

[0017] An example of a 2D keypoint is defined on Kaggle as follows: -Each predicted keypoint is specified by a real-valued pair of (x,y) values ​​in the pixel index space. There are 15 keypoints, which represent the following elements of the face: left_eye_center, right_eye_center, left_eye_inner_corner, left_eye_outer_corner, right_eye_inner_corner, right_eye_outer_corner, left_eyebrow_inner_end, left_ey ebrow_outer_end, right_eyebrow_inner_end, right_eyebrow_outer_end, nose_tip, mouth_left_corner, mouth_right_corner, mouth_center_top_lip, mouth_center_bottom_lip

[0018] Using 3D keypoints (or point clouds) can potentially achieve better accuracy than their 2D counterparts when dealing with larger head poses such as rotation and translation. Sometimes, merging 2D and 3D keypoints is beneficial due to a trade-off between better accuracy and complexity (reference

[21] ). Landmarks can be considered a special type of keypoint with semantic significance.

[0019] Affine transformations in each local neighborhood are used to describe local motion. A simple 2x2 affine transformation (Jacobian matrix) can be used for representation (reference [1]). Affine transformations can be applied globally (i.e., one affine transformation per image) or associated with each keypoint (reference [1]).

[0020] In some cases, domain-based features may be used instead of keypoints. Multiple types of matrices can be used as region matrices, such as shift matrices (1×2), covariance matrices (2×2), and affine transformation matrices (2×2) (Reference [2]). For face generation, the number of regions can be predefined, for example, 10 (Reference [2]).

[0021] The learned features of neural networks (such as U-Net in reference [4]) are generally learned in an end-to-end manner and can represent temporal movement very sparsely. The features can be further compressed / compacted to a corresponding size (such as 4x4 in reference [4]) using a learning-based compression method (autoencoder) (reference

[22] ). The compacted features can then be packed into a matrix (such as 4x4 in reference [4]).

[0022] Intrinsic visual representations, or facial semantics, have the advantage of explicitly representing facial expressions. In reference [5], facial semantics are represented using 14 parameters, as shown in Table 1. [Table 1]

[0023] Reference [5] relies on a 3D Morphable Model (3DMM) model for 3D face reconstruction. Deep learning-based methods significantly improve 3D face mesh reconstruction performance and greatly advance the development of generated face videos by directly regressing 3DMM coefficients from the input, transforming the face template, and reconstructing the corresponding face mesh in a supervised or unsupervised manner.

[0024] The Facial Action Coding System (FACS), based on a system originally developed by Swedish anatomist Carl-Herman Hjortsjo, is a system that classifies human facial movements based on their appearance on the face. Using FACS, a human coder can manually code almost any anatomically possible facial expression and break it down into specific "action units" (AUs) and the time segments that produced that expression. Since AUs are independent of any interpretation, they can be used for any higher-order decision-making process, including the recognition of basic emotions or pre-programmed commands for the surrounding intelligent environment.

[0025] To clarify, while FACS is an indicator of facial expression, it does not actually provide biomechanical information about the degree of muscle activation. Muscle activation is not part of FACS, but the major muscles involved in facial expression are added here.

[0026] An action unit (AU) is the basic movement of an individual muscle or group of muscles. An action descriptor (AD) is a unit movement that may include the movement of several muscle groups (e.g., the forward propulsion of the jaw). The muscular basis for these movements is not specified, and specific behaviors are not distinguished as precisely as AUs.

[0027] FACS intensity is annotated by adding letters A to E (minimum to maximum intensity) to the operating unit number (for example, AU 1A is the weakest trace of AU 1, and AU 1E is the maximum intensity possible for an individual). A-Trace B-Slight C-Marked or pronounced D-Severe or extreme

[0028] FACS codes for expressing emotions have other modifiers, such as "R" for actions occurring on the right side of the face and "L" for actions occurring on the left side. Actions that are unilateral (occurring on only one side of the face) but do not have a specific side are indicated by "U," and actions that are bilateral but have a stronger side are indicated by the asymmetrical "A." List of AU and AD (~100 codes) Main code Head movement code • Eye movement codes • Visibility code • Overall behavior code

[0029] The segmentation map may be generated for distinct areas of the face. For example, a map labeled with 15 categories (eyes, hair, ears, etc.) may be created.

[0030] MRAA (reference [2]) proposes using meaningful object parts such as the torso, upper arm, forearm, etc. These may have a variable number of pixels. A potential option is to normalize the number of pixels for each region, thereby making it deterministic.

[0031] Reference [3] includes 12-dimensional head transformation parameters, namely a 3x3 rotation matrix and a 3x1 translation matrix. The head pose may be defined using three values ​​related to parameters that define the yaw, roll, and pitch of the 3D head pose.

[0032] In addition to image / video features, some technologies use audio signals to generate a head in speech. Note that some of the above features can, in some cases, be merged into a single category.

[0033] [Overview of the reconstruction algorithm] Almost all techniques use a generator network to reconstruct the final output image. The FOMM method (reference [1]) uses a dense motion network that takes face landmarks from the source / reference frame and driving frame (other frames) as input, along with the source frame, to generate a dense motion field and occlusion map. These are then provided to a generation module for further processing. Regarding the motion module, reference [1] describes five convolutions within the encoder. 3x3 -bn-relu-avg-pool 2x2 The block and the five upsamples within the decoder 2x2 -conv 3x3 It uses an architecture based on U-Net (reference

[18] ) which has -bn-relu blocks.

[0034] The generator module takes a source frame, a dense motion field, and an occlusion map as input. The dense motion field is used to align / warp the feature map calculated from the source frame with the object pose in the driving frame. The feature map is obtained from the source frame after two downsampling convolutional blocks. The warped feature map is then multiplied with the occlusion map to mask out feature map regions that should be inpainted because they cannot be restored using image warping. For the generator network, FOMM uses a Johnson architecture (reference

[19] ) with two downsampling blocks, six residual blocks, and two upsampling blocks.

[0035] References [2,9,10] use a reconstruction mechanism similar to that of reference [1]. Reference [8] uses a slightly different mechanism in dense motion network modules. It does not use information about the affine transformation matrix of keypoints while predicting flow and occlusion maps. Instead, it relies solely on keypoints.

[0036] Similar to reference [8], reference [7] also does not use affine transformation matrices during flow estimation. Furthermore, it modifies the FOMM architecture by adding a SPADE (Spatial Adaptation) (reference

[20] ) normalization layer to the upsampling block of the decoder network. They use face keypoints and draw polygons for the inside of the eyes, eyebrows, lips and mouth, and use them as semantic maps for SPADE. The generator network consists of a stack of three upsampling blocks and five residual blocks to which SPADE normalization is applied.

[0037] Similar to reference [1], reference [3] also uses source keypoints and driving keypoints to estimate the optical flow used to warp source features. This is fed into a motion estimation network to generate a flow configuration mask, which is then used to warp the 3D source features. The generator then converts the warped features into an output image.

[0038] In addition to motion feature maps (similar to reference [1]), reference

[12] uses attention maps and speech as further features. These are passed to a generator module in addition to the source frame to create the output. The generator module consists of an identity encoder (in which the source frame is supplied) which differs from reference [1].

[0039] [Video coding based on generated face video] In one embodiment, a general video compression method using generated face video has at least the following requirements: 1) Keyframes must be compressed using an image or video codec. For video codecs, keyframes are generally coded as intraframes. Since the decoded image is available to the decoder, facial features can be derived in the decoder to save some bits. To reduce the complexity of face reconstruction, facial features from keyframes can also be extracted in the encoder and sent to the decoder in a bitstream. Note that keyframes can also be coded as interframes. When coded as interframes, the reference picture must be handled carefully to ensure it is present in the decoder picture buffer (DPB) when interframe prediction is applied. For example, the reference picture may be set as a long-term reference picture. 2) Regarding the driving frame, the basic idea is to send only the facial features to a coded bitstream. The decoder then decodes the facial features and generates the driving frame via the generator block, using the information from the input generation for the generator block. The facial features are carried in High Level Syntax (HLS) within the video codec, such as video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), picture header (PH), supplemental enhancement information (SEI), or other means of metadata. In the proposed exemplary embodiments, but not limited to, the facial features are carried in new SEI messages. In some cases, it is proposed to code the driving frame at a much lower bitrate using interframe coding to preserve facial details (such as wrinkles that are different from keyframes), or to cover parts of the background occlusion due to head pose movement or camera movement, or to cover the appearance of new objects, or simply to code the missing details. For example, several preprocessing techniques can be applied, such as blurring the parts that can be generated by the generator and leaving only the parts that cannot be covered by the generator for compression. Alternatively, the residue between the original image and the generated image can simply be coded. In the decoder, the generated driving frames can be fused together with the decoded driving frames using simple addition and filtering or some more advanced neural network (NN) techniques. This has another advantage in that if the SEI message is lost or not understood by the decoder, the decoder can still output a video that has some meaning.Note that in video codec standards such as AVC, HEVC, and VVC, for any access unit, the SEI message must be bundled with the primary coded picture (see, for example, the structure of the access unit in AVC in Figure 7.1 of reference

[25] ). Therefore, if it is decided not to code the driving frame, a dummy picture still needs to be coded as the primary coded picture. The decoder must be notified via SEI messaging that the dummy picture will neither be displayed nor used for any processing.

[0040] To further reduce the bitrate, in one embodiment, it is proposed to use a spatial and / or temporal downsampling scheme in the overall coding framework. Furthermore, the complexity in the decoder for generated face video reconstruction can be reduced. In the first exemplary embodiment, the video is first downsampled (spatially and / or temporally), and then the proposed generated face video compression scheme can be used. In this case, both the coded video and the extracted face features are downsampled spatially and / or temporally. On the decoder side, the reconstructed video can be upsampled to the original spatial resolution and / or frame rate. The downsampling and upsampling processes can use a neural network method, and in some cases, can be absorbed into a neural network for feature generation and generator reconstruction.

[0041] In a second exemplary embodiment, the video is simply spatially downsampled for video compression. Facial features are extracted at their original resolution. Temporal downsampling can still be applied to both the video and facial features. In the decoder, the decoded video is spatially and temporally upsampled (if the driving frames were coded and temporally downsampled). Temporal downsampled facial features are interpolated for the full frame rate. The driving frames are then reconstructed using a generator at full resolution and / or full frame rate. Alternatively, instead of first temporally interpolating the features and then generating full-frame-rate driving frames, the driving frames may be generated first and then interpolated for full frame rate using a frame interpolation method.

[0042] Figure 2 shows an exemplary framework for a first exemplary embodiment including an encoder and decoder, where both video compression and feature extraction are at the same resolution and frame rate. As shown in Figure 2, the encoder side has two modules for processing the input video. • Spatial / Temporal Downsampling (205): This module performs downsampling to reduce spatial resolution and / or temporal frame rate. As described above, its purpose is to reduce bitrate and, in some cases, decoder complexity. This module is optional. • Face Feature Extraction (210): This module is for extracting facial and / or body / hand features. These extracted features are compressed and signaled with SEI messages (222) generated by the GFV SEI generation block (220).

[0043] The multiplexer (muxer 225) multiplexes the SEI messaging (222) to the output of the video compression block (215) to generate a coded video bitstream (227). In Figure 2, "face video" represents either a keyframe, a driving frame, or a dummy frame.

[0044] On the decoder side, the compressed bitstream and SEI messaging (227) are demultiplexed by the demultiplexer (230). Three main modules are required. • Spatial / Temporal Upsampling (245): This module performs upsampling in both the spatial and / or temporal domains (e.g., using super-resolution) to restore video frames to their original resolution and / or frame rate. This module is optional. • GFV SEI reconstruction (235): This module decompresses the compressed features stored in the SEI message to generate reconstructed facial features (237). • Face reconstruction (240): This module can receive input from decompressed SEI features (237) and decoded face video, reconstruct a high-quality face video, and, if necessary, upsample it temporally and spatially to produce a final reconstructed face video (250) at full spatial resolution and / or frame rate.

[0045] Figure 3 shows an exemplary framework for a second exemplary embodiment, where feature extraction (210) is performed on full-resolution video, and video compression (215) is performed at the downsampled resolution after spatial downsampling (207). Temporal downsampling (206) can be applied before both video compression and feature extraction.

[0046] In the decoder, spatial upsampling (247) follows video decoding, while temporal upsampling (246) (if present) follows face reconstruction.

[0047] [Proposed metadata syntax and semantics] JVET has made two proposals regarding generated facial videos (references [23-24]). These proposed SEI messages cover only facial features. The supported facial features are listed in Table 2. [Table 2]

[0048] In an exemplary embodiment, a series of novel SEI messages are proposed according to the processing pipeline shown in Figures 2 and 3. In addition to the facial features proposed by references [23,24], the novel SEI messages cover the following aspects: 1) A single face or multiple faces 2) Spatial sampling 3) Time sampling 4) Primary coded picture characteristics and driving picture processing 5) Background processing 6) Sustainability of SEI 7) Facial features and compression 8) How do NN post-filter representations (e.g., NNPFC and NNPFA SEI messaging (reference

[26] )) work?

[0049] [Multiple facial IDs] In video chats or conferences, more than one face may be present in the same video. References [23,24] propose the syntax gfv_id, which is used to identify the generated face video filter. gfv_id contains an identification number that can be used to identify the generated face video filter. The value of gfv_id is between 0 and 2. 32 It must be within the range of -2 or less.

[0050] In exemplary embodiments, two alternative designs are proposed. In the first design, gfv_id is repurposed to represent both face_id and generated face video filter. In one embodiment, bits 0 through 7 may be used to identify the generated face video filter, and bits 9 through 15 may be used to identify the face ID. In the second design, the number of faces is explicitly signaled using the syntax gfv_num_faces_minus1, and the number of faces is looped using face_id in a single generated face video SEI message. For example, in a frame with two faces (e.g., person 1 and person 2), gfv_num_faces_minus_1 can be set to 1, assigning a face_id equal to 0 to person 1 and a face_id equal to 1 to person 2. It is also necessary to specify the order of the people in the picture. In one embodiment, a "Z" scan order from left to right, then top to bottom, may be followed. Thus, Adding 1 to gfv_num_faces_minus1 specifies the number of faces described in this SEI message.

[0051] [Spatial sampling (gfv_spatial_sampling_flag)] In the encoder, spatial downsampling (205 or 207) of the input signal may be used. Generally, the generated face video network is trained end-to-end (e.g., from the input video in the encoder to the output of the reconstructed video in the decoder). It is useful to specify the input resolution to the generated face video network and the output resolution after the generated face video network. For example, in reference [4], the input resolution of the network is 256 × 256 and the output resolution of the network is 256 × 256. It is proposed to use presence flags for signaling spatial sampling and presence resolution flags for the input picture resolution and output picture resolution. Resolution can be explicitly signaled by width and height, or implicitly signaled by the scaling ratio of width and height. Exemplary syntax is shown in Table 3. [Table 3] A gfv_spatial_sampling_flag equal to 1 indicates that spatial sampling information exists for the generated face video. A gfv_spatial_sampling_flag equal to 0 indicates that spatial sampling information does not exist for the generated face video. A gfv_input_resolution_present_flag equal to 1 indicates that the syntax gfv_input_pic_width_in_luma_samples and gfv_input_pic_height_in_luma_samples exist. A gfv_input_resolution_present_flag equal to 0 indicates that the syntax gfv_input_pic_width_in_luma_samples and gfv_input_pic_height_in_luma_samples do not exist. gfv_input_pic_width_in_luma_samples and gfv_input_pic_height_in_luma_samples specify the width and height of the luma sample array of the input picture to the generated face network in the SEI message, respectively. If they are not present, they are assumed to be equal to CroppedWidth and CroppedHeight, respectively, which are the width and height of the decoded cropped picture in units of luma samples. The same semantics may be defined for the output resolution parameter.

[0052] [Time sampling (gfv_temporal_sampling_flag)] To further reduce the bitrate, time subsampling (205, 206) (frame rate reduction) can be used in the encoder to reduce the bitrate and transmit facial features. In the decoder, frame interpolation can be applied to interpolate missing pictures. An alternative method is to interpolate the facial features (keypoints, etc.) of the missing picture and then generate the missing picture using GFV.

[0053] Instructions for time rate reduction can be implicit or explicit. For implicit signaling, one example is when a primary coded picture may not be coded, and therefore the SEI is not available for the missing frame. The decoder learns of the missing frame from system-level information or other means. For explicit signaling, there are two exemplary methods: 1) When a primary picture is coded as an intra-random access picture (IRAP), a time sublayer can be used to indicate whether a GFV SEI is transmitted for that sublayer. This explicitly signals if the GFV SEI is intentionally missing for a particular picture and has not been dropped. 2) A new flag (e.g., gfv_temporal_sampling_flag) is signaled in the SEI to indicate whether feature information is signaled in this SEI. This method is explicit only if a primary coded picture exists. Examples of explicit signaling methods are shown in Table 4. [Table 4] The gfv_IRAP_flag is equal to 1 when the primary coded picture is an intra random access picture (IRAP). The gfv_IRAP_flag is equal to 0 when the primary coded picture is not an IRAP picture. Adding 1 to gfv_max_sub_layers_minus1 specifies the maximum number of time sublayers containing GFV SEI. Note that if you need more flexibility in coding keyframes, you can use gfv_key_flag instead of gfv_IRAP_flag. gfv_key_flag is equal to 1 when the primary coded picture is a key picture and is used for face generation for the driving picture. gfv_key_flag is equal to 0 when the primary coded picture is not a key picture.

[0054] [Primary coded picture characteristics and driving picture processing (gfv_drive_pic_idc)] In an exemplary embodiment, there are four types of primary coded pictures: 1) Keyframe: coded as an intra picture. 2) Dummy picture: this picture cannot be used for display or other post-processing. The driving picture is generated using facial features transported by SEI and generated by the generator. 3) A driving picture that is fused with the generated driving picture from the generator. The coded driving picture can be used to improve facial details or to process background changes, etc. 4) A driving picture that is displayed alone. This is not intended to be fused with the generated driving image. Keyframes can be derived from the primary coded picture type, or gfv_IRAP_flag can be used. For driving pictures, a new syntax element (gfv_drive_pic_idc) is proposed to distinguish between the three types. gfv_drive_pic_idc indicates a driving picture type as specified in Table 5. [Table 5]

[0055] In typical use cases, keyframes are used by default to generate driving frames (pictures). However, in some cases, compared to keyframes, driving frames can function as better "key" frames that are closer to the current frame, such as facial expressions or posture. Alternatively, driving frames can work with keyframes for better face generation, for example, to make it smoother, or to better handle the background. In this case, it is necessary to indicate whether the generated driving frame should replace the decoded one in the DPB. The syntax in SEI messaging also needs to signal which frame (uni_pred), multiple frames (bi_pred), or any number of reference pictures in the DPB are being used. The delta_poc (difference between picture order count (POC)) between the current frame and the reference frame (ref_frame) can be used to indicate which reference picture is being used. An example is given in Table 6. [Table 6] A gfv_drive_pic_adv_flag value equal to 1 indicates the presence of syntax related to advanced drive picture processing. A gfv_drive_pic_adv_flag value equal to 0 indicates the absence of syntax related to advanced drive picture processing. A gfv_drive_pic_ref_flag equal to 1 indicates that the generated driving picture may function as a reference picture for face generation for subsequent pictures in the decoded order. A gfv_drive_pic_ref_flag equal to 0 indicates that the generated driving picture will not function as a reference picture for face generation for subsequent pictures in the decoded order. A gfv_bipred_flag equal to 0 specifies that face generation will use one reference picture. A gfv_bipred_flag equal to 1 specifies that face generation will use two reference pictures. delta_poc0 and delta_poc1 specify the POC difference between the current picture and the 0th and 1st reference pictures, respectively.

[0056] To handle backward compatibility with existing decoders that do not understand GFV SEI messages, or in the event that GFV SEI messages are lost, several methods can be applied.

[0057] Method 1: Mark the decrypted picture as a non-output picture. If the currently decoded driving picture is a dummy picture or is used for the purpose of merging with a generated picture, the decoded picture can be marked as a non-output picture to prevent a normal decoder from outputting it. Therefore, a normal decoder will only output the key picture. Method 2: Modifying VUI signaling The vui_non_packed_constraint_flag can be reused. A vui_non_packed_constraint_flag equal to 1 specifies that no frame packing placement SEI messages or GFV SEIs exist in the bitstream to which CLVS is applied. A vui_non_packed_constraint_flag equal to 0 means no such constraint is imposed.

[0058] [Background processing (gfv_bg_flag)] In video conferencing, the background can occupy a large portion of the image. Sometimes, even small background movement between frames, or changes in a person's head posture between frames, can cause undesirable artifacts, such as occlusion. Several methods can be used to handle background changes. As discussed above, background changes can be handled using coded driving pictures. Alternatively, background changes can be predicted using affine transformation matrices. The two methods can work together and therefore allow for coding fewer bits in the coded driving picture. In general, background processing is much simpler than processing human faces and heads. It is proposed to explicitly signal the affine transformation matrices for the background. For example, the gfv_bg_flag flag may be used to signal the presence of background processing. A gfv_bg_flag equal to 1 indicates that syntax related to background features corresponding to the generated face video exists. A gfv_bg_flag equal to 0 indicates that syntax related to background features corresponding to the generated face video does not exist.

[0059] [Sustainability of GFV SEI] In GFV communications, features are expected to be updated frame by frame, and therefore, it is appropriate to consider the persistence of GFV SEI messages to be only for the current frame. However, there may be cases where it is desirable to save bits by not transmitting updated features, or where the difference in features from a previous frame is too small to transmit. In such cases, the frame can be completely dropped using time sampling, or SEI persistence can be applied.

[0060] [Facial features] This section discusses facial features not covered in references [23-24], and then proposes a revised syntax for facial features. Table 2 lists the features supported in references [23-24]. It is proposed to add two new features to GFV SEI. -FACS: The facial motion coding system was discussed above. Using FACS, almost any anatomically possible facial expression can be manually coded. This feature is not present in Table 2. - Background Features: As mentioned above, there are advantages to listing background features. The use of background features can be very different from that of faces. Therefore, it is proposed to separate these features from facial features.

[0061] Furthermore, a new design of syntax elements for all features is proposed. In reference

[24] , all matrices are mixed together using the matrix_idx variable, as shown in Table 7. Some parameters in the table can work with keypoints such as affine transformation matrices or covariance matrices, but some parameters such as head rotation and transformation matrices or compact feature matrices cannot. For easier understanding or clearer presentation and combination, separate syntax is proposed. [Table 7]

[0062] In one embodiment, features are classified into six classes: 1) keypoints, 2) region matrices, 3) 3D head transformation matrices, 4) compact features, 5) FACS and face semantics, and 6) background features. Furthermore, while references [23-24] directly code all features using ue(v) (an unsigned integer using exponential Golomb coding), a more efficient coding scheme is proposed to reduce the bitrate of GFV SEI messaging. The proposed compression process is shown in Figure 4 and will be described in more detail below.

[0063] As shown in Figure 4, on the encoder side, there are three main steps after the arbitrarily selected feature preprocessing (405). 1) Quantization of coded feature parameters (410) 2) Entropy reduction of quantized parameters (415) (e.g., coding feature residuals using difference coding instead of directly coding features) 3) Entropy coding (420) of quantized feature parameters to generate a coded bitstream (422)

[0064] On the decoder side, given a coded bitstream (422), the process is reversed. 1) Feature Entropy Decoding (425) 2) Restoration of entropy reduction (430) (when entropy reduction (415) is applied in the encoder) 3) Inverse quantization (435), followed by an optional feature post-processing (440)

[0065] In the alternative method, the order of steps 1)(410) and 2)(415) in the encoder can be swapped. The main difference is whether entropy reduction coding is performed before or after quantization. As shown in Figure 4, feature preprocessing (405) may be added in the encoder before step 1)(410) to preprocess or reshape feature data for easier compression. Examples of preprocessing include lumamapping or "image reshaping" (e.g., luma-mapping chroma scaling (LMCS) in VVC) to improve coding efficiency.

[0066] The six main categories proposed and their corresponding syntax elements will be discussed next. 1) Key points in 2D and 3D space (gfv_kp_flag) 2) Region matrix (gfv_rm_flag) in 2D and 3D space 3) 3D head transformation matrix (gfv_ht_flag) 4) Compact features (gfv_cf_flag) 5) FACS and facial semantics (gfv_st_flag) 6) Background processing features (gfv_bg_flag)

[0067] [1] Key points in 2D and 3D space (gfv_kp_flag) ] Keypoints can be facial landmarks or other keypoints that do not have semantic meaning. Compared to the syntax design in reference

[24] , the following is proposed: 1) To make coding keypoints more efficient, or to simplify GFV parsing in decoders, it is beneficial to signal whether or not a keypoint is a facial landmark. 2) For each keypoint, an additional local matrix, such as an affine matrix or covariance matrix, can be optionally supported. 3) Improve keypoint compression to save bits, and further allow 2D keypoints and 3D keypoints to be specified independently (exclusively) or together.

[0068] For example, only 2D keypoints are considered, but this technology can be easily extended to 3D keypoints. For 3D keypoints, it may be necessary to specify a coordinate system, for example, using the OMAF (ISO / IEC 23000-20 Omnidirectional Media Format) coordinate system.

[0069] In the encoder, 2D keypoints are specified in grid[-1 1]x[-1 1]. Assuming that the quantization coefficient is specified using q_step (where q_step is an integer), the quantization in the encoder for each coordinate is specified as follows: KP_quant=Round((KP+1)*q_step) In the decoder, inverse quantization is specified as follows: KP_dec=((float)KP_quant / (float)q_step)-1 Here, KP is a floating-point value in the range [-1 1], q_step is a positive integer quantization coefficient, KP_quant is a quantized integer value in the range [0, 2 * q_step], and KP_dec is an inversely quantized floating-point value in the range [-1, 1].

[0070] KP_quant can be coded directly with ue(v). Alternatively, differential coding or residual coding can be applied before entropy coding. The encoder can apply spatial prediction, temporal prediction, or template prediction to generate the residual as specified below. KP_residue=KP_quant-ref_quant Here, ref_quant can be a quantized spatial predictor, a temporal predictor, or a template predictor. In the decoder, the following applies: KP_quant = KP_residue + ref_quant

[0071] For spatial prediction, previously numbered keypoints for the current picture can function as references (ref_quant) to the current keypoint. Sometimes, keypoints can be clustered into groups. Therefore, spatial prediction occurs only within groups. For example, if 2D face landmarks are used as keypoints, using the Dlib library as an example, 68 face landmarks can be clustered into six groups, namely 1-17, 18-27, 2-36, 37-42, 43-48, and 49-68. In each group, the first keypoint is either absolutely coded or spatially predicted by a specified KP; for example, for 18, 1 is specified as the spatial predictor. For the other keypoints within the group, the previously numbered keypoints are used as spatial predictors.

[0072] For time prediction, the literature (references [3-6]) suggests that features from previously reconstructed pictures are used as references. The previous picture can be the temporally closest decoded picture, or it can be adaptively selected from previously decoded pictures in the DPB. Such a method is undesirable in coding standards due to error tolerance issues. It is proposed to use features from keyframes as time predictors / references. 1) Keyframes have coded source images. If they are lost, the GFV system is destroyed anyway. 2) If the keyframe SEI is lost, the decoder can use the coded image as error hiding to detect keypoints.

[0073] In addition to spatial and temporal predictions, a third method called a template predictor is proposed. The positions of all keypoints within a normalized template can be predefined and serve as references for prediction. In one embodiment, each keypoint or group of keypoints can be made to adaptively select a reference predictor. Regarding residual coding, there may be many zeros, especially when the number of keypoints is relatively large or when time prediction is used. In such cases, it is sufficient to simply code the keypoint numbers that have non-zero residual values. In one embodiment, conventional run-length coding can be applied. In another embodiment, the "non-zero residual" keypoint numbers and the residual values ​​may be coded separately.

[0074] For entropy coding, it is preferable to apply exponential Golomb coding. Typically, zero-order exponential Golomb codes such as ue(v) and se(v) are used. The remainder can be negative, zero, or positive. It can be coded as se(v), or its absolute value can be coded first. If the absolute value is not equal to zero, its sign is also coded. Examples of syntax for coding keypoints are specified in Table 8. [Table 8-1] [Table 8-2]

[0075] For the syntax examples shown in Table 8, the number of keypoints for facial landmarks is generally assumed to be greater than the number for other cases. Therefore, when facial landmarks are used, or when the number of keypoints is greater than a predefined threshold TH_KP (e.g., 10, or additional syntax can be added to signal TH_KP), spatial residual coding may be used if it is an IRAP picture, and otherwise, temporal residual coding may be applied. If the keypoints are not facial landmarks, direct coding is used if it is an IRAP picture, and otherwise, temporal residual coding is used for all keypoints.

[0076] The key point is that both facial and non-facial landmarks are permitted. When standard facial landmarks are used for spatial group prediction, this case can be categorized as a non-facial landmark.

[0077] Note that when scanning a matrix from a 2D to a 1D representation, row-direction scanning is used.

[0078] Note that for the quantization coefficient (q_step), we can choose a power of 2 to replace division with a binary shift. In that case, we simply signal log2(q_step) to save more bits.

[0079] The exemplary semantics in Table 8 are defined as follows: A gfv_kp_flag equal to 1 indicates that the syntax element associated with the keypoint exists in the SEI. A gfv_kp_flag equal to 0 indicates that the syntax element associated with the keypoint does not exist in the SEI. gfv_log2_kp_quant_factor specifies the value of the keypoint quantization coefficient qKp. qKp=2 (gfv_log2_kp_quant_factor) A gfv_facial_landmark_flag equal to 1 indicates that the keypoint is a facial landmark. A gfv_facial_landmark_flag equal to 0 indicates that the keypoint is not a facial landmark. A gfv_3D_kp_flag equal to 1 specifies that the keypoint is in 3D coordinates. A gfv_3D_kp_flag equal to 0 specifies that the keypoint is in 2D coordinates. Adding 1 to gfv_num_kp_minus1 specifies the total number of keypoints. Each keypoint is associated with a numbered identifier, identified as the kpID, which ranges from 0 to gfv_num_kp_minus1. A gfv_kp_local_matrix_flag equal to 1 indicates that the keypoint has an associated local matrix. A gfv_kp_local_matrix_flag equal to 0 indicates that the keypoint does not have an associated local matrix. delta_x_coordinator, delta_y_coordinator, and delta_z_coordinator indicate that the x, y, and z coordinates of the keypoints are coded as the residue. x_coordinator, y_coordinator, and z_coordinator indicate that the x, y, and z coordinates of the keypoints are directly coded. Adding 1 to gfv_num_kp_update_minus1 specifies the number of keypoints whose coordinate values ​​differ from the reference. gfv_kp_id[j] specifies the kpID number of the j-th updated keypoint. gfv_kp_local_matrix[i][j] specifies the value of the j-th local matrix coefficient of the i-th keypoint.

[0080] [2] Region matrices (gfv_rm_flag) in 2D and 3D space ] The characteristics of each region or segmentation can be represented by a matrix. To save bits, it is not necessary to code the details of the region segmentation. The decoder can derive the same region / segmentation information from the key (intra) frame. The improvements compared to reference

[24] are as follows: 1) Reference

[24] only considers the 2D case and does not support the 3D case. 2) Reference

[24] mixes all matrices together, such as keypoint-based, region-based, and head-pose matrices. The matrices are separated to improve clarity. Table 9 shows an example of the proposed syntax. [Table 9] [Table 10]

[0081] In the example above, matrix coefficients are coded directly, but note that differential coding can be used, as with keypoints. The semantics are defined as follows: A gfv_rm_flag equal to 1 indicates that the syntax elements related to the region matrix are present in the SEI. A gfv_rm_flag equal to 0 indicates that the syntax elements related to the region matrix are not present in the SEI. gfv_log2_rm_quant_factor specifies the value of the domain matrix quantization coefficient qRm. qRm=2 (gfv_log2_rm_quant_factor) A gfv_rm_3D_flag equal to 1 specifies a 3D region matrix. A gfv_rm_3D_flag equal to 0 specifies a 2D region matrix. Adding 1 to gfv_num_region_minus1 specifies the total number of regions. Each region is associated with a numbered identifier ranging from 0 to gfv_num_region_minus1, which is identified as the region ID. Adding 1 to gfv_num_rm_minus1 specifies the total number of region matrices. Each region matrix is ​​associated with a numbered identifier within the range of 0 to gfv_num_rm_minus1. gfv_rm_id[k] specifies the rmID of the k-th region matrix. The rmID is defined as shown in Table 10. gfv_rm_coeff[i][k][j] specifies the i-th region, the k-th matrix, and the j-th coefficient value.

[0082] [3] 3D head transformation matrix (gfv_ht_flag) ] 3D head transformation plays a crucial role in facial video. A 3D head transformation matrix typically includes a rotation matrix (3x3) and a translation matrix (1x3), resulting in a total of 12 parameters. The 3D head transformation matrix can work in conjunction with other features. An exemplary syntax is shown in Table 11. [Table 11]

[0083] Note that in the example above, the coefficients of the 3D head transformation matrix are coded directly. Differential coding can be used, as with keypoints. The semantics are defined as follows: A gfv_ht_flag equal to 1 indicates that syntax elements related to the 3D head transformation matrix exist in the SEI. A gfv_ht_flag equal to 0 indicates that syntax elements related to the 3D head transformation matrix do not exist in the SEI. The gfv_log2_ht_quant_factor specifies the value of the 3D head transform quantization coefficient qHt. qHt = 2 (gfv_log2_ht_quant_factor) The gfv_ht_coeff[i] specifies the value of the i-th coefficient.

[0084] [4) Compact features (gfv_cf_flag) Compact features generally represent learning-based features from a neural network. An alternative term could be "compact learning features". Compared to reference

[24] , flexibility should be allowed to specify the dimensions of the compact features and parameter values. An exemplary syntax is shown in Table 12.

Table 12

[0085] Note that in the above example, the compact feature parameters are directly coded. Differential coding can be used as in the case of keypoints. The semantics are defined as follows. The gfv_cf_flag equal to 1 specifies that the syntax element related to the compact feature is present in the SEI. The gfv_cf_flag equal to 0 specifies that the syntax element related to the compact feature is not present in the SEI. The gfv_log2_cf_quant_factor specifies the value of the compact feature quantization coefficient qCf. qCf = 2 (gfv_log2_cf_quant_factor) One plus gfv_cf_length_minus1 specifies the length of the compact feature. The gfv_cf_coeff[i] specifies the value of the i-th coefficient.

[0086] [5) FACS and facial semantics (gfv_st_flag) ​​FACS parameters may be coded based on reference [4]. There are 99 AU numbers, some of which may have multiple sub-AUs, such as the M prefix. There are five intensity scores, namely A, B, C, D, and E (for minimum-maximum intensity). There are four modifiers, namely L, R, U, and A. A simple coding using u(v) is to assign 7 bits to the AU number, 1 bit to the M prefix, 3 bits to the intensity score, and 2 bits to the modifier. Therefore, in total, 13 bits are needed to code it.

[0087] The facial semantics used in reference

[24] are considered a simpler case for FACS, as shown in Table 1. This is based on a 3D Morphable Model (3DMM). For the head, 1x3 dimensions (yaw, roll, and pitch) are used to identify head rotation, and 1x1 dimensions are used to locate the face from the input image. This representation is different from the head pose matrix. When facial semantics are used as in Table 1, gfv_hp_flag is assumed to be set to 0. Note that the matrix dimensions are fixed in Table 1. Therefore, it is not necessary to signal the matrix dimensions as in reference

[24] . An exemplary syntax is shown in Table 13. [Table 13]

[0088] Note that in the example above, the coding of face semantics coefficients is done directly. Differential coding can be used, as with keypoints. The semantics are defined as follows: A gfv_st_flag equal to 1 indicates that syntax related to face semantics exists in the SEI. A gfv_st_flag equal to 0 indicates that syntax related to face semantics does not exist in the SEI. gfv_log2_st_quant_factor specifies the value of the face semantics quantization coefficient qSt. qSt=2 (gfv_log2_st_quant_factor) A gfv_facs_flag value equal to 1 indicates that FACS-related syntax exists. A gfv_facs_flag value equal to 0 indicates that FACS-related syntax does not exist. Adding 1 to gfc_facs_num_minus1 specifies the total number of FACS parameters. gfv_facs_coeff[i] specifies the i-th FACS parameter. gfv_fs_coeff[i][j] specifies the i-th semantic and the value of the j coefficient.

[0089] [6] Background processing features (gfv_bg_flag) ] For the background, it is proposed to use an affine background transformation. The dimension is 2 × 3, and therefore there are a total of 6 parameters. An exemplary syntax is shown in Table 14. [Table 14]

[0090] Note that in the example above, the background matrix coefficients are coded directly. Differential coding can be used, as with keypoints. The semantics are defined as follows: A gfv_bg_flag equal to 1 indicates that syntax related to background features corresponding to the generated face video exists. A gfv_bg_flag equal to 0 indicates that syntax related to background features corresponding to the generated face video does not exist. gfv_log2_bg_quant_factor specifies the value of the background matrix quantization coefficient qBg. qBg=2 (gfv_log2_bg_quant_factor) gfv_bg_coeff[i] specifies the value of the i-th coefficient.

[0091] While voice / audio features can also be added to the syntax to support generated facial videos, it should be noted that such syntax goes beyond the scope of the proposed embodiments.

[0092] [Overall Syntax Overview] Bringing all the syntax elements together, Table 15 provides an example of an overall syntax for GFV metadata using SEI messaging. [Table 15-1] [Table 15-2]

[0093] In another embodiment, Table 16 provides an alternative, simplified version of Table 15. [Table 16]

[0094] The Generative Face Video (GFV) SEI message specifies a neural network, indicated as Generator(), which may be used to generate a new output picture using the indicated face feature information of the source picture before encoding and the previously decoded output picture. For example, Generator() may be a generative adversarial network (GAN).

[0095] Using this SEI message requires the definition of the following variables. -Here, the input picture width and height are shown in lumens, indicated by CroppedWidth and CroppedHeight, respectively. - Lunar sample array keyCroppedYPic and chromatic sample arrays keyCroppedCbPic and keyCroppedCrPic for the decoded output picture, indicated as KeyPicture, corresponding to the source key picture. - Lunar sample array driveCroppedYPic and chroma sample array driveCroppedCbPic and driveCroppedCrPic for the decoded output picture, shown as DrivePicture, corresponding to the source driving picture. - BitDepth for the lumens sample array of the input picture Y - BitDepth for the chroma sample array of the input picture, if present. C -The chroma format indicator described herein by ChromaFormatIdc, as described in Section 7.3 of H.274(VSEI) (Reference

[27] ). The variables SubWidthC and SubHeightC are derived from ChromaFormatIdc as specified in Table 2 of H.274(VSEI) (reference

[27] ). gfv_id contains an identification number that can be used to identify facial feature information and specify a neural network that can be used as Generator(). The value of gfv_id is between 0 and 2. 32 It must be within the range of -2 or less. 256 or more and 511 or less and 2 31 The above 2 32 gfv_id values ​​less than or equal to -2 are reserved for future use by ITU-T|ISO / IEC. Decoders conforming to this edition of this document are in the range of 256 to 511 or 2 31 The above 2 32 When a GFV SEI message with a gfv_id within the range of -2 or less is encountered, that SEI message will be ignored. Note - Different values ​​for gfv_id in different GFV SEI messages may be used to identify different faces, for example, when two or more faces are present in the output picture. gfv_nn_base_flag, gfv_nn_mode_idc, gfv_nn_reserved_zero_bit_a, gfv_nn_tag_uri, gfv_nn_uri, and gfv_nn_payload_byte[i] specify a neural network that can be used as Generator(). gfv_nn_base_flag, gfv_nn_mode_idc, gfv_nn_reserved_zero_bit_a, gfv_nn_tag_uri, gfv_nn_uri, and gfv_nn_payload_byte[i] have the same syntax and semantics as nnpfc_base_flag, nnpfc_mode_idc, nnpfc_reserved_zero_bit_a, nnpfc_tag_uri, nnpfc_uri, and nnpfc_payload_byte[i], respectively. A gfv_key_pic_flag equal to 1 indicates that the current decoded output picture corresponds to the key picture. A gfv_key_pic_flag equal to 0 indicates that the current decoded output picture does not correspond to the key picture. If present, a gfv_drive_pic_flag equal to 1 indicates that the current decoded output picture corresponding to the driving picture may be input to Generator(). A gfv_drive_pic_flag equal to 0 indicates that the current decoded output picture should not be input to Generator(). Note-1: The gfv_drive_pic_flag value may be used to indicate that the currently decoded output picture may be used, for example, to improve facial detail or to handle background changes. Facial feature information is coded as 2D keypoints specified using (x,y) coordinate pairs within a grid [-1,+1]×[-1,+1]. gfv_log2_kp_quant_factor specifies the quantization coefficient qKp, which is determined as follows. qKp=2 (gfv_log2_kp_quant_factor) The inverse quantization function DeqKp() is specified as follows: DeqKp(x)=Clip3(-1,1,(x÷qKp)-1) Adding 1 to gfv_num_kp_minus1 specifies the number of 2D keypoints specified in the GFV SEI message. A gfv_kp_pred_flag equal to 1 indicates that the syntax elements gfv_kp_grid_pos_dx[i] and gfv_kp_grid_pos_dy[i] are present in the GFV SEI message and that the values ​​of the syntax elements gfv_kp_grid_pos_x[i] and gfv_kp_grid_pos_y[i] are derived. A gfv_kp_pred_flag equal to 0 indicates that the syntax elements gfv_kp_grid_pos_dx[i] and gfv_kp_grid_pos_dy[i] are not present, but the syntax elements gfv_kp_grid_pos_x[i] and gfv_kp_grid_pos_y[i] are present in the GFV SEI message. gfv_kp_grid_pos_x[i] specifies the x component of the (x,y) coordinate pair of the i-th 2D keypoint. gfv_kp_grid_pos_y[i] specifies the y component of the (x,y) coordinate pair of the i-th 2D keypoint. If present, gfv_kp_grid_pos_dx[i] specifies the delta x component of the (x, y) coordinate pair of the i-th 2D keypoint. If present, gfv_kp_grid_pos_dy[i] specifies the delta y component of the (x, y) coordinate pair of the i-th 2D keypoint. When gfv_kp_pred_flag is equal to 1, gfv_kp_grid_pos_x[i] and gfv_kp_grid_pos_y[i] are derived as follows:

number

number

number

number

[0096] In one embodiment, the following syntax elements can also be part of the SEI message. fdi_type_idc indicates the type of feature descriptor information determined by the content provider, as specified in Table 17, for example. [Table 17] fdi_purpose_idc indicates the purpose of the feature descriptor, as determined by the content provider, for example, as specified in Table 18. [Table 18]

[0097] [Explanation of Generator() input and output] The input value to Generator() is a real number, and the functions InpY() and InpC() are specified as follows: InpY(x) = x ÷ ((1< <BitDepth Y )-1) InpC(x) = x ÷ ((1< <BitDepth C )-1) The output value from Generator() is a real number, and the functions OutY() and OutC() are specified as follows. OutY(x) = Clip3(0,(1< <BitDepth Y )-1,x*((1< <BitDepth Y )-1) OutC(x) = Clip3(0,(1< <BitDepth C )-1,x*((1< <BitDepth C )-1)

[0098] When gfv_key_pic_flag is equal to 1, the KeyPicture luma sample array KeyY is derived as follows: KeyY[x][y]=InpY(keyCroppedYPic[x][y]), where x=0..CroppedWidth-1 and y=0..CroppedHeight-1. When gfv_key_pic_flag is equal to 1 and chromaIDC is not equal to 0, the KeyPicture chroma sample arrays KeyCb and KeyCr are derived as follows: KeyCb[x][y]=InpC(keyCroppedCbPic[x][y]), where x=0..CroppedWidth / SubWidthC-1 and y=0..CroppedHeight / SubHeightC-1. KeyCr[x][y]=InpC(keyCroppedCrPic[x][y]), where x=0..CroppedWidth / SubWidthC-1 and y=0..CroppedHeight / SubHeightC-1.

[0099] When gfv_drive_pic_flag is equal to 1, the DrivePicture lunar sample array driveY is derived as follows: driveY[x][y]=InpY(driveCroppedYPic[x][y]), where x=0..CroppedWidth-1 and y=0..CroppedHeight-1. When gfv_drive_pic_flag is equal to 1 and chromaIDC is not equal to 0, the DrivePicture chroma sample arrays driveCb and driveCr are derived as follows: driveCb[x][y]=InpC(driveCroppedCbPic[x][y]), where x=0..CroppedWidth / SubWidthC-1 and y=0..CroppedHeight / SubHeightC-1. driveCr[x][y]=InpC(driveCroppedCrPic[x][y]), where x=0..CroppedWidth / SubWidthC-1 and y=0..CroppedHeight / SubHeightC-1.

[0100] The input to Generator() is as follows: -When gfv_key_pic_flag is equal to 0 and gfv_drive_pic_flag is equal to 0, deqFeature, KeyDeqFeature, KeyY, KeyCb, KeyCr When gfv_key_pic_flag is equal to 0 and gfv_drive_pic_flag is equal to 1, deqFeature, KeyDeqFeature, KeyY, KeyCb, KeyCr, driveY, driveCb and driveCr The output of Generator() is as follows: -When chromaIDC is equal to 0, the chroma sample array genY -When chromaIDC is not equal to 0, the chroma sample array genY and the chroma sample arrays genCb and genCr are used. The arrays outYPic[x][y], outYPic[x][y], and outYPic[x][y] are derived as follows. -If gfv_key_pic_flag is equal to 0, the following applies: -outYPic[x][y]=OutY(genY[x][y]), where x=0..CroppedWidth-1 and y=0.CroppedHeight-1. -If chromaIDC is not equal to 0, the following applies: - outCbPic[x][y]=OutC(genCb[x][y]), where x=0..CroppedWidth / SubWidthC-1 and y=0..CroppedHeight / SubHeightC-1. -outCrPic[x][y]=OutC(genCr[x][y]), where x=0..CroppedWidth / SubWidthC-1 and y=0..CroppedHeight / SubHeightC-1. -Otherwise (if gfv_key_pic_flag is equal to 1), the following applies: -outYPic[x][y]=keyCroppedYPic[x][y] -If chromaIDC is not equal to 0, the following applies: -outCbPic[x][y]=keyCroppedCbPic[x][y] -outCrPic[x][y]=keyCroppedCrPic[x][y]

[0101] In one embodiment, exemplary pseudocode for using Generator() may include the following: Parse and process GFV SEI messages { If the currently decrypted output picture corresponds to the key picture { Stores decoded 2D keypoints representing the facial features of the key picture. Store the current decrypted output picture corresponding to the key picture. } Otherwise, / * the currently decoded output picture corresponds to the driving picture * / Stores decoded 2D keypoints representing facial feature information from the driving picture. Optionally, store the currently decoded output picture corresponding to the key picture. Prepare the input for Generator(). Decrypted picture corresponding to the key picture Decrypted 2D keypoints for key pictures Decoded 2D keypoints for driving pictures Optionally, decoded picture corresponding to the driving picture The input is applied to Generator() to generate an output picture that approximates the driving picture. Store the generated output picture. } }

[0102] Table 15 uses multiple flags to detect whether support exists for various features. An alternative method is to use bitmasking, which is easier for feature expansion and combination. gfv_feature represents the feature specified in Table 15, and a value not equal to 0 (gfv_feature&bitMask) indicates that the GFV will use the feature associated with the bitMask in Table 19. [Table 19]

[0103] It should be noted that most of the proposed features, such as keypoints, region matrices, and compact features, do not necessarily have to be limited to faces. Further features may be used for generated image / video compression. The proposed SEI messages can be generalized as generated video SEI messages (GV SEI). A new syntax, gv_purpose_idc, can be defined to indicate the purpose of the generated video. gv_purpose_idc indicates the purpose of the GV message, for example, as specified in Table 20. [Table 20]

[0104] [Relationship with Neural Network Post-Filter SEI Messages] To generate generated face videos using GFV SEI, it may be necessary to access a neural network generator model. For interoperability, it may be necessary to specify how to access the network model according to the standard. NNPFC SEI (reference

[26] ) provides such capability using the syntax element nnpfc_mode_idc. To use NNPFC SEI for GFV, the following aspects need to be updated.

[0105] In the currently proposed modifications to the VSEI (H.274) specification (reference

[26] ), the neural network post-filter SEI message is designed to enable post-filtering using a neural network. This includes two SEIs: the neural-network post-filter characteristics SEI message (NNPFC) and the neural-network post-filter activation (NNPFA). The neural-network post-filter characteristics SEI message specifies the neural network that can be used as a post-processing filter. The use of a specified neural-network post-processing filter (NNPF) for a particular picture is indicated by the neural-network post-filter activation (NNPFA) SEI message.

[0106] Appearance 1: In NNPFC, the syntax nnpfc_purpose exists. As stated in reference

[26] , "nnpfc_purpose indicates the purpose of the NNPF as specified in Table 20 (copied here as Table 21), and (nnpfc_purpose&bitMask) not equal to 0 indicates that the NNPF has a purpose associated with the bitMask value in Table 20. When nnpfc_purpose is greater than 0 and (nnpfc_purpose&bitMask) is equal to 0, the purpose associated with the bitMask value is not applicable to the NNPF. When nnpfc_pospose is equal to 0, the NNPF may be used as determined by the application." The value of nnpfc_purpose shall be between 0 and 63 in the bitstream conforming to this version of this document. Values ​​for nnpfc_purpose between 64 and 65535 are reserved for future use by ITU-T|ISO / IEC and shall not exist in the bitstream conforming to this version of this document. Decoders conforming to this version of this document shall ignore NNPFC SEI messages with nnpfc_purpose between 64 and 65535. [Table 21]

[0107] To enable the use of generated face videos, it is proposed to add one purpose for generated face videos for bitMark 0x40, shown in bold italics in the table below. For broader use, the syntax name "Generate AI" may be used as an example. [Table 22]

[0108] Embodiment 2: The NNPFC is designed without considering further input of facial features. It is proposed to use the syntax nnpfc_auxiliary_inp_idc to transport facial feature information from GFV_SEI. According to reference

[26] , "A value of nnpfc_auxiliary_inp_idc greater than 0 indicates that the auxiliary input data is present in the NNPF input tensor. A value of nnpfc_auxiliary_inp_idc equal to 0 indicates that the auxiliary input data is not present in the input tensor. A value of nnpfc_auxiliary_inp_idc equal to 1 indicates that the auxiliary input data is derived as specified in equation 85." The value of nnpfc_auxiliary_inp_idc shall be between 0 and 1 in the bitstream conforming to this version of this document. Values ​​of nnpfc_auxiliary_inp_idc between 2 and 255 are reserved for future use by ITU-T|ISO / IEC and do not exist in the bitstream conforming to this version of this document. Decoders conforming to this version of this document shall ignore NNPFC SEI messages with nnpfc_auxiliary_inp_idc between 2 and 255. Values ​​of nnpfc_auxiliary_inp_idc greater than 255 do not exist in the bitstream conforming to this version of this document and are not reserved for future use. To show the proposed updated syntax, use bold italics or underline Using this, the syntax of nnpfc_auxiliary_inp_idc can be updated as follows: A value of nnpfc_auxiliary_inp_idc equal to 0 indicates that no auxiliary input data is present in the input tensor. A value of nnpfc_auxiliary_inp_idc equal to 1 indicates that the auxiliary input data is derived as specified in Equation 85. A value of nnpfc_auxiliary_inp_idc equal to 2 specifies that the auxiliary input data is derived from the GFV SEI. The value of nnpfc_auxiliary_inp_idc shall be between 0 and 2 in the bitstream conforming to this version of this document. Values ​​of nnpfc_auxiliary_inp_idc between 3 and 255 are reserved for future use by ITU-T|ISO / IEC and do not exist in the bitstream conforming to this version of this document. Decoders conforming to this version of this document shall ignore NNPFC SEI messages with nnpfc_auxiliary_inp_idc between 3 and 255. Values ​​of nnpfc_auxiliary_inp_idc greater than 255 do not exist in the bitstream conforming to this version of this document and are not reserved for future use.

[0109] The NNPFC SEI also specifies NNPF properties using nnpfc_property_present_flag, which includes changes in input resolution and frame rate, and therefore, these can be signaled within the GFV SEI using the syntax within the NNPFC SEI instead. Alternatively, GFV SEI can be implemented without relying on NNPFC SEI. In that case, the syntax and semantics related to nnpfc_mode_idc can be incorporated into GFV SEI, as shown in Table 23. [Table 23]

[0110] Embodiment 3: As part of neural network post-processing, GV SEI messaging is reused in addition to picturerate upsampling.

[0111] As described in this application, the generated face video can be considered a form of picturerate upsampling, but requires further features such as keypoints, region matrices, etc. In one example, POC0 and POC30 may be key pictures, while POC1, POC2, ... and POC29 may be marked as interpolation pictures using the generated video. GFV SEI messages need to be associated with POC1, POC2, ... and POC29. Since there are no primary coded pictures associated with the interpolation pictures, it is proposed to add a syntax element gfv_poc_delta to each GFV SEI message to associate the timing with those GFV SEIs. The POC for a generated video using one GFV SEI can be calculated as base_POC + gfv_poc_delta, where base_POC is the picture order count (POC) of the currently decoded picture.

[0112] Note: GFV video-related parameters are described herein with respect to SEI messaging; however, such parameter representations may be readily adapted to alternative metadata and file formats suitable for alternative video codecs such as AV1, AVS, etc.

[0113] [Other applications of GFV SEI messaging] As explained above, GFV SEI messages (see, for example, Table 15 or Table 16) carry two types (or parts) of information. 1) Codified spatial and temporal metadata for identifying and representing major parts of the human body 2) Information identifying the driving picture and corresponding properties within the coded video.

[0114] Regarding communication using generated video, both parts #1 and #2 are used, but the information transmitted in part #1 may be used alone for other purposes such as machine analysis (e.g., athlete tracking, lip-sync monitoring, meeting participant engagement, etc.) and / or for improving coded video (e.g., facial recognition super-resolution, frame rate upsampling, etc.).

[0115] The information transmitted in part #1 may be signaled by a separate SEI, for example, a "human body feature information" (HBFI) SEI message, or it may be signaled in a GFV SEI message using a conditional flag so that the driving picture information (part #2) does not need to be signaled. Another alternative is to signal part #2 with a standalone SEI message, for example, a "generative driving picture information" (GDPI) SEI message, which can be used in conjunction with an HBFI SEI message to achieve the full functionality of a GFV SEI message.

[0116] For example, the syntax and semantics of an HBFI SEI message can be as described above, starting with the "Facial Features" section and including the syntax elements listed in Tables 7-13 (i.e., parts of the document related to coding such as landmarks and key points). Details of two exemplary applications are described below.

[0117] [Human Body Feature Information (HBFI) SEI Message for Machine Analysis] Figure 5 illustrates an exemplary process for facilitating machine analysis using GFV-related messaging according to an exemplary embodiment. As shown in Figure 5, in such a scenario, feature extraction and coding are performed on the encoder side to reduce processing requirements at the receiver. The decoded bitstream can also be displayed, but this is not required, and the extracted HBFI SEI messages can be fed directly into a machine analysis block to generate relevant information such as viewer attention detection (e.g., while driving), lip-sync monitoring, security analysis, etc.

[0118] [Human Body Feature Information (HBFI) SEI message for video enhancement] Figure 6 shows an exemplary process using GFV-related messaging to facilitate video improvement or enhancement according to another embodiment. GFV-related messaging can be applied to improve image quality or enhance picture characteristics such as frame rate, spatial resolution, depth of field, and auto-panning and scanning (e.g., tracking individual faces of athletes). In one embodiment, the decoder may also include neural-network post filtering (NNPF), which is signaled by NNPC SEI messages and activated by NNPFA SEI messages. In NNPF-related use cases, the information transmitted by HBFI SEI messages can be considered auxiliary information, as described in the paragraph "Aspect 2" following Table 22.

[0119] [Considerations for SEI messaging implementation] During the JVET meeting in January 2024, a proposal for SEI messages for generated facial videos was adopted as a technology under consideration (references [28-29]). In that proposal, gfv_drive_pic_fusion_flag is defined as follows: If present, a gfv_drive_pic_fusion_flag equal to 1 indicates that the currently decoded picture corresponding to a driving picture that may be used for fusion may be input to GenerativeNN(). A gfv_drive_pic_fusion_flag equal to 0 indicates that the currently decoded picture should not be input to GenerativeNN(). The value gfv_drive_pic_fusion_flag in Note 3-1 can be used, for example, to indicate that the currently decoded picture can be used to improve facial details or to process background changes. Note 4 - The fusion takes three inputs, namely the base picture, features from keypoints and / or matrices carried in GFV SEI messages, and the currently decoded picture, and outputs the picture. Note 5 - When the currently decoded picture corresponds to a driving picture, it should be marked as not intended for output.

[0120] The purpose of this flag is similar to setting gfv_drive_pic_idc, as defined above in Table 5, to 1. The problem with Note 5 is that there is no clear definition in the current VSEI specification (reference

[27] ) on how to comply with this note. In other words, given a fused driving picture, it is then necessary to decode the primary coded picture so that it can be fused together, which means that in the codec (e.g., VVC), ph_pic_output_flag=1 must be set or inferred for that picture, i.e., it must be output from the decoder but not displayed on the screen. It does not need to be displayed. This means that this is not currently supported by the current VSEI specification.

[0121] In HEVC (H.265), the "no_display" SEI message is defined as follows:

number

[0122] Introducing the same SEI message to VSEI should resolve the issue of exhibiting the behavior described in Note 5, and further, it suggests that Note 5 should be replaced with the following requirements. "When the currently decrypted picture corresponds to a driving picture, a 'no_display' SEI message should be sent for the associated picture."

[0123] In one embodiment, the "no_display" message may be nested within another SEI message, for example, as follows:

number

number

[0124] In another embodiment, assuming the diffusion of possible post-processing operations (e.g., generative fusion, auxiliary input to neural networks, pattern matching for copyright, source, or authentication) even when the decoded frame is not displayed, alternative versions of the no_display SEI message may be represented as shown in Table 24. [Table 24] The syntax parameter no_display_idc indicates that the picture may currently be available for post-processing, as specified in Table 25, for example. [Table 25]

[0125] Alternatively, for simplicity, the syntax parameter no_display_idc can be replaced by the no_display_flag flag, which has possible values ​​of 0 and 1 corresponding to the values ​​of 0 and 1 in Table 25.

[0126] Reference

[30] indicates that the syntax element gfv_drive_pic_fusion_flag, if present and equal to 1, may be input to a face picture generator neural network, indicated as GenerativeNN(), to improve the background texture and face details.

[0127] In some cases, a driving picture is intended solely as input to GenerativeNN() and not to be displayed. For example, a driving picture representing only a background texture may be input to GenerativeNN() for fusion with the generated face. In such scenarios, it is helpful to indicate that the unfused background texture should not be displayed directly.

[0128] In one embodiment, instead of sending a separate no-display SEI message, the no-display flag may be part of the syntax elements of the generative_face_video() message, as discussed above. For example, as shown in Table 26, the no-display message may be indicated by gfv_drive_pic_no_display_flag, which is defined as follows: If present, a gfv_drive_pic_no_display_flag equal to 1 indicates that the currently decoded picture corresponding to the driving picture should not be displayed. A gfv_drive_pic_no_display_flag equal to 0 indicates that the currently decoded picture may be displayed. This indicates that. For example, a gfv_drive_pic_no_display_flag value of 1 can be used to indicate that the currently decoded picture should only be used as input to GenerativeNN(). [Table 26]

[0129] From reference

[30] , the remaining syntax parameters in Table 26 are given by: A gfv_base_pic_flag equal to 1 indicates that the current decoded output picture corresponds to the base picture. A gfv_base_pic_flag equal to 0 indicates that the current decoded output picture does not correspond to the base picture, or that this SEI message does not specify a syntax element for the base picture. If gfv_base_pic_flag is not present, it is presumed to be equal to 0. If present, a gfv_drive_pic_fusion_flag equal to 1 indicates that the current decoded picture corresponding to a driving picture that may be used for fusion may be input to GenerativeNN(). A gfv_drive_pic_fusion_flag equal to 0 indicates that the current decoded picture should not be input to GenerativeNN(). Note-1: The gfv_drive_pic_fusion_flag value can be used, for example, to indicate that the currently decoded picture can be used to improve facial details or to process background changes. Note - The fusion takes three inputs: a base picture, features from keypoints and / or matrices carried in GFV SEI messages, and the currently decoded picture, and outputs a picture.

[0130] [References] Each of these references is included in its entirety by reference. JVET refers to the Joint Video Experts Team, ITU-T SG 16 WP3 and ISO / IEC JTC 1 / SC 29. [1]A. Siarohin, et al., “First order motion model for image animation,” Advances in Neural Information Processing Systems, 2019, 32 [2]A. Siarohin, et al., “Motion representations for articulated animation,” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 13653-13662, arXiv:2104.11280v1, 22 Apr. 2021 [3]TC. Wang, et al., “One-shot free-view neural talking-head synthesis for video conferencing,” Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 10039-10049, arXiv:2011.15126v3, 2 April 2021 [4]B. Chen, et al. “Beyond keypoint coding: Temporal evolution inference with compact feature representation for talking face video compression,” 2022 Data Compression Conference (DCC). IEEE, 2022: 13-22 [5]B. Chen, et al. “Interactive face video coding: a generative compression Framework,” arXiv:2302.09919v1, 20 Feb. 2023 [6]D. Feng, et al. “A generative compression framework for low bandwidth video conference,” 2021 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). IEEE, 2021: 1-6 [7]M. Oquab, et al., “Low Bandwidth video-chat compression using deep generative models,” CVPRW 2021 [8]A. Tang, et al., “Generative compression for face video: a hybrid scheme,” ICME 2022, arXiv:2204.10055v2, 26 Apr. 2022 [9]G. Konuko, et al., “Ultra-low bitrate video conferencing using deep image animation, IEEE ICASSP 2021, arXiv:2012.00346v1, 1 Dec. 2020

[10] G. Konuko, et al., “A hybrid deep animation codec for low bitrate video conferencing,” IEEE ICIP 2022

[11] M. Agarwal, et al., “Compressing video calls using synthetic talking heads,” BMVC 2022

[12] M. Agarwal, et al., “Audio-visual face reenactment,” WACV 2023, arXiv:2210.02755v1, 6 Oct., 2022

[13] K.R. Prajwal et al., “A lip sync expert is all you need for speech to lip generation in the wild,” ACM MM 2020, arXiv:2008.10010v1, 23 Aug. 2020

[14] Y. Zhou, et al., “MakeitTalk: Speaker-aware talking head animation,” SIGGRAPH Asia 2020, arXiv:2004.12992v3, 25 Feb. 2021

[15] S. B. Hegde, et al., “Extreme-scale talking-face video upsampling with audio-visual priors,” ACM MM 2022, arXiv: 2208.08118v1, 17 Aug. 2022

[16] Y. Guo, et al., “AD_NeRF: Audio driven neural radiance fields for talking head synthesis,” ICCV 2021, arXiv:2103.11078v3, 19 Aug. 2021

[17] X. Ji, et al., “Audio-driven emotional video portraits,” CVPR 2021, arXiv:2104.07452v2, 20 May, 2021

[18] O. Ronneberger, et al. “U-net: Convolutional networks for biomedical image segmentation,” in MICCAI, 2015, arXiv:1505.04597v1, 18 May 2015

[19] J. Johnson, et al., “Perceptual losses for real-time style transfer and super resolution,” in ECCV, 2016, arXiv:1603.08155v1, 27 Mar. 2016

[20] T. Park, et al., “Semantic image synthesis with spatially-adaptive normalization,” in CVPR 2019, arXiv:1903.07291v2, 5 Nov. 2019

[21] J. M. D. Barros, et al., "Fusion of keypoint tracking and facial landmark detection for real-time head pose estimation," 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA, 2018, pp. 2028-2037, doi: 10.1109 / WACV.2018.00224

[22] J. Balle’, et al., “Density modeling of images using a generalized normalization transformation,” in International Conference on Learning Representations, 2016, arXiv:1511.06281v4, 29 Feb. 2016

[23] B. Chen, et al., “AHG9: Generative face video SEI Message,” doc. no. JVET-AC0088, 29th meeting, by teleconference, Jan. 11-20, 2023

[24] B. Chen, et al. “AHG9: Common SEI Message of Generative Face Video,”, doc. no. JVET-AD0051, 30th meeting, Antalya, Turkey, 21-28 April 2023

[25] H.264, “Infrastructure of audiovisual services - coding of moving video,” ITU-T (08 / 2021)

[26] S. McCarthy, et al., “Additional SEI messages for VSEI (Draft 4),” doc. No. JVET-AD2006, 30th meeting, Antalya, Turkey, May 2023

[27] H.274,”Versatile supplemental enhancement information messages for coded video bitstreams,” ITU-T (05 / 2022)

[28] J. Chen, et al., “AHG9 / AHG16: Common text for proposed generative face video SEI message,” doc. No. JVET-AG0203, 33d JVET meeting, by teleconference, 17-26 Jan 2024

[29] S. McCarthy, et al., “Technologies under consideration for future extensions of VSEI (version 3),” output document No. JVET-AG2032, 33d JVET meeting, by teleconference, uploaded March 29, 2024

[30] S. McCarthy, et al., “Technologies under consideration for future extensions of VSEI (version 4),” output document No. JVET-AH2032, 34d JVET meeting, Rennes, FR, uploaded June 5, 2024

[0131] [Implementation of an exemplary computer system] Embodiments of the present invention may be implemented using a device including a computer system, a system composed of electronic circuits and components, a microcontroller, a field programmable gate array (FPGA), or another integrated circuit (IC) device such as a configurable or programmable logic device (PLD), a discrete-time or digital signal processor (DSP), an application specific IC (ASIC), and / or one or more of such systems, devices or components. The computer and / or IC may implement, control or execute instructions related to video coding and metadata using GFV as described herein. The computer and / or IC may calculate any of various parameters or values related to video coding and metadata using the GFV process described herein. Embodiments of images and videos may be implemented in hardware, software, firmware, and various combinations thereof.

[0132] A particular implementation of the present invention includes a computer processor that executes software instructions causing the processor to perform the method of the present invention. For example, one or more processors in a display, encoder, set-top box, transcoder, etc., may perform a method related to video coding and metadata using the GFV process described above by executing software instructions in program memory accessible to the processor. The present invention may be provided in the form of a program product. The program product may include any tangible, non-temporary medium that, when executed by a data processor, carries a set of computer-readable signals containing instructions causing the data processor to perform the method of the present invention. The program product according to the present invention may be in any of the wide variety of tangible forms. The program product may include physical media such as, for example, magnetic data storage media including floppy disks, hard disk drives, optical data storage media including CD-ROMs and DVDs, and electronic data storage media including ROMs and flash RAM. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0133] Where components (e.g., software modules, processors, assemblies, devices, circuits, etc.) are referred to above, unless otherwise indicated, references to such components (including references to “means”) should be interpreted as including any components that perform the function of the described component (e.g., functionally equivalent), including components that are not structurally equivalent to the structure of the disclosure performing the function in the illustrated exemplary embodiments of the present invention.

[0134] [Equivalents, extensions, substitutes, and others] Exemplary embodiments relating to video coding and metadata using the GFV process are described herein. In the above specification, embodiments of the invention are described with reference to numerous specific details, which may vary from implementation to implementation. Therefore, the set of claims issued pursuant to this application, including any subsequent amendments in the particular form in which such claims are issued, is the sole and exclusive indication of what the invention is and what the applicant intends to be the invention. Any definitions expressly provided herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Therefore, no limitations, elements, characteristics, features, advantages, or attributes expressly stated in the claims shall in any way limit the scope of such claims. Accordingly, the specification and drawings should be considered illustrative and not restrictive.

Claims

1. A method for generating a video bitstream using generated face video (GFV), Steps include receiving a sequence of video pictures, The steps include generating a face video picture based on the sequence of video pictures, A step of generating facial features for the aforementioned facial video picture, The steps include generating metadata representing the facial features, The steps include: encoding the face video picture using a video encoder to generate a coded face video picture; The steps include multiplexing the coded face video picture and the metadata to generate a GFV coded picture, and A method that includes this.

2. The method according to claim 1, further comprising the step of applying spatial and / or temporal downsampling to a sequence of video pictures before generating the face video pictures and the facial features.

3. The method according to claim 1, wherein the face video picture includes one of a keyframe picture, a driving frame picture, or a dummy frame picture.

4. The steps include applying time downsampling to the sequence of video pictures before generating the face video picture and the face features for the face video picture, The steps include: applying spatial downsampling before encoding the face video picture and generating the coded face picture; The method according to claim 1, further comprising:

5. A method for generating a picture decoded from a generated face video (GFV) coded bitstream, The steps include extracting a GFV-coded picture from the GFV-coded bitstream, The steps include generating a decoded face video picture and metadata representing the facial features within the decoded face video picture from the GFV-coded picture, A step of generating an output face picture based on the decoded face video image and the metadata. A method that includes this.

6. The steps include generating an intermediate face picture based on the decoded face video picture and the metadata, The steps include: applying spatial and / or temporal upsampling to the intermediate face picture to generate the output face picture; The method according to claim 5, further comprising:

7. The steps include applying spatial upsampling to the decoded face video picture to generate an upsampled face video picture, The steps include generating an intermediate face picture based on the upsampled face video picture and the metadata, The steps include applying time upsampling to the intermediate face picture to generate the output face picture, and The method according to claim 5, further comprising:

8. The method according to any one of claims 1 to 7, wherein the metadata includes supplemental enhancement information (SEI) messaging via GFV SEI messages.

9. The method according to claim 8, wherein the GFV SEI message includes syntax elements describing a facial feature and at least one of the following: the presence of one or more faces, spatial sampling, temporal sampling, primary code picture characteristics and driving picture processing, background processing, SEI persistence, and compression parameters for the facial feature.

10. The method according to claim 8, wherein the GFV SEI message further includes parameters representing a neural network post-filtering message.

11. The method according to claim 9, wherein the presence of one or more faces is described by a syntax parameter specifying how many faces are described in the GFV SEI message, and each described face is associated with a face ID.

12. The method according to claim 9, wherein the spatial sampling is described by syntax parameters specifying the width and height of an input picture to the GFV network and / or the width and height of an output picture to the GFV network.

13. The aforementioned time sampling is Syntax parameter indicating whether time sampling is enabled or not. Syntax parameter indicating the maximum number of time sublayers containing GFV SEI messages. The GFV IRAP flag indicates that the primary coded picture is an intracoded picture, or GFV key flag indicating that the primary coded picture is a key picture. The method according to claim 9, described by parameters including one or more of the following.

14. The aforementioned primary code picture characteristics and driving picture processing are as follows: Syntax parameter indicating whether the driving picture contains a dummy picture or a picture that is intended to be merged with or not merged with the generated picture. A flag indicating whether advanced processing of the driving picture is required. A flag indicating that the generated driving picture can function as a reference picture for face generation for subsequent pictures in the decoded order. A flag indicating that face generation uses one or two reference pictures, or The delta value indicates the picture order count difference between the current picture and the referenced picture. The method according to claim 9, described by parameters including one or more of the following.

15. The method according to claim 9, wherein the background processing is described by a parameter including a flag indicating whether or not there is syntax related to background features corresponding to the generated face video.

16. The method according to claim 9, wherein the persistence of the SEI is described by a parameter including a flag indicating that a parameter related to the GFV SEI persists in more than one picture.

17. The method according to claim 9, wherein the facial features are described by parameters that describe one or more of the following: keypoints, region matrices, 3D head transformation matrices, compact features, facial semantics, or background features.

18. The aforementioned key points are, Keypoint quantization coefficients for improving the coding efficiency of the aforementioned keypoints, A flag indicating whether or not a key point is a facial landmark. A flag indicating whether the aforementioned keypoint is represented in 2D space or 3D space, A flag indicating whether or not a local matrix is ​​associated. The remaining coordinates for coding the keypoint using differential coding, or Indexes related to coding the aforementioned keypoints using differential coding. The method according to claim 17, described by parameters including one or more of the following.

19. The aforementioned region matrix is, area matrix ID, A flag indicating whether the region matrix is ​​represented in 2D space or 3D space. A variable indicating the total number of region matrices. Array rmID for two or more region matrices, or Coefficient array for each of the aforementioned region matrices The method according to claim 17, described by parameters including one or more of the following.

20. The aforementioned 3D head transformation matrix is: The quantization coefficient for the 3D head transformation matrix, or Coefficient parameters for the 3D head transformation matrix The method according to claim 17, described by parameters including one or more of the following.

21. The aforementioned compact features are, Variables indicating the dimension of compact features, The array of coefficients for the compact feature and The method according to claim 17, described by parameters including the following:

22. The aforementioned facial semantics are, A flag indicating whether the aforementioned facial semantics are related to a facial motion coding system (FACS), The number representing the total FACS semantics. An array of FACS parameters, or Array of FACS coefficients for the FACS parameters The method according to claim 17, described by parameters including one or more of the following.

23. The method according to claim 17, wherein the background features are described by a parameter including an array of background coefficients representing a 2x3 transformation matrix of the background.

24. The method according to claim 17, wherein support for specific facial features is indicated via bitmasking.

25. The above output face picture is generated by The method according to claim 5, further comprising applying the metadata and the decoded face video picture to a neural network to generate the output face picture, wherein the neural network is determined from the neural network post-filtering metadata received in the GFV coded bitstream.

26. The method according to claim 5, further comprising the step of performing machine analysis of the facial features based on the metadata received in the GFV-coded bitstream.

27. The method according to any one of claims 1 to 26, wherein the neural network model for generating video is communicated to the decoder via neural network post-filter characteristic (NNPFC) SEI messages and / or neural network post-filter activation (NNPFA) messages.

28. The method according to claim 27, wherein the NNPFC message includes an NNPFC purpose syntax element for indicating the use of generated video.

29. The method according to claim 6, wherein time upsampling is performed using neural network-based post-processing.

30. The method according to any one of claims 1 to 26, wherein the encoded or decoded picture includes a human face, a human body, an animal, or an animated video.

31. The method according to claim 27, wherein processing facial features is replaced by processing human body features or vehicle features.

32. The method according to claim 14, wherein if syntax parameters indicate that the driving picture should not be displayed, a hide message for the driving picture is further transmitted to the decoder via metadata.

33. The method according to claim 32, wherein the hidden message includes a syntax parameter indicating whether the driving picture is a) not intended for post-processing use or b) available for post-processing use.

34. The method according to claim 32, wherein the hidden message is a separate SEI message from the GFV SEI message.

35. The method according to claim 32, wherein the syntax parameter indicating that a driving picture should not be displayed is part of the GFV SEI message.

36. An apparatus including a processor, configured to perform the method described in any one of claims 1 to 27.

37. A non-temporary computer-readable storage medium storing computer-executable instructions for performing the method according to any one of claims 1 to 27 using one or more processors.

Images