Rigid mesh linking for scene description

Abstract

Systems and methods for encoding and decoding a mesh animation. In an example method, information is obtained defining a reference mesh and a dependent mesh. The dependent mesh includes at least three binding vertices. Correspondence information is obtained that can be used to determine, for each of the binding vertices of the dependent mesh in a reference pose, a respective position relative to the reference mesh. For each frame in an animation, an animated pose of the reference mesh is determined, and animated positions of the binding vertices are determined based on the animated pose of the reference mesh and on the correspondence information. A transformation is determined that substantially transforms the binding vertices of the dependent mesh in a reference pose to the respective first animated positions of the binding vertices, and an animated pose of the dependent mesh is determined by applying the first transformation to all the vertices of the dependent mesh.

Description

RIGID MESH LINKING FOR SCENE DESCRIPTION CROSS-REFERENCE

[0001] This application claims the priority of European Patent Application No. 23306625.7, filed 28 September 2023, entitled “Rigid Mesh Linking For Scene Description,” which is incorporated herein by reference in its entirety. BACKGROUND

[0002] The domain of this disclosure is the representation of digital humans and their interaction with 3D virtual environments. Specifically, the disclosure deals with the animation of avatars. The present document is also understood in the context of the formatting and the playing of augmented reality and extended reality applications when rendered on end-user devices such as mobile devices or Head-Mounted Displays (HMD). 3D scenes may by represented by a runtime asset delivery file such as a glTF file. A runtime asset delivery file can provide information on a 3D scene, including the geometry and organization of objects in the scene and information used to animate those objects. Scene graphs (such as the one proposed by Khronos / glTF and its extensions defined in MPEG Scene Description format or Apple / USDZ for instance) are a possible way to represent the content to be rendered. They combine a declarative description of the scene structure linking real-environment objects and virtual objects on one hand, and binary representations of the virtual content on the other hand. SUMMARY

[0003] A method according to some embodiments includes: obtaining information defining a reference mesh and a dependent mesh in a reference pose, the dependent mesh in a reference pose including at least three binding vertices; obtaining correspondence information usable for determining, for each of the binding vertices, a respective position relative to the reference mesh; determining a first animated pose of the reference mesh in at least a first frame of an animation; determining first animated positions of the binding vertices based on the first animated pose of the reference mesh and the correspondence information; determining a first transformation, the first transformation being a transformation that substantially transforms the binding vertices of the dependent mesh in a reference pose to the respective first animated positions of the binding vertices; and determining a first animatedpose of the dependent mesh by applying the first transformation to all the vertices of the dependent mesh.

[0004] Such embodiments may further include rendering the first frame of the animation using the first animated pose of the reference mesh and the first animated pose of the dependent mesh.

[0005] Some embodiments further include: determining a second animated pose of the reference mesh in at least a second frame of the animation; determining second animated positions of the binding vertices based on the second animated pose of the reference mesh and the correspondence information; determining a second transformation, the second transformation being a transformation that substantially transforms the binding vertices of the dependent mesh in a reference pose to the respective second animated positions of the binding vertices; and determining a second animated pose of the dependent mesh by applying the first transformation to all the vertices of the dependent mesh. Such embodiments may further include rendering the second frame of the animation including the second animated pose of the reference mesh and the second animated pose of the dependent mesh. An analogous process may be performed for additional frames (and even for all frames) of an animation, and many or all of those frames may use the same correspondence information.

[0006] In some embodiments, the correspondence information identifies, for each of the binding vertices, a respective nearest vertex of the reference mesh. In some embodiments, the correspondence information identifies, for each of the binding vertices, a respective position on a face of the reference mesh. In some embodiments, the correspondence information further includes distance information indicating a distance of each respective binding vertex from the reference mesh.

[0007] A method according to some embodiments includes: selecting at least three vertices of a dependent mesh as binding vertices; selecting a pose of the dependent mesh relative to a reference mesh; and encoding correspondence information indicating, for each of the binding vertices, a position of the respective binding vertex relative to the reference mesh while the dependent mesh is in the selected pose.

[0008] In some such embodiments, fewer than all vertices on the dependent mesh are selected as binding vertices. In some embodiments, only three vertices of the dependent mesh are selected as binding vertices.

[0009] Some embodiments include encoding an animation of a scene including the reference mesh and the dependent mesh.

[0010] Additional embodiments include an apparatus comprising one or more processors configured to perform any of the methods described herein.

[0011] Additional embodiments include an apparatus comprising at least one processor and a computer-readable medium (which may be non-transitory) storing instructions for performing any of the methods described herein.

[0012] Additional embodiments include a computer-readable medium storing instructions for performing any of the methods described herein.

[0013] Additional embodiments include a computer-readable medium storing a scene encoded according to any of the methods described herein.

[0014] Additional embodiments include a signal conveying a scene encoded according to any of the methods disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG.1 illustrates a skeleton representation of an avatar, that can be used to drive its animation.

[0016] FIG.2 schematically illustrates an example avatar mesh geometry that may be used in some embodiments.

[0017] FIG. 3 illustrates an example of mesh registration through vertex-to-vertex correspondence. Vertex V’6on M’ maps to position P6on M. The correspondence of V’6on M is set to the closest vertex to P6on M, which is V5.

[0018] FIG.4 illustrates mesh registration through vertex-to-face correspondence. Vertex V’6on M’ maps to position P6on M. The correspondence of V’6on M is a parametric model of the location of P6on M, in the form of barycentric coordinates (u6, v6, w6) of P6in triangle (V5V7V9).

[0019] FIG. 5 illustrates an extension of vertex-to-vertex correspondence to non-coinciding vertex pairs. Vertex V’Ion dependent mesh M’ lies on the normal (Li) to reference mesh M at point Pithat coincides with vertex Aiof the reference mesh. (Li) can be computed as the normalized average of unit normals of the triangles sharing Aias a vertex that are represented on the figure.

[0020] FIG. 6 illustrates an extension of vertex-to-face correspondence to non-coinciding vertex pairs. Vertex V’Ion dependent mesh M’ lies on the normal (Li) to triangle Ti= (Ai, Bi, Ci)of reference mesh M at point Pi. The location of Piis referenced by the index tiof triangle Tiin the reference mesh and the barycentric coordinates (ui, vi, wi) on Ti.

[0021] FIG.7 illustrates a hat mesh tied to an avatar head mesh during animation.

[0022] FIG.8 illustrates a hat mesh in a “reference pose” and in an ”animation pose”.

[0023] FIG. 9 is a flow diagram illustrating a method of parsing of the “MPEG_mesh_rigid_linking” extension.

[0024] FIG. 10 is a flow diagram of an example processing model for rigid mesh animation based on mesh correspondence data.

[0025] FIG.11 illustrates three-point correspondence on a rigid object.

[0026] FIG. 12 illustrates three-point correspondences after rotation of the reference pose plane to the animation pose plane.

[0027] FIG.13 is a functional block diagram of a system that may be used to implement some embodiments. DETAILED DESCRIPTION Overview of Avatar Animation

[0028] The animation of an avatar body is conventionally driven by the motion of a skeleton. The skeleton is a directed acyclic graph of joint nodes linked by bone edges. The joints are often mapped to physical human body joints such as elbow, ankle, and wrist, as seen in FIG. 1. Note that the naming of joints may differ across different skeleton representations. The root joint of the graph is usually chosen to be the pelvis, named “hips” on FIG.1. The translation and rotation of each joint with respect to its parent joint unambiguously defines the posture of the skeleton. FIG.1 illustrates a skeleton representation that could be used for instance in the MPEG-I-SD avatar model, from “[SD] Update of Annex to ISO / IEC 23090-14:2022 – MPEG Reference Humanoid Avatar”, MPEG meeting #142 input document m62956, April 2023.

[0029] The geometry of the envelope of the avatar body is represented by a textured mesh. An example of an avatar mesh geometry is shown schematically in FIG.2. The animation of the envelope mesh is controlled by the motion of the skeleton joints through a process known as skinning. In this process, the position of each vertex of the mesh is bound to the positions of a pre-determined set of influence joints associated with the vertex, through a pre- determined binding matrix. For a given pose of the skeleton, the positions of the mesh vertices are computed from the joint positions through the binding matrices and provide the geometryof the avatar envelope. After texturing and lighting, the image of the avatar in a picture is obtained by rendering the textured mesh from a given camera viewpoint.

[0030] While FIG.2 illustrates a generic avatar mesh model, some embodiments use one or more of the topologies of avatar mesh models provided as informative references for the MPEG-I Scene Description standard, which describes different avatar mesh models with various different levels of detail (e.g. high, medium and low levels of detail), from “[SD] Update of Annex to ISO / IEC 23090-14:2022 – MPEG Reference Humanoid Avatar”, MPEG meeting #142 input document m62956, April 2023.

[0031] Morph targets provide another way to animate an avatar mesh, and particularly to sculpt facial expressions. They may be described as 3D deformations with respect to the geometry of a base mesh, represented by the 3D offsets of each mesh vertex. Typically, the base mesh is a model of the avatar with a neutral face expression that shows no emotion. Morph targets are provided in addition to the base mesh. Each morph target is associated to a scalar weight. The desired shape of the mesh is obtained by adding, to the vertex positions of the base mesh, a linear combination of several morph target offsets weighted by their corresponding weights. In a typical use case, morph targets are mapped to deformations of the neutral face geometry incurred by the activation of a group of facial muscles. Combining these deformations provides a convenient way for artists to sculpt the shape of the face mesh to reproduce a desired facial expression. Overview of Mesh Correspondence

[0032] During the animation of an avatar, it may be desirable to bind an object, whose geometry is represented by a 3D mesh, to the mesh of the avatar, as the avatar moves. To this purpose, it is advantageous to bind the positions of some of the object mesh vertices to corresponding positions on the surface of the mesh of the avatar. This binding information defines a correspondence between the two meshes.

[0033] To illustrate mesh correspondence, consider a reference mesh M with vertices {Vi} and a dependent mesh M’ with vertices {V’i}. The correspondence on M of some vertex V’iof M’ is defined as the location Pion M that corresponds to V’i. For instance, if M is the mesh of an avatar and M’ the mesh of an object to be bound to the avatar, such as a hat, V’icould represent a vertex of the mesh of the hat in contact with the head of the avatar, and Piwould be the location on the surface of the avatar mesh that coincides with V’iwhen the hat is tied to the avatar head.

[0034] For triangular meshes there are two standard ways to represent locations Piwith respect to the geometry of M.

[0035] A first option is to represent Piby the vertex of the reference mesh M that is closest to Pion the surface of M. This representation called vertex-to-vertex correspondence is illustrated on FIG. 3. FIG. 3 provides an illustration of mesh registration through vertex-to- vertex correspondence. Vertex V’6on M’ maps to position P6on M. The correspondence of V’6on M is set to the closest vertex to P6on M, which is V5.

[0036] Vertex V’6of M’ has corresponding point P6on the reference mesh M. The closest vertex of M to P6is V5. Thus, the location on M corresponding to V’6is V6V_to_V= V5. The vertex- to-vertex correspondence data for the registration of dependent mesh M’ to reference mesh M are represented by the set of {ViV_to_V} for each vertex V’I of the dependent mesh M’.

[0037] The second representation is a set of vertex-to-face correspondences represented by barycentric coordinates, illustrated on FIG. 4. FIG. 4 provides an illustration of mesh registration through vertex-to-face correspondence. Vertex V’6on M’ maps to position P6on M. The correspondence of V’6on M is a parametric model of the location of P6on M, in the form of barycentric coordinates (u6, v6, w6) of P6in triangle (V5V7V9). The corresponding point on the reference mesh M to vertex V’i= V’6of M’ is Pi= P6. The triangle Ti= (Ai, Bi, Ci) of M in which P6lies is determined as T6= (V5, V7, V9). Thus, Ai= V5, Bi= V7and Ci= V9. Let Sibe the area of Ti, and Ui(respectively Vi, Wi) be the area of the triangles formed by Piand the two vertices of Tiopposite to its first (respectively second, third) vertex. On FIG.4, Tiis (V5, V7, V9), Pi= P6, hence Uiis the area of triangle (P6, V7, V9), Viis the area of triangle (V5, P6, V9), and Wiis the area of triangle (V5, V7, P6). Further, let ui= Ui / Si, vi= Vi / Siand wi= Wi / Si. (ui, vi, wi) are the barycentric coordinates of Piin Ti. The position of Piinside Tiis obtained from its barycentric coordinates (ui, vi, wi) as: ^^^^ ൌ ^^^ ^^^ ^ ^^^ ^^^ ^ ^^^ ^^^^^ ^ ^^ ^ ^ (1)^ ^ ^^ ൌ 1up coordinates correspondence data for each vertex V'iof M’ are represented by the set of {(ti, ui, vi, wi)}. Each (ti, ui, vi, wi) unambiguously determines the position of Pi, the corresponding point to V'ion mesh M.

[0039] Mesh correspondences can be extended to the case where the positions of a vertex on the reference mesh and its corresponding vertex on the dependent mesh do not coincide. In this case, the position of the corresponding vertex V’ion the dependent mesh can be referenced as a point on the normal (Li) to the reference mesh at a projection point Pion the reference mesh. This projection point Piis described as explained above using vertex-to- vertex correspondence (as in FIG. 3) or vertex-to-face correspondence (as in FIG. 4). Thelocation of the correspondence on the normal to the reference mesh at the projection point may be described by a signed distance dito the projection point. The sign indicates whether the correspondence point is located on the outward or inward normal to the mesh. A proposed convention for the sign is that if the triangle vertices are ordered as (Ai, Bi, Ci), the direction ofpositive distances is defined by the normal vector^^^^^^ప^^^^^^ప⃗⋀^^^^^^ప^^^^^^ప⃗. Alternatively stated, letting^ ^^^^⃗^ ^^^^ഢ^^^^^^ഢ⃗ ⋀ ^ప ൌ be the unit normal to thedi is defined as ^^^ ≜ ^^^^^ప⃗ .^^^^^ప^^^^^^′^ప⃗. (3)

[0040] This extension is illustrated on FIG.5 in the case where Piis defined using vertex-to- vertex correspondence as one of the vertices of the reference mesh, Aion the figure. FIG.5 illustrates an extension of vertex-to-vertex correspondence to non-coinciding vertex pairs. Vertex V’I on dependent mesh M’ lies on the normal (Li) to reference mesh M at point Pi that coincides with vertex Aiof the reference mesh. (Li) can be computed as the average of unit normals of the triangles sharing Aias a vertex that are represented on the figure. As an example, for triangle (Ai, Bi, Ci) the unit normal vector is defined by equation (2). On FIG.5, the unit normal at Aiwould be defined as the normalized average of the unit normals to triangles (Ai, Bi, Ci), (Ai, Ci, Di), (Ai, Di, Ei) and (Ai, Ei, Fi).

[0041] FIG. 6 illustrates an extension of vertex-to-face correspondence to non-coinciding vertex pairs. Vertex V’Ion dependent mesh M’ lies on the normal (Li) to triangle Ti= (Ai, Bi, Ci) of reference mesh M at point Pi. The location of Piis referenced by the index tiof triangle Tiin the reference mesh and the barycentric coordinates (ui, vi, wi) on Ti. The specification of the correspondence for a vertex V’Iof the dependent mesh not located on the surface of the reference mesh, in the case of vertex-to-face correspondence, consists of the index tiof triangle Tiin the reference mesh, the barycentric coordinates (ui, vi, wi) and triangle index of the normal projection Piof V’Ion the reference mesh, as well as the signed distance d(Pi, V’I) on the normal to the reference mesh at Pi. Overview of Avatar Mesh Correspondence in MPEG-I Scene Description

[0042] A description scheme for avatars in 3D virtual environments is provided in the MPEG- I Scene Description (SD) standard, described at ISO / IEC DIS 23090-14:2021(E) Information technology – Coded representation of immersive media – Part 14: Scene description for MPEG media: FDIS comments. This builds on the Khronos glTF specification. The “MPEG_node_avatar” extension of MPEG-I SD, described at ISO / IEC 23090-14 CDAM 2: Support for Haptics, Augmented Reality, Avatars, Interactivity, MPEG-I Audio, and Lighting,specifies a reference avatar model with a mesh topology and an animation rig, typically built on skeletal animation, skinning and morph target mesh deformations. The avatar model is referenced by a Uniform Resource Name specified by the type attribute of the ”MPEG_node_avatar” extension.

[0043] However, users may want to animate custom avatars whose mesh topology differs from the reference avatar model. Such custom avatars may be obtained, for instance, by capturing the geometry of a person using a 3D scanning device or reconstructing this geometry based on photographs of the person taken from different viewpoints using photogrammetric reconstruction techniques. In these cases, no animation rig is available for the custom avatar model by default.

[0044] The “MPEG_mesh_linking” extension of the “mesh“ object in MPEG-I SD provides a way to animate the custom avatar using the rig of the reference avatar model. To this end, the mesh of the custom avatar is registered to the mesh of the reference avatar. This is obtained by providing mesh correspondence data that bind each vertex of the custom avatar mesh, referred to as the dependent mesh, to a corresponding location on the surface of the reference avatar mesh, referred to as the shadow mesh. If the custom avatar is to be animated, correspondence data is provided for each frame of the animation.

[0045] Specifically, the “MPEG_mesh_linking” extension holds the following attributes: ^ Amesh index pointing to the shadow mesh object in the glTF “meshes” array, ^ Apose accessor pointing to the transforms of the shadow mesh skeleton joints to be applied to reproduce the desired pose of the dependent mesh in the current frame of the animation, ^ Aweights accessor pointing to the weights to be applied to the morph targets of the shadow mesh to reproduce the desired dependent mesh shape in the current frame, and ^ Acorrespondence accessor pointing to the per-vertex correspondence data that provide the locations of each vertex of the dependent mesh relative to the surface of the shadow mesh after its pose and shape has been transformed by skinning and morph target deformation using the pose and weights attributes.

[0046] The animation transfer at each frame proceeds in two stages: ^ First, the mesh of the reference avatar model is transformed, based on its skinning and morph target attributes and the corresponding pose and weights data, to the desired pose and expression of the custom avatar, ^ Second, the transformed reference avatar mesh is re-topologized to the topology of the custom avatar mesh using the data held by the buffer pointed to by the correspondence attribute. These data allow for computation of the position of each vertex of the custom mesh on the surface of the transformed reference avatar mesh. The mesh obtained after this re-topologization operation has thedesired shape of the transformed reference avatar mesh and the topology of the custom mesh. Issued Addressed in Some Embodiments

[0047] Consider an application relying on the MPEG-I Scene Description (SD) standard that allows users to select a customizable avatar and animate it in a virtual environment. Note that MPEG-I SD supports animations as instances of glTF “Animation” objects. Users may want to enrich their avatars with pieces of apparel such as glasses, headcover, or earrings. To this end, the application could provide a library of such items as 3D textured mesh representations. During the animation of the avatar, the mesh representing an apparel item would then need to be rigidly bound to the mesh of the avatar. Specifically, a rigid 3D geometric transform, defined as the composition of a 3D rotation and a 3D translation, would have to be applied to the mesh of the apparel item so that its position and pose is consistent with the position and pose of the body or head of the avatar at each frame.

[0048] The present disclosure addresses the problem of binding the mesh of a second rigid object, hereafter referred to as the dependent mesh, to the mesh of a first object, hereafter referred to as the reference mesh, when the reference mesh is animated. The geometry of the dependent mesh is provided in a pre-determined “reference pose”. At each frame of the animation, the pose of the dependent mesh is computed so that it is consistently bound to the reference mesh. For instance, if in the first frame the dependent mesh is in contact with the reference mesh at specific locations on this reference mesh, the points of contact approximately remain the same throughout the animation.

[0049] One way to address this problem would be to precompute and store the animation of the dependent mesh for all animations of the reference mesh it is to be bound to. However, because of the large number of possible combinations of reference and dependent meshes, this would involve storing a large volume of data. Besides, during the animation, the precomputed pose of the dependent mesh would have to be transmitted at each frame to render the correct binding. Typically, this precomputed pose is represented by a rigid 3D transform with respect to a reference pose of the dependent object. This rigid 3D transform is the composition of a 3D rotation and a 3D translation. Overview of Example Embodiments

[0050] Example embodiments described herein aim to provide a more efficient alternative. In some embodiments, correspondence data between the reference mesh and the dependent mesh that are independent of any animation are transmitted infrequently (e.g. only once) to the device in charge of reconstructing the animation geometry. From these correspondence data, the device is able to reconstruct the correct pose of the dependent mesh at each frameof any animation of the reference mesh, based on the geometry of the reference mesh at the considered frame.

[0051] Some embodiments provide a representation format for these correspondence data.

[0052] In example embodiments, the geometry of the dependent mesh is provided in a fixed reference pose. The mesh may be a rigid mesh, in which case the only allowable change to the dependent mesh geometry during an animation is a global 3D transform defined as the composition of a global 3D rotation of the mesh and a global 3D translation of the mesh.

[0053] In some embodiments, the dependent mesh is adapted to non-rigid deformations of the reference mesh in the course of animations, which may often be small. These adaptations will translate into slight changes of the relative pose of the dependent mesh with respect to the reference mesh, or slight deformations of the dependent mesh if it is not strictly rigid. An animation where the reference mesh is the head of an avatar, and the dependent mesh represents a cap on the avatar head, illustrates this issue. As the avatar speaks and expresses emotions, the shape of the forehead part of the avatar head will undergo slight non-rigid deformations. As a result, the positioning of the cap on the head will change slightly. For example, if the avatar head expresses surprise, the brow raiser muscles will be activated, raising the upper part of the forehead, while the back of the head remains undeformed. As a result, the cap may be slightly displaced towards the back of the head. Concurrently, in some embodiments, the mesh representing the cap may also slightly stretch to accommodate the small inflation of the avatar head at the level of the forehead.

[0054] In example embodiments, the reference mesh is a triangular mesh.

[0055] In some embodiments, the binding information that binds the dependent mesh to the reference mesh is pre-computed prior to and independently of any animation of the reference mesh and is transmitted only once. This reduces or removes the need for any transmission of pose information for the dependent mesh during any animation of the reference mesh. This results in a reduction of the volume of transmitted data for the animation. It further reduces the transmission bandwidth requirements with respect to an approach where the animations of the dependent mesh would be precomputed as a rigid 3D transform and transmitted at each frame of each animation of the reference mesh.

[0056] In some embodiments, the binding information that binds the rigid dependent mesh to the reference mesh includes correspondence data for three “binding vertices” on the dependent mesh. These three binding vertices may be selected (e.g. by a user) with the constraint that they should not be collinear. The correspondence data provides the relative positions of the binding vertices of the dependent mesh with respect to the triangles on the surface of the reference mesh. In some embodiments, these relative positions are providedfor each of the three binding vertices as a corresponding location on the surface of the reference mesh and a signed distance between the considered vertex and its corresponding location on the reference mesh.

[0057] The processing of the correspondence data whose representation format is proposed in some embodiments can be described as follows.

[0058] Prior to an animation of the reference mesh, the positions of the three correspondence vertices in the provided reference pose of the dependent mesh are retrieved, as well as the correspondence data for the three binding vertices.

[0059] At each frame of an animation of the reference mesh, the pose of the dependent mesh that ensures a proper rigid binding to the reference mesh is computed. In some embodiments, the pose is determined according to the following steps: ^ First, obtain the positions of the mesh vertices of the reference mesh in the current frame. ^ Second, based on the correspondence data for the three binding vertices and on the positions of the vertices of the reference mesh, compute the positions of the three binding vertices of the dependent mesh for the current frame. ^ Third, from the positions of the three binding vertices in the reference pose of the dependent mesh and in the current frame, compute the parameters of the rigid 3D transform that brings the dependent mesh from its reference pose to the animated pose in the current frame. ^ Fourth, apply the rigid transform computed in the third step to all the vertices of the dependent mesh in the reference pose to obtain the animation pose of the dependent mesh for the current frame.

[0060] The rigid 3D transform computed in the third step and applied in the fourth step may only be an approximation of the actual position and shape of the dependent mesh relative to the reference mesh when this latter mesh undergoes non-rigid deformations. However, in typical use cases, for instance when the reference mesh represents the head of a character and the dependent mesh represents facial apparel such as headcover or glasses, the non- rigid deformations of the reference mesh will be very small and the representation of the pose of the dependent mesh relative to the reference mesh by a constant rigid 3D transform throughout an animation will provide a plausible approximation of reality. Thus, following these steps, the dependent mesh will remain consistently bound to the reference mesh throughout the animation, in a way that is consistent with the user specification of the locations of the three binding vertices of the dependent mesh relative to the surface of the reference mesh. Representation Format for Mesh Correspondence Data

[0061] This disclosure proposes a representation format for mesh correspondence data between a rigid object and another object. As described above, these correspondence datamay be used to provide an advantageous representation of the data to bind the mesh of the rigid object to another object when this other object is animated. In some embodiments, this representation depends only on the meshes of the rigid object and the other object, regardless of their position and pose variations during the animation. As a result, the correspondence data may be transmitted once, prior to any animation. The availability of these data during an animation of the other object provides enough information to allow for the computation at every frame of the pose of the rigid object mesh that binds it correctly to the mesh of the other object. This reduces the volume of data to be transmitted compared to the alternative solution that consists in pre-computing the animation of the rigid object mesh, which calls for transmitting the pose of this rigid object mesh at every frame of every animation.

[0062] Consistent with the previous sections, in the following description the mesh of the rigid object is named “dependent mesh” and the mesh of the animated object on which the rigid object is to be bound is called the “reference mesh”.

[0063] For the sake of clarity, example embodiments are described herein using a format that is compliant with the MPEG-I Scene Description format, but the principles described herein are not restricted to the use of any particular format. Proposed representations according to embodiments described herein may be encoded in any other scene description format, such as XML or USD.

[0064] The MPEG-I Scene Description already features an “MPEG_mesh_linking” extension for the encoding of mesh correspondence data. However, this description is meant for a specific use case where both meshes represent an avatar. It calls for one of the meshes to be equipped with an animation rig – consisting of a set of blendshapes and a skinning framework, and correspondences are provided for all the vertices of one of the meshes at all frames of an animation. This is different from embodiments disclosed herein, where correspondence data is provided for, e.g., only three vertices of the mesh of a rigid object to another mesh, and do not depend on the pose and position of the meshes during an animation. In addition, no animation rig is required either for the reference mesh or the dependent mesh.

[0065] For clarity, the proposed mesh correspondence data format is described herein as a new extension of the “mesh” glTF object that may be called “MPEG_mesh_rigid_linking”. However, it should be understood that other description options within MPEG-I-SD are possible, such as by modifying the content of the existing “MPEG_mesh_linking” extension to accommodate both the avatar linking and the rigid object linking use cases.Correspondence for rigid object binding

[0066] The rigid object to be bound to the reference mesh is modelled by a dependent mesh in a fixed, pre-determined “reference pose”. During the course of an animation, the dependent mesh will take other poses which we refer to as “animation poses”. For instance, the dependent mesh could model a hat and the reference mesh could model an avatar. In this example, the correspondence data considered in an example embodiment would be used to ensure that the hat takes the appropriate animation pose on the head of the avatar as the avatar moves (see FIG.7 as a schematic example). FIG.8 illustrates the reference pose and an animation pose of the rigid hat mesh in this example.

[0067] The reference mesh is assumed to be a triangular mesh.

[0068] In the proposed representation format, the reference pose of the dependent mesh is described in the glTF “mesh” instance describing the geometry of the rigid object. This instance contains an “MPEG_mesh_rigid_linking” extension.

[0069] In some embodiments, the dependent mesh is rigid. This means that it will not deform during an animation. Alternatively stated, at each frame of an animation the geometry of the dependent mesh will be obtained by applying a rigid 3D transformation to some reference pose of this mesh. In general, a rigid transformation in 3D can be specified by the combination of a rotation, a scaling and a translation. In some example use cases, the scaling component is removed since the object is rigid, hence always retains the same dimensions. Thus, in some predefined 3D coordinate system, the position riof vertex i of the rigid object mesh in the animation pose is computed from its position rref, iin the reference pose as ^^^ൌ ^^൫^^^^^,^൯^ ^^^(4)

[0070] where t0is a 3Dby one or more matrices).

[0071] The values of t0and R can be computed from the knowledge of corresponding positions of three non-collinear vertices of the dependent mesh in the reference pose and the animation pose. The calculation may be made in some embodiments using techniques described in “Closed-form solution of absolute orientation using unit quaternions”, B.K.P. Horn, Journal of the Optical Society of America, Vol.4, pp.629 sqq, April 1987. These three vertices will be referred to hereafter as “binding vertices”. An example of this computation process is provided below. Once t0and R are determined, the positions of all the rigid object mesh vertices in the animation pose can be computed by applying R followed by t0to their positions in the reference pose. The positions of the binding vertices relative to the surface of the reference mesh may be determined in advance, e.g. manually by a user designing the scene. These relative positions can be described by the vertex-to-vertex or vertex-to-facecorrespondence data on the surface of the reference mesh, complemented by a signed distance to the surface of the reference mesh. Note that these descriptions assume that the reference mesh is a triangular mesh.

[0072] Further note that, as mentioned above, modeling the relative pose of the dependent mesh relative to the reference mesh by a constant rigid 3D transformation consisting in the composition of a 3D rotation and a 3D translation is only an approximation of reality if the reference mesh undergoes non-rigid deformations. In general, the non-rigid deformations of the reference mesh will be sufficiently small for this approximation to remain valid. In some embodiments, a scaling component may be included in the 3D rigid transformation model to allow for a small shrinking or expansion of the dependent mesh to better accommodate the deformations of the reference mesh. This scaling component may be expressed as a scalar scaling factor s that can be computed in closed form, together with the 3D translation t0and the 3D rotation matrix R in equation (4), from the knowledge of corresponding positions of three binding vertices of the dependent mesh in the reference and animation poses. In some embodiments, the value of s is calculated using techniques described in Horn, cited above.

[0073] FIG.7 illustrates a hat mesh tied to avatar head mesh during animation

[0074] FIG.8 illustrates a hat mesh in “reference pose” 802 and ”animation pose” 804. Example Scene Description File Extension

[0075] In some embodiments, the dependent mesh in some reference pose is described by a glTF “mesh” instance extended by an “MPEG_mesh_rigid_linking” extension. The choice of the reference pose may be made arbitrarily. The extension contains a pointer to the reference mesh and correspondence information that defines the location of the three non- collinear binding vertices of the dependent mesh relative to the reference mesh. This correspondence information allows for the computation of rigid transforms that bring the dependent mesh from its reference pose to the animation pose where it is properly positioned and rotated with respect to the reference mesh.

[0076] In some embodiments, the ”MPEG_mesh_rigid_linking” extension may be configured as shown in Table 1..Name Type Usage Description correspondence_ indicesinteger[3] M Indices of the binding vertices of the dependent meshcorrespondence_ typeinteger M Type of mesh correspondence data (see Table 2)correspondenceintegerMIndex of the accessor to the correspondence dataIndex in the “meshes” array of reference mesh to meshinteger Mwhich the dependent mesh instance extended by the current “MPEG_mesh_rigid_linking” extension is to be bound Table 1.

[0077] Table 1 illustrates attributes of the proposed “MPEG_mesh_rigid_linking” extension according to some embodiments. In the Usage column, “M” identifies attributes whose inclusion is mandatory according to the syntax of this particular embodiment, and “O” identifies attributes that are optional in this embodiment, but it should be understood that attributes that are mandatory to the syntax of one embodiment are not necessarily mandatory to (or even used in) any other embodiment. correspondence_type value Description correspondence_type = 0Vertex-to-vertex correspondencecorrespondence_type = 1Vertex-to-face correspondenceTable 2.

[0078] Table 2 illustrates semantics of a proposedcorrespondence_type attribute in some embodiments. Name Type Usage Description if (correspondence_type == 0) {vertex-to-vertex correspondenceindex of the closest vertex on the closest_vertex_indicesinteger[3] Mreference mesh for each of the three binding vertices of the dependent mesh } if (correspondence_type == 1) {vertex-to-face correspondence index of the face (triangle) of the reference mesh holding the face_indicesinteger[3]Mcorrespondence point Pifor each of the three binding vertices of the dependent mesh, i in {1, 2, 3} first two barycentric coordinates (ui, vi) barycentric_ float[ Mof the correspondence point Pion the cords6]reference mesh triangle indexed by face_indices[i] for each of thethree binding vertices V’I of the dependent mesh, i in {1, 2, 3}. The third barycentric coordinate is obtained as wi= 1. – ui– vi. } signed distance from Pito V’Ialong the normal to reference mesh M at Pi, signed_distancefloat[3] Ofor each of the three binding vertices of the dependent mesh, i in {1, 2, 3}. Table 3.

[0079] Table 3 illustrates contents of the correspondence buffer in some embodiments. In the Usage column, “M” identifies attributes whose inclusion is mandatory according to the syntax of this particular embodiment, and “O” identifies attributes that are optional in this embodiment, but it should be understood that attributes that are mandatory to the syntax of one embodiment are not necessarily mandatory to (or even used in) any other embodiment. The barycentric_coords and signed_distance attributes are described above in the overview of mesh correspondence.

[0080] The “MPEG_mesh_rigid_linking” extension presented in Table 1 extends the description of the dependent mesh, represented by an MPEG-SD “mesh” object. It defines a representation for the correspondence data between the dependent mesh and a reference mesh pointed to by the “mesh” attribute of the extension.

[0081] These correspondence data are described by the “correspondence“ attribute of the “MPEG_mesh_rigid_linking” extension, which defines an accessor that provides the locations of exactly three binding vertices of the dependent mesh relative to the surface of the reference mesh. The indices of these three binding vertices in the dependent mesh are provided through the “correspondence_indices“ attribute of the extension. The locations described in the “correspondence“ attribute can be represented as vertex-to-vertex or vertex-to-face correspondences, as described above. The choice of representation is specified by the correspondence_type attribute, whose allowed values are provided in Table 2.

[0082] The contents of the correspondence buffer data pointed to by the “correspondence” attribute is described in Table 3. The buffer holds two pieces of information: ^ The parametric representation of the corresponding locations on the reference mesh of the three binding vertices of the dependent mesh. Depending on the value of the “correspondence_type” attribute in the “MPEG_mesh_rigid_linking” extension, this parametric representation is provided for each binding vertex either as the index of the corresponding vertex on the reference mesh(correspondence_type = 0), or as the index of the corresponding face on the reference mesh and the barycentric coordinates of the corresponding point on this face (correspondence_type = 1). ^ Second, the signed distance from the corresponding point to each binding vertex, to the location of the binding vertex, for each of the three binding vertices. The determination of the sign is described above. Example Implementation

[0083] In this example we want to add a hat to an animated avatar. The MPEG-I SD description of the scene features a mesh for the avatar and a mesh for the hat. The hat in this example is a rigid object. When the avatar is animated and moves in its 3D environment, it is desirable for the hat mesh to be attached to the mesh of the avatar head. To do this, three binding vertices of the hat mesh are registered to the avatar mesh through correspondence data provided in the “MPEG_mesh_rigid_linking” extension of the glTF “mesh” object representing the hat mesh. In this example, the correspondence is encoded as vertex-to- vertex correspondences. {   "scene": 0,      "nodes": [      {       "name": "avatar",        "mesh": 0,        "extensions": {         "MPEG_node_avatar": {           "isAvatar": true,            "type": “urn:my_avatar_type”         }       }     },      {        "name": "hat",        "mesh": 1     }   ],   "scenes": [      {       "nodes": [0,1]     }    ],      "meshes": [      {       "name": "full_avatar",      "primitives": [          {           "attributes":              {               "POSITION": 0,               "TEXCOORD_0": 1,             },           "indices": 2         }        ]     },      {       "name": "hat",       "primitives": [          {           "attributes":              {               "POSITION": 3,               "TEXCOORD_0": 4,             },           "indices": 5         }        ],        "extensions": {         "MPEG_mesh_rigid_linking": {            "mesh": 0,            "correspondence_indices": [17,39,87],            "correspondence_type": 0,           "correspondence": 6          }     },    ],   "extensionsUsed": [        "MPEG_node_avatar",       "MPEG_mesh_rigid_linking”    ],      "extensionsRequired": [        "MPEG_node_avatar",       "MPEG_mesh_rigid_linking”    ]  }

[0084] In some embodiments that make use of MPEG-I SD, the correspondence information that binds the dependent mesh to the reference mesh may be encoded in an “MPEG_rigid_mesh_linking” extension to the description of the dependent mesh, as described above.

[0085] The only scene in this description contains two root nodes: an avatar node with the “MPEG_node_avatar” extension and a hat node. Note that the “MPEG_node_avatar” extension is provided for consistency with the presented use case but is not required by example embodiments.

[0086] The avatar node points to mesh #0 in the “meshes” array, named full_avatar and representing the avatar, the hat node points to mesh #1 in the same array, named hat and representing the rigid hat object to be tied to the avatar geometry during the animation. These two meshes hold attributes that describe the positions of their vertices (“POSITION”) and their color through texture coordinates in an UV map (“TEXCOORD_0”). For both meshes, the “indices” attribute holds the triplets of vertex indices for each mesh triangle that define the indexed mesh geometry.

[0087] The hat mesh that is to be bound to the full_avatar mesh has an “MPEG_rigid_mesh_linking” extension. This extension provides the following attributes: ^ “mesh” points to mesh of index 0 in the “meshes” array that represents the mesh of the avatar. ^ “correspondence_indices” holds the indices of the three binding vertices of the hat mesh whose locations on the reference mesh determine the binding between the two meshes. ^ “correspondence_type” defines the nature of the correspondence data that specify the location of the hat vertices on the mesh of the avatar, here 0 for vertex-to- vertex correspondence (see Table 2). ^ “correspondence” is an accessor that points to the vertex-to-vertex correspondence data, i.e., the index of the closest vertex on the avatar mesh, for the three binding vertices of the hat mesh whose indices are specified by the “correspondence_indices” attribute.

[0088] For clarity, the content of the accessors is not exposed in the example scene description. Example Parsing Method

[0089] Parsing of the description in the “MPEG_mesh_rigid_linking” extension may proceed as illustrated in the block diagram of FIG. 9. In a method of parsing a glTF mesh node, a determination is made of whether the node includes an MPEG_rigid_mesh_linking extension. If not, the remaining operations of FIG.9 need not be performed. If there is such an extension, then the correspondence_type, mesh, indices, and correspondence attributes may be parsed. If the correspondence type is 0, then the buffer pointed to by "correspondence" is parsed to provide the vertex indices and, optionally, signeddistance. If the correspondence type is 1, then the buffer pointed to by "correspondence" is parsed to provide the face indices, pairs of barycentric coordinates, and, optionally, signed distance. Example Runtime Processing

[0090] In an example embodiment, at runtime, the Presentation Engine consumes the scene description that includes a “mesh” instance representing the 3D geometry of the reference mesh in the current frame and another “mesh” instance representing the 3D geometry of the dependent mesh in a pre-determined “reference pose”. The latter “mesh” instance representing the dependent mesh contains an “MPEG_mesh_rigid_linking” extension.

[0091] At each frame of the animation, the geometry of the dependent mesh is computed in the “animation pose” where it is bound to the reference mesh. The dependent mesh can then be rendered in the adequate position and pose relative to the geometry of the reference mesh in the current frame. This computation may performed using a method such as the method described below with respect to the block diagram of FIG.10.

[0092] At 110, the “mesh” instance extended by “MPEG_mesh_rigid_linking” extension is parsed to obtain the positions of the vertices of the dependent mesh in the reference pose. This may be performed prior to the computation and rendering of the animation.

[0093] At 120, the data in the buffer pointed to by the “correspondence_indices” accessor of the “MPEG_mesh_rigid_linking” extension of the “mesh” instance are parsed to determine the indices of the three binding vertices of the dependent mesh used to specify the correspondence. The positions of these binding vertices in the reference pose of the dependent mesh are looked up in the dependent mesh data parsed at 110. This may be performed prior to the computation and rendering of the animation.

[0094] At 130, the “correspondence_type” attribute and the correspondence data in the buffer pointer to by the “correspondence” accessor, both in the “MPEG_mesh_rigid_linking” extension of the dependent “mesh”, are parsed. This may be performed prior to the computation and rendering of the animation.

[0095] At 140, the positions of the vertices of the reference mesh in the current frame of the animation are obtained from the attributes of the “mesh” instance of the “MPEG_mesh_rigid_linking” extension and the attributes of the animation for the frame.

[0096] At 150, the absolute positions of the three binding vertices of the dependent mesh in their animation pose are computed for the current frame, based on the absolute positions ofthe vertices of the reference mesh obtained at 140 and the correspondence data parsed at 130.

[0097] At 160, the parameters of the rigid transform that maps the reference pose of the dependent mesh to its animated pose are computed for the current frame. This computation may be based on the positions of the three binding vertices of the dependent mesh in the reference pose, obtained at 110 and 120, and on the corresponding positions of these same three binding vertices in the animation pose, computed at 150 for the current frame. Example calculations that may be used in this process are described in greater detail below.

[0098] At 170, the rigid transformation obtained at 160 is applied to the positions of all the vertices of the dependent mesh in the reference pose parsed at 110, to compute their positions in the animated pose. This determines the full 3D geometry of the dependent mesh in the animated pose, bound to the reference mesh in the current frame. Example Processing for Encoding

[0099] In some embodiments, to encode an animation, at least three (or in some embodiments, exactly three) vertices of a dependent mesh are selected to be used as binding vertices. A pose of the dependent mesh is selected relative to a reference mesh. A user may employ an authoring tool or other technique to move the dependent mesh (e.g. using a rigid transformation) into various different poses relative to the reference mesh until an appropriate pose is selected (for example, orienting a dependent mesh representing a hat on a reference mesh representing a head). The positions of the binding vertices are determined while the dependent mesh is in the selected pose. Correspondence information is determined, where the correspondence information indicates, for each of the binding vertices, a position of the respective binding vertex relative to the reference mesh. Correspondence information does not need to be encoded for all vertices of the dependent mesh and in some embodiments is encoded for only three binding vertices. As described in greater detail above, the correspondence information for a vertex may identify a respective nearest vertex of the reference mesh, a respective position on a face of the reference mesh, and / or a distance of each binding vertex to the surface of the reference mesh.

[0100] Using such embodiments, an animation that includes both the reference mesh and the dependent mesh may be encoded without explicitly coding motion information for the dependent mesh. Instead, the pose of the dependent mesh in each frame may be derived from the binding information (which may remain the same for multiple frames, or even for the entire animation) and the geometry of the reference mesh.Example Calculations for Rigid Object Transforms

[0101] This section provides further detail on calculations involving rigid object transforms that may be performed in some embodiments. However, embodiments disclosed herein are not limited to the use of the precise calculation techniques described below.

[0102] For use in an example calculation, FIG.11 shows two instances of a 3D rigid object, here a hat. The first instance in the “reference pose” provides a reference geometry for the object. In the second instance in an “animated pose” the object has been moved and rotated to another position and attitude, as happens in an animation. The 3D object is represented in a Cartesian coordinate frame Oxyz. Three predetermined points represented by vectors r1R, r2Rand r3Rare given on the surface of the object in the reference pose. These points are assumed not to be collinear. Let r1A, r2Aand r3Abe the corresponding positions of these three points in the animated pose. It can be demonstrated that the rigid transform that maps the position of any point on the object surface in the reference pose to its position in the animated pose can be computed from the corresponding positions of three points in these two poses. The outline of the demonstration is sketched below for completeness.

[0103] First, we introduce the following notations. We define normalized sets of coordinates by offsetting the three points in the reference pose and the animated pose with respect to their centroids ^ത^ோൌ1 ଷ^^^^ , ^ത^^ൌ1 ଷோ^ ^^^^as:and similarly ^ത^^^ൌ ^^^^െ ^ത^^.

[0104] We further define two vectors xiRand xiAin the reference and animated pose, respectively, as ^^^ோൌ ^^ோଶ െ ^^^ோ, ^^ோଶ ൌ ^^ோଷ െ ^^^ோand ^^^^ൌ ^^ଶ^െ ^^^^, ^^ଶ^ൌ ^^ଷ^െ ^^^^.

[0105] The rigid transformation that maps a point on the surface of the object in the reference pose to the corresponding point on the surface of the object in the animated pose can be expressed as: ^^ ^^ ^ ^^ where R is a 3D rotation matrix, t is a 3D translation vector and s is a scaling factor. Since the size of the rigid object does not change during the animation, s is equal to 1 in this example.

[0106] We want to compute R and t by minimizing the sum over the three points of square mapping errors ^^^^ൌ ^^^^– R( ^^ோ^) – t.

[0107] This sum of square mapping errors E can be written as a function of the centered positions as ଷ ^^ ൌ ^ฮ ^ത^^^െ ^^൫ ^̅^^ோ൯ െ ^ ^^ െ ^ത^^^ ^^^ ^ത^ோ^^ฮଶ.

[0108] The determinationas follows. Developing the square norm as a dot product, E can be rewritten as ଷ ^^ ൌ ^ฮ ^ത^^^െ ^^൫ ^̅^^ோ൯ฮଶ

[0109] of the centroid,ଷ ଷ ^^ത^^^ൌ 0, ^ ^ത^^ோൌ 0 , and since the rotation is a linear term is 0. The first term does not depend on t. The minimization ofthe third term, yielding the optimal value t* for t as: ^^∗ൌ ^ത^^െ ^^^^ത^ோ^.

[0110] The determination of the rotation matrix ^^ may be performed as follows. The three points on the surface of the rigid object are non-collinear, so they define a plane in the reference pose and another plane in the animated pose. Let us first determine the rotation R1that brings the former plane to the latter plane.

[0111] The normals to the planes for the reference pose and the animated pose are obtained respectively as ^^ோൌ ^^^ோ∧ ^^ோଶ ^^ ^^ ^^ ^^^ൌ ^^^^∧ ^^ଶ^.

[0112] The line of intersection of the two planes is perpendicular to both of these normals. Thus, its direction is represented by the unit vector ^^ೃ ಲ^^ ൌ∧ ^^ฮ^^ೃ ∧ ^^ಲฮ.

[0113] The angle Φ in the range [0,π] pose plane to the animated poseplane is also the rotation angle between it is given by^^ோ . ^ì ^^ï cos ^^ ൌ^^ோ .^

[0114] Thus, a first rotation R the three object points in reference pose to the plane definedobject points in the animated pose is defined by the unit vector a along its axis and the rotation angle Φ. Given a vector v in 3D space, its rotated version R1(v) can be expressed using Rodrigues’ rotation formula: ^^^^^^^ൌ ^^ cos ^^ ^^^^ ∧ ^^^sin ^^ ^ ^^ .^^^ . ^^^^1 െ cos ^^^

[0115] Using this formula, the vectors ^ത^^ோ, ^ത^ோଶ and ^ത^ோଶ defining the positions of the three points on the surface of the hat in the reference pose can be rotated to their respective positions ^ഥ^′^ோ,^ഥ^′ோଶand ^ഥ^′ோଷin the plane of the corresponding three points in the animated pose (FIG. 11).

[0116] Having determined R1, what remains to be done to compute the rotation matrix R that maps the surface of the rigid object from its reference pose to its animated pose is to find the rotation R2in the animated pose plane that best aligns the ^ഥ^′ோ^ to the ^ത^^^in this plane. Indeed, R = R2.R1. To this end, we want to minimize ଷ ^ฮ ^ത^^തோ ଶ^ െ ^^′^ฮ

[0117] Letting αj be the angle FIG. 12). FIG. 12 illustrates three-pointafter rotation ofplane to the animation pose plane. Using the cosine rule for triangles, each term of the above optimization criterion expands to ฮ^ത^^^ฮଶ^ฮ ^ഥ^′^ோฮଶെ2ฮ ^ത^^^ฮ ^ ^ഥ^ᇱோ^^ cos ^^^

[0118] Let θ be the angle of When R2isto the ^ഥ^′ோ^, the criterion to be∑ଷ^ฮ ^ത^^ฮଶ^ฮ ^ഥ^ோ ଶ ^ ഥᇱோ^ୀ^ ^ ′^ ฮ െ 2 ฮ ^ത^^ ฮ ^ ^^^ ^ cos൫ ^^^ െ ^^൯^ .

[0119] Minimizing this∑^ୀ^ ฮ^ത^^^ฮ ^^ഥ^ᇱ^ ^cos൫^^^െ ^^൯ൌ ^^ ^^ ^^ ^^ ^^ ^ ^^ ^^ ^^ ^^ ^^where^^ ൌ ^ฮ ^ത^^ ฮ ^ ^^^ ^ cos ^^^ , ^^ ൌ ^ฮ ^ത^^ ฮ ^ ^^^ ^ sin ^^^.

[0120] Nulling the this expressionis found to be such ^^ sin ^^ ൌ ^^ cos ^^ or^^ ^^sin ^^ ൌ േ√^^ଶ ^ ^^ଶ, cos ^^ ൌ ൌ േ√^^ଶ ^ ^^ଶ.

[0121] Since, with these optima, ଶଶ^^ ^^ ^^ ^^ ^^ ^ ^^ ^^ ^^ ^^ ^^ ൌ േ ^^ ^ ^^ൌ േ ^ ^ ଶ ଶ^^ଶ ^ ^^ଶ^ ^ ^^ ,and we want to compute the optimal value of θ. This definesmatrix R may be determined as the composition of rotations R1and R2, such that R = R2R1. Example Systems

[0122] The encoding, decoding, processing, and rendering of scene descriptions as described herein may be implemented using a system such as the system of FIG.13. FIG.13 is a block diagram of an example of a system in which various aspects and embodiments are implemented. System 1000 can be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this document. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 1000, singly or in combination, can be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, the system 1000 is communicatively coupled to one or more other systems, or other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various embodiments, the system 1000 is configured to implement one or more of the aspects described in this document.

[0123] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this document. Processor 1010 can include embedded memory, input output interface, and various other circuitries as known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device, and / or a non-volatile memory device). System 1000 includes a storage device 1040, which can include non-volatile memory and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drive, and / or optical disk drive. The storagedevice 1040 can include an internal storage device, an attached storage device (including detachable and non-detachable storage devices), and / or a network accessible storage device, as non-limiting examples.

[0124] System 1000 includes an encoder / decoder module 1030 configured, for example, to process data to provide an encoded 3D scene or decoded 3D scene, and the encoder / decoder module 1030 can include its own processor and memory. The encoder / decoder module 1030 represents module(s) that can be included in a device to perform the encoding and / or decoding functions. As is known, a device can include one or both of the encoding and decoding modules. Additionally, encoder / decoder module 1030 can be implemented as a separate element of system 1000 or can be incorporated within processor 1010 as a combination of hardware and software as known to those skilled in the art.

[0125] Program code to be loaded onto processor 1010 or encoder / decoder 1030 to perform the various aspects described in this document can be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. In accordance with various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 can store one or more of various items during the performance of the processes described in this document. Such stored items can include, but are not limited to, the input scene, the decoded scene or portions of the decoded scene, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0126] In some embodiments, memory inside of the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other embodiments, however, a memory external to the processing device (for example, the processing device can be either the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory can be the memory 1020 and / or the storage device 1040, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of, for example, a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for video coding and decoding operations, such as for MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also referred to as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET, the Joint Video Experts Team).

[0127] The input to the elements of system 1000 can be provided through various input devices as indicated in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Component (COMP) input terminal (or a set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples include composite video.

[0128] In various embodiments, the input devices of block 1130 have associated respective input processing elements as known in the art. For example, the RF portion can be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select (for example) a signal frequency band which can be referred to as a channel in certain embodiments, (iv) demodulating the downconverted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner that performs various of these functions, including, for example, downconverting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, downconverting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements can include inserting elements in between existing elements, such as, for example, inserting amplifiers and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna.

[0129] Additionally, the USB and / or HDMI terminals can include respective interface processors for connecting system 1000 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed-Solomon error correction, can be implemented, for example, within a separate input processing IC or within processor 1010 as necessary. Similarly, aspects of USB or HDMI interface processing can be implemented within separate interface ICs or within processor 1010 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 1010, and encoder / decoder1030 operating in combination with the memory and storage elements to process the datastream as necessary for presentation on an output device.

[0130] Various elements of system 1000 can be provided within an integrated housing, Within the integrated housing, the various elements can be interconnected and transmit data therebetween using suitable connection arrangement 1140, for example, an internal bus as known in the art, including the Inter-IC (I2C) bus, wiring, and printed circuit boards.

[0131] The system 1000 includes communication interface 1050 that enables communication with other devices via communication channel 1060. The communication interface 1050 can include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 1060. The communication interface 1050 can include, but is not limited to, a modem or network card and the communication channel 1060 can be implemented, for example, within a wired and / or a wireless medium.

[0132] Data is streamed, or otherwise provided, to the system 1000, in various embodiments, using a wireless network such as a Wi-Fi network, for example IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal of these embodiments is received over the communications channel 1060 and the communications interface 1050 which are adapted for Wi-Fi communications. The communications channel 1060 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet for allowing streaming applications and other over- the-top communications. Other embodiments provide streamed data to the system 1000 using a set-top box that delivers the data over the HDMI connection of the input block 1130. Still other embodiments provide streamed data to the system 1000 using the RF connection of the input block 1130. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, for example a cellular network or a Bluetooth network.

[0133] The system 1000 can provide an output signal to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. The display 1100 of various embodiments includes one or more of, for example, a touchscreen display, an organic light- emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 can be for a television, a tablet, a laptop, a cell phone (mobile phone), or other device. The display 1100 can also be integrated with other components (for example, as in a smart phone), or separate (for example, an external monitor for a laptop). The other peripheral devices 1120 include, in various examples of embodiments, one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide a functionbased on the output of the system 1000. For example, a disk player performs the function of playing the output of the system 1000.

[0134] In various embodiments, control signals are communicated between the system 1000 and the display 1100, speakers 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communications protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, the output devices can be connected to system 1000 using the communications channel 1060 via the communications interface 1050. The display 1100 and speakers 1110 can be integrated in a single unit with the other components of system 1000 in an electronic device such as, for example, a television. In various embodiments, the display interface 1070 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0135] The display 1100 and speaker 1110 can alternatively be separate from one or more of the other components, for example, if the RF portion of input 1130 is part of a separate set- top box. In various embodiments in which the display 1100 and speakers 1110 are external components, the output signal can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0136] The system 1000 may include one or more sensor devices 1095. Examples of sensor devices that may be used include one or more GPS sensors, gyroscopic sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors may be used to determine information such as user’s position and orientation. Where the system 1000 is used as the control module for an augmented reality display (such as control modules 124, 1254), the user’s position and orientation may be used in determining how to render image data such that the user perceives the correct portion of a virtual object or virtual scene from the correct point of view. In the case of head- mounted display devices, the position and orientation of the device itself may be used to determine the position and orientation of the user for the purpose of rendering virtual content. In the case of other display devices, such as a phone, a tablet, a computer monitor, or a television, other inputs may be used to determine the position and orientation of the user for the purpose of rendering content. For example, a user may select and / or adjust a desired viewpoint and / or viewing direction with the use of a touch screen, keypad or keyboard, trackball, joystick, or other input. Where the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation used for the purpose of rendering content may be selected and / or adjusted based on motion of the display device.

[0137] The embodiments can be carried out by computer software implemented by the processor 1010 or by hardware, or by a combination of hardware and software. As a non- limiting example, the embodiments can be implemented by one or more integrated circuits. The memory 1020 can be of any type appropriate to the technical environment and can be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 1010 can be of any type appropriate to the technical environment, and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples. Further Embodiments.

[0138] This disclosure describes a variety of aspects, including tools, features, embodiments, models, approaches, etc. Many of these aspects are described with specificity and, at least to show the individual characteristics, are often described in a manner that may sound limiting. However, this is for purposes of clarity in description, and does not limit the disclosure or scope of those aspects. Indeed, all of the different aspects can be combined and interchanged to provide further aspects. Moreover, the aspects can be combined and interchanged with aspects described in earlier filings as well.

[0139] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are illustrated specifically, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of the implementations. At least one of the aspects generally relates to 3D scene encoding and decoding, and at least one other aspect generally relates to transmitting a bitstream generated or encoded. These and other aspects can be implemented as a method, an apparatus, a computer readable storage medium having stored thereon instructions for encoding or decoding scene description data according to any of the methods described, and / or a computer readable storage medium having stored thereon a bitstream generated according to any of the methods described.

[0140] Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as “first”, “second”, etc. may be used in various embodiments to modify an element, component, step, operation, etc., such as, for example, a “first decoding” and a “second decoding”. Use of such terms does notimply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or in an overlapping time period with the second decoding.

[0141] Various numeric values may be used in the present disclosure, for example. The specific values are for example purposes and the aspects described are not limited to these specific values.

[0142] Embodiments described herein may be carried out by computer software implemented by a processor or other hardware, or by a combination of hardware and software. As a non- limiting example, the embodiments can be implemented by one or more integrated circuits. The processor can be of any type appropriate to the technical environment and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples.

[0143] When a figure is presented as a flow diagram, it should be understood that it also provides a block diagram of a corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow diagram of a corresponding method / process.

[0144] The implementations and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed can also be implemented in other forms (for example, an apparatus or program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.

[0145] Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout this disclosure are not necessarily all referring to the same embodiment.

[0146] Additionally, this disclosure may refer to “determining” various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0147] Further, this disclosure may refer to “accessing” various pieces of information. Accessing the information can include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0148] Additionally, this disclosure may refer to “receiving” various pieces of information. Receiving is, as with “accessing”, intended to be a broad term. Receiving the information can include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0149] It is to be appreciated that the use of any of the following “ / ”, “and / or”, and “at least one of”, for example, in the cases of “A / B”, “A and / or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended for as many items as are listed.

[0150] Also, as used herein, the word “signal” refers to, among other things, indicating something to a corresponding decoder. For example, in certain embodiments the encoder signals a particular one of a plurality of parameters for region-based filter parameter selection for de-artifact filtering. In this way, in an embodiment the same parameter is used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as wellas others, then signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual functions, a bit savings is realized in various embodiments. It is to be appreciated that signaling can be accomplished in a variety of ways. For example, one or more syntax elements, flags, and so forth are used to signal information to a corresponding decoder in various embodiments. While the preceding relates to the verb form of the word “signal”, the word “signal” can also be used herein as a noun.

[0151] Implementations can produce a variety of signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal can be formatted to carry the bitstream of a described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0152] We describe a number of embodiments. Features of these embodiments can be provided alone or in any combination, across various claim categories and types. Further, embodiments can include one or more of the following features, devices, or aspects, alone or in any combination, across various claim categories and types: ^ A bitstream or signal that includes one or more of the described syntax elements, or variations thereof. ^ A bitstream or signal that includes syntax conveying information generated according to any of the embodiments described. ^ Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the described syntax elements, or variations thereof. ^ Creating and / or transmitting and / or receiving and / or decoding according to any of the embodiments described. ^ A method, process, apparatus, medium storing instructions, medium storing data, or signal according to any of the embodiments described.

[0153] Note that various hardware elements of one or more of the described embodiments may be referred to as “modules” that carry out (i.e., perform, execute, and the like) various functions that are described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, oneor more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable for a given implementation. Each described module may also include instructions executable for carrying out the one or more functions described as being carried out by the respective module, and it is noted that those instructions could take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or the like, and may be stored in any suitable non-transitory computer- readable medium or media, such as commonly referred to as RAM, ROM, etc.

[0154] Although features and elements are described above in particular combinations, each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

CLAIMS 1. A method comprising: obtaining information defining a reference mesh and a dependent mesh, the dependent mesh including at least three binding vertices; obtaining correspondence information usable for determining, for each of the binding vertices, a respective animated position relative to the reference mesh; determining a first animated pose of the reference mesh in at least a first frame of an animation; determining first animated positions of the binding vertices based on the first animated pose of the reference mesh and the correspondence information; determining a first transformation, the first transformation being a transformation that transforms the binding vertices of the dependent mesh toward the respective first animated positions of the binding vertices; and determining a first animated pose of the dependent mesh by applying the first transformation to the dependent mesh.

2. An apparatus comprising one or more processors configured to perform at least: obtaining information defining a reference mesh and a dependent mesh, the dependent mesh including at least three binding vertices; obtaining correspondence information usable for determining, for each of the binding vertices, a respective animated position relative to the reference mesh; determining a first animated pose of the reference mesh in at least a first frame of an animation; determining first animated positions of the binding vertices based on the first animated pose of the reference mesh and the correspondence information; determining a first transformation, the first transformation being a transformation that transforms the binding vertices of the dependent mesh toward the respective first animated positions of the binding vertices; and determining a first animated pose of the dependent mesh by applying the first transformation to the dependent mesh.

3. The method of claim 1 or the apparatus of claim 2, further comprising rendering the first frame of the animation using the first animated pose of the reference mesh and the first animated pose of the dependent mesh.

4. The method of claim 1, or claim 3 as it depends from claim 1, or the apparatus of claim 2, or claim 3 as it depends from claim 2, further comprising: determining a second animated pose of the reference mesh in at least a second frame of the animation; determining second animated positions of the binding vertices based on the second animated pose of the reference mesh and the correspondence information; determining a second transformation, the second transformation being a transformation that transforms the binding vertices of the dependent mesh toward the respective second animated positions of the binding vertices; and determining a second animated pose of the dependent mesh by applying the first transformation to the dependent mesh.

5. The method of claim 4 as it depends from claim 1, or the apparatus of claim 4 as it depends from claim 2, further comprising rendering the second frame of the animation including the second animated pose of the reference mesh and the second animated pose of the dependent mesh.

6. The method of claim 1, or claims 3-5 as they depend from claim 1, or the apparatus of claim 2, or claims 3-5 as they depend from claim 2, wherein the first transformation is a rigid transformation.

7. The method of claim 1, or claims 3-6 as they depend from claim 1, or the apparatus of claim 2, or claims 3-6 as they depend from claim 2, wherein the first transformation comprises a rotation and a translation.

8. A method comprising: selecting at least three vertices of a dependent mesh as binding vertices; selecting a pose of the dependent mesh relative to a reference mesh; and encoding correspondence information indicating, for each of the binding vertices, a position of the respective binding vertex relative to the reference mesh while the dependent mesh is in the selected pose.

9. An apparatus comprising one or more processors configured to perform at least: selecting at least three vertices of a dependent mesh as binding vertices; selecting a pose of the dependent mesh relative to a reference mesh; andencoding correspondence information indicating, for each of the binding vertices, a position of the respective binding vertex relative to the reference mesh while the dependent mesh is in the selected pose.

10. The method of claim 8 or the apparatus of claim 9, wherein fewer than all vertices on the dependent mesh are selected as binding vertices.

11. The method of claim 8, or claim 10 as it depends from claim 8, or the apparatus of claim 9, or claim 10 as it depends from claim 9, wherein only three vertices of the dependent mesh are selected as binding vertices.

12. The method of claim 8, or claims 10-11 as they depend from claim 8, or the apparatus of claim 9, or claims 10-11 as they depend from claim 9, further comprising encoding an animation of a scene including the reference mesh and the dependent mesh.

13. The method of claim 1 or claims 3-7 as they depend from claim 1, or the method of claim 8 or claims 10-12 as they depend from claim 8, or the apparatus of claim 2 or claims 3-7 as they depend from claim 2, or the apparatus of claim 9 or claims 10-12 as they depend from claim 8, wherein the correspondence information identifies, for at least one of the binding vertices, a respective nearest vertex of the reference mesh.

14. The method of claim 1 or claims 3-7 as they depend from claim 1, or the method of claim 8 or claims 10-12 as they depend from claim 8, or the apparatus of claim 2 or claims 3-7 as they depend from claim 2, or the apparatus of claim 9 or claims 10-12 as they depend from claim 8, wherein the correspondence information identifies, for at least one of the binding vertices, a respective position on a face of the reference mesh.

15. The method of claim 1 or claims 3-7 as they depend from claim 1, or the method of claim 8 or claims 10-12 as they depend from claim 8, or the apparatus of claim 2 or claims 3-7 as they depend from claim 2, or the apparatus of claim 9 or claims 10-12 as they depend from claim 8, wherein the correspondence information further includes distance information indicating a distance of at least one of the binding vertices from the reference mesh.

Images