Rigid mesh linking for scene description

Abstract

An apparatus and method for encoding and decoding a mesh animation are provided. In an example method, information defining a reference grid and a slave grid is obtained. The subordinate grid comprises at least three binding vertexes. Correlation information is obtained that can be used to determine, for each of the binding vertices of the slave grid in a reference pose, a respective position relative to the reference grid. And for each frame in the animation, determining an animation attitude of the reference grid, and determining an animation position of the binding vertex based on the animation attitude of the reference grid and the corresponding relation information. A transformation that substantially transforms the binding vertex of the slave grid in a reference pose to a respective first animated position of the binding vertex is determined, and an animated pose of the slave grid is determined by applying a first transformation to all vertices of the slave grid.

Description

[0001] Cross-references This application claims priority to European Patent Application No. 23306625.7, filed on September 28, 2023, entitled "Rigid Mesh Linking For Scene Description", which is incorporated herein by reference in its entirety. Background Technology

[0002] This disclosure relates to the representation of digital humans and their interaction with 3D virtual environments. Specifically, this disclosure relates to the animation of virtual characters. This document can also be understood in the context of augmented reality and extended reality applications being rendered and formatted and played on end-user devices such as mobile devices or head-mounted displays (HMDs). 3D scenes can be represented by runtime asset delivery files (such as glTF files). Runtime asset delivery files can provide information about the 3D scene, including the geometry and organization of objects in the scene, as well as information for animateting these objects. Scene graphs (such as those proposed by Khronos / glTF and their extensions defined in the MPEG scene description format or Apple / USDZ) are possible ways to represent content to be rendered. They combine, on the one hand, an explanatory description of the scene structure linking real-world objects and virtual objects, and on the other hand, a binary representation of the virtual content. Summary of the Invention

[0003] A method according to some embodiments includes: obtaining information defining a reference mesh and a dependent mesh in a reference pose, the dependent mesh in the reference pose including at least three bound vertices; obtaining correspondence information capable of determining a corresponding position relative to the reference mesh for each of the bound vertices; determining a first animated pose of the reference mesh in at least a first frame of an animation; determining a first animated position of the bound vertices based on the first animated pose of the reference mesh and the correspondence information; determining a first transformation, the first transformation being a transformation that substantially transforms the bound vertices of the dependent mesh in the reference pose to the corresponding first animated positions of the bound vertices; and determining the first animated pose of the dependent mesh by applying the first transformation to all vertices of the dependent mesh.

[0004] Such embodiments may also include rendering the first frame of the animation using the first animated pose of the reference mesh and the first animated pose of the dependent mesh.

[0005] Some embodiments further include: determining a second animated pose of the reference mesh in at least a second frame of the animation; determining a second animated position of the bound vertex based on the second animated pose of the reference mesh and the correspondence information; determining a second transformation, the second transformation being a transformation that substantially transforms the bound vertex of the dependent mesh to the corresponding second animated position of the bound vertex; and determining the second animated pose of the dependent mesh by applying the first transformation to all vertices of the dependent mesh. Such embodiments may also include rendering a second frame of the animation, the second frame including the second animated pose of the reference mesh and the second animated pose of the dependent mesh. A similar process can be performed on additional frames (and even all frames) of the animation, and many or all of these frames may use the same correspondence information.

[0006] In some embodiments, for each of the bound vertices, the correspondence information identifies the corresponding nearest vertex of the reference mesh. In some embodiments, for each of the bound vertices, the correspondence information identifies the corresponding position on a face of the reference mesh. In some embodiments, the correspondence information further includes distance information indicating the distance between each corresponding bound vertex and the reference mesh.

[0007] One method according to some embodiments includes: selecting at least three vertices of a dependent mesh as bound vertices; selecting the pose of the dependent mesh relative to a reference mesh; and encoding correspondence information, the correspondence information indicating for each of the bound vertices the position of the corresponding bound vertex relative to the reference mesh when the dependent mesh is in the selected pose.

[0008] In some such embodiments, fewer than all vertices on the dependent mesh are selected as bound vertices. In some embodiments, only three vertices of the dependent mesh are selected as bound vertices.

[0009] Some embodiments include encoding animation of a scene that includes the reference mesh and the subordinate mesh.

[0010] Additional embodiments include an apparatus comprising one or more processors configured to perform any of the methods described herein.

[0011] Additional embodiments include an apparatus comprising at least one processor and a computer-readable medium (which may be non-transitory) storing instructions for performing any of the methods described herein.

[0012] Additional embodiments include a computer-readable medium storing instructions for performing any of the methods described herein.

[0013] Additional embodiments include a computer-readable medium that stores a scenario encoded according to any of the methods described herein.

[0014] Additional embodiments include a signal that transmits a scene encoded according to any of the methods described herein. Attached Figure Description

[0015] Figure 1 The diagram shows a skeletal representation of a virtual character that can be used to drive the animation of the virtual character.

[0016] Figure 2 An example virtual character mesh geometry that may be used in some embodiments is illustrated schematically.

[0017] Figure 3 An example of mesh registration using vertex-to-vertex correspondence is shown. M’ The top of the vertex V’ 6 mapped to M The position above P 6. V’ 6 in M The correspondence on is set to M Above and P The closest vertex to 6, that is, V 5.

[0018] Figure 4 This illustrates mesh registration via vertex-to-face correspondence. M’ The top of the vertex V’ 6 mapped to M The position above P 6. V’ 6 in M The correspondence above is M On P The parametric model for position 6 is adopted. P 6 in triangle （V 5 V 7 V 9 ） barycentric coordinates （u 6 ，v 6 ，w 6 ） In the form of.

[0019] Figure 5 This illustrates the extension of vertex-to-vertex correspondences to non-coincident vertex pairs. Dependent Mesh M’ The top of the vertex V’I Located on the reference grid M The vertex of the reference mesh A i Overlapping points P i normal at the location （L i ） superior. （L i ） It can be calculated as the sharing represented in the diagram. A i The normalized average of the unit normals of the triangle that serves as the vertices.

[0020] Figure 6 This illustrates the extension of the vertex-to-face correspondence to non-coincident vertex pairs. Dependent Mesh M’ The top of the vertex V’ I Located on the reference grid M triangle T i =（A i ，B i ，C i ） At point P i normal at the location （L i ） superior. P i The position is determined by the triangle in the reference grid. T i index t i and T i The barycentric coordinates on （u i ，v i ，w i ） Sure.

[0021] Figure 7 The top hat mesh is shown as being bound to the virtual character's head mesh during animation.

[0022] Figure 8 The top hat mesh is shown in both "reference pose" and "animation pose".

[0023] Figure 9 This is a flowchart illustrating the method for parsing the "MPEG_mesh_rigid_linking" extension.

[0024] Figure 10 This is a flowchart of an example processing model for rigid mesh animation based on mesh correspondence data.

[0025] Figure 11 The correspondence between three points on a rigid object is shown.

[0026] Figure 12 The diagram shows the correspondence between three points after the reference pose plane is rotated to the animation pose plane.

[0027] Figure 13 This is a functional block diagram of a system that can be used to implement some of the embodiments. Detailed Implementation

[0028] Overview of Virtual Character Animation The animation of a virtual character's body is typically driven by the movement of its skeleton. The skeleton is a directed acyclic graph of joint nodes connected by bone edges. Joints are usually mapped to physical human joints, such as the elbow, ankle, and wrist joints, as in... Figure 1 As seen in the diagram. It should be noted that joint names can differ in different skeletal representations. The root joint in the diagram is typically chosen as the pelvis. Figure 1 The term "hip" is used in this context. The translation and rotation of each joint relative to its parent joint clearly define the posture of the bone. Figure 1 The skeletal representation is shown, which can be used, for example, for the MPEG-I-SD virtual character model, from “[SD] Update of Annex to ISO / IEC 23090-14:2022 – MPEG Reference Humanoid Avatar”, MPEG Conference #142 Input Document m62956, April 2023.

[0029] The geometry of the virtual character's body envelope is represented by a textured mesh. Figure 2 The diagram schematically illustrates an example of the mesh geometry of a virtual character. The animation of the envelope mesh is controlled by the motion of the skeletal joints through a process called skinning. In this process, the position of each vertex of the mesh is bound to the positions of a set of predetermined influencing joints associated with that vertex via a predetermined binding matrix. For a given pose of the skeleton, the positions of the mesh vertices are calculated based on the joint positions using the binding matrix, and these positions provide the geometry of the virtual character's envelope. After texturing and lighting, an image of the virtual character in the picture is obtained by rendering the textured mesh from a given camera viewpoint.

[0030] Although Figure 2A general virtual character mesh model is shown, but some embodiments use one or more topologies of a virtual character mesh model that is provided as an information reference for the MPEG-I scene description standard, which describes different virtual character mesh models with various levels of detail (e.g., high, medium, and low levels of detail), from “[SD] Update of Annex to ISO / IEC 23090-14:2022 – MPEG Reference Humanoid Avatar”, MPEG Conference #142 Input Document m62956, April 2023.

[0031] Morphological targets provide an alternative way to animate virtual character meshes, and particularly to shape facial expressions. They can be described as 3D deformations of the geometry relative to a base mesh, represented by 3D offsets of each mesh vertex. Typically, the base mesh is a model of a virtual character with a neutral facial expression that does not express emotion. In addition to the base mesh, morphological targets are provided. Each morphological target is associated with a scalar weight. The desired shape of the mesh is obtained by adding a linear combination of several morphological target offsets, weighted by their corresponding weights, to the vertex positions of the base mesh. In typical use cases, morphological targets are mapped to deformations of neutral facial geometry caused by the activation of a set of facial muscles. Combining these deformations provides an easy way for artists to shape facial meshes to reproduce desired facial expressions.

[0032] Overview of grid correspondence During the animation of a virtual character, it may be desirable to bind objects whose geometry is represented by a 3D mesh to the virtual character's mesh as the virtual character moves. For this purpose, it is advantageous to bind the positions of some object mesh vertices to corresponding positions on the virtual character's mesh surface. This binding information defines the correspondence between the two meshes.

[0033] To illustrate the mesh correspondence, consider meshes with vertices. {V i } Reference mesh M and having vertices {V’ i } Subordinate grid M’ . M’ a certain vertex V’ i exist M The correspondence on is defined as M Above and V’ i Corresponding positionP i For example, if M It is a grid of virtual characters and M’ If it is a grid of objects (such as top hats) to be bound to a virtual character, then V’ i The vertices of the mesh that can represent the top hat in contact with the head of the virtual character, and P i It will be when the top hat is attached to the virtual character's head and... V’ i The positions of overlapping virtual characters on the surface of the grid.

[0034] For triangular meshes, there are two standard ways to represent them relative to... M The position of the geometric shape P i .

[0035] The first option is to use a reference grid. M In M The surface closest to P i Represented by vertices P i This representation, known as vertex-to-vertex correspondence, is... Figure 3 As shown in the image. Figure 3 A diagram illustrating mesh registration using vertex-to-vertex correspondence is provided. M’ The top of the vertex V’ 6 mapped to M The position above P 6. V’ 6 in M The correspondence on is set to M Above and P The closest vertex to 6, that is, V 5.

[0036] The vertex of M' V’ 6 in the reference grid M There are corresponding points on it. P 6. M and P The nearest vertex to 6 is V 5. Therefore, M Above and V’ The position corresponding to 6 is V 6 V_to_V =V 5. For subordinate grids M’ Each vertex V’ I , used to subordinate mesh M’ With reference grid MThe registered vertex-to-vertex correspondence data is provided by {V i V_to_V } The set is represented by .

[0037] The second representation is a set of vertex-to-face correspondences represented by barycentric coordinates. Figure 4 As shown in the image. Figure 4 A diagram is provided illustrating mesh registration based on vertex-to-face correspondence. M’ The top of the vertex V’ 6 mapped to M The position above P 6. V’ 6 in M The correspondence above is M On P The parametric model for position 6 is adopted. P 6 in triangle （V 5 V 7 V 9 ） barycentric coordinates （u 6 ，v 6 ， w 6 ） The form. Reference grid. M Above and M’ vertex V’ i = V’ The point corresponding to 6 is P i = P 6. P 6 is located M triangle T i = （A i ，B i ，C i ） identified as T 6 =（V 5 ，V 7 ，V 9 ） .therefore, A i = V 5 ，B i = V 7 and C i = V 9. Let... S i for Ti The area, and let U i (They are respectively) V i , W i ) for P i and its first (second and third) opposite vertex T i The area of ​​the triangle formed by the two vertices of the triangle. Figure 4 middle, T i yes （V 5 ，V 7 ，V 9 ） , P i = P 6, therefore U i It is a triangle （P 6 ，V 7 ，V 9 ） area, V i It is a triangle （V 5 ，P 6 ，V 9 ） The area, and W i It is a triangle （V 5 ，V 7 ，P 6 ） The area. Furthermore, let... u i = U i / S i , v i = V i / S i and w i = W i / S i . （u i ，v i ，w i ） yes P i exist T i The coordinates of the centroid in P.i exist T i The position inside is from its centroid coordinates （u i ，v i ，w i ） The following was obtained: (1) exist Figure 4 middle, u 6 = Area((P 6 ，V 7 ，V 9 )) / Area ((V 5 ，V 7 ，V 9 ）） , v 6 = Area((V 5 ，P 6 ，V 9 ）） / Area (V 5 ，V 7 ，V 9 ），w 6 = Area(V 5 ，V 7 ，P 6 )) / Area ((V 5 ，V 7 ，V 9 ） ,and P 6 = u 6 V 5 + v 6 V 7 + w 6 V 9. Let... t i To construct a reference grid M A set of triangles T i The index. M’ Each vertex V' i The vertex-to-face coordinate correspondence data is composed of a set of {（t i ，u i ，v i ，w i ）} Indicates. Each. (t i ，u i ，v i ，wi ) Clearly determine P i (i.e., grid) M Above and V' i The location of the corresponding point.

[0038] The mesh correspondence can be extended to cases where the position of a vertex on the reference mesh does not coincide with its corresponding vertex on the dependent mesh. In this case, the corresponding vertex on the dependent mesh... V’ i The position can be referenced as the projection point of the reference grid onto the reference grid. P i normal at the location （L i ） The point on the projection. As explained above, this projection point P i Use vertex-to-vertex mappings (such as in...) Figure 3 (in the middle) or vertex-to-face correspondence (such as in Figure 4 The location of the correspondence between the reference mesh and the normal at the projection point can be described by the signed distance to the projection point. d i To describe. This symbol indicates whether the corresponding point lies on the outer or inner normal of the mesh. The suggested convention for this symbol is if the triangle vertex is... （A i ，B i ，C i ） Sort by the positive distance direction, which is determined by the normal vector. Definition. Or rather, let... (2) Is the reference mesh in P i The unit normal at point , where ∧ represents the cross product of vectors. d i Defined as (3) Figure 5 In China P i The vertex-to-vertex correspondence is defined as one of the vertices of the reference mesh in the graph. A i This extension is illustrated in the case of [example missing]. Figure 5 This illustrates the extension of vertex-to-vertex correspondences to non-coincident vertex pairs. Dependent Mesh M’ The top of the vertex V’ ILocated on the reference grid M The vertex of the reference mesh A i Overlapping points P i normal at the location （L i ） superior. （L i ） It can be calculated as the sharing represented in the diagram. A i The average value of the unit normals of the triangle that serves as its vertices. For example, for a triangle... （A i ，B i ，C i ） The unit normal is defined by equation (2). Figure 5 middle, A i The unit normal at that location will be defined as a triangle. （A i ，B i ， C i ）、（A i ，C i ，D i ）、（A i ，D i ，E i ） and （A i ，E i ，F i ） The normalized average of the unit normal.

[0039] Figure 6 This illustrates the extension of the vertex-to-face correspondence to non-coincident vertex pairs. Dependent Mesh M’ The top of the vertex V’ I Located on the reference grid M triangle T i = （A i ，B i ，C i） At point P i normal at the location （L i ） superior. P i The position is determined by the triangle in the reference grid. T i index t i and T i The barycentric coordinates on （u i ，v i ，w i ） Confirmed. In the case of vertex-to-face correspondence, the vertices of the subordinate mesh. V’ I The correspondence specification on surfaces not located on the reference mesh is determined by triangles in the reference mesh. T i index t i , V’ I Normal projection on the reference mesh P i barycentric coordinates （u i ，v i ，w i ） and triangle index and reference grid in P i Signed distance on the normal at the location d(P) i ，V’ I ） composition.

[0040] An overview of the virtual character mesh correspondence in MPEG-I scene description. The scheme for describing virtual characters in 3D virtual environments is provided in the MPEG-I Scene Description (SD) standard described in ISO / IEC DIS 23090-14:2021(E) Information technology—Encoded representation of immersive media—Part 14: Scene description of MPEG media: FDIS Commentary. This is based on the Khronos glTF specification. The “MPEG_node_avatar” extension of the MPEG-I SD described in ISO / IEC 23090-14 CDAM 2: Support for haptics, augmented reality, virtual characters, interactivity, MPEG-I audio and lighting specifies a reference virtual character model with mesh topology and animation rig, which is typically constructed based on skeletal animation, skinning, and morphological target mesh deformation. The virtual character model is referenced by a Uniform Resource Name specified by the type attribute of the “MPEG_node_avatar” extension.

[0041] However, users may wish to animate custom virtual characters whose mesh topology differs from the reference virtual character model. For example, such custom virtual characters can be obtained by capturing the geometry of a person using a 3D scanning device or by reconstructing that geometry based on photographs of a person taken from different viewpoints using photogrammetric reconstruction techniques. In these cases, the default custom virtual character model does not have available animation rigs.

[0042] The "MPEG_mesh_linking" extension of the "mesh" object in MPEG-I SD provides a way to animate custom virtual characters using a reference virtual character model. To do this, the mesh of the custom virtual character is registered with the mesh of the reference virtual character. This is obtained by providing mesh correspondence data, which will animate the custom virtual character's mesh (referred to as...). subordinate Each vertex of the mesh is bound to a reference virtual character mesh (referred to as the "mesh"). shadow The corresponding position on the surface of the grid. If you want to animate a custom virtual character, provide the corresponding relationship data for each frame of the animation.

[0043] Specifically, the "MPEG_mesh_linking" extension has the following attributes: • The grid index, which points to the shaded grid object in the glTF "grid" array. • Pose accessor, which points to the transformation of the shadow mesh's skeletal joints to be applied in order to reproduce the desired pose of the dependent mesh in the current frame of the animation. • A weight accessor that points to the weights of the shape target to be applied to the shadow mesh in order to reproduce the desired subordinate mesh shape in the current frame, and • A correspondence accessor that points to per-vertex correspondence data, which provides the position of each vertex relative to the surface of the shaded mesh after the pose and shape of each vertex of the dependent mesh have been transformed by skinning and morphological target deformation using pose and weight attributes.

[0044] The animation transition at each frame is performed in two stages: First, the mesh of the reference virtual character model is transformed into the desired pose and expression of the custom virtual character based on its skinning and shape target attributes, as well as the corresponding pose and weight data. Second, the transformed reference avatar mesh is retopologically retopologically converted to the topology of the custom avatar mesh using the data contained in the buffer pointed to by the corresponding attribute. This data allows the calculation of the position of each vertex of the custom mesh on the surface of the transformed reference avatar mesh. The resulting mesh after this retopology operation has the desired shape of the transformed reference avatar mesh and the topology of the custom mesh.

[0045] Problems solved in some embodiments Consider an application that relies on the MPEG-I Scene Description (SD) standard, allowing users to select customizable virtual characters and animate them within a virtual environment. It should be noted that MPEG-I SD supports animation as instances of glTF "Animation" objects. Users might want to enrich their virtual characters with multiple accessories such as glasses, headbands, or earrings. For this purpose, the application can provide a library of such items as 3D textured mesh representations. During the animation of the virtual character, the mesh representing the accessory item then needs to be rigidly bound to the virtual character's mesh. Specifically, a rigid 3D geometric transformation (defined as a combination of 3D rotation and 3D translation) must be applied to the accessory item's mesh so that its position and pose are consistent with the position and pose of the virtual character's body or head in each frame.

[0046] This disclosure solves the problem of binding a mesh of a second rigid object (hereinafter referred to as the dependent mesh) to the mesh of a first object (hereinafter referred to as the reference mesh) when a reference mesh is animated. The geometry of the dependent mesh is provided with a predetermined "reference pose". At each frame of the animation, the pose of the dependent mesh is calculated such that it is always bound to the reference mesh. For example, if in the first frame the dependent mesh contacts the reference mesh at a specific location on the reference mesh, the contact point remains substantially unchanged throughout the animation.

[0047] One way to solve this problem would be to pre-compute and store the animation of the dependent mesh for all animations of the reference mesh to which it will be bound. However, this would involve storing a large amount of data due to the large number of possible combinations of reference and dependent meshes. Furthermore, during animation, the pre-computed pose of the dependent mesh must be transmitted every frame to render the correct binding. Typically, this pre-computed pose is represented by a rigid 3D transformation relative to the reference pose of the dependent object. This rigid 3D transformation is a combination of 3D rotation and 3D translation.

[0048] Overview of Example Implementations The example embodiments described herein are intended to provide a more efficient alternative. In some embodiments, correspondence data between a reference mesh and a dependent mesh, independent of any animation, is transmitted to the device responsible for reconstructing the animation geometry only a small amount (e.g., only once). Based on this correspondence data, the device is able to reconstruct the correct pose of the dependent mesh at each frame of any animation of the reference mesh, based on the geometry of the reference mesh at the frame under consideration.

[0049] Some embodiments provide a representation format for these correspondence data.

[0050] In the example embodiment, the geometry of the dependent mesh is provided with a fixed reference pose. The mesh can be a rigid mesh, in which case the only permissible change to the geometry of the dependent mesh during animation is a global 3D transformation, which is defined as a combination of a global 3D rotation and a global 3D translation of the mesh.

[0051] In some embodiments, the dependent mesh adapts to the non-rigid deformation of the reference mesh during animation, which is typically small. These adaptations translate into a slight change in the relative pose of the dependent mesh relative to the reference mesh, or, if the dependent mesh is not strictly rigid, a slight change in the dependent mesh itself. This is illustrated in an animation where the reference mesh is the head of a virtual character and the dependent mesh represents the hat on the virtual character's head. When the virtual character speaks and expresses emotions, the shape of the forehead portion of the virtual character's head undergoes slight non-rigid deformation. Therefore, the position of the hat on the head will change slightly. For example, if the virtual character's head expresses surprise, the brow muscles are activated, raising the upper portion of the forehead while the back of the head remains unchanged. Thus, the hat may shift slightly towards the back of the head. Simultaneously, in some embodiments, the mesh representing the hat may also be slightly stretched to accommodate a small expansion of the virtual character's head at the level of the forehead.

[0052] In the example embodiment, the reference mesh is a triangular mesh.

[0053] In some embodiments, the binding information that binds the subordinate mesh to the reference mesh is pre-computed independently and prior to any animation of the reference mesh, and is transmitted only once. This reduces or eliminates the need for any transmission of the subordinate mesh's pose information during any animation of the reference mesh. This results in a reduction in the amount of animation data transmitted. This further reduces transmission bandwidth requirements compared to methods that pre-compute the animation of the subordinate mesh as a rigid 3D transformation and transmit it at every frame of every animation on the reference mesh.

[0054] In some embodiments, binding information for rigidly binding a dependent mesh to a reference mesh includes correspondence data for three “bound vertices” on the dependent mesh. These three bound vertices may be selected (e.g., by a user) according to constraints that they should not be collinear. The correspondence data provides the relative positions of the bound vertices of the dependent mesh with respect to triangles on the surface of the reference mesh. In some embodiments, these relative positions are provided for each of the three bound vertices as their corresponding positions on the surface of the reference mesh and the signed distance between the vertex under consideration and its corresponding position on the reference mesh.

[0055] In some embodiments, the processing of correspondence data whose representation format is proposed can be described as follows.

[0056] Before animate the reference mesh, retrieve the positions of the three corresponding vertices in the reference pose provided by the subordinate mesh, as well as the correspondence data of the three bound vertices.

[0057] At each frame of the animation of the reference mesh, the pose of the dependent mesh, ensuring proper rigidity binding with the reference mesh, is calculated. In some embodiments, the pose is determined according to the following steps: First, obtain the position of the reference mesh vertices in the current frame.

[0058] Second, based on the correspondence data of the three bound vertices and the positions of the vertices of the reference mesh, calculate the positions of the three bound vertices of the dependent mesh for the current frame.

[0059] Third, based on the positions of the three bound vertices in the subordinate mesh's reference pose and the current frame, calculate the parameters of the rigid 3D transformation that brings the subordinate mesh from its reference pose to the animated pose in the current frame.

[0060] Fourth, the rigid transformation calculated in the third step is applied to all vertices of the dependent mesh in the reference pose to obtain the animated pose of the dependent mesh for the current frame.

[0061] When the reference mesh undergoes non-rigid deformation, the rigid 3D transformation calculated in the third step and applied in the fourth step may only approximate the actual position and shape of the dependent mesh relative to the reference mesh. However, in typical use cases, such as when the reference mesh represents a character's head and the dependent mesh represents facial adornments, such as a hood or glasses, the non-rigid deformation of the reference mesh will be very small, and a representation of the dependent mesh's pose relative to the reference mesh through a constant rigid 3D transformation throughout the animation will provide a reasonable approximation of reality. Therefore, after these steps, the dependent mesh will remain bound to the reference mesh throughout the animation in a manner consistent with the user-specified positions of the dependent mesh's three bound vertices relative to the surface of the reference mesh.

[0062] Representation format of grid correspondence data This disclosure presents a representation format for mesh correspondence data between a rigid object and another object. As described above, this correspondence data can be used to provide a favorable representation of data to bind the mesh of the rigid object to the other object when the other object is animated. In some embodiments, this representation depends only on the meshes of the rigid object and the other object, regardless of their position and pose changes during animation. Therefore, the correspondence data can be transmitted once before any animation. During the animation of the other object, the availability of this data provides sufficient information to allow calculations to be performed on the pose of the rigid object's mesh at each frame, thereby correctly binding it to the mesh of the other object. This reduces the amount of data to be transmitted compared to alternative solutions that involve pre-compiling the animation of the rigid object's mesh, which requires transmitting the pose of the rigid object's mesh at each frame of each animation.

[0063] Consistent with previous chapters, in the following description, the mesh of a rigid object is referred to as the “dependent mesh”, and the mesh of the animated object to which the rigid object is bound is referred to as the “reference mesh”.

[0064] For clarity, this document uses a format conforming to the MPEG-I scene description format to describe example embodiments, but the principles described herein are not limited to using any particular format. The representations proposed according to the embodiments described herein can be encoded in any other scene description format, such as XML or USD.

[0065] MPEG-I scene descriptions already have an "MPEG_mesh_linking" extension for encoding mesh correspondence data. However, this description is for a specific use case where both meshes represent virtual characters. It requires one of the meshes to be equipped with an animated rig consisting of a set of mixed shapes and skinned frames, and to provide correspondences for all vertices of one mesh at all frames of the animation. This differs from the embodiments disclosed herein, which provide correspondence data from only three vertices of a mesh to another mesh, for example, a rigid object, and this correspondence data is independent of the mesh's pose and position during animation. Furthermore, neither the reference mesh nor the dependent mesh requires an animated rig.

[0066] For clarity, this paper describes the proposed mesh correspondence data format as a new extension of the “mesh” glTF object, which may be referred to as “MPEG_mesh_rigid_linking”. However, it should be understood that other description options within MPEG-I-SD are possible, such as by modifying the content of the existing “MPEG_mesh_linking” extension to accommodate both virtual role linking and rigid object linking use cases.

[0067] rigid object binding correspondence A rigid object to be bound to a reference mesh is modeled by a subordinate mesh in a fixed, pre-defined "reference pose." During animation, the subordinate mesh will assume other poses, which we call "animated poses." For example, the subordinate mesh could model a top hat, and the reference mesh could model a virtual character. In this example, the correspondence data considered in the example embodiment will be used to ensure that the top hat assumes an appropriate animated pose on the virtual character's head as the virtual character moves (see [link to example]). Figure 7 (As an illustrative example). Figure 8 The reference pose and animated pose of the rigid top hat mesh in this example are shown.

[0068] Assume the reference mesh is a triangular mesh.

[0069] In the proposed representation format, the reference pose of the subordinate mesh is described in a glTF "mesh" instance that describes the geometry of the rigid object. This instance includes the "MPEG_mesh_rigid_linking" extension.

[0070] In some embodiments, the dependent mesh is rigid. This means it does not deform during animation. Or, in other words, at each frame of the animation, the geometry of the mesh is obtained by applying a rigid 3D transformation to a reference pose of the dependent mesh. Generally, rigid transformations in 3D can be specified through a combination of rotation, scaling, and translation. In some example use cases, because the object is rigid, the scaling component is removed, thus maintaining the same size throughout. Therefore, in some predefined 3D coordinate system, the vertices of the rigid object mesh... i Position in animated poses r i It is based on its position in the reference attitude. r ref, i The calculation is as follows: (4) in t 0 represents 3D translation offset, and R It is a 3D rotation (which can be represented by one or more matrices).

[0071] t 0 and R The value can be calculated based on knowledge of the corresponding positions of the three non-collinear vertices of the dependent mesh in the reference pose and the animated pose. In some embodiments, the calculation can be performed using the technique described in "Closed-form solution of absolute orientation using unit quaternions," BKPHorn, Journal of the Optical Society of America, Vol. 4, pp. 629 and subsequent, April 1987. These three vertices will be referred to below as "bound vertices." An example of this calculation process is provided below. Once determined... t 0 and R The positions of all rigid object mesh vertices in the animation pose can be determined by... R Next is t 0 is applied to their positions in the reference pose for calculation. The positions of the bound vertices relative to the reference mesh surface can be determined in advance, for example, manually by the user designing the scenario. These relative positions can be described by vertex-to-vertex or vertex-to-face correspondence data on the reference mesh surface, supplemented by signed distances to the reference mesh surface. It should be noted that these descriptions assume the reference mesh is a triangular mesh.

[0072] It should also be noted that, as mentioned above, if the reference mesh undergoes non-rigid deformation, modeling the relative pose of the subordinate mesh with respect to the reference mesh through a constant rigid 3D transformation consisting of a combination of 3D rotation and 3D translation is only an approximation of reality. Generally, the non-rigid deformation of the reference mesh will be small enough for this approximation to remain valid. In some embodiments, a scaling component can be included in the 3D rigid transformation model to allow for small contractions or expansions of the subordinate mesh, thus better accommodating the deformation of the reference mesh. This scaling component can be expressed as a scalar scaling factor. s The scalar scaling factor can be based on the knowledge of the corresponding positions of the three bound vertices of the subordinate mesh in the reference and animation poses, and the 3D translation in equation (4). t 0 and 3D rotation matrix R Together, they are calculated in a closed-form manner. In some embodiments, the techniques described above by Horn are used for calculation. s The value of .

[0073] Figure 7 The top hat mesh is shown as being bound to the virtual character's head mesh during the animation process.

[0074] Figure 8 The top hat mesh is shown in "reference pose" 802 and "animation pose" 804.

[0075] Example Scenario Description File Extension In some embodiments, a dependent mesh in a reference pose is described by a glTF "mesh" instance extended by the "MPEG_mesh_rigid_linking" extension. The reference pose can be arbitrarily chosen. This extension contains pointers to the reference mesh and mapping information defining the positions of the three non-collinear bound vertices of the dependent mesh relative to the reference mesh. This mapping information allows for the computation of a rigid transformation that carries the dependent mesh from its reference pose to an animated pose in which it is correctly positioned and rotated relative to the reference mesh.

[0076] In some embodiments, the "MPEG_mesh_rigid_linking" extension can be configured as shown in Table 1.

[0077]

[0078] Table 1. Table 1 shows the attributes of the proposed "MPEG_mesh_rigid_linking" extension according to some embodiments. In the usage column, "M" identifies an attribute that must be included in the syntax according to that particular embodiment, and "O" identifies an attribute that is optional in that embodiment. However, it should be understood that an attribute that is syntactically mandatory in one embodiment is not necessarily mandatory (or even used) in any other embodiment.

[0079]

[0080] Table 2. Table 2 illustrates the semantics of the correspondence_type attribute as proposed in some embodiments.

[0081]

[0082] Table 3. Table 3 shows the contents of the correspondence buffer in some embodiments. In the usage column, "M" identifies an attribute that must be included according to the syntax of that particular embodiment, and "O" identifies an attribute that is optional in that embodiment. However, it should be understood that an attribute that is grammatically mandatory in one embodiment is not necessarily mandatory (or even used) in any other embodiment.

[0083] The barycentric_coords and signed_distance attributes were described above in the overview of grid correspondences.

[0084] The “MPEG_mesh_rigid_linking” extension presented in Table 1 expands the description of dependent meshes represented by MPEG-SD “mesh” objects. It defines the representation of the correspondence data between dependent meshes and the reference mesh pointed to by the extended “mesh” attribute.

[0085] These correspondences are described by the "Correspondence" attribute, an extension of "MPEG_mesh_rigid_linking," which defines an accessor providing the positions of exactly three bound vertices of the dependent mesh relative to the surface of the reference mesh. The indices of these three bound vertices in the dependent mesh are provided through the extended "correspondence_indices" attribute. The positions described in the "Correspondence" attribute can be represented as vertex-to-vertex or vertex-to-face correspondences, as described above. The choice of representation is specified by the correspondence_type attribute, whose allowed values ​​are provided in Table 2.

[0086] Table 3 describes the contents of the corresponding buffer data pointed to by the "correspondence" attribute. The buffer contains two pieces of information: • Parametric representation of the corresponding positions of the three bound vertices of the subordinate mesh on the reference mesh. This parameter representation is provided for each bound vertex based on the value of the "correspondence_type" attribute in the "MPEG_mesh_rigid_linking" extension, either as the index of the corresponding vertex on the reference mesh (correspondence_type = 0), or as the index of the corresponding face on the reference mesh and the barycentric coordinates of the corresponding point on that face (correspondence_type = 1).

[0087] Second, for each of the three bound vertices, the signed distance from the corresponding point of each bound vertex to the position of the bound vertex. The determination of the sign has been described above.

[0088] Example implementation In this example, we want to add a top hat to an animated virtual character. The MPEG-I SD description of the scene has a mesh for the virtual character and a mesh for the top hat. The top hat in this example is a rigid object. As the virtual character is animated and moves within its 3D environment, it is desired to attach the top hat mesh to the mesh on the virtual character's head. To do this, the three bound vertices of the top hat mesh are registered to the virtual character mesh using mapping data provided in the "MPEG_mesh_rigid_linking" extension of the glTF "mesh" object representing the top hat mesh. In this example, the mapping is encoded as a vertex-to-vertex mapping.

[0089] { "scene": 0, "nodes": [ { "name":"avatar", "mesh": 0, "extensions": { "MPEG_node_avatar": { "isAvatar":true, "type":"urn:my_avatar_type" } } }, { "name":"hat", "mesh": 1 } ], "scenes": [ { "nodes": [0,1] } ], "meshes": [ { "name":"full_avatar", "primitives": [ { "attributes": { "POSITION": 0, "TEXCOORD_0": 1, }, "indices": 2 } ] }, { "name":"hat", "primitives": [ { "attributes": { "POSITION": 3, "TEXCOORD_0": 4, }, "indices": 5 } ], "extensions": { "MPEG_mesh_rigid_linking": { "mesh": 0, "correspondence_indices": [17,39,87], "correspondence_type": 0, "correspondence": 6 } }, ], "extensionsUsed": [ "MPEG_node_avatar", "MPEG_mesh_rigid_linking” ], "extensionsRequired": [ "MPEG_node_avatar", "MPEG_mesh_rigid_linking" ] } In some embodiments utilizing MPEG-I SD, the mapping information between the subordinate mesh and the reference mesh can be encoded in the "MPEG_rigid_mesh_linking" extension of the subordinate mesh description, as described above.

[0090] The only scenario in this description contains two root nodes: one with the "MPEG_node_avatar" extension. virtual characters Nodes and Top hat Node. Note that the "MPEG_node_avatar" extension is provided for consistency with the presented use case, but the example implementation does not require this extension.

[0091] virtual characters The node points to grid #0 in the "grid" array and is named full_avatar And it refers to virtual characters. Top hat The node points to grid #1 in the same array and is named Top hat It also indicates that a rigid top hat object will be bound to the virtual character's geometry during animation. These two meshes have an attribute ("POSITION") describing the position of their vertices and an attribute ("TEXCOORD_0") describing their color via texture coordinates in the UV map. For both meshes, the "index" attribute has a triplet defining the vertex indices of each mesh triangle of the indexed mesh geometry.

[0092] To bind to full_avatar Grid Top hat The mesh has the "MPEG_rigid_mesh_linking" extension. This extension provides the following properties: • “Grid” refers to the grid at index 0 in the “Grid” array that represents the grid of the virtual character.

[0093] • “correspondence_indices” contains the indices of the three bound vertices of the hat mesh, whose positions on the reference mesh determine the binding between the two meshes.

[0094] • “correspondence_type” defines the nature of the correspondence data of the position of the top hat vertex on the grid of the virtual character. Here, 0 indicates a vertex-to-vertex correspondence (see Table 2).

[0095] • “Correspondence” refers to the vertex-to-vertex correspondence data pointing to the three bound vertices of the top hat mesh. Right now The accessor for the index of the nearest vertex on the virtual character's mesh, whose indices are specified by the "correspondence_indices" attribute.

[0096] For clarity, the accessor content is not shown in the example scenario description.

[0097] Example parsing method The parsing of the description in the "MPEG_mesh_rigid_linking" extension can be as follows: Figure 9 The process is illustrated in the block diagram. In the method for parsing glTF mesh nodes, it is determined whether the node includes the MPEG_rigid_mesh_linking extension. If it does not, no further action is required. Figure 9 The remaining operations are then performed. If such an extension exists, the correspondence_type, mesh, index, and correspondence attributes can be resolved. If the correspondence type is 0, the buffer pointed to by "correspondence" is resolved to provide vertex indices and optionally signed distances. If the correspondence type is 1, the buffer pointed to by "correspondence" is resolved to provide face indices, barycentric coordinate pairs, and optionally signed distances.

[0098] Example runtime processing In the example embodiment, at runtime, the demonstration engine uses a scene description that includes a "mesh" instance representing the 3D geometry of a reference mesh in the current frame and another "mesh" instance representing the 3D geometry of a dependent mesh in a predetermined "reference pose". The latter "mesh" instance representing the dependent mesh contains the "MPEG_mesh_rigid_linking" extension.

[0099] In each frame of the animation, the geometry of the dependent mesh is calculated in the "animation pose" where the dependent mesh is bound to the reference mesh. The dependent mesh can then be rendered in the current frame with the geometry relative to the reference mesh in the appropriate position and pose. This calculation can use references such as those described below. Figure 10 The block diagram describes the methods, etc., for execution.

[0100] At 110, the "mesh" instance, extended via the "MPEG_mesh_rigid_linking" extension, is resolved to obtain the positions of the vertices of the subordinate mesh in the reference pose. This can be performed before calculating and rendering the animation.

[0101] At 120, data in the buffer pointed to by the "correspondence_indices" accessor of the "MPEG_mesh_rigid_linking" extension of the "mesh" instance is parsed to determine the indices of the three bound vertices of the dependent mesh used to specify the correspondence. At 110, the positions of these bound vertices in the reference pose of the dependent mesh are found within the parsed dependent mesh data. This can be performed before calculating and rendering the animation.

[0102] At 130, the "correspondence_type" attribute in the "MPEG_mesh_rigid_linking" extension of the subordinate "mesh" is parsed, along with the corresponding data in the buffer pointed to by the "correspondence" accessor. This can be performed before calculating and rendering the animation.

[0103] At 140, the position of the reference mesh vertices in the current frame of the animation is obtained from the properties of the "mesh" instance extended by "MPEG_mesh_rigid_linking" and the animation properties of the frame.

[0104] At 150, for the current frame, the absolute positions of the three bound vertices of the dependent mesh in its animated pose are calculated based on the absolute positions of the vertices of the reference mesh obtained at 140 and the corresponding data resolved at 130.

[0105] At position 160, parameters for the rigid transformation mapping the reference pose of the dependent mesh to its animated pose are calculated for the current frame. This calculation can be based on the positions of the three bound vertices of the dependent mesh in the reference pose obtained at positions 110 and 120, and the corresponding positions of these three identical bound vertices in the animated pose calculated for the current frame at position 150. An example calculation that can be used for this process is described in more detail below.

[0106] At 170, the rigid transformation obtained at 160 is applied to the positions of all vertices of the dependent mesh resolved at 110 in the reference pose to calculate their positions in the animated pose. This determines the complete 3D geometry of the dependent mesh in the animated pose, which is bound to the reference mesh in the current frame.

[0107] Example processing for encoding In some embodiments, for encoding the animation, at least three (or exactly three) vertices of the dependent mesh are selected as bound vertices. The pose of the dependent mesh is selected relative to a reference mesh. The user can use authoring tools or other techniques to move the dependent mesh (e.g., using a rigid transformation) to various different poses relative to the reference mesh until an appropriate pose is selected (e.g., orienting the dependent mesh representing a top hat onto the reference mesh representing a head). The positions of the bound vertices are determined when the dependent mesh is in the selected pose. Correspondence information is determined, wherein for each bound vertex, the correspondence information indicates the position of the corresponding bound vertex relative to the reference mesh. It is not necessary to encode the correspondence information for all vertices of the dependent mesh, and in some embodiments, only three bound vertices are encoded. As described in more detail above, the vertex correspondence information may identify the corresponding nearest vertex of the reference mesh, the corresponding position on a face of the reference mesh, and / or the distance of each bound vertex to the surface of the reference mesh.

[0108] Using this type of implementation, animations that include both a reference mesh and a subordinate mesh can be encoded without explicitly encoding the motion information of the subordinate mesh. Instead, the pose of the subordinate mesh in each frame can be derived from rigging information (which can remain constant across multiple frames or even the entire animation) and the geometry of the reference mesh.

[0109] Example calculations for rigid object transformation This section provides additional details on calculations involving the transformation of rigid objects that can be performed in some embodiments. However, the embodiments disclosed herein are not limited to the use of the precise calculation techniques described below.

[0110] For use in example calculations, Figure 11 Two instances of a 3D rigid object (here, a top hat) are shown. The first instance, in "Reference Pose," provides the object with reference geometry. In the second instance, in "Animated Pose," the object has been moved and rotated to another position and pose, just as happens in the animation. (In Cartesian coordinate system) Oxyz The vector represents a 3D object. The surface of the object, representing a reference pose, is given by a vector. r 1 R , r 2 R and r 3 R Let three predetermined points be represented. Assume these points are not collinear. Let... r 1 A , r 2 A and r 3 ALet be the corresponding positions of these three points in the animated pose. It can be proven that a rigid transformation mapping the position of any point on the object's surface in the reference pose to its position in the animated pose can be calculated based on the corresponding positions of the three points in these two poses. For completeness, a brief overview of the proof is given below.

[0111] First, we introduce the following notation. A normalized coordinate set is defined by offsetting three points relative to their centroids in both the reference pose and the animated pose.

[0112] as follows:

[0113] And similarly

[0114] We further define two vectors in the reference pose and the animated pose, respectively. x i R and x i A ,as follows

[0115] and

[0116] The rigid transformation that maps points on the surface of an object in a reference pose to corresponding points on the surface of an object in an animated pose can be expressed as:

[0117] in R It is a 3D rotation matrix. t It is a 3D translation vector and s It is the scaling factor. Since the size of a rigid object does not change during animation, in this example, s equal 1 .

[0118] We want to calculate by minimizing the sum of the squared mapping errors at the three points. R and t –R( ) –t.

[0119] The sum of this squared mapping error E It can be written as a function of the center position, as follows:

[0120] The translation vector can be performed as follows: The determination of the square norm. Expanding the square norm into a dot product, E It can be rewritten as

[0121] The sum of the second term in this expression is zero because, according to the definition of the centroid,

[0122] And because rotation is a linear operator, the second term is 0. The first term does not depend on... t Minimizing the total error is equivalent to minimizing the third term, thus obtaining... t Optimal value t* ,as follows:

[0123] The rotation matrix can be performed as follows: The determination of the planes is as follows: Three points on the surface of a rigid object are not collinear, therefore they define one plane in the reference pose and another plane in the animated pose. We first determine the rotation that brings the former plane to the latter. R 1.

[0124] The normals of the planes for the reference pose and the animated pose are obtained as follows.

[0125] The line of intersection of the two planes is perpendicular to the two normals. Therefore, its direction is represented by a unit vector. .

[0126] From the reference pose plane to the animation pose plane in the range [0，π] Inner angle Φ It is also the rotation angle between the normals. Therefore, it is given by the following formula.

[0127] Therefore, the first rotation from the plane defined by the three object points in the reference pose to the plane defined by the corresponding three object points in the animated pose. R 1 is a unit vector along its axis a and rotation angle Φ Definition. Given a vector in 3D space v Its rotating version R 1 （v） This can be expressed using the Rodriguez rotation formula:

[0128] Using this formula, we define the vectors representing the positions of three points on the surface of the hat in the reference pose. , and They can be rotated to their corresponding positions in the plane of the three corresponding points in the animated pose. , and ( Figure 11 ).

[0129] After confirming R After step 1, the rotation matrix that maps the surface of the rigid object from its reference pose to its animated pose needs to be calculated. R The goal is to find the plane containing the animated pose. and Optimal alignment rotation R 2. In fact, R = R 2 .R 1. Therefore, we want to minimize

[0130] set up α j for and The angle between (see) Figure 12 ). Figure 12 The diagram illustrates the correspondence between three points after rotation from the reference pose plane to the animation pose plane. Using the cosine rule of triangles, each term of the above optimization criterion is expanded to...

[0131] set up θ In order to be in Figure 11 The rotation sought in the plane R Angle 2. When R 2. Applied to At that time, the criterion to be minimized becomes .

[0132] Minimizing this standard is equivalent to maximizing it.

[0133] in

[0134] By relative θ Setting the derivative to zero, we find that maximizing this expression... θ The value of makes

[0135] or

[0136] By utilizing these optimal values,

[0137] And we want to maximize this expression, so we choose the plus sign for calculation. θ The optimal value is defined. Figure 12 rotation in the plane R 2. Final rotation matrix R It can be determined as rotation R 1 and R The composite of 2 makes R = R 2 R 1.

[0138] Example System The encoding, decoding, processing, and rendering of the scenario described in this article can use methods such as... Figure 13 This is achieved through systems such as [system name missing]. Figure 13 This is a block diagram illustrating examples of systems implementing various aspects and embodiments. System 1000 may be embodied as a device including the various components described below and configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 1000 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0139] System 1000 includes at least one processor 1010 configured to execute instructions loaded thereon to implement various aspects described herein, such as those described. Processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1000 includes a storage device 1040 that may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 1040 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0140] System 1000 includes an encoder / decoder module 1030 configured to, for example, process data to provide encoded or decoded 3D scenes, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module that can be included in a device to perform encoding and / or decoding functions. It is well known that a device may include one or both encoding and decoding modules. Alternatively, the encoder / decoder module 1030 may be implemented as a separate element of system 1000, or it may be incorporated within processor 1010 as a combination of hardware and software known to those skilled in the art.

[0141] Program code to be loaded onto processor 1010 or encoder / decoder 1030 to execute the various aspects described herein may be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. According to various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more of various items during the execution of the processes described herein. Such stored items may include, but are not limited to, input scenes, decoding scenes or portions of decoding scenes, bitstreams, matrices, variables, and intermediate or final results from equations, formulas, operations, and operational logic processing.

[0142] In some embodiments, the memory within the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. External memory may be memory 1020 and / or storage device 1040, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Various Video Coding, i.e., a new standard developed by JVET (Joint Video Experts Group)).

[0143] Input to the components of system 1000 can be provided through various input devices, as indicated in box 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Other examples include composite video.

[0144] In various embodiments, the input device of block 1130 has corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting the signal band to a band), (ii) down-converting the selected signal, (iii) further band-limiting to a narrower band to select (e.g.,) a signal band that may be referred to as a channel in some embodiments), (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired data packet stream. The RF section of various embodiments includes one or more elements performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners performing various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and filtering again to the desired frequency band. Various embodiments rearrange the order of the above (and other) components, remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as insert amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0145] Additionally, USB and / or HDMI terminals may include corresponding interface processors for connecting system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, within a separate input processing IC or within processor 1010 as needed. Similarly, aspects of USB or HDMI interface processing may be implemented, as needed, within a separate interface IC or within processor 1010. The demodulated, error-corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 1010 and an encoder / decoder 1030 operating in conjunction with memory and storage elements, to process the data stream as needed for presentation on the output device.

[0146] Various components of system 1000 can be housed within an integrated housing. Within the integrated housing, the various components can be interconnected and transmit data between them using suitable connection devices 1140 (e.g., internal buses known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0147] System 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0148] In various embodiments, data is streamed to or otherwise provided to system 1000 using a wireless network, such as a Wi-Fi network, for example, IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). In these embodiments, the Wi-Fi signal is received via a communication channel 1060 and a communication interface 1050 adapted for Wi-Fi communication. The communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to system 1000, delivering data via an HDMI connection to input block 1130. Still other embodiments use an RF connection to input block 1130 to provide streaming data to system 1000. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0149] System 1000 can provide output signals to various output devices, including display 1100, speaker 1110, and other peripheral devices 1120. Display 1100 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1100 can be used in televisions, tablets, laptops, mobile phones, or other devices. Display 1100 can also be integrated with other components (e.g., in a smartphone) or standalone (e.g., an external monitor for a laptop computer). In various examples of embodiments, other peripheral devices 1120 include one or more of a standalone digital video disc (or digital versatile disc) (DVR, for both terms), an optical disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 to provide functionality based on the output of system 1000. For example, an optical disc player performs the function of playing the output of system 1000.

[0150] In various embodiments, signaling (such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention) is used to transmit control signals between system 1000 and display 1100, speaker 1110, or other peripheral devices 1120. Output devices may be communicatively coupled to system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to system 1000 via communication interface 1050 using communication channel 1060. Display 1100 and speaker 1110 may be integrated into a single unit along with other components of system 1000 in an electronic device, such as a television. In various embodiments, display interface 1070 includes a display driver, such as a timing controller (TCon) chip.

[0151] For example, if the RF input section 1130 is part of a separate set-top box, the display 1100 and speaker 1110 can alternatively be separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal can be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0152] System 1000 may include one or more sensor devices 1095. Examples of sensor devices that may be used include one or more GPS sensors, gyroscope sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors can be used to determine information such as the user's position and orientation. Where system 1000 is used as a control module (such as control modules 124, 1254) for an augmented reality display, the user's position and orientation can be used to determine how to render image data so that the user perceives the correct portion of a virtual object or scene from the correct perspective. In the case of a head-mounted display device, the position and orientation of the device itself can be used to determine the user's position and orientation for rendering virtual content. In the case of other display devices (such as telephones, tablets, computer monitors, or televisions), other inputs can be used to determine the user's position and orientation for rendering content. For example, a user can select and / or adjust the desired viewpoint and / or viewing direction by using a touchscreen, keypad or keyboard, trackball, joystick, or other inputs. Where the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation can be selected and / or adjusted based on the movement of the display device for rendering content.

[0153] The embodiments can be executed by processor 1010 or by computer software implemented by hardware or a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, memory 1020 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 1010 can be of any type suitable for the technical environment and can encompass one or more microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0154] Another embodiment. This disclosure describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described in a specific manner and are generally described in a way that may sound limiting, at least for the purpose of illustrating the various features. However, this is for the purpose of clarity of description and does not limit the disclosure or scope of these aspects. In fact, all the different aspects can be combined and interchanged to provide other aspects. Furthermore, these aspects can also be combined and interchanged with aspects described in previous documents.

[0155] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are specifically shown, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of implementations. At least one aspect generally relates to 3D scene encoding and decoding, and at least one other aspect generally relates to transmitting the generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions thereon stored thereon for encoding or decoding scene description data according to any of the methods, and / or computer-readable storage media having bitstreams generated according to any of the methods stored thereon.

[0156] This document describes various methods, each of which includes one or more steps or actions to implement the described method. The order and / or use of specific steps and / or actions can be modified or combined unless the correct operation of the method requires a specific order of steps or actions. Additionally, in various embodiments, terms such as "first," "second," etc., may be used to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless specifically required, the use of such terms does not imply an ordering of the modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding, but can occur, for example, before, during, or in a time period overlapping with the second decoding.

[0157] For example, various numerical values ​​may be used in this disclosure. Specific values ​​are for illustrative purposes, and the aspects described are not limited to these specific values.

[0158] The embodiments described herein can be executed by computer software implemented by a processor or other hardware or a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, the processor can be of any type suitable for the technical environment and can encompass one or more microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0159] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0160] The implementations and aspects described herein can be implemented, for example, in methods or processes, apparatuses, software programs, data streams, or signals. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), the implementation of the features in question can also be implemented in other forms (e.g., apparatuses or programs). Apparatuses can be implemented, for example, with appropriate hardware, software, and firmware. Methods can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0161] The references to "an embodiment" or "an embodiment" or "an implementation" or "an implementation," and other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the phrases "in an embodiment" or "in an embodiment" or "in an implementation" or "in an implementation," and any other variations appearing throughout this disclosure, do not necessarily refer to the same embodiment.

[0162] Additionally, this disclosure may relate to “determining” various types of information. Determining information may include one or more of the following: for example, estimated information, calculated information, predicted information, or information retrieved from memory.

[0163] Furthermore, this disclosure may relate to “accessing” various types of information. Accessing information may include one or more of the following: for example, receiving information, retrieving information (e.g., retrieving from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0164] Additionally, this disclosure may relate to "receiving" various types of information. Like "access," the intent to receive is a broad term. Receiving information may include one or more of the following: for example, accessing information or retrieving information (e.g., retrieving from memory). Furthermore, "receiving" is generally referred to in one or more ways during operation, such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0165] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As yet another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” this wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). This can be extended to as many as the items listed.

[0166] Furthermore, as used herein, the term “signaling” specifically refers to instructing the corresponding decoder to do something. For example, in some embodiments, the encoder signals a particular parameter among several parameters for region-based filter parameter selection for artifact removal filtering. In this way, in embodiments, the same parameter is used at both the encoder and decoder sides. Thus, for example, the encoder can send (explicit signaling) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling can be used without sending (implicit signaling) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various embodiments by avoiding the transmission of any actual functionality. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntactic elements, flags, etc., are used to send information to the corresponding decoder. Although the verb form of the word “signaling” has been referred to above, the word “signal” can also be used as a noun herein.

[0167] Implementations can generate various signals that are formatted to carry information, such as information that can be stored or transmitted. The information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, the signal may be formatted to carry a bitstream of the described embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding the data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0168] Several embodiments are described. Features of these embodiments may be provided individually or in any combination across various claim classes and types. Furthermore, embodiments may include one or more of the following features, devices, or aspects individually or in any combination across various claim classes and types: • A bitstream or signal that includes one or more of the described syntactic elements or their variants.

[0169] • A bitstream or signal, which includes syntax for conveying information generated according to any embodiment of the described embodiments.

[0170] • Create and / or send and / or receive and / or decode bitstreams or signals, which include one or more of the described syntactic elements or their variants.

[0171] • Create and / or send and / or receive and / or decode according to any of the described embodiments.

[0172] • A method, process, apparatus, medium for storing instructions, medium for storing data, or signal according to any of the described embodiments.

[0173] It should be noted that the various hardware elements in one or more of the described embodiments may be referred to as “modules”, which implement (i.e., perform, execute, etc.) the various functions described herein in conjunction with the respective modules. As used herein, a module includes hardware considered suitable for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices). Each described module may also include executable instructions for performing one or more functions described as being performed by the respective module, and it should be noted that these instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., and may be stored in any suitable non-transitory computer-readable medium such as RAM, ROM, etc., commonly referred to as RAM, ROM, etc.

[0174] Although the features and elements have been described above in specific combinations, each feature or element may be used alone or in any combination with other features and elements. Furthermore, the methods described herein can be implemented in a computer program, software, or firmware incorporated into a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM discs and digital multifunction disks (DVDs). The processor associated with the software can be used to implement a radio frequency transceiver for a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method comprising: Obtain information defining a reference mesh and a dependent mesh, wherein the dependent mesh includes at least three bound vertices; Obtain correspondence information that can be used to determine the corresponding animated position relative to the reference mesh for each of the bound vertices; Determine the first animated pose of the reference mesh in at least the first frame of the animation; The first animated position of the bound vertex is determined based on the first animated pose of the reference mesh and the correspondence information; A first transformation is determined, wherein the first transformation is a transformation that changes the bound vertex of the dependent mesh toward the corresponding first animated position of the bound vertex; as well as The first animation pose of the subordinate mesh is determined by applying the first transformation to the subordinate mesh.

2. An apparatus comprising one or more processors, said one or more processors being configured to perform at least: Obtain information defining a reference mesh and a dependent mesh, wherein the dependent mesh includes at least three bound vertices; Obtain correspondence information that can be used to determine the corresponding animated position relative to the reference mesh for each of the bound vertices; Determine the first animated pose of the reference mesh in at least the first frame of the animation; The first animated position of the bound vertex is determined based on the first animated pose of the reference mesh and the correspondence information; A first transformation is determined, wherein the first transformation is a transformation that changes the bound vertex of the dependent mesh toward the corresponding first animated position of the bound vertex; as well as The first animation pose of the subordinate mesh is determined by applying the first transformation to the subordinate mesh.

3. The method of claim 1, or the apparatus of claim 2, further comprising rendering the first frame of the animation using the first animated pose of the reference mesh and the first animated pose of the dependent mesh.

4. The method according to claim 1 or claim 3 which is dependent on claim 1, or the apparatus according to claim 2 or claim 3 which is dependent on claim 2, wherein the method or the apparatus further comprises: A second animated pose of the reference mesh is determined in at least the second frame of the animation; The second animated position of the bound vertex is determined based on the second animated pose of the reference mesh and the correspondence information; Determine a second transformation, which is a transformation that changes the bound vertex of the dependent mesh toward the corresponding second animated position of the bound vertex; as well as The second animated pose of the subordinate mesh is determined by applying the first transformation to the subordinate mesh.

5. The method according to claim 4 of claim 1, or the apparatus according to claim 4 of claim 2, wherein the method or the apparatus further comprises rendering a second frame of the animation, the second frame including the second animated pose of the reference mesh and the second animated pose of the dependent mesh.

6. The method according to claim 1 or claims 3 to 5 which are dependent on claim 1, or the apparatus according to claim 2 or claims 3 to 5 which are dependent on claim 2, wherein, The first transformation is a rigid transformation.

7. The method according to claim 1 or claims 3 to 6 which are dependent on claim 1, or the apparatus according to claim 2 or claims 3 to 6 which are dependent on claim 2, wherein, The first transformation includes rotation and translation.

8. A method comprising: Select at least three vertices of the subordinate mesh as bound vertices; Select the orientation of the subordinate mesh relative to the reference mesh; as well as The correspondence information is encoded, which indicates the position of the corresponding bound vertex relative to the reference grid when the subordinate grid is in the selected pose.

9. An apparatus comprising one or more processors, said one or more processors being configured to perform at least: Select at least three vertices of the subordinate mesh as bound vertices; Select the orientation of the subordinate mesh relative to the reference mesh; as well as The correspondence information is encoded, which indicates the position of the corresponding bound vertex relative to the reference grid when the subordinate grid is in the selected pose.

10. The method according to claim 8, or the apparatus according to claim 9, wherein, Select fewer than all vertices on the dependent mesh as bound vertices.

11. The method according to claim 8 or claim 10 which is a dependent of claim 8, or the apparatus according to claim 9 or claim 10 which is a dependent of claim 9, wherein, Only three vertices of the dependent mesh are selected as the bound vertices.

12. The method according to claim 8 or claims 10 to 11 which are dependent on claim 8, or the apparatus according to claim 9 or claims 10 to 11 which are dependent on claim 9, wherein the method or the apparatus further comprises encoding an animation of a scene including the reference mesh and the dependent mesh.

13. The method according to claim 1 or claims 3 to 7 of claim 1, or the method according to claim 8 or claims 10 to 12 of claim 8, or the apparatus according to claim 2 or claims 3 to 7 of claim 2, or the apparatus according to claim 9 or claims 10 to 12 of claim 8, wherein, For at least one of the bound vertices, the correspondence information identifies the corresponding nearest vertex of the reference mesh.

14. The method according to claim 1 or claims 3 to 7 of claim 1, or the method according to claim 8 or claims 10 to 12 of claim 8, or the apparatus according to claim 2 or claims 3 to 7 of claim 2, or the apparatus according to claim 9 or claims 10 to 12 of claim 8, wherein, For at least one of the bound vertices, the correspondence information identifies the corresponding position on the face of the reference mesh.

15. The method according to claim 1 or claims 3 to 7 of claim 1, or the method according to claim 8 or claims 10 to 12 of claim 8, or the apparatus according to claim 2 or claims 3 to 7 of claim 2, or the apparatus according to claim 9 or claims 10 to 12 of claim 8, wherein, The correspondence information also includes distance information, which indicates the distance between at least one of the bound vertices and the reference mesh.

Images