Virtualized avatar controller in a scene description

Abstract

In an example embodiment, a runtime asset delivery file for a 3D scene, such as a glTF file, includes a plurality of first syntax structures each defining an association between an input weight value and a corresponding transform of at least one node among a plurality of nodes in the scene, and at least one second syntax structure associating a respective semantic label with at least one first syntax structure. In an example method, such a runtime asset delivery file is obtained; at least one of the first syntax structures is selected based on the associated semantic label; and a transform indicated by the selected first syntax structure is applied to the node.

Description

[0001] Cross-references This application claims the benefit of European Patent Application No. 24305458.2, filed on March 27, 2024, the entire disclosure of which is incorporated herein by reference. Background Technology

[0002] This disclosure relates to the encoding of virtual avatar 3D models in scene descriptions.

[0003] Various technologies can be used to generate, process, and render virtual 3D scenes. The information characterizing a 3D scene (called a scene description) can be time-dependent, allowing the 3D scene to change in a manner similar to video playback. This behavior can be achieved by relying on a framework defined in the MPEG media scene description document (Information technology - Coded representation of immersive media - Part 14: Scene Description for MPEG media, ISO / IEC DIS23090-14:2021 (E)). Scene update mechanisms based on JSON patching protocols as defined in IETF RFC 6902 can be used to synchronize virtual content to MPEG media streams.

[0004] The current MPEG-I Scene Description (SD) format allows animation to be stored in "Animation" assets within glTF files. Each animation defines a named function that can modify the mesh given a time between zero and a maximum value. A common use of these animations is to provide different actions for the mesh, such as a character with walking, running, and idle animations. Another use is to provide animations that result in different cinematic art styles (e.g., like a short film, but rendered in real time).

[0005] Current animation coding allows functions to modify one or more properties of a mesh, such as translation, rotation, or mesh target weights. Extensions like KHR_animation_pointers can be used to change even more properties. Each animation can have a name, but there is no standard to define the meaning of these names. Summary of the Invention

[0006] A method according to some embodiments includes obtaining a runtime asset delivery file for a 3D scene, the file including: a plurality of first syntactic structures, each first syntactic structure defining an association between input weight values ​​and corresponding transformations of at least one of a plurality of nodes in the scene; and at least one second syntactic structure, the at least one second syntactic structure associating corresponding semantic tags with at least one of the first syntactic structures. The method further includes: selecting at least one first syntactic structure based on the associated semantic tags; and applying a transformation indicated by the selected first syntactic structure.

[0007] An apparatus according to some embodiments includes one or more processors, configured to at least perform: obtaining a runtime asset delivery file for a 3D scene, the file including: a plurality of first syntactic structures, each first syntactic structure defining an association between input weight values ​​and corresponding transformations of at least one of a plurality of nodes in the scene; and at least one second syntactic structure associating corresponding semantic tags with at least one of the first syntactic structures. The apparatus is further configured to: select at least one of the first syntactic structures based on the associated semantic tags; and apply a transformation indicated by the selected first syntactic structure.

[0008] In some embodiments, the first syntax structure is the controller structure in the MPEG_controllers extension.

[0009] In some embodiments, the file further includes a third syntactic structure associated with nodes in the scene, wherein the third syntactic structure includes a second syntactic structure, and wherein the transformation corresponding to the first syntactic structure is a transformation of the mesh in the node.

[0010] In some embodiments, the third syntax structure includes information indicating that the node represents a virtual avatar. In some embodiments, the third syntax structure is an extension of MPEG_node_avatar.

[0011] In some embodiments, the third syntax structure is an extension of MPEG_node_controllers.

[0012] In some embodiments, the document further includes a fourth syntactic structure that associates the corresponding semantic tag with a set including a plurality of first syntactic structures.

[0013] In some embodiments, each of the first syntactic structures in the first syntactic structure has a corresponding index, and wherein the second syntactic structure includes an index of at least one of the first syntactic structures in the first syntactic structure.

[0014] In some embodiments, the runtime asset delivery file is a glTF file.

[0015] In some embodiments, semantic tags include usage attributes.

[0016] Some embodiments further include animing at least the nodes associated with the selected first syntactic structure by applying the varying weights to the transformation corresponding to the selected first syntactic structure.

[0017] Some embodiments further include causing at least the node associated with the selected first syntactic structure to be displayed.

[0018] Some embodiments further include: obtaining a plurality of runtime asset delivery files, each runtime asset delivery file including: a corresponding node representing a virtual avatar, at least one first syntactic structure defining the association between input weight values ​​and corresponding transformations of the corresponding virtual avatar, and at least one second syntactic structure associating a corresponding semantic tag with the first syntactic structure; presenting a virtual environment including the virtual avatars of the corresponding runtime asset delivery files; selecting an action to be performed on the selected virtual avatar; identifying the first syntactic structure corresponding to the selected action using the semantic tag of the second syntactic structure associated with the selected virtual avatar; and applying the transformation indicated by the identified first syntactic structure.

[0019] According to some embodiments, a computer-readable medium stores a runtime asset delivery file for a 3D scene, the file comprising: a plurality of first syntactic structures, each first syntactic structure defining an association between an input weight value and a corresponding transformation of at least one of a plurality of nodes in the scene; and at least one second syntactic structure associating a corresponding semantic tag with at least one of the first syntactic structures.

[0020] The method according to some embodiments includes encoding runtime asset delivery files according to any embodiment of the embodiments described herein.

[0021] The device according to some embodiments includes one or more processors configured to encode, decode, and / or parse runtime asset delivery files according to any of the embodiments described herein. Attached Figure Description

[0022] Figure 1A-1C This describes a virtual avatar, where the controller is used to move the virtual avatar's eyes from the right side of the avatar's gaze ( Figure 1A Move to gaze ahead ( Figure 1B ), to gaze to the left ( Figure 1C ).

[0023] Figure 2A-2CThis describes a virtual avatar, where a controller is used to move the avatar's eyes from a downward gaze ( Figure 2A Move to gaze ahead ( Figure 2B ), to looking upwards ( Figure 2C ).

[0024] Figure 3 This is a flowchart illustrating a method for resolving nodes according to an example embodiment.

[0025] Figure 4 This is a flowchart illustrating a method for parsing a set of virtual avatar controllers according to an example embodiment.

[0026] Figure 5 This is a flowchart illustrating a method for parsing a virtual avatar controller according to an example embodiment.

[0027] Figure 6 It is a schematic illustration of the logical structure of the relevant syntactic elements in the runtime asset delivery file.

[0028] Figure 7 This is a schematic illustration of the logical structure of relevant syntactic elements in a runtime asset delivery file, including semantic tags associated with a controller, according to the embodiments described herein.

[0029] Figure 9 This is a flowchart illustrating a method for delivering files using several runtime assets according to an example embodiment.

[0030] Figure 10 This is a functional block diagram illustrating an example of a device that can be used to encode, decode, and / or process scene description files according to an example embodiment. Detailed Implementation

[0031] The current scene description format does not clearly define how animations can be combined, especially for modifying or animates of an avatar's appearance. For example, if a user wants to use two animations of an avatar at once, the current scene description format does not clearly define how the properties that are changing should be updated in the order the animations begin.

[0032] This disclosure describes devices and methods for encoding virtual avatars with controllers, and how to use them. The controller can be described as a high-level controllable animation, for example, a function that modifies the virtual avatar model given one or more weights.

[0033] The example embodiments can be implemented in an environment utilizing the current MPEG-I Scene Description (SD) format along with the “MPEG_controllers” extension as described in European Patent Application No. EP 23306167.0, filed July 10, 2023. The example embodiments can further utilize the “MPEG_node_avatar” extension as described in European Patent Application No. EP23305405.5, filed March 24, 2023.

[0034] In traditional glTF systems, if someone provides additional information, the application can use a virtual avatar with controllers included in the glTF file. For example, an application can load a glTF file with virtual avatars and controllers, and let the user define which controllers to use and for what purpose. For instance, if a controller makes the virtual avatar move, the user must know or guess it.

[0035] One goal of virtual avatar coding in scene descriptions is to allow the exchange of virtual avatar instances in heterogeneous environments. For example, a person might want the ability to bring their virtual avatar into a virtual world and move around. A user might want to change their facial expressions or identity. Controllers can be used to implement these features. However, the application must know which controllers are associated with which movements or expressions and what weights to use.

[0036] In the current description, the proposed format conforms to the glTF format and is compatible with recent MPEG-I SD works that extend glTF using MPEG-I SD extensions. Although glTF is used herein to describe example embodiments, it should be understood that other embodiments based on the same principles can be implemented using other formats (e.g., XML, USD, etc.).

[0037] Overview of Virtual Avatar Signaling In the current MPEG format, no virtual avatar is defined, and a virtual avatar can take any form, such as a humanoid and non-humanoid version, or it can take the shape of any 3D object with the attribute "isAvatar" set to "true". Therefore, it is helpful for applications to understand what to expect from "MPEG_node_avatar". In some embodiments, it may be expected that the virtual avatar conforms to some or all of the following characteristics: • Virtual avatars are represented by the node “MPEG_node_avatar”, and roles outside this scope may not qualify as “virtual avatars” under the MPEG-I SD standard. • Reconstruction and animation conform to MPEG scene descriptions and / or primitives supported in other standards and animation formats. • A virtual avatar can be explicitly represented, or not, through a virtual avatar type attribute that may be accompanied by a corresponding URI to the 3D asset. • If the virtual avatar has a virtual avatar type attribute set, the default can be "MPEG_reference_avatar". In some embodiments, the type of virtual avatar representation is provided as a URN that uniquely identifies the virtual avatar representation scheme. The virtual avatar representation scheme can define the format of all components used to reconstruct and animate the virtual avatar.

[0038] In some embodiments, given some user input, the avatar type is reconstructed and redirected to a reference 3D avatar model, which may include, for example, a full-body mesh representation, skeletal and skinning weights, facial blending shapes and landmarks, and additional geometry including eyes, chin, teeth, and tongue. Such embodiments allow for a shared geometry and feature representation for any humanoid character. Appropriate deformation allows for shape adaptation. This common underlying representation facilitates animation and representation across different forms or contexts.

[0039] In some embodiments, an extension such as the "MPEG_node_avatar" extension can be used to identify the virtual avatar being used. Given such an extension attribute, the client will know which virtual avatar to rebuild / render.

[0040] In some embodiments, attributes such as isAvatar can be used with syntax and semantics such as the following: .

[0041] In some embodiments, the MPEG_node_avatar extension may have the following syntax and semantics: .

[0042] In some embodiments, a mapping object may have the following syntax and semantics: .

[0043] Overview of controllers and animations In glTF, animations are conventionally structured in JSON (JavaScript Object Notation) format as follows. The "Animations" array contains one or more "Animation" objects. Each "Animation" object includes information identifying the component to be animated (called the "target") and information identifying the corresponding data used to perform the animation (called the "sampler"). This connection between the animation target and the sampler is established in the "Channel" object. The Channel object further includes information indicating how the data affects the animation target. For example, in the case where the animation target is a mesh, the Channel object can indicate whether the animation data is used to translate the mesh, rotate the mesh, or change the displacement of certain vertices of the mesh using "deformation target weights" already defined for the mesh.

[0044] The "sampler" object, which provides data for animation, has "input" and "output" properties. The data associated with the "output" property indicates the transformation to be applied to the components of the scene. For example, in the case of a translation, the "output" data can provide components of different vectors representing different displacements. In the case of a rotation, the "output" data can provide components of different rotation matrices. In the case of a transformation applied to a deformable target, the "output" data can provide different weights. The "input" property is used to help determine which of these output vectors, matrices, or weights should be applied to the relevant components of the scene. The input property can correspond to a list of times. As an example, a sampler can identify inputs with five different time values ​​and outputs with five different rotation matrices. Heuristically, these can be thought of as columns of a table, where the system performing the animation looks up the current time (e.g., the time since the start of the autodraw) in the "input" column and finds the appropriate rotation matrix in the "output" column. The time explicitly listed in the "input" column is called the time of the animation keyframe. When the current time is not one of the times explicitly listed in the "input," interpolation is used, where the type of interpolation (e.g., "step," "linear," or "cubic spline") is indicated in the sampler object.

[0045] (More specifically, the characteristics of the "input" and "output" properties of a "sampler" object are not necessarily the relevant input or output data itself; instead, the characteristics can be indexes to objects called "accessors," which are used to access the relevant data through one or more additional layers of abstraction. However, such details are not necessary for understanding the principles described in this paper.) A "controller" can be described as an object that resembles an animation but transforms mesh properties given input weights instead of input time. For example, a controller object can include one or more sampler objects with corresponding inputs and outputs, and one or more channel objects that associate these samplers with corresponding mesh properties. Given input weights, each sampler provides a corresponding transformation as its output, and this transformation can be applied to the appropriate mesh property identified by the channel object. Controller objects offer greater flexibility than animation objects, at least in part because controller weights are inherently not functions of time (although controller weights can be animated as functions of time). Therefore, the details of controller-mediated transformations can be modified independently of the timeline of the entire scene animation.

[0046] The current MPEG_Controllers extension has the following characteristics: • Controller: Each controller defines a function that changes the mesh properties by giving a weight between a minimum and a maximum value. • Controller weights: Each value defines the weight of each controller.

[0047] Example embodiments may optionally include components that define a user interface for a controller, which can be used to change one or more controller weights.

[0048] The “MPEG_controllers” attribute can be described as shown in Table 1.

[0049] Table 1 describes the characteristics of the MPEG_controllers attributes, including the "ui" attribute. Note that in this and all other tables in this disclosure, the column "Required" (or M for "Mandatory") indicates whether a feature is required or mandatory under the specific syntax described in the table. The indication that a particular feature is "required" or "mandatory" for that particular syntax does not mean that the feature is required in all embodiments. There may be other embodiments in which such features are not required.

[0050] In the example embodiment, the number of items in "controller" and "weight" is constrained to be the same.

[0051] In the example implementation, each controller in the "controllers" list of MPEG_controllers defines a function that changes one or more properties of the mesh. For example, in some implementations, the controller changes the position of the mesh vertices and thus models the deformation of the mesh.

[0052] The shape of the function associated with the controller is defined by a set of (input weights, output attribute changes) values. These values ​​can be encoded for compatibility with the keyframe scheme used in the corresponding animation (e.g., glTF animation). Keyframe timings pointed to by sampler input features are mapped to function input weights. Animation values ​​pointed to by channel target features are mapped to output attribute change values. Thus, animation frameworks (e.g., glTF animation frameworks) are used to shape each controller as a possible non-linear function of input weight control values.

[0053] In some embodiments, controllers are defined in the runtime asset delivery file. A controller is defined through an instance of the "Controller" attribute. A controller attribute can include one or more attributes. The "Required" column indicates the attribute that is required in at least some implementations of the controller attribute, although the same attribute may not be required in other implementations.

[0054] In some embodiments, the controller property has the same characteristics as the standard animation property of glTF, plus two additional properties: "animation" and "minimum".

[0055] In some embodiments, there are two modes for this attribute, depending on the definition of the "Animation" attribute. If this attribute does not exist, the controller directly defines the function; otherwise, it uses the function of an existing animation. In some implementations, the controller definition is constrained to be in one of two modes, where other combinations result in an error. In all modes, the "Name" attribute can define the name of the controller. If it is not defined, the application can select a name based on the controller's index, such as "controller_23" or "controller_0023".

[0056] In some embodiments, the "Animation" property is not defined in the controller; in this case, the controller directly defines its function. The "Minimum" property is ignored in this case. All other properties ("Channel," "Sampler," "Name," "Extended," "Extra") can be used based on those in the standard animation properties. Common animation parsers can be run, except that time is treated as an input weight value, and its minimum value can be negative.

[0057] In other embodiments, an "animation" property is defined in the controller. In such embodiments, the controller may not define its own function, but instead use the animation's function. In this case, the following properties can be omitted: "channel" and "sampler". However, the properties "name", "extended", and "additional" can still be used when defining the animation property.

[0058] The "Animation" property defines the index of the animation within the "Animation" asset. In this case, the function defined by the animation is used as the controller, except that time is treated as an input weight value, and its minimum value is shifted by the value of the "Minimum" property. For example, if the "Minimum" is -1.3, the animation time is reduced by -1.3, as if the animation started at -1.3 seconds. This property allows defining controllers with negative input weight values ​​while referencing glTF animations whose time cannot be negative.

[0059] In some embodiments, the “weights” property of MPEG_controllers contains an array of numbers (e.g., floating-point values) that define the weights associated with each controller. The weights can be provided in the same order as the controllers in the “controllers” list: weights[0] for controllers[0], weights[1] for controllers[1], and so on.

[0060] These weights provide information about how nodes targeted by the controller change. In some embodiments, each weight is used to apply its corresponding controller, iterating from the first to the last. For example, weights[0] can first be used to apply the functions of controllers[0] to update the nodes. Then, weights[1] can be used to modify these updated nodes through the functions of controllers[1], and so on. These updates can be expressed as follows: in f i It is a function of controllers[i]. w i It is weights[i], scene in It is the initial node and scene out It is a node that uses controllers and weights for updates.

[0061] In some embodiments, controller changes occur at the same steps as the animation: they are applied to the initial node content and before any transformations in other glTF properties (like those in the “node” asset).

[0062] In some embodiments, the controller is not activated when glTF parsing is complete; similar to animation, the application selects whether to enable the controller.

[0063] In some embodiments, the controller is not used with the animation; the application can choose to enable either the controller or the animation.

[0064] In some embodiments, an extension such as KHR_animation_pointer is used to animate the controller. In this case, the extension is used to define animations to animate the controller weights (similar to the use of deformable target weights).

[0065] Signaling using controller set In the example embodiment, a new list of "controller sets" is added to the MPEG_node_avatar extension. An MPEG_node_avatar extension according to such an embodiment may have the following syntax and semantics: Table 3 - MPEG_node_avatar attribute with a new list of "controller sets".

[0066] In this and other tables used herein, the indication that a parameter is “required” indicates only that it is required by the syntax of a particular illustrated embodiment; there may be other embodiments in which the syntax does not require the parameter.

[0067] Each item in the "Controller Sets" list defines a controller set for the avatar and provides one or more semantic tags indicating how to use the controller set. A avatar controller set reorganizes several avatar controllers for a given purpose; for example, one avatar controller set might be dedicated to avatar movement (walking, running, idling, etc.), while another avatar controller set might be dedicated to facial expressions. Syntax elements in the scene description file can be used to associate semantic tags with different controllers and / or sets of controllers.

[0068] In some embodiments, the virtual avatar controller set is replaced with AvatarController The list is provided below. These embodiments offer fewer features because there is a single set of virtualization controllers, but they can use less memory.

[0069] In the example embodiment, it is defined as shown in Table 4. AvatarControllerSet property.

[0070] In the example embodiment, the "Description" attribute describes the collection of virtual avatar controllers and their usage. This is merely illustrative and, for example, can be used in the user interface to help users understand what the controller collection is intended for.

[0071] In an example embodiment, the "Purpose" attribute contains a string identifying the purpose of the collection of virtualization controllers. In some embodiments, the purpose attribute is constrained to follow predetermined values ​​defined by the application or standards. In some embodiments, this attribute is of type enumeration (or "enum") variable type. The "Purpose" attribute and / or the "Description" attribute can be used as semantic tags to provide information that allows the system to associate the collection of controllers with desired operations.

[0072] In the example embodiment, as described in more detail below, the “controller” list defines the controllers to be used and indicates how to use them.

[0073] In the example implementation, the "Weight" attribute defines the default controller weights to be used. The length of this array can be constrained to be equal to the number of items in "Controller". These controller weights can replace the corresponding weights in the "Weight" attribute of "MPEG_controllers".

[0074] In the example embodiment, AvatarController Attributes can have syntax and semantics as shown in Table 5.

[0075] In the example embodiment, the "Description" attribute describes the virtual avatar controller and its usage. This is merely illustrative and, for example, can be used in a user interface to help users understand what the controller is intended for. The description may include human-readable text that is not necessarily constrained to predetermined values.

[0076] In an example embodiment, the "Purpose" attribute contains a string identifying the purpose of the virtualization controller. It may be constrained to follow predetermined values ​​defined by the application or standards. In some embodiments, this attribute is of type enumeration (or "enum") variable type. The "Purpose" attribute and / or the "Description" attribute can be used as semantic labels to provide information that allows the system to associate a controller with the desired operation. Where a "Purpose" attribute does not exist for a particular controller, the controller may be assigned the "Purpose" attribute of the corresponding controller set to which it belongs. Therefore, the semantic label associated with the controller set can be used as the default semantic label for controllers within the set.

[0077] In the example embodiment, the "controller" attribute references a controller in the "controller" list of the "MPEG_controllers" extension.

[0078] In the example implementation, the "range" attribute defines the minimum and maximum values ​​of the controller weights.

[0079] In some embodiments, the controller set information described above can be provided to non-virtualized nodes, instead of providing such information to virtualized nodes, or other than providing such information to virtualized nodes. As an example, a controller set can be added to a glTF node by introducing a new "MPEG_node_controllers" extension. This allows a controller set to be added to any glTF node. For clarity, this document follows the glTF format and describes such embodiments with compatibility with the use of MPEG-ISD extensions. However, it should be understood that the principles described herein are not limited to the use of glTF, and the principles described herein can be implemented using other encoding formats (e.g., XML, USD, etc.).

[0080] In some embodiments, the "MPEG_node_controllers" extension can be used to implement a set of controllers for nodes (which may or may not be avatar nodes) in a scene description file. In an example embodiment, such an extension can be used to extend any glTF node. This extension can be used to define a set of controllers that an application can use to animate the mesh in the node. In an example embodiment, data in these controller sets includes one or more semantic tags that define the purpose of the controllers. The application can read such semantic tags to identify which controller should be used to implement the desired function.

[0081] In the example embodiment, the controller set is implemented as an extension of "MPEG_node_controllers" with the following attributes: Table 6 - Attributes of the MPEG_node_controllers extension.

[0082] The "Description" attribute describes the controller collection and its usage. This is purely illustrative; for example, the description attribute can be used in the user interface to help users understand the expected functions of the controller collection.

[0083] The "Purpose" attribute contains a string identifying the purpose of the controller set. In some embodiments, this attribute is of type enumeration (or "enum") variable type. The "Purpose" attribute and / or the "Description" attribute can be used as semantic labels to provide information that allows the system to associate controllers with desired operations. Where a "Purpose" attribute does not exist for a particular controller, the controller can be assigned the "Purpose" attribute of the corresponding controller set to which it belongs. Therefore, the semantic label associated with a controller set can be used as the default semantic label for controllers within the set.

[0084] Each item in the "Controller Collections" list defines a controller collection for a node and how to use it. A controller collection provides a set of one or more controllers for a given purpose. A controller can belong to multiple controller collections. (And some controllers may not belong to any controller collection.) In some embodiments, the controller set is replaced by a list of “controller” attributes as described below. Such embodiments offer fewer features because there is no grouping of controllers (except that they are included in a list), but such embodiments can use less memory.

[0085] In the example embodiment, Controller set The attribute is defined as follows: Table 7 - Controller Collection Attributes.

[0086] The "Description" attribute describes the controller collection and its usage. This is purely illustrative; for example, it can be used in the user interface to help users understand the expected functions implemented by the controller collection.

[0087] The "Purpose" attribute in this embodiment contains a string identifying the purpose of the controller set. It must conform to a predetermined value defined by the application or standard. In some embodiments, this attribute is of type enumeration (or "enum") variable type. The "Purpose" attribute and / or the "Description" attribute can be used as semantic labels to provide information that allows the system to associate controllers with desired operations. Where a "Purpose" attribute does not exist for a particular controller, the controller can be assigned the "Purpose" attribute of the corresponding controller set to which it belongs. Therefore, the semantic label associated with a controller set can be used as the default semantic label for controllers within the set.

[0088] The “Controllers” list in this embodiment defines the controllers to be used and how to use them.

[0089] The “Weight” attribute in this example provides the default controller weights to be used. The length of this array can be constrained to be equal to the number of items in “Controllers”. These controller weights replace the corresponding weights in the “Weight” attribute of “MPEG_controllers”.

[0090] In the example embodiment, controller The attribute is defined as follows: Table 8 - Controller Properties.

[0091] The "Description" property describes the controller and how it is used. This is purely illustrative; for example, it can be used in the user interface to help users understand what the controller is intended for.

[0092] The “Purpose” attribute contains a string identifying the purpose of the controller. In some embodiments, the purpose attribute is constrained to follow predetermined values ​​defined by the application or standards. In some embodiments, this attribute is of type enumeration (or “enum”) variable type. The “Purpose” attribute and / or the “Description” attribute can be used as semantic labels to provide information that allows the system to associate a set of controllers with desired operations.

[0093] In this embodiment, the “controller” attribute refers to a controller in the “controller” list of the “MPEG_controllers” extension.

[0094] In this embodiment, the "range" attribute defines the minimum and maximum values ​​that the controller weights can achieve.

[0095] Examples of glTF schemes based on embodiments disclosed herein are provided below. For clarity, these examples show only the portions of the glTF file related to the virtual avatar and controller. Typically, the file also includes information about the scene having one or more meshes representing the virtual avatars.

[0096] In the example, it might be desirable to use two controllers to control the avatar's eyes. The first controller (index 0) controls horizontal eye movement, and the second controller (index 1) controls vertical eye movement. (As in...) Figure 1A-1C and Figure 2A-2C As shown, the controller changes the weights of the eye rotation and the two deformable targets corresponding to eyelid movement.

[0097] The glTF file is configured with the following content: • The head mesh (index 0) has four deformation targets: º AU61_Eyes_turn_L: This deformable target definition has an animation of the eyelid moving to the left. º AU61_Eyes_turn_R: This deformable target definition has an animation of the eyelid moving to the right. º AU63_eyeUp: This deformable target definition has an animation of the eyelid moving upwards. º AU63_eyeDown: This deformable target definition has an animation of the eyelid moving downwards. • The eye grid for the left eye (index 1), centered at the origin. • The eye grid for the right eye (index 2), centered at the origin.

[0098] In this example, encoding is provided for these controllers (using "MPEG_controllers") and the virtual avatar controllers that reference them and give them descriptions and purposes.

[0099] In the example embodiment, the controller and the corresponding virtual avatar controller are encoded in the following manner:

[0100] Example implementations use one or more semantic tags associated with controllers and / or a set of controllers to identify operations that can be performed on nodes in the scene. Application software rendering and / or animates the scene can determine the function to be executed and can identify one or more controllers to be used to execute the function based on the semantic tags.

[0101] Regarding the semantic tags applied to a set of controllers, in some embodiments, each semantic tag can be provided using a "purpose" attribute, which can be a property of AvatarControllerSet as described above. In example embodiments, the value of the purpose attribute is constrained to a value selected from a predetermined set of values ​​defined by the application or a standard. The following table presents examples of such values: .

[0102] Regarding semantic labels applied to individual controllers (which may or may not be in the set of controllers), in some embodiments, the "purpose" attribute of the AvatarController is used to provide semantic labels to define how an application can use the virtualization controller. In example embodiments, the value of the purpose attribute is constrained to a value selected from a predetermined set of values ​​defined by the application or a standard. The following table presents examples of such values: .

[0103] The scenario description file according to the embodiments disclosed herein can be parsed as follows. In an example embodiment, the glTF parser decodes all content (including "MPEG_controllers") except for content from the "MPEG_node_avatar" extension. It then decodes the "MPEG_node_avatar" content found in the first node. It indicates that the node is a virtual avatar ("isAvatar" is true), the type is a standard type ("type" is "urn:mpeg:sd:2023:avatar"), and that a portion corresponding to the header exists ("mapping" attribute).

[0104] Parse the "Controller Set" attribute of "MPEG_node_avatar". In this example, there is a single AvatarControllerSet In this example, the message "Update Gaze" ("Description" is "Update Gaze") can be used to describe a collection of avatar controllers. As indicated by the encoded semantic tags, the purpose of this collection of avatar controllers is to change the gaze of the avatar ("Purpose" is "Gaze"). The string "Gaze" is an example of the value of the "Purpose" attribute. The syntax and semantics of such values ​​can be defined by the application or a standard.

[0105] The parsing continues with the two items in the "controller set" list.

[0106] The first item in the “Controller Set” list is the avatar controller, and it can be described using the string “Horizontal Gaze” (the “Description” is “Horizontal Gaze”). The purpose of this avatar controller is to change the avatar gaze along the horizontal axis (the “Purpose” is in “Gaze: Horizontal”). The “Controller” attribute references the first controller extended by “MPEG_controller”. Therefore, an application can use the first controller extended by “MPEG_controllers” to change the avatar gaze along the horizontal axis.

[0107] The second item in the “Controller Set” list is the avatar controller, and it can be described using the string “Vertical Gaze” (the “Description” is “Vertical Gaze”). This avatar controller’s purpose is to change the avatar’s gaze along the vertical axis (the “Purpose” is in “Gaze: Vertical”). The “Controller” attribute references a second controller extended by “MPEG_controllers”. Therefore, an application can use the second controller extended by “MPEG_controllers” to change the avatar’s gaze along the vertical axis.

[0108] Therefore, using data from the avatar controller, the application can change the avatar's gaze to the right / left and up / down.

[0109] It can be as follows Figure 3 The method for parsing the MPEG_node_avatar extension is shown in the flowchart.

[0110] The parsing of nodes in the glTF file begins at 501. At 502, the glTF loader decodes standard attributes and possible extensions. If the MPEG_node_avatar extension is determined to exist at 503, the loader parses it; otherwise, parsing ends. At 504, the "isAvatar" attribute is parsed. In embodiments where this attribute is required, an error can be signaled if it does not exist. At 505, the "type" attribute is parsed. In embodiments where this attribute is required, an error can be signaled if it does not exist. At 506, the "mapped" attribute is parsed. In embodiments where this attribute is required, an error can be signaled if it does not exist.

[0111] At point 507, determine if the "Controller Set" list exists. If it does, resolve it at point 508. Figure 4 This explains the parsing of each item in the "Controller Collection" list. At 509, the results of the data found in the MPEG_node_avatar extension (if any) are stored in a dedicated structure and should not change the scene.

[0112] exist Figure 4 The flowchart illustrates an example method for parsing the list of "controller sets" in the MPEG_node_avatar.

[0113] In the glTF file AvatarControllerSet Attribute parsing begins at 601. At 602, the "Purpose" attribute is parsed. In embodiments where this attribute is required, an error can be signaled if it does not exist. At 603, it is determined whether the "Description" attribute exists; if so, it is parsed at 604. At 605, it is determined whether the "Weight" attribute exists; if so, it is parsed at 606. At 607, as in... Figure 5 The parsing of each item in the "controller" list is described in more detail below. The parsing ends at 608.

[0114] Figure 5 This is an explanation AvatarControllerSet The flowchart shows an example of the parsing of items in the "Controllers" list.

[0115] AvatarControllerAttribute parsing begins at 701. At 702, the "Controller" attribute is parsed. In embodiments where this attribute is required, an error can be signaled if it is absent. The value of "Controller" can be constrained to a valid index in the "Controllers" list extended by "MPEG_controllers". At 703, it is determined whether the "Description" attribute exists; if so, it is parsed at 704. At 705, it is determined whether the "Purpose" attribute exists; if so, it is parsed at 706. If "Purpose" does not exist, the purpose of this controller is defined by the set of controllers that embed it and its index within the set. For example, if controller #2 has no purpose, the "Purpose" of the controller set can define a default purpose for controller #2. At 707, it is determined whether the "Scope" attribute exists; if so, it is parsed at 708. Parsing ends at 709.

[0116] Figure 6 This illustration demonstrates the structure of a 3D scene as represented in a runtime asset delivery file (e.g., a glTF file). The scene in this example comprises four meshes. As described above, three of the meshes (i.e., left eye mesh 1202, right eye mesh 1204, and head mesh 1206) are associated with avatar nodes through references in the MPEG_node_avatar extension. For illustration, this scene also includes nodes that are not avatar nodes and are associated with... Figure 6 Another node associated with grid 1208, which is called the scene grid. Figure 6 The scenario includes the four controllers defined in the MPEG_controllers extension as described above. One of the controllers 1210 is used to control the vertical eye movement of the avatar, such as applying rotation matrices to the left and right eyes while applying morphing target weights to the appropriate target (e.g., eyelid) of the head grid, wherein the appropriate rotation matrix and morphing target weights are determined by the controller based on the weight input to that controller. Another of the controllers 1212 is used to control the horizontal eye movement of the avatar, such as applying rotation matrices to the left and right eyes while applying morphing target weights to the appropriate target (e.g., eyelid) of the head grid, wherein the appropriate rotation matrix and morphing target weights are determined by the controller based on the weight input to that controller.

[0117] Figure 6 The two additional controllers 1214 and 1216 can apply appropriate transformations (e.g., translation, rotation, and / or deformation target weights) to the scene mesh based on the weighted inputs. As described in more detail above, the controllers can utilize the sampler in determining the appropriate correspondence between the input weights and the output transformations applied to the mesh, but not in... Figure 6 The text explains such details.

[0118] Figure 6 One characteristic of the scene is that it does not include explicit information about the purpose of the different controllers. If the scene were represented using glTF's traditional JSON format, each controller would be represented by an index indicating its position within an array of controllers within the MPEG_controllers extension, where the first controller in the array has an index of zero, the second has an index of 1, and so on. In strictly controlled application environments, such an arrangement might be sufficient, where the application processing the scene can rely on information indicating which controller in the array applies which transformation to which (or those) meshes. Such information could instruct the first controller with index 0 to control horizontal eye movement, and so on. However, this arrangement hinders the portability of the scene for use by other applications. For example, if an application only has information about the indices of the different controllers (which can be implicitly determined from the glTF file), it might not have enough information to determine which controller to use, for example, to perform horizontal eye movement. Applying weights to the wrong controller could have unintended effects on the scene by applying incorrect transformations and / or completely transforming the wrong meshes.

[0119] Figure 7 Schematic illustration Figure 6 The scene is encoded, in which virtual avatar signaling with a set of controllers is added according to the example embodiments described herein. Figure 7 In the example, the MPEG_node_avatar extension 1300 of the scene includes additional semantic tag information providing descriptions and purposes for the controllers associated with the virtual avatar node. Specifically, the controller for horizontal eye movement is provided with a "Description" attribute for "Horizontal Gaze" and a "Purpose" attribute for "Gaze: Horizontal". Similarly, the controller for vertical eye movement is provided with a "Description" attribute for "Vertical Gaze" and a "Purpose" attribute for "Gaze: Vertical". In this example, the set of two controllers associated with the virtual avatar is provided with semantic tags in the form of a "Description" attribute for "Update Gaze" and a "Purpose" attribute for "Gaze". Figure 7 In the example, semantic tags are organized in a hierarchical structure, where the more general semantic tag “gaze” is provided in syntax structure 1302 associated with the set of controllers, and the more specific semantic tags “gaze:vertical” and “gaze:horizontal” are provided by syntax structures 1304 and 1306, respectively.

[0120] In some embodiments, the “description” and “purpose” attributes of a controller are explicitly provided in the MPEG_node_avatar extension, while the controller is referenced within the MPEG_node_avatar extension by its index (e.g., indicating their order within the MPEG_controllers extension where they are defined). However, variations of this configuration are possible. For example, the “description” and “purpose” attributes may be provided within the same data structure used to define the controller (such as the MPEG_controllers extension), or the controller may be defined in a data structure that provides avatar attributes (such as in the MPEG_node_avatar extension). Some embodiments do not necessarily provide both the “description” and “purpose” attributes. For example, some embodiments may provide only one attribute (e.g., the “purpose” attribute), which may be constrained to be selected from a predetermined set of attributes. Of course, attributes such as “description” and “purpose” may have different names in different embodiments.

[0121] The principles described in this paper do not require that semantic tags be provided solely by extensions associated with virtual avatar nodes. For example, this can be modified by replacing the MPEG_node_avatar extension 1300 with the MPEG_node_controllers extension or any other syntactic structure that conveys the semantic tags of the controller. Figure 7 Examples of implementations.

[0122] As in Figure 8 As described, in a method according to some embodiments, a runtime asset delivery file is obtained for a 3D scene at step 802. The file can be obtained by a server or by a client device for displaying and / or rendering 3D scene information. In an example embodiment, the file includes a plurality of first syntactic structures, which may be controllers in an MPEG_controllers structure. Each first syntactic structure defines an association between input weight values ​​and a corresponding transformation of at least one of a plurality of nodes in the scene. The file further includes at least one second syntactic structure, which may be a set of controllers and / or one or more controller attributes as described above, as well as other possibilities. The second syntactic structure associates a corresponding semantic tag (e.g., the “purpose” attribute described above, although other configurations are possible) with at least one of the first syntactic structures. At step 804, at least one of the first syntactic structures is selected based on the associated semantic tag. At step 806, the selected first syntactic structure indicates the transformation.

[0123] As a non-limiting example, the second syntactic structure may associate the semantic label "gaze:level" with one of the first syntactic structures. The selection of the first syntactic structure may be based on this semantic label. For example, in response to a determination by a client or server device that the horizontal direction of the gaze should be adjusted for a mesh representing a virtual avatar (or other eyed creature), the device in this example searches for second syntactic structures to identify one of them that includes the label "gaze:level", selects which first syntactic structure is associated with that label, and applies a transformation indicated by that first syntactic structure (e.g., by applying changed input weights to that structure).

[0124] In some embodiments, the runtime asset delivery file further includes a third syntax structure, such as the MPEG_node_avatar or MPEG_node_controllers structure described above, which is associated with a node in the scene. In some such embodiments, a second syntax structure is included in the third syntax structure. In such embodiments, the transformation corresponding to the first syntax structure may be a transformation of the mesh in the node.

[0125] In some embodiments, the third syntax structure includes information indicating that the node represents a virtual avatar.

[0126] In some embodiments, the document further includes a fourth syntactic structure, which may be a controller set syntactic structure as described above, which associates a corresponding semantic tag with a set including a plurality of first syntactic structures.

[0127] In some embodiments, each of the first syntactic structures has a corresponding index, and the second syntactic structure includes an index of at least one of the first syntactic structures.

[0128] In some embodiments, the runtime asset delivery file is a glTF file.

[0129] In some embodiments, semantic tags include the “purpose” attribute as described above. Semantic tags and any other syntactic features described herein may have different names in different embodiments.

[0130] In some embodiments, a client or server device may operate to animate at least the nodes associated with the selected first syntax structure by applying varying weights to a transformation corresponding to the selected first syntax structure.

[0131] Some embodiments further include causing at least the node associated with the selected first syntactic structure to be displayed. Such embodiments may include, for example, actually displaying the rendering of the node on a display screen, providing the rendering of the node to a display device, providing the display device with image or video data of the node, or other actions.

[0132] As in Figure 9 As shown, in some embodiments, a device such as a client or server device receives multiple runtime asset delivery files at 902. These files may be received directly from different users 904, 906, 908, or they may be stored in a database (e.g., 910) where they may be associated with a specific user or may be selectable as options by more than one user. Each runtime asset delivery file includes a corresponding node representing virtual avatars 912, 914, 916, 918. Each runtime asset delivery file also includes at least one first syntactic structure (e.g., a controller in the MPEG_controllers extension) defining the association between input weight values ​​and corresponding transformations of the corresponding virtual avatar, and at least one second syntactic structure (e.g., a controller set and / or controller attributes) associating corresponding semantic tags (e.g., a "purpose" attribute) with the first syntactic structure.

[0133] At position 920, the device presents a virtual environment including virtual avatars 912, 914, 916, and 918 as described in the corresponding runtime asset delivery files. The virtual environment can be, for example, a teleconference, a virtual world, a virtual store, or other options.

[0134] At 922, the device selects the action to be performed on the selected virtual avatar. This selection can be performed at least in part based on input from the user. As an example, user 912 can provide input indicating that he / she wishes to turn his / her head.

[0135] At point 924, using a semantic tag for the second syntactic structure associated with the selected avatar, a first syntactic structure corresponding to the selected action is selected. For example, the first syntactic structure could be associated with a semantic tag indicating that the corresponding transformation results in a head-turning motion. It's possible, for instance, that different files from different users all contain controllers for turning the avatar's head, but those different controllers may have different indices in those different files. Using semantic tags associated with those controllers makes it possible to identify the appropriate controller, despite the lack of a standardized controller indexing scheme.

[0136] At position 926, the transformation indicated by the first recognized grammatical structure is applied to the virtual avatar.

[0137] Use case: Processing workflows When creating 3D assets, a common approach is to break the process down into several stages. Each stage focuses on a specific task, such as mesh modeling, texturing, animation, and lighting. These stages are often handled by different people or teams, or by the same individual at different times. To manage this, one solution is to store the results of each stage in a new file. Using the current glTF standard, this approach is already feasible for some tasks. However, this method is not effective for other tasks, such as the complex animation of virtual avatars.

[0138] Given this context, one approach is to use a set of basic animations or controllers for each model. For example, a face is associated with a set of facial expressions; each is encoded as an animation. During the early steps of the workflow, artists must design these animations, in which case the current glTF standard is sufficient. During later stages, artists can combine these basic animations to create the final animated scene. In this final case, and if this is the final stage, the animation combination can be baked into the final animation in the current glTF file. However, if this is not the final stage, and if another artist needs to update the final animation (e.g., during the lighting stage), utilizing the standard approach is no longer possible: they must ask the artist from the earlier stages to update the animation and bake it again.

[0139] Using the "MPEG_controllers" extension, it becomes possible to store useful information about the animation creation process in the glTF file. In this context, the basic animation is called a controller, and combinations of animations are the animations of controllers. However, using only the standard and the "MPEG_controllers" extension, there is no information about the role of each controller in the glTF file. Then, when artists obtain avatars from previous stages, they (or applications) must have separate information indicating the purpose of each controller in the glTF file, because the file itself does not include such information. Furthermore, without this information, different glTF files would need to have the same number of controllers and the same order. If, for some reason, a new controller has to be added to the current controller list, the entire processing chain must be updated. In complex environments involving many teams or even many companies, performing such updates can be very difficult or even completely impractical.

[0140] Using the proposed embodiments described herein, virtual avatar controllers can be associated with meaningful information. For example, a set of purpose identifiers can be defined for each controller type (e.g., using...). AvatarControllerThe "Purpose" attribute. Alternatively, or additionally, each controller can be described, providing the artist with information about the controller's purpose (e.g., using...). AvatarController (The "Description" attribute). Furthermore, the example embodiments make it possible to provide proposed extensions to different controller sets. In such embodiments, each controller set can be dedicated to a specific purpose, such as modifying only the avatar's identity or pose. In some embodiments, controller sets can be used for version control. This allows artists to modify the animation of the controllers themselves in the next stage without having to guess the purpose of each controller and set.

[0141] Use case: Virtual avatars in the virtual world Some example implementations are useful in virtual worlds provided by different companies, governments, or users. Using such implementations, users can import their virtual avatars into these virtual worlds, and they will function correctly in each virtual world that conforms to the same standards.

[0142] Many characteristics can be expected from these virtual avatars, such as the ability to change their identity (face shape, nose size, etc.). To correctly change their identity, applications can use a dedicated set of controllers for the identity. Using this set of controllers according to the example embodiment, a user can change their identity within a virtual world. Furthermore, the user can update the virtual avatar's glTF file with new weights for this set of controllers and use it in another virtual world that supports the standard.

[0143] These mechanisms also apply to other modifications, including live modifications like facial expressions. In these cases, a set of controllers can be dedicated to these modifications and used by the application to animate the avatar. Such a set of controllers can be updated: for example, if a user wants to change how their avatar smiles, the controller dedicated to that expression can be modified. Again, it will update the avatar in the current virtual world, but it will also update avatars in other virtual worlds that support the standard.

[0144] Use case: Virtual avatars in virtual stores Some implementations can be used in situations where virtual stores exist, managed by different companies, governments, or users. Using the implementations described herein, users can import their avatars into these virtual stores, and they will function correctly in each virtual store that conforms to the same standards.

[0145] People can buy clothing in a virtual store. They first import their avatar into the store, and then they can try on clothes, assuming their avatar has their body measurements. They can change the avatar's pose to see how the clothes fit depending on the situation. For example, the avatar controller can model different common poses (standing, sitting, etc.) to help them see how clothing can fit their bodies.

[0146] Use case: Virtual avatars in video conferencing In some embodiments, virtual avatars can be used in video conferencing, where the avatar is used to represent a user. It is desirable to first animate the avatar using facial expressions that match the user's voice. Therefore, at least one set of avatar controllers can be dedicated to facial expressions; however, several sets of controllers dedicated to facial expressions can also be placed in the glTF file, each for a specific emotion: a set of facial expressions for happiness, a set for sadness, a set for anger, etc. This principle can be repeated indefinitely, beyond mere imagination, such as making the avatar stand, move, adopt funny poses, etc.

[0147] Example system hardware Systems such as Figure 10 The system is used to implement devices for displaying and / or rendering 3D scene information, along with its control electronics. Figure 10 This is a block diagram illustrating examples of systems implementing various aspects and embodiments. System 1000 may be embodied as an apparatus including the various components described below and configured to perform one or more aspects described in this document. Examples of such apparatus include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 1000 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0148] System 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing various aspects, such as those described in this document. Processor 1010 may include embedded memory, input / output interfaces, and various other circuitry as known in the art. System 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 1040 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0149] System 1000 includes an encoder / decoder module 1030 configured, for example, to process data to provide encoding or decoding scenarios, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents one or more modules that can be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both encoding and decoding modules. Furthermore, the encoder / decoder module 1030 may be implemented as a separate element of system 1000 or may be incorporated into processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0150] Program code to be loaded onto processor 1010 or encoder / decoder 1030 to execute the various aspects described in this document may be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. According to various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more of various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input scene description files, decoded scenes or portions of decoded scenes, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0151] In some embodiments, the memory within processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., processor 1010 or encoder / decoder module 1030) is used for one or more of these functions. External memory may be memory 1020 and / or storage device 1040, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory, such as RAM, is used as working memory for encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Multi-Functional Video Coding, a new standard being developed by JVET (Joint Video Experts Group)).

[0152] Inputs to the components of system 1000 can be provided by various input devices as indicated in box 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcaster, (ii) a component input terminal (or a collection of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Other examples include composite video.

[0153] In various embodiments, the input device of block 1130 has associated corresponding input processing elements as known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band), (ii) down-converting the selected signal, (iii) band-limiting it again to a narrower band to select (e.g.,) a signal band that may be referred to as a channel in some embodiments), (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements for performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners that perform various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or to baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the aforementioned (and other) components, remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0154] Additionally, USB and / or HDMI terminals may include corresponding interface processors for connecting system 1000 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) can be implemented, for example, within a separate input processing IC or within processor 1010 as needed. Similarly, aspects of USB or HDMI interface processing can be implemented, as needed, within a separate interface IC or within processor 1010. Demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 1010, and encoder / decoder 1030, which operates in combination with memory and storage elements to process the necessary data streams for presentation on an output device.

[0155] Various components of system 1000 can be provided within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data therebetween using a suitable connection arrangement 1140 (e.g., internal buses as known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0156] System 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0157] In various embodiments, data is streamed or otherwise provided to system 1000 using a wireless network such as Wi-Fi (e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)). Wi-Fi signals in these embodiments are received via a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks (including the Internet) to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that transmits data via an HDMI connection of input box 1130 to provide streaming data to system 1000. Still other embodiments use an RF connection of input box 1130 to provide streaming data to system 1000. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0158] System 1000 can provide output signals to various output devices, including display 1100, speaker 1110, and other peripheral devices 1120. Display 1100 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1100 can be used for a television, tablet computer, laptop computer, mobile phone, or other device. Display 1100 can also be integrated with other components (e.g., in a smartphone) or separate (e.g., an external monitor for a laptop computer). Other peripheral devices 1120 include one or more of a standalone digital video disc (or digital versatile disc) (DVR for both terms), an optical disc player, a stereo system, and / or a lighting system in various examples of embodiments. Various embodiments use one or more peripheral devices 1120 to provide functionality based on the output of system 1000. For example, an optical disc player performs the function of playing the output of system 1000.

[0159] In various embodiments, control signals are transmitted between system 1000 and display 1100, speaker 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to system 1000 via communication interface 1050 using communication channel 1060. Display 1100 and speaker 1110 can be integrated into a single unit along with other components of system 1000 in an electronic device such as, for example, a television. In various embodiments, display interface 1070 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0160] Display 1100 and speaker 1110 may alternatively be separate from one or more other components, for example, if the RF section of input 1130 is part of a separate set-top box. In various embodiments where display 1100 and speaker 1110 are external components, output signals may be provided via dedicated output connections including, for example, HDMI ports, USB ports, or COMP outputs.

[0161] System 1000 may include one or more sensor devices 1095. Examples of sensor devices that may be used include one or more GPS sensors, gyroscope sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors can be used to determine information such as the user's position and orientation. In the case where system 1000 is used as a control module for an extended reality display, the user's position and orientation can be used to determine how image data is rendered so that the user perceives the correct portion of a virtual object or scene from the correct angle. In the case of a head-mounted display device, the position and orientation of the device itself can be used to determine the user's position and orientation for the purpose of rendering virtual content. In the case of other display devices (such as telephones, tablets, computer monitors, or televisions), other inputs can be used to determine the user's position and orientation for the purpose of rendering content. For example, the user can use a touchscreen, keyboard or keypad, trackball, joystick, or other inputs to select and / or adjust the desired viewpoint and / or viewing direction. In the case where the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation for the purpose of rendering content can be selected and / or adjusted based on the movement of the display device.

[0162] The embodiments can be executed by computer software implemented by processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, memory 1020 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 1010 can be of any type suitable for the technical environment and can include one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0163] Other embodiments This disclosure describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described in detail, and often in a manner that may sound restrictive, at least to illustrate individual characteristics. However, this is for the purpose of clarity of description and does not limit the scope of this disclosure or those aspects. In fact, all the different aspects can be combined and interchanged to provide other aspects. Furthermore, the aspects described can also be combined and interchanged with those described in earlier applications.

[0164] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments have been specifically described, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of implementation. At least one aspect generally relates to the encoding, decoding, and rendering of scene description information, and at least one other aspect generally relates to the transmission of generated or encoded files and / or bitstreams. These and other aspects can be implemented as methods, apparatuses, computer-readable storage media having instructions thereon stored for encoding, decoding, or rendering scene description data according to any of the methods described, and / or computer-readable storage media having bitstreams generated according to any of the methods described.

[0165] Various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as "first," "second," etc., may be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless specifically required, the use of such terms does not imply a reordering of the modified operations. Thus, in this example, the first decoding does not need to be performed before the second decoding, and the first decoding may occur, for example, before the second decoding, during the second decoding, or within a time period overlapping with the second decoding.

[0166] For example, various numerical values ​​may be used in this disclosure. Specific values ​​are for illustrative purposes and the aspects described are not limited to these specific values.

[0167] The embodiments described herein can be executed by computer software implemented by a processor or other hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. As a non-limiting example, the processor can be of any type suitable for the technical environment and can include one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0168] When a diagram is presented as a flowchart, it should be understood that a block diagram of the corresponding device is also provided. Similarly, when a diagram is presented as a block diagram, it should be understood that a flowchart of the corresponding method / process is also provided.

[0169] The implementations and aspects described herein can be implemented in, for example, methods or processes, devices, software programs, data streams, or signals. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), implementations of the discussed features can be implemented in other forms (e.g., devices or programs). Devices can be implemented in, for example, suitable hardware, software, and firmware. Methods can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate the transfer of information between end users.

[0170] The references to "an embodiment" or "an embodiment" or "an implementation" or "implementation" and their other variations mean that a particular feature, structure, characteristic, etc., described in relation to an embodiment is included in at least one embodiment. Therefore, the appearance of the phrases "in an embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" throughout this disclosure does not necessarily all refer to the same embodiment.

[0171] Additionally, this disclosure may relate to "determining" various pieces of information. Determining information may include, for example, one or more of the following: estimated information, calculated information, predicted information, or information retrieved from memory.

[0172] Furthermore, this disclosure may relate to “accessing” various information fragments. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or one or more of these.

[0173] Additionally, this disclosure may relate to "receiving" various pieces of information. Like "access," receiving is intended as a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory). Furthermore, "receiving" is generally referred to in one or more ways during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0174] It should be recognized that, for example, the use of " / ", "and / or", and "...at least one of A and B" in the context of "A / B", "A and / or B", and "at least one of A and B" is intended to include selecting only the option listed in the first column (A), or only the option listed in the second column (B), or selecting both options (A and B). As another example, in the context of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to include selecting only the option listed in the first column (A), or only the option listed in the second column (B), or only the option listed in the third column (C), or only the options listed in the first and second columns (A and B), or only the options listed in the first and third columns (A and C), or only the options listed in the second and third columns (B and C), or selecting all three options (A, B, and C). This can be extended to as many items as are listed.

[0175] Moreover, as used herein, among other things, the word “signal” refers to the corresponding decoder instructing something. For example, in some embodiments, the encoder signals a specific parameter among a plurality of parameters for region-based filter parameter selection for artifact removal filtering. In this way, in embodiments, the same parameter is used at both the encoder and decoder sides. Thus, for example, the encoder may transmit (explicit signaling) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling may be used without transmitting (implicit signaling) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various embodiments by avoiding the transmission of any actual function. It should be appreciated that signaling can be done in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. While the foregoing refers to the verb form of the word “signal,” the word “signal” may also be used as a noun herein.

[0176] The implementation can generate a wide variety of signals formatted to carry information that can be stored or transmitted, for example. The information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, the signal may be formatted to carry a bit stream of the described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave using the encoded data stream. The information carried by the signal may be, for example, analog or digital information. The signal can be transmitted via a wide variety of wired or wireless links as known. The signal may be stored on a processor-readable medium.

[0177] We have described several embodiments. Features of these embodiments may be provided individually or in any combination, across various claim classes and types. Furthermore, embodiments may include one or more of the following features, means, or aspects individually or in any combination, across various claim classes and types: • Includes one or more bitstreams or signals of the described syntax elements or their variants. • Includes bitstreams or signals that convey information generated according to any embodiment of the described embodiments. • Create and / or transmit and / or receive and / or decode bit streams or signals including one or more of the described syntax elements or their variants. • Create and / or transmit and / or receive and / or decode according to any embodiment of the described embodiments. • Methods, processes, apparatus, media for storing instructions, media for storing data, or signals according to any of the embodiments described.

[0178] Note that the various hardware elements of one or more embodiments described in the embodiments are referred to as “modules”, which implement (i.e., operate, perform, etc.) the various functions described herein in relation to the respective modules. As used herein, a module includes hardware considered suitable for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices). Each described module may also include instructions executable to perform one or more functions described as implemented by the respective module, and note that those instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or the like, or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or the like, and may be stored in any suitable one or more non-transitory computer-readable media such as RAM, ROM, etc.

[0179] Although features and elements have been described above in specific combinations, each feature or element may be used individually or in any combination with other features and elements. Additionally, the methods described herein may be implemented using computer programs, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM discs and digital multifunction discs (DVDs). A processor associated with the software may be used to implement a radio frequency transceiver for a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method comprising: Obtain runtime asset delivery files for the 3D scene, the files including: Multiple first syntactic structures, each defining the association between input weight values ​​and the corresponding transformation of at least one of the multiple nodes in the scene; and At least one second syntactic structure, which associates a corresponding semantic tag with at least one first syntactic structure in the first syntactic structure; Select at least one first syntactic structure from the first syntactic structures based on associated semantic tags; and The transformation indicated by the selected first syntactic structure is applied.

2. An apparatus comprising one or more processors, the apparatus being configured to perform at least: Obtain runtime asset delivery files for the 3D scene, the files including: Multiple first syntax structures, each first syntax structure defining the association between the input weight value and the corresponding transformation of at least one of the multiple nodes in the scene; as well as At least one second syntactic structure, which associates a corresponding semantic tag with at least one first syntactic structure in the first syntactic structure; Select at least one first syntactic structure from the first syntactic structures based on the associated semantic tags; as well as The transformation indicated by the selected first syntactic structure is applied.

3. The method of claim 1 or the device of claim 2, wherein the first syntax structure is a controller structure in the MPEG_controllers extension.

4. The method of claim 1 or the method of claim 3 when it is dependent on claim 1, or the device of claim 2 or the device of claim 3 when it is dependent on claim 2, wherein the document further includes a third syntactic structure associated with a node in the scene, wherein the second syntactic structure is included in the third syntactic structure, and wherein the transformation corresponding to the first syntactic structure is a transformation of the mesh in the node.

5. The method of claim 4 when it is subordinate to claim 1, or the device of claim 4 when it is subordinate to claim 2, wherein the third syntax structure includes information indicating that the node represents a virtual avatar.

6. The method of claim 4 as it is dependent on claim 1, or the device of claim 4 as it is dependent on claim 2, wherein the third syntax structure is an MPEG_node_avatar extension.

7. The method of claim 4 as it is dependent on claim 1, or the device of claim 4 as it is dependent on claim 2, wherein the third syntax structure is an MPEG_node_controllers extension.

8. The method of claim 1 or the method of any one of claims 3-7 when it is dependent on claim 1, or the device of claim 2 or the device of any one of claims 3-7 when it is dependent on claim 2, wherein the document further comprises a fourth syntactic structure that associates the corresponding semantic tags with a set including a plurality of the first syntactic structures.

9. The method of claim 1 or the method of any one of claims 3-8 when it is dependent on claim 1, or the device of claim 2 or the device of any one of claims 3-8 when it is dependent on claim 2, wherein each of the first syntactic structures has a corresponding index, and wherein the second syntactic structure includes the index of at least one of the first syntactic structures in the first syntactic structure.

10. The method of claim 1 or the method of any one of claims 3-9 when it is dependent on claim 1, or the device of claim 2 or the device of any one of claims 3-9 when it is dependent on claim 2, wherein the runtime asset delivery file is a glTF file.

11. The method of claim 1 or the method of any one of claims 3-10 when it is dependent on claim 1, or the device of claim 2 or the device of any one of claims 3-10 when it is dependent on claim 2, wherein the semantic tag includes a usage attribute.

12. The method of claim 1 or the method of any one of claims 3-11 when it is dependent on claim 1, or the device of claim 2 or the device of any one of claims 3-11 when it is dependent on claim 2, further comprising at least animing the node associated with the selected first syntactic structure by applying varying weights to the transformation corresponding to the selected first syntactic structure.

13. The method of claim 1 or the method of any one of claims 3-12 when it is dependent on claim 1, or the device of claim 2 or the device of any one of claims 3-12 when it is dependent on claim 2, further comprising causing at least the node associated with the selected first grammatical structure to be displayed.

14. The method of claim 1 or the method of any one of claims 3-13 when it is dependent on claim 1, or the apparatus of claim 2 or the apparatus of any one of claims 3-13 when it is dependent on claim 2, further comprising: A plurality of runtime asset delivery files are obtained, each runtime asset delivery file comprising: a corresponding node representing a virtual avatar, at least one first syntactic structure defining the association between input weight values ​​and corresponding transformations of the corresponding virtual avatar, and at least one second syntactic structure associating a corresponding semantic tag with the first syntactic structure. Presents a virtual environment containing the virtual avatar, including the corresponding runtime asset delivery files; Select the action to be performed on the selected virtual avatar among the virtual avatars; Using the semantic tag of the second syntactic structure associated with the selected virtual avatar, identify the first syntactic structure corresponding to the selected action; and The transformation indicated by the first recognized grammatical structure is applied.

15. A computer-readable medium storing runtime asset delivery files for a 3D scene, the files comprising: Multiple first syntax structures, each first syntax structure defining the association between the input weight value and the corresponding transformation of at least one of the multiple nodes in the scene; as well as At least one second syntactic structure, which associates a corresponding semantic tag with at least one first syntactic structure in the first syntactic structure.

Images