Controller user interface coding in scene description

Abstract

In an exemplary method, a runtime asset delivery file includes a controller syntax structure that defines an association between a range of weights and respective output transformations. The file also includes a user interface syntax structure that identifies at least one associated controller syntax structure and at least one location. At least one user interface element is displayed at a location identified by the first user interface syntax structure. A user input indicating a first weight is received through a first user interface element. A first output transform associated with the first weight is determined based on a first controller syntax structure identified by the first user interface syntax structure. At least one node in the scene is transformed by applying the first output transformation to nodes associated with the first controller syntax structure.

Description

Cross-references to related applications

[0001] This application claims priority to European Patent Application No. 23306701.6, filed on October 4, 2023, entitled “Controller User Interface Encoding in Scene Description,” which is incorporated herein by reference in its entirety. Background Technology

[0002] This disclosure relates to the encoding and decoding of 3D scene representations. A 3D scene can be represented by a runtime asset delivery file (such as a glTF file). A glTF or other runtime asset delivery file can provide information about the 3D scene, including the geometric data and organization of objects in the scene, as well as information used to animate these objects.

[0003] While the glTF format has proven usable for delivering final-form 3D scenes to users, it performs poorly as a format for those designing and implementing 3D scenes. In particular, the structure of animations in glTF makes it difficult for designers to make minor modifications without recalculating the entire animation. Summary of the Invention

[0004] A method according to some embodiments includes: acquiring a runtime asset delivery file for a 3D scene, the file including: a plurality of controller syntax structures, each controller syntax structure defining an association between a weight range and a corresponding output transformation of at least one associated node in the scene; and a plurality of user interface syntax structures, each user interface syntax structure including information identifying at least one associated controller syntax structure and at least one location. The method further includes: for at least one first user interface syntax structure in the user interface syntax structures, displaying at least one first user interface element at a location identified by the first user interface syntax structure; receiving user input indicating a first weight through the first user interface element; determining a first output transformation associated with the first weight based on the first controller syntax structure identified by the first user interface syntax structure; and generating a transformed node by applying the first output transformation to the node associated with the first controller syntax structure.

[0005] Some embodiments also include displaying the transformed node. In some embodiments, the transformed node and the user interface element are displayed simultaneously.

[0006] In some embodiments, the nodes associated with the first controller syntax structure include a grid.

[0007] In some embodiments, the first user interface element includes a movable cursor within a range defined by the first user interface syntax structure, and wherein the first weight is indicated by the position of the cursor within that range.

[0008] In some embodiments, the first user interface syntax structure indicates a start point and an end point, wherein the first user interface element includes a cursor movable along a line between the start point and the end point, and wherein the first weight is indicated by the position of the cursor within that range.

[0009] In some embodiments, the first user interface syntax structure defines a rectangle; the first user interface element includes a movable cursor within the rectangle; the first weight is indicated by the vertical position of the cursor within the rectangle; and the second weight is indicated by the horizontal position of the cursor within the rectangle.

[0010] In some embodiments, the first user interface syntax structure defines an ellipse; the first user interface element includes a movable cursor within the ellipse; the first weight is indicated by the vertical position of the cursor within the ellipse; and the second weight is indicated by the horizontal position of the cursor within the ellipse.

[0011] Some embodiments further include: determining a second output transformation associated with the second weight based on the second controller syntax structure in the runtime asset delivery file; wherein generating the transformed node further includes applying the second output transformation to the node associated with the first controller syntax structure.

[0012] In some embodiments, the transformation includes at least one of the following: rotation, translation, scaling change, or change of the weight of the deformable target.

[0013] Some embodiments include storing the first weight in the runtime asset delivery file after receiving user input indicating the first weight.

[0014] The method according to some embodiments includes: encoding at least one node in a runtime asset delivery file for a 3D scene; encoding a plurality of controller syntax structures in the runtime asset delivery file, each controller syntax structure defining an association between an input weight range and a corresponding output transformation of at least one associated node in the scene; and encoding a plurality of user interface syntax structures in the runtime asset delivery file, each user interface syntax structure including information identifying at least one associated controller syntax structure and at least one location, each user interface syntax structure defining a corresponding user interface element for modifying the input weights of the corresponding controller syntax structure.

[0015] According to some embodiments, the signal includes a runtime asset delivery file for a 3D scene, wherein the scene description data includes: at least one node; a plurality of controller syntax structures, each controller syntax structure defining an association between an input weight range and a corresponding output transformation of at least one associated node in the scene; and a plurality of user interface syntax structures, each user interface syntax structure including information identifying at least one associated controller syntax structure and at least one location, each user interface syntax structure defining a corresponding user interface element for modifying the input weights of the corresponding controller syntax structure. In some embodiments, the asset delivery file is a JSON file. In some embodiments, the runtime asset delivery file is a glTF file.

[0016] Further embodiments include one or more processors configured to perform any of the methods described herein.

[0017] Further embodiments include a computer-readable medium (which may be non-transitory) containing instructions for causing one or more processors to perform any of the methods described herein.

[0018] Further embodiments include a computer program product containing instructions that, when executed by one or more processors, cause the one or more processors to perform any of the methods described herein. Attached Figure Description

[0019] Figure 1 An example of a controller user interface (UI) for animate facial meshes is illustrated according to some embodiments.

[0020] Figure 2 The illustration shows an example of a controller user interface (UI) for animate a facial mesh, according to some embodiments. The left side shows the rendering of the face, and the right side shows the controller defined in a glTF file according to some embodiments.

[0021] Figure 3 A graph is provided indicating how the operation of controller 44 modifies the target weight of the outer eyebrow deformation.

[0022] Figure 4 The illustration shows an example of a controller user interface (UI) for animate facial meshes according to some embodiments, where the s1 control is located at the start of the line, resulting in a controller weight of 0.5.

[0023] Figure 5 An example of a controller user interface (UI) for animate facial meshes according to some embodiments is illustrated, where the s1 control is located at the end of a line, resulting in a controller weight of 1.0.

[0024] Figure 6This is a graph showing how the operation of controller 28 changes the "puckerOpen" deformable target.

[0025] Figure 7 This is a graph showing how the operation of controller 29 changes the deformable target "AU18_lip_pucker".

[0026] Figure 8 The illustration shows an example of a controller user interface for animate a face mesh with a first controller weight of 1.0 (X-axis) and a second controller weight of 0.0 (Y-axis) according to some embodiments, where the UI control m34 (the rectangle in the lower left corner) is used.

[0027] Figure 9 The illustration shows an example of a controller user interface for animate a face mesh with a first controller weight of 0.0 (X-axis) and a second controller weight of 1.0 (Y-axis) according to some embodiments, where the UI control m34 (the rectangle in the lower left corner) is used.

[0028] Figure 10 The illustration shows an example of a controller user interface for animate a face mesh with a first controller weight of 1.0 (X-axis) and a second controller weight of 1.0 (Y-axis) according to some embodiments, where the UI control m34 (the rectangle in the lower left corner) is used.

[0029] Figure 11 A graph is provided illustrating how controller 0 changes the weights of the deformable targets “AU61_Eyes_turn_L” and “AU61_Eyes_turn_R”.

[0030] Figure 12 A graph is provided illustrating how controller 1 changes the weights of the deformable targets “AU63_eyeUp” and “AU63_eyeDown”.

[0031] Figure 13 The illustration shows an example of a controller user interface for animate a facial mesh with a first controller weight of 1.0 (X-axis) and a second controller weight of 0.0 (Y-axis) according to some embodiments, where the UI control gaze (the ellipse in the middle) is used.

[0032] Figure 14 The illustration shows an example of a controller user interface for animate a face mesh with a first controller weight of 0.0 (X-axis) and a second controller weight of -1.0 (Y-axis) according to some embodiments, where the UI control gaze (the ellipse in the middle) is used.

[0033] Figure 15The illustration shows an example of a controller user interface for animate a facial mesh, with a first controller (X-axis) having high positive weights and a second controller (Y-axis) having high negative weights, according to some embodiments, where a UI control gaze (an ellipse in the middle) is used. In this example, the weight values ​​are constrained by the ellipse boundary, so both weights are not allowed to have a value of 1.0 simultaneously.

[0034] Figure 16 This is a flowchart illustrating a method for parsing "MPEG controllers" extensions according to some embodiments.

[0035] Figure 17 This is a flowchart illustrating a parsing controller according to an exemplary embodiment.

[0036] Figure 18 This is a flowchart illustrating the parsing of user interface controls according to some embodiments.

[0037] Figure 19 This is a functional block diagram of an apparatus that can be used to implement some of the embodiments described herein. Detailed Implementation

[0038] Exemplary embodiments involve encoding controllers in a 3D scene representation. Controllers are named functions that apply transformations to change a mesh given weights. They are typically associated with UI components. This disclosure describes how to encode these UI components in a scene description.

[0039] An overview of animation in scene description The current MPEG-I Scene Description (SD) format allows animation to be stored in glTF files. animations Within the assets. Each animation definition can modify a mesh's named function for a given time period between zero and a maximum value. A common use of these animations is to provide different actions for the mesh, such as walking, running, and idle animations for a character. Another use is to provide animations that produce different cutscenes, which can be described as short films rendered in real time.

[0040] Current animation coding allows functions to modify one or more properties of a mesh, such as by applying changes to translation, rotation, or mesh target weights. Extensions such as KHR_animation_pointers can be used to modify even more properties. Each animation can have a name, but there is no standard to define the meaning of these names.

[0041] The current format does not explicitly define how animations should be combined. For example, if you want to use two animations simultaneously, it is unclear how the properties they change should be updated, and firstly, the order of the animations.

[0042] Furthermore, it is not feasible to combine animations with different time ranges (e.g., a combination of animation 1 in the encoding range [1,2] and animation 2 in the range [2,3]).

[0043] Finally, encoding optional UI elements in the glTF file that would facilitate animation is not feasible. Figure 1 An example of this UI is shown, where each control is linked to one or two controllers.

[0044] Overview of controller extensions in the scene description For clarity, exemplary embodiments are described with reference to the glTF format and the MPEG-I SD extension for glTF. However, it should be understood that the principles described herein are not limited to any particular format and can be implemented alternatively using other formats such as XML or USD.

[0045] The current MPEG_Controllers extension has the following characteristics: • Controller: Each controller defines a function that outputs a transformation to change mesh properties, given weights between minimum and maximum values.

[0046] • Controller weights: Each value defines the weight of each controller.

[0047] An exemplary embodiment includes components that define a user interface for the controller.

[0048] •UI: Defines a set of controls that define user interface components that change the weight of one or more controllers.

[0049] Some implementations can be partially achieved by modifying the root of the glTF file in the "extensions" attribute, so that the "MPEG_controllers" attribute can be as described in Table 1.

[0050]

[0051] Table 1 describes the characteristics of the MPEG_controllers attribute, including the “ui” attribute. Note that in this table and all other tables in this disclosure, the “Required” column indicates whether the feature is required for the particular syntax shown in the table. The syntax indicating that a particular feature is “required” does not mean that the feature is required in all embodiments. There may be other embodiments where these features are not required. In exemplary embodiments, the number of items in “controllers” and “weights” is constrained to be the same.

[0052] Controller Overview In an exemplary embodiment, each controller in the "controllers" list of MPEG_controllers defines a function that transforms one or more properties of the mesh. For example, in some embodiments, the controller changes the position of the mesh vertices, thereby modeling the deformation of the mesh.

[0053] The shape of the function associated with the controller is defined by a set of values ​​(input weights, output attribute changes). These values ​​are encoded to be compatible with the keyframe scheme used in the corresponding animation (e.g., glTF animation). The keyframe times pointed to by the sampler input attributes are mapped to the function input weights. The animation values ​​pointed to by the channel target attributes are mapped to the output attribute change values. Therefore, an animation framework (e.g., the glTF animation framework) is used to shape each controller as a possible non-linear function of the input weight control values.

[0054] In some embodiments, the controller is defined in the runtime asset delivery file. The controller is defined by an instance of the "controller" property. A controller property may include one or more properties. The "Required" column indicates the properties that are required in at least some implementations of the controller property, although the same properties may not be required in other implementations.

[0055]

[0056] In some embodiments, the Controller property has the same properties as the standard Animation property in glTF, plus two additional properties: "animation" and "min".

[0057] In some embodiments, this attribute has two modes, depending on the definition of the "animation" attribute. If this attribute does not exist, the controller directly defines the function; otherwise, it uses the function of an existing animation. In some implementations, the controller definition is constrained to be in one of the two modes, with other combinations resulting in an error. In all modes, the "name" attribute can define the name of the controller. If not defined, the application can choose a name based on the controller index, such as "controller_23" or "controller_0023".

[0058] In some embodiments, the "animation" property is not defined in the controller; in this case, the controller directly defines its function. The "min" property is ignored in this case. All other properties ("channels", "samplers", "name", "extensions", "extras") can be used according to the properties of the standard Animation property. A normal animation parser can be run, except that time is treated as an input weight value, and its minimum value can be negative.

[0059] In other embodiments, the "animation" property is defined in the controller. In such embodiments, the controller may not define its own function but instead use an animation function. In this case, the following properties can be ignored: "channels" and "samplers". However, when the "animation" property is defined, the properties "name", "extensions", and "extras" can still be used.

[0060] The "animation" property defines the animation index within the "animations" asset. In this case, the function defined for the animation is used as the controller, except that time is treated as an input weight value, and its minimum value is offset by the value of the "min" property. For example, if "min" is -1.3, the animation time is reduced by -1.3, as if the animation started at -1.3 seconds. This property allows defining controllers with negative input weight values ​​while referencing glTF animations whose time cannot be negative.

[0061] In some embodiments, the “weights” attribute of MPEG_controllers contains an array of numbers (e.g., floating-point values) that defines the weights associated with each controller. The weights can be provided in the same order as the controllers in the “controllers” list: weights[0] for controllers[0], weights[1] for controllers[1], and so on.

[0062] These weights provide information about how the nodes targeted by the controller change. In some embodiments, each weight is used to apply its corresponding controller, iterating from the first to the last. For example, a node can first be updated by applying the function of controllers[0] with weights[0]. Then, these updated nodes are modified by the function of controllers[1] with weights[1], and so on. These updates can be expressed as follows:

[0063] in It is a function of controllers[i]. It is weights[i], It is the initial node. It is the node after updating with the controller and weights.

[0064] In some embodiments, controller changes occur in the same step as the animation: they are applied to the initial node content and before any transformations in other glTF properties (such as properties in the “nodes” asset).

[0065] In some embodiments, the controller is not activated when glTF parsing is complete; similar to animation, the application chooses whether to enable the controller.

[0066] In some embodiments, the controller is not used with the animation; the application can choose to enable either the controller or the animation.

[0067] In some embodiments, an extension (such as KHR_animation_pointer) is used to animate the controller, in which case the extension is used to define animations to animate the controller weights (similar to the use of deformable target weights).

[0068] Controller User Interface Overview In an exemplary embodiment, ControllersUI Attributes define a 2D user interface that allows the user to update controller weights within the user interface. Example ControllersUI Attributes can have the attributes shown in Table 3.

[0069]

[0070] The "size" attribute is used to define the main frame, where controls are defined. In some embodiments, the coordinates of the top-left corner are always (0, 0), and the coordinates of the bottom-right corner are (size[0], size[1]). The coordinates of the controls below are related to the size attribute and should be within the frame. The size does not need to be a pixel or integer value. In some embodiments, the application scales the user interface to any desired size, such as the size of the window.

[0071] The "controls" property can be used to define controls. Each item in this example is... Control Instances of the attribute are shown in Table 4.

[0072]

[0073] In this example, the "name" attribute defines the name used for the control.

[0074] In this example, the "type" attribute defines the type of the control, as detailed below.

[0075] In this example, the "points" property defines the coordinates of the control. It has four values: (x1, y1, x2, y2). (x1, y1) are the coordinates of the first point; (x2, y2) are the coordinates of the second point. The semantics of the "points" property depend on the control it specifies, and examples will be described in the following paragraphs. Points should be as follows: ControllersUI The "size" property is defined within the UI box.

[0076] In this example, the "controllers" property lists the controllers controlled by this control. The size of the list depends on the type of control.

[0077] Line control. One type of control used in some embodiments is the line control. When "type" is "line", the control is a line on which the user can slide the cursor. In this example, the "controllers" list is limited to a single item, which identifies the controller whose weights are controlled by this control. The line extends from point (x1, y1) to (x2, y2). The starting point of the line at (x1, y1) corresponds to the minimum weight value, and the ending point at (x2, y2) corresponds to the maximum weight value. The weight values ​​are linearly interpolated at intermediate positions. In some embodiments, the minimum and maximum weight values ​​are defined by the sampler's input properties (and can be modified by the "min" property). If no animation properties are defined in the controller, the sampler is in the "samplers" property. If animation properties are defined ("animation" is present), the sampler used to determine the minimum and maximum weight values ​​is the animation sampler. The input properties contain the weight / time values ​​of the controller / animation. The minimum and maximum weight values ​​are the minimum and maximum values ​​of these input values. These sampler inputs can be provided according to the glTF 2.0 standard.

[0078] A rectangular control. One type of control used in some embodiments is the rectangular control. When "type" is "rectangle," the control is a rectangle in which the user can move the cursor. In this example, the "controllers" list has two items that jointly identify two controllers whose weights are controlled by this control. The first corner of the rectangle is (x1, y1), and the opposite corner (the other side of the diagonal) is (x2, y2). The cursor's position within the rectangle defines the weights assigned to the two controllers. The horizontal axis corresponds to the first, and the vertical axis to the second. The first rectangle corner at (x1, y1) corresponds to the minimum weight value, and the opposite corner at (x2, y2) corresponds to the maximum weight value.

[0079] Ellipse Control. One type of control used in some embodiments is the ellipse control. When "type" is "ellipse," the control is an ellipse in which the user can move the cursor. The "controllers" list in this example has exactly two items, identifying the controllers whose weights are controlled by this control. The shape of the ellipse is determined by the coordinates in the "points" property. The shape of the ellipse can be determined differently in different embodiments. In one embodiment, the ellipse is the largest ellipse inside a fitted rectangle, with one corner at (x1, y1) and the opposite corner (the other side of the diagonal) at (x2, y2). In this example, the cursor's position inside the ellipse indicates the weights used for the two controllers, where the horizontal axis corresponds to the first weight and the vertical axis corresponds to the second weight. The weight values ​​can be calculated as in the example with the bounding rectangle, except that the cursor is constrained to remain inside the ellipse.

[0080] Controller User Interface Example For clarity, these examples only show the portions of the glTF file related to the controller user interface. In an actual implementation, the file also includes a scene defining one or more grids and a controller defined in the MPEG_Controllers extension. Therefore, the following only describes the loading of the controller user interface components.

[0081] In this example, we want to define a controller to animate the facial apparatus, and define controls to drive them in the user interface: Figure 2 The left side shows the facial equipment, and the right side shows the controls defined in the glTF file.

[0082] In the exemplary use case, the glTF file includes: • Head mesh (index 0), with 84 deformable targets.

[0083] • The eye grid (index 1) for the left eye is centered at the origin.

[0084] • The eye grid for the right eye (index 2) is centered at the origin.

[0085] • MPEG_controllers extension, with: 55 controllers.

[0086] o55 zero-value weights.

[0087] There are 52 controls (1 ellipse, 2 rectangles and 49 lines).

[0088] The following script represents a portion of an exemplary glTF file that includes three UI controls:

[0089] Line control. This section describes the first UI control (index 0) in the example above.

[0090] The name of this control is "s1". It is a "line" control with coordinates (154, 166) -> (118, 99). Figure 2 The line coordinates are shown in the upper left part of the control window. Note that the actual line coordinates in the control window may not be (154, 166) -> (118, 99), but rather scaled to the window size, such as x <- windowWidth. x / uiWidth, where uiWidth is the UI width defined in the "size" attribute of "ui" in MPEG_controllers.

[0091] The “controllers” list contains a single index: 44. This indicates that the UI control drives the controller at index 44. (The code for the controller itself is not shown in the code list.) In the example, the UI changes the weights of two deformable targets associated with the right outer eyebrow (we assume they are named AU4_innerbrowrlow_R1 and AU2_outerbrowraiser_R1 in the glTF file). Figure 3 The graph shows how this controller updates the deformable target, with weight values ​​ranging from -0.5 to 1.0. In its initial state, controller 44 has a weight of 0, mapping to zero weights in the deformable target. The initial weights of this control and other controls can be provided by the "weights" attribute in MPEG_controllers. In this state, the control displays a cursor on the line, approximately 1 / 3 from the starting point (154, 166) and approximately 2 / 3 from the ending point (118, 99). This cursor position is the result of linear interpolation between the minimum and maximum weight values.

[0092] When the user moves the cursor to the starting point at (154, 166) of the line, the weight of controller 44 becomes -0.5, which is the minimum weight value. Figure 4 This state is illustrated. When the user moves the cursor to the end point of the line at (118, 99), the weight of controller 44 becomes 1.0, which is the maximum weight value. Figure 5 This state is illustrated in the diagram.

[0093] Rectangular control. This section describes the second UI control (index 1).

[0094] In this example, the name of this control is "m34". It is a "rectangle" control with coordinates (96, 577) -> (52, 531). Figure 2This is visible in the lower left part of the control window. This control drives the controllers at indices 28 and 29, which change the weights of two morphing targets related to mouth movements. Controller 28 changes the mouth opening (morphing target "puckerOpen"), and controller 29 makes it look like a kiss (morphing target "AU18_lip_pucker"). Figure 6 This demonstrates how controller 28 updates the first deformable target within the range of 0.0 to 1.0. Figure 7 The same situation is shown for controller 29 and the second deformed target.

[0095] In the initial state, both controllers have zero weights, and the cursor is located at (96, 577), which is the bottom right corner of the rectangular control (e.g., Figure 2 As shown). If the user moves the cursor horizontally, the weight of the first controller (index 28) increases (as shown). Figure 8 As shown). If the user moves the cursor vertically, the weight of the second controller (index 29) increases (e.g.) Figure 9 As shown). If the user moves the cursor horizontally and vertically, both weights increase (e.g., Figure 10 (As shown).

[0096] Ellipse control. This section describes the third UI control, index 2.

[0097] In this example, the name of this control is "gaze". It is an "ellipse" control with coordinates (200, 262) -> (296, 218). Figure 2 The controller is visible in the middle of the control window. This control drives controllers at indices 0 and 1, which change the weights of the deformable targets related to eye movement and eye rotation. Controller 0 changes the eyelids (deformable targets "AU61_Eyes_turn_L" and "AU61_Eyes_turn_R") and rotates the eyeball around the Y-axis. Controller 1 changes the eyelids (deformable targets "AU63_eyeUp" and "AU63_eyeDown") and rotates the eyeball around the X-axis. Figure 11 This shows how controller 0 updates the first deformable target within the range of -1.0 to 1.0. Figure 12 The same situation was shown for controller 1 and the final deformed target.

[0098] In the initial state, both controllers can have zero weights, and the cursor is located at (248, 240), which is the middle of the elliptical control (e.g., ...). Figure 6 As shown). If the user moves the cursor to the right, the weight of the first controller (index 0) increases, and the character looks to the left (as shown). Figure 13 As shown). If the user moves the cursor down, the weight of the second controller (index 1) increases, and the character looks down (as shown). Figure 14As shown). If the user moves the cursor as far to the lower right as possible, both weights increase (e.g., ...). Figure 15 (As shown). In the last case, because the cursor cannot leave the ellipse, the control does not allow selecting the maximum weight value for two controllers simultaneously.

[0099] Exemplary parsing method The exemplary embodiment further includes parsing MPEG_controllers objects (such as those shown in Table 1). Parsing of MPEG_controllers extensions can be performed immediately after the normal parsing of the glTF file. Figure 16 The diagram illustrates a flowchart of an exemplary method for parsing MPEG_controllers extensions. Parsing of the extensions can be performed if the "extensions" attribute exists at the root of the glTF file, and if it contains the "MPEG_controllers" attribute.

[0100] The process parses the "weights" list in MPEG_controllers. Then, iterates through all items in the "controllers" list. Each item is a Controller, and its parsing is detailed below. Next, if an "ui" attribute exists in MPEG_controllers, the UI size is obtained from the "size" attribute within "ui," and all items in the "controls" list are iterated through. Each item in this list is a Control, and its parsing is detailed below.

[0101] Controller resolution. This section describes the resolution of the Controller (as shown in Table 2). Figure 17 A flowchart of an exemplary parsing process is provided. If it exists, the "name" attribute is parsed. Then, if the "animation" attribute exists, its value is used from the glTF file. animations The function in the asset retrieves the updated scene. Next, if a "min" property exists, it is used to offset the time (e.g., the value of the sampler input). If no "animation" property exists, the controller property is treated as an animation, the difference being that negative time values ​​are acceptable.

[0102] UI control parsing. This section describes the parsing of UI controls (as shown in Table 4). Figure 18 A flowchart of an exemplary parsing process is provided. In this process, the "name" attribute (if any) is parsed. Then the "type", "points", and "controllers" attributes are parsed.

[0103] Exemplary processing workflow When a user or team is creating or modifying 3D assets, one approach involves breaking the process down into several stages. Each stage focuses on a specific task, such as handling mesh modeling, texturing, animation, lighting, etc. These stages are often handled by different people or teams, or by the same person at different times. To manage this, one solution is to store the results of each stage in a new file. Using the current glTF standard, this approach is suitable for several tasks. However, the current glTF is not well-suited for other tasks, such as complex animations.

[0104] Given this situation, one approach involves using a set of animations (or controllers) for each model. For example, character animations like walking, jumping, or idling could be used. Faces are also associated with a set of facial expressions; each expression is encoded as an animation. In the early steps of the workflow, artists design these animations, where the current glTF standard is sufficient. In later stages, artists typically combine these animations to create the final animated scene. In the latter case, if this is the final stage, the combination of animations can be processed into the final animation in the current glTF file. However, if this is not the final stage, and if other artists want the ability to modify the final animation (e.g., in the lighting stage), this approach is no longer feasible. Instead, it requires artists in earlier stages to update the individual animations and process them again into the final animation.

[0105] However, using the exemplary embodiments described herein, the animation creation process can be stored in a glTF file. In this context, the basic animation is referred to as a controller, and the combination of animations is the controller's animation. The next stage artists can then modify the controller's animation itself. In some embodiments, the controller's animation can be performed by animatenizing the controller weights using a KHR_animation_pointer. For example, the "pointer" attribute of the KHR_animation_pointer can be set to MPEG controllers weights, which can be represented as " / extensions / MPEG_controllers / weights". An example of an animation for a controller identified in the MPEG_controllers extension can be arranged as follows:

[0106] Furthermore, 3D asset software that supports this extension can use the UI definitions coded in the proposed extension. This allows artists designing controllers to also design the user interface for later stages, making the creation and updating of controller animations easier.

[0107] In some embodiments, the controller user interface may be provided in 3D asset files generated by freelancers or other artists for sale (e.g., through an asset database), thereby allowing asset buyers to more easily modify their work and thus increase the value of the asset.

[0108] In some embodiments, the controller user interface can be used by end users, for example, to modify a 3D model of their own home (possibly to experiment with different designs and layouts) or to personalize their 3D avatars.

[0109] Exemplary System The scene described in this article can be encoded, decoded, processed, and rendered using systems such as... Figure 19 This is achieved through a system (such as a .). Figure 19 This is a block diagram of an exemplary system in which various aspects and embodiments are implemented. System 1000 may be embodied as a device including a variety of components and configured to perform one or more aspects described herein. Examples of such devices include, but are not limited to, a variety of electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, networked home appliances, and servers. Elements of System 1000 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete devices. For example, in at least one embodiment, the processing and encoder / decoder elements of System 1000 are distributed across multiple ICs and / or discrete devices. In various embodiments, System 1000 may be communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or via dedicated input and / or output ports. In various embodiments, System 1000 is configured to implement one or more aspects described herein.

[0110] System 1000 includes at least one processor 1010 configured to execute instructions loaded thereon to implement various aspects, such as those described herein. Processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 1000 includes at least one memory 1020 (e.g., volatile and / or non-volatile memory devices). System 1000 includes a storage device 1040, which may include non-volatile and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. Storage device 1040 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices, as non-limiting examples.

[0111] System 1000 includes an encoder / decoder module 1030 configured to, for example, process data to provide encoding or decoding scenarios, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module that can be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Furthermore, the encoder / decoder module 1030 may be implemented as a separate element of system 1000, or it may be incorporated within processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0112] Program code to be loaded onto processor 1010 or encoder / decoder 1030 to execute the various aspects described herein may be stored in storage device 1040 and then loaded into memory 1020 for execution by processor 1010. According to various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more items during the execution of the processes described herein. These stored items may include, but are not limited to, input scenes, decoded scenes or portions thereof, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operational logic.

[0113] In some embodiments, the memory within the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory (e.g., RAM) is used as working memory for encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2) or VVC (Various Video Coding, a new standard being developed by JVET (Joint Video Experts Group)).

[0114] Inputs to the components of system 1000 can be provided by a variety of input devices as shown in box 1130. These input devices include, but are not limited to: (i) an RF section that receives radio frequency (RF) signals, for example, transmitted over the air by a broadcaster; (ii) component input terminals (or sets of COMP input terminals); (iii) universal serial bus (USB) input terminals; and / or (iv) high-definition multimedia interface (HDMI) input terminals. Other examples include composite video.

[0115] In various embodiments, the input device of block 1130 has associated corresponding input processing elements (as known in the art). For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or band-limiting a signal to a frequency band); (ii) down-converting the selected signal; (iii) band-limiting it again to a narrower frequency band to select, for example, a signal band, which may be referred to as a channel in some embodiments; (iv) demodulating the down-converted and band-limited signal; (v) performing error correction; and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include a tuner that performs a variety of these functions, such as down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above (and other) components, remove some of them, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as inserting amplifiers and analog-to-digital converters. In many embodiments, the RF section includes an antenna.

[0116] Furthermore, USB and / or HDMI terminals may include corresponding interface processors for connecting System 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented as needed, for example, within a separate input processing IC or within processor 1010. Similarly, aspects of USB or HDMI interface processing may be implemented as needed, either within a separate interface IC or within processor 1010. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including processor 1010 and encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on the output device.

[0117] Various components of system 1000 can be housed within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data therebetween using suitable connection means 1140 (e.g., internal buses known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0118] System 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 1060 may be implemented, for example, in a wired and / or wireless medium.

[0119] In various embodiments, data is streamed to system 1000 using a wireless network such as Wi-Fi (e.g., IEEE 802.11 (IEEE stands for Institute of Electrical and Electronics Engineers)) or otherwise provided. In these embodiments, Wi-Fi signals are received via a communication channel 1060 and a communication interface 1050 adapted for Wi-Fi communication. The communication channel 1060 in these embodiments is typically connected to an access point or router, providing access to external networks (including the Internet) to allow streaming applications and other over-the-top services to communicate. Other embodiments use a set-top box to provide streaming data to system 1000, which transmits data via an HDMI connection of input box 1130. Still other embodiments use an RF connection of input box 1130 to provide streaming data to system 1000. As described above, various embodiments provide data in non-streaming modes. Furthermore, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0120] System 1000 can provide output signals to a variety of output devices, including display 1100, speaker 1110, and other peripheral devices 1120. Display 1100 in various embodiments includes, for example, one or more of a touchscreen display, organic light-emitting diode (OLED) display, curved display, and / or foldable display. Display 1100 can be a television, tablet computer, laptop computer, mobile phone, or other device. Display 1100 can also be integrated with other components (e.g., in a smartphone) or separate (e.g., an external monitor for a laptop computer). Other peripheral devices 1120 include, in various exemplary embodiments, one or more of a standalone digital video disc (or digital multi-function disc) (DVR, used for both terms), optical disc player, stereo system, and / or lighting system. Various embodiments use one or more peripheral devices 1120 that provide functionality based on the output of system 1000. For example, an optical disc player performs the function of playing the output of system 1000.

[0121] In various embodiments, control signals are transmitted between system 1000 and display 1100, speaker 1110, or other peripheral devices 1120 using communication protocols such as AV.Link, Consumer Electronics Control (CEC), or others that enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to system 1000 via communication interface 1050 using communication channel 1060. Display 1100 and speaker 1110 can be integrated into a single unit within an electronic device such as a television set, along with other components of system 1000. In various embodiments, display interface 1070 includes a display driver, such as a timing controller (TCon) chip.

[0122] Display 1100 and speaker 1110 may alternatively be separated from one or more other components, for example, if the RF portion of input 1130 is part of a separate set-top box. In various embodiments where display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection, such as including an HDMI port, a USB port, or a COMP output.

[0123] System 1000 may include one or more sensor devices 1095. Examples of sensor devices that may be used include one or more of a GPS sensor, gyroscope sensor, accelerometer, light sensor, camera, depth camera, microphone, and / or magnetometer. These sensors can be used to determine information such as the user's position and orientation. When system 1000 is used as a control module (e.g., control modules 124, 1254) for an augmented reality display, the user's position and orientation can be used to determine how to render image data so that the user perceives the correct portion of a virtual object or scene from the correct perspective. In the case of a head-mounted display device, the position and orientation of the device itself can be used to determine the user's position and orientation in order to render virtual content. In the case of other display devices (e.g., telephone, tablet, computer monitor, or television), other inputs can be used to determine the user's position and orientation in order to render content. For example, the user can use a touchscreen, keypad or keyboard, trackball, joystick, or other inputs to select and / or adjust the desired viewpoint and / or viewing direction. If the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation used to render content can be selected and / or adjusted based on the movement of the display device.

[0124] The embodiments may be executed by computer software or hardware, or a combination of hardware and software, implemented by processor 1010. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. Memory 1020 may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, such as optical storage devices, magnetic storage devices, semiconductor-based storage devices, fixed memory, and removable memory, as a non-limiting example. Processor 1010 may be of any type suitable for the technical environment and may include one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures, as a non-limiting example.

[0125] Further Examples A method according to some embodiments includes: acquiring a runtime asset delivery file for a 3D scene, the file including: a plurality of controller syntax structures, each controller syntax structure defining an association between a weight range and a corresponding output transformation of at least one associated node in the scene; and a plurality of user interface syntax structures, each user interface syntax structure including information identifying at least one associated controller syntax structure and at least one location. The method further includes: for at least a first user interface syntax structure in the user interface syntax structures, causing at least a first user interface element to be displayed at a location identified by the first user interface syntax structure; receiving user input indicating a first weight through the first user interface element; determining a first output transformation associated with the first weight based on the first controller syntax structure identified by the first user interface syntax structure; and generating a transformed node by applying the first output transformation to the node associated with the first controller syntax structure.

[0126] Some embodiments also include displaying the transformed node. In some embodiments, the transformed node and the user interface element are displayed simultaneously.

[0127] In some embodiments, the nodes associated with the first controller syntax structure include a grid.

[0128] In some embodiments, the first user interface element includes a movable cursor within a range defined by the first user interface syntax structure, and wherein the first weight is indicated by the position of the cursor within that range.

[0129] In some embodiments, the first user interface syntax structure indicates a start point and an end point, wherein the first user interface element includes a cursor movable along a line between the start point and the end point, and wherein the first weight is indicated by the position of the cursor within that range.

[0130] In some embodiments, the first user interface syntax structure defines a rectangle; the first user interface element includes a movable cursor within the rectangle; the first weight is indicated by the vertical position of the cursor within the rectangle; and the second weight is indicated by the horizontal position of the cursor within the rectangle.

[0131] In some embodiments, the first user interface syntax structure defines an ellipse; the first user interface element includes a movable cursor within the ellipse; the first weight is indicated by the vertical position of the cursor within the ellipse; and the second weight is indicated by the horizontal position of the cursor within the ellipse.

[0132] Some embodiments further include: determining a second output transformation associated with the second weight based on the second controller syntax structure in the runtime asset delivery file; wherein generating the transformed node further includes applying the second output transformation to the node associated with the first controller syntax structure.

[0133] In some embodiments, the transformation includes at least one of the following: rotation, translation, scaling change, or change of the weight of the deformable target.

[0134] Some embodiments include storing the first weight in the runtime asset delivery file after receiving user input indicating the first weight.

[0135] The method according to some embodiments includes: encoding at least one node in a runtime asset delivery file for a 3D scene; encoding a plurality of controller syntax structures in the runtime asset delivery file, each controller syntax structure defining an association between an input weight range and a corresponding output transformation of at least one associated node in the scene; and encoding a plurality of user interface syntax structures in the runtime asset delivery file, each user interface syntax structure including information identifying at least one associated controller syntax structure and at least one location, each user interface syntax structure defining a corresponding user interface element for modifying the input weights of the corresponding controller syntax structure.

[0136] According to some embodiments, the signal includes a runtime asset delivery file for a 3D scene, wherein the scene description data includes: at least one node; a plurality of controller syntax structures, each controller syntax structure defining an association between an input weight range and a corresponding output transformation of at least one associated node in the scene; and a plurality of user interface syntax structures, each user interface syntax structure including information identifying at least one associated controller syntax structure and at least one location, each user interface syntax structure defining a corresponding user interface element for modifying the input weights of the corresponding controller syntax structure. In some embodiments, the asset delivery file is a JSON file. In some embodiments, the runtime asset delivery file is a glTF file.

[0137] Further embodiments include one or more processors configured to perform any of the methods described herein.

[0138] Further embodiments include a computer-readable medium (which may be non-transitory) containing instructions for causing one or more processors to perform any of the methods described herein.

[0139] Further embodiments include a computer program product containing instructions that, when executed by one or more processors, cause the one or more processors to perform any of the methods described herein.

[0140] This disclosure describes a variety of aspects, including tools, features, embodiments, models, solutions, etc. Many of these aspects are described in detail and are generally described in a manner that sounds restrictive, at least to show their respective characteristics. However, this is for clarity of description and does not limit the scope of this disclosure or those aspects. In fact, all the different aspects can be combined and interchanged to provide further aspects. Furthermore, these aspects can be combined and interchanged with aspects described in previous applications.

[0141] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are specifically illustrated, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of implementation. At least one aspect relates generally to video encoding and decoding, and at least another aspect relates generally to transmitting generated or encoded bitstreams. These and other aspects can be implemented as methods, apparatus, computer-readable storage media storing instructions for encoding or decoding video data according to any described method, and / or computer-readable storage media storing bitstreams generated according to any described method.

[0142] This document describes various methods, each including one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for the method to operate correctly, the order and / or use of specific steps and / or actions can be modified or combined. Furthermore, terms such as "first" and "second" can be used in various embodiments to modify elements, components, steps, operations, etc., e.g., "first decoding" and "second decoding." Unless specifically required, the use of these terms does not imply an ordering of the modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding and can occur, for example, before, during, or overlapping with the second decoding.

[0143] Various numerical values ​​may be used in this disclosure. Specific values ​​are for illustrative purposes only, and the aspects described are not limited to these specific values.

[0144] The embodiments described herein can be executed by computer software or a combination of hardware and software, which is executed by a processor or other hardware. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The processor can be of any type suitable for the technical environment and can include one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures, as a non-limiting example.

[0145] When accompanying drawings are presented in flowchart form, it should be understood that a block diagram of the corresponding apparatus is also provided. Similarly, when accompanying drawings are presented in block diagram form, it should be understood that a flowchart of the corresponding method / process is also provided.

[0146] The implementations and aspects described herein can be implemented, for example, in methods or processes, apparatuses, software programs, data streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the features in question can also be implemented in other forms (e.g., apparatuses or programs). Apparatuses can be implemented, for example, in appropriate hardware, software, and firmware. These methods can be implemented, for example, in a processor, which generally refers to a processing device, such as a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0147] References to “an embodiment” or “an embodiment” or “an implementation” or “an implementation” and other variations mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the phrases “in an embodiment” or “in an embodiment” or “in an implementation” or “in an implementation” and any other variations appearing throughout this disclosure do not necessarily refer to the same embodiment.

[0148] Furthermore, this disclosure may involve "determining" various types of information. Determining information may include, for example, one or more of the following: estimated information, calculated information, predicted information, or information retrieved from memory.

[0149] Furthermore, this disclosure may involve "accessing" various types of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or one or more of these.

[0150] Furthermore, this disclosure may relate to "receiving" various types of information. Receiving, like "accessing," is a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory). Moreover, "receiving" generally involves operations in some way, such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0151] It should be understood that the use of any of the following " / ", "and / or", and "at least one of", such as in "A / B", "A and / or B", and "at least one of A and B", is intended to include selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As another example, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). This can be extended to any number of listed items.

[0152] Furthermore, as used herein, the term "signal" refers to instructing the corresponding decoder to do something. For example, in some embodiments, the encoder signals a specific one of several parameters used for region-based filter parameter selection for artifact removal filtering. Thus, the same parameter is used on both the encoder and decoder sides in this embodiment. Therefore, for example, the encoder can transmit (explicit signaling) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has that specific parameter along with others, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various embodiments by avoiding the transmission of any actual function. It should be understood that signaling can be done in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. While the foregoing refers to the verb form of the term "signal," the term "signal" may also be used herein as a noun.

[0153] The implementation can generate signals in various formats to carry information that can be stored or transmitted, for example. This information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, the signal can be formatted to carry a bit stream of the described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. The signal can be transmitted via a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0154] We have described several embodiments. The features of these embodiments may be provided individually or in any combination across multiple claim classes and types. Furthermore, embodiments may include one or more of the following features, devices, or aspects individually or in any combination across multiple claim classes and types.

[0155] • Includes a bitstream or signal of one or more described syntax elements or their variants.

[0156] • Includes a bitstream or signal carrying a syntax for information generated according to any of the described embodiments.

[0157] • Creating and / or transmitting and / or receiving and / or decoding bitstreams or signals that include one or more of the described syntax elements or variations thereof.

[0158] • Create and / or transmit and / or receive and / or decode according to any of the described embodiments.

[0159] • Methods, processes, apparatus, media for storing instructions, media for storing data, or signals according to any of the described embodiments.

[0160] Note that the various hardware elements of one or more of the described embodiments may be referred to as “modules” that perform (i.e., carry out, implement, etc.) the various functions described herein in connection with the respective modules. As used herein, a module includes hardware deemed suitable for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more storage devices). Each described module may also include executable instructions for performing one or more functions described as being performed by the respective module, and note that these instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., and may be stored in any suitable non-transitory computer-readable medium (e.g., commonly referred to as RAM, ROM, etc.).

[0161] Although the features and elements have been described above in specific combinations, each feature or element can be used alone or in any combination with other features and elements. Furthermore, the methods described herein can be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor storage devices, magnetic media (such as internal hard disks and removable disks), magneto-optical media, and optical media (such as CD-ROMs and digital versatile optical discs (DVDs)). The processor associated with the software can be used to implement a radio frequency transceiver used in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method comprising: Obtain runtime asset delivery files for a 3D scene, the files including: multiple controller syntax structures, each controller syntax structure defining the association between a weight range and the corresponding output transformation of at least one associated node in the scene; and multiple user interface syntax structures, each user interface syntax structure including information identifying at least one associated controller syntax structure and at least one associated location; For at least a first user interface syntax structure in the user interface syntax structure, such that at least a first user interface element is displayed at the associated location; The user input indicating the first weight is received through the first user interface element; Based on the first controller syntax structure identified by the first user interface syntax structure, determine the first output transformation associated with the first weight; and Transformed nodes are generated by applying the first output transformation to the nodes associated with the first controller syntax structure.

2. An apparatus comprising one or more processors, said one or more processors configured to perform at least: Obtain runtime asset delivery files for the 3D scene, the files including: Multiple controller syntax structures, each controller syntax structure defining the association between a weight range and the corresponding output transformation of at least one associated node in the scene; And multiple user interface syntax structures, each user interface syntax structure including information identifying at least one associated controller syntax structure and at least one associated location; For at least a first user interface syntax structure in the user interface syntax structure, such that at least a first user interface element is displayed at the associated location; The user input indicating the first weight is received through the first user interface element; Based on the first controller syntax structure identified by the first user interface syntax structure, a first output transformation associated with the first weight is determined; as well as Transformed nodes are generated by applying the first output transformation to the nodes associated with the first controller syntax structure.

3. The method of claim 1 or the apparatus of claim 2, further comprising displaying the transformed node.

4. The method of claim 1, or claim 3 which is a subset of claim 1, or the apparatus of claim 2, or claim 3 which is a subset of claim 2, wherein the transformed node and the user interface element are displayed simultaneously.

5. The method of claim 1, or the apparatus of claims 3-4 of claim 1, or the apparatus of claim 2, or the apparatus of claims 3-4 of claim 2, wherein the first user interface element includes a cursor movable along a line between a start point and an end point, and wherein the first weight is indicated by the position of the cursor within the range.

6. The method as claimed in claim 1, or the apparatus as claimed in claim 3-4, which are dependent on claim 1, or the apparatus as claimed in claim 2, or the apparatus as claimed in claim 3-4, which are dependent on claim 2, wherein: The first user interface syntax structure defines a rectangular or elliptical two-dimensional shape; The first user interface element includes a cursor that can move within the two-dimensional shape; The first weight is indicated by the vertical position of the cursor within the two-dimensional shape; and The second weight is indicated by the horizontal position of the cursor within the two-dimensional shape.

7. The method of claim 1, or the apparatus of claims 3-6, which are dependent on claim 1, or the apparatus of claim 2, or the apparatus of claims 3-6, which are dependent on claim 2, further comprising: Based on the second controller syntax structure in the runtime asset delivery file, determine the second output transformation associated with the second weight; Generating the transformed node further includes applying the second output transformation to the node associated with the first controller syntax structure.

8. The method of claim 1, or the apparatus of claims 3-7 of claim 1, or the apparatus of claim 2, or the apparatus of claims 3-7 of claim 2, wherein the first output transformation includes at least one of the following: rotation, translation, scaling change, and change of deformation target weight.

9. The method of claim 1, or the apparatus of claims 3-8 of claim 1, or the apparatus of claim 2, or the apparatus of claims 3-8 of claim 2, further comprising, after receiving the user input indicating the first weight, storing the first weight in the runtime asset delivery file.

10. A method comprising: Encode at least one node in the runtime asset delivery file used for 3D scenes; Multiple controller syntax structures are encoded in the runtime asset delivery file, each controller syntax structure defining the association between the input weight range and the corresponding output transformation of at least one associated node in the scene; as well as The runtime asset delivery file encodes multiple user interface syntax structures, each user interface syntax structure including information identifying at least one associated controller syntax structure and at least one location, and each user interface syntax structure defining a corresponding user interface element for modifying the input weights of the corresponding controller syntax structure.

11. An apparatus comprising one or more processors, said one or more processors configured to perform at least: Encode at least one node in the runtime asset delivery file used for 3D scenes; The runtime asset delivery file encodes multiple controller syntax structures, each controller syntax structure defining the association between the input weight range and the corresponding output transformation of at least one associated node in the scene; and The runtime asset delivery file encodes multiple user interface syntax structures, each user interface syntax structure including information identifying at least one associated controller syntax structure and at least one location, and each user interface syntax structure defining a corresponding user interface element for modifying the input weights of the corresponding controller syntax structure.

12. The method of claim 10 or the apparatus of claim 11, wherein at least one of the user interface syntax structures indicates a start point and an end point, wherein the corresponding user interface element includes a cursor movable along a line between the start point and the end point, and wherein the corresponding input weight is indicated by the position of the cursor within the range.

13. The method of claim 10 or the apparatus of claim 11, wherein: At least one of the user interface syntax structures defines a rectangular or elliptical two-dimensional shape; The corresponding user interface elements include a cursor that can move within the two-dimensional shape; The first input weight is indicated by the vertical position of the cursor within the two-dimensional shape; and The second input weight is indicated by the horizontal position of the cursor within the two-dimensional shape.

14. The method of claim 10, or claims 12-13 which are dependent on claim 10, or the apparatus of claim 11, or claims 12-13 which are dependent on claim 11, wherein the transformation includes at least one of the following: rotation, translation, scaling change, and change of deformation target weight.

15. A signal including a runtime asset delivery file for a 3D scene, wherein the runtime asset delivery file includes: At least one node; Multiple controller syntax structures, each controller syntax structure defining the association between the input weight range and the corresponding output transformation of at least one associated node in the scene; as well as Multiple user interface syntax structures, each user interface syntax structure including information identifying at least one associated controller syntax structure and at least one location, and each user interface syntax structure defining a corresponding user interface element for modifying the input weights of the corresponding controller syntax structure.

Images