How to parse a scene description file

Abstract

The present disclosure provides a method and apparatus for generating a scene description file, the apparatus including: determining a haptic media accessor associated with a target action node when a three-dimensional scene to be rendered performs a target haptic action; the haptic media accessor being an accessor for accessing haptic rendering data of a haptic media file declared in an MPEG media description module associated with the target action node when the target haptic action is performed; generating a target haptic object description module based on the haptic media accessor and adding the target haptic object description module to the MPEG haptic description module; generating a target haptic action node array based on an index value of a node description module corresponding to the target action node and an index value of the target haptic object description module; and adding the target haptic action node array to a haptic action node list of the target action description module.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) Original Vow claims priority from the following Chinese patent applications: Chinese patent application having application number 202310036790.8 filed on January 10, 2023; Chinese patent application having application number 202310383151.9 filed on April 11, 2023; Chinese patent application having application number 202310362214.2 filed on April 6, 2023; Chinese patent application having application number 202311236289.2 filed on September 22, 2023; and Chinese patent application having application number 202310275300.X filed on March 20, 2023, and the contents of each prior application are incorporated by reference. Original Vow will be incorporated into

[0002] Original Vow Some embodiments of the present invention relate to the field of video processing, in particular to the processing of scene description files. analysis direction By law Regarding. [Background technology]

[0003] As 3D media content becomes increasingly richer, users have higher requirements for the realism and richness of virtual scenes. As part of the sensory experience, haptics is an object experience that needs to be considered and added to immersive media. For example, users can experience an explosion effect in a virtual scene, which not only requires the involvement of vision and hearing, but also the involvement of haptics as a sensory experience, enhancing the realism and presence of the virtual explosion effect.

[0004] In related technical standards, each haptic material attribute is represented by a texture array, and texture arrays of different attributes are combined to create a specific haptic sensation. Each array element includes a haptic texture and a haptic material attribute type represented as an enumeration, which includes high resolution, low resolution, reference, and other. When the haptic material attribute type is high resolution or low resolution, haptic values, rather than color values, are stored in the texture map or haptic texture material data packet of the texture index. These haptic values ​​can be directly retrieved and haptically rendered by the display engine. When the haptic material attribute type is reference, each pixel in the texture map or haptic texture material data packet stores an index value of the haptic media file of the rendering object, rather than a directly stored haptic value. As mentioned above, when the haptic material attribute type is reference, the data obtained from the texture map or haptic texture material data packet is an index value of the haptic media, and based on the index value, only the encoded and packaged haptic media file can be obtained, but the decoded haptic value usable for rendering cannot be obtained, thereby causing the display engine to be unable to perform rendering successfully. Summary of the Invention

[0005] In a first aspect, Original Vow Some embodiments of the scene description file analysis A method is provided, the method comprising: The method includes: obtaining a target haptic action node array from a haptic action node list of a target action description module of a Moving Picture Experts Group MPEG interactive description module of a scene description file of a three-dimensional scene to be rendered; determining an index value of a target haptic object description module corresponding to a haptic object associated with the target action node based on the target haptic action node array; obtaining the target haptic object description module from the MPEG haptic description module of the scene description file based on the index value of the target haptic object description module; and obtaining, based on the target haptic object description module, description information of a haptic media accessor associated with the target action node when the three-dimensional scene to be rendered performs a target haptic action, wherein the target haptic action is a haptic action corresponding to the target action description module, and the haptic media accessor is an accessor for accessing haptic rendering data of a haptic media file declared in the MPEG media description module associated with the target action node when the target haptic action is performed. .

[0006] In a second aspect, Original Vow Some embodiments of the present invention further comprise a memory configured to store a computer program, and a computer program that, when invoked, reads the scene description file. analysis a scene description file according to the first aspect of the present invention; analysisa processor configured to implement the method; and a scene description file including analysis A device is provided.

[0007] In a third aspect, Original Vow Some embodiments of the present invention provide a computer-readable storage medium having stored thereon a computer program, the computer program being configured to, when executed by a computing device, cause the computing device to read a scene description file according to the first aspect. analysis Make the method happen.

[0008] In a fourth aspect, Original Vow Some embodiments of the present invention provide a computer program product, which when executed on a computer, causes the computer to analysis Make the method happen. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 shows a structural diagram of an immersive media description framework in some embodiments of the present application. [Figure 2] FIG. 2 shows a schematic diagram of the structure of a scene description file in some embodiments of the present application. [Figure 3] FIG. 3 shows a structural diagram of a scene description file in another embodiment of the present application. [Figure 4] FIG. 4 shows a data model diagram of haptic data in some embodiments of the present application. [Figure 5] FIG. 5 shows a schematic diagram of the human body hierarchy in some embodiments of the present application. [Figure 6] FIG. 6 is a structural schematic diagram of a haptic signal encoder according to some embodiments of the present application. [Figure 7] FIG. 7 shows a structural schematic diagram of a haptic signal decoder in some embodiments of the present application. [Figure 8] FIG. 8 shows a schematic diagram of the structure of a scene description file in another embodiment of the present application. [Figure 9] FIG. 9 shows a structural diagram of a scene description file in another embodiment of the present application. [Figure 10] FIG. 10 shows a flowchart of the steps of a method for generating a scene description file in some embodiments of the present application. [Figure 11] FIG. 11 shows a flowchart of the steps of a method for analyzing a scene description file in some embodiments of the present application. [Figure 12] FIG. 12 shows a step flow chart of a method for processing haptic media files in some embodiments of the present application. [Figure 13] FIG. 13 shows a step flow chart of a three-dimensional scene rendering method in some embodiments of the present application. [Figure 14] FIG. 14 shows a step flow chart of a buffer management method in some embodiments of the present application. [Figure 15] FIG. 15 shows a schematic diagram of the structure of a scene description file in another embodiment of the present application. [Figure 16] FIG. 16 shows a flowchart of the steps of a method for generating a scene description file in some embodiments of the present application. [Figure 17] FIG. 17 shows a flowchart of the steps of a scene description file analysis method in some embodiments of the present application. [Figure 18] FIG. 18 shows a step flow chart of a scene description file analysis method in another embodiment of the present application. [Figure 19] FIG. 19 shows a step flow chart of a method for processing haptic media files in some embodiments of the present application. [Figure 20] FIG. 20 shows a step flow chart of a three-dimensional scene rendering method in some embodiments of the present application. [Figure 21] FIG. 21 shows a step flow chart of a buffer management method in some embodiments of the present application. [Figure 22]FIG. 22 shows an interaction flowchart of a three-dimensional scene rendering method in some embodiments of the present application. [Figure 23] FIG. 23 shows a schematic diagram of the structure of a scene description file in another embodiment of the present invention. [Figure 24] FIG. 24 shows a flowchart of steps in a method for generating a scene description file in some embodiments of the present application. [Figure 25] FIG. 25 shows a flowchart of the steps of a scene description file analysis method in some embodiments of the present application. [Figure 26] FIG. 26 shows a step flow chart of a scene description file analysis method in another embodiment of the present application. [Figure 27] FIG. 27 shows a step flow chart of a method for processing haptic media files in some embodiments of the present application. [Figure 28] FIG. 28 shows a step flow chart of a three-dimensional scene rendering method in some embodiments of the present application. [Figure 29] FIG. 29 shows a step flow chart of a buffer management method in some embodiments of the present application. [Figure 30] FIG. 30 shows an interaction flowchart of a three-dimensional scene rendering method in some embodiments of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0010] Original Vow In order to clarify the purpose and embodiments of the present invention, the following description will be given. Original Vow With reference to the drawings in an exemplary embodiment of Original Vow The exemplary embodiments of the present invention have been clearly and completely described, and it is clear that the exemplary embodiments described Original Vow These are only some examples, not all examples.

[0011] In addition, Original Vow The general explanation of terms in the following is provided to facilitate understanding of the embodiments described below. Original Vow Unless otherwise explained, these terms should be understood in accordance with their ordinary and customary meaning.

[0012] The terms "comprises" and "having" and any variations thereof are intended to cover but not exclusively include, for example, a product or device including a set of components is not necessarily limited to all components expressly listed, but may include other components not expressly listed or inherent to those products or devices.

[0013] References in the specification to "some implementations," "some embodiments," and the like indicate that the described implementations or embodiments may include a particular feature, structure, or characteristic, but do not necessarily mean that every embodiment includes that particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same implementation. Additionally, when a particular feature, structure, or characteristic is described in connection with one embodiment, it is believed to be within the knowledge of one skilled in the art that such feature, structure, or characteristic may also be implemented in connection with other implementations (whether or not expressly described herein).

[0014] Original VowSome embodiments relate to a scene description for immersive media. Referring to the immersive media scene description framework shown in FIG. 1 , the immersive media scene description framework decouples media file access and processing from media file rendering so that the display engine 11 can focus on media rendering, and designs a media access function (MAF) 12 to handle media file access and processing functions. At the same time, a media access function application programming interface (API) is designed, and command interaction is performed between the display engine 11 and the media access function 12 via the media access function API. The display engine 11 can send commands to the media access function 12 via the media access function API, and the media access function 12 can also request commands from the display engine 11 via the media access function API.

[0015] The general workflow of the immersive media scene description framework is as follows: 1) the display engine 11 reads the scene description file (Scene Description File) provided by the immersive media service provider; 2) the display engine 11 parses the scene description file to obtain parameters or information such as the access address of the media file, attribute information of the media file (such as media type and codec parameters), and a format request for the processed media file, and calls the media access function API to transmit all or part of the information obtained by parsing the scene description file to the media access function 12; 3) the media access function 12 requests download of the specified media file from the media resource server or obtains the specified media file locally based on the information transmitted from the display engine 11, establishes a corresponding pipeline for the media file, and then performs processes such as depackaging, decryption, and decoding on the media file in the pipeline to convert the media file from the package format to a format specified for the display engine 11; 4) the pipeline stores the output data for which all processing has been completed in a specified buffer; and 5) finally, the display engine 11 reads the completely processed data from the specified buffer and renders the media file based on the data read from the buffer.

[0016] The files and functional modules associated with the immersive media scene description framework are further described below.

[0017] 1. Scene description file

[0018] In the workflow of the immersive media scene description framework, a scene description file is used to describe the contents of a 3D scene, such as its structure (whose features can be described by a 3D mesh), textures (e.g., texture resources), animations (rotation, translation), and camera viewpoint (rendering angle).

[0019] In the related art, the GL Transmission Format 2.0 (glTF2.0) has been determined as a candidate format for scene description files, and can meet the requirements of Motion Picture Experts Group - Immersive (MPEG-I) and Six Degrees of Freedom (6DoF) applications. For example, glTF2.0 is described in the Khronos Group's GL Transmission Format (glTF) Version 2.0, available at github.com / KhronosGroup / glTF / tree / master / specification / 2.0#specifying-extensions. Referring to Figure 2, Figure 2 is a schematic diagram of the structure of a scene description file in the glTF2.0 scene description standard (ISO / IEC 12113). As shown in FIG. 2, a scene description file in the glTF2.0 scene description standard includes, but is not limited to, a scene description module (scene) 201, a node description module (node) 202, a mesh description module (mesh) 203, an accessor description module (accessor) 204, a buffer slice description module (bufferView) 205, a buffer description module (buffer) 206, a camera description module (camera) 207, a light illumination description module (light) 208, a material description module (material) 209, a texture description module (texture) 210, a sampler description module (sampler) 211, a texture resource description module (image) 212, an animation description module (animation) 213, and a skin description module (skin) 214.

[0020] 2 is used to describe a 3D scene included in the scene description file. One scene description file may include any number of 3D scenes, and each 3D scene is described using one scene description module 201. There is a parallel relationship between the scene description modules 201, i.e., there is a parallel relationship between 3D scenes.

[0021] The node description module (node) 202 in the scene description file shown in FIG. 2 is the next hierarchical description module of the scene description module 201, and is used to describe the objects included in the 3D scene described by the scene description module 201. Each 3D scene may contain many concrete objects, such as a virtual digital human, nearby 3D objects, and distant background images. The scene description file is structured as follows: 202 These concrete objects are described by the following expressions. Each node description module 202 can represent one object or a set of objects consisting of multiple objects. The relationships between the node description modules 202 reflect the relationships between the components in the 3D scene described by the scene description module 201. A 3D scene described by a single scene description module 201 may contain one or multiple nodes. Multiple nodes may have parallel or hierarchical relationships, i.e., a "contains" and "is contained" relationship exists between the node description modules 202, thereby describing multiple concrete objects collectively or individually. When one node is contained within another node, the contained node is called a subnode, and the word "node" is replaced with "children" for the subnode. Nodes and subnodes can be flexibly combined to form a hierarchical node structure, thereby expressing rich scene content.

[0022] 2 is a description module at the next level after the node description module 202, and is used to describe the characteristics of the object represented by the node description module 202. The mesh description module 203 is a collection of one or more primitives, and each primitive may include one attribute, which defines the attributes that need to be used by a graphics processing unit (GPU) when rendering. The attributes may include position (three-dimensional coordinates), normal (normal vector), tangent (tangent vector), texcoord_n (texture coordinates), color_n (color: RGB or RGBA), joints_n (attributes related to the skin description module 214), and weights_n (attributes related to the skin description module 214). Because the mesh description module 203 contains a large number of vertices, each containing various attribute information, it is inconvenient to directly store the large amount of media data contained in the media file in the mesh description module 203 of the scene description file. Instead, the access address (URI) of the media file is indicated in the scene description file, and data in the media file can be downloaded when needed, thereby realizing the separation of the scene description file and the media file. Therefore, generally, the mesh description module 203 does not store media data, but stores the index value of the accessor description module (accessor) 204 corresponding to each attribute, and the accessor description module 204 points to the corresponding data in the buffer slice (bufferView) of the buffer (buffer).

[0023] In some embodiments, the scene description file and the media files may be merged to form a single binary file, thereby reducing the number and variety of files.

[0024] Furthermore, a primitive in the mesh description module 203 may have a syntax element called mode, which is used to describe the topology structure when a graphics processing unit (GPU) draws a 3D mesh, for example, mode=0 represents a scattered point, mode=1 represents a line, and mode=4 represents a triangle.

[0025] The definitions of the syntax elements in the primitive attributes (mash.primitives.attributes) of the mesh description module 203 are shown in Table 1 below.

[0026] [Table 1]

[0027] The definition of the indexed access types in the primitive attributes (mash.primitives.attributes) of the mesh description module 203 is shown in Table 2 below.

[0028] [Table 2]

[0029] The component type definitions in the primitive attributes (mash.primitives.attributes) of the mesh description module 203 are shown in Table 3 below.

[0030] [Table 3]

[0031] For example, the following is a JSON example for the mesh description module 203: [Table A]

[0032] In the mesh description module 203, the value of “position” is 1, which points to the accessor description module whose index is 1 and ultimately points to the vertex coordinate data stored in the buffer, and the value of “color_0” is 2, which points to the accessor description module whose index is 2 and ultimately points to the color data stored in the buffer.

[0033] 2, the accessor description module (accessor) 204, buffer slice description module (bufferView) 205, and buffer description module (buffer) 206 are used to realize layer-by-layer refinement indexing of rendering data of a media file by the mesh description module 203. As described above, specific media data is not stored in the mesh description module 203, but the index value of the corresponding accessor description module 204 is stored, and specific media data is accessed by the accessor description module 204 indexed by the index value. The process of indexing media data by the mesh description module 203 includes first, the index value declared in the syntax element of the primitive attribute of the mesh description module 203 points to the corresponding accessor description module 204, then, the accessor description module 204 points to the corresponding buffer slice description module 205, and finally, the buffer slice description module 205 points to the corresponding buffer description module 206. 2, the buffer description module 206 mainly serves to point to the corresponding media file, and includes the URI and byte length of the media file, etc., and is used to describe a buffer for buffering the rendering data of the media file. A buffer may be divided into multiple buffer slices, and the buffer slice description module 205 mainly performs partial access to the media file rendering data in the buffer, including the start byte offset of the access data and the byte length of the access data, etc., and partial access to the rendering data of the media file can be realized by the buffer slice description module 205 and the buffer description module 206. The accessor description module 204 mainly serves to add additional information to the portion of data defined in the buffer slice description module 205, such as the data type, the number of data of that type, and the numerical range of that type of data.Such a three-layer structure can realize the function of retrieving part of the data from one media file, which is advantageous for accurate retrieval of data and for reducing the number of media files.

[0034] 2 is a description module at the next level after the node description module 202, and is used to describe visual information such as the viewpoint and viewing angle when a user views an object described by the node description module 202. In order to enable a user to be placed in a three-dimensional scene and to view the three-dimensional scene, the node description module 202 points to the camera description module 207, which can also describe visual information such as the viewpoint and viewing angle when a user views an object described by the node description module 202.

[0035] The light illumination description module (light) 208 in the scene description file shown in Figure 2 is a description module at the next level after the node description module 202, and is used to describe information about illumination such as the light illumination intensity, ambient light color, light illumination direction, and light source position of the object described by the node description module 202.

[0036] The material description module (material) 209 in the scene description file shown in FIG. 2 is a description module at the next level after the mesh description module 203 and is used to describe material information of the 3D object described by the mesh description module 203. When describing a 3D object, describing the geometric information of the 3D object using the mesh description module 203 or simply defining the color and / or position of the 3D object cannot improve the realism of the 3D object; more information must be added to the surface of the 3D object. In typical 3D modeling techniques, this process may be abbreviated as texture mapping or texture addition. The scene description file in the glTF 2.0 scene description standard also uses this description module. The material description module 209 defines materials using a set of common parameters and describes the material information of geometric objects appearing in a 3D scene. The material description module 209 generally uses a metallic-roughness model to describe the material of a virtual object, and the material property parameters based on the metallic-roughness model are represented by a material based on the widely used physically based rendering (PBR). Based on this, the material description module 209 describes the metallic-roughness material attributes of the object in detail, and the definitions of the syntax elements in the material description module 209 are shown in Table 4.

[0037] [Table 4]

[0038] In some embodiments, definitions of syntax elements in material.Pbr Metallic Roughness in the material description module 209 are shown in Table 5 below.

[0039] [Table 5]

[0040] The value of each attribute in the Metallic Roughness (material.Pbr Metarial Roughness) of the material description module 209 may be defined using coefficients and / or textures (e.g., base Color Texture and base Color Factor). If a texture is not provided, all corresponding texture components in this material model must assume a value of 1.0. If a coefficient and texture are present at the same time, the coefficient value is a linear multiplier of the corresponding texture value. A texture binding is defined by a texture object index and a selectable texture coordinate index.

[0041] Illustratively, the following is a JSON example of the material description module 209: [Table B]

[0042] By analyzing the material description module 209, it can be determined that the current material is named “gold” based on the material name syntax element and its value (“name”: “gold”), that the base color value of the current material is [1.000, 0.766, 0.336, 1.0] based on the color syntax element in the metal-roughness array (pbr Metallic Roughness) and its value (“basecolorFactor”: [1.000, 0.766, 0.336, 1.0]), that the metalness value of the current material is “1.0” based on the metallicity syntax element in the metal-roughness array (pbr Metallic Roughness) and its value (“metalnessFactor”: 1.0), and that the roughness value of the current material is “0.0” based on the roughness syntax element in the metal-roughness array (pbr MetallicRoughness) and its value (“roughnessFactor”: 0.0).

[0043] The texture description module (texture) 210 in the scene description file shown in FIG. 2 is the next hierarchical description module after the material description module 209 and is used to describe the color and other characteristics used in material definition of the three-dimensional object described by the material description module 209. Texture is one of the important aspects that gives an object a realistic appearance. By defining the object's primary color and other characteristics used in material definition through texture, the appearance of the rendered object can be accurately described. A texture itself can define multiple texture objects, which can be used as textures for virtual objects during rendering and can be used to encode different texture attributes. The texture description module 210 references one sampler description module (sampler) 211 and one texture resource description module (image) 212 using the sampler index and texture resource index syntax elements. The texture resource description module 212 contains a uniform resource identifier (URI) linked to the texture image or binary file package actually used by the texture description module 210. The sampler description module 211 is a filtering and packaging module for describing textures. The roles and cooperation of the material description module 209, texture description module 210, sampler description module 211, and texture resource description module 212 include the following: The material description module 209, together with the texture description module 210, defines the color and physical information of the object surface. The sampler description module 211 defines how to apply a texture resource to the object surface.The texture description module 210 specifies the sampler description module 212 and the texture resource description module 212, which realizes adding textures through the texture resource description module 212, which uses URIs for identification and indexing, and uses the accessor description module 204 for data access. The sampler description module 211 realizes the specific adjustment and packaging of textures. The definitions of the syntax elements in the texture description module 210 are shown in Table 6 below.

[0044] [Table 6]

[0045] In some embodiments, the definitions of syntax elements in the sampler (texture.sample) of the texture description module 210 are shown in Table 7 below.

[0046] [Table 7]

[0047] Illustratively, the following are JSON examples of one material description module 209, texture description module 210, sampler description module 211, and texture resource description module 212: [Table C]

[0048] 2 is a description module in the next hierarchical level after the node description module 202, and is used to describe animation information to be added to an object described by the node description module 202. Since animation can be added to the object described by the node description module 202 so that the object represented by the node description module 202 is not limited to a static state, the description hierarchy of the animation description module 213 in the scene description file is specified by the node description module 202; that is, the animation description module 213 is a description module in the next hierarchical level after the node description module 202, and similarly has a correspondence relationship with the mesh description module 203. The animation description module 213 describes animation in three ways: position movement, angle rotation, and size scaling, and can specify the start and end times of the animation and the implementation method of the animation. For example, when an animation is added to a mesh description module 203 representing a three-dimensional object, the three-dimensional object represented by the mesh description module 203 can complete a predetermined animation process by combining position movement, angle rotation, and size scaling within a specified time window.

[0049] 2, the skin description module (skin) 214 is a description module at the next level after the node description module 202 and is used to describe the motion relationship between the skeleton added by the mesh description module 203 and the mesh representing the object surface information. When the mesh description module 203 designated by the node description module 202 represents an object with a large degree of freedom of motion, such as a person, animal, or machine, the skeleton can be filled into the object to optimize the motion representation effect of the object, and the mesh description module 203 representing the object surface information then conceptually becomes the skin. The description level called the skin description module 214 is specified by the node description module 202; that is, the skin description module 214 is a description module at the next level after the node description module 202, and there is a correspondence between the skin description module 214 and the mesh description module 203. By moving the mesh on the surface of an object in conjunction with the movement of the skeleton and combining it with a bionic simulation design, it is possible to achieve a relatively realistic movement effect. For example, when a person's hand makes a fist, the skin on the surface stretches along with the internal skeleton, causing changes such as occlusion. At this time, by further defining the cooperative relationship between the skeleton and the skin for the skeleton pre-filled in the hand model, a realistic simulation of this movement can be achieved.

[0050] Each description module in the scene description file in the glTF 2.0 scene description standard only has the most basic capabilities for describing 3D objects, and has problems such as being unable to support dynamic 3D immersive media, audio files, or scene updates. glTF also states that each object attribute has one selectable extension object attribute, allowing any part of the object attribute to be extended using extensions to achieve more complete functionality. Modules such as the scene description module (scene), node description module (node), mesh description module (mesh), accessor description module (accessor), buffer description module (buffer), and animation description module (animation), as well as their internally defined syntax elements, all have selectable extension object attributes to support certain functional extensions based on glTF 2.0.

[0051] Currently, mainstream immersive media is mainly point clouds, 3D meshes, 6 DoF panoramic video, and MIV. In 3D scenes, multiple types of immersive media often coexist. This requires rendering engines to support multiple immersive media codecs. Different rendering engines are created depending on the type and number of supported codecs. Rendering engines designed by different vendors support different media types. To address the cross-platform description of 3D scenes composed of different media types, the Moving Picture Experts Group (MPEG) initiated the creation of the MPEG Scene Description Standard, designated ISO / IEC 23090-14. This standard primarily addresses the cross-platform description problem of 3D scenes in MPEG media (including codecs developed by MPEG, MPEG file formats, and MPEG transmission mechanisms).

[0052] The MPEG#128th meeting resolution is to create an MPEG-I Scene Description standard based on glTF2.0 (ISO / IEC12113). The first version of the MPEG Scene Description standard is currently in the FDIS balloting stage. Based on the first version standard, the MPEG Scene Description standard adds corresponding extensions to address requirements not yet realized in cross-platform 3D scene description, such as interactivity, AR anchors, user and avatar representations, haptic support, and extended support for immersive media codecs.

[0053] The first edition of the MPEG scene description standard created mainly creates the following contents:

[0054] 1) The MPEG Scene Description Standard defines a scene description file format for describing immersive 3D scenes. This format combines the content of the original glTF2.0 (ISO / IEC12113) and makes a series of extensions based on it.

[0055] 2) MPEG scene description defines a scene description framework and an application program interface (API) for inter-module communication within it, enabling decoupling of the immersive media acquisition and processing process from the media rendering process, and is beneficial for optimizing aspects such as adapting to different network conditions for immersive media, acquiring partial immersive media files, accessing different levels of immersive media detail, and adjusting content quality. Decoupling between the immersive media acquisition and processing process and the immersive media rendering process is the key to realizing cross-platform 3D scene description.

[0056] 3) For MPEG scene description, a series of extensions based on the International Standardization Organization Base Media File Format (ISOBMFF) (ISO / IEC 14496-12) have been proposed for the transmission of immersive media content.

[0057] As shown in Figure 3, based on the scene description file shown in Figure 2, the scene description file in the MPEG scene description standard is extended, and compared with the scene description file in the glTF2.0 scene description standard (the scene description file shown in Figure 2), the extensions of the scene description file in the MPEG scene description standard can be divided into two groups.

[0058] The first group of extensions includes MPEG media (MPEG_media) 301, MPEG time-varying accessor (MPEG_accessor_timed) 302, and MPEG ring buffer (MPEG_buffer_circular) 303. Here, MPEG media 301 is an independent extension for referencing external media sources, MPEG time-varying accessor 302 is an extension of the accessor hierarchy for accessing time-varying media, and MPEG ring buffer is an extension of the buffer hierarchy for supporting circular buffers. The first group of extensions provides basic descriptions and formats for media in a scene, meeting the basic needs of describing time-varying immersive media in a scene description framework. Among them, MPEG time-varying accessor (MPEG_accessor_timed) 302 is for accessing time-varying media. Because the glTF 2.0 scene description standard does not support time-varying media, if media data needs to change over time, it must be achieved by updating the scene description file in the glTF 2.0 scene description standard. For example, in the glTF2.0 scene description standard, the texture map of an object surface can change over time, so if the texture map of the object surface needs to be updated, the scene description file in the glTF2.0 scene description standard must be updated. Frequent updates to the scene description file require frequent parsing, processing, and transmission of the scene description file, which increases performance overhead in the 3D scene rendering process. Based on this, MPEG designed an MPEG time-varying accessor (MPEG_accessor_timed) 302, which allows parameters in the MPEG time-varying accessor to change over time, changing the access method for media data and enabling the accessed data to change over time, thereby avoiding the frequent parsing, processing, and transmission of the scene description file.

[0059] The second group of extensions includes MPEG dynamic scene (MPEG_scene_dynamic) 304, MPEG texture (MPEG_texture_video) 305, MPEG audio spatial (MPEG_audio_spatial) 306, MPEG view recommendation (MPEG_viewport_recommended) 307, MPEG mesh mapping (MPEG_mesh_linking) 308, and MPEG animation timing (MPEG_animation_timing) 309. Here, MPEG_scene_dynamic 304 is a scene hierarchy extension to support dynamic scene updates, MPEG_texture_video 305 is a texture hierarchy extension to support video-style textures, MPEG_audio_spatial 306 is a node hierarchy and camera hierarchy extension to support spatial 3D audio, MPEG_viewport_recommended 307 is a scene hierarchy extension to support describing recommended viewing angles for 2D displays, MPEG_mesh_linking 308 is a mesh hierarchy extension to support linking two meshes to provide mapping information, and MPEG_animation_timing 309 is a scene hierarchy extension to support controlling animation timelines.

[0060] Each of the above extensions will be described in detail below.

[0061] The MPEG media in the MPEG scene description file is used to describe the type of media file, and by providing the necessary explanations for the MPEG type media file, these MPEG type media files can be subsequently obtained. Here, the definitions of the syntax elements of the first layer of the MPEG media description module (MPEG_media) are shown in Table 8 below.

[0062] [Table 8]

[0063] The definitions of the syntax elements in the media list (MPEG_media.media) of the MPEG media description module are shown in Table 9 below.

[0064] [Table 9]

[0065] The definitions of the syntax elements in the option version list (MPEG_media.alternatives) in the media list of the MPEG media description module are shown in Table 10 below.

[0066] [Table 10]

[0067] The definitions of the syntax elements in the Tracks array in the Option Version List of the Media List (MPEG_media.media.alternatives.tracks) of the MPEG media description module are shown in Table 11 below.

[0068] [Table 11]

[0069] ISO / IEC 23090-14 also defines a transmission format for the delivery of scene description files and data related to extensions to glTF 2.0, based on ISOBMFF (ISO / IEC 14496-12). To facilitate the delivery of scene description files to clients, ISO / IEC 23090-14 defines how glTF files and related data are packaged into ISOBMFF files as non-time-varying and time-varying data (e.g., as track samples). MPEG_scene_dynamic, MPEG_mesh_linking, and MPEG_animation_timing provide time-varying data in specific formats to the display engine, which should perform corresponding operations based on this changed information. ISO / IEC 23090-14 also defines the format of each extended time-varying data and how it is packaged into ISOBMFF files. MPEG G MPEG_media can reference external media streams delivered over protocols such as RTP / SRTP or MPEG-DASH. To enable addressing media streams without knowing the actual protocol, hostname, or port values, ISO / IEC 23090-14 defines a new URL scheme. This scheme requires that a single stream identifier be present in the query part, but does not specify a specific type of identifier; it allows the use of a Media Stream Identification scheme (RFC5888), a labeling scheme (RFC4575), or a zero-based indexing scheme.

[0070] To illustrate, the following is a JSON example of an MPEG media description module: [Table D]

[0071] Haptics provides users with additional entertainment and sensory immersion, and accurately adding haptics to audio / video content can significantly enhance the experience and enjoyment that media content provides to users. The display of haptic data has a standard unified data model, and standard decoding of haptic data can be achieved when the data model contains all the information required for a specified synthesizer to present haptic effects.

[0072] Referring to FIG. 4, FIG. 4 is a structural diagram of a haptic data model. As shown in FIG. 4, the haptic data model includes a description section 41 and an encoding section 42, and employs a hierarchical structure to describe haptic data layer by layer. Here, the description section 41 mainly describes sensor and haptic channel information in the data structure, while the encoding section 42 mainly describes haptic bands, haptic effects, and keyframes in the data structure. The top layer of the description section 41 includes an overall description of the content contained in the entire data structure and is typically used to describe the entire haptic experience defined in a file or media stream. The haptic experience includes some high-level metadata and provides a predefined avatar list (i.e., a representation of a body part) to facilitate the assignment of subsequent haptic stimulation to a realization location on the body. The haptic data is described by a sensor list (sensor 1-sensor a). Each haptic sensor in the haptic sensor list corresponds to a haptic signal associated with a specific sensing mode (e.g., vibration, force, position, velocity, temperature, etc.). Each sense in the sense list includes metadata information, such as several characteristics associated with the haptic device and an avatar associated with the sense. In addition to the metadata, the sense signal itself is composed of a mixture of multiple haptic channels (haptic channel 1-haptic channel b). Each haptic channel includes not only metadata on the channel configuration (e.g., mixing gain, etc.), but also a haptic band list (haptic band 1-haptic band c) that breaks down the data into bands, with the band data for each haptic band providing a portion of the signal within a frequency range. A haptic band is described using band data and a haptic effect list (haptic effect 1-haptic effect d), and each haptic effect in the haptic effect list includes one keyframe list (keyframe 1-keyframe e). Different bands contain different signals, and the haptic signal in a channel can be reconstructed by combining the data from the different bands.

[0073] In some embodiments, to achieve channel-level haptic effects for specific body parts, a hierarchical structure may be adopted to describe body parts layer by layer. As shown in Figure 5, the hierarchical structure of the human body has the following structure: a first layer 51 includes the entire human body; a second layer 52 includes the upper and lower body; a third layer 53 includes the arms, chest, and head of the upper body, and the legs and waist of the lower body; a fourth layer 54 includes the upper arms, forearms, hands, neck and lower neck of the head, and the thighs, lower legs, and feet of the legs; a fifth layer 55 includes the palms and fingers of the hands, and the soles and toes of the feet; a sixth layer 56 includes the thumb, index finger, middle finger, ring finger, and little finger of the fingers, and the hallux, big toe, middle toe, ring toe, and little toe of the feet; and a seventh layer 57 includes the proximal phalanges of the thumb, the second knuckles, and the third knuckles of the knuckles.

[0074] The overall sensing modes for tactile sensing in the tactile data model are shown in Table 12 below.

[0075] [Table 12]

[0076] Digital encoding is crucial for enabling the operation of digital haptic devices. As a primary media type similar to audio and video, haptic media files also store haptic data in a digital format. Currently, haptic media files lack a unified encoding format. The lack of uniform encoding formats developed and used by various technology manufacturers causes incompatibility issues in the decoder-side formats of haptic media files, hindering the widespread adoption of haptics in the market. Therefore, the ISO / IEC Moving Picture Experts Group (MPEG) Working Group 7 emphasizes the need for a standardized haptic coding representation by developing ISO / IEC 23090-31 Haptic Standardization Behavior. A standardized haptic coding format (and associated standardized decoder) will contribute to the incorporation of haptics into the ISOBMFF, MPEG-DASH, and MPEG-I standards, making it easier for content creators and media / streaming content providers to integrate haptics in a standardized manner and improve the overall user experience.

[0077] The ISO / IEC Moving Picture Experts Group (MPEG) Working Group 7 developed a proposed codec architecture in the ISO / IEC 23090-31 haptic codec standard, which can process Pulse Code Modulation (PCM) files (WAV files / .wav files) or descriptive haptic files (AHAP files / .ahap files, IVS files / .ivs files, or HJIF files / .hjif files). Metadata information is provided to the codec in the form of .ohm files. Haptic coding allows for the encoding of descriptive and quantized data in a human-readable JSON format, and the encoding process is shown in Figure 6.

[0078] Referring to FIG. 6, FIG. 6 is a schematic diagram of an encoding process for encoding a haptic signal. As shown in FIG. 6, the haptic signal encoder 600 processes PCM haptic files and descriptive haptic files in different manners. For a descriptive haptic file (.ivs file 61, .ahap file 62, or .hjif file 63), the haptic signal encoder 600 first identifies the file format of the haptic file. If the haptic file to be encoded is an .hjif file 63, the .hjif file 63 and a metafile are directly packed into an MIHS file 64 by a binary compression step S601 and a packing step S602 without transcoding. Here, the metafile is a file obtained by performing metafile extraction on an .ohm file 65 by a metafile extraction step S603. If the haptic file to be encoded is an .ahap file 62 or an .ivs file 61, transcoding is required. The haptic signal encoder 600 first semantically analyzes the haptic file to be encoded in a semantic analysis step S604, and then transcodes and formats the file into the data model shown in Fig. 4 in a transcoding step S605 and a formatting step S606. After transcoding, the data may be derived as an interchange file format (.hjif) 66 encoded in a JSON-readable format, a distribution format (.hmpg) 67 encoded in a compressed binary file, or a stream format (MIHS) 64 defining a packetized bitstream. For a PCM file (.wav) 68, the signal analysis process of the haptic signal encoder 600 is divided into two sub-processes. The first sub-process performs a band decomposition step S607 on the original signal and then encodes the low-frequency signal obtained by the band decomposition based on a keyframe extraction step S608. The second method reconstructs the low-frequency signal, calculates the residual between the reconstructed low-frequency signal and the original low-frequency signal to obtain a residual signal, adds the residual signal to the original high-frequency signal, performs encoding using a wavelet transform step S609, and outputs the encoding result in at least one format of an .hjif file 66, an .hmpg file 67, or an MIHS file 64.

[0079] Referring to FIG. 7, FIG. 7 is a schematic diagram of a decoding process for decoding a haptic signal. As shown in FIG. 7, a haptic signal decoder 700 may decompress and decompress a packetized stream-formatted input (MIHS file 71) through a packetization step S701 and a binary decompression step S702 to obtain an .hjif file 72, or may select an .hmpg file 73 as input and decompress it through a binary decompression step S702 to output the .hjif file 72. The haptic data contained in the hjif file 72 may then be presented directly on a haptic device, or a .wav file 74 may be output and rendered on a haptic device through a synthesis step S703 using an intermediate synthesizer that generates PCM data. Note that the .hjif format is a human-readable format based on JSON and is not optimized for memory usage. This makes it easy to parse and manually edit, convenient for content design and creation, and an ideal exchange format. For distribution, the .hjif data can be compressed into a binary .hmpg bitstream, which has higher memory efficiency. This compression is lossy, and different parameters affect the coding depth of the amplitude and frequency that make up the bitstream. For stream transmission, the data can be compressed and packaged into an MPEG-I haptic stream (MIHS). This method meets the expectations of related technologies for the descriptive and quantization format compatibility of haptic media files, as well as interoperability between haptic terminal devices, enabling 3D immersive experiences and distribution purposes. The MIHS haptic stream is also the basis for packaging haptic media files into ISOBMFF.

[0080] In some embodiments, as shown in FIG. 8, a proposal to add MPEG_haptic 801 and MPEG_material_haptic 802 to an MPEG scene description file (the scene description file shown in FIG. 3) was passed in the MPEG #140 conference to associate haptic media files with objects in a scene and make haptics part of an immersive media experience. Based on the new version of the haptic extension of glTF, haptic support can be enabled in the scene description. Here, MPEG_haptic 801 can obtain accessor description information for accessing the rendering data of haptic media files stored in a buffer by indexing an accessor description module in the accessory list of the scene description file. MPEG_material_haptic 802 is a description module next in level to the mesh description module and supports users in obtaining corresponding haptic feedback through active and / or passive interactions in a virtual scene.

[0081] Texture mapping is a technique for efficiently and realistically displaying details in a three-dimensional model without requiring high-resolution meshes. This method was originally developed in computer graphics and is now widely applied in haptic rendering. The existence of texture mapping allows virtual objects to be enhanced with additional haptic material attributes distributed on their surfaces, enhancing the realism of the virtual objects. Based on texture mapping, related technologies also provide the concept of haptic materials. In computer graphics, a material is an easy-to-use data packet that contains all the data required for the visualization and rendering of a virtual object. Relatively speaking, haptic materials should provide all the elements required for haptic rendering, and different haptic materials contain different haptic features. These different haptic material attributes are stored in different haptic material packets, which is convenient for the related tasks of haptic design and realization. The definitions of haptic material attributes are shown in Table 13 below.

[0082] [Table 13]

[0083] Each haptic material attribute in Table 13 above is represented by a texture array, and texture arrays of different attributes are combined to form a specific haptic sensation. A haptic material attribute may provide an associated haptic texture value as a traditional two-dimensional texture, or it may be a haptic element mapping. Each pixel point in the haptic element mapping includes multiple haptic unit elements associated with ordinal indices corresponding to haptic objects in MPEG_haptic. The rendering engine can select and render the optimal texture array. Thus, each array element includes one haptic texture and a texture type represented as an enumeration. The enumerated haptic material texture types include:

[0084] High_resolution: Haptic textures are high-resolution two-dimensional textures that directly store tactile values.

[0085] Low_resolution: A haptic texture is a low-resolution two-dimensional texture that directly stores haptic values.

[0086] Reference: A haptic texture is a 2D haptic element mapping that contains a haptic signal reference. Each pixel in the haptic texture corresponds to an accessor order index within a haptic Object in MPEG_haptic.

[0087] Other: Unknown proprietary texture format.

[0088] In order to interpret the data contained in 2D textures, the bit depth and range of these textures must be specified. Tables 14 and 15 below show the bit depth and range of each haptic material attribute for low-resolution and high-resolution haptic textures, respectively.

[0089] [Table 14]

[0090] [Table 15]

[0091] As shown in Table 15 above, the value of each texture map pixel in the high resolution vibration texture is split into a first byte for describing the amplitude value and a second byte for describing the frequency value.

[0092] The definitions of syntax elements in MPEG Haptic Material (MPEG_material_haptic) 802 are shown in Table 16 below.

[0093] [Table 16]

[0094] Table 16 above shows the meaning of the syntax elements included in MPEG Haptic Material (MPEG_material_haptic) 802, and the enumerated values ​​in Table 16 are the haptic material attributes included in Table 13. MPEG Haptic Material (MPEG_material_haptic) 802 in an MPEG scene description file can associate haptic materials with objects in a scene, and can provide corresponding haptic sensations when the user interacts with the objects.

[0095] Although each haptic material attribute contains different haptic characteristics, the internal storage method is consistent for all of them. Table 17 below further explains the stiffness (MPEG_material_haptic.stiffness) listed in the MPEG haptic material, using stiffness as an example in the haptic material attribute.

[0096] [Table 17]

[0097] 2. Display engine

[0098] As shown in FIG. 1, in the workflow of the scene description framework for immersive media, the display engine 11 mainly includes acquiring a scene description file, analyzing the acquired scene description file to acquire the structural structure of the 3D scene to be rendered and detailed information about the 3D scene to be rendered, and rendering and displaying the 3D scene to be rendered based on the information acquired by analyzing the scene description file. Original Vow In this embodiment, the specific workflow and principles of the display engine 11 are not limited, but are based on the display engine 11 analyzing the scene description document, sending commands to the media access function 12 via the media access function API, sending commands to the buffer management module 13 via the buffer API, and obtaining the processed data from the buffer to complete the rendering and display of the 3D scene and the objects therein.

[0099] 3. Media Access Function

[0100] In the workflow of the immersive media scene description framework, the media access function 12 can receive commands from the display engine 11 and complete functions of accessing and processing media files based on the commands sent by the display engine 11. Specifically, this includes obtaining media files and then processing the media files. There are significant differences in the processing processes for different types of media files. In order to support a wide range of media types and also take into account the work efficiency of the media access function, various pipelines are designed in the media access function, and only the pipeline that matches the media type needs to be enabled in the processing process.

[0101] The input of the pipeline is media files downloaded from a server or read from a local storage control. These media files generally have complex structures and cannot be directly used by the display engine 11, so the main function of the pipeline is to process the data of such media files and make the data of the media files conform to the requirements of the display engine 11.

[0102] In the workflow of the immersive media scene description framework, media data that has completed pipeline processing must be passed to the display engine 11 in a standard array structure for use, which requires the participation of a buffer API and a buffer management module 13. The buffer API and buffer management create corresponding buffers based on the format of the processed media data and are responsible for subsequent management of the buffers, such as operations to update and release. The buffer management module 13 may communicate with the media access function 12 or the display engine 11 via the buffer API, and the goal of communication with the display engine 11 and / or the media access function 12 is to achieve buffer management. When the buffer management module 13 communicates with the media access function 12, the display engine 11 must first send buffer management-related commands to the media access function 12 via the media access function API, and the media access function 12 must then send buffer management-related commands to the buffer management module 13 via the buffer API. When the buffer management module 13 communicates with the display engine 11, the display engine 11 may directly transmit the buffer management description information parsed from the scene description document to the buffer management module 13 via the buffer API.

[0103] As shown in Figure 8, the current MPEG #140 meeting has confirmed the proposal to add MPEG Haptic (MPEG_haptic) 801 and MPEG Haptic Material (MPEG_material_haptic) 802, and allows MPEG Haptic 801 to be indexed into the accessor description module in the accessory list of the scene description file to obtain the haptic rendering data of the haptic media file. The haptic rendering data of the haptic media file often describes the overall haptic experience associated with an event or environment. The haptic rendering data of the indexed haptic media file can be considered as a haptic signal, which can essentially directly drive a haptic device to generate corresponding haptic feedback for the human body.

[0104] As mentioned above, in the related art, each array element includes one haptic texture and one haptic material attribute type represented as an enumeration, and the haptic material attribute types include high resolution (High_resolution), low resolution (Low_resolution), reference (Reference), and other (Other). When the haptic material attribute type is high resolution or low resolution, it is the haptic value, not the color value, that is stored in the texture image or haptic texture material packet of the texture index. The haptic value only represents the magnitude of a specific attribute (e.g., the magnitude of the friction coefficient, the magnitude of the stiffness coefficient), and can be directly obtained by the display engine for haptic rendering. However, when the haptic material attribute type is reference, each pixel in the texture image or haptic texture material data packet does not directly store the haptic value, but each pixel in the texture image or haptic texture material data packet points to a haptic media accessor. The haptic value of a haptic material attribute whose type is reference needs to be obtained from a haptic media file. Therefore, how to support haptic media files in a scene description framework and how to store haptic media files whose type is reference is discussed. is Supporting tactile material attributes has been one of the focuses of research in this field.

[0105] Original Vow Some embodiments of the present invention provide a scene description framework that supports haptic media files, including specific content such as supporting haptic media files with a scene description file, supporting haptic media files with a media access function API, supporting haptic media files with media access functions, supporting haptic media files with a buffer API, and supporting haptic media files with buffer management.

[0106] The process of indexing haptic media files in a 3D scene based on the scene description framework and rendering and displaying the 3D scene based on the haptic media files includes: first, a display engine acquires a scene description file provided by a 3D scene provider by a method such as network protocol download or local storage read. Here, the scene description file acquired by the display engine includes description information for the entire 3D scene and the haptic media files in the 3D scene, and the description information for the haptic media files includes the access address of the haptic media file, attribute information of the haptic media file (media type, used codec parameters, etc.), etc. At the same time, the process further includes a format request for haptic rendering data obtained by decoding the haptic media file, and the haptic media file is displayed. for a fileThe format request for the haptic rendering data obtained by the decoding process further includes format requests (such as data type, data length, and data access method) for information data such as the description portion and encoding portion of the haptic data obtained by processing the haptic media file (i.e., description portion 41 and encoding portion 42 in FIG. 4, hereinafter collectively referred to as the haptic media file), as well as format requests and addresses for other media data, including data such as texture resources to be used later and description information of haptic material attributes. After completing the analysis of the scene description file, the display engine calls the media access function API to transmit information such as the access address of the haptic media file and attribute information of the haptic media file (such as media type and codec used) to the media access function. At the same time, the display engine also transmits format requests for the decoded haptic media data and other media data specified in the analyzed scene description file to the media access function by calling the media access function API. Then, the media access function requests the server to download the specified haptic media file based on the access address of the haptic media file transmitted from the display engine, or obtains the specified haptic media file from a local location. Then, the media access function builds a corresponding pipeline for the media file based on information such as the media type and codec parameters, and converts the obtained media data from the package format into the format required by the scene description file, i.e., converts the haptic media file into the format (data type, data length, access method, etc.) specified in the scene description file in the corresponding pipeline. In the pipeline, operations such as decapsulation, decryption, decoding, and post-processing can be performed on the haptic media file. The pipeline then stores the haptic rendering data and other media data of the haptic media file, which have been processed into the predetermined format of the scene description file, in a specified buffer.The display engine can create a buffer by calling the buffer management module through the buffer API, and if the display engine does not create the corresponding buffer, the media access function can create the corresponding buffer by calling the buffer management module through the buffer API. The purpose of both is to create, update, and release a buffer. Finally, the display engine reads the haptic rendering data of the haptic media file in the specified buffer in the format specified in the scene description file to complete the rendering of the haptic part in the 3D scene.

[0107] The following describes the scene description file, media access function API, media access functions, buffer API, and buffer management that support haptic media files.

[0108] 1. Scene description files supporting haptic media files

[0109] As shown in Figure 9, Original Vow9 is a schematic diagram of the structure of a scene description file supporting a haptic media file according to an embodiment of the present invention. As shown in Fig. 9, the scene description file supporting a haptic media file includes an MPEG media description module (MPEG_media) 901, an MPEG interactivity description module (MPEG_interactivity) 902, an MPEG haptic description module (MPEG_haptic) 903, a scene description module (scene) 904, a node description module (node) 905, a mesh description module (mesh) 906, an accessor description module (accessor) 907, a buffer slice description module (bufferView) 908, a buffer description module (buffer) 909, a camera description module (camera) 910, a light illumination description module (light) 911, an MPEG haptic material (MPEG_material_haptic) 912, a material description module (material) 913, a texture description module (texture) 914, a sampler description module (sampler) 915, a texture resource description module (TMP) 916, a texture description module (TMP) 917, a texture resource description module (TMP) 918, a texture resource description module (TMP) 919, a texture resource description module (TMP) 920, a texture resource description module (TMP) 921, a texture resource description module (TMP) 922, a texture resource description module (TMP) 923, a texture resource description module (TMP) 924, a texture resource description module (TMP) 925, a texture resource description module (TMP) 926, a texture resource description module (TMP) 927, a texture resource description module (TMP) 928, a texture resource description module (TMP) 929, a texture resource description module (TMP) 930, a texture resource description module (TMP) 931, a texture resource description module (TMP) 932, a texture resource Description Module These include, but are not limited to, an image description module (image) 916, an animation description module (animation) 917, and a skin description module (skin) 918. Each part of the scene description module shown in FIG.

[0110] So that the scene description file can accurately describe the relevant information of the haptic media file in the 3D scene, Original Vow Some embodiments of the present invention extend the values ​​of syntax elements in the MPEG media description module (MPEG_media) 901 of the scene description file, and the specific extensions include at least one of the following:

[0111] Extension 1 extends the value of the media type syntax element (mime Type) to declare the media resource type of a media file in the option version list (alternatives) of the media list (media) of the MPEG media description module (MPEG_media) of the scene description file. The extension of the media type syntax element (mime Type) includes adding a value associated with a haptic media file to the media type syntax element (mime Type). For example, if the media list of the MPEG media description module of a scene description file includes haptic media and the value of the media type syntax element (mime type) for declaring the media resource type of the media file complies with the relevant provisions of the ISO / IEC 23090-32 standard, when the media file is a media file containing audio, video, and haptic data, the value of the media type syntax element (mime type) is "video / mp4", when the media file is a media file containing only audio and haptic data, the value of the media type syntax element (mime type) is "video / mp4", and when the media file is a media file containing only haptic data, the value of the media type syntax element (mime type) is "haptics / mp4".

[0112] Extension 2 extends the value of the first track index syntax element (tracks) to declare track information of a media file in the track array (tracks) of the optional version list (alternatives) of the media list (media) of the MPEG media description module (MPEG_media) of a scene description file. The extension of the first track index syntax element (track) includes following the reference track selection method in the ISO / IEC 23090-32 standard when an encoded haptic media file is referenced by a scene description file as one of the arrays in the optional version list of the media list of the MPEG media description module. For example, if haptic media is packaged into one MIHS track by ISOBMFF, this MIHS track should be referenced in the MPEG media description module; if haptic media is packaged into multiple MIHS tracks by ISOBMFF, subsequent supplementary content of ISO / IEC 23090-32 must be referenced.

[0113] Extension 3 extends the value of the codec parameter syntax element (codecs) to declare the codec type of the media data included in the bitstream track in the track array (tracks) of the option version list (alternatives) of the media list (media) of the MPEG media description module (MPEG_media) of the scene description file. IETF RFC 6381 defines the codec parameters of the media files included in the bitstream track. If the codestream track contains different types of codecs (e.g., if an AdaptationSet contains representations with different codecs), the codec parameter syntax element (codecs) can be expressed as a comma-separated list of codec values. That is, extending the value of the codec parameter syntax element (codecs) includes specifying that if the media data is haptic data and the packaging complies with the ISO / IEC 23090-32 standard, the value of the codec parameter syntax element (codecs) should be set in accordance with the provisions of the ISO / IEC 23090-32 Carriage of Haptics Data standard. For example, if haptic data is packaged in ISMBFF, the first group of elements in the codec parameter syntax element (codecs) is a code representing a codec in ISO / IEC 23090-31, and the parameters and their format of the codec parameter syntax element (codecs) are "codecs":"mih1.oo", where "mih1" is a description of the data format in which the haptic data is packaged in ISOBMFF. This value is not unique and can be replaced based on the name of a sample format that matches other ISOBMFF packages. Here, "oo" is an object type indicator defined on the "Object Types" page of the MP4 Registration Authority website.

[0114] That is, to ensure that the scene description file accurately describes the haptic media file, Original VowSome embodiments of extend the values ​​of syntax elements in the MPEG media description module (MPEG_media) in the scene description file, and specifically include one or more of the extensions shown in Table 18 below.

[0115] [Table 18]

[0116] The haptic media file package is based on the data structure shown in Figure 4. The haptic media file is permitted to have not only an information section but also a metadata placement section. Original Vow The embodiment does not limit the packaging method of the haptic media file, but is based on the fact that the encoded and packaged haptic media file can be obtained based on the access address of the haptic media file in the MPEG media description module.

[0117] So that the scene description file can accurately describe haptic-related information in a 3D scene, Original Vow In some embodiments, the scene description file is independently extended with an MPEG Interactivity Description Module (MPEG_interactivity) 902. The MPEG Interactivity Description Module (MPEG_interactivity) 902 is used to describe interactive triggers, interactive actions, and the association between the interactive triggers and the interactive actions. Here, the definitions of the syntax elements of the first layer of the MPEG Interactivity Description Module (MPEG_interactivity) 902 are shown in Table 19 below.

[0118] [Table 19]

[0119] Note that Table 19 above only shows a portion of the content related to supporting haptic media, and the first layer of the MPEG Interactivity Description Module (MPEG_interactivity) 902 may further include other syntax elements, such as: Original Vow The embodiment is not limited thereto. Definitions of syntax elements in triggers of the MPEG_interactivity description module (MPEG_interactivity) 902 are shown in Table 20 below.

[0120] [Table 20]

[0121] The definitions of the syntax elements in the actions of the MPEG interactivity description module (MPEG_interactivity) 902 are shown in Table 21 below.

[0122] [Table 21]

[0123] The definitions of the syntax elements in the haptic action node list (haptic Action Nodes) in the action list (actions) of the MPEG interactivity description module (MPEG_interactivity) 902 are shown in Table 22 below.

[0124] [Table 22]

[0125] The definitions of the syntax elements in the behaviors of the MPEG_interactivity description module (MPEG_interactivity) 902 are shown in Table 23 below.

[0126] [Table 23]

[0127] To support haptic media files in the scene description framework, Original Vow In some embodiments, the scene description file is independently extended with an MPEG haptic description module (MPEG_haptic) 903. Compared with the MPEG haptic (MPEG_haptic) 801 in the scene description file shown in FIG. Original Vow In this embodiment, the MPEG haptic description module (MPEG_haptic) 903 is not a description module at the next level below the node description module (node), but is the top-level description module of the scene description file, and is not a media description module in the media list for indexing the MPEG media description module (MPEG_media) 901. It declares an accessor for accessing the haptic rendering data of the haptic media file in units of haptic objects. The top-level syntax elements in the MPEG haptic description module (MPEG_haptic) 903 are shown in Table 24 below.

[0128] [Table 24]

[0129] The syntax elements in the haptic object of the MPEG haptic description module (MPEG_haptic) 903 are shown in Table 25 below.

[0130] [Table 25]

[0131] In some embodiments, the description methods of the scene description module (scene) 904 and node description module (node) 905 in a scene description file supporting a haptic media file include describing the overall structure of the 3D scene and the structural hierarchy and position of the haptic media file in the 3D scene using the scene and node description methods when the haptic media file is included in the 3D scene. Each 3D scene is described using an independent scene description module. A 3D scene may contain many specific objects, such as a virtual digital human, nearby 3D objects, a distant background image, and a virtual camera representing the user's viewpoint and viewing angle. The scene description file treats these specific objects as nodes in the 3D scene, and each node can represent one object or a set of objects consisting of multiple objects. The relationships between the nodes reflect the relationships between the components in the 3D scene. Each scene description file can describe one or more 3D scenes, and the 3D scenes have only a parallel relationship, not a hierarchical relationship, i.e., there is no "contains" or "contained" relationship between the scenes. Nodes can have parallel or hierarchical relationships, i.e., "contains" and "is" relationships between nodes, allowing multiple concrete objects to be collectively represented. When one node is contained within another, the contained node is called a subnode, and the word "node" is replaced with "children" to refer to the subnode. Nodes and subnodes can be flexibly combined to form a hierarchical node structure.

[0132] In some embodiments, a description method of a mesh description module (mesh) 906 in a scene description file supporting a haptic media file includes describing a node representing a three-dimensional object using a three-dimensional mesh data format, i.e., using the mesh description module (mesh) 906. The mesh description module includes multiple specific pieces of information, such as three-dimensional coordinates and color, which all belong to syntax elements in the primitive attribute (mesh.primitives.attribute) of the mesh description module (mesh). The most basic mesh description module includes the three-dimensional coordinates of vertices, the connectivity between vertices, the color information of vertices, and the three-dimensional coordinates requiring texture values. At the mesh description module level, it is necessary to enable the texture coordinate (Texcoord) of the three-dimensional coordinate set of the desired texture value, and the haptic value obtained from the subsequent texture is added to the corresponding three-dimensional coordinate of the three-dimensional object. Information such as the texture coordinate (Texcoord) and color data are all attributes (mesh.primitives.attribute) of the primitive in the mesh description module. Haptic media temporarily supports further definition of haptic materials by simply enabling the MPEG haptic description module (MPEG_material_haptic) 912 and supplementing it with parallel content in the material description module (material) 913, without the need to reference and define attributes such as the number of colors at the mesh description module level.

[0133] In some embodiments, the description method of the material description module (material) 913 and the texture description module (texture) 914 in the scene description file supporting the haptic media file includes describing additional information of the surface of a three-dimensional object using the material description module (material) 913 and the texture description module (texture) 914. When describing a typical three-dimensional object, describing the geometric information of the three-dimensional object using only a mesh description module or simply defining the colors of the vertices in the three-dimensional mesh is far insufficient to improve the realism of the virtual object, and more information needs to be added to the surface of the three-dimensional object. In typical three-dimensional modeling techniques, this process may be abbreviated as mapping or texturing. That is, the scene description file supporting the haptic media file is based on glTF2.0 and also uses a material description module (material) 913 and a texture description module (texture) 914. The material description module (material) 913 and the texture description module (texture) 914 cooperate to define the color and physical information of the object surface. The texture description module (texture) 914 specifies a sampler description module (sampler) 915 and a texture resource description module (image) 916. The sampler description module (sampler) 915 defines how to map a texture image to the object surface and realizes specific adjustment and packaging of the texture, and the texture resource description module (image) 916 identifies and indexes the texture resource using a ULR.

[0134] In some embodiments, the description method of the MPEG haptic description module (MPEG_material_haptic) 912 in a scene description file supporting a haptic media file includes further describing haptic material attributes by the MPEG haptic description module (MPEG_material_haptic) 912. When a haptic media file is included in a three-dimensional scene, haptic material attributes need to be considered in the material properties. The MPEG haptic material (MPEG_material_haptic) 912 includes a haptic material attribute description module corresponding to each haptic material attribute (stiffness, friction, haptic texture, temperature, vibration, and custom), and the haptic material attribute description module includes an attribute type syntax element (type), whose value may be high resolution, low resolution, or reference. Objects in a three-dimensional scene that include haptic feedback may include different types of haptic material attributes. Different haptic material attributes on the same object can obtain haptic values ​​or haptic media that are felt by the user in different ways, thereby realizing the associated haptic media in the scene description file to obtain haptic feedback.

[0135] As can be seen from Table 17, the MPEG haptic description module (MPEG_material_haptic) 912 includes haptic material attribute description modules corresponding to multiple haptic material attributes (stiffness, friction, haptic texture, temperature, vibration, and custom), and the haptic material attribute description module corresponding to each haptic material attribute includes the type of the haptic material attribute, specifically, low_resolution, high_resolution, reference, and other. If the haptic material texture type of a haptic material attribute is low resolution or high resolution, it indicates that the haptic value at that texture coordinate point can be obtained from a texture image or another haptic material texture data packet. The value contained in each pixel point in the texture image or haptic material texture data packet represents the haptic value. As shown in Tables 15 and 16, the bit depth of the haptic value for different haptic material texture types is different. For example, the bit depth of the haptic value for a high-resolution (High_resolution) haptic material attribute is 16 bits, and the bit depth of the attribute value for a low-resolution (Low_resolution) haptic material attribute is 8 bits. If the haptic material attribute type is Reference, it is required that the haptic value for that texture coordinate point or a set of texture coordinate points can be indexed into a set of temporal or spatial haptic values. In this case, the MPEG Interactivity Description Module (MPEG_interactivity) 902 and the MPEG Haptic Description Module (MPEG_haptic) 903 provide related supplementary descriptions. Here, the MPEG Interactivity Description Module 902 and the MPEG Haptic Description Module 903 is enabled when a scene description file contains haptic media files, and the type of each haptic material attribute is included within the MPEG haptic description module (MPEG_material_haptic) 912.

[0136] In some embodiments, a scene description file supporting a haptic media file includes an accessor description module (accessor) 907, a buffer slice description module (bufferView) 908, and a buffer description module (buffer). 909 The description method of includes pointing to a media description module corresponding to a haptic media file in the MPEG media description module (MPEG_media) by a value of the media index syntax element (buffer.MPEG_buffer_circular.media) of the MPEG circular buffer in the buffer description module. That is, the haptic media file needs to be specified in the buffer description module (buffer), but instead of directly adding a uniform resource locator (URL) of the haptic media file to the buffer description module (buffer), the index value of the media description module corresponding to the corresponding haptic media file in the MPEG media description module (MPEG_media) is declared via the media index syntax element (MPEG_buffer_circular.media) of the MPEG ring buffer in the buffer description module (buffer).

[0137] For example, if the media description module corresponding to a haptic media file in an MPEG media description module (MPEG_media) is the third media description module in the media list of the MPEG media (MPEG_media), the value of the media index syntax element (buffer.MPEG_buffer_circular.media) in the MPEG ring buffer of the buffer description module can be set to “2”, thereby indexing the access address in the third media description module in the MPEG media description module in the MPEG ring buffer of the buffer description module, and indexing the haptic media file by the (MPEG_buffer_circular) media index syntax element in the MPEG ring buffer of the buffer description module.

[0138] In some embodiments, the description methods of the accessor description module (accessor) 907, buffer slice description module (bufferView) 908, and buffer description module (buffer) 908 in a scene description file supporting a haptic media file include indicating track information when accessing a package file of a haptic media file using a track (buffer.MPEG_buffer_circular.tracks.track) in the MPEG ring buffer track array of the buffer description module. Specifically, if the MPEG ring buffer track array (buffer.MPEG_buffer_circular.tracks) in the buffer description module includes multiple tracks, the pipeline performs the necessary processing on all referenced tracks to generate the data format required by the buffer description module. If a media file referenced in a track (buffer.MPEG_buffer_circular.tracks) in the MPEG ring buffer track array of the buffer description module has multiple optional versions (alternatives), each optional version (alternative) should record the track selection status. If the track arrangement of the MPEG circular buffer (buffer.MPEG_buffer_circular.tracks) in the buffer description module is not defined, the pipeline will perform the necessary processing for all tracks in the MPEG media description module (MPEG_media) to generate the data format required by the buffer description module (buffer), i.e., guide how the buffer description module (buffer) correctly stores the data in the MPEG media description module (MPEG_media).

[0139] The syntax elements contained in the MPEG circular buffer (MPEG_buffer_circular) are shown in Table 26.

[0140] [Table 26]

[0141] That is, a haptic media file is indexed in the buffer description module by setting the value of the media index syntax element (media) in Table 26 to the index value of the media description module corresponding to the corresponding haptic media file in the MPEG media description module (MPEG_media) based on the value setting rule for the media index syntax element (media) in Table 26. A track of a haptic media file is indexed in the buffer description module by setting the value of the syntax element "tracks" in Table 26 to the track index containing the haptic media file in the MPEG media description module based on the value setting rule for the syntax element "tracks" in Table 26.

[0142] In some embodiments, the description method of the camera description module (camera) 910 that supports the scene description file of the haptic media file includes defining visual information associated with viewing, such as the viewpoint, viewing angle, etc. of the node, by the camera description module (camera) 910.

[0143] In some embodiments, the description method of the animation description module (animation) 917 that supports the scene description file of the haptic media file includes defining animations to be added to nodes by the animation description module (animation) 917.

[0144] In some implementations, an animation description module (animation) 917 can describe animations to add to a node by one or more of position translation, angle rotation, and size scaling.

[0145] In some embodiments, the animation description module (animation) 917 may indicate at least one of the start time, end time, and animation implementation method of an animation added to a node.

[0146] That is, in a scene description file supporting a haptic media file, animation can be added to a node representing an object in a three-dimensional object. The animation description module (animation) 917 describes the animation added to the node in three ways: position movement, angle rotation, and size scaling, and may also specify the start and end times of the animation and the implementation method of the animation.

[0147] In some embodiments, a description method of a skin description module (skin) 918 that supports a scene description file of a haptic media file includes defining, by the skin description module (skin) 918, movement and deformation relationships between a 3D mesh mounted on a node and corresponding bones.

[0148] For illustrative purposes, the scene description file shown in FIG. 9 will be described below by combining an example of one scene description file. [Table E] TIFF2026501767000243.tif249167TIFF2026501767000244.tif249168TIFF2026501767000245.tif250167TIFF2026501767 000246.tif249168TIFF2026501767000247.tif250168TIFF2026501767000248.tif250168TIFF2026501767000249.tif94168

[0149] The main contents of the scene description file shown in Figure 9 are contained within a pair of square brackets on lines 1-320 in the above example. A scene description file that supports haptic media files includes a digital asset description module ("asset":{}), a usage extension list ("extensions Used":[]), an extension description module ("extensions":{}), a scene statement (scene), a scene list ("scenes":[]), a node list ("nodes":[]), a mesh list ("meshes":[]), a haptic material list ("MPEG_material_Happtic":[]), a texture list ("textures":[]), a texture resource list ("images":[]), a sampler list ("samplers":[]), an accessory list ("accessors":[]), a buffer slice list ("bufferViews":[]), and a buffer list ("buffers":[]). The following describes the information contained in each part of the content and the analysis angle.

[0150] 1. Digital asset description module ("asset":{}): The digital asset description module is lines 2-4. The "version":"2.0" on the third line of the digital asset description module indicates that the scene description file is based on the glTF2.0 version, which is also the reference version of the scene description standard. From an analysis perspective, the display engine can determine which parser to select to parse the scene description file based on the digital asset description module.

[0151] 2. Used Extension List ("extensions Used":[]): The used extension list is lines 6-13. The used extension list includes six extensions: MPEG media (MPEG_media), MPEG interactive (MPEG_interactivity), MPEG haptics (MPEG_Haptic), MPEG haptic material (MPEG_material_Haptic), MPEG ring buffer (MPEG_buffer_circular), and MPEG time-varying accessor (MPEG_accessor_timed). Therefore, it can be determined that six MPEG extensions are used in this scene description file: MPEG media (MPEG_media), MPEG interactive (MPEG_interactivity), MPEG haptics (MPEG_Haptic), MPEG haptic material (MPEG_material_Haptic), MPEG ring buffer (MPEG_buffer_circular), and MPEG time-varying accessor (MPEG_accessor_timed). From an analysis perspective, the display engine can know in advance, based on the content using the extension list (extension Used), that the extension items associated with subsequent analysis include MPEG media, MPEG interaction, MPEG haptics, MPEG haptic material, MPEG ring buffer, and MPEG time-varying accessor.

[0152] 3. Extension Description Module ("extensions":{}): The extension list is lines 15-94. The extension description module includes three extension description modules: MPEG Interactivity Description Module ("MPEG_interactivity":{}), MPEG Media Description Module ("MPEG_media":{}), and MPEG Haptic Description Module ("MPEG_haptic":{}).

[0153] 3.1, MPEG Interactivity Description Module ("MPEG_interactivity":{}): The MPEG Interactivity Description Module is in lines 16-60. The MPEG Interactivity Description Module contains three parts of content: a trigger list ("triggers":[]), an action list ("actions":[]), and a behavior list ("behaviors":[]). Here, the trigger list ("triggers":[]) is in lines 17-24. The trigger list contains two parallel square brackets. Two triggers are declared in the trigger list by two trigger description modules, and the "type":"TRIGGER_COLLISION" in lines 19 and 22 indicates that the type of the two triggers is a collision. The action list (" The action list contains two parallel square brackets, indicating that two actions are declared in the action list by two action description modules. The first action description module (the content enclosed in square brackets in lines 26-41) first indicates that the type of the action corresponding to the first action description module is a haptic action by "type":"ACTION_SET_HAPTIC" in line 27, and then adds the haptic action node list ("haptic Action The haptic actions are indicated by three haptic action node arrays (the bracketed content on lines 29-32, the bracketed content on lines 33-36, and the bracketed content on lines 37-39) in the MPEG haptic description module ("MPEG_haptic":{}), and the nodes that need to enable the haptic actions include nodes corresponding to node description modules with index values ​​of 0, 1, and 2 (the first, second, and third node description modules in the node list). The node corresponding to the node description module with index value 0 needs to enable the haptic media file and access the haptic rendering data of the haptic media file based on the first haptic object description module ("hapticObject":[]) in the MPEG haptic description module ("MPEG_haptic":{}).The node corresponding to the node description module with an index value of 1 also needs to enable a haptic media file and access the haptic rendering data of the haptic media file based on the second haptic object description module ("hapticObject":[]) in the MPEG haptic description module ("MPEG_haptic":{}), while the node corresponding to the node description module with an index value of 3 also needs to enable a haptic media file, but there is no index statement for the haptic object description module, and the node corresponding to the node description module with an index value of 3 states that it will access the haptic rendering data of the haptic media file through the MPEG haptic material (MPEG_material_Haptic) in the corresponding mesh description module. The content stated by the second action description module (the content enclosed in square brackets on lines 42-50) is similar to the content stated by the first action array, and to avoid duplication, detailed description will be omitted. The behavior list ("behaviors":[]) is on lines 52-59. The behavior list contains two parallel square brackets, indicating that two behaviors are stated in the behavior list by two behavior arrays. The first behavior array (the content enclosed in square brackets on lines 53-55) realizes the correlation between the first trigger description module in the trigger list ("triggers":[]) and the first action description module and the second behavior description module in the action list ("actions":[]) by "triggers":0 on line 53 and "actions":0, l on line 54. The second action description module realizes the correlation between the second trigger description module in the trigger list ("triggers":[]) and the second action description module in the action list ("actions":[]) by "triggers":1 on line 56 and "actions":1 on line 57.

[0154] 3.2, MPEG Media Description Module ("MPEG_media":{}): The MPEG media description module is in lines 61-81. The MPEG media description module realizes statements about the media files included in a 3D scene. The media list ("media":[]) in the MPEG media description module contains only one bracket (one media description module), indicating that only one media file is described in the MPEG media of the scene description file. In the media description module, "mimeType":"video / mp4" on line 69 indicates that the file type of the package file corresponding to the media file is MP4, "uri":"http: / / www.example.com / Hapticexample.mp4" on line 70 indicates the access address of the corresponding media file, "track":"track Index=1" on line 73 indicates the track information of the corresponding media file, "codecs":"mih1.oo" on line 74 indicates the decoding information of the corresponding media file, "name":"Hapticexample" on line 64 indicates the name of the corresponding media file, "autoplay":true on line 65 indicates that the corresponding media file should be auto-played, and "loop":true on line 66 indicates that the corresponding media file should be played cyclically. From the analysis perspective, the display engine can determine that one haptic media file exists in the 3D scene to be rendered by analyzing the MPEG media, and know how to access and parse this haptic media file.

[0155] 3.3. MPEG Haptic Description Module ("MPEG_haptic":{}): MPEG Haptic contains two haptic object description modules ("hapticObject":[]). The first haptic object description module declares the accessors corresponding to the second and fifth accessor description modules in the accessory list via "accessor":1,4 on line 85. By referring to the contents of the haptic action node list of the first action description module in the action list ("actions":[]), it can be seen that the node corresponding to the node description module with index value 0 needs to access the haptic rendering data of the haptic media file based on the accessors corresponding to the second and fifth accessor description modules in the accessory list. The second haptic object array declares the accessors corresponding to the third and fourth accessor description modules in the accessory list via "accessor":2,3 on line 90, and by referring to the contents of the haptic action node list in the second action array of the action list ("actions":[]), it can be seen that the node corresponding to the node description module with index value 1 needs to access the haptic rendering data of the haptic media file via the accessors corresponding to the third and fourth accessor description modules in the accessory list.

[0156] 4. Scene Statement (scene): The scene statement is on line 96. Theoretically, one scene description file can contain scene description modules for multiple 3D scenes. Therefore, the scene description file first indicates by "scene":0 on line 96 that the scene to be subsequently processed and rendered (the 3D scene to be rendered) is the 3D scene corresponding to the first scene description module in the scene list ("scenes":[]) of the scene description file, i.e., the scene described in the scene description file enclosed in square brackets on lines 99-103. From an analytical perspective, this scene statement makes it clear that the entire scene description framework should select the 3D scene corresponding to the first scene description module in the scene list ("scenes":[]) for subsequent processing and rendering.

[0157] 5. Scene List ("scenes":[]): The scene list is on lines 98-104. The scene list contains only one bracket, and the scene description file contains only one scene description module (scene), which describes only one 3D scene. The brackets (scene description module) contain a node index list on lines 100-102, with its contents "nodes":[0,1,2], which indicates that the 3D scene contains three nodes, each corresponding to the first, second, and third node description modules in the node list ("nodes":[]). From an analytical perspective, this scene list ("scenes":[]) clarifies the overall structure of the 3D scene and points to more detailed node description modules (nodes) at lower levels.

[0158] 6. Node List ("nodes":[]): The node list is on lines 106-119. The three brackets in the node list indicate that the scene description file contains three node description modules, the 3D scene has three nodes, and the three nodes correspond one-to-one to the nodes indexed in the node index list of the only scene description module in the scene list ("scenes":[]), and the two are related by index. In the first node description module (the content enclosed in brackets on lines 107-110), "name":"Hapticexample_node1" on line 108 indicates that the name of the corresponding node is "Hapticexample_node1", and "mesh":"0" on line 109 indicates that the content mounted on the corresponding node is the first 3D mesh in the mesh list ("meshes":[]), which corresponds to the mesh description module in the next layer. From an analytical point of view, the first node description module in the node list ("nodes":[]) indicates that the content mounted on the node is a 3D mesh, and the index of the mesh description module for this 3D mesh is 0. The content declared by the second node description module (content enclosed in square brackets on lines 111-114) and the third node description module (content enclosed in square brackets on lines 115-118) is similar to the content declared by the first node description module, and to avoid duplication, detailed descriptions will be omitted here.

[0159] 7. Mesh List ("meshes":[]): The mesh list is on lines 121-159. It contains three square brackets, indicating that the scene description file contains three mesh description modules (mesh). The 3D scene has three 3D meshes, and each of the three 3D meshes is a mesh description module whose node description module is indexed by the mesh index syntax element ("mesh"). In the target mesh description module (the content enclosed in square brackets on lines 122-132), "name":"Hapticexample_mesh1" on line 123 indicates that the name of the corresponding 3D mesh is "Hapticexample_mesh1", which is used only as an identification mark. "primitives" on line 124 indicates that the corresponding 3D mesh has primitive information, i.e., primitives. "attributes" on line 126 and "mode" on line 129 indicate that primitives have two types of information: attributes and modes. "TEXCOORD_0":0 on line 127 indicates that the accessor corresponding to the texture coordinate is the first accessor description module in the accessory list. "mode":0 on line 129 indicates that the topological structure of the corresponding 3D mesh is a scattered point structure.The content stated by the second mesh description module (content enclosed in square brackets on lines 133-158) and the third mesh description module (content enclosed in square brackets on lines 133-132) is similar to the content stated by the first mesh description module, except that the third mesh description module further includes an MPEG material haptic (MPEG_material_Haptic) extension in its primitives, and the MPEG haptic description module ("MPEG_material_Haptic":{}) on lines 153 and 154 indicates that the 3D mesh corresponding to the third mesh description module carries haptic material attributes and points to the first haptic material array in the more detailed next-level haptic material list ("MPEG_material_Haptic":[]).

[0160] 8. Haptic Material List ("MPEG_material_Haptic":[]): The haptic material list is on lines 161-187. The haptic material list (MPEG_material_Haptic) contains only one bracket, indicating that the scene description file contains only one haptic material array (the content contained in the brackets on lines 162-185). In the haptic material array, three haptic material description modules on lines 164-170, 171-177, and 178-184 define three haptic material attributes, namely stiffness, temperature, and vibration. Each haptic material attribute contains two pieces of information: a texture array ("texture":{}) and a haptic material type.The description module corresponding to stiffness is on lines 164-170, where "index":0 on line 166 indicates that stiffness refers to the first texture description module in the texture list, "texCoord":0 on line 167 indicates that stiffness refers to the texture coordinate accessed by the accessor indexed by the first texture coordinate syntax element (Tex Coord_n) in the mesh list as the coordinate reference for adding texture to the haptic texture attribute, "type":Low_resolution on line 169 defines that the type of the haptic texture attribute for stiffness is low_resolution, and the length of the haptic value of the stiffness attribute obtained later is all 8 bits long, and the description module corresponding to temperature is on lines 171-177, where "index":1 on line 173 indicates that temperature refers to the second texture description module in the texture list, and " The "type":Reference defines that the type of the tactile material texture for temperature is Reference, and the description module corresponding to vibration is in lines 178-185. "index":2 in line 180 defines that vibration points to the third texture description module (texture) in the texture list, and "type":Reference in line 183 defines that the type of the tactile material texture for vibration is Reference. From an analytical perspective, the texture type and source of the tactile material attribute contained in the object can be determined based on the tactile material list (MPEG_material_Happtic), and points to the more detailed description module, the texture description module, in the next layer.

[0161] 9. Texture List ("textures":[]): The texture list is on lines 189-202. The texture list contains three brackets. The scene description file contains three texture description modules, each corresponding to stiffness, temperature, and vibration. The texture description modules corresponding to stiffness, temperature, and vibration each contain a sampler syntax element (sampler) and a source syntax element (source). The sampler index syntax element (sampler) and texture resource index syntax element (source) determine the access addresses of the samplers and texture resources for stiffness, temperature, and vibration. The access addresses of the texture resources correspond to the texture resource list ("images":[]) at the next level, and the sampler information corresponds to the sampler list ("samplers":[]) at the next level.

[0162] 10. Texture Resource List ("images":[]): The texture resource list is on lines 204-214. The texture resource list contains three square brackets, which respectively correspond to the access addresses of the texture resources cited in the texture description modules for stiffness, temperature, and vibration. From {"uri":"stiffnessTexture.png"} on lines 205-207, we can determine that the access address of the texture resource for stiffness is stiffnessTexture.png. From {"uri":"temperatureTexture.png"} on lines 208-210, we can determine that the access address of the texture resource for temperature is temperatureTexture.png. From {"uri":"Texture.png"} on lines 211-213, we can determine that the access address of the texture resource for vibration is vibrationTexture.png.

[0163] 11. Sampler List ("samplers":[]). The sampler list is on lines 216-221. The sampler list contains one square bracket, which indicates that the scene description file contains one sampler description module. Only one sampler is needed to display the 3D scene. The corresponding sampler description information can be determined by "magFilter":9729, "minFilter":9987, "wrapS":33648, and "wrapT":33648 on lines 217-220.

[0164] 12. Buffer List ("buffers":[]): The buffer list is in lines 304-319. The buffer list contains two brackets, indicating that the scene description file contains two buffer description modules and that two buffers are required to display the 3D scene. Based on the first buffer description module (the content enclosed in brackets on lines 305-308), it can be determined that the size of the first buffer is 1000, the address of the media file to be stored is "AnimatedBody.bin", and the media file contains coordinate content that requires adding texture information to specific 3D coordinates. Based on the MPEG ring buffer ("MPEG_buffer_circular":{}) in the second buffer description module (the content enclosed in brackets on lines 309-318), it can be determined that the second buffer is a ring buffer modified using MPEG extensions. "media:0" on line 314 indicates that the data source in the ring buffer is the first media file declared in the aforementioned MPEG media (MPEG_media), and "tracks":"#trackIndex=1" on line 315 indicates that the track with index 1 should be referenced when accessing the haptic media package file. Here, the track with index 1 is not limited and may be the only track of a haptic file in a single-track package or the reference track of a haptic file in a multi-track package. Also, based on "count":3 on line 313, it can be determined that the MPEG ring buffer has three memory sections, and based on "byteLength":11000 on line 310, it can be determined that the byte length (capacity) of the MPEG ring buffer is 11000 bytes. From an analytical perspective, the buffer list realizes that haptic media files declared in MPEG media (MPEG_media) correspond to buffers, or that buffers reference haptic media files that were previously declared but not used.It should be noted that the haptic media files cited herein are package files of unprocessed haptic media files, and the package files of haptic media files must be processed by media access functions before information that can be directly used for rendering can be extracted.

[0165] 13. Buffer slice list ("bufferViews":[]): The buffer slice list is in lines 276-302. The buffer slice list contains five parallel square brackets, which indicates that the scene description file contains five buffer slice description modules. The "buffer" :0 on line 278 of the first buffer slice description module (the content enclosed in square brackets on lines 277-281) determines that the buffer slice corresponding to the first buffer slice description module is the buffer slice of the buffer corresponding to the first buffer description module in the buffer list, and the "byteLength" :1000 on line 279 and the "byteOffset" :0 on line 280 state that the data slice range of the corresponding buffer slice is the first 1000 bytes of the first buffer. The "buffer" :1 on line 283 of the second buffer slice description module (the content enclosed in square brackets on lines 282-286) determines that the buffer slice corresponding to the second buffer slice description module is the buffer slice of the buffer corresponding to the second buffer description module in the buffer list, and the "byteLength" :2000 on line 284 and the "byteOffset" :0 on line 285 state that the data slice range of the corresponding buffer slice is the first 1000 bytes of the first buffer. :0 declares that the data slice range of the corresponding buffer slice is the first 2000 bytes of the second buffer. The contents declared by the third, fourth, and fifth buffer slice description modules are similar to the contents declared by the second buffer slice description module. To avoid duplication, detailed descriptions are omitted here. From an analysis perspective, the buffer slice list groups the data of the media file and is useful for detailed definition of the subsequent accessor description modules.

[0166] 14. Accessory List ("accessors":[]): The accessory list is on lines 223-274. The accessory list has a structure similar to that of the buffer slice list, and each contains five parallel square brackets, indicating that the 3D scene description file contains five accessor description modules (accessors), and that the 3D scene display must access media data through three accessors. The first accessor description module (the content enclosed in square brackets on lines 224-229) indicates via "bufferView":0 on line 225 that the accessor corresponding to the first accessor description module is for accessing the first buffer slice in the buffer slice list, and further indicates via "componentType":5121 on line 226 that the data format of the data accessed by the accessor corresponding to the first accessor description module is an 8-bit unsigned short (8 bits), and via the fact that the type of "type":"VEC2" on line 227 is a two-dimensional vector (VEC2), and thus via "count":500 on line 228 that there are 500 pieces of data that the accessor needs to access. The second accessor description module (the content enclosed in square brackets on lines 230-240), the third accessor description module (the content enclosed in square brackets on lines 241-251), the fourth accessor description module (the content enclosed in square brackets on lines 252-262), and the fifth accessor description module (the content enclosed in square brackets on lines 263-273) all have an MPEG time-varying accessor ("MPEG_accessor_timed"), and the corresponding accessors are all MPEG time-varying accessors, and all refer to MPEG-defined time-varying media.The second accessor description module determines that the corresponding accessor accesses the buffer slice corresponding to the second buffer slice description module in the buffer slice list ("bufferViews":[]) by "bufferView":1 on line 236, and indicates that the value of the syntax element in the corresponding accessor will not change over time by "immutable":true on line 237. Similarly, "componentType", "type", and "count" can declare the data type of the data accessed by the corresponding accessor, the corresponding accessor type, and the number of data accessed by the corresponding accessor. The contents declared in the third accessor description module, the fourth accessor description module, and the fifth accessor description module are similar to the contents declared in the second accessor description module, and to avoid duplication, detailed descriptions will be omitted here. From an analysis perspective, this accessor description module fully improves the complete definition of the data required for rendering. For example, the buffer slice and the data type missing from the buffer are defined in the corresponding accessor description module.

[0167] 2. Display engine supporting haptic media files

[0168] In the workflow of the immersive media scene description framework, the main functions of the display engine include: parsing a scene description file to obtain a rendering 3D scene; sending media access commands or media data processing commands to a media access function via a media access function API; sending buffer management commands to a buffer management module via a buffer API; obtaining processed data from the buffer, and completing the rendering and display of the 3D scene and the objects in the 3D scene based on the read data. Therefore, the functions of a display engine that supports haptic media include: 1. analyzing a scene description file containing haptic media and obtaining a corresponding 3D scene rendering method; 2. transmitting a media access command or a media data processing command to the media access function via the media access function API, where the media access command or the media data processing command is derived from the analysis result of the scene description file containing haptic media; 3. sending a buffer management command to a buffer management module via a buffer management API; and 4. obtaining haptic rendering data of the processed haptic media file from the buffer, and completing the rendering and display of the 3D scene and the haptic media file included in the 3D scene based on the obtained haptic rendering data.

[0169] 3. Media access function API supporting haptic media files

[0170] In the workflow of the immersive media scene description framework, the display engine can obtain how to render a three-dimensional scene by analyzing the scene description file, and needs to communicate how to render the three-dimensional scene to the media access function or send commands to the media access function based on the method of rendering the three-dimensional scene, and the process of communicating how to render the three-dimensional scene to the media access function or sending commands to the media access function based on the method of rendering the three-dimensional scene is realized by the media access function API.

[0171] In some embodiments, the display engine may send media access commands or media data processing commands to the media access function via the media access function API, where the commands sent by the display engine are derived from the analysis results of a scene description file containing attribute information of haptic media, and these commands may include addresses of the media files, attribute information of the media files (media type, codec used, etc.), formatting requests for the processed haptic media data and other media data, etc.

[0172] In some embodiments, the media access function may request media access commands or media data processing commands from the display engine via the media access function API.

[0173] 4. Media access functions to support haptic media files

[0174] In the workflow of the immersive media scene description framework, the media access function receives a media access command or a media data processing command sent by the display engine via the media access function API, and then executes the media access command or the media data processing command sent by the display engine via the media access function API, for example, by obtaining a haptic media file, establishing an appropriate pipeline for the haptic media file to convert data of the haptic media file into a format specified in the command, and allocating appropriate buffers for the haptic media data and other media data processed into the format specified in the scene description file.

[0175] In some embodiments, the media access function obtaining the haptic media file includes downloading the haptic media file from a server using a network transmission service.

[0176] In some embodiments, the media access function obtaining the haptic media file includes reading the haptic media file from a local storage space.

[0177] After the media access function obtains the haptic media file, it needs to process the haptic media file. There are significant differences in the processing process of different types of media files. In order to support a wide range of media types and also take into account the work efficiency of the media access function, various pipelines are designed in the media access function, and only the pipeline that matches the media type needs to be enabled in the media file processing process.

[0178] 5. Buffer API to support haptic media files

[0179] After the media access function is processed by the data of the haptic media file, the media access function needs to transmit the processed data to the display engine in a standard array structure, which means that the processed data needs to be accurately stored in the buffer. This work is completed by the buffer management module, which needs to receive buffer management commands from the media access function or the display engine via the buffer API.

[0180] In some embodiments, the media access function can send a buffer management command to the buffer management module via a buffer API, where the buffer management command is a buffer management command sent by the display engine to the media access function via the media access function API.

[0181] In some embodiments, the display engine can send buffer management commands to the buffer management module via the buffer API.

[0182] That is, the buffer management module may communicate with the media access function via the buffer API, or may communicate with the display engine via the buffer API, and the purpose of communicating with the media access function or the display engine is to realize buffer management. When the buffer management module communicates with the media access function via the buffer API, the display engine must first send buffer management commands to the media access function via the media access function API, and the media access function must then send the buffer management commands to the buffer management module via the buffer API. When the buffer management module communicates with the display engine via the buffer API, the display engine must generate buffer management commands based on buffer management information parsed from the scene description file and send them to the buffer management module via the buffer API.

[0183] In some embodiments, the buffer management commands may include one or more of a create buffer command, an update buffer command, and a release buffer command.

[0184] 6. Buffer management module to support haptic media files

[0185] In the workflow of the immersive media scene description framework, after the media access function completes processing the data of the haptic media file through the pipeline, the processed data needs to be passed to the display engine in a standard array structure, which requires the processed data to be accurately stored in a buffer, and this task is handled by the buffer management module.

[0186] The buffer management module implements management operations such as creating, updating, and releasing buffers, and operation commands are received via the buffer API. Buffer management rules are recorded in a scene description document, analyzed by the display engine, and finally sent to the buffer management module by the display engine or the media access function. After media files are processed by the media access function, they need to be stored in appropriate buffers and used by the display engine. The role of buffer management is to properly manage these buffers to match the format of the processed media data without disrupting the processed media data. For specific design methods of the media management module, please refer to the designs of the display engine and the media access function.

[0187] Based on the above, Original Vow Some embodiments of the present invention provide a method for generating a scene description file, and as shown in FIG. 10, the method for generating a scene description file includes the following steps S101-S105.

[0188] S101, determining a haptic media accessor associated with a target action node when a three-dimensional scene to be rendered performs a target haptic action.

[0189] Here, the haptic media accessor is an accessor for accessing haptic rendering data of a haptic media file declared in an MPEG media description module ("MPEG_media":{}) associated with the target action node when performing the target haptic action.

[0190] Original Vow In this embodiment, the haptic rendering data of the haptic media file is data that is directly used to render a three-dimensional scene obtained by processing the haptic media file, such as depackaging, decoding, decoding, format conversion, etc., and may also be data that is directly used to render a three-dimensional scene obtained by performing other processing on the haptic media file. The haptic rendering data of the haptic media file may include haptic values ​​of haptic material attributes, which may include one or more of stiffness, friction, tactile texture, temperature, vibration, and custom attributes.

[0191] Note that when a three-dimensional scene to be rendered performs different haptic actions, the haptic rendering data of a haptic media file associated with the same node in the three-dimensional scene to be rendered may be different (the haptic rendering data of the haptic media file for rendering the node may be different), and therefore when a three-dimensional scene to be rendered performs different haptic actions, the haptic media accessors associated with the same node in the three-dimensional scene to be rendered may be different. For example, when a three-dimensional scene to be rendered performs haptic action A, the haptic media accessors associated with node 1 in the three-dimensional scene to be rendered are accessor 1 and accessor 2, and when a three-dimensional scene to be rendered performs haptic action B, the haptic media accessors associated with node 1 in the three-dimensional scene to be rendered are accessor 2 and accessor 3. As another example, when the three-dimensional scene to be rendered performs haptic action C, the haptic media accessors associated with node 2 in the three-dimensional scene to be rendered are accessor 0 and accessor 1, and when the three-dimensional scene to be rendered performs haptic action D, node 2 in the three-dimensional scene to be rendered does not need to perform haptic feedback and does not perform haptic rendering, so node 2 is not associated with a haptic media accessor.

[0192] It should be further noted that there may be any number of haptic media accessors associated with the target action node when the target haptic action is performed, i.e., there may be one or more haptic media accessors associated with the target action node when performing the target haptic action.

[0193] S102, generate a target haptic object description module ("hapticObject":[]) based on the index value of the accessor description module corresponding to the haptic media accessor, and add the target haptic object description module to an MPEG haptic description module ("MPEG_haptic":{}) in a scene description file of the 3D scene to be rendered.

[0194] That is, to associate haptic rendering data of a haptic media file with a node in a 3D scene, Original Vow The embodiment of MPEG_haptic introduces the concept of a haptic object, which associates haptic rendering data of a haptic media file with a node in a 3D scene.

[0195] In some embodiments, the step of generating a target haptic object description module based on an index value of an accessor description module corresponding to the haptic media accessor includes adding an accessor index syntax element (accessor) to the target haptic object description module and setting the value of the accessor index syntax element to the index value of an accessor description module corresponding to the haptic media accessor.

[0196] For example, if two tactile media accessors are included and the index values ​​of the accessor description modules corresponding to the two tactile media accessors are 1 and 4, respectively, the target tactile object description module generated based on the index values ​​of the accessor description modules corresponding to the tactile media accessors can be shown as follows: [Table F]

[0197] As explained in the above example, index values ​​of accessor description modules corresponding to multiple haptic media accessors may be separated by ",".

[0198] S103: Generate a target haptic action node array based on the index value of the node description module corresponding to the target action node and the index value of the target haptic object description module, and add the target haptic action node array to the haptic action node list ("hapticActionNodes":[]) of the target action description module of the MPEG interactivity description module ("MPEG_interactivity":{}) of the scene description file.

[0199] Here, the target action description module is an action description module corresponding to the target haptic action.

[0200] In some embodiments, generating a target haptic action node array based on the index value of the node description module corresponding to the target action node and the index value of the target haptic object description module includes the following steps 1031 and 1032.

[0201] Step 1031: Add a node index syntax element (node) to the target haptic action node array, and set the value of the node index syntax element to the index value of a node description module corresponding to the target action node.

[0202] Step 1032: Add a haptic object index syntax element (haptic Object) to the target haptic action node array, and set the value of the haptic object index syntax element as the index value of the target haptic object description module.

[0203] Exemplarily, if the index value of the node description module corresponding to the target action node is 0 and the index value of the target tactile object description module is 1, the target tactile action node array generated based on the index value of the node description module corresponding to the target action node and the index value of the target tactile object description module can be shown as follows: [Table G]

[0204] In addition, the 3D scene to be rendered may contain multiple nodes that need to be haptic rendered when performing a target haptic action, and the way some nodes obtain haptic values ​​is to obtain them from haptic textures (the haptic values ​​associated with them are obtained by haptic materials in the mesh description module of the 3D mesh mounted on them), but since there is no need to associate haptic objects, the haptic action node list of the target action description module may contain multiple haptic action node arrays, and the multiple haptic action node arrays may include haptic action node arrays that do not include the haptic object index syntax element (haptic Object).

[0205] For example, if a three-dimensional scene to be rendered includes three nodes, node A, node B, and node C, that need to be rendered when performing a certain haptic action, and the index values ​​of the node description modules corresponding to node A, node B, and node C are 0, 1, and 2, respectively, and the index value of the haptic object description module corresponding to the haptic object associated with node A is 0, and the index value of the haptic object description module corresponding to the haptic object associated with node B is 1, and node C realizes obtaining the haptic value associated with the haptic material in the mesh description module of its mounted three-dimensional mesh and is not associated with a haptic object, then the haptic action node list of the action description module corresponding to the haptic action can be shown as follows: [Table H]

[0206] Here, the contents of rows n+02 to n+05 are the tactile action node array corresponding to node A, the contents of rows n+06 to n+09 are the tactile action node array corresponding to node B, and the contents of rows n+10 to n+12 are the tactile action node array corresponding to node C.

[0207] Original VowIn a scene description file generation method according to an embodiment of the present invention, when generating a scene description file for a 3D scene to be rendered, first, a haptic media accessor associated with a target action node is determined when the 3D scene to be rendered performs a target haptic action, the haptic media accessor being an accessor for accessing haptic rendering data of a haptic media file declared in a Motion Picture Experts Group (MPEG) media description module associated with the target action node when the target haptic action is performed, then a target haptic object description module is generated based on an index value of an accessor description module corresponding to the haptic media accessor, and the target haptic object description module is added to the MPEG haptic description module of the scene description file for the 3D scene to be rendered; while generating a target haptic action node array based on the index value of a node description module corresponding to the target action node and the index value of the target haptic object description module, and the target haptic action node array is added to the haptic action node list of the target action description module of the MPEG interactive description module of the scene description file.Since the haptic media accessor is an accessor for accessing the haptic rendering data of the haptic media file associated with the target action node when performing the target haptic action, and the target action description module is an action description module corresponding to the target haptic action, when triggering the three-dimensional scene to be rendered to perform the target haptic action, it can be determined that the target action node needs to be rendered based on the target haptic action node array and the target haptic object description module, and the data for rendering the target action node is the haptic rendering data of the haptic media file accessed by the haptic media accessor, so the above embodiment realizes an association between haptic actions, nodes, and haptic rendering data of haptic media files, and realizes supporting haptic media files in the scene description framework.

[0208] In some embodiments, the method for generating the scene description file adds an action type syntax element (type) to the target action description module and sets a value of the action type syntax element based on the action type of the target action.

[0209] As described in the above example, after adding an action type syntax element to the target action description module and setting the value of the action type syntax element based on the action type of the target action, the target action description module can be shown as follows: [Table I]

[0210] In some embodiments, the method for generating the scene description file further includes generating a target behavior array corresponding to the target behavior based on an index value of a trigger description module corresponding to a trigger associated with the target behavior and an index value of an action description module corresponding to an action associated with the target behavior, and adding the target behavior array to a behavior list (behaviors":[]) of the MPEG interactivity description module ("MPEG_interactivity":{}), where the target behavior is any behavior in the three-dimensional scene to be rendered.

[0211] In some embodiments, generating a target behavior array corresponding to the target behavior based on an index value of a trigger description module corresponding to a trigger associated with the target behavior and an index value of an action description module corresponding to an action associated with the target behavior includes the following steps 1 and 2.

[0212] In step 1, a trigger index syntax element (triggers) is added to the target behavior array, and the value of the trigger index syntax element is set to the index value of a trigger description module corresponding to a trigger associated with the target behavior.

[0213] In step 2, an action index syntax element (actions) is added to the target behavior array, and the value of the action index syntax element is set to the index value of the action description module corresponding to the action to be associated with the target behavior.

[0214] Note that one action may be associated with one or more behaviors.

[0215] For example, if the index value of the trigger description module corresponding to the trigger associated with the target behavior is 0, two actions are associated with the target behavior, and the index values ​​of the action description modules corresponding to the two actions are 1 and 2, respectively, the target behavior array corresponding to the target behavior is shown as follows: [Table J]

[0216] In some embodiments, the method for generating the scene description file further includes: generating a target trigger description module corresponding to the target trigger based on a type of the target trigger, and adding the target trigger description module to a trigger list ("triggers":[]) of the MPEG interactivity description module ("MPEG_interactivity":{}), where the target trigger is any trigger in the three-dimensional scene to be rendered.

[0217] In some embodiments, generating a target trigger description module corresponding to the target trigger based on the type of the target trigger includes adding a trigger type syntax element (type) to the target trigger description module and setting a value of the trigger type syntax element based on the type of the target trigger.

[0218] As shown in Table 20 above, if the target trigger type is collision, the trigger type syntax element and its value are set to "type":"TRIGGER_COLLISION", if the target trigger type is proximity, the trigger type syntax element and its value are set to "type":"TRIGGER_PROXIMITY", if the target trigger type is user access, the trigger type syntax element and its value are set to "type":"RIGGER_USER_INPUT", and if the target trigger type is visibility, the trigger type syntax element and its value are set to "type":"TRIGGER_VISIBILITY".

[0219] For example, if the target trigger type is collision, the target trigger description module corresponding to the target trigger can be expressed as follows: [Table K]

[0220] In some embodiments, the method for generating the scene description file further includes generating a target media description module corresponding to the haptic media file based on description information of the haptic media file, and adding the target media description module to a media list ("media") of an MPEG media description module ("MPEG_media":{}) of the scene description file.

[0221] In some embodiments, generating a target media description module corresponding to the haptic media file based on description information of the haptic media file includes at least one of the following steps 31-35.

[0222] In step 31, a media name syntax element (name) is added to the target media description module, and the value of the media name syntax element is set based on the name of the haptic media file.

[0223] For example, if the name of the haptic media file is "Haptic example", add a media name syntax element (name) to the target media description module, and set the media name syntax element and its value to "name":"Haptic example".

[0224] In step 32, an autoplay syntax element (autoplay) is added to the target media description module, and the value of the autoplay syntax element is set based on whether the haptic media file needs to be autoplayed.

[0225] For example, if the haptic media file needs to be autoplayed, add the syntax element "autoplay" to the target media description module and set the autoplay syntax element and its value to "autoplay":true or "autoplay":1.

[0226] As another example, if the haptic media file does not need to be autoplayed, the syntax element "autoplay" is added to the target media description module, and the autoplay syntax element and its value are set to "autoplay":false or "autoplay":0.

[0227] In step 33, in the target media description module circulation regeneration Syntax element (loop) addition and sets the value of the cyclic playback syntax element based on whether the haptic media file needs to be cyclically played.

[0228] For example, if the haptic media file needs to be played in a circular fashion, add a syntax element "loop" to the target media description module and set the circular playback syntax element and its value to "loop":true or "autoplay":1.

[0229] In another example, if the haptic media file does not require circular playback, the syntax element "loop" is added to the target media description module, and the circular playback syntax element and its value are set to "loop":false or "autoplay":0.

[0230] In step 34, an optional version list ("alternatives":[]) is added to the target media description module.

[0231] In step 35, based on the description information of each optional version of the haptic media file, an optional version description module corresponding to each optional version of the haptic media file is generated, and the optional version description module corresponding to each optional version of the haptic media file is added to the optional version list (“alternatives”:[]).

[0232] In some embodiments, in the above step 35, generating an optional version description module corresponding to each optional version of the haptic media file based on the description information of each optional version of the haptic media file includes at least one of the following steps 351 to 355.

[0233] In step 351, a media type syntax element (mime Type) is added to a target option version description module corresponding to the target option version, and the value of the media type syntax element is set based on the data contained in the target option version.

[0234] Here, the target optional version is any optional version of the haptic media file. That is, for each optional version of the haptic media file, Original Vow This embodiment allows for the creation of an optional version description module.

[0235] In some embodiments, setting the value of the media type syntax element based on data included in the target option version includes setting the value of the media type syntax element to a first preset value if the haptic media file includes audio and / or video data, and setting the value of the media type syntax element to a second preset value if the haptic media file does not include audio and video data.

[0236] Exemplarily, the first preset value may be "video / mp4" and the second preset value may be "haptic / mp4".

[0237] In step 352, add a uniform resource identifier syntax element (uri) to the target option version description module and set the value of the uniform resource identifier syntax element based on the uniform resource identifier (URI) of the target option version.

[0238] For example, if the uniform resource identifier of the target option version of the haptic media file is "http: / www.Hapticexample.mp4", add a uniform resource identifier syntax element (uri) to the target option version description module and set the uniform resource identifier syntax element and its value to "uri":"http: / www.Hapticexample.mp4".

[0239] In step 353, a tracks array (tracks[]) is added to the target option version description module.

[0240] In step 354, a first track index syntax element (track) is added to the tracks array (tracks[]) and the value of the first track index syntax element is set to the index value of each bitstream track of the target option version.

[0241] That is, if the encoded haptic media is referenced by the scene description file as one of MPEG_media.alternative.tracks and the referenced item conforms to the specifications for tracks in ISOBMFF of the ISO / IEC 23090-32 standard, for a single-track packaged haptic media file, the track referenced in MPEG_media is the single track in the haptic package. For a multi-track packaged haptic media file, the track referenced in MPEG_media is the multiple tracks in which the haptic media is packaged. For example, a non-zero value is predefined in the Track Header Box of the alternate_group configuration information of the packaging file to indicate that the haptic media is packaged into multiple MIHS tracks and that these tracks are also tracks indexed in MPEG_media.

[0242] In step 355, a codec parameter syntax element (codecs) is added to the track array (tracks[]), and the value of the codec parameter syntax element is set based on the encoding parameters of the data in each bitstream track of the target option version and the provisions of the ISO / IEC 23090-32 haptic data transmission standard.

[0243] Codec parameters for the media contained in the track array (tracks), as defined in IETF RFC 6381. If the track array contains codecs of different types (e.g., if an AdaptationSet contains representations with different codecs), the codecparameters syntax element may be represented as a comma-separated list of codec values.

[0244] If the media data is haptic data, it should follow the relevant detailed specifications in the ISO / IEC 23090-32 Carriage of Haptics Data standard. For example, if the haptic data is packaged in ISMBOFF, the first group of elements in the codec parameter syntax element is a code representing the codec in ISO / IEC 23090-31, and the value format of the codec parameter syntax element is "codec=mih1.oo", where "mih1" is a description of one type of data format in which haptic data is packaged in ISOBMFF. This value is not unique and allows replacement with sample format names that match other ISOBMFF packaging. Here, "oo" is an object type indicator defined on the "Object Types" page of the MP4 Registration Authority website.

[0245] For example, if the name of the haptic media file is Haptic example, the haptic media file requires automatic playback and circular playback, the haptic media file contains audio data and / or video data, the access address of the haptic media file is www.example.com / Haptic example.mp4, the haptic media file is a single-track package file, and the index value of the bitstream track is 1, the target media description module can be shown as follows: [Table L]

[0246] In some embodiments, the method for generating a scene description file further comprises:

[0247] Based on the description information of the three-dimensional scene to be rendered, a target scene description module corresponding to the three-dimensional scene to be rendered is generated, and the target scene description module is added to a scene list ("scenes":[]) of the scene description file.

[0248] In some embodiments, generating a target scene description module corresponding to the three-dimensional scene to be rendered based on the description information of the three-dimensional scene to be rendered includes adding a node index list (“nodes”:[]) to the target scene description module, and adding index values ​​of node description modules corresponding to each top node in the three-dimensional scene to the node index list.

[0249] In addition, Original Vow The embodiment adds to the node index list only the index value of the node description module corresponding to the top node in the 3D scene to be rendered, not the index values ​​of the node description modules corresponding to all nodes (not including the index values ​​of the node description modules corresponding to subnodes).

[0250] For example, if the scene to be rendered includes three top-level nodes and the index values ​​of the node description modules corresponding to the three top-level nodes are 0, 1, and 2, respectively, the target scene description module is shown as follows: [Table M]

[0251] In some embodiments, the method for generating a scene description file further includes: generating a target node description module corresponding to the target node based on description information of the target node, and adding the target node description module to a node list ("nodes":[]) of the scene description file, where the target node is an arbitrary node in the three-dimensional scene to be rendered.

[0252] In some embodiments, generating a target node description module corresponding to the target node based on the description information of the target node includes at least one of the following steps 41-44:

[0253] In step 41, a node name syntax element (name) is added to the target node description module, and the value of the node name syntax element is set based on the name of the target node.

[0254] For example, if the name of the target node is "Haptic_node", add a node name syntax element (name) to the target media description module, and set the node name syntax element and its value in the target node description module to "name":"Haptic_node".

[0255] In step 42, a subnode index list ("children":[]) is added to the target node description module, and the index value of the node description module corresponding to each subnode mounted on the target node is added to the subnode index list.

[0256] For example, the target node mounts two subnodes, the index value of the node description module corresponding to one subnode is 1, and the index value of the node description module corresponding to another subnode is 2, and a part of the subnode index list of the target node description module is shown as follows: [Table N]

[0257] In step 43, a mesh index syntax element (mesh) is added to the target node description module, and the value of the mesh index syntax element (mesh) is set as the index value of the mesh description module corresponding to the 3D mesh mounted on the target node.

[0258] For example, if one 3D mesh is mounted on the target node and the index value of the mesh description module corresponding to the 3D mesh is 0, add a mesh index syntax element (mesh) to the target media description module, and set the mesh index syntax element and its value in the target node description module to "mesh":0.

[0259] In step 44, a position offset syntax element (translation) is added to the target node description module, and the value of the position offset syntax element is set based on the spatial position offset of the target node relative to its parent node.

[0260] It should be noted that what is directly obtained based on the value of the position offset syntax element in the node description module is the spatial position offset of the node relative to its parent node, which is always the spatial position offset of the node relative to the node representing the target digital human, and the spatial position offset of the node relative to the node representing the target digital human is the superposition of the spatial offsets of each node on the path from the node to the node representing the target digital human.

[0261] For example, if node A is a subnode of the node representing the target digital human, node B is a subnode of node A, the position offset syntax element in the node description module corresponding to node A is "translation":[0.0,10.0,25.0], and the position offset syntax element in the node description module corresponding to node B is "translation":[0.0,10.0,20.0], first, obtain that the offset of node A relative to the node representing the target digital human is [0.0,10.0,25.0], and obtain that the offset of node B relative to node A is [0.0,00.0,20.0], and further combine the position offsets to obtain that the offset of node B relative to the node representing the target digital human is [0.0,10.0,45.0].

[0262] For example, if the name of the target node is "Hapticexample_mesh", the target node includes two subnodes, and the index values ​​of the node description modules corresponding to the two subnodes are 1 and 2, respectively, the index value of the mesh description module corresponding to the 3D mesh mounted on the target node is 0, and the spatial position offset of the target node relative to its parent node is [0.0, 10.0, 25.0], the node description module generated based on the description information of the target node can be shown as follows: [Table O]

[0263] In some embodiments, the method for generating the scene description file further includes: generating a target mesh description module corresponding to the target 3D mesh based on description information of the target 3D mesh, and adding the target mesh description module to a mesh list ("meshes":[]) of the scene description file, where the target 3D mesh is any 3D mesh in the 3D scene to be rendered.

[0264] In some embodiments, generating a target mesh description module corresponding to the target 3D mesh based on description information of the target 3D mesh includes at least one of the following steps 51 to 56:

[0265] In step 51, a mesh name syntax element (name) is added to the target mesh description module, and the value of the mesh name syntax element is set based on the name of the target 3D mesh.

[0266] For example, if the name of the target 3D mesh is "example_mesh", add a mesh name syntax element to the target mesh description module, and set the mesh name syntax element and its value to "name":"example_mesh".

[0267] In step 52, a texture coordinate syntax element (TEX COORD_n) is added to the attributes of the primitives of the target mesh description module, and the texture coordinate syntax element is set to the index value of an accessor description module corresponding to an accessor for accessing the texture coordinates of the target 3D mesh.

[0268] In step 53, a mode syntax element (mode) is added to the primitive of the target mesh description module, and the value of the mode syntax element is set based on the topology type of the target 3D mesh.

[0269] In step 54, an MPEG haptic description module ("MPEG_material_Happtic":{}) is added to the primitives of the target mesh description module, a haptic material array index syntax element (index) is added to the MPEG haptic description module ("MPEG_material_Happtic":{}), and the value of the haptic material array index syntax element is set to the index value of the haptic material array associated with the target 3D mesh.

[0270] For example, if the name of the target 3D mesh is Haptic example_mesh, the topological structure of the target 3D mesh is scattered, and the index value of the haptic material attribute array associated with the target 3D mesh is 2, the target mesh description module corresponding to the target 3D mesh can be shown as follows: [Table P]

[0271] In some embodiments, the method for generating a scene description file further includes the following steps 6 to 8.

[0272] In step 6, a haptic material attribute description module corresponding to each haptic material attribute mounted on the target 3D mesh is generated based on the description information of each haptic material attribute mounted on the target 3D mesh.

[0273] In some embodiments, step 6 above (generating a tactile material attribute description module corresponding to each tactile material attribute to be mounted on the target 3D mesh based on description information of each tactile material attribute to be mounted on the target 3D mesh) includes at least one of the following steps 61 to 64:

[0274] In step 61, a texture array is added to the target haptic material attribute description module corresponding to the target haptic material attribute.

[0275] Here, the target haptic material attribute is any haptic material attribute that is mounted on the target 3D mesh.

[0276] In step 62, a texture index syntax element (index) is added to the texture array, and the value of the texture index syntax element is set as the index value of the texture array associated with the target haptic material attribute.

[0277] In step 63, a texture coordinate index syntax element (tex Coord) is added to the texture array and the value of the texture index syntax element is set to the index value of the texture coordinate syntax element (TEX COORD_n) associated with the target haptic material attribute.

[0278] In step 64, a haptic material type syntax element (type) is added to the target haptic material attribute description module, and the value of the haptic material type syntax element is set based on the type of the target haptic material attribute.

[0279] In some embodiments, the types of haptic material attributes include High_resolution, Low_resolution, and Reference.

[0280] For example, for a low resolution haptic material type of rigid, when the index value of the associated texture array is 0 and the index value of the associated texture coordinate syntax element is 0, the haptic material attribute description module corresponding to rigidity can be expressed as follows: [Table Q]

[0281] In step 7, a haptic material array associated with the target 3D mesh is generated based on the haptic material attribute description module corresponding to each haptic material attribute mounted on the target 3D mesh, and the haptic material array associated with the target 3D mesh is added to the haptic material list ("MPEG_material_Happtic":[]) of the scene description file.

[0282] For example, the tactile material attributes mounted on the target 3D mesh include stiffness and temperature, and the stiffness type is low resolution and the temperature type is reference, the tactile material attribute array of the target 3D mesh can be shown as follows: [Table R]

[0283] For example, if haptic material attributes are mounted on two 3D meshes in the scene to be rendered, and the haptic material attribute mounted on one 3D mesh is rigid and of type low resolution, and the haptic material attribute mounted on the other mesh description module is temperature and of type reference, the haptic material array list in the scene description file can be shown as follows: [Table S]

[0284] In some embodiments, the method for generating the scene description file further includes generating a texture array associated with the target tactile material attribute based on texture description information of the target tactile material attribute, and adding the texture array associated with the target tactile material attribute to a texture list (“textures”:[]) of the scene description file.

[0285] In some embodiments, generating a texture array associated with the target haptic material attribute based on texture description information of the target haptic material attribute includes at least one of the following steps 81 and 82.

[0286] In step 81, a sampler index syntax element (sampler) is added to the texture array associated with the target haptic material attribute, and the value of the sampler index syntax element is set to the index value of the sampler description module corresponding to the sampler of the target haptic material attribute.

[0287] In step 82, a texture resource index syntax element (source) is added to the texture array associated with the target haptic material attribute, and the value of the texture resource index syntax element is set to the index value of a texture resource description module corresponding to the texture resource of the target haptic material attribute.

[0288] For example, if the index value of the texture resource description module corresponding to the texture resource of the target tactile material attribute is 2 and the index value of the sampler description module corresponding to the sampler of the target tactile material attribute is 3, generating a texture array associated with the target tactile material attribute based on the texture description information of the target tactile material attribute can be shown as follows:

[0289] [Table T]

[0290] In some embodiments, the method for generating the scene description file further includes generating a sampler description module corresponding to the sampler of the target tactile material attribute based on description information of the sampler of the target tactile material attribute, and adding the sampler description module corresponding to the sampler of the target tactile material attribute to a sampler list ("samplers":[]) of the scene description file.

[0291] For example, if the Mag filter parameter of a sampler for a target tactile material attribute is :9729, the Min filter parameter is :9987, the horizontal coverage parameter is :33648, and the numeric coverage parameter is :33648, the sampler description module corresponding to the sampler for the target tactile material attribute can be shown as follows: [Table U]

[0292] In some embodiments, the method for generating the scene description file further includes: generating a texture resource description module corresponding to the texture resource of the target haptic material attribute based on an access address of the texture resource of the target haptic material attribute; and adding the texture resource description module corresponding to the texture resource of the target haptic material attribute to a texture resource list ("images":[]) of the scene description file.

[0293] For example, if the access address of the texture resource of the target tactile material attribute is temperatureTexture.png, generating a texture resource description module corresponding to the texture resource of the target tactile material attribute based on the access address of the texture resource of the target tactile material attribute can be shown as follows: [Table V]

[0294] In some embodiments, the method for generating the scene description file further includes: generating a target accessor description module corresponding to the target accessor based on description information of the target accessor, and adding the target accessor description module to an accessories list ("accessors":[]) of the scene description file, where the target accessor is any accessor for realizing rendering of the three-dimensional scene to be rendered.

[0295] In some embodiments, generating a target accessor description module corresponding to the target accessor based on description information of the target accessor includes at least one of the following steps 91-96.

[0296] In step 91, a data type syntax element (component Type) is added to the target accessor description module, and the value of the data type syntax element is set based on the data type of the data accessed by the target accessor.

[0297] For example, if the data type of data accessed by a certain accessor is 5126, the data type syntax element and its value in the accessor description module corresponding to that accessor are set to "componentType":5126.

[0298] In step 92, an accessor type syntax element (type) is added to the target accessor description module, and the value of the accessor type syntax element is set based on the type of the target accessor.

[0299] For example, if the accessor type accessed by a certain accessor is a three-dimensional vector, the accessor type syntax element (type) and its value in the accessor description module corresponding to the accessor are set to "type":"VEC3".

[0300] In step 93, the target accessor description module data A quantity syntax element (count) is added, and the value of the data quantity syntax element is set based on the quantity of data accessed by the target accessor.

[0301] For example, if the number of data accessed by a certain accessor is 1024, the syntax element (count) for the number of data in the accessor description module corresponding to that accessor and its value are set to "count":1024.

[0302] In step 94, a first buffer slice index syntax element (buffer View) is added to the target accessor description module, and the value of the first buffer slice index syntax element is set based on an index value of the buffer slice description module corresponding to a buffer slice for buffering data accessed by the target accessor.

[0303] For example, if the buffer slice description module corresponding to the buffer slice that buffers data accessed by a certain accessor is the fourth buffer slice description module (with an index value of 3) in the buffer slice list (buffer Views) of the scene description file, the target buffer slice index syntax element and its value in the accessor description module corresponding to that accessor are set to "buffer View":3.

[0304] In step 95, an MPEG time-varying accessor ("MPEG_accessor_timed":{}) is added to the extensions list (extensions":{}) of the target accessor description module.

[0305] In step 96, a second buffer slice index syntax element (buffer View) is added to the MPEG time-varying accessor, and the value of the second buffer slice index syntax element is set based on the index value of a buffer slice description module corresponding to a buffer slice for buffering the time-varying parameters of the target accessor.

[0306] For example, if the buffer slice description module corresponding to the buffer slice that buffers the time-varying parameters of a certain accessor is the second buffer slice description module in the buffer slice list (buffer Views) of the scene description file, the second buffer slice index syntax element in the accessor description module corresponding to that accessor and its value are set to "bufferView":1.

[0307] In step 97, a time-varying syntax element (immutable) is added to the MPEG time-varying accessor, and the value of the time-varying syntax element is set based on whether the value of the syntax element in the target accessor changes over time.

[0308] In some embodiments, setting the value of the time-varying syntax element based on whether the value of the syntax element in the target accessor changes over time includes setting the time-varying syntax element in the MPEG time-varying accessor of the target accessor description module and its value to "immutable":true or "immutable":1 if the value of the syntax element in the target accessor does not change over time, and setting the time-varying syntax element in the MPEG time-varying accessor of the target accessor description module and its value to "immutable":false or "immutable":0 if the value of the syntax element in the target accessor changes over time.

[0309] In step 98, an accessor name syntax element (name) is added to the target accessor description module, and the value of the accessor name syntax element is set based on the name of the target accessor.

[0310] For example, if the type of data accessed by a certain accessor is 5126, the name of the accessor is "haptic Accessor", the type of the accessor is VEC3, the number of data accessed by the accessor is 1828, the index value of the buffer slice description module corresponding to the buffer slice that buffers the data accessed by the accessor is 0, the value of the syntax element in the accessor changes over time, and the index value of the buffer slice description module corresponding to the buffer slice that buffers the time-varying parameters of the accessor is 2, the accessor description module corresponding to the accessor can be shown as follows: [Table W]

[0311] In some embodiments, Original VowThe scene description file generating method according to the embodiment further includes: generating a target buffer description module corresponding to the target buffer based on description information of the target buffer, and adding the target buffer description module to a buffer list ("buffers":[]) of the scene description file, where the target buffer is any buffer for realizing the three-dimensional scene rendering to be rendered.

[0312] In some embodiments, generating a target buffer description module corresponding to the target buffer based on the description information of the target buffer includes at least one of the following steps 101 to 107:

[0313] In step 101, a first byte length syntax element (byte Length) is added to the target buffer description module, and the value of the first byte length syntax element is set based on the capacity of the target buffer.

[0314] For example, if the capacity of a buffer is 15000 bytes, the first byte length syntax element in the buffer description module corresponding to the buffer is set to "byteLenth":15000.

[0315] In step 102, an MPEG circular buffer ("MPEG_buffer_circular":{}) is added to the target buffer description module.

[0316] In step 103, a segment count syntax element (count) is added to the MPEG circular buffer ("MPEG_buffer_circular":{}), and the value of the segment count syntax element is set based on the number of stored segments in the target buffer.

[0317] For example, if the number of storage segments in a certain buffer is 5, the number of segments syntax element in the MPEG ring buffer of the buffer description module corresponding to that buffer and its value are set to "count":5.

[0318] In step 104, a media index syntax element (media) is added to the MPEG ring buffer, and the value of the media index syntax element is set based on the index value of the media description module corresponding to the media file to which the source data of the data buffered in the target buffer belongs.

[0319] For example, if the index value of the media description module corresponding to the media file to which the source data of the data buffered in a certain buffer belongs is 0, the media index syntax element and its value in the MPEG ring buffer of the buffer description module corresponding to the buffer are set to "media":0.

[0320] In step 105, a track index syntax element (tracks) is added to the MPEG ring buffer, and the value of the track index syntax element is set based on the track index value of the source data of the data buffered in the target buffer.

[0321] In step 106, add a buffer name syntax element (name) to the buffer description module corresponding to the target buffer, and set the buffer name syntax element based on the name of the target buffer.

[0322] In step 107, a uniform resource identifier syntax element (uri) is added to the target buffer description module, and the value of the uniform resource identifier syntax element is set to the uniform resource identifier of the source data of the data buffered in the target buffer.

[0323] For example, if the capacity of a buffer called "hapticBuffer" is 108940 bytes, the number of storage segments of the buffer is 3, and the index value of the media description module corresponding to the media file to which the source data of the data buffered in the buffer belongs is 1, the buffer description module corresponding to the buffer to be added to the buffer list of the scene description file can be shown as follows: [Table X]

[0324] In some embodiments, the method for generating the scene description file further includes: generating a target buffer slice description module corresponding to the target buffer slice based on description information of the target buffer slice, and adding the target buffer slice description module to a buffer slice list (“bufferViews”:[]) of the scene description file, where the target buffer slice is any buffer slice of a buffer used to render the three-dimensional scene to be rendered.

[0325] In some embodiments, generating a target buffer slice description module corresponding to the target buffer slice based on description information of the target buffer slice includes at least one of the following steps 111 to 114:

[0326] Steps 111 In the method, a buffer index syntax element (buffer) is added to the target buffer slice description module, and a value of the buffer index syntax element is set based on an index value of a buffer description module corresponding to a buffer to which the target buffer slice belongs.

[0327] For example, if the index value of a buffer description module corresponding to a buffer is 2, the buffer index syntax element and its value in the buffer slice description module corresponding to the buffer slice of that buffer are set to "buffer":2.

[0328] In step 112, a second byte length syntax element (byte Length) is added to a buffer slice description module corresponding to the target buffer slice, and the value of the second byte length syntax element (byte Length) is set based on the capacity of the target buffer slice.

[0329] In step 113, an offset syntax element (byte Offset) is added to the target buffer slice description module, and the value of the offset syntax element is set based on the offset of the data buffered in the target buffer slice.

[0330] In step 114, a buffer slice name syntax element (name) is added to the target buffer slice description module, and the value of the buffer slice name syntax element is set based on the name of the target buffer slice.

[0331] For example, if the index value of a buffer description module corresponding to a certain buffer is 0, the capacity of the buffer is 43972, the buffer includes three buffer slices, the name of the first buffer slice is "hapticbuffer Views1", the capacity is 21936, and the offset is 0, the name of the second buffer slice is "hapticbuffer Views2", the capacity is 21936, and the offset is 21936, and the name of the third buffer slice is "hapticbuffer Views3", the capacity is 100, and the offset is 43872, the buffer slice description module portion corresponding to the buffer slices of the buffer in the scene description file can be shown as follows: [Table Y]

[0332] In some embodiments, the method for generating the scene description file further includes adding a digital asset description module (“asset”:{}) to the scene description file, adding a version syntax element (version) to the digital asset description module, and setting a value of the version syntax element based on version information of the scene description file.

[0333] For example, if the scene description file creates a scene description statement based on the glTF 2.0 version, the value of the version syntax element is set to 2.0.

[0334] Exemplarily, the digital asset description module added to the scene description file can be shown as follows: [Table Z]

[0335] In some embodiments, the method for generating a scene description file further includes adding an extensions used list ("extensions Used":[]) to the scene description file, and adding to the extensions used list a top-level extension item of the MPEG to glTF 2.0 version scene description file used by the scene description file.

[0336] For example, MPEG extension items used in a scene description file include MPEG media (MPEG_media), MPEG interaction (MPEG_interactivity), MPEG haptics (MPEG_Haptic), MPEG haptic material (MPEG_material_Haptic), MPEG ring buffer (MPEG_buffer_circular), and MPEG time-varying accessor (MPEG_accessor_timed), and the extension usage list added to the scene description file can be shown as follows: [Table AA]

[0337] In some embodiments, the method for generating the scene description file further comprises adding a scene statement (scene) to the scene description file and setting the value of the scene statement to an index value of a scene description module corresponding to the scene to be rendered.

[0338] Illustratively, if the index value of the scene description module corresponding to the scene to be rendered is 0, adding a scene statement to the scene description file can be shown as follows: [Table AB]

[0339] Original VowSome embodiments of the present invention further provide a scene description file analysis method, which, as shown in FIG. 11, includes the following steps S111-S112: 114 Includes.

[0340] In S111, a target haptic action node array is obtained from the haptic action node list ("hapticActionNodes":[]) of the target action description module of the MPEG interactive description module ("MPEG_interactivity":{}) of the scene description file of the 3D scene to be rendered.

[0341] Here, the target action description module may be any action description module in an MPEG interactive description module, and the target haptic action node array may be any haptic action node array in the haptic action node list of the target action description module.

[0342] Illustratively, the target action description module can be shown as follows: [Table AC]

[0343] Although the third haptic action node array (lines n+12 to n+14) in the haptic action node list ("haptic Action Nodes":[]) of the target action description module does not contain a haptic object syntax element (haptic Object), haptic data must also be associated with the node declared in the third haptic action node array (the node corresponding to the node description module with an index value of 2), and the haptic data associated with that node is associated with the MPEG haptic material (MPEG_haptic_material) in the mesh description module.

[0344] At S112, an index value of a target haptic object description module corresponding to a haptic object associated with the target action node is determined based on the target haptic action node array.

[0345] For example, a target haptic action node array is shown below: [Table AD]

[0346] The node corresponding to the node description module with an index value of 0 (the first node description module in the node list) is the target action node, and the index value of the target haptic object description module corresponding to the haptic object associated with the target action node is 0 (the first haptic object description module in the MPEG haptic description module).

[0347] In S113, the target haptic object description module is obtained from the MPEG haptic description module ("MPEG_haptic":{}) of the scene description file based on the index value of the target haptic object description module.

[0348] In some embodiments, determining an index value of a target haptic object description module corresponding to a haptic object associated with a target action node based on the target haptic action node array includes obtaining a value of a haptic object index syntax element (haptic Object) in the target haptic action node array, and determining an index value of the target haptic object description module based on the value of the haptic object index syntax element.

[0349] As described in the above example, since the haptic object index syntax element and its value in the target haptic action node array are "hapticObject":0, it can be determined that the index value of the target haptic object description module is 0, and further, the first haptic object description module can be obtained from the MPEG haptic description modules ("MPEG_haptic":{}) in the scene description file as the target haptic object description module.

[0350] In S114, based on the target haptic object description module, description information of a haptic media accessor associated with the target action node when the three-dimensional scene to be rendered performs a target haptic action is obtained.

[0351] Here, the target haptic action is a haptic action corresponding to the target action description module, and the haptic media accessor is an accessor for accessing haptic rendering data of a haptic media file declared in an MPEG media description module ("MPEG_media":{}) associated with the target action node when executing the target haptic action.

[0352] It should be noted that the haptic media accessor associated with an action node may be one accessor or multiple accessors.

[0353] In some embodiments, the above step S114 (obtaining description information of the haptic media accessor associated with the target action node when the three-dimensional scene to be rendered performs the target haptic action based on the target haptic object description module) includes the following steps 1141 to 1143.

[0354] In step 1141, the value of the accessor index syntax element (accessor) in the target haptic object description module is obtained.

[0355] In step 1142, an accessor description module corresponding to the haptic media accessor is obtained from the accessories list ("accessors":[]) of the scene description file based on the value of the accessor index syntax element.

[0356] In step 1143, description information of the haptic media accessor is obtained based on an accessor description module corresponding to the haptic media accessor.

[0357] Illustratively, the target haptic object description module is shown below: [Table AE]

[0358] The above steps 1141 to 1143 include obtaining the second accessor description module and the fifth accessor description module from the accessories list ("accessors":[]) of the scene description file based on the values ​​(1 and 4) of the accessor index syntax element (accessor), and obtaining description information of the tactile media accessor based on the second accessor description module and the fifth accessor description module.

[0359] Original VowA scene description file analysis method according to an embodiment of the present invention includes obtaining a target haptic action node array from a haptic action node list of a target action description module of an MPEG interactive description module of a scene description file of a 3D scene to be rendered, determining an index value of a target haptic object description module corresponding to a haptic object associated with the target action node based on the target haptic action node array, obtaining the target haptic object description module from the MPEG haptic description module of the scene description file based on the index value of the target haptic object description module, and obtaining description information of an accessor for accessing haptic rendering data of a haptic media file declared in the MPEG media description module associated with the target action node based on the target haptic object description module when the 3D scene to be rendered performs a haptic action corresponding to the target action description module. Original Vow The scene description file analysis method provided by the embodiment can obtain the description information of the accessor for accessing the haptic rendering data of the haptic media file associated with the action node when the three-dimensional scene to be rendered performs a haptic action, and realizes the association between the haptic action, the node and the haptic rendering data of the haptic media file, so that the above embodiment realizes the support of haptic media files in the scene description framework.

[0360] In some embodiments, the method for parsing the scene description file further includes obtaining a value of an action type syntax element (type) in the target action description module, and obtaining an action type of the target action based on the value of the action type syntax element.

[0361] For example, if the action type syntax element and its value are "type":"ACTION_HAPTIC", it determines that the action type of the target action is obtaining haptic feedback at a set of nodes.

[0362] In some embodiments, the method for analyzing the scene description file further includes obtaining a target behavior sequence from a behavior list (behaviors":[]) of the MPEG interactivity description module ("MPEG_interactivity":{}), and obtaining description information of a target behavior corresponding to the target behavior sequence based on the target behavior sequence, where the target behavior sequence is any of the behavior sequences in the behavior list (behaviors":[]).

[0363] By way of example, the target behavior sequence can be shown as follows: [Table AF]

[0364] In some embodiments, obtaining description information of the target behavior corresponding to the target behavior sequence based on the target behavior sequence includes at least one of the following steps a1 and a2:

[0365] In step a1, an index value of a trigger description module corresponding to a trigger of the target behavior is determined based on the value of a trigger index syntax element (triggers) in the target behavior sequence.

[0366] As described in the above example, since the value of the trigger index syntax element (triggers) in the target behavior array is 0, it can be determined that the index value of the trigger description module corresponding to the trigger of the target behavior is 0.

[0367] In step a2, an index value of an action description module corresponding to an action of the target behavior is determined based on the value of an action index syntax element (actions) in the target behavior sequence.

[0368] As described in the above example, since the values ​​of the trigger index syntax element (triggers) in the target behavior array are 0 and 1, it can be determined that the target behavior includes two actions, and that the two actions correspond to the action description modules with index values ​​0 and 1, respectively.

[0369] In some embodiments, the scene description file parsing method further includes obtaining a target trigger description module from a trigger list ("triggers":[]) of the MPEG interactivity description module ("MPEG_interactivity":{}), and obtaining description information of a target trigger corresponding to the target trigger description module based on the target trigger description module.

[0370] Here, the target trigger description module is any trigger description module in the trigger list.

[0371] Illustratively, the target trigger description module can be shown as follows: [Table AG]

[0372] In some embodiments, obtaining description information of a target trigger corresponding to the target trigger description module based on the target trigger description module includes obtaining a type of the target trigger based on a value of a trigger type syntax element (type) in the target trigger description module.

[0373] For example, if the trigger type syntax element in the target trigger description module is "type":"TRIGGER_COLLISION", it can be determined that the type of the target trigger is collision.

[0374] In another example, if a trigger type syntax element in a target trigger description module is "type":"TRIGGER_VISIBILITY", the type of the target trigger may be determined to be visible.

[0375] In some embodiments, the method for analyzing the scene description file further includes obtaining a target media description module corresponding to the haptic media file from a media list ("media":[]) of an MPEG media description module ("MPEG_media":{}) of the scene description file, and obtaining description information of the haptic media file based on the target media description module.

[0376] In some embodiments, obtaining description information of the haptic media file based on the target media description module includes at least one of the following steps b1 to b4:

[0377] In step b1, obtain the name of the haptic media file according to the value of a media name syntax element (name) in the target media description module.

[0378] For example, if the media name syntax element in the target media description module and its value are "name":"Haptic example", it can be determined that the name of the haptic media file is Haptic example.

[0379] In step b2, it is determined whether the haptic media file needs to be autoplayed based on the value of an autoplay syntax element (autoplay) in the target media description module.

[0380] In some embodiments, determining whether the haptic media file needs to be autoplayed based on the value of an autoplay syntax element (autoplay) in the target media description module includes determining that the haptic media file needs to be autoplayed if the autoplay syntax element (autoplay) and its value in the target media description module is "autoplay":true or "autoplay":1, and determining that the haptic media file does not need to be autoplayed if the autoplay syntax element (autoplay) and its value in the target media description module is "autoplay":false or "autoplay":0.

[0381] In step b3, it is determined whether the haptic media file needs to be played in a circular manner based on the value of a loop syntax element (loop) in the target media description module.

[0382] In some embodiments, determining whether the haptic media file needs to be played in a circular fashion based on the value of a circular playback syntax element (loop) in the target media description module includes determining that the haptic media file needs to be played in a circular fashion if the circular playback syntax element (loop) in the target media description module and its value is "loop":true or "loop":1, and determining that the haptic media file does not need to be played in a circular fashion if the circular playback syntax element (loop) in the target media description module and its value is "loop":false or "loop":0.

[0383] In step b4, description information of each optional version of the haptic media file is obtained according to each optional version description module in the optional version list ("alternatives":[]) of the target media description module.

[0384] In some embodiments, the above step b4 (obtaining description information for each optional version of the haptic media file based on each optional version description module in the optional version list of the target media description module) may include at least one of the following steps b41-b44.

[0385] In step b41, data included in the target option version corresponding to the target option version description module is obtained based on the value of the media type syntax element (mime Type) in the target option version description module.

[0386] Here, the target option version description module is any option version description module in the option version list.

[0387] In some embodiments, obtaining data included in a target option version corresponding to a target option version description module based on a value of a media type syntax element (mime Type) in the target option version description module includes determining that the target option version includes haptic data and audio and / or video data if the value of the media type syntax element is a first preset value, and determining that the target option version includes haptic data if the value of the media type syntax element is a second preset value.

[0388] Illustratively, the first preset value may be video / mp4, and the second preset value may be haptic / mp4.

[0389] In step b42, obtain the uniform resource identifier (URI) of the target option version according to the value of the uniform resource identifier syntax element (uri) in the target option version description module.

[0390] For example, if the uniform resource identifier syntax element (uri) and its value in an optional version description module in the optional version list ("alternatives":[]) of the target media description module is "uri":"http: / / www.example.com / Hapticexample.mp4", it can be determined that the uniform resource identifier (access address) of the optional version of the haptic media file corresponding to the optional version description module is "http: / / www.example.com / Hapticexample.mp4".

[0391] In step b43, obtain the index value of each bitstream track of the target option version according to the value of the first track index syntax element (track) in the track array ("tracks":[]) of the target option version description module.

[0392] In step b44, the decoding parameters of the bitstream of the target option version are obtained based on the value of the codec parameter syntax element (codecs) in the track array ("tracks":[]) of the target option version description module and the provisions of the ISO / IEC 23090-32 haptic data transmission standard.

[0393] In some embodiments, Original VowThe scene description file analyzing method according to the embodiment further includes: obtaining a target scene description module corresponding to the three-dimensional scene to be rendered from a scene list (“scenes”:[]) of the scene description file; and obtaining description information of the three-dimensional scene to be rendered based on the target scene description module.

[0394] In some embodiments, obtaining description information of the three-dimensional scene to be rendered based on the target scene description module includes determining an index value of a node description module corresponding to each top-level node in the three-dimensional scene to be rendered based on the index values ​​stated in a node index list ("nodes":[]) of the target scene description module.

[0395] For example, if the node index list of the target scene description module and the index value of its statement are "nodes":[0], it can be determined that the 3D scene to be rendered contains only one top-level node, and the node description module corresponding to the top-level node is the first node description module in the node list of the scene description file.

[0396] As another example, if the node index list of the target scene description module and the index value of its statement are "nodes":[0,4], it can be determined that the 3D scene to be rendered includes two top-level nodes, and the node description modules corresponding to the two top-level nodes are the first node description module and the fifth node description module in the node list of the scene description file, respectively.

[0397] In some embodiments, Original VowThe scene description file analysis method according to the embodiment further includes: obtaining a target node description module from a node list ("nodes":[]) of the scene description file; and obtaining description information of a target node corresponding to the target node description module based on the target node description module.

[0398] Here, the target node description module is any node description module in the node list.

[0399] In some embodiments, obtaining description information of the target node corresponding to the target node description module based on the target node description module includes at least one of the following steps c1 to c4:

[0400] In step c1, obtain the name of the target node based on the value of the node name syntax element (name) in the target node description module.

[0401] For example, if the node name syntax element and its value in the target node description module are "name": "Haptic_node1", it can be obtained that the name of the target node is Haptic_node1 based on the value of the node name syntax element in the target node description module.

[0402] In step c2, the index values ​​of the node description modules corresponding to each subnode mounted on the target node are obtained based on the index values ​​stated in the subnode index list ("children":[]) of the target node description module.

[0403] For example, if the subnode index list in the target node description module and its declared index value are "children": "1, 2", it can be determined that two subnodes are mounted on the target node based on the index values ​​declared in the subnode index list of the target node description module, and that the two subnodes mounted on the target node are nodes corresponding to the second node description module and the third node description module in the node list of the scene description file, respectively.

[0404] In step c3, the index value of the mesh description module corresponding to each 3D mesh mounted on the target node is obtained based on the index value declared in the mesh index syntax element (mesh) of the target node description module.

[0405] For example, if the mesh index syntax element in the target node description module and its value are "mesh":0, it can be determined that the 3D mesh mounted on the target node is the 3D mesh corresponding to the first mesh description module in the mesh list ("meshes":[]) of the scene description file based on the index value declared in the mesh index syntax element of the target node description module.

[0406] In step c4, a spatial position offset of the target node relative to the parent node is obtained based on the value of a position offset syntax element (translation) in the target node description module.

[0407] For example, if the position offset syntax element in the target node description module and its value are "translation":"0.0,0.0,20.0", the spatial position offset [0.0,0.0,20.0] relative to the parent node of the target node can be obtained based on the value of the position offset syntax element in the target node description module.

[0408] It should be noted that what is directly obtained based on the value of the position offset syntax element in the target node description module is the spatial position offset of the target node relative to its parent node, and if the parent node of the target node does not represent a root node, this spatial position offset may not be the offset of the target node relative to the root node, and the offset of the target node relative to the root node is a superposition of the spatial offsets of each node on the path from the target node to the root node.

[0409] For example, if node A is a subnode of the root node, node B is a subnode of node A, the position offset syntax element in the node description module corresponding to node A is "translation":[0.0,10.0,25.0], and the position offset syntax element in the node description module corresponding to node B is "translation":[0.0,10.0,20.0], then first obtain the offset of node A relative to the target node as [0.0,10.0,25.0], then obtain the offset of node B relative to node A as [0.0,00.0,20.0], and then further combine the position offsets to obtain the offset of node B relative to the root node as [0.0,10.0,45.0].

[0410] In some embodiments, the method for analyzing the scene description file further includes: obtaining a target mesh description module from a mesh list ("meshes":[]) of the scene description file; and obtaining description information of a target 3D mesh corresponding to the target mesh description module based on the target mesh description module, where the target mesh description module is any mesh description module in the mesh list.

[0411] In some embodiments, obtaining, based on the target mesh description module, description information of the target 3D mesh corresponding to the target mesh description module includes at least one of the following steps d1-d5:

[0412] In step d1, the name of the target 3D mesh is obtained based on the value of the mesh name syntax element (name) in the target mesh description module.

[0413] For example, if the mesh name syntax element and its value in a mesh description module in a scene description file are "name":"Haptic_mesh1", the name of the 3D mesh corresponding to the mesh description module, Haptic_mesh1, can be obtained based on the value of the mesh name syntax element in the mesh description module.

[0414] In step d2, an index value of an accessor description module corresponding to an accessor for accessing the texture coordinates of the target 3D mesh is obtained based on the value of the texture coordinate syntax element (TEX COORD_n) in the attributes of the primitives of the target mesh description module.

[0415] For example, if the texture coordinate syntax element in the attributes of a primitive in a mesh description module in a scene description file and its value is "TEXCOORD_0":1, it can be determined that the accessor for accessing the texture coordinates of the 3D mesh corresponding to that mesh description module is the accessor corresponding to the second accessor description module in the accessories list of the scene description file.

[0416] In step d3, the type of the topological structure of the target 3D mesh is obtained based on the value of a mode syntax element (mode) in the primitives (primitives) of the target mesh description module.

[0417] For example, if the mode syntax element and its value in the primitive of a mesh description module in a scene description file are "mode":4, it can be obtained that the type of the topological structure of the 3D mesh corresponding to the mesh description module is a triangle face sheet based on the value of the mode syntax element in the primitive of the mesh description module.

[0418] In step d4, description information of the haptic material attributes to be mounted on the target 3D mesh is obtained based on the value of the haptic material array index syntax element (index) in the MPEG haptic description module ("MPEG_material_Haptic":{}) of the primitive of the target mesh description module.

[0419] In some embodiments, the above step d4 (obtaining description information of haptic material attributes to be mounted on the target 3D mesh based on the value of the haptic material array index syntax element in the MPEG haptic description module of the primitive of the target mesh description module) includes the following steps d41 to d43.

[0420] In step d41, the haptic material array associated with the target 3D mesh is obtained from the haptic material list ("MPEG_material_Happtic":[]) in the scene description file based on the value of the haptic material array index syntax element ("index").

[0421] For example, if the haptic material array index syntax element and its value are "index":0, the first haptic material array in the haptic material list of the scene description file is taken as the haptic material array associated with the target 3D mesh.

[0422] By way of example, the haptic material arrangement associated with the target 3D mesh can be shown as follows: [Table AH]

[0423] In step d42, a haptic material attribute description module corresponding to the haptic material attributes mounted on the target 3D mesh is obtained from the haptic material array associated with the target 3D mesh.

[0424] As described in the above example, tactile material attribute description modules corresponding to the stiffness, temperature, and vibration mounted on the target 3D mesh can be obtained from the tactile material array associated with the target 3D mesh, and the tactile material attribute description module corresponding to stiffness is the content of lines n+02 to n+08, the tactile material attribute description module corresponding to temperature is the content of lines n+09 to n+15, and the tactile material attribute description module corresponding to vibration is the content of lines n+16 to n+22.

[0425] In step d43, description information of the haptic material attributes mounted on the target 3D mesh is obtained based on the haptic material attribute description module corresponding to each haptic material attribute mounted on the target 3D mesh.

[0426] In some embodiments, the above step d43 (obtaining description information of tactile material attributes to be mounted on the target 3D mesh based on a tactile material attribute description module corresponding to each tactile material attribute to be mounted on the target 3D mesh) includes at least one of the following steps d431-d433.

[0427] In step d431, texture description information of the texture associated with the target tactile material corresponding to the target tactile material attribute description module is obtained based on the value of the texture index syntax element (index) in the texture array ("texture":{}) of the target tactile material attribute description module.

[0428] Here, the target haptic material attribute description module is any haptic material attribute description module in a haptic material array associated with the target three-dimensional mesh.

[0429] In some embodiments, the above step d431 (obtaining texture description information of a texture associated with a target haptic material corresponding to the target haptic material attribute description module based on the value of the texture index syntax element in the texture array of the target haptic material attribute description module) includes the following steps d4311 and d4312.

[0430] In Step d4311, the target texture description module is obtained from the texture list ("textures":[]) of the scene description file based on the value of the texture index syntax element.

[0431] For example, if the texture index syntax element and its value are "index":2, the third texture description module is obtained from the texture list ("textures":[]) of the scene description file as the target texture description module.

[0432] In Step d4312, texture description information of a texture associated with the target haptic material is obtained based on the target texture description module.

[0433] In some embodiments, the above step d4312 (obtaining texture description information of a texture associated with the target tactile material based on the target texture description module) includes at least one of the following steps e and f.

[0434] In step e, description information of a sampler of the target haptic material is obtained based on the value of a sampler index syntax element (sampler) in the target texture description module.

[0435] In some embodiments, the above step e (obtaining description information of a sampler of the target haptic material based on the value of a sampler index syntax element in the target texture description module) includes the following steps e1 and e2.

[0436] In step e1, a sampler description module corresponding to the sampler of the target haptic material is obtained from the sampler list (“samplers”:[]) of the scene description file based on the value of the sampler index syntax element (sampler) in the target texture description module.

[0437] Illustratively, a sampler description module corresponding to a sampler for the target haptic material can be shown as follows: [Table AI]

[0438] In step e2, description information of the sampler of the target haptic material is obtained based on the sampler description module corresponding to the sampler of the target haptic material.

[0439] As described in the above example, the description information of the target tactile material sampler that can be obtained by the sampler description module corresponding to the target tactile material sampler includes selecting the nearest point mode for texture map enlargement, selecting the Linear-mipmap-linear mode for texture map reduction, the horizontal texture covering method being 33648, and the vertical texture covering method being 33648.

[0440] In step f, description information of a texture resource corresponding to the texture resource of the target haptic material is obtained based on the value of a texture resource index syntax element (source) in the target texture description module.

[0441] In some embodiments, the above step f (obtaining description information of a texture resource corresponding to the texture resource of the target tactile material based on the value of a texture resource index syntax element in the target texture description module) includes the following steps f1 and f2.

[0442] In step f1, a texture resource description module corresponding to the texture resource of the target haptic material is obtained from the texture resource list (“images”:[]) of the scene description file based on the value of the texture resource index syntax element (source) in the target texture description module.

[0443] Exemplarily, a texture resource description module corresponding to the texture resource of the target haptic material can be shown as follows: [Table AJ]

[0444] In step f2, description information of the texture resource of the target haptic material is obtained based on a texture resource description module corresponding to the texture resource of the target haptic material.

[0445] As described in the above example, based on the value of the uniform address identifier syntax element (uri) in the texture resource description module corresponding to the texture resource of the target tactile material, it can be obtained that the access address of the texture resource of the target tactile material is stiffnessTexture.png.

[0446] In step d432, the index value of the texture coordinate syntax element (TEX COORD_n) associated with the target tactile material is obtained based on the value of the texture coordinate index syntax element (tex Coord) in the texture array of the target tactile material attribute description module.

[0447] In step d433, the type of the target haptic material attribute is obtained based on the value of the haptic material type syntax element (type) in the texture array of the target haptic material attribute description module.

[0448] As described in the above example, the rigid tactile material attribute type can be determined to be low resolution based on "type":Low_resolution on line n+07, the temperature tactile material attribute type can be determined to be reference based on "type":Reference on line n+14, and the vibration tactile material attribute type can be determined to be reference based on "type":Reference on line n+21.

[0449] In some embodiments, the method for parsing the scene description file further includes the steps of obtaining a target accessor description module from an accessories list ("accessors":[]) of the scene description file, and obtaining description information of a target accessor corresponding to the target accessor description module based on the target accessor description module, where the target accessor description module is any accessor description module in the accessories list ("accessors":[]) of the scene description file.

[0450] In some embodiments, obtaining, based on the target accessor description module, description information of the target accessor corresponding to the target accessor description module includes at least one of the following steps g1-g8:

[0451] In step g1, the data type of the data accessed by the target accessor is obtained based on the value of a data type syntax element (component Type) in the target accessor description module.

[0452] For example, if the data type syntax element and its value in a certain accessor description module is "componentType":5126, it can be determined that the data type of the data accessed by the accessor corresponding to that accessor description module is a 32-bit floating-point number (float).

[0453] In step g2, the type of the target accessor is determined based on the value of the accessor type syntax element (type) in the target accessor description module.

[0454] For example, if the value of the accessor type syntax element in a certain accessor description module is "type":"VEC3", it can be determined that the type of the accessor corresponding to that accessor description module is a three-dimensional vector.

[0455] In step g3, the quantity of data accessed by the target accessor is determined based on the value of the data quantity syntax element (count) in the target accessor description module.

[0456] For example, if the data quantity syntax element and its value in a certain accessor description module are "count":1828, it can be determined that the quantity of data accessed by the accessor corresponding to that accessor description module is 1828.

[0457] In step g4, an index value of a buffer slice description module corresponding to a buffer slice that buffers the data accessed by the target accessor is determined based on the value of a first buffer slice index syntax element (buffer View) in the target accessor description module.

[0458] For example, if the first buffer slice index syntax element in an accessor description module and its value is "bufferView":1, it can be determined that the data accessed by the accessor corresponding to that accessor description module is buffered in the buffer slice corresponding to the second buffer slice description module in the buffer slice list ("bufferViews":[]).

[0459] In step g5, it is determined whether the target accessor is a time-varying accessor based on MPEG extension modification based on whether the target accessor description module includes an MPEG time-varying accessor (MPEG_accessor_timed).

[0460] In some embodiments, determining whether the target accessor is a time-varying accessor based on an MPEG extension adaptation based on whether the target accessor description module includes an MPEG time-varying accessor ("MPEG_accessor_timed":{}) includes determining that the target accessor is a time-varying accessor based on an MPEG extension adaptation if the target accessor description module includes an MPEG time-varying accessor, and determining that the target accessor is not a time-varying accessor based on an MPEG extension adaptation if the target accessor description module does not include an MPEG time-varying accessor.

[0461] In step g6, a buffer slice description module corresponding to a buffer slice that buffers the time-varying parameters of the target accessor is determined based on the value of the second buffer slice index syntax element (buffer View) in the MPEG time-varying accessor (MPEG_accessor_timed) of the target accessor description module.

[0462] For example, if the second buffer slice index syntax element in an MPEG time-varying accessor of an accessor description module and its value is "bufferView":3, it can be determined that the time-varying parameters of the accessor corresponding to that accessor description module are buffered in the buffer slice corresponding to the fourth buffer slice description module in the buffer slice list ("bufferViews":[]).

[0463] In step g7, it is determined whether the value of the syntax element in the target accessor changes over time based on the value of the time-varying syntax element (immutable) in the MPEG time-varying accessor (MPEG_accessor_timed) of the target accessor description module.

[0464] In some embodiments, determining whether the value of a syntax element in a target accessor changes over time based on the value of a time-varying syntax element (immutable) in an MPEG time-varying accessor of a target accessor description module includes determining that the value of a syntax element in the target accessor does not change over time if the time-varying syntax element in the MPEG time-varying accessor of the target accessor description module and its value is "immutable":true or "immutable":1, and determining that the value of a syntax element in the target accessor changes over time if the time-varying syntax element in the MPEG time-varying accessor of the target accessor description module and its value is "immutable":false or "immutable":0.

[0465] In step g8, the name of the target accessor is determined based on the value of the accessor name syntax element (name) in the target accessor description module.

[0466] In some embodiments, the scene description file parsing method further includes: obtaining a target buffer description module from a buffer list (“buffers”:[]) of the scene description file; and obtaining description information of a target buffer corresponding to the target buffer description module based on the target buffer description module.

[0467] Here, the target buffer description module is any buffer description module obtained from the buffer list ("buffers":[]) of the scene description file.

[0468] In some embodiments, obtaining description information of the target buffer corresponding to the target buffer description module based on the target buffer description module includes at least one of the following steps h1-h7:

[0469] In step h1, the name of the target buffer is obtained based on the value of the buffer name syntax element (name) in the target buffer description module.

[0470] For example, if the buffer name syntax element in the target buffer description module and its value are "name":"haptic_buffer", it can be determined that the name of the target buffer is "name":"haptic_buffer".

[0471] In step h2, the capacity of the target buffer corresponding to the target buffer description module is obtained according to the value of the first byte length syntax element (byte Length) in the target buffer description module.

[0472] For example, if the first byte length syntax element in a buffer description module and its value is "byteLength":43972, it can be determined that the capacity of the buffer corresponding to the buffer description module is 43972 bytes.

[0473] In step h3, it is determined whether the target buffer is a ring buffer based on MPEG extension modification according to whether the target buffer description module includes an MPEG ring buffer (MPEG_buffer_circular).

[0474] In some embodiments, determining whether the target buffer is a ring buffer based on MPEG extension modification based on whether the target buffer description module includes an MPEG ring buffer includes determining that the target buffer is a ring buffer based on MPEG extension modification if the target buffer description module includes an MPEG ring buffer, and determining that the target buffer is not a ring buffer based on MPEG extension modification if the target buffer description module does not include an MPEG ring buffer.

[0475] In step h4, the number of storage segments of the target buffer is obtained based on the value of the segment count syntax element (count) in the MPEG ring buffer of the target buffer description module.

[0476] For example, if the Number of Segments in MPEG Ring Buffer syntax element of a buffer description module and its value is "count":3, it can be determined that the buffer corresponding to that buffer description module contains three storage segments.

[0477] In step h5, based on the value of the media index syntax element (media) in the MPEG ring buffer of the target buffer description module, obtain the index value of the media description module corresponding to the media file to which the source data of the data buffered in the target buffer belongs.

[0478] In step h6, obtain a track index value of the source data of the data buffered in the target buffer according to the value of the second track index syntax element (tracks) in the MPEG ring buffer of the target buffer description module.

[0479] In step h7, obtain a uniform resource identifier of the source data of the data buffered in the target buffer according to the value of a uniform resource identifier syntax element (uri) in the target buffer description module.

[0480] In some embodiments, the scene description file parsing method further includes: obtaining a target buffer slice description module from a buffer slice list (“bufferViews”:[]) of the scene description file; and obtaining description information of a target buffer slice corresponding to the target buffer slice description module based on the target buffer slice description module.

[0481] Here, the target buffer slice description module is any buffer slice description module in the buffer slice list.

[0482] In some embodiments, obtaining, based on the target buffer slice description module, description information of the target buffer slice corresponding to the target buffer slice description module includes at least one of the following steps j1-j3:

[0483] In step j1, obtain the capacity of the target buffer slice corresponding to the target buffer slice description module according to the value of a second byte length syntax element (byte Length) in the target buffer slice description module.

[0484] For example, if the second byte length syntax element in a buffer slice description module and its value is "byteLength":21936, it can be determined that the capacity of the buffer slice corresponding to the buffer slice description module is 21936 bytes.

[0485] In step j2, the offset of the data buffered in the target buffer slice is obtained based on the value of an offset syntax element (byte Offset) in the target buffer slice description module.

[0486] For example, if the offset syntax element in a buffer slice description module and its value are "byteOffset":0, it can be determined that the offset of the data buffered in the buffer slice corresponding to the buffer description module is 0.

[0487] In step j3, the name of the target buffer slice is obtained based on the value of a buffer slice name syntax element (name) in the target buffer slice description module.

[0488] In some embodiments, the method for parsing the scene description file further includes determining a version number of the scene description file based on a version syntax element (version) and its value in a digital asset description module (asset) in the scene description file.

[0489] For example, the digital assets of a scene description file are shown below: [Table AK]

[0490] In this case, it can be determined that the scene description file was created based on the glTF2.0 version, and that the version of the scene description file is the reference version of the scene description standard.

[0491] In some embodiments, the scene description file parsing method further comprises: obtaining extension items used by the scene description file based on an extension used list in the scene description file.

[0492] For example, the extension used list in the scene description file is shown below: [Table AL]

[0493] Therefore, the extension items used by the scene description file may include six: MPEG media (MPEG_media), MPEG interaction (MPEG_interactivity), MPEG haptics (MPEG_Haptic), MPEG haptic material (MPEG_material_Haptic), MPEG ring buffer (MPEG_buffer_circular), and MPEG time-varying accessor (MPEG_accessor_timed).

[0494] In some embodiments, the method of parsing the scene description file further comprises determining, based on a scene statement in the scene description file, a scene description module corresponding to the three-dimensional scene to be rendered.

[0495] For example, if the scene statement of the scene description file is "scene":0, it can be determined that the scene description module corresponding to the 3D scene to be rendered is the first scene description module in the scene list ("scenes":[]) of the scene description file.

[0496] Original Vow Some embodiments further provide a method for rendering a three-dimensional scene, where the execution entity of the three-dimensional scene rendering method is a display engine in an immersive media description framework, and as shown in FIG. 12, the three-dimensional scene rendering method includes the following steps S121-S124:

[0497] In S121, based on the MPEG interactive description module ("MPEG_interactivity":{}) of the scene description file of the 3D scene to be rendered, when the 3D scene to be rendered performs a target haptic action, an index value of the target haptic object description module corresponding to the haptic object associated with the target action node is obtained.

[0498] In some embodiments, the above step S121 (obtaining, based on the MPEG interactive description module of the scene description file of the 3D scene to be rendered, the index value of the target haptic object description module corresponding to the haptic object associated with the target action node when the 3D scene to be rendered performs the target haptic action) includes the following steps 1211-1213:

[0499] In step 1211, a target action description module is obtained from the MPEG interactivity description module ("MPEG_interactivity":{}) of the scene description file of the 3D scene to be rendered.

[0500] Here, the target action description module is an action description module corresponding to the target haptic action.

[0501] In step 1212, a target haptic action node array is obtained from the haptic action node list (“hapticActionNodes”:[]) of the action description module.

[0502] Here, the target haptic action node array is a haptic action node array in which the value of a node index syntax element (node) is the index value of a node description module corresponding to the target action node.

[0503] In step 1213, the value of the haptic object index syntax element in the target haptic action node array is determined as the index value of the target haptic object description module.

[0504] In step S122, the target haptic object description module is obtained from the MPEG haptic description module ("MPEG_haptic":{}) of the scene description file based on the index value of the target haptic object description module.

[0505] For example, if the index value of the target haptic object description module is 1, the second haptic object description module from the MPEG haptic description modules of the scene description file is read as the target haptic object description module.

[0506] In step S123, based on the target haptic object description module, description information of a haptic media accessor associated with the target action node when the three-dimensional scene to be rendered performs the target haptic action is obtained.

[0507] In some embodiments, the above step S123 (obtaining description information of a haptic media accessor associated with the target action node when the three-dimensional scene to be rendered performs the target haptic action based on the target haptic object description module) includes the following steps 1231 to 1233.

[0508] In step 1231, the value of the accessor index syntax element (accessor) in the target haptic object description module is obtained.

[0509] In step 1232, an accessor description module corresponding to the haptic media accessor is obtained from the accessories list ("accessors":[]) of the scene description file based on the value of the accessor index syntax element (accessor).

[0510] In step 1233, description information of the haptic media accessor is obtained based on an accessor description module corresponding to the haptic media accessor.

[0511] The implementation method for obtaining description information of the tactile media accessor based on the accessor description module corresponding to the tactile media accessor can refer to at least one of steps g1 to g8 in the above-mentioned embodiments, and will not be described in detail here to avoid duplication.

[0512] In step S124, when triggering the three-dimensional scene to be rendered to perform the target haptic action, haptic rendering data of the haptic media file declared in the MPEG media description module ("MPEG_media":{}) associated with the target action node is accessed from the haptic media buffer based on the description information of the haptic media accessor, and the target action node is rendered based on the haptic rendering data associated with the target action node.

[0513] Original Vow A three-dimensional scene rendering method according to an embodiment of the present invention can obtain, based on an MPEG interactive description module of a scene description file of the three-dimensional scene to be rendered, an index value of a target haptic object description module corresponding to a haptic object associated with a target action node when the three-dimensional scene to be rendered performs a target haptic action; obtain the target haptic object description module from the MPEG haptic description module of the scene description file based on the index value of the target haptic object description module; obtain description information of a haptic media accessor associated with the target action node when the three-dimensional scene to be rendered performs the target haptic action based on the target haptic object description module; access haptic rendering data of a haptic media file declared in the MPEG media description module associated with the target action node from a haptic media buffer based on the description information of the haptic media accessor when triggering the three-dimensional scene to be rendered to perform the target haptic action; and render the target action node based on the haptic rendering data of the haptic media file associated with the target action node. Original Vow The three-dimensional scene rendering method according to the embodiment of the present invention can trigger the three-dimensional scene to be rendered to perform the target haptic action, access haptic rendering data of a haptic media file associated with the action node, and render the action node according to the haptic rendering data of the haptic media file associated with the action node, and obtain the association relationship between the haptic action, the node, and the haptic rendering data of the haptic media file according to a scene description file: Original Vow An embodiment of the present invention can support haptic media files in a scene description file.

[0514] Since, when triggering the three-dimensional scene to be rendered and executing the target haptic action, the display engine needs to access haptic rendering data of the haptic media file associated with the target action node from a haptic media buffer based on the description information of the haptic media accessor, in some embodiments, the method for rendering the three-dimensional scene further includes performing the following steps (1)-(3) to obtain the haptic rendering data of the haptic media file associated with the get action node before accessing the haptic rendering data of the haptic media file from a haptic media buffer based on the description information of the haptic media accessor.

[0515] In step (1), a target media description module corresponding to the haptic media file is obtained from the media list ("media":[]) of the MPEG media description module ("MPEG_media":{}) of the scene description file.

[0516] In step (2), description information of the haptic media file is obtained based on the target media description module.

[0517] The implementation method of obtaining the description information of the haptic media file based on the target media description module can refer to at least one of steps b1 to b4 in the above embodiment, and to avoid redundancy, the description will not be repeated here.

[0518] In step (3), the description information of the haptic media file is sent to the media access function so that the media access function obtains the haptic media file based on the description information of the haptic media file, and the haptic rendering data of the haptic media file is obtained by unpackaging and decrypting the haptic media file.

[0519] In some embodiments, sending the description information of the haptic media file to a media access function includes sending the description information of the haptic media file to the media access function via a media access function Application Program Interface (API).

[0520] Since, when triggering the three-dimensional scene to be rendered and executing the target haptic action, the display engine needs to access haptic rendering data of the haptic media file associated with the target action node from a haptic media buffer based on the description information of the haptic media accessor, in some embodiments, the method for rendering a three-dimensional scene further includes performing the following steps (4) and (5) to write the haptic rendering data of the haptic media file to a haptic media buffer before accessing the haptic rendering data of the haptic media file associated with the target action node from the haptic media buffer based on the description information of the haptic media accessor.

[0521] In step (4), description information of the haptic media buffer and description information of buffer slices of the haptic media buffer are obtained based on the scene description file.

[0522] In some embodiments, obtaining description information of the haptic media buffer and description information of buffer slices of the haptic media buffer based on the scene description file includes obtaining a buffer description module corresponding to the haptic media buffer from a buffer list ("buffers":[]) of the scene description file, obtaining buffer slice description modules corresponding to buffer slices of the haptic media buffer from a buffer slice list ("bufferViews":[]) of the scene description file, and obtaining description information of the haptic media buffer and description information of buffer slices of the haptic media buffer based on the buffer description module corresponding to the haptic media buffer and the buffer slice description module corresponding to buffer slices of the haptic media buffer.

[0523] In step (5), the description information of the haptic media buffer, the description information of the buffer slices of the haptic media buffer, and the description information of the haptic media accessor are sent to a media access function so that the media access function writes haptic rendering data of the haptic media file to the haptic media buffer based on the description information of the haptic media buffer, the description information of the buffer slices of the haptic media buffer, and the description information of the haptic media accessor.

[0524] In some embodiments, sending the description information of the haptic media buffer, the description information of the buffer slices of the haptic media buffer, and the description information of the haptic media accessor to a media access function includes sending the description information of the haptic media buffer, the description information of the buffer slices of the haptic media buffer, and the description information of the haptic media accessor to a media access function via a media access function API.

[0525] Since, when triggering the three-dimensional scene to be rendered and executing the target haptic action, the display engine needs to access haptic rendering data of the haptic media file associated with the target action node from a haptic media buffer based on description information of the haptic media accessor, in some embodiments, the method for rendering the three-dimensional scene further includes performing the following steps (6) and (7) to create the haptic media buffer before accessing the haptic rendering data of the haptic media file associated with the target action node from the haptic media buffer based on description information of the haptic media accessor.

[0526] In step (6), description information of the haptic media buffer and description information of buffer slices of the haptic media buffer are obtained based on the scene description file.

[0527] In step (7), the description information of the haptic media buffer and the description information of the buffer slices of the haptic media buffer are sent to a buffer management module, so that the buffer management module creates the haptic media buffer based on the description information of the haptic media buffer and the description information of the buffer slices of the haptic media buffer.

[0528] In some embodiments, sending the description information of the haptic media buffer and the description information of buffer slices of the haptic media buffer to a buffer management module includes sending the description information of the haptic media buffer and the description information of buffer slices of the haptic media buffer to the buffer management module via a buffer API.

[0529] Original VowSome embodiments of the present invention further provide a method for processing a haptic media file, and the method for processing a haptic media file is performed by a media access function in an immersive media description framework, as shown in FIG. 13 . Processing haptic media files includes the following steps S131-S133.

[0530] In step S131, description information of the haptic media file declared in the MPEG media description module ("MPEG_media":{}) of the scene description file transmitted by the display engine is received.

[0531] In some embodiments, receiving description information for a haptic media file declared in an MPEG media description module of a scene description file transmitted from a display engine includes receiving, via a media access function API, description information for a haptic media file declared in an MPEG media description module of a scene description file transmitted from the display engine.

[0532] In step S132, haptic rendering data of the haptic media file is obtained based on the description information of the haptic media file.

[0533] In some embodiments, the above step S132 (obtaining haptic rendering data for the haptic media file based on the description information of the haptic media file) includes creating a haptic media pipeline for processing the haptic media file based on the description information of the haptic media file, obtaining the haptic media file through the haptic media pipeline, and depackaging and decrypting the haptic media file to obtain the haptic rendering data for the haptic media file.

[0534] In some embodiments, the description information of the haptic media file includes an access address of the haptic media file, and the above step S132 (obtaining haptic rendering data of the haptic media file based on the description information of the haptic media file) includes obtaining the haptic media file based on the access address of the haptic media file.

[0535] In some embodiments, obtaining the haptic media file based on an access address of the haptic media file includes obtaining the haptic media file from a media resource server based on an access address of the haptic media file.

[0536] In some embodiments, obtaining the haptic media file based on an access address of the haptic media file includes reading the haptic media file from a local storage space based on an access address of the haptic media file.

[0537] In some embodiments, the description information of the haptic media file further includes an index value of each bitstream track of the haptic media file, and the above step S132 (obtaining haptic rendering data of the haptic media file based on the description information of the haptic media file) further includes depackaging the haptic media file based on the index value of each bitstream track of the haptic media file to obtain a bitstream of each bitstream track of the haptic media file.

[0538] In some embodiments, the description information of the haptic media file further includes codec parameters of each bitstream track of the haptic media file, and the above step S132 (obtaining haptic rendering data of the haptic media file based on the description information of the haptic media file) further includes: decoding the bitstream of each bitstream track of the haptic media file based on the codec parameters of each bitstream track of the haptic media file, and obtaining corresponding decoded data of each bitstream track of the haptic media file to obtain the haptic rendering data of the haptic media file.

[0539] In step S133, when triggering the three-dimensional scene to be rendered to perform the target haptic action, the display engine writes the haptic rendering data of the haptic media file to the haptic media buffer, so as to access the haptic rendering data of the haptic media file associated with the target action node from the haptic media buffer based on the description information of the haptic media accessor.

[0540] Here, the description information of the haptic media accessor is accessor description information obtained by the display engine based on a target haptic object description module in the MPEG haptic description module ("MPEG_interactivity":{}) of the three-dimensional scene to be rendered, and the target haptic object description module is a haptic object description module corresponding to a haptic object associated with the target action node when the three-dimensional scene to be rendered performs the target haptic action.

[0541] Original VowA haptic media file processing method according to the embodiment can receive description information of a haptic media file declared in an MPEG media description module of a scene description file transmitted from a display engine, obtain haptic rendering data of the haptic media file based on the description information of the haptic media file, and write the haptic rendering data of the haptic media file to a haptic media buffer. Original Vow The haptic media file processing method according to the embodiment can obtain haptic rendering data of a haptic media file and write the haptic rendering data of the haptic media file to a haptic media buffer, so that a display engine can read the haptic rendering data of a haptic media file associated with a target action node from the haptic media buffer based on the description information of the haptic media accessor obtained based on the target haptic object description module in the MPEG haptic description module of the three-dimensional scene to be rendered, and when triggering the three-dimensional scene to be rendered to perform a target haptic action, can further read the haptic rendering data of the haptic media file associated with the target action node from the haptic media buffer based on the description information of the haptic media accessor, and render the target action node based on the haptic rendering data of the haptic media file associated with the target action node; Original Vow An embodiment of the present invention can support haptic media files in a scene description file.

[0542] Original Vow The embodiment further provides a buffer management method, and the execution body of the buffer management method is a buffer management module in an immersive media description framework. As shown in FIG. 14, the buffer management method includes the following steps S141 and S142:

[0543] In step S141, description information of a haptic media buffer and description information of buffer slices of the haptic media buffer are received.

[0544] Here, the haptic media buffer is a buffer for buffering haptic rendering data of a haptic media file declared in an MPEG media description module ("MPEG_media":{}).

[0545] In some embodiments, the above step S141 (receiving description information of a haptic media buffer and description information of buffer slices of the haptic media buffer) includes receiving description information of the haptic media buffer and description information of buffer slices of the haptic media buffer sent by a display engine.

[0546] That is, the display engine analyzes the scene description file to obtain description information of the haptic media buffer and description information of the buffer slices of the haptic media buffer, and then sends the description information of the haptic media buffer and description information of the buffer slices of the haptic media buffer directly to the buffer management module.

[0547] In some embodiments, receiving the description information of the haptic media buffer and the description information of buffer slices of the haptic media buffer sent by a display engine includes receiving the description information of the haptic media buffer and the description information of buffer slices of the haptic media buffer sent by a display engine via a buffer API.

[0548] In some embodiments, in some embodiments, the above step S141 (receiving description information of a haptic media buffer and description information of buffer slices of the haptic media buffer) includes receiving description information of the haptic media buffer and description information of buffer slices of the haptic media buffer sent by a media access function.

[0549] That is, the display engine analyzes the scene description file to obtain the description information of the haptic media buffer and the description information of the buffer slices of the haptic media buffer, and then first sends the description information of the haptic media buffer and the description information of the buffer slices of the haptic media buffer to a media access function, and then the media access function transfers the description information of the haptic media buffer and the description information of the buffer slices of the haptic media buffer to a buffer management module.

[0550] In some embodiments, receiving the description information of the haptic media buffer and the description information of buffer slices of the haptic media buffer sent by a media access function includes receiving the description information of the haptic media buffer and the description information of buffer slices of the haptic media buffer sent by a media access function via a buffer API.

[0551] In some embodiments, the description information of the haptic media buffer includes at least one of a name of the haptic media buffer, a capacity of the haptic media buffer, whether the haptic media buffer is a ring buffer based on MPEG extensions, the number of storage segments of the haptic media buffer, an index value of a media description module corresponding to a media file to which source data of data buffered in the haptic media buffer belongs, and a track index value of the source data of data buffered in the haptic media buffer.

[0552] In some embodiments, the description information of a buffer slice of the haptic media buffer includes at least one of a name of the buffer slice, an index value of a buffer description module corresponding to the haptic media buffer to which the buffer slice belongs, a capacity of the buffer slice, and an offset of the buffer slice.

[0553] In step S142, a media access function creates a haptic media buffer based on description information of the haptic media buffer and description information of a buffer slice of the haptic media buffer, so that the haptic rendering data of the haptic media file is written to the haptic media buffer, and when triggering a three-dimensional scene to be rendered to perform a target haptic action, the display engine accesses the haptic rendering data associated with the target action node from the haptic media buffer based on description information of the haptic media accessor.

[0554] Here, the description information of the haptic media accessor is accessor description information obtained by the display engine based on a target haptic object description module in an MPEG haptic description module of the three-dimensional scene to be rendered, and the target haptic object description module is a haptic object description module corresponding to a haptic object associated with the target action node when the three-dimensional scene to be rendered performs the target haptic action.

[0555] In some embodiments, Original Vow The buffer management method according to the embodiment further includes receiving description information of a target accessor, which is an accessor for accessing a dynamic three-dimensional model of the target component, sent from the display engine and / or the media access function, and buffer managing the target buffer based on the description information of the target accessor.

[0556] Here, the description information of the target accessor includes at least one of an index value of a buffer slice description module corresponding to a buffer slice that stores data accessed by the haptic media accessor, a data type of the data accessed by the haptic media accessor, a type of the haptic media accessor, the number of data accessed by the haptic media accessor, whether the haptic media accessor is a time-varying accessor based on MPEG extensions, an index value of a buffer slice description module corresponding to a buffer slice that stores time-varying parameters of the haptic media accessor, and whether the parameters of the haptic media accessor change over time.

[0557] In some embodiments, Original Vow The buffer management method according to the embodiment further includes receiving a buffer management command to instruct the target buffer to release or update data buffered in the target buffer, and releasing the target buffer or updating the data buffered in the target buffer in response to the buffer management command.

[0558] Original VowA buffer management method according to an embodiment of the present invention includes receiving description information of a haptic media buffer for buffering haptic rendering data of a haptic media file and description information of a buffer slice of the haptic media buffer, which are declared in an MPEG media description module, and then creating the haptic media buffer based on the description information of the haptic media buffer and the description information of the buffer slice of the haptic media buffer, so that a media access function can write the haptic rendering data of the haptic media file to the haptic media buffer; a display engine obtains description information of a haptic media accessor based on a target haptic object description module in the MPEG haptic description module of the three-dimensional scene to be rendered; and, when triggering the three-dimensional scene to be rendered to perform a target haptic action, accesses the haptic rendering data of the haptic media file associated with a target action node from the haptic media buffer based on the description information of the haptic media accessor, and renders the target action node based on the haptic rendering data of the haptic media file associated with the target action node; Original Vow An embodiment of the present invention can support haptic media files in a scene description file.

[0559] As 3D media content becomes increasingly richer, users have higher requirements for the realism and richness of virtual scenes. As part of the sensory experience, haptics is an object experience that needs to be considered and added to immersive media. For example, users can experience an explosion effect in a virtual scene, which not only requires the involvement of vision and hearing, but also the involvement of haptics as a sensory experience, enhancing the realism and presence of the virtual explosion effect.

[0560] In related technical standards, each haptic material attribute is represented by a texture array, and texture arrays of different attributes are combined to create a specific haptic sensation. Each array element includes a haptic texture and a haptic material attribute type represented as an enumeration, which includes high resolution, low resolution, reference, and other. When the haptic material attribute type is high resolution or low resolution, haptic values, rather than color values, are stored in the texture map or haptic texture material data packet of the texture index. These haptic values ​​can be directly retrieved and haptically rendered by the display engine. When the haptic material attribute type is reference, each pixel in the texture map or haptic texture material data packet stores an index value of the haptic media file to be rendered, rather than a directly stored haptic value. As mentioned above, when the haptic material attribute type is reference, the data obtained from the texture map or haptic texture material data packet is an index value of the haptic media, and based on the index value, only the encoded and packaged haptic media file can be obtained, but the decoded haptic value usable for rendering cannot be obtained, thereby causing the display engine to be unable to perform rendering successfully.

[0561] As shown in Figure 8, the MPEG #140 meeting proposed adding MPEG_haptic 801 and MPEG_material_haptic 802 to MPEG scene description files to associate haptic media files with objects in a scene and make haptics part of an immersive media experience. Haptic support can be enabled in scene descriptions based on a new version of the glTF haptic extension. Here, MPEG_haptic 801 obtains the access address of a haptic media file by indexing a media description module in the media list of MPEG_media. Note that the haptic media indexed from MPEG_media is the haptic media for which haptic encoding and packaging have been completed. MPEG_material stores haptic material attribute values ​​as supplementary descriptions of the material attributes of the original virtual object and supports direct indexing to haptic media files.

[0562] MPEG_haptic 801 is the next hierarchical description module after the node description module, and adds haptic information by supporting the indexing of media files associated with haptics in MPEG_media. The definitions of the syntax elements in MPEG_haptic 801 are shown in Table 27 below.

[0563] [Table 27]

[0564] MPEG haptic material (MPEG_material_haptic) 802 is a description module at the next level after the mesh description module, which supports users to obtain corresponding haptic feedback through active and / or passive interactions in the virtual scene.

[0565] As mentioned above, the MPEG #140 conference is currently proposing the addition of MPEG_haptic 801 and MPEG_material_haptic 802, which would allow indexing of MPEG_haptic 801 into haptic media files in MPEG_media. Many haptic media files describe an overall haptic experience associated with an event or environment, and the indexed haptic media file can be considered a haptic signal, essentially directly driving a haptic device to generate corresponding haptic feedback for the human body. In related technology, each array element includes one haptic texture and one haptic material attribute type represented as an enumeration, and the haptic material attribute types include high resolution, low resolution, reference, and other. When the haptic material attribute type is high resolution or low resolution, it is the haptic value, not the color value, that is stored in the texture map of the texture index or the haptic texture material data packet. A haptic value represents only the magnitude of a specific attribute (e.g., the magnitude of the friction coefficient, the magnitude of the stiffness coefficient) and can be directly obtained by a display engine for haptic rendering. However, when the haptic material attribute type is reference, each pixel in a texture map or haptic texture material data packet stores an index value of a haptic media file to be rendered rather than directly storing a haptic value. Therefore, when the haptic material attribute type is reference, only the encoded and packaged haptic media file can be obtained based on the index value, and a decoded haptic value usable for rendering cannot be obtained, preventing the display engine from rendering normally. In other words, the related art does not allow the haptic value of a reference-type haptic material attribute to be obtained.

[0566] Original VowSome embodiments of the present invention provide a scene description framework that supports obtaining tactile values ​​of reference-type tactile material attributes, and specific content includes support for obtaining tactile values ​​of reference-type tactile material attributes through a scene description file, support for obtaining tactile values ​​of reference-type tactile material attributes through a media access function API, support for obtaining tactile values ​​of reference-type tactile material attributes through a media access function, support for obtaining tactile values ​​of reference-type tactile material attributes through a buffer API, and support for obtaining tactile values ​​of reference-type tactile material attributes through buffer management.

[0567] The process of indexing, rendering, and displaying haptic media files in a 3D scene based on the scene description framework first involves the display engine acquiring a scene description file from a 3D scene provider by a method such as network protocol download or local storage read. The scene description file acquired by the display engine includes descriptive information for the entire 3D scene and the haptic media files in the scene, the descriptive information for the haptic media files including the access addresses of the haptic media files and access attribute information for the haptic media files (e.g., media type, codec parameters used, etc.). At the same time, the scene description file further includes a format request for data obtained by decoding the haptic media files. The format request for the data obtained by decoding the haptic media files further includes format requests (e.g., data type, data length, data access method, etc.) for information data such as a description portion and an encoding portion of the haptic data obtained by processing the haptic media files (i.e., description portion 41 and encoding portion 42 in FIG. 4, hereinafter collectively referred to as the haptic media file), as well as format requests and addresses for other media data, including data such as texture maps and description information of haptic material attributes that need to be used later. After completing the analysis of the scene description file, the display engine calls the media access function API to transmit information such as the access address of the haptic media file and attribute information of the haptic media file (media type, codec used, etc.) to the media access function. At the same time, the display engine also transmits a format request for the decoded haptic media data specified in the analyzed scene description file and a format request for other media data to the media access function by calling the media access function API.The media access function then requests the server to download the specified haptic media file based on the access address of the haptic media file transmitted from the display engine, or obtains the specified haptic media file from a local server. The media access function then establishes a corresponding pipeline for the media file based on information such as the media type and codec parameters, and converts the obtained media data from the package format into the format required by the scene description file. That is, the haptic media file should be converted into the format (data type, data length, access method, etc.) specified in the scene description file in the pipeline. The pipeline can perform operations such as depackaging, decoding, decoding, and post-processing on the media file. The pipeline then stores the decoded haptic media information data and other media data in a specified format processed as the scene description file in a specified buffer. The display engine can create the buffer by calling a buffer management module via a buffer API. If the display engine does not create the corresponding buffer, the media access function can create the corresponding buffer by calling a buffer management module via the buffer API. The purpose is to realize buffer creation, update, release, etc. Finally, the display engine reads the haptic media files in the format specified in the scene description file in the specified buffer to complete the rendering of the haptic portion of the 3D scene.

[0568] Below, we will explain the scene description file, media access function API, media access functions, buffer API, and buffer management that support obtaining haptic values ​​for reference-type haptic material attributes.

[0569] 1. Scene description file that supports retrieving tactile values ​​for reference-type tactile material attributes

[0570] As shown in Figure 15, Original Vow 15 is a structural schematic diagram of a scene description file supporting retrieval of haptic values ​​of reference-type haptic material attributes according to an embodiment of the present invention. As shown in Fig. 15, the scene description file supporting retrieval of haptic values ​​of reference-type haptic material attributes includes, but is not limited to, an MPEG media (MPEG_media) 2601, a scene description module (scene) 2602, a node description module (node) 2603, a mesh description module (mesh) 2604, an accessor description module (accessor) 2605, a buffer slice description module (bufferView) 2606, a buffer description module (buffer) 2607, a material description module 2608, an MPEG haptic material (MPEG_material_haptic) 2609, a texture description module (texture) 2610, an MPEG haptic (MPEG_haptic) 2611, a camera description module (camera) 2612, a skin description module (skin) 2613, and an animation description module (animation) 2614. Each description module of the scene description file shown in FIG. 15 will be explained below.

[0571] So that the scene description file can accurately describe the haptic media file, Original Vow Some embodiments of extend the values ​​of syntax elements in the MPEG media (MPEG_media) 2601 of the scene description file, and specific extensions include at least one of the following:

[0572] Extension 1: Extends the value of the media type syntax element (MPEG_media.media.alternatives.mimeType) to describe the media resource type of the media in the media list options (MPEG_media.media.alternatives) of the MPEG media in the scene description file. The extension of the media type syntax element (mime Type) includes adding values ​​associated with haptic media files to the media type syntax element (mime Type). For example, when the media list of MPEG media in a scene description file includes haptic media, if the value of the media type syntax element (MPEG_media.media.alternatives.mimeType) used to describe the media resource type of the media follows the relevant provisions of the ISO / IEC23090-32 standard, the value of "mimeType" is "video / mp4" when the media resource is a media resource that includes audio, video, and haptics, the value of "mimeType" is "video / mp4" when the media resource is a media resource that includes only audio and haptics, and the value of "mimeType" is "haptics / mp4" when the media resource is a media resource that includes only haptics.

[0573] Extension 2 extends the value of the track syntax element (MPEG_media.media.alternatives.tracks.tracks) to declare track information for media files in the optional track array (MPEG_media.media.alternatives.tracks) of the media list of MPEG media in a scene description file. The extension to the track syntax element (track) includes following the selection method for reference tracks in the ISO / IEC 23090-32 standard when an encoded haptic media file is referenced by a scene description file as one of the selectable tracks (MPEG_media.media.alternatives.tracks) of the media list of MPEG media. For example, if haptic media is packaged into one MIHS track by ISOBMFF, this MIHS track should be referenced in the MPEG media; if haptic media is packaged into multiple MIHS tracks by ISOBMFF, a subsequent supplement to ISO / IEC 23090-32 must be referenced.

[0574] Extension 3 extends the value of the codec parameter syntax element (MPEG_media.media.alternatives.tracks.codecs) to describe the codec type of the media data included in the bitstream track in the optional track array (MPEG_media.media.alternatives.tracks) of the media list of the MPEG media in the scene description file. IETF RFC6381 defines the codec parameters of the media files included in the bitstream track. When the codestream track contains different types of codecs (e.g., an AdaptationSet contains representations with different codecs), the codec parameter syntax element (codecs) can be expressed as a comma-separated codec value list. Therefore, extending the value of the codec parameter syntax element (codecs) includes specifying that if the media data is haptic data and the packaging rules follow the 23090-32 standard, the value of the codec parameter syntax element (codecs) should be set based on the specifications in the ISO / IEC 23090-32 Carriage of Happtics Data standard. For example, if haptic data is packaged in ISMBFF, the first group of elements in the codec parameter syntax element (codecs) is a code representing a codec in ISO / IEC 23090-31, and the parameter format of the codec parameter syntax element (codecs) is "codecs":"mih1.oo", where "mih1" is a description of one type of data format in which haptic data is packaged in ISOBMFF. This value is not unique and allows substitution with the name of a sample format that matches other ISOBMFF packages. Here, "oo" is an object type indicator defined on the "Object Types" page of the MP4 Registration Authority website.

[0575] That is, to ensure that the scene description file accurately describes the haptic media file, Original VowSome embodiments of extend the values ​​of syntax elements in MPEG media in a scene description file, and specific extensions include one or more of the extensions shown in Table 28 below.

[0576] [Table 28]

[0577] By executing at least one of the above extensions 1 to 3 on the values ​​of syntax elements in MPEG_media in the scene description file, the MPEG media in the scene description file can realize a description for a haptic media file.

[0578] The haptic media file package is based on the data structure shown in Figure 4. The haptic media file is permitted to have not only an information section but also a metadata placement section. Original Vow The embodiment does not limit the packaging method of the haptic media file, but is based on the ability to obtain the encoded and packaged haptic media file based on the access address of the haptic media file in the MPEG media.

[0579] In some embodiments, the description methods of the scene description module (scene) 2602 and the node description module (node) 2603 in the scene description file that support obtaining haptic values ​​for reference-type haptic material attributes include, when a haptic media file is included in the 3D scene, describing the overall structure of the 3D scene and the structural hierarchy and position of the haptic media file in the 3D scene using the scene and node description methods. If there is a node description module (node) that needs to index haptic media files in the 3D scene, it enables MPEG haptics (MPEG_haptic) and indexes the required haptic media files in MPEG haptics. The MPEG haptic (MPEG_haptic) 2611 of the node description module (node) of the scene description file shown in Figure 15 includes a media reference (Media_reference), and the media reference (Media_reference) includes a haptic media identification syntax element (haptic_id) 26111, the value (haptic_id) of the haptic media identification syntax element 26111, an accessor index syntax element (accessor) 26112, and the value of the accessor index syntax element (accessor). A description method of MPEG haptic (MPEG_haptic) 2611 in a scene description file that supports obtaining haptic values ​​of reference-type haptic material attributes includes using a haptic media identification syntax element (haptic_id) and its value in MPEG haptic (MPEG_haptic) 2611 to index a texture coordinate point into one encoded and packaged haptic media file, and using an accessor index syntax element (accessor) and its value in MPEG haptic (MPEG_haptic) 2611 to index a texture coordinate point into a group of haptic values ​​in the decoded data of the corresponding haptic media file.That is, if the type of haptic material attribute mounted by a certain 3D mesh is Reference, one haptic media identification syntax element (haptic_id) and one accessor index syntax element (accessor) must be extended in the media reference (Media_reference) of MPEG_haptic (MPEG_haptic) 2611 of the node description module corresponding to the node containing the 3D mesh, and the relevant supplementary explanation is provided by the value pair of the haptic media identification syntax element (haptic_id) and the accessor index syntax element (accessor). The related supplementary explanation specifically includes clarifying a media description module corresponding to a haptic media file whose texture coordinate points need to be indexed by a haptic media identification syntax element (accessor) 26112 and its value in the media reference (Media_reference) of MPEG haptic (MPEG_haptic) 2611, and clarifying an accessor description module corresponding to an accessor that can actually access the decoded data of a haptic media file whose texture coordinate points need to be indexed by an accessor index syntax element (accessor) 26111 and its value in the media reference (Media_reference) of MPEG haptic (MPEG_haptic) 2611. This function corresponds to descriptions in the buffer description module (buffer) and buffer slice description module (buffer view) in the scene description file, and jointly realizes direct indexing of the decoded data of a haptic media file by texture coordinate points.

[0580] Furthermore, each 3D scene is described using an independent scene description module. A 3D scene may contain many concrete objects, such as a virtual digital human, nearby 3D objects, a distant background image, and a virtual camera representing the user's viewpoint and viewing angle. The scene description file treats these concrete objects as nodes in the 3D scene, with each node representing one object or a set of objects, and the relationships between the nodes reflect the relationships between the components of the 3D scene. Each scene description file can describe one or more 3D scenes, and the 3D scenes are only parallel, not hierarchical, i.e., there are no "contains" or "contained" relationships between them. The relationships between nodes may be parallel or hierarchical, i.e., there are "contains" and "contained" relationships between nodes, allowing multiple concrete objects to be represented collectively. When one node is contained within another, the contained node is called a subnode, and the subnode is represented by replacing "node" with "children." Nodes and subnodes can be flexibly combined to form a hierarchical node structure.

[0581] In some embodiments, a description method of the mesh description module (mesh) 2604 in a scene description file that supports obtaining haptic values ​​for reference-type haptic material attributes includes describing the scene description file using a 3D mesh data format, i.e., using the mesh description module (mesh) 2604, for nodes representing 3D objects. The mesh description module includes multiple specific pieces of information, such as 3D coordinates and color, which all belong to syntax elements in the primitive attribute (mesh.primitives.attribute) of the mesh description module 2604. The most basic mesh description module includes the 3D coordinates of vertices, the connectivity between vertices, vertex color information, and the 3D coordinates requiring texture values. At the mesh description module level, it is necessary to enable the texture coordinate (Texcoord) of the 3D coordinate set of the desired texture value, and then the haptic value obtained from the texture is added to the corresponding 3D coordinate of the 3D object. Information such as the texture coordinate (Texcoord) and color data are all attributes (mesh.primitives.attribute) of the primitive in the mesh description module. Haptic media temporarily supports further definition of haptic materials by simply enabling MPEG haptic material (MPEG_material_haptic) 2609 and supplementing it as parallel content with the material description module (material) 2604, without the need to reference and define attributes such as the number of colors at the mesh description module level.

[0582] In some embodiments, a description method of a material description module (material) 2608 and a texture description module (texture) 2610 in a scene description file that supports obtaining tactile values ​​for reference-type tactile material attributes includes describing additional information about the surface of a three-dimensional object using the material description module (material) 2608 and the texture description module (texture) 2610. When describing a typical three-dimensional object, describing the geometric information of the three-dimensional object using only a mesh description module or simply defining the colors of vertices in a three-dimensional mesh is far insufficient to improve the realism of the virtual object, and more information needs to be added to the surface of the three-dimensional object. In typical three-dimensional modeling techniques, this process may be abbreviated as mapping, texturing, or the like. That is, a scene description file that supports obtaining tactile values ​​for reference-type tactile material attributes also uses a material description module (material) 2608 and a texture description module (texture) 2610 based on glTF 2.0. The material description module (material) 2608 and the texture description module (texture) 2610 have a collaborative relationship in which they jointly define the color and physical information of an object surface. The texture description module (texture) 2610 specifies a sampler description module (sampler) and a texture map description module (image). The sampler description module (sampler) defines how a texture map is mapped to an object surface, enabling specific texture adjustment and packaging. The texture map description module (image) identifies and indexes the texture map using a ULR.

[0583] In some embodiments, a method for describing MPEG haptic material (MPEG_material_haptic) 2609 in a scene description file that supports obtaining haptic values ​​for reference-type haptic material attributes includes further describing haptic material attributes using MPEG haptic material (MPEG_material_haptic) 2609. When a haptic media file is included in a three-dimensional scene, haptic material attributes need to be considered in the material properties. MPEG haptic material (MPEG_material_haptic) 2609 includes a haptic material attribute description module corresponding to each haptic material attribute (stiffness, friction, haptic texture, temperature, vibration, and custom), and the haptic material attribute description module includes an attribute type syntax element, and the value of the attribute type syntax element may be high resolution, low resolution, or reference. Objects in a three-dimensional scene that include haptic feedback may include different types of haptic material attributes. Different haptic material attributes on the same object can obtain haptic values ​​or haptic media that are felt by the user in different ways, thereby realizing the associated haptic media in the scene description file to obtain haptic feedback.

[0584] As can be seen from Table 16, the MPEG haptic material (MPEG_material_haptic) includes haptic material attribute description modules corresponding to multiple haptic material attributes (stiffness, friction, haptic texture, temperature, vibration, and custom), and the haptic material attribute description module corresponding to each haptic material attribute includes the type of the haptic material attribute, specifically including Low_resolution, High_resolution, Reference, and Other. If the haptic material texture type of a haptic material attribute is Low_resolution or High_resolution, this indicates that the haptic value at the texture coordinate point can be obtained from a texture map or other haptic material texture data packet. The value included at each pixel point in the texture map or haptic material texture data packet represents the haptic value. As shown in Tables 14 and 15, the bit depth of the haptic value for different haptic material texture types is different, for example, , high The bit depth of the haptic values ​​of haptic material attributes with a high resolution is 16 bits, and the bit depth of the attribute values ​​of haptic material attributes with a low resolution is 8 bits.

[0585] If the type of a haptic material attribute is Reference, it is required that the haptic value of the texture coordinate point or a set of texture coordinate points can be indexed to a set of temporal or spatial haptic values. In this case, the haptic media identification syntax element (haptic_id) and the accessor index syntax element (accessor) in the Media_reference of MPEG Haptic (MPEG_haptic) 2611 must be enabled and related supplementary explanations must be provided. The type of each haptic material attribute is included within MPEG Haptic Material (MPEG_material_haptic) 2609. If the type of a haptic material attribute is Reference, it is necessary to add the haptic media identification syntax element and the accessor index syntax element to the Media_reference, with the purpose of realizing that the texture coordinate point can index to a set of encoded and packaged haptic media files and an accessor for accessing the decoded data of the haptic media files. That is, the Media_reference of MPEG_haptic 2611 has one syntax element named "accessor" and one syntax element named "haptic_id," where "haptic_id" specifies the media description module of the haptic media file whose texture coordinate points need to be indexed, and "accessor" specifies the accessor description module corresponding to the accessor that can actually access the decoded data of the haptic media file whose texture coordinate points need to be indexed. This function corresponds to descriptions in the buffer description module (buffer) and buffer slice description module (buffer view) in the scene description file, and jointly realizes direct indexing of texture coordinate points to the haptic media file. The definitions of the syntax elements in the Media_reference of MPEG_haptic 2611 are shown in Table 29 below.

[0586] [Table 29]

[0587] In some embodiments, the description methods of the accessor description module (accessor) 2605, buffer slice description module (bufferView) 2606, and buffer description module (buffer) 2607 in a scene description file that support obtaining haptic values ​​of reference-type haptic material attributes include pointing to an index of a media description module corresponding to a haptic media file in MPEG media (MPEG_media) by a value of the media index syntax element (buffer.MPEG_buffer_circular.media) of the MPEG ring buffer in the buffer description module. That is, the haptic media file needs to be specified in the buffer description module (buffer), but instead of directly adding a uniform resource locator (URL) of the haptic media file to the buffer description module (buffer), the index value of the media description module corresponding to the corresponding haptic media file in MPEG media (MPEG_media) is declared via the media index syntax element (MPEG_buffer_circular.media) of the MPEG ring buffer in the buffer description module (buffer).

[0588] For example, if the media description module corresponding to a haptic media file in MPEG media (MPEG_media) is the third media description module in the media list of MPEG media (MPEG_media), the value of the media index syntax element (buffer.MPEG_buffer_circular.media) in the MPEG ring buffer of the buffer description module can be set to “2” to index the access address in the third media description module in MPEG media in the MPEG ring buffer of the buffer description module, and the haptic media file can be indexed by the (MPEG_buffer_circular) media index syntax element in the MPEG ring buffer of the buffer description module.

[0589] In some embodiments, the description methods of the accessor description module (accessor) 2605, buffer slice description module (bufferView) 2606, and buffer description module (buffer) 2607 in a scene description file that supports obtaining haptic values ​​for reference-type haptic material attributes include indicating track information when accessing a package file of a haptic media file using a track (buffer.MPEG_buffer_circular.tracks.track) in the MPEG ring buffer track array of the buffer description module. Specifically, if the MPEG ring buffer track array (buffer.MPEG_buffer_circular.tracks) in the buffer description module includes multiple tracks, the pipeline performs necessary processing on all referenced tracks to generate a data format required by the buffer description module. If a media file referenced in a track (buffer.MPEG_buffer_circular.tracks) in the MPEG ring buffer track array of the buffer description module has multiple options (alternatives), each option (alternative) should record the track selection status. If the track arrangement of the MPEG circular buffer (buffer.MPEG_buffer_circular.tracks) in the buffer description module is not defined, the pipeline will perform the necessary processing for all tracks in the MPEG media (MPEG_media) and generate the data format required by the buffer description module (buffer), i.e., guide how the buffer description module (buffer) correctly stores the data in the MPEG media (MPEG_media).

[0590] Table 30 shows the syntax elements included in the MPEG circular buffer (MPEG_buffer_circular).

[0591] [Table 30]

[0592] That is, a haptic media file is indexed in a buffer description module by setting the value of the media index syntax element (media) in Table 30 to the index value of a media description module corresponding to the corresponding haptic media file in MPEG media (MPEG_media) based on the value setting rule for the media index syntax element (media) in Table 30. A track of a haptic media file is indexed in a buffer description module by setting the value of the syntax element "tracks" in Table 30 to the track index containing the haptic media file in MPEG media based on the value setting rule for the syntax element "tracks" in Table 30.

[0593] In some embodiments, a method for describing a camera description module (camera) 2612 in a scene description file that supports obtaining haptic values ​​for reference-type haptic material attributes includes defining visual information associated with viewing, such as viewpoint, viewing angle, etc., of a node description module (node) using the camera description module (camera) 2612.

[0594] In some embodiments, a method of describing an animation description module (animation) 2613 of a scene description file that supports obtaining tactile values ​​for reference-type tactile material attributes includes defining an animation to be added to a node description module (node) by the animation description module (animation) 2613.

[0595] In some implementations, the animation description module (animation) can describe the animation to be added to the node description module (node) by one or more of position movement, angle rotation, and size scaling.

[0596] In some embodiments, the animation description module (animation) may indicate at least one of the start time, end time, and animation implementation method of the animation added to the node description module (node).

[0597] That is, in a scene description file that supports obtaining tactile values ​​of reference-type tactile material attributes, animation can be added to a node description module (node) that represents an object in a three-dimensional object. The animation description module (animation) describes the animation added to the node in three ways: position movement, angle rotation, and size scaling, and may also specify the start and end times of the animation and the implementation method of the animation.

[0598] In some embodiments, a description method of a skin description module (skin) 2614 of a scene description file that supports obtaining tactile values ​​for reference-type tactile material attributes includes defining movement and deformation relationships between meshes and corresponding bones in a node description module (node) by the skin description module (skin) 2614.

[0599] Based on the improvements and extensions of the Motion Picture Experts Group media (MPEG_media), scene, node, mesh, accessor, buffer slice, buffer view, buffer, skin, animation, camera, material, texture, sampler, and texture map (image) in the scene description file according to the above embodiment, the scene description file can accurately describe haptic media files.

[0600] By way of example, the scene description file shown in FIG. 15 is described below in combination with an example scene description file for haptic media in a single track package, including stiffness, temperature, and vibration haptic material attributes. [Table AM] TIFF2026501767000288.tif251167TIFF2026501767000289.tif251167TIFF20265017670 00290.tif251168TIFF2026501767000291.tif249168TIFF2026501767000292.tif111168

[0601] The pair of square brackets on lines 1 and 241 in the above example contains the main contents of the scene description file shown in Figure 15. A scene description file that supports obtaining haptic values ​​for reference-type haptic material attributes includes a digital asset description module (asset), an extension description module (extension Used), MPEG media (MPEG_media), a scene statement (scene), a scene list (scenes), a node list (nodes), a mesh list (meshes), a haptic material array list (MPEG_material_Haptic), a texture list (textures), a texture map list (images), a sampler list (samplers), an accessory list (accessors), a buffer slice list (bufferView), and a buffer list (buffers). Below, we will explain the contents of each description module and the information contained in each description module from an analytical perspective.

[0602] 1. Regarding the digital asset description module (asset), the digital asset description module is in lines 2-4. The "version":"2.0" in the third line of the digital asset description module indicates that the scene description file is based on the glTF2.0 version, which is also the reference version of the scene description standard. From an analysis perspective, the display engine can determine which parser to select to parse the scene description file based on the digital asset description module.

[0603] 2. Regarding the extension description module usage, the extension description module usage is in lines 6-12. The extension description module usage includes five extensions: MPEG media (MPEG_media), MPEG ring buffer (MPEG_buffer_circular), MPEG time-varying accessor (MPEG_accessor_timed), MPEG haptics (MPEG_Haptic), and MPEG haptic material (MPEG_material_Haptic). Therefore, it can be determined that the scene description file uses five MPEG extensions: MPEG media, MPEG ring buffer, MPEG time-varying accessor, MPEG haptics, and MPEG haptic material. From an analysis perspective, the display engine can know in advance based on the contents of the extension description module usage that the extension items related to subsequent analysis include MPEG media, MPEG ring buffer, MPEG time-varying accessor, MPEG haptics, and MPEG haptic material.

[0604] 3. MPEG Media (MPEG_media) The MPEG media is in lines 14-53. The MPEG media implements the description of the haptic media files included in the 3D scene. The media list (media) of the MPEG media contains two brackets (two media description modules, the media description module in lines 17-33 and the media description module in lines 34-50) indicating that two media files are described in the MPEG media of the scene description file. The media description module on lines 17-33 indicates the file type of the package file corresponding to the media file with "mimeType":"haptic / mp4" on line 23, the access address of the corresponding media file with "uri":"http: / www.example.com / Hapticexample.mp4" on line 24, the track information of the corresponding media file with "track":"track Index=1" on line 27, the decoding information of the corresponding media file with "codecs":"mih1.oo" on line 28, the name of the corresponding media file with "name":"Hapticexample1" on line 18, the name of the corresponding media file with "autoplay":true on line 19, and the loop playback with "loop":true on line 20.The media description module on lines 34-50 indicates the file type of the package file corresponding to the media file with "mimeType":"video / mp4" on line 40, the access address of the corresponding media file with "uri":"http: / www.example.com / Hapticexample.mp4" on line 41, the track information of the corresponding media file with "track":"track Index=1" on line 44, the decoding information of the corresponding media file with "codecs":"mih1.oo" on line 45, the name of the corresponding media file with "name":"Hapticexample2" on line 35, the name of the corresponding media file with "autoplay":true on line 36, and the loop with "loop":true on line 37. From an analysis perspective, by analyzing the MPEG media, the display engine can determine that two haptic media files exist in the 3D scene to be rendered, and know how to access and parse these two haptic media files.

[0605] 4. Regarding the scene statement (scene), the scene statement is on line 55. In theory, one scene description file can contain multiple 3D scenes, so the above scene description file first indicates by "scene":0 on line 55 that the scene described in the scene description file is the 3D scene corresponding to the first scene description module in the scene list, i.e., the scene described in the scene description file included in the square brackets on lines 57-63.

[0606] 5. Scene List (scenes): The scene list is on lines 57-63. The fact that the scene list contains only one bracket indicates that the scene description file contains only one scene description module (scene) and only one 3D scene. Within the brackets (scene description module), "nodes":[0] on lines 58-61 indicates that the 3D scene contains only one node with index 0. From an analytical perspective, the scene list clarifies that the entire scene description framework should select the first 3D scene in the scene list for subsequent processing and rendering, clarifying the overall structure of the 3D scene and pointing to a more detailed node description module (node) in the next layer.

[0607] 6. Regarding the node list (nodes), the node list is on lines 65-83. The node list contains only one bracket, indicating that the scene description file contains only one node description module, the 3D scene contains only one node, and that this node and the node in the scene description module with an index value of 0 are the same node, and the two are associated using the index method. In the brackets (node ​​description module) representing the node, "name":"Hapticexample_node" on line 67 indicates that the name of the node is "Hapticexample_node", and "mesh":"0" on line 68 indicates that the content mounted on the node is the first 3D mesh in the mesh list (meshes), which corresponds to the mesh description module in the next layer. At the same time, lines 69-81 of the node description module further include MPEG haptics (MPEG_Happtic), and the media reference list (Media_reference) of MPEG haptics includes two square brackets, indicating that the node description module includes two media reference arrays (the media reference array on lines 72-75 and the media reference array on lines 76-79). The media reference array on lines 72-75 indicates, via "haptic_id":0 on line 73, that the media file corresponding to the media reference array on lines 72-75 is the media file described by the first media description module in the media list of MPEG media, and via "accessor":1 on line 74, that the accessor for accessing the analysis data of the corresponding media file is the accessor described by the second accessor description module in the accessories list.The media reference array on lines 76-79 indicates, via "haptic_id":1 on line 77, that the media file corresponding to the media reference array on lines 76-79 is the media file described by the second media description module in the media list of the MPEG media, and via "accessor":2 on line 78, that the accessor for accessing the analysis data of the corresponding media file is the accessor described by the third accessor description module in the accessories list. From an analysis perspective, the node description module in the node list indicates that the content mounted on the node is a 3D mesh and that the index of the mesh description module of the 3D mesh is 0, and further indicates that the haptic media files that can be indexed from the MPEG media are the first media file and the second media file in the MPEG media, and at the same time, indicates that the accessor description modules of the accessor accessing the first media file and the accessor accessing the second media file are the accessors described in the second accessor description module and the third accessor description module in the accessories list, respectively.

[0608] 7. Regarding the mesh list (meshes), the mesh list is on lines 85-102, and contains only one square bracket, indicating that the scene description file contains only one mesh description module (mesh), that there is only one 3D mesh in the 3D scene, and that this 3D mesh and the 3D mesh with an index value of 0 in the node description module are the same mesh. In the square brackets (mesh description module) representing the mesh, "name":"Haptic example_mesh" on line 87 indicates that the name of the 3D mesh is "Haptic example_mesh", and this name is used only as an identification mark, and "primitives" on line 88 indicates that the 3D mesh has primitives, i.e., primitives. "attributes" on line 90, "mode" on line 93, and "MPEG_material_Haptic" on line 95 indicate that the primitive has three types of information: attribute, mode, and MPEG material haptic (MPEG_material_Haptic). "TEXCOORD_0":0 on line 91 indicates that the accessor description module corresponding to the texture coordinate is the first accessor description module in the accessory list. "mode":0 on line 93 indicates that the topology of the 3D mesh is a scattered structure. "MPEG_material_Happtic":{"index":0} on lines 95-97 indicates that the 3D mesh corresponding to the mesh description module mounts MPEG haptic material attributes and points to a more detailed haptic material array list (MPEG_material_Happtic) in the next layer. From an analytical perspective, the mesh list clarifies the actual data type and topology type of the 3D mesh.

[0609] 8. Regarding the haptic material array list (MPEG_material_Haptic), the haptic material array list is on lines 104-130. The haptic material array list (MPEG_material_Happtic) contains only one bracket, indicating that the scene description file contains only one haptic material array (the content contained in the brackets on lines 106-128). In the haptic material array, three types of haptic material attributes are defined on lines 107-113, 114-120, and 121-127, respectively: stiffness, temperature, and vibration. Each type of haptic material attribute contains two pieces of information: the texture coordinate index of the texture array. The description module corresponding to stiffness is on lines 107-113, and the "index" and its value on line 109 define that stiffness points to the first texture description module in the texture list. The "texCoord" on line 110 defines the coordinate reference for the tactile material attribute of stiffness, which texture maps the first texture coordinate set data packet as the tactile material attribute. The "type":Low_resolution on line 112 defines that the type of the tactile material attribute of stiffness is low resolution, and the length of the tactile value of the stiffness attribute obtained later is all 8 bits long. The description module corresponding to temperature is on lines 114-120, and the "index" and its value on line 116 defines that temperature points to the texture bicycleThe "type":Reference on line 119 defines that the type of the tactile material texture for temperature is Reference, and the description module for vibration is on lines 121-127. The "index" and its value on line 123 define that vibration points to the third texture description module (texture) in the texture description list, and the "type" and its value on line 107 define that the type of the tactile material texture for vibration is Reference. From an analytical perspective, the texture type and source of the tactile material attribute contained in the object can be determined based on the tactile material array list (MPEG_material_Happtic), and points to a more detailed texture description module in the next layer description module.

[0610] Regarding 9. Texture List (textures), the texture list is on lines 132-145. The texture list contains three brackets, indicating that the scene description file contains three texture description modules, each describing texture information for stiffness, temperature, and vibration. The texture description content for stiffness and temperature includes a sampler syntax element (sampler) and a texture map syntax element (source). The access addresses of the stiffness and temperature sampler description modules and texture maps can be determined based on the values ​​of the sampler syntax element (sampler) and texture map syntax element (source). The texture map source corresponds to the texture map list (images) at the next level.

[0611] Regarding 10. Texture Map List (images), the texture map list is on lines 147-157. The texture map list contains three brackets, each corresponding to the access address of the texture map referenced in the texture description modules for stiffness, temperature, and vibration. From {" uri":" stiffnessTexture.png"} on lines 148-150, we can determine that the access address of the stiffness texture map is stiffnessTexture.png. From {" uri":" temperatureTexture.png"} on lines 151-153, we can determine that th...

Claims

1. 1. A method for generating a scene description file, comprising: determining a haptic media accessor associated with a target action node when the three-dimensional scene to be rendered performs a target haptic action, the haptic media accessor being an accessor for accessing haptic rendering data of a haptic media file declared in a Moving Picture Experts Group MPEG Media Description Module associated with the target action node when the three-dimensional scene to be rendered performs the target haptic action; generating a target haptic object description module based on an index value of an accessor description module corresponding to the haptic media accessor, and adding the target haptic object description module to an MPEG haptic description module in a scene description file for the three-dimensional scene to be rendered; generating a target haptic action node array based on an index value of a node description module corresponding to the target action node and an index value of the target haptic object description module, and adding the target haptic action node array to a haptic action node list of a target action description module of an MPEG interactive description module of the scene description file, wherein the target action description module is an action description module corresponding to the target haptic action; How to generate a scene description file.

2. generating a target haptic object description module based on an index value of an accessor description module corresponding to the haptic media accessor, adding an accessor index syntax element to the target haptic object description module and setting the value of the accessor index syntax element to an index value of an accessor description module corresponding to the haptic media accessor. The method of claim 1.

3. generating a target haptic action node array based on index values ​​of a node description module corresponding to the target action node and index values ​​of the target haptic object description module; adding a node index syntax element to the target haptic action node array and setting the value of the node index syntax element to the index value of a node description module that corresponds to the target action node; adding a haptic object index syntax element to the target haptic action node array and setting the value of the haptic object index syntax element to the index value of the target haptic object description module. The method of claim 1.

4. The method further includes adding an action type syntax element to the target action description module and setting a value of the action type syntax element based on an action type of the target action. The method of claim 1.

5. The method further includes generating a target behavior sequence corresponding to the target behavior based on an index value of a trigger description module corresponding to a trigger associated with the target behavior and an index value of an action description module corresponding to an action associated with the target behavior, and adding the target behavior sequence to a behavior list of the MPEG interactive description module; wherein the target behavior is any behavior in the three-dimensional scene to be rendered, and generating a target behavior sequence corresponding to the target behavior based on an index value of a trigger description module corresponding to a trigger associated with the target behavior and an index value of an action description module corresponding to an action associated with the target behavior includes: adding a trigger index syntax element to the target behavior array and setting the value of the trigger index syntax element to an index value of a trigger description module corresponding to a trigger associated with the target behavior; adding an action index syntax element to the target behavior array and setting a value of the action index syntax element to an index value of an action description module corresponding to an action associated with the target behavior. The method of claim 1.

6. The method further includes generating a target trigger description module corresponding to the target trigger based on a type of the target trigger, and adding the target trigger description module to a trigger list of the MPEG interactive description module; wherein the target trigger is any trigger in the three-dimensional scene to be rendered, and generating a target trigger description module corresponding to the target trigger based on a type of the target trigger includes: adding a trigger type syntax element to the target trigger description module and setting a value of the trigger type syntax element based on the type of the target trigger. The method of claim 1.

7. The method further includes generating a target media description module corresponding to the haptic media file based on description information of the haptic media file, and adding the target media description module to a media list of an MPEG media description module of the scene description file; wherein generating a target media description module corresponding to the haptic media file based on description information of the haptic media file includes: adding a media name syntax element to the target media description module and setting a value of the media name syntax element based on a name of the haptic media file; adding an autoplay syntax element to the target media description module and setting a value of the autoplay syntax element based on whether the haptic media file needs to be autoplayed; adding a cyclic playback syntax element to the target media description module and setting a value of the cyclic playback syntax element based on whether the haptic media file needs to be cyclically played; adding an option version list to the target media description module; generating an optional version description module corresponding to each optional version of the haptic media file based on description information of each optional version of the haptic media file, and adding the optional version description module corresponding to each optional version of the haptic media file to the optional version list. The method of claim 1.

8. generating an optional version description module corresponding to each optional version of the haptic media file based on description information of each optional version of the haptic media file, a media type syntax element is added to a target option version description module corresponding to a target option version of the haptic media file, and a value of the media type syntax element is set based on data included in the target option version, the target option version being any option version of the target media file; adding a uniform resource identifier syntax element to the target option version description module and setting a value of the uniform resource identifier syntax element based on the uniform resource identifier of the target option version; adding a track array to the target option version description module; adding a first track index syntax element to the track array, the value of the first track index syntax element being set to an index value of each bitstream track of the target option version; a codec parameter syntax element is added to the track array, and a value of the codec parameter syntax element is set based on encoding parameters of data in each bitstream track of the target option version and the provisions of the ISO / IEC 23090-32 haptic data transmission standard. The method of claim 7.

9. setting a value of the media type syntax element based on data included in the target option version, If the haptic media file includes audio and / or video data, the value of the media type syntax element is set to a first preset value; If the haptic media file does not contain audio and video data, the value of the media type syntax element is set to a second preset value. The method of claim 8.

10. The method further includes generating a target scene description module corresponding to the three-dimensional scene to be rendered based on description information of the three-dimensional scene to be rendered, and adding the target scene description module to a scene list of the scene description file; Here, generating a target scene description module corresponding to the three-dimensional scene to be rendered based on the description information of the three-dimensional scene to be rendered includes adding a node index list to the target scene description module, and adding to the node index list index values ​​of node description modules corresponding to each top layer node in the three-dimensional scene to be rendered.

10. The method of any one of claims 1-9.

11. The method further includes: generating a target node description module corresponding to the target node based on description information of the target node; and adding the target node description module to a node list of the scene description file; wherein the target node is any node in the three-dimensional scene to be rendered, and generating a target node description module corresponding to the target node based on description information of the target node includes: adding a node name syntax element to the target node description module and setting a value of the node name syntax element based on the name of the target node; adding a subnode index list to the target node description module, and adding an index value of a node description module corresponding to each subnode mounted on the target node to the subnode index list; adding a mesh index syntax element to the target node description module and setting the value of the mesh index syntax element to an index value of a mesh description module corresponding to a three-dimensional mesh mounted on the target node; adding a position offset syntax element to the target node description module and setting a value of the position offset syntax element based on a spatial position offset of the target node relative to its parent node.

10. The method of any one of claims 1-9.

12. The method further includes: generating a target mesh description module corresponding to the target 3D mesh based on description information of the target 3D mesh; and adding the target mesh description module to a mesh list of the scene description file; wherein the target 3D mesh is an arbitrary 3D mesh in the 3D scene to be rendered, and generating a target mesh description module corresponding to the target 3D mesh based on description information of the target 3D mesh includes: adding a mesh name syntax element to the target mesh description module and setting a value of the mesh name syntax element based on a name of the target 3D mesh; adding a texture coordinate syntax element to an attribute of a primitive in the target mesh description module, and setting the texture coordinate syntax element to an index value of an accessor description module corresponding to an accessor for accessing texture coordinates of the target 3D mesh; adding pattern syntax elements to primitives of the target mesh description module and setting values ​​of the pattern syntax elements based on a topology type of the target 3D mesh; adding an MPEG haptic description module to a primitive of the target mesh description module; adding a haptic material array index syntax element to the MPEG haptic description module; and setting a value of the haptic material array index syntax element to an index value of a haptic material array associated with the target three-dimensional mesh.

10. The method of any one of claims 1-9.

13. The method comprises: generating a haptic material attribute description module corresponding to each haptic material attribute mounted on the target three-dimensional mesh based on description information of each haptic material attribute mounted on the target three-dimensional mesh; generating a haptic material array associated with the target three-dimensional mesh based on a haptic material attribute description module corresponding to each haptic material attribute to be mounted on the target three-dimensional mesh; adding a haptic material array associated with the target three-dimensional mesh to a haptic material list in the scene description file; Here, generating a haptic material attribute description module corresponding to each haptic material attribute mounted on the target three-dimensional mesh based on description information of each haptic material attribute mounted on the target three-dimensional mesh includes: Adding a texture array to a target haptic material attribute description module corresponding to a target haptic material attribute, the target haptic material attribute being any haptic material attribute mounted on the target three-dimensional mesh; adding a texture index syntax element to the texture array and setting the value of the texture index syntax element to an index value of the texture array associated with the target haptic material attribute; adding a texture coordinate index syntax element to the texture array and setting the value of the texture index syntax element to the index value of the texture coordinate syntax element associated with the target haptic material attribute; adding a haptic material type syntax element to the target haptic material attribute description module and setting a value of the haptic material type syntax element based on a type of the target haptic material attribute. The method of claim 12.

14. The method comprises: generating a texture array associated with the target haptic material attribute based on texture description information for the target haptic material attribute; adding a texture array associated with the target haptic material attribute to a texture list in the scene description file; wherein generating a texture array associated with the target haptic material attribute based on texture description information of the target haptic material attribute includes: adding a sampler index syntax element to a texture array associated with the target haptic material attribute and setting the value of the sampler index syntax element to an index value of a sampler description module corresponding to a sampler for the target haptic material attribute; adding a texture resource index syntax element to a texture array associated with the target haptic material attribute and setting the value of the texture resource index syntax element to an index value of a texture resource description module corresponding to a texture resource of the target haptic material attribute. The method of claim 13.

15. The method comprises: generating a sampler description module corresponding to the target haptic material attribute sampler based on description information of the target haptic material attribute sampler; adding a sampler description module corresponding to a sampler of the target haptic material attribute to a sampler list in the scene description file. The method of claim 14.

16. The method comprises: generating a texture resource description module corresponding to the texture resource of the target haptic material attribute based on an access address of the texture resource of the target haptic material attribute; adding a texture resource description module corresponding to the texture resource of the target haptic material attribute to a texture resource list of the scene description file. The method of claim 14.

17. The method further includes generating a target accessor description module corresponding to the target accessor based on description information of the target accessor, and adding the target accessor description module to an accessories list of the scene description file; Here, the target accessor is any accessor for realizing rendering of the three-dimensional scene to be rendered, and generating a target accessor description module corresponding to the target accessor based on description information of the target accessor includes: adding a data type syntax element to the target accessor description module and setting a value of the data type syntax element based on a data type of data accessed by the target accessor; adding an accessor type syntax element to the target accessor description module and setting a value of the accessor type syntax element based on the type of the target accessor; adding a data number syntax element to the target accessor description module and setting a value of the data number syntax element based on the number of data accessed by the target accessor; adding a first buffer slice index syntax element to the target accessor description module, wherein a value of the first buffer slice index syntax element is set based on an index value of a buffer slice description module corresponding to a buffer slice for buffering data to be accessed by the target accessor; adding an MPEG time-varying accessor to the extension list of said target accessor description module; adding a second buffer slice index syntax element to the MPEG time-varying accessor and setting a value of the second buffer slice index syntax element based on an index value of a buffer slice description module corresponding to a buffer slice for buffering the time-varying parameters of the target accessor; adding a time-varying syntax element to the MPEG time-varying accessor and setting the value of the time-varying syntax element based on whether the value of a syntax element in the target accessor changes over time; adding an accessor name syntax element to the target accessor description module and setting a value of the accessor name syntax element based on the name of the target accessor.

10. The method of any one of claims 1-9.

18. The method further includes generating a target buffer description module corresponding to the target buffer based on description information of the target buffer, and adding the target buffer description module to a buffer list of the scene description file; wherein the target buffer is any buffer for realizing rendering of the three-dimensional scene to be rendered, and generating a target buffer description module corresponding to the target buffer based on description information of the target buffer includes: adding a first byte length syntax element to the target buffer description module and setting a value of the first byte length syntax element based on a capacity of the target buffer; adding an MPEG ring buffer to a buffer description module corresponding to said target buffer; adding a number-of-segments syntax element to the MPEG ring buffer and setting a value of the number-of-segments syntax element based on the number of stored segments in the target buffer; adding a media index syntax element to the MPEG ring buffer and setting a value of the media index syntax element based on an index value of a media description module corresponding to a media file to which source data of the data buffered in the target buffer belongs; adding a track index syntax element to the MPEG ring buffer and setting a value of the track index syntax element based on a track index value of source data for data buffered in the target buffer; adding a buffer name syntax element to a buffer description module corresponding to the target buffer and configuring the buffer name syntax element based on the name of the target buffer; adding a uniform resource identifier syntax element to the target buffer description module and setting a value of the uniform resource identifier syntax element to a uniform resource identifier of source data of data buffered in the target buffer.

10. The method of any one of claims 1-9.

19. The method comprises: generating a target buffer slice description module corresponding to the target buffer slice based on description information of the target buffer slice, and adding the target buffer slice description module to a buffer slice list of the scene description file; wherein the target buffer slice is any one of buffer slices for realizing the rendering of the three-dimensional scene to be rendered, and generating a target buffer slice description module corresponding to the target buffer slice based on description information of the target buffer slice includes: adding a buffer index syntax element to the target buffer slice description module and setting a value of the buffer index syntax element based on an index value of a buffer description module corresponding to a buffer to which the target buffer slice belongs; adding a second byte length syntax element to the target buffer slice description module and setting a value of the second byte length syntax element based on a capacity of the target buffer slice; adding an offset syntax element to the target buffer slice description module and setting a value of the offset syntax element based on an offset of data buffered in the target buffer slice; adding a buffer slice name syntax element to the target buffer slice description module and setting a value of the buffer slice name syntax element based on a name of the target buffer slice.

10. The method of any one of claims 1-9.

20. A scene description file generation device, comprising: a memory configured to store a computer program; a processor configured to, when invoking a computer program, cause the scene description file generation device to implement the scene description file generation method of any one of claims 1 to 19. Scene description file generator.

Images