Method and apparatus for generating scene description files

Abstract

Some embodiments of the present application provide a method and apparatus for generating a scene description file in the video processing technical field, which includes determining a type of a media file in a 3D scene to be rendered, and if the type of the target media file in the 3D scene to be rendered is a geometry-based point cloud compressed G-PCC coded point cloud, generating a scene description file corresponding to the target media file based on description information of the target media file. Target Media Description Module and adding the target media description module to a media list of MPEG media in a scene description file of the three-dimensional scene to be rendered.

Description

Cross-reference to Related Applications

[0001] This application claims the priority of a Chinese patent application with application number 202310474240.4 filed on April 27, 2023, and the contents of these applications are incorporated herein by reference. This is the national phase of a PCT application filed on June 1, 2023, with application number PCT / CN2023 / 097873, and the said PCT application was filed on January 10, 2023, with application number 202310036790.8 Chinese patent applications, and This application claims the priority of a Chinese patent application with application number 202310474240.4 filed on April 27, 2023, and the contents of these applications are incorporated herein by reference.

Technical Field

[0002] Some embodiments of this application relate to the field of video processing technology, and particularly to a method and apparatus for generating a scene description file.

Background Art

[0003] A point cloud is a collection of a large number of three-dimensional points. Compression standards for point clouds mainly include geometry-based point cloud compression (G-PCC) and video-based point cloud compression (V-PCC).

[0004] With the development of immersive media and applications, the types of immersive media are increasing, and currently, the mainstream immersive media mainly include point clouds, 3D meshes, 6 DoF panoramic video, and MPEG Immersive Video (MIV). In 3D scenes, various types of immersive media often coexist simultaneously. This requires rendering engines to support multiple different types of immersive media codecs, and different types of rendering engines are generated depending on the types and number of codecs supported. The media types supported by rendering engines designed by different vendors differ, and in order to implement cross-platform description of 3D scenes composed of different types of media, the Moving Picture Experts Group (MPEG) initiated the creation of an MPEG scene description standard, with standard number ISO / IEC 23090-14. This standard mainly solves the cross-platform description problem in 3D scenes of MPEG media (including the codecs, MPEG file format, and MPEG transmission mechanism established by MPEG). While the extensions provided by the first edition of the ISO / IEC 23090-14 MPEG-I scene description standard met the critical needs of immersive scene description solutions, the current scene description standard does not support media files of the G-PCC encoded point cloud type. Since point clouds are an important three-dimensional media format and G-PCC is currently one of the mainstream point cloud compression algorithms, supporting media files of the G-PCC encoded point cloud type in a scene description framework has significant implications and value. [Overview of the project]

[0005] According to the first aspect, some embodiments of the present application are methods for generating scene description files, Determine the type of media file in the 3D scene to be rendered. If the type of target media file in the rendering target 3D scene is a G-PCC encoded point cloud based on geometric point cloud compression, then the target media file will be handled according to the description information of the target media file. Target media description module To generate, Add the target media description module to the media list of the MPEG media in the scene description file of the 3D scene to be rendered. This provides a method for generating a scene description file that includes this information.

[0006] According to a second aspect, some embodiments of the present application are scene description file generation devices, Memory configured to store computer programs, When a computer program is called, a processor configured to cause the scene description file generation device to implement the scene description file generation method described in the first embodiment, This provides a device for generating scene description files that include this functionality. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 shows a schematic diagram of an immersive media description framework based on several examples. [Figure 2] Figure 2 shows a schematic diagram of the scene description file structure in several examples. [Figure 3] Figure 3 shows a schematic diagram of the scene description file according to another embodiment of the present invention. [Figure 4] Figure 4 shows schematic diagrams of G-PCC encoders according to several embodiments. [Figure 5] Figure 5 shows a schematic diagram of the LOD partitioning process in several examples. [Figure 6] Figure 6 shows a schematic diagram of the lifting transformation process in several embodiments. [Figure 7] Figure 7 shows a schematic diagram of the RAHT conversion process according to several examples. [Figure 8] Figure 8 shows schematic diagrams of G-PCC decoders according to several embodiments. [Figure 9] Figure 9 shows a schematic diagram of the scene description file in another embodiment. [Figure 10] Figure 10 shows a schematic diagram of the scene description file in another embodiment. [Figure 11] Figure 11 shows schematic diagrams of pipelines corresponding to media files of the G-PCC encoded point cloud type in several examples. [Figure 12] Figure 12 shows a step flowchart of the method for generating a scene description file according to several examples. [Figure 13] Figure 13 shows a step flowchart of a scene description file analysis method according to several embodiments. [Figure 14] Figure 14 shows a step flowchart of a media file processing method according to several embodiments. [Figure 15] Figure 15 shows a step flowchart of a 3D scene rendering method in several examples. [Figure 16] Figure 16 shows a step flowchart of a buffer management method according to several embodiments. [Figure 17] Figure 17 shows interaction flowcharts for rendering 3D scenes in several examples. [Modes for carrying out the invention]

[0008] To further clarify the purpose and embodiments of this application, the exemplary embodiments of this application will be clearly and completely described below with reference to the drawings of the exemplary embodiments of this application, and it should be noted that the exemplary embodiments described are only a selection of embodiments of this application, not all embodiments.

[0009] Note that the schematic explanations of the terms in this application are for facilitating the understanding of the embodiments described below and do not limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0010] The terms "comprising" and "having" and any variations thereof are intended to cover but not exclusively include. For example, a product or device comprising a series of components is not necessarily limited to all the components explicitly listed, and may include other components not explicitly listed or specific to these products or devices.

[0011] The "some implementation manners", "some embodiments", etc. referred to in this specification indicate that the described implementation manners or embodiments may include specific features, structures, or characteristics, but each embodiment does not necessarily include this specific feature, structure, or characteristic. Also, such phrases do not necessarily refer to the same implementation manner. In addition, when a specific feature, structure, or characteristic is described in relation to one embodiment, it is considered within the knowledge scope of those skilled in the art that such feature, structure, or characteristic can be implemented in relation to other implementation manners (regardless of whether it is clearly described in this specification).

[0012] The specification contains numerous brackets, some of which contain English interpretations of the aforementioned terms, such as Media Access Function (MAF), Scene Description Documents, and Application Programming Interface (API), while others contain abbreviations used when the parameter is actually used in a computer program, code, or other context, such as scene, node, mesh, and accessor. Note that the above examples illustrate only some of the expressions used in this disclosure, and understanding more of the bracket content requires contextual interpretation.

[0013] Some embodiments of the present application relate to the scene description of immersive media. Referring to the scene description framework of immersive media shown in FIG. 1, in order for the display engine 11 to focus on media rendering, the scene description framework of immersive media decouples the access and processing of media files and the rendering of media files, and designs a media access function (MAF) 12 to be responsible for the access and processing functions of media files. At the same time, a media access function application programming interface (API) is designed, and command interaction is performed between the display engine 11 and the media access function 12 through the media access function API. The display engine 11 can send commands to the media access function 12 through the media access function API, and the media access function 12 can also request commands from the display engine 11 through the media access function API.

[0014] A typical workflow for an immersive media scene description framework is as follows: 1) The display engine 11 receives a scene description file (Scene Description) provided by the immersive media service provider. 1) Obtaining a Document; 2) The display engine 11 analyzes the scene description file and obtains parameters or information such as the media file's access address, media file attribute information (media type and codec parameters, etc.), and the format request for the processed media file, and calls the media access function API to transmit all or part of the information obtained by analyzing the scene description file to the media access function 12; 3) The media access function 12, based on the information transmitted from the display engine 11, requests the download of a specified media file from the media resource server or obtains a specified media file from locally, establishes a corresponding pipeline for the media file, and then performs processing such as decapsulation, decoding, and post-processing on the media file in the pipeline to convert the media file from the encapsulated format to the format specified by the display engine 11; 4) The pipeline stores the output data after all processing is complete in a specified buffer; 5) Finally, the display engine 11 reads the fully processed data from the specified buffer and renders the media file based on the data read from the buffer.

[0015] The following provides further details about the files and functional modules related to the immersive media scene description framework.

[0016] Scene description file

[0017] In the workflow of an immersive media scene description framework, scene description files are used to describe content such as the structure of a 3D scene (its features can be described using a 3D mesh), textures (e.g., texture mapping), animations (rotation, translation), and camera viewpoint position (rendering angle).

[0018] In related technical fields, GL Transmission Format 2.0 (glTF2.0) has been selected as a candidate format for scene description files and can meet the requirements of video experts - immersive (MPEG-I) and 6-degrees-of-freedom (6DoF) applications. For example, glTF2.0 is described in the Khronos Group's GL Transmission Format (glTF) Version 2.0, which can be found at github.com / KhronosGroup / glTF / tree / master / specification / 2.0#specifying-extensions. Referring to Figure 2, which is a schematic diagram of the scene description file structure in the glTF2.0 scene description standard (ISO / IEC12113). As shown in Figure 2, a scene description file in the glTF2.0 scene description standard includes, but is not limited to, a scene description module (scene) 201, a node description module (node) 202, a mesh description module (mesh) 203, an accessor description module (accessor) 204, a buffer slice description module (buffer View) 205, a buffer description module (buffer) 206, a camera description module (camera) 207, a light illumination description module (light) 208, a material description module (material) 209, a texture description module (texture) 210, a sampler description module (sampler) 211, and a texture mapping description module (image) 212, an animation description module (animation) 213, and a skin description module (skin) 214.

[0019] In the scene description file shown in Figure 2, the scene description module (scene) 201 is used to describe the 3D scenes contained in the scene description file. A single scene description file can contain any number of 3D scenes, and each 3D scene is represented by one scene description module 201. There is a parallel relationship between scene description modules 201 and scene description modules 201, that is, there is a parallel relationship between 3D scenes and 3D scenes.

[0020] In the scene description file shown in Figure 2, node description module (node) 202 is the next hierarchical description module after scene description module 201 and is used to describe the objects contained in the 3D scene described by scene description module 201. Each 3D scene may contain many specific objects, such as a virtual digital human, nearby 3D objects, and distant background images, and the scene description file describes these specific objects using node description modules 202. Each node description module 202 can represent one object or a set of objects consisting of multiple objects, and the relationships between node description modules 202 reflect the relationships between each component in the 3D scene described by scene description module 201, and a scene described by one scene description module 201 may contain one or more nodes. The relationships between multiple nodes may be parallel or hierarchical, that is, there may be "container" and "contained" relationships between node description modules 202, which allows for the description of multiple specific objects together or for the description of multiple specific objects individually. When one node is contained within another node, the contained node is called a subnode (children), and the subnode is represented by replacing "node" with "children". By flexibly combining nodes and subnodes, a hierarchical node structure can be constructed, thereby expressing a rich range of scene content.

[0021] In the scene description file shown in Figure 2, the mesh description module (mesh) 203 is the next hierarchical description module after the node description module 202 and is used to describe the features of the object represented by the node description module 202. The mesh description module 203 is a collection of one or more primitives, each primitive may contain one attribute, the primitive's attribute defining the attribute that the graphics processing unit (GPU) needs to use when rendering. The attribute may include position (3D coordinates), normal (normal vector), tangent (tangent vector), texcoord_n (texture coordinates), color_n (color: RGB or RGBA), joints_n (attributes related to the skin description module 214), and weights_n (attributes related to the skin description module 214). Because the mesh description module 203 contains a very large number of vertices, and each vertex contains various attribute information, it is inconvenient to directly store the large amount of media data contained in the media file in the mesh description module 203 of the scene description file. Therefore, the scene description file indicates the access address (Uniform Resource Identifier, URI) of the media file, and data in the media file can be downloaded when necessary, thus achieving separation of the scene description file and the media file. Accordingly, generally, the mesh description module 203 does not store media data, but stores the index value of the accessor description module 204 corresponding to each attribute, and the accessor description module 204 points to the corresponding data in the buffer slice (buffer view) of the buffer.

[0022] In some implementations, the number and types of files can be reduced by merging the scene description file and media files into a single binary file.

[0023] Additionally, the primitives in the mesh description module 203 may have a syntax element called "mode." The mode syntax element is used to describe the topological structure when a graphics processing unit (GPU) renders a 3D mesh. For example, mode=0 represents scattered points, mode=1 represents a line, and mode=4 represents a triangle.

[0024] For example, the following is an example of JSON for mesh description module 203. JPEG0007872437000001.jpg85164

[0025] In the mesh description module 203 described above, the value of "position" is 1, which refers to the accessor description module 204 with index 1, and ultimately refers to the vertex coordinate data stored in the buffer. The value of "color_0" is 2, which refers to the accessor description module 204 with index 2, and ultimately refers to the color data stored in the buffer.

[0026] The definitions of syntax elements in the primitive attributes (mash.primitives.attributes) of the mesh description module 203 are shown in Table 1 below. [Table 1]

[0027] The definitions of the types of accessors indexed in the primitive attributes (mash.primitives.attributes) of the mesh description module 203 are shown in Table 2 below. [Table 2]

[0028] The definitions of data types in the primitive attributes (mash.primitives.attributes) of the mesh description module 203 are shown in Table 3 below. [Table 3]

[0029] In the scene description file shown in Figure 2, the accessor description module 204, buffer slice description module 205, and buffer description module 206 all implement sophisticated indexing for each layer of media file data by the mesh description module 203. As described above, the mesh description module 203 does not store specific media data, but stores the index value of the corresponding accessor description module 204, and accesses the specific media data through the accessor described by the accessor description module 204 indexed by the index value. The media data indexing process by the mesh description module 203 includes, first, that the index value declared in the syntax element of the mesh description module 203 points to the corresponding accessor description module 204, then that the accessor description module 204 points to the corresponding buffer slice description module 205, and finally that the buffer slice description module 205 points to the corresponding buffer description module 206. In the scene description file shown in Figure 2, the buffer description module 206 primarily serves to point to the corresponding media file and includes information such as the media file's URI and byte length. It is used to describe a buffer that buffers the media data of the media file. A single buffer may be divided into one or more buffer slices. The buffer slice description module 205 primarily performs partial access to the media data in the buffer and includes information such as the start byte offset and byte length of the accessed data. Partial access to the data of the media file can be achieved through this buffer slice description module 205 and the buffer description module 206. The accessor description module 204 primarily serves to add additional information to a portion of the data defined in the buffer slice description module 205, such as the data type, the number of data of that type, and the numerical range of data of that type.This three-tiered structure enables the retrieval of partial data from a single media file, which is advantageous for accurate data acquisition and also reduces the number of media files.

[0030] In the scene description file shown in Figure 2, the camera description module (camera) 207 is the next-level description module after the node description module 202, and is used to describe visual information such as the viewpoint and viewing angle when the user views an object described by the node description module 202. In order for the user to be placed in and able to view the 3D scene, the node description module 202 points to the camera description module 207, and the camera description module 207 can also describe visual information such as the viewpoint and viewing angle when the user views an object described by the node description module 202.

[0031] In the scene description file shown in Figure 2, the light illumination description module (light) 208 is the next-level description module after the node description module 202, and is used to describe information about light illumination, such as the light illumination intensity, ambient light color, light illumination direction, and light source position of the object described by the node description module 202.

[0032] In the scene description file shown in Figure 2, the material description module (material) 209 is the next-level description module after the mesh description module 203 and is used to describe the material information of the 3D object described by the mesh description module 203. When describing a 3D object, simply describing the geometric information of the 3D object by the mesh description module 203, or monotonically defining the color and / or position of the 3D object, is insufficient to improve the realism of the 3D object; more information needs to be added to the surface of the 3D object. For 3D modeling techniques such as 3D mesh models, this process may be abbreviated as texture mapping or texture addition. The scene description file in the glTF2.0 scene description standard also utilizes this description module. The material description module 209 defines a material using a set of common parameters and describes the material information of geometric objects that appear in the 3D scene. The material description module 209 generally describes the material of a virtual object using a metallic-roughness model, and the material property parameters based on the metallic-roughness model are represented by materials in the widely used physically based rendering (PBR) system. Based on this, the material description module 209 describes the metallic-roughness material attributes of an object in detail, and the definitions of the syntax elements in the material description module 209 are shown in Table 4. [Table 4]

[0033] In some examples, the definitions of the syntax elements in material.PbrMetarialRoughness of material description module 209 are shown in Table 5 below. [Table 5]

[0034] The values ​​of each attribute in the Metal-Roughness section of Material Description Module 209 may be defined using coefficients and / or textures (e.g., baseColorTexture and baseColorFactor). If no texture is provided, the values ​​of all corresponding texture components in this material model can be determined to be 1.0. If both coefficients and textures are present, the coefficient value becomes a linear multiplier of the corresponding texture value. Texture binding is defined by the index of the texture object and the selectable texture coordinate index.

[0035] As an example, the following is a JSON example of material description module 209. JPEG0007872437000007.jpg59164

[0036] By analyzing the above material description module 209, it can be determined that the current material is named "gold" based on the material name syntax element and its value ("name": "gold"), further determining that the base color value of the current material is [1.000, 0.766, 0.336, 1.0] based on the color syntax element and its value in the pbrMetallicRoughness array ("basecolorFactor": [1.000, 0.766, 0.336, 1.0]), determining that the metallicity value of the current material is "1.0" based on the metallicity syntax element and its value in the pbrMetallicRoughness array ("metalnessFactor": 1.0), and determining that the roughness value of the current material is "0.0" based on the roughness syntax element and its value in the pbrMetalRoughness array ("roughnessFactor": 0.0).

[0037] In the scene description file shown in Figure 2, the texture description module (texture) 210 is the next-level description module after the material description module 209 and is used to describe the color of the 3D object described by the material description module 209 and other properties used in the material definition. Textures are one of the important aspects that give an object a realistic appearance. By defining the main color of an object and other properties used in the material definition through textures, the appearance of the rendered object can be accurately described. A material itself can define multiple texture objects, and these texture objects can be used as textures for virtual objects during rendering and can be used to encode different material attributes. The texture description module 210 references one sampler description module (sampler) 211 and one texture mapping description module (image) 212 using sampler syntax elements and texture mapping syntax element indices. The texture mapping description module 212 contains a Uniform Resource Identifier (URI) linked to the texture map or binary file capsule actually used by the texture description module 210. The sampler description module 211 is a filtering and encapsulation mode for describing textures. The roles and cooperation of the Material Description Module 209, Texture Description Module 210, Sampler Description Module 211, and Texture Mapping Description Module 212 are as follows: Material Description Module 209 and Texture Description Module 210 together define the color and physical information of the object surface. Sampler Description Module 211 defines how texture mapping is applied to the object surface.The texture description module 210 specifies the sampler description module 211 and the texture mapping description module 212. The texture mapping description module 212 adds textures, uses URIs for identification and indexing, and accesses the data using the accessor description module 204. The sampler description module 211 implements the specific adjustment and encapsulation of textures. The definitions of the syntax elements in the texture description module 210 are shown in Table 6 below. [Table 6]

[0038] In some embodiments, the definitions of syntax elements in the sample(texture.sample) of the texture description module 210 are shown in Table 7 below. [Table 7]

[0039] As an example, the following is a JSON example of a single material description module 209, texture description module 210, sampler description module 211, and texture mapping description module 212. JPEG0007872437000010.jpg162166

[0040] In the scene description file shown in Figure 2, the animation description module (animation) 213 is the next-level description module after the node description module 202, and is used to describe animation information added to the object described by the node description module 202. To ensure that the object represented by the node description module 202 is not limited to a stationary state, animation can be added to the object described by the node description module 202. Therefore, the description hierarchy of animation description module 213 in the scene description file is specified by the node description module 202; that is, animation description module 213 is the next-level description module after the node description module 202, and animation description module 213 similarly corresponds to mesh description module 203. Animation description module 213 can describe animation in three ways: position movement, angle rotation, and size scaling, and can also define the start and end times of the animation and the method of animation implementation. For example, by adding an animation to a mesh description module 203 that represents a 3D object, the 3D object represented by the mesh description module 203 can complete a predetermined animation process within a specified time window by combining positional movement, angular rotation, and scaling.

[0041] In the scene description file shown in Figure 2, the skin description module (skin) 214 is the next-level description module after the node description module 202. It is used to describe the motion coordination relationship between the skeleton added to the node described by the node description module 202 and the mesh representing the object's surface information. When the node described by the node description module 202 represents an object with a large degree of freedom of movement, such as a person, animal, or machine, a skeleton can be filled inside the object to optimize the motion representation effect of these objects. In this case, the 3D mesh representing the object's surface information conceptually becomes the skin. The description hierarchy of the skin description module 214 is specified by the node description module 202; that is, the skin description module 214 is the next-level description module after the node description module 202, and there is a correspondence between the skin description module 214 and the mesh description module 203. By moving the mesh on the object's surface in conjunction with the movement of the skeleton, and further combining this with the design of the simulation bionic, relatively realistic motion effects can be achieved. For example, when a person's hand makes a fist, the surface skin stretches along with the internal skeleton, causing changes such as occlusion. At this time, by redefining the cooperative relationship between the skeleton and the skin for the skeleton pre-filled in the hand model, a realistic simulation of this motion can be achieved.

[0042] Each description module in the glTF 2.0 scene description standard has only the most basic ability to describe 3D objects and has problems such as not being able to support dynamic 3D immersive media, audio files, or scene updates. It is also stated that glTF has one selectable extension object attribute for each of its object attributes, and more complete functionality can be achieved by extending any part of it using extensions. Scene description modules (scene), node description modules (node), mesh description modules (mesh), accessor description modules (accessor), buffer description modules (buffer), animation description modules (animation), etc., and including the syntax elements defined within them, all have selectable extension object attributes to support certain functional extensions based on glTF 2.0.

[0043] Currently, rendering engines designed by different vendors support different media types. To achieve cross-platform description of 3D scenes composed of different types of media, the Moving Picture Experts Group (MPEG) has begun developing an MPEG scene description standard, ISO / IEC 23090-14. This standard primarily addresses the cross-platform description problem in 3D scenes of MPEG media (including codecs created by MPEG, MPEG file formats, and MPEG transmission mechanisms).

[0044] The MPEG #128 meeting resolution stipulates that the MPEG-I Scene Description standard will be created based on glTF2.0 (ISO / IEC 12113). Currently, the first version of the MPEG Scene Description standard has been created and is in the FDIS voting stage. The MPEG Scene Description standard will address requirements not yet realized in 3D scene cross-platform descriptions, including interactivity, AR anchors, user and avatar representation, haptic support, and support for immersive media codecs, by adding corresponding extensions to the first version of the standard.

[0045] The first version of the MPEG scene description standard created will primarily include the following:

[0046] The MPEG Scene Description standard defines a scene description file format for describing immersive 3D scenes. This format combines the contents of the original glTF 2.0 (ISO / IEC 12113) and makes a series of extensions based on it.

[0047] MPEG scene description defines a scene description framework and an Application Program Interface (API) for inter-module communication within it. This enables decoupling of the immersive media acquisition and processing process with the media rendering process, which is beneficial for optimizing aspects such as adapting immersive media to different network conditions, acquiring partial immersive media files, accessing different detail levels of immersive media, and adjusting content quality. Decoupling between the immersive media acquisition and processing process and the immersive media rendering process is a key point in achieving cross-platform description of 3D scenes.

[0048] In MPEG scene descriptions, extensions based on a series of International Standardization Organization Base Media File Formats (ISOBMFF) (ISO / IEC 14496-12) have been proposed for use in the transmission of immersive media content.

[0049] As shown in Figure 3, the scene description file is extended in the MPEG scene description standard based on the scene description file shown in Figure 2, and the extension of the scene description file in the MPEG scene description standard can be divided into two groups compared to the scene description file in the glTF2.0 scene description standard (the scene description file shown in Figure 2).

[0050] The first group of extensions includes MPEG media (MPEG_media) 301, MPEG timed accessor (MPEG_accessor_timed) 302, and MPEG ring buffer (MPEG_buffer_circular) 303. Here, MPEG media 301 is a standalone extension for referencing external media sources, MPEG timed accessor 302 is an extension of the accessor hierarchy for accessing timed media, and MPEG ring buffer is an extension of the buffer hierarchy for supporting ring buffers. The first group of extensions provides a basic description and format for media in a scene and satisfies the basic need to describe immersive media that changes over time in a scene description framework. Of these, MPEG timed accessor (MPEG_accessor_timed) 302 is for accessing timed media. Since the glTF2.0 scene description standard does not support timed media, if media data needs to change over time, this must be achieved by updating the scene description file in the glTF2.0 scene description standard. For example, in the glTF2.0 scene description standard, if it is necessary to update the texture mapping of an object's surface so that the texture mapping of the object's surface changes over time, the scene description file in the glTF2.0 scene description standard must be updated. Frequently updating the scene description file requires frequent parsing, processing, and transmission of the scene description file, increasing the performance overhead in the 3D scene rendering process. Based on this, the MPEG timed accessor (MPEG_accessor_timed)302 was designed in MPEG so that parameters within the MPEG timed accessor can change over time, changing the media data access method and enabling the accessed data to change over time, thus avoiding frequent parsing, processing, and transmission of the scene description file.

[0051] The extensions to the second group include MPEG dynamic scenes (MPEG_scene_dynamic)304, MPEG textures (MPEG_texture_video)305, MPEG audio spaces (MPEG_audio_spatial)306, MPEG viewport recommendations (MPEG_viewport_recommended)307, MPEG mesh mapping (MPEG_mesh_linking)308, and MPEG animation timing (MPEG_animation_timing)309. Here, MPEG_scene_dynamic304 is a scene hierarchy extension to support dynamic scene updates, MPEG_texture_video305 is a texture hierarchy extension to support textures in video format, MPEG_audio_spatial306 is a node hierarchy and camera hierarchy extension to support spatial 3D audio, MPEG_viewport_recommended307 is a scene hierarchy extension to support describing recommended viewing angles when displaying in 2D, MPEG_mesh_linking308 is a mesh hierarchy extension to support linking two meshes to provide mapping information, and MPEG_animation_timing309 is a scene hierarchy extension to support controlling the animation timeline.

[0052] The following provides a detailed explanation of each of the above-mentioned extensions.

[0053] In an MPEG scene description file, the MPEG media is used to describe the type of media file, and by providing the necessary description for MPEG-type media files, these MPEG-type media files are retrieved subsequently. The definition of the first-level syntax elements of MPEG media is shown in Table 8 below. [Table 8]

[0054] The definitions of syntax elements in the media list (MPEG_media.media) of MPEG media are shown in Table 9 below. [Table 9]

[0055] The definitions of syntax elements within the options (MPEG_media.alternatives) in the media list of MPEG media are shown in Table 10 below. [Table 10]

[0056] The definitions of syntax elements in the track array (MPEG_media.alternatives.tracks) in the media list options for MPEG media are shown in Table 11 below. [Table 11]

[0057] Furthermore, ISO / IEC 23090-14 also defines the transmission format for data distribution related to the delivery of scene description files and extensions to glTF 2.0, based on ISOBMFF (ISO / IEC 14496-12). To facilitate the delivery of scene description files to clients, ISO / IEC 23090-14 defines how glTF files and associated data are encapsulated in ISOBMFF files as non-time-varying and time-varying data (e.g., as track samples). MPEG_scene_dynamic, MPEG_mesh_linking, and MPEG_animation_timing provide the display engine with time-varying data in specific formats, and the display engine 11 should perform corresponding operations based on this changed information. ISO / IEC 23090-14 also defines the format of each extension time-varying data and the method for encapsulating it in ISOBMFF files. MPEF media (MPEG_media) can reference external media streams delivered via protocols such as RTP / SRTP and MPEG-DASH. To enable addressing media flows without knowing the actual protocol solution, hostname, or port value, ISO / IEC 23090-14 defines a new Uniform Resource Locator (URL) scheme. This scheme requires that the query portion contains a single stream identifier, but does not specify a particular type of identifier, and allows the use of a Media Stream Identification scheme (RFC5888), a labeling scheme (RFC4575), or a zero-based index scheme.

[0058] Display engine

[0059] As shown in Figure 1, in the workflow of the immersive media scene description framework, the display engine 11 mainly includes acquiring a scene description file, analyzing the acquired scene description file to obtain the constituent structure of the rendering target 3D scene and detailed information in the rendering target 3D scene, and rendering and displaying the rendering target 3D scene based on the information obtained by analyzing the scene description file. In the embodiments of the present application, the specific workflow and principles of the display engine 11 are not limited, and the basis is that the display engine 11 analyzes the scene description document, sends commands to the media access function 12 via the media access function API, sends commands to the buffer management module 13 via the buffer API, acquires the processed data from the buffer, and completes the rendering and display of the 3D scene and the objects within it.

[0060] Media access function

[0061] In the workflow of the immersive media scene description framework, the media access function 12 receives commands from the display engine 11 and can complete the access and processing functions for media files according to the commands sent by the display engine 11. Specifically, this includes obtaining the media file and then processing the media file. There are significant differences in the processing processes for different types of media files, and in order to support a wide range of media types, and also considering the work efficiency of the media access function, it is sufficient to design various pipelines in the media access function and enable the pipeline that matches the media type in the processing process.

[0062] The input to the pipeline is media files downloaded from a server or media files read from a local storage control. These media files generally have complex structures and cannot be used directly by the display engine 11. Therefore, the main function of the pipeline is to process the data in such media files and match the data in the media files to the requirements of the display engine 11.

[0063] In the workflow of the immersive media scene description framework, the media data after pipeline processing must be passed to the display engine 11 in a standard array structure for use, which requires the participation of the buffer API and buffer management module 13. module Based on the format of the processed media data, the buffer management module 13 creates a corresponding buffer and is responsible for subsequent management operations on the buffer, such as updating and releasing. The buffer management module 13 may communicate with the media access function 12 via the buffer API or with the display engine 11, and the goal of communication with the display engine 11 and / or the media access function 12 is to implement buffer management. When the buffer management module 13 communicates with the media access function 12, the display engine 11 must first send buffer management-related commands to the media access function 12 via the media access function API, and the media access function 12 must then send buffer management-related commands to the buffer management module 13 via the buffer API. When the buffer management module 13 communicates with the display engine 11, the display engine 11 may directly send buffer management description information parsed from the scene description document to the buffer management module 13 via the buffer API.

[0064] The above embodiments introduce the basic flow of scene description framework rendering, including a 3D scene of immersive media, and the contents and roles of each functional module or file in the scene description framework. Immersive media in a 3D scene may be point cloud-based media files, 3D mesh-based media files, 6DoF-based media files, MIV media files, etc. Since some embodiments of this application relate to rendering a 3D scene including a point cloud based on the scene description framework, the point cloud-related content will be described first below.

[0065] A point cloud is a collection of a large number of three-dimensional points. After obtaining the spatial coordinates of each sampling point on an object's surface, the resulting set of points is called a point cloud. In addition to geometric coordinates, points in a point cloud may further include other attribute information such as color, normal vector, reflectivity, transparency, and material type. Point clouds can be obtained in various ways. In some embodiments, the implementation of point cloud acquisition involves observing an object using a camera array whose fixed position in space is known, and obtaining a point cloud corresponding to the object by obtaining a three-dimensional representation of the object using some relevant algorithms based on the two-dimensional image captured by the camera array. In some other embodiments, the implementation of point cloud acquisition involves obtaining a point cloud corresponding to an object using a laser radar scanning device. The sensors of the laser radar scanning device obtain volume information of the object by recording electromagnetic waves reflected from the object's surface after the electromagnetic waves from the radar are captured, and then obtain a point cloud corresponding to the object based on the volume information of the object. In another embodiment, the implementation of point cloud acquisition may further include obtaining a point cloud corresponding to an object by creating three-dimensional volume information based on a two-dimensional image using artificial intelligence or computer vision algorithms.

[0066] Point clouds provide a high-precision 3D representation for the detailed digitization of the physical world and are widely applied in fields such as 3D modeling, smart cities, autonomous navigation systems, and augmented reality. However, due to their characteristics such as large data volume, unstructured nature, and uneven density, the storage and transmission of point clouds face enormous challenges. Therefore, it is necessary to efficiently compress point clouds, and currently there are mainly two types of compression standards for point clouds: geometry-based point cloud compression (G-PCC) and video-based point cloud compression (V-PCC). The principles and related algorithms of G-PCC will be further explained below.

[0067] As shown in Figure 4, the G-PCC encoder 400 may be divided into two parts: a geometric coding module 41 and an attribute coding module 42. The geometric coding module 41 may be further divided into a geometric coding unit 411 based on an octree and a geometric coding unit 412 based on a prediction tree.

[0068] As shown in Figure 4, the main encoding steps by the G-PCC encoder's geometric encoding module 41 to encode the geometric information of the point cloud include step S401, which extracts geometric information (positions) in the point cloud to be encoded; step S402, which performs a coordinate transformation on the geometric information to include all points in the point cloud to be encoded in a single bounding box; and step S403, which performs voxelization on the geometric information after the coordinate transformation. That is, first the geometric information after the coordinate transformation is quantized to scale the point cloud to be encoded. Due to rounding during quantization, some points in the point cloud to be encoded will have the same position, so in order to quantize the geometric information after the coordinate transformation, it is necessary to decide whether or not to remove duplicate points based on parameters, and the process of quantization and removal of duplicate points is called the voxelization process. After the voxelization of the geometric information is completed, encoding is performed by the geometric encoding unit 411 based on an octvine and the geometric encoding unit 412 based on a prediction tree, respectively, to obtain a geometric information code stream of the point cloud to be encoded.

[0069] The encoding process of the octvine-based geometric encoding unit 411 includes, in S404, tree partitioning, and involves continuously performing tree partitioning (octvine / quadvine / binary tree) on the bounding box in a breath-first search order, and encoding the bit code of each node. That is, the bounding box is sequentially partitioned to obtain subcubes, and non-empty subcubes (containing points in the point cloud) are continued to be partitioned until the partitioned leaf nodes are 1x1x1 unit cubes. Next, the number of points contained in the leaf nodes is encoded, finally completing the encoding of the geometric octvine and generating a binary code stream. In S405, surface fitting is performed on the geometric information based on a triangular pool (trisoup). Similarly, surface fitting also begins with octvine partitioning, but it does not require stepwise division of the point cloud to be encoded into unit cubes with side lengths of 1x1x1. Instead, the partitioning stops when the side length of a subblock reaches a predetermined value. Then, based on the surface formed by the distribution of points in each subblock, up to 12 intersection points (vertex) generated by the 12 edges of the subblock are obtained. The coordinates of the intersection points of each subblock are then sequentially encoded to generate a binary code stream.

[0070] The coding process of the geometric coding unit 412 based on the prediction tree includes the following steps: S406, construct the prediction tree structure. This includes sorting the points in the point cloud to be coded, and the sorting method includes methods such as unordered, Morton order, azimuthal order, and radial distance order, and the prediction tree structure is constructed using two different methods (high-latency slow method and low-latency fast method). S407, based on the structure of the prediction tree, traverse each node in the prediction tree, select a different prediction mode to predict the geometric position information of the node and obtain the prediction residual, and quantize the geometric prediction residual using quantization parameters. S408, arithmetic coding is performed, which includes arithmetic coding of the prediction residual of the prediction tree node position information, the prediction tree structure, and quantization parameters, etc., through continuous iteration, and generating a binary geometric information code stream.

[0071] As shown in Figure 4, the process by which the G-PCC encoder attribute coding module 42 encodes attribute information of the point cloud to be encoded mainly includes S408, extracting attribute information (attributes) in the point cloud to be encoded; S409, performing attribute prediction on the attribute information; S410, performing a lifting transformation on the attribute information; S411, performing a Region Adaptive Hierarchical Transform (RAHT) transformation on the attribute information; S412, quantizing the coefficients of the RAHT transformation and the lifting transformation; and S413, performing arithmetic coding on the quantized coefficients of the RAHT transformation and the lifting transformation to obtain an attribute information code stream. Furthermore, since the attribute coding module 42 processes based on the reconstructed geometric information, after lossy geometric coding is completed, it is necessary to perform steps S414: reconstruct the geometric information based on the geometric code stream and match the original attribute information (attributes) with the reconstructed geometric information; and step S415: recolor the geometric information. Here, the recoloring step in S415 involves using the original point cloud to add attribute information to the reconstructed point cloud, with the aim of minimizing errors by making the attribute values ​​of the reconstructed point cloud as similar as possible to the attribute values ​​of the point cloud to be encoded.

[0072] The attribute prediction algorithm is an algorithm that obtains the predicted attribute value of the current target point by weighting and summing it with the reconstructed attribute values ​​of the reconstructed points in three-dimensional space. The attribute prediction algorithm can effectively remove redundancy in the attribute space and achieve the objective of compressing attribute information. In some embodiments, the method of realizing attribute prediction may include the following: First, a hierarchical partitioning of the point cloud to be encoded is performed using a Level of Detail (LOD) algorithm to construct a hierarchical structure of the point cloud to be encoded. Next, points in the lower levels are encoded and decoded first, and progressive coding is realized by predicting points in the higher levels using the reconstructed points of the same level as the lower level points. Here, the method of realizing hierarchical partitioning of the point cloud to be encoded using the LOD algorithm may include the following: First, all points in the point cloud to be encoded are marked as unaccessed, and the accessed point cloud is displayed as V. In the initial state, the accessed point cloud is empty. We traverse cyclically through all unaccessed points in the point cloud to be encoded, calculate the minimum distance D from the current point to the accessed point cloud V. If D is less than the threshold distance, we ignore the current point; otherwise, we mark the current point as accessed and add it to the accessed point cloud V and the current subspace. Finally, we merge the points in each subspace and all the subspaces preceding each subspace to obtain the hierarchical structure of the point cloud to be encoded.

[0073] As an example, as shown in Figure 5, Figure 5 illustrates that the point cloud to be encoded includes points P1-P9. In the distance-based LOD partitioning process, in the first circular traverse, points P0, P2, P4, and P5 are sequentially added to the accessed point cloud V and hierarchical R0; in the second circular traverse, points P1, P3, and P8 are sequentially added to the accessed point cloud V and hierarchical R1; in the third circular traverse, the traverse of all points is completed, and points P6, P7, and P9 are sequentially added to the accessed point cloud V and hierarchical R2; and finally, points in each hierarchical and all hierarchical levels preceding each hierarchical are merged to obtain the hierarchical structure of the point cloud to be encoded, which includes three hierarchical levels. Here, the first level is LOD0, which includes points P0, P2, P4, and P5; the second level is LOD1, which includes points P0, P2, P4, P5, P1, P3, and P8; and the third level is LOD2, which includes P0, P2, P4, P5, P1, P3, P8, P6, P7, and P9.

[0074] The lifting transform is built upon the predictive transform and consists of three parts: partitioning, prediction, and updating. As shown in Figure 6, the partitioning module 61 spatially partitions the point cloud to be encoded into two parts: a high-level point cloud H(N) and a low-level point cloud L(N), with a certain correlation between the two. The prediction module 62 uses the attributes of the low-level point cloud L(N) to perform attribute prediction on the high-level point cloud H(N), obtaining a prediction residual D(N) = H(N) - P(N). Here, P(N) is the feature output by the prediction module 62 after predicting the low-level point cloud L(N). In the processes of the partitioning module 61 and the prediction module 62, because the prediction strategy in LOD partitioning gives more influence to points in the low-level LOD layer, the update module 63 recursively updates each point by defining its influence weight based on the prediction residual D(N) and the distance between the predicted point and its neighbors. Here, recursive updating refers to performing multiple lifting transformations, where the output data of the previous lifting transformation is the input data for the next lifting transformation. Recursive updating by defining the influence weight of each point based on the predicted residual D(N) and the distance between the predicted point and its adjacent points includes recursive updating by defining the influence weight of each point based on the predicted residual D(N), the distance between the predicted point and its adjacent points, and the formula L'(N)=L(N)+U(N), where U(N) is a feature output by the update module 63 after predicting the predicted residual D(N).

[0075] The RAHT transform is a hierarchical domain adaptive transform algorithm based on the Haar wavelet transform. Based on a hierarchical tree structure, it recursively transforms occupied subnodes along each dimension from bottom to top within the same parent node, transmits the low-frequency coefficients obtained by the transform to the next level of the transform process, and performs quantization and entropy coding on the high-frequency coefficients.

[0076] In some embodiments, the above RAHT transformation can be achieved by RAHT transformation based on upsampling prediction. In RAHT transformation based on upsampling prediction, the tree structure of the entire RAHT transformation is changed from bottom-up to top-down, and the transformation is still performed within a 2x2x2 block. As shown in Figure 7, within a single 2x2x2 block, the transformation flow includes the following: First, an RAHT transformation is performed on voxel block 71 in the first direction. If there are adjacent voxel blocks in the first direction, both undergo RAHT to obtain the weighted average value (DC coefficient) and residual (AC coefficient) of the attribute values ​​of the two adjacent points. Here, the obtained DC coefficient exists as attribute information of the parent node voxel block 122, and the RAHT transformation of the next layer is performed, and the AC coefficient is reserved for final encoding. If there are no adjacent points, the attribute value of the voxel block 71 is directly transmitted to the second layer parent node. During the RAHT transformation of the second layer, it is performed along the second direction. If there are adjacent voxel blocks in the second direction, both undergo the RAHT transformation to obtain the weighted average value (DC coefficient) and residual (AC coefficient) of the attribute values ​​of the two adjacent points. Subsequently, the RAHT transformation of the third layer is performed along the third direction. The parent node voxel block 73, where the three different color depths intersect, is obtained as a subnode of the next layer in the octree, and the RAHT transformation is repeated along the first, second, and third directions until only one parent node exists in the entire point cloud to be encoded.

[0077] As shown in Figure 8, the G-PCC decoder 800 may be divided into a geometric decoding module 81 and an attribute decoding module 82, and the geometric decoding module 81 may be further divided into a geometric decoding unit 811 based on an octvine and a geometric decoding unit 812 based on a prediction tree.

[0078] As shown in Figure 8, the main steps by which the G-PCC decoder decodes the geometric information code stream using the geometric decoding unit 811 based on an octree of the geometric decoding module 81 include steps S801, arithmetic decoding, S802, octree synthesis, S803, surface fitting, S804, reconstruction geometry, and S805, inverse coordinate transformation, to obtain the geometric information of the point cloud. Here, the geometric decoding by the geometric decoding unit 811 based on an octree involves obtaining the bit code of each node by continuous analysis in the order of breadth-first traverse, and continuously dividing the nodes, stopping the division until a 1x1x1 unit cube is divided, obtaining the number of points contained in each leaf node by analysis, and finally restoring the geometrically reconstructed point cloud information. The main steps of the G-PCC decoder decoding the geometric information code stream using the geometric decoding unit 812 based on the prediction tree of the geometric decoding module 81 include steps S801, arithmetic decoding, S806, reconstruction prediction tree, S807, residual calculation, S804, reconstruction geometry, and S805, inverse coordinate transformation, to obtain the geometric information of the point cloud. The main steps of attribute decoding performed by the attribute decoding module 82 based on the G-PCC decoder 800 include S808, arithmetic decoding, S809, inverse quantization, execution of steps S810 and S811, or execution of step S812, S810, attribute prediction, S811, boosting transformation, S812, inverse transformation based on RAHT, and S813, inverse color transformation to obtain the attribute information of the point cloud. Finally, a three-dimensional image model of the point cloud data to be encoded is reconstructed based on the geometric information and attribute information. The main steps of the G-PCC decoder decoding the attribute information code stream based on the attribute decoding module 82 and the main steps of the G-PCC encoder encoding the attribute information based on the attribute encoding module 82 are inverse processes of each other, so their explanation is omitted here.

[0079] Currently, the extensions provided by the first edition of the ISO / IEC 23090-14 MPEG-I scene description standard meet important needs for immersive scene description solutions, addressing needs such as interaction with virtual scenes, AR anchors, display of virtual users, haptic support, and support for immersive codecs. Since point clouds are an important immersive 3D media format in 3D environments, supporting the display of point cloud media in the scene description standard is a crucial aspect of scene description. Geometry-based point cloud compression algorithms (G-PCC) are currently one of the mainstream point cloud compression algorithms, and supporting media files of type G-PCC encoded point clouds in scene descriptions has significant meaning and value.

[0080] Some embodiments of the present invention provide support for a scene description framework that includes a point cloud code stream obtained by the G-PCC compression standard, specifically including support for media files of type G-PCC encoded point clouds via scene description files, support for media files of type G-PCC encoded point clouds via media access function APIs, support for media files of type G-PCC encoded point clouds via media access functions, support for media files of type G-PCC encoded point clouds via buffer APIs, and support for media files of type G-PCC encoded point clouds via buffer management.

[0081] The process of rendering media files of type G-PCC encoded point clouds in a 3D scene based on a scene description framework includes the following: First, the display engine obtains the scene description file by means of download or local read. Here, the scene description file contains description information for the entire 3D scene and the media files of type G-PCC encoded point clouds contained in the scene, and the description information for the media files of type G-PCC encoded point clouds may include the access address of the media files of type G-PCC encoded point clouds, the storage format of the decoded data of the processed media files of type G-PCC encoded point clouds, the playback time of the media files of type G-PCC encoded point clouds, the playback frame rate, etc. After parsing the scene description file, the display engine transmits the description information for the media files of type G-PCC encoded point clouds contained in the scene description to the media access function via the media access function API. At the same time, the display engine may call the buffer management module via the buffer API to allocate buffers and transmit buffer information to the media access function, and the media access function calls the buffer management module via the buffer API to allocate buffers. After receiving descriptive information transmitted from the display engine, the media access function first requests the server to download a media file whose type is a G-PCC encoded point cloud, or reads a media file whose type is a G-PCC encoded point cloud from a local file. After obtaining a media file whose type is a G-PCC encoded point cloud, the media access function creates and starts the corresponding pipeline to process the media file whose type is a G-PCC encoded point cloud. The input to the pipeline is the encapsulation file of the media file whose type is a G-PCC encoded point cloud, and the pipeline sequentially performs processes such as decapsulation, G-PCC decoding, and post-processing, and then stores the processed data in a specified buffer.Ultimately, the display engine retrieves decoded data from a media file of the G-PCC encoded point cloud type from the specified buffer, and renders and displays the 3D scene based on the data retrieved from the buffer.

[0082] The following sections describe the scene description file, media access function API, media access function, buffer API, and buffer management that support media files of type G-PCC encoded point clouds.

[0083] Scene description file that supports media files of type G-PCC encoded point clouds.

[0084] To enable the scene description file to accurately describe a media file of the type G-PCC encoded point cloud, some embodiments of the present application extend the values ​​of the syntax elements in the MPEG media (MPEG_media) of the scene description file, specifically the extension including at least one of the following:

[0085] Extension 1 extends the media type syntax element (MPEG_media.media.alternatives.mimeType) for declaring the encapsulation format of a media file in the media list (media) options (MPEG_media.media.alternatives) of the MPEG media (MPEG_media) in the scene description file. The extension of the media type syntax element (mimeType) includes extending the value "application / mp4" associated with G-PCC coded point clouds to the media type syntax element (mimeType). If the type of media file is a G-PCC coded point cloud, the value of the media type syntax element (mimeType) will be "application / mp4". For example, mimeType: application / mp4.

[0086] Extension 2 extends the value of the first track index syntax element (MPEG_media.media.alternatives.tracks.tracks) for declaring track information of a media file in the optional track array (MPEG_media.media.alternatives.tracks) of the media list (media) of the MPEG media (MPEG_media) in the scene description file. The extension of the first track index syntax element (MPEG_media.media.alternatives.tracks.track) includes the fact that, when G-PCC data is referenced by the scene description file as one of the optional track arrays in the media list of the MPEG media, and the reference conforms to the track specification in the International Standardization Organization Base Media File Format (ISOBMFF), for single-track encapsulated G-PCC data, the track referenced in the MPEG media is a G-PCC code stream track, and for multi-track encapsulated G-PCC data, the track referenced in the MPEG media is a G-PCC geometric code stream track.

[0087] Extension 3 extends the codec parameter syntax element (MPEG_media.media.alternatives.tracks.codecs) to describe the codec parameters of media data contained in code stream tracks in the track array (tracks) of the media list (media) of the MPEG media (MPEG_media) in the scene description file. Specifically, the extension includes extending the codec parameters of media files contained in code stream tracks as defined in IETF RFC6381. If a code stream track contains multiple different types of codec parameters (for example, when encapsulating a G-PCC coded point cloud using DASH, the AdaptationSet contains representations with different codecs), the codec parameter syntax element (codecs) can be represented as a comma-separated list of codec values. Therefore, extending the retrieval of the codec parameter syntax element (codecs) includes the requirement that, if the media file type is a G-PCC coded point cloud, the value of the codec parameter syntax element (codecs) should be set according to the provisions of the ISO / IEC 23090-18 G-PCC Data Transmission (Carriage of Geometry-based Point Cloud Compression Data) standard. For example, if G-PCC data adopts DASH capsules, and G-PCC pre-selection signaling is used in the Media Presentation Description (MPD) file, the "codecs" attribute of the pre-selection signaling should be set to 'gpc1' to indicate that the pre-selection media is based on geometric point clouds. If there are multiple G-PCC Tile tracks in the G-PCC container, the "codecs" attribute of the Main G-PCC Adaptation Set should be set to 'gpcb' or 'gpeb' to indicate that the adaptation set contains G-PCC Tile base track data.If Tile Component Adaptation Sets signal only a single G-PCC component data, the "codecs" attribute of the Main G-PCC adaptivesset should be set to 'gpcb'. If Tile Component Adaptation Sets signal all G-PCC component data, the "codecs" attribute of the Main G-PCC Adaptation Set should be set to 'gpeb'. If G-PCC Tile pre-selection signaling is used in the MPD file, the "codecs" attribute of the pre-selection signaling should be set to 'gpt1', indicating that the pre-selection media is a geometric point cloud fragment.

[0088] In summary, in order to enable scene description files to accurately describe media files of the G-PCC encoded point cloud type, some implementations of the present invention extend the values ​​of syntax elements in MPEG media (MPEG_media) within the scene description file, and specific extensions include one or more extensions shown in Table 12 below. [Table 12]

[0089] By performing at least one of the above extensions 1-3 to retrieve the values ​​of syntax elements within MPEG media (MPEG_media) in the scene description file, MPEG media (MPEG_media) in the scene description file partially supports media files of type G-PCC encoded point clouds.

[0090] In some embodiments, the method for describing scenes and nodes in a scene description file containing media files of type G-PCC encoded point clouds includes, when a 3D scene contains media files of type G-PCC encoded point clouds, describing the overall structure of the 3D scene and the structural hierarchy and location of the media files of type G-PCC encoded point clouds within the 3D scene using the scene and node description method. Here, describing the overall structure of the 3D scene and the structural hierarchy and location of the media files of type G-PCC encoded point clouds within the 3D scene using the description methods of the scene description module and node description module includes describing one 3D scene using one scene description module. Each scene description file can describe one or more 3D scenes, and the relationships between 3D scenes are parallel only, not hierarchical. The relationships between nodes may be parallel or hierarchical.

[0091] In some embodiments, the method for describing a 3D mesh in a scene description file that supports media files of type G-PCC encoded point clouds involves multiplexing each type of data in the media file where the syntax element description type in the primitive attributes (mesh.primitives.attributes) of the mesh description module is G-PCC encoded point cloud. Specifically, since a point cloud is a distributed data structure, and a collection of many scattered points is a point cloud, describing a media file of type G-PCC encoded point cloud is equivalent to describing the data at each point in the point cloud. Generally, each point in a media file of type G-PCC encoded point cloud has two types of information: geometric information and attribute information. Geometric information represents the 3D coordinates of the point in space, and attribute information represents information attached to the point, such as color, reflectance, and normal direction. Since the data at points in a media file whose type is G-PCC encoded point cloud and the syntax elements included in the attributes of primitives in the mesh description module are similar to the attributes that can be declared, when describing data at points in a media file whose type is G-PCC encoded point cloud in the mesh description module (mesh), the syntax elements in the attributes (mesh.primitives.attribute) of the primitives (primitives) of the mesh description module (mesh) can be multiplexed to describe the data at points in a media file whose type is G-PCC encoded point cloud.

[0092] For example, the value of the position syntax element (position, the first entry in Table 1 above) in the primitive attribute of the mesh description module is a 3D vector consisting of floating-point numbers. Since such a data structure can similarly represent the geometric information of a G-PCC encoded point cloud, the position syntax element (position) in the primitive attribute (mesh.primitives.attribute) of the mesh description module can be multiplexed to represent the geometric information of a point in a media file of type G-PCC encoded point cloud. Alternatively, for example, the color value of a point in a media file of type G-PCC encoded point cloud may be represented by multiplexing the color syntax element (color_n, the fifth entry in Table 1 above) in the primitive attribute (mesh.primitives.attribute) of the mesh description module. Alternatively, for example, the normal vectors of points in a media file whose type is a G-PCC encoded point cloud may be represented by multiplexing the normal vector syntax element (normal, the third entry in Table 1 above) in the primitive attribute (mesh.primitives.attribute) of the mesh description module.

[0093] If we define the first syntax element set as the set of syntax elements supported in the attributes of primitives in the mesh description module of a scene description file specified in the ISO / IEC 23090-14 MPEG-I scene description standard, then a method for describing a 3D mesh that supports a media file of type G-PCC encoded point cloud includes adding syntax elements corresponding to each type of data that the 3D mesh has to the attributes of primitives in the mesh description module corresponding to the 3D mesh, based on the syntax elements in the first syntax element set. As shown in Table 13 below, Table 13 shows a method for describing some of the data of points in a media file of type G-PCC encoded point cloud by multiplexing syntax elements in the attributes (mesh.primitives.attribute) of primitives in the mesh description module. [Table 13]

[0094] The above examples and Table 13 illustrate a method in which only a portion of the G-PCC encoded point cloud data is described by multiplexing the syntax elements in the attributes of the primitives in the mesh description module. The G-PCC encoded point cloud data may also include other data, and the other data in the G-PCC encoded point cloud may be described by multiplexing the syntax elements in the attributes of the primitives in the mesh description module. Examples include texture coordinates (texcoord_n), joints (joints_n), and weights (weights_n).

[0095] In another embodiment, a 3D mesh description method that supports a media file of type G-PCC encoded point cloud includes adding a target extension array to the primitive extension list (mesh.primitives.extensions) of the mesh description module, adding syntax elements to the target extension array corresponding to each type of data contained in the 3D mesh in the media file of type G-PCC encoded point cloud, and describing, respectively, data such as geometric information, color data, and normal vectors associated with each vertex of the 3D mesh in the media file of type G-PCC encoded point cloud using the syntax elements corresponding to each type of data.

[0096] In some embodiments, adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the target extension array includes adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the target extension array, based on syntax elements in a first set of syntax elements. Here, the first set of syntax elements is a set of syntax elements supported in the attributes of primitives of the mesh description module of a scene description file as defined in the ISO / IEC 23090-14 MPEG-I scene description standard.

[0097] In some embodiments, adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the target extended array includes adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the target extended array based on a second set of syntax elements consisting of syntax elements corresponding to a pre-configured G-PCC coded point cloud.

[0098] If we define a syntax element representing the geometric information associated with each vertex as the first syntax element, a syntax element representing the color data associated with each vertex as the second syntax element, and a syntax element representing the normal vector associated with each vertex as the third syntax element, then the syntax elements added to the target extension array of the primitive extension list (mesh.primitives.extensions) of some mesh description modules will include the following, as shown in Table 14 below: [Table 14]

[0099] As shown in Figure 9, Figure 9 is a schematic diagram of the scene description file structure after adding a target extension array to the primitive extension list (mesh.primitives.extensions) of the mesh description module based on the above embodiment, and extending the target extension array with a first syntax element, a second syntax element, and a third syntax element. A scene description file includes, but is not limited to, MPEG media (MPEG_media)901, scene description module (scene)902, node description module (node)903, mesh description module (mesh)904, accessor description module (accessor)905, buffer slice description module (buffer View)906, buffer description module (buffer)907, skin description module (skin)908, animation description module (animation)909, camera description module (camera)910, material description module (material)911, texture description module (texture)912, sampler description module (sampler)913, and texture mapping description module (image)914. Here, the extended list of primitive attributes in the mesh description module 904 includes a target extension array 9000, and the extended syntax elements within the target extension array 9000 include a first syntax element 9001 for representing geometric information associated with each vertex, a second syntax element 9002 for representing color data associated with each vertex, and a third syntax element 9003 for representing the normal vector associated with each vertex. In addition to the above extensions, information such as the roles of other elements in the scene description file shown in Figure 9, accessor types, and data types are similar to the scene description file shown in Figure 3 and are therefore omitted from this explanation.

[0100] In some embodiments, a mesh description method that supports a media file of type G-PCC encoded point cloud includes pre-setting syntax elements corresponding to each type of data in the G-PCC encoded point cloud, and adding syntax elements corresponding to each type of data to the attributes of primitives in a mesh description module corresponding to a 3D mesh in the G-PCC encoded point cloud, based on the pre-set syntax elements corresponding to each type of data in the G-PCC encoded point cloud.

[0101] Exemplary, the syntax elements corresponding to each type of data in a pre-configured G-PCC coded point cloud include a fourth syntax element for representing geometric information associated with each vertex, a fifth syntax element for representing color data associated with each vertex, and a sixth syntax element for representing the normal vector associated with each vertex. Adding the syntax elements corresponding to each type of data to the attributes of a primitive in a mesh description module corresponding to a 3D mesh in the G-PCC coded point cloud, based on the syntax elements corresponding to each type of data in the pre-configured G-PCC coded point cloud, includes adding at least one of the fourth, fifth, and sixth syntax elements to the attributes of a primitive in a mesh description module corresponding to a 3D mesh in the G-PCC coded point cloud.

[0102] If we define a syntax element to represent the geometric information associated with each vertex corresponding to the G-PCC coded point cloud as the fourth syntax element, a syntax element to represent the color data associated with each vertex corresponding to the G-PCC coded point cloud as the fifth syntax element, and a syntax element to represent the normal vector associated with each vertex corresponding to the G-PCC coded point cloud as the sixth syntax element, then, as shown in Table 15 below, the description method of syntax elements in the attributes of primitives in some mesh description modules includes the following: [Table 15]

[0103] As shown in Figure 10, Figure 10 is a schematic diagram of the structure of a scene description file after extending the syntax elements in the primitive attributes (mesh.primitives.attribute) of the mesh description module based on the above embodiment. The scene description file consists of MPEG media (MPEG_media) 101, scene description module (scene) 102, and node description module (node) 103 , the primitive attributes (mesh.primitives.attribute) of the mesh description module 104 include, but are not limited to, a mesh description module (mesh) 104, an accessor description module (accessor) 105, a buffer slice description module (buffer View) 106, a buffer description module (buffer) 107, a skin description module (skin) 108, an animation description module (animation) 109, a camera description module (camera) 110, a material description module (material) 111, a texture description module (texture) 112, a sampler description module (sampler) 113, and a texture mapping description module (image) 114. Here, the primitive attributes (mesh.primitives.attribute) of the mesh description module 104 include an extended fourth syntax element 1041 for representing geometric information associated with each vertex, a fifth syntax element 1042 for representing color data associated with each vertex, and a fifth syntax element 1043 for representing the normal vector associated with each vertex. In addition to the above extensions, the information regarding the roles of other elements, accessor types, and data types in the scene description file shown in Figure 10 is similar to that in the scene description file shown in Figure 3, and therefore will not be explained here.

[0104] Furthermore, when a scene description file describes a 3D scene containing a media file of the G-PCC encoded point cloud type, regardless of whether the G-PCC encoded point cloud data is described by multiplexing syntax elements in the attributes of the primitives in the mesh description module, by adding a target extension array to the primitives in the mesh description module, or by extending the attributes of the primitives in the mesh description module with new syntax elements to describe a media file of type G-PCC encoded point cloud, the mesh description module (mesh) will always contain a large number of points in the G-PCC encoded point cloud, and each point will contain at least geometric information and attribute information. Therefore, it is inconvenient to directly store the data of a media file of type G-PCC encoded point cloud in the scene description framework. When it is necessary to point out a link to a media file of type G-PCC encoded point cloud in the scene description framework and obtain the G-PCC encoded point cloud data, the media file is downloaded.

[0105] In some embodiments, the scene description file and the media file, which is of type G-PCC encoded point cloud, may be merged to form a single binary file, thereby reducing the number and types of files.

[0106] In some embodiments, the description of an accessor, buffer slice, and buffer module that supports a media file of type G-PCC encoded point cloud includes the index value declared by the media index syntax element (media) of the MPEG ring buffer (MPEG_buffer_circular) of the buffer module (buffer) pointing to a media module corresponding to a media file of type G-PCC encoded point cloud in the MPEG media (MPEG_media).

[0107] In other words, media files whose type is G-PCC encoded point clouds must be specified within the buffer description module (buffer). However, instead of directly adding the Uniform Resource Locator (URL) of the media file whose type is G-PCC encoded point clouds to the buffer description module, the value of the media index syntax element (media) in the MPEG ring buffer (MPEG_buffer_circular) within the buffer description module (buffer) points to the media description module corresponding to the media file whose type is G-PCC encoded point clouds in the MPEG media (MPEG_media).

[0108] For example, if the value of the Uniform Resource Identifier syntax element (uri) in the options of the media description module corresponding to a media file whose type in the media list (media) of MPEG media (MPEG_media) is G-PCC encoded point cloud is "http: / / www.example.com / G-PCCexample.mp4" and is the first media description module in MPEG media, then setting the value of the Media Index syntax element (media) in the MPEG ring buffer (MPEG_buffer_circular) to "0" will index the link of the first media file in MPEG media in the MPEG ring buffer of the buffer description module, and the Media Index syntax element (media) in the MPEG ring buffer (MPEG_buffer_circular.media) of the buffer description module (buffer) will index the media description module corresponding to the media file whose type in MPEG media (MPEG_media) is G-PCC encoded point cloud.

[0109] In some embodiments, an accessor, buffer view, and buffer description method that supports a media file of type G-PCC encoded point cloud includes track information of the data buffered in the buffer by the value of the second track index syntax element (tracks) of the track array (tracks) of the MPEG ring buffer (MPEG_buffer_circular) of the buffer description module (buffer).

[0110] On glTF 2.0, the scene description technique proposed by MPEG includes an extension called the MPEG ring buffer (MPEG_buffer_circular). The MPEG ring buffer is used to reduce the number of buffers used, assuming the data buffer is guaranteed. The MPEG ring buffer can be considered as connecting the head and tail of a normal buffer to form a single ring, and writing to the ring buffer and reading data from the ring buffer realizes a simultaneous writing and reading operation using write and read pointers. The syntax elements included in the MPEG ring buffer (MPEG_buffer_circular) are shown in Table 16. [Table 16]

[0111] In other words, based on the setting rules for the value of the syntax element "media" in Table 16, if the value of the media index syntax element (media) in Table 16 is set to the index value of the media description module corresponding to a media file whose type declared in MPEG media (MPEG_media) is a G-PCC encoded point cloud, then the media file whose type is a G-PCC encoded point cloud can be indexed within the buffer description module (buffer). Similarly, based on the setting rules for the value of the track index syntax element (tracks) in Table 16, if the value of the track index syntax element (tracks) in Table 16 is set to the index value of one or more code stream tracks of a media file whose type is a G-PCC encoded point cloud, then the decoded data of that one or more code stream tracks can be buffered in the corresponding buffer.

[0112] In some embodiments, a method for describing materials, textures, samplers, and texture mappings that supports a media file of type G-PCC encoded point clouds includes not describing the 3D scene using materials, textures, samplers, and texture mappings when the scene description file is used to describe a 3D scene of G-PCC encoded point clouds.

[0113] Since the G-PCC encoded point cloud has a scattered topological structure, it does not actually have the concept of a surface. Various additional information is also directly represented by points, and material, texture, sampler, and image are all additional information to surfaces. Therefore, we reserve the right to define material, texture, sampler, and image, but we do not use material, texture, sampler, and image to describe a 3D scene.

[0114] In some embodiments, a camera description module (camera) description method that supports media files of type G-PCC encoded point clouds includes defining viewing-related visual information such as viewpoint and viewing angle of nodes in a 3D scene by the camera description module.

[0115] In some embodiments, an animation description module (animation) description method that supports media files of type G-PCC encoded point clouds includes animations added to node description modules (nodes) in a 3D scene by the animation description module (animation).

[0116] In some embodiments, an animation description module can describe animations to be added to a node description module (node) by one or more of the following: positioning, rotation, and scaling.

[0117] In some embodiments, the animation description module may specify at least one of the start time, end time, and animation implementation method of the animation added to the node description module (node).

[0118] That is, in a scene description file that supports media files of type G-PCC encoded point clouds, animation may similarly be added to nodes representing objects in a 3D object. The animation may describe the animation added to the node in three ways: position translation, angular rotation, and size scaling, and may also specify the start and end times of the animation and the method of implementing the animation.

[0119] In some embodiments, a method for describing a skin description module (skin) that supports a media file of type G-PCC encoded point cloud includes defining motion and distortion relationships between a mesh and a corresponding skeleton in a node description module (node) using the skin description module (skin).

[0120] Based on the above embodiment, improvements and extensions have been made to the video expert group media (MPEG_media), scene description module (scene), node description module (node), mesh description module (mesh), accessor description module (accessor), buffer slice description module (buffer View), buffer description module (buffer), skin description module (skin), animation description module (animation), camera description module (camera), material description module (material), texture description module (texture), sampler description module (sampler), and texture mapping description module (image) in the scene description file, so that the scene description file can accurately describe media files of type G-PCC encoded point clouds.

[0121] As an example, the following describes a scene description file that supports a media file whose type according to the embodiment of the present invention is a G-PCC encoded point cloud, with reference to a specific scene description file.

[0122] JPEG0007872437000020.jpg152166JPEG0007872437000021.jpg148164JPEG0007872437000022.jpg148170JPEG0007872437000023.jpg150164

[0123] The pair of square brackets on lines 1 and 118 in the example above contain the main contents of a scene description file that supports media files of type G-PCC encoded point clouds. A scene description file that supports media files of type G-PCC encoded point clouds includes a digital asset description module (asset), an extension description module (extension Used), MPEG media (MPEG_media), a scene statement (scene), a scene list (scenes), a node list (nodes), a mesh list (meshes), an access list (accessors), a buffer slice list (buffer Views), and a buffer list (buffers). The contents of each part and the information contained in each list at the analysis angle will be explained below.

[0124] Digital Asset Description Module (asset): The digital asset description module is located on lines 2-4. The "version":"2.0" on line 3 of the digital asset description module confirms that the scene description file was created based on glTF version 2.0, and that this version is also the reference version of the scene description standard. From an analysis perspective, the display engine can determine which parser to select to analyze the scene description file according to the digital asset description module.

[0125] Extension Used: The extension used is found in lines 6-10. Since the extension used includes three syntax elements—MPEG media, MPEG ring buffer, and MPEG timed accessor—it can be determined that the scene description file uses these three MPEG extensions. From an analysis perspective, the display engine can know in advance, based on the contents of the extension used, that subsequent analysis will include extensions related to MPEG media, MPEG ring buffer, and MPEG timed accessor.

[0126] MPEG media (MPEG_media): The MPEG media is located in lines 12-34. The MPEG media statement is for media files whose type is G-PCC encoded point clouds included in a 3D scene. The media type syntax element on line 21 and its value "mimeType": "application / mp4" indicates the capsule format of the media file containing the media file whose type is G-PCC encoded point clouds. The "uri" on line 22: "http: / / www.exp.com / G-PCCexp.mp4" indicates the access address of the media file whose type is G-PCC encoded point clouds. The "track" on line 25: "trackIndex=1" indicates the track information of the media file whose type is G-PCC encoded point clouds. The "codecs" on line 26: "gpc1" indicates the codec parameters of the media file whose type is G-PCC encoded point clouds. The "name" on line 16: "G-PCCexample" indicates the name of the media file whose type is G-PCC encoded point clouds. The "autoplay" on line 17: The value `true` indicates that media files of type G-PCC encoded point clouds should be automatically played, and the value `loop: true` on line 18 indicates that G-PCC encoded point cloud files should be played in a loop. From an analysis perspective, the display engine can determine, by analyzing the MPEG media, that media files of type G-PCC encoded point clouds exist in the 3D scene to be rendered, and obtain a method to access and analyze those media files of type G-PCC encoded point clouds.

[0127] Scene statement: The scene statement is on line 36. Since a single scene description file can theoretically contain multiple 3D scenes, in the above scene description file, the scene statement on line 36 and its "scene": 0 first indicate that the 3D scene to be processed and rendered later, based on the scene description file, is the first 3D scene in the scene list, i.e., the 3D scene enclosed in square brackets on lines 39-43.

[0128] Scene List (scenes): The scene list is lines 38-44. The fact that the scene list contains only one set of square brackets indicates that the scene list contains only one scene description module, that the scene description file contains only one 3D scene, and that within the square brackets, lines 40-42, "nodes":[0], indicate that the 3D scene contains only one node, and that the index value of the node description module corresponding to that node is 0. From an analytical standpoint, the contents of the scene list clarify that the entire scene description framework should select the first 3D scene in the scene list (the 3D scene with index 0) for subsequent processing and rendering, clarify the overall structure of the 3D scene, and point to the next layer of more detailed node description modules (nodes).

[0129] Node List (nodes): The node list is lines 46-51. The fact that the node list contains only one set of square brackets indicates that the node list contains only one node description module, the 3D scene contains only one node, and that node and the node whose index value in the node description module in the scene description module is 0 are the same node, and the two are related in an index manner. In the square brackets representing the node, line 48's "name": "G-PCCexample_node" indicates that the node's name is "G-PCC example_node", and line 49's "mesh": 0 indicates that the content mounted on the node is a 3D mesh corresponding to the first mesh description module in the mesh list, which corresponds to the mesh description module of the next layer. From an analytical standpoint, the content of this node list indicates that the content mounted on the node is a 3D mesh, and that the 3D mesh is a 3D mesh corresponding to the first mesh description module in the mesh list.

[0130] Mesh List: The mesh list is lines 53-66, and the fact that the mesh list contains only one set of square brackets indicates that the mesh list contains only one mesh description module, that the 3D scene has only one 3D mesh, and that this 3D mesh and the 3D mesh with an index value of 0 in the node description module are the same 3D mesh. In the square brackets (mesh description module) describing this 3D mesh, line 55, "name": "G-PCC example_mesh", indicates that the name of this 3D mesh is "G-PCC example_mesh", and this name is used only as an identification mark. Line 56, "primitives", indicates that this 3D mesh has primitives. Line 58, "attributes", and line 62, "mode", respectively, indicate the primitives AttributesThis indicates that there are two types of information: (attribute) and (mode). Line 59, "position", and line 60, "color_0", respectively, indicate that the 3D mesh has geometric coordinates and color data. Line 59, "position":0, and line 60, "color_0":1, respectively, indicate that the accessor corresponding to the geometric coordinates is the accessor corresponding to the first accessor description module in the Accessorist, and the accessor corresponding to the color data is the accessor corresponding to the second accessor description module in the Accessorist. Additionally, line 62, "mode":0, confirms that the topology of the 3D mesh is a scattered structure. From an analytical standpoint, this mesh list clarifies the actual data types and topology types of the 3D mesh in the scene description file.

[0131] Buffer List (buffers): The buffer list is located on lines 106-117. The presence of only one set of square brackets in the buffer list indicates that the scene description file contains only one buffer description module, and that the display of the 3D scene requires access to only one media file. These square brackets use an extension called MPEG_buffer_circular, indicating that this buffer is a ring buffer modified using the MPEG extension. Line 112, "media":0, indicates that the data source in the ring buffer is the media file corresponding to the first media description module declared in the aforementioned MPEG media, and line 113, "tracks":"#trackIndex=1", indicates that when accessing the media file, the track with index value 1 should be referenced. Here, the track with index value 1 is not limited; it may be the only track in a media file whose single-track encapsulated type is a G-PCC encoded point cloud, or it may be a geometric code stream track in a media file whose multi-track encapsulated type is a G-PCC encoded point cloud. Furthermore, based on the syntax element "count": 5 in the MPEG ring buffer, it can be determined that the MPEG ring buffer has 5 memory segments, and based on the syntax element "byteLength": 15000 in the MPEG ring buffer, it can be determined that the byte length (capacity) of the MPEG ring buffer is 15000 bytes. From an analytical standpoint, the buffer list enables the mapping of media files whose declared type in the MPEG media is a G-PCC encoded point cloud to a buffer, or enables the referencing of media files whose declared type is a G-PCC encoded point cloud that has not been used by the buffer.Note that the media files of type G-PCC encoded point clouds cited here are unprocessed G-PCC capsule files, and without processing by the media access function, it is not possible to extract information directly usable for rendering, such as position coordinates and color values ​​(color_0) mentioned in the mesh description module.

[0132] Buffer Slice List (buffer Views): The buffer slice list is on lines 93-104. The buffer slice column contains two parallel square brackets, and there is only one buffer determined by the buffer description module, indicating that the buffer for storing media files of type G-PCC encoded point clouds is divided into two buffer slices, and the point cloud data in media files of type G-PCC encoded point clouds is stored in two buffer slices. In the first square bracket (first buffer slice description module), first, buffer:0 on line 95 refers to the buffer description module with index 0, i.e., the only buffer description module mentioned in the buffer list, and then the two parameters byte length and byte offset on lines 96 and 94 limit the data slice range of the corresponding buffer slice to the first 12,000 bytes. The contents of the second square bracket (second buffer slice description module) are similar to the first square bracket, but the data slice range is defined as the last 3,000 bytes. From an analytical standpoint, the buffer slice list groups point cloud data in media files whose type is G-PCC encoded point clouds, contributing to the detailed definition of subsequent accessor description modules.

[0133] Accessors: The accessors are located on lines 68-91. The structure of the accessors is similar to that of the buffer slice list, and the presence of two parallel sets of square brackets indicates that the accessors contain two accessor description modules, and that the display of the 3D scene requires access to media data via two accessors. Furthermore, both sets of square brackets (accessor description modules) have an extension called MPEG_accessor_timed, which explains that both of these accessors refer to MPEG-defined time-varying media. In the first set of square brackets, the content of the MPEG time-varying accessor refers to the buffer slice description module with an index value of 0. In the first set of square brackets (the first accessor description module), the "componentType":5126 on line 70 and the "type":"VEC3" on line 71 further explain that the data format stored in the accessor is a three-dimensional vector consisting of 32-bit floating-point numbers, and "count":1000 explains that there are 1000 pieces of data that need to be accessed by an accessor of this format, with each 32-bit floating-point number occupying 4 bytes. Therefore, the accessor corresponding to this accessor description module contains 12000 bytes of data, which corresponds to the setting in the buffer slice description module where the index value is 0. The contents of the second set of square brackets (the second accessor description module) are similar, changing the index value of the buffer slice description module to 1 and redefining the data type. From an analytical standpoint, the accessors improve the complete definition of the data required for rendering; for example, data types missing in the buffer slice description module and buffer description module are defined in the corresponding accessor description module.

[0134] A display engine that supports media files of type G-PCC encoded point clouds.

[0135] In the workflow of the immersive media scene description framework, the main function of the display engine is to support the functions of the display engine for media files of type G-PCC encoded point clouds, and is similar to the main function of the display engine in the workflow of the immersive media scene description framework described above: 1. It can analyze the scene description file of a media file of type G-PCC encoded point clouds and obtain the rendering method for the corresponding 3D scene; 2. It can transmit media access functions and media access commands or media data processing commands via the media access function API, of which the media access commands or media data processing commands are derived from the analysis results of the scene description file of the media file of type G-PCC encoded point clouds; 3. It can send buffer management commands to the buffer management module via the buffer API; and 4. It can obtain the processed G-PCC encoded point cloud data from the buffer and complete the rendering and display of the 3D scene and objects in the 3D scene based on the read data. The details of the processing process will not be elaborated here.

[0136] Media access function API that supports media files of type G-PCC encoded point clouds.

[0137] In the workflow of the immersive media scene description framework, the display engine can obtain a method for rendering a 3D scene containing media files of type G-PCC media files by parsing the scene description file, and then needs to transmit the method for rendering the 3D scene to the media access function, or send commands to the media access function based on the method for rendering the 3D scene. This process of transmitting the method for rendering the 3D scene to the media access function, or sending commands to the media access function based on the method for rendering the 3D scene, is implemented by the media access function API.

[0138] In some embodiments, the display engine may send media access commands or media data processing commands to the media access function via the media access function API. The media access commands or media data processing commands that the display engine sends to the media access function via the media access function API are derived from the results of parsing a scene description file of a media file whose type is a G-PCC encoded point cloud, and the media access commands or media data processing commands may include the index of the media file whose type is a G-PCC encoded point cloud, the URL of the media file whose type is a G-PCC encoded point cloud, attribute information of the media file whose type is a G-PCC encoded point cloud, the display time window of the media file whose type is a G-PCC encoded point cloud, and format requests for the processed media file whose type is a G-PCC encoded point cloud.

[0139] In some embodiments, the media access function may request a media access command or a media data processing command from the display engine via the media access function API.

[0140] Media access functions that support media files of type G-PCC encoded point clouds.

[0141] In the workflow of an immersive media scene description framework, a media access function receives a media access command or media data processing command sent by the display engine via the media access function API, and then executes the media access command or media data processing command sent by the display engine via the media access function API. For example, it might retrieve a media file whose type is a G-PCC encoded point cloud, establish an appropriate pipeline for the media file whose type is a G-PCC encoded point cloud, and allocate an appropriate buffer for the processed media file whose type is a G-PCC encoded point cloud.

[0142] In some embodiments, a media access function obtaining a media file of type G-PCC encoded point cloud involves using a network transmission service to download a media file of type G-PCC encoded point cloud from a server.

[0143] In some embodiments, the media access function retrieving a media file of the type of G-PCC encoded point cloud includes reading a media file of the type of G-PCC encoded point cloud from local storage.

[0144] The media access function needs to process media files of type G-PCC encoded point clouds after obtaining them. There are significant differences in the processing of different media file types, and considering the efficiency of the media access function in order to support a wide range of media types, it is necessary to design various pipelines within the media access function and enable only the pipeline that matches the media type during the media file processing process. If the media file is of type G-PCC encoded point clouds, the media access function establishes a corresponding pipeline for media files of type G-PCC encoded point clouds, and the established pipeline needs to perform processes such as decapsulation, G-PCC decoding, and post-processing on media files of type G-PCC encoded point clouds. After completing processing for media files of type G-PCC encoded point clouds, the media access function processes the media file data of type G-PCC encoded point clouds into a data format for direct rendering by the display engine.

[0145] As shown in Figure 11, Figure 11 is a schematic diagram of the structure of a pipeline corresponding to a G-PCC encoded point cloud in some embodiments of the present invention. As shown in Figure 11, the pipeline 1100 that supports a media file of type G-PCC encoded point cloud includes an input module 111, a decapsulation module 112, a geometric decoder 113, an attribute decoder 114, a first post-processing module 115, and a second post-processing module 116.

[0146] The input module 111 receives a G-PCC capsule file and inputs it to the decapsulation module 112. Here, the G-PCC capsule file is a file obtained by encapsulating a G-PCC code stream obtained by G-PCC encoding point cloud data. Since the G-PCC capsule file is displayed in track format, what the input module 111 receives is the track stream of the G-PCC capsule file. Furthermore, as can be seen from the encapsulation rules of the G-PCC code stream, the G-PCC capsule file may be a single track or a multi-track file. Therefore, in the embodiment of this application, the G-PCC capsule file received by the input module 111 may be a single track or a multi-track file, and the embodiment of this application is not limited to this.

[0147] The decapsulation module 112 decapsulates the G-PCC capsule file input from the input module 111 to obtain a G-PCC code stream (including a geometric information code stream and an attribute information code stream), inputs the geometric information code stream to the geometric decoder 113, and inputs the attribute information code stream to the attribute decoder 114. Furthermore, as related technologies develop, it is possible that code streams of other information may be added to the G-PCC code stream. If the G-PCC code stream contains additional code streams of other information, the decapsulation module 112 decapsulates the G-PCC capsule file to obtain the code streams of the other information and inputs them to the corresponding decoders.

[0148] The geometric decoder 113 decodes the geometric information code stream output from the decapsulation module 112 to obtain the geometric information of the point cloud. Here, the main steps in which the geometric decoder 113 decodes the geometric information code stream include obtaining the geometric information of the point cloud through arithmetic decoding, octree synthesis, surface fitting, geometric reconstruction, inverse coordinate transformation, etc. A concrete implementation of the geometric decoder 113 decodes the geometric information code stream can be seen by referring to the workflow of the geometric decoding module 81 in Figure 8, and a detailed explanation is omitted here.

[0149] The attribute decoder 114 decodes the attribute information code stream input from the decapsulation module 112 to obtain the attribute information of the point cloud. Here, the main steps by which the attribute decoder 114 decodes the geometric information code stream include attribute prediction, enhancement, and inverse RAHT transformation to obtain the attribute information code stream. The specific implementation of the attribute decoder 114 decoding the attribute information code stream can be seen in the workflow of the attribute decoding module 82 in Figure 8, and a detailed explanation is omitted here.

[0150] The first post-processing module 115 processes the geometric information output from the geometric decoder 113. After completing the decoding of the geometric information code stream, the geometric information of points in the G-PCC coded point cloud can be obtained, and in some cases the obtained geometric information can be used directly by the display engine. However, since the scene description framework does not impose excessive restrictions or special definitions on the display engine, various types of display engines may appear. Because the requirements for input data of these different display engines may differ, the first post-processing module 115 is added after completing the decoding of the geometric information code stream to ensure that the geometric information output of the pipeline is available to any display engine. In some embodiments, the processing of geometric information by the first post-processing module 115 includes performing a format conversion on the geometric information.

[0151] The second post-processing module 116 is configured to process the attribute information output by the attribute decoder 114. After decoding the attribute information code stream, the attribute information of the points in the G-PCC encoded point cloud can be obtained, and in some cases, the attribute information can be used directly by the display engine. However, since the scene description frame does not impose excessive restrictions or special definitions on the display engine, various types of display engines may appear. Because the requirements for the input data of these different display engines may differ, adding the second post-processing module 116 after decoding the attribute information code stream ensures that the attribute information of the pipeline output is available to any display engine. In some embodiments, the processing of geometric information by the first post-processing module 115 includes performing format conversion on the attribute information.

[0152] Finally, by writing the processed geometric information output by the first post-processing module 115 and the processed attribute information output by the second post-processing module 116 to the buffer 117, the display engine 118 reads the geometric information and attribute information from the buffer as needed, and renders and displays the G-PCC encoded point cloud in the 3D scene based on the read geometric information and attribute information.

[0153] Buffer API that supports media files of type G-PCC encoded point clouds.

[0154] After the media access function completes processing of the G-PCC encoded point cloud data via the pipeline, the media access function needs to transmit the processed data to the display engine in a standard array structure. This requires the processed G-PCC encoded point cloud data to be accurately stored in a buffer, a task completed by the buffer management module, which needs to receive buffer management instructions from the media access function or the display engine via the buffer API.

[0155] In some embodiments, the media access function can send buffer management instructions to the buffer management module via the buffer API. Here, the buffer management instructions are those sent by the display engine to the media access function via the media access function API.

[0156] In some embodiments, the display engine can send buffer management commands to the buffer management module via the buffer API.

[0157] In other words, the buffer management module may communicate with the media access function via the buffer API, or it may communicate with the display engine via the buffer API, and the purpose of communicating with the media access function or the display engine is to implement buffer management. When the buffer management module communicates with the media access function via the buffer API, the display engine must first send a buffer management command to the media access function via the media access function API, and the media access function must then send a buffer management command to the buffer management module via the buffer API. When the buffer management module communicates with the display engine via the buffer API, the display engine should generate a buffer management command based on the buffer management information parsed from the scene description file and send it to the buffer management module via the buffer API.

[0158] In some embodiments, the buffer management instruction may include one or more of the following: buffer creation instruction, buffer update instruction, and buffer release instruction.

[0159] A buffer management module that supports media files of type G-PCC encoded point clouds.

[0160] In the workflow of an immersive media scene description framework, after the media access function completes processing of G-PCC encoded point cloud data via the pipeline, the processed G-PCC encoded point cloud data needs to be passed to the display engine in a standard array structure. This requires the processed G-PCC encoded point cloud data to be accurately stored in a buffer, and this task is handled by the buffer management module.

[0161] The buffer management module handles management operations such as buffer creation, updating, and release, receiving command instructions via the buffer API. Buffer management rules are recorded in the scene description document, parsed by the display engine, and finally sent to the buffer management module by the display engine or media access functions. Media files, after being processed by media access functions, must be stored in appropriate buffers and used by the display engine. The role of buffer management is to manage these buffers to match the format of the processed media data without compromising it. For specific design methods of the media management module, refer to the design of the display engine and media access functions.

[0162] Based on the above, some embodiments of the present application provide a method for generating a scene description file, and as shown in Figure 12, the method for generating a scene description file includes the following steps S121-S123.

[0163] S121 determines the type of media file in the 3D scene to be rendered.

[0164] The media file types in the embodiments of this application may include one or more of the following: G-PCC encoded point clouds, V-PCC encoded point clouds, haptic media files, 6DoF videos, MIV videos, etc., and any number of media files of the same type may be included. For example, the rendering target 3D scene may include only one media file whose type is a G-PCC encoded point cloud. Alternatively, for example, the rendering target 3D scene may include one media file whose type is a G-PCC encoded point cloud and one media file whose type is a V-PCC encoded point cloud. Alternatively, for example, the rendering target 3D scene may include two media files whose type is a G-PCC encoded point cloud and one haptic media file.

[0165] In step S121 above, if the type of the target media file in the 3D scene to be rendered is a G-PCC encoded point cloud, the following step S122 is executed.

[0166] S122, Based on the description information of the target media file, the corresponding Target media description module Generates.

[0167] In some embodiments, the description information for the target media file includes one or more of the following: the name of the target media file, whether or not the target media file needs to be played automatically, whether or not the target media file needs to be played in a cycle, the capsule format of the target media file, the type of code stream of the target media file, and the encoding parameters of the target media file.

[0168] In some embodiments, step S122 (based on the description information of the target media file, the corresponding to the target media file Target media description module The process of generating ( ) includes at least one of the following steps 1221-1229.

[0169] In step 1221, a media name syntax element (name) is added to the target media description module, and the value of the media name syntax element is set based on the name of the target media file.

[0170] For example, if the media name syntax element in the target media description module is "name" and the name of the target media file is "G-PCC example", then the syntax element "name" is added to the target media description module and the value of the syntax element "name" is set to "G-PCC example".

[0171] In step 1222, an autoplay syntax element is added to the target media description module, and the value of the autoplay syntax element is set according to whether the target media file should be autoplayed.

[0172] For example, if the autoplay syntax element in the target media description module is "autoplay" and the target media file needs to be played automatically, the syntax element "autoplay" is added to the target media description module and the value of the syntax element "autoplay" is set to "true".

[0173] Furthermore, for example, if the autoplay syntax element in the target media description module is "autoplay" and the target media file does not need to be autoplayed, the syntax element "autoplay" is added to the target media description module and the value of the syntax element "autoplay" is set to "false".

[0174] In step 1223, within the target media description module circulation regeneration Syntax element (loop) addition The value of the loop playback syntax element is set according to whether the target media file needs to be played in a loop.

[0175] For example, if the autoplay syntax element in the target media description module is "loop" and it is necessary to play the target media file in a loop, the syntax element "loop" is added to the target media description module and the value of the syntax element "loop" is set to "true".

[0176] Furthermore, for example, if the autoplay syntax element in the target media description module is "loop" and the target media file does not need to be played back in a loop, the syntax element "loop" is added to the target media description module and the value of the syntax element "loop" is set to "false".

[0177] In step 1224, options (alternatives) are added to the target media description module.

[0178] In step 1225, a media type syntax element (mime Type) is added to the options (alternatives), and the value of the media type syntax element is set to the capsule format value corresponding to the G-PCC encoded point cloud.

[0179] In some examples, the capsule format corresponding to the G-PCC encoded point cloud is MP4, and the capsule format value corresponding to the G-PCC encoded point cloud is application / mp4.

[0180] For example, if the media type syntax element is "mimeType" and the capsule format value corresponding to the G-PCC encoded point cloud is "application / mp4", the syntax element "mimeType" is added to the options of the target media description module, and the value of the syntax element "mimeType" is set to "application / mp4".

[0181] In step 1226, the uniform resource identifier syntax element (uri) is added to the options (alternatives), and the value of the uniform resource identifier syntax element is set to the access address of the target media file.

[0182] For example, if the uniform resource identifier syntax element is "uri" and the access address of the target media file is "http: / www.exp.com / G-PCCexp.mp4", then the syntax element "uri" is added to the options of the target media description module, and the value of the syntax element "uri" is set to http: / www.exp.com / G-PCCexp.mp4.

[0183] In step 1227, track arrays are added to the alternatives.

[0184] In step 1228, a first track index syntax element (track) is added to the track array (track) of the options (alternatives) of the target media description module, and the value of the first track index syntax element (track) is set according to the encapsulation method of the target media file.

[0185] In some embodiments, the step of setting the value of the first track index syntax element (track) based on the encapsulation method of the target media file is to set the value of the first track index syntax element to the index value of the code stream track of the target media file if the target media file is a single-track encapsulation file, and to set the value of the first track index syntax element to the index value of the geometric code stream track of the target media file if the target media file is a multi-track encapsulation file.

[0186] In other words, if an encoded G-PCC coded point cloud is referenced by the scene description file as one of the MPEG_media.alternative.tracks, and the referenced track satisfies the definition of a track in ISOBMFF, then for single-track encapsulated G-PCC data, the track referenced in MPEG_media is the G-PCC code stream track. For example, if G-PCC data is encapsulated in a single MIHS track by ISOBMFF, the track referenced in MPEG_media is this barcode stream track. In the case of multi-track encapsulated G-PCC data, the track referenced in MPEG_media is the G-PCC geometric code stream track.

[0187] In the embodiments of the present application, the encapsulation method for the G-PCC coded point cloud includes single-track capsules and multi-track capsules. Here, a single-track capsule refers to a encapsulation method in which the geometric code stream and attribute code stream of the G-PCC coded point cloud are encapsulated in the same code stream track, and a multi-track capsule refers to a encapsulation method in which the geometric code stream and attribute code stream of the G-PCC coded point cloud are each encapsulated in multiple code stream tracks.

[0188] In step 1229, a codec parameter syntax element (codecs) is added to the optional track array of the target media description module, and the value of the codec parameter syntax element is set based on the encoding parameters of the target media file, the type of code stream of the target media file, and the ISO / IEC 23090-18G-PCC data transmission standard.

[0189] For example, the ISO / IEC 23090-18 G-PCC data transmission standard specifies that when a G-PCC coded point cloud uses DASH capsules, and G-PCC pre-selection signaling is used in an MPD file, the "codecs" attribute of the pre-selection signaling should be set to 'gpc1' to indicate that the pre-selection medium is a geometric point cloud; and when there are multiple G-PCC Tile tracks in a G-PCC container, the "codecs" attribute of the Main G-PCC Adaptation Set should be set to 'gpcb' or 'gpeb' to indicate that the adaptive set contains G-PCC Tile base track data. If Tile Component Adaptation Sets signal only a single G-PCC component data, the "codecs" attribute of the Main G-PCC adaptivesset should be set to 'gpcb'. If Tile Component Adaptation Sets signal all G-PCC component data, the "codecs" attribute of the Main G-PCC Adaptation Set should be set to 'gpeb'. When using G-PCC Tile pre-selection signaling in an MPD file, the "codecs" attribute of the pre-selection signaling should be set to 'gpt1', indicating that the pre-selection media is a geometric point cloud fragment, and the G-PCC encoded point cloud employs a DASH capsule. When using G-PCC pre-selection signaling in an MPD file, the value of "codecs" in "tracks" of "alternatives" in the target media description module can be set to "gpc1".

[0190] For example, if the media files in the 3D scene to be rendered consist only of a target media file of type G-PCC encoded point cloud, the capsule format value corresponding to the G-PCC encoded point cloud is "application / mp4", the name of the target media file is "G-PCC example", the target media file is automatically and cyclically played, the access address of the target media file is http: / / www.exp.com / G-PCCexp.mp4, the target media file is a single-track capsule file, the index value of the code stream track of the target media file is 1, the target media file employs a DASH capsule, and G-PCC pre-selection signaling is used in the MPD file, then the target media description module corresponding to the target media file can be shown as follows. JPEG0007872437000024.jpg96164

[0191] In S123, the target media description module is added to the media list (media) of the MPEG media (MPEG_media) of the scene description file of the rendering target 3D scene.

[0192] Here, the target media description module is a media description module generated based on the description information of the target media file.

[0193] For example, if the media files in the 3D scene to be rendered include only one type of target media file which is a G-PCC encoded point cloud, the capsule format value corresponding to the G-PCC encoded point cloud is application / mp4, the name of the target media file is "G-PCCexample1", the target media file is autoplayed and looped, the access address of the target media file is "uri":http: / www.exp.com / G-PCCexp.mp4, the target media file is a single-track capsule file, the index value of the code stream track of the target media file is 1, the target media file is DASH encapsulated, and G-PCC pre-selection signaling is used in the MPD file, then the MPEG media of the scene description file can be shown as follows. JPEG0007872437000025.jpg122165

[0194] In some embodiments, the rendering target 3D scene may further include multiple media files, and one or more of the media files are of type G-PCC encoded point clouds. When generating the scene description file, it is necessary to add a media description module corresponding to the media file of type G-PCC encoded point cloud based on the above embodiments, and to add media description modules corresponding to other types of media files based on the scene description file generation method for other types of media files.

[0195] For example, if the media files in the 3D scene to be rendered include a target media file of type G-PCC encoded point cloud and one haptic media file, the capsule format value corresponding to the G-PCC encoded point cloud is "application / mp4", the name of the target media file is "G-PCC example", the target media file is autoplayable and loopable, the access address of the target media file is "uri":http: / www.exp.com / G-PCCexp.mp4, the target media file is a single-track capsule file, the index value of the code stream track of the target media file is 1, the target media file is DASH encapsulated, and G-PCC pre-selection signaling is used in the MPD file, then the MPEG media of the scene description file can be shown as follows. JPEG0007872437000026.jpg111164JPEG0007872437000027.jpg92164

[0196] In the example above, the media list (media) for MPEG media contains two sets of square brackets: the first set of square brackets (lines n+2-n+18) contains the media description module corresponding to the target media file whose type is a G-PCC encoded point cloud, and the second set of square brackets (lines n+19-n+35) contains the media description module corresponding to the haptic media file.

[0197] The method for generating a scene description file according to the embodiment of the present application, when generating a scene description file for a 3D scene to be rendered, first determines the type of media file in the 3D scene to be rendered, and if the type of target media file in the 3D scene to be rendered is a G-PCC encoded point cloud, then, based on the description information of the target media file, the method for generating a scene description file for the target media file Target media description moduleThe present embodiment generates a target media description module and adds it to the media list of the MPEG media in the scene description file of the rendering target 3D scene.In the present embodiment, if the media file in the rendering target 3D scene includes a target media file of the type G-PCC encoded point cloud, the present embodiment generates a target media description module corresponding to the target media file based on the description information of the target media file and adds the target media description module to the media list of the MPEG media in the scene description file of the rendering target 3D scene. Target media description module By adding this, a media description module corresponding to the target media file can be added to the media description module list of the MPEG media in the scene description file. Therefore, the embodiment of the present invention can generate a scene description file containing a 3D scene whose type is a G-PCC encoded point cloud, and realizes support for media files whose type is a G-PCC encoded point cloud by the scene description file.

[0198] In some embodiments, the method for generating the scene description file further includes the following steps:

[0199] The scene description module (scene) corresponding to the rendering target 3D scene is added to the scene list (scenes) of the scene description file, and the index value of the node description module corresponding to the node in the rendering target scene is added to the node list (nodes) of the target scene description module.

[0200] For example, if the rendering target 3D scene includes two nodes, and the index values ​​of the node description modules (nodes) corresponding to those two nodes are 0 and 1 respectively, the target scene description module corresponding to the rendering target 3D scene added to the scene description file can be shown as follows. JPEG0007872437000028.jpg59165

[0201] In the example above, the 3D scene to be rendered contains two nodes, and the index values ​​of the node description module corresponding to the two nodes are 0 and 1, respectively. Therefore, the two index values ​​0 and 1 are added to the node list (nodes) of the scene description module corresponding to the 3D scene to be rendered.

[0202] In some embodiments, the method for generating the scene description file further includes the following steps:

[0203] The node description module corresponding to the nodes in the rendering target scene is added to the node list (nodes) of the scene description file, and the index value of the mesh description module corresponding to the 3D mesh mounted on the node is added to the mesh index list (mesh) of the node description module.

[0204] In some embodiments, the method for generating the scene description file further includes the following steps:

[0205] A node name syntax element (name) is added to the node description module, and the value of the node name syntax element (name) in the corresponding node description module is set based on the name of the node.

[0206] For example, the rendering target 3D scene includes two nodes, named G-PCCexp_node1 and G-PCCexp_node2, with the index values ​​of the mesh description modules corresponding to the 3D mesh contained in node G-PCCexp_node1 being 0 and 1 respectively, and the index value of the mesh description module corresponding to the 3D mesh contained in node G-PCCexp_node2 being 2. The node list (nodes) portion of the scene description file can be shown as follows. JPEG0007872437000029.jpg65165

[0207] In the example above, the node list (nodes) in the scene description file corresponding to the 3D scene to be rendered contains two node description modules. The first node description module is contained within the square brackets on lines n+2-n+5, and the second node description module is contained within the square brackets on lines n+6-n+9. The value of the node name syntax element (name) in the first node description module is set to correspond to the node name "G-PCCexp_node1", and the value of the mesh index syntax element (mesh) in the first node description module is set to correspond to index values ​​0 and 1 of the mesh description module of the 3D mesh mounted on the node. The value of the node name syntax element (name) in the second node description module is set to correspond to the node name "G-PCCexp_node2", and the value of the mesh index syntax element (mesh) in the second node description module is set to correspond to index value 2 of the mesh description module of the 3D mesh mounted on the node.

[0208] In some embodiments, the method for generating the scene description file further includes the following steps:

[0209] A mesh description module (mesh) corresponding to the 3D mesh in the rendering target scene is added to the mesh list (meshes) of the scene description file, syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the mesh description module are added to the mesh description module, and the values ​​of the syntax elements corresponding to each type of data are set to the index values ​​of the accessor description module corresponding to the accessor for accessing each type of data.

[0210] In the embodiments of this application, the data included in the 3D mesh may include one or more of the following: geometric coordinates (position), color values ​​(color), normal vectors (normal), tangent vectors (tangent), texture coordinates (texcoord), joints (joints), and weights (weights).

[0211] In some embodiments, adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the mesh description module includes the following steps:

[0212] An extensions list is added to the primitives of the mesh description module corresponding to the 3D mesh in the target media file, a target extension array is added to the extensions list, and syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the target extension array are added.

[0213] In some embodiments, the target extension sequence may be MPEG_primitve_GPCC.

[0214] In some embodiments, adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the target extension array includes adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the target extension array, based on syntax elements in a first set of syntax elements. Here, the first set of syntax elements is a set of syntax elements supported in the attributes of primitives of the mesh description module of a scene description file as defined in the ISO / IEC 23090-14 MPEG-I scene description standard.

[0215] Specifically, the syntax elements supported by the primitive attributes of the mesh description module in a scene description file as defined in the ISO / IEC 23090-14 MPEG-I scene description standard include position, color_n, normal, tangent, texcoord, joints, and weights, and therefore the first set of syntax elements is {position, color_n, normal, tangent, texcoord, joints, weights}.

[0216] For example, if a certain 3D mesh includes geometric coordinates and color data, and the index value of the accessor description module corresponding to the accessor for accessing the geometric coordinates is 0, and the index value of the accessor description module corresponding to the accessor for accessing the color data is 1, then, after adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the target extension array based on a first set of syntax elements, the mesh description module corresponding to the 3D mesh can be represented as follows: JPEG0007872437000030.jpg75164

[0217] In some embodiments, adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the target extended array includes adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the target extended array based on a second set of syntax elements consisting of syntax elements corresponding to a pre-configured G-PCC coded point cloud.

[0218] Exemplary, the syntax elements corresponding to the G-PCC coded point cloud may include G-PCC_position, G-PCC_color_n, G-PCC_normal, G-PCC_tangent, G-PCC_texcoord, G-PCC_joints, and G-PCC_weights, and correspondingly, the second set of syntax elements is {G-PCC_position, G-PCC_color_n, G-PCC_normal, G-PCC_tangent, G-PCC_texcoord, G-PCC_joints, G-PCC_weights}.

[0219] For example, if a certain 3D mesh includes geometric coordinates and color data, and the index value of the accessor description module corresponding to the accessor for accessing the geometric coordinates is 0, and the index value of the accessor description module corresponding to the accessor for accessing the color data is 1, then, after adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the target extension array based on a second set of syntax elements, the mesh description module corresponding to the 3D mesh can be represented as follows: JPEG0007872437000031.jpg75164

[0220] In some embodiments, adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the mesh description module includes adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the mesh description module to the attributes of the primitives of the mesh description module.

[0221] In some embodiments, adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the mesh description module to the attributes of the primitives of the mesh description module includes adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the mesh description module to the attributes of the primitives of the mesh description module based on the first set of syntax elements. Here, the first set of syntax elements is a set of syntax elements supported in the attributes of the primitives of the mesh description module in a scene description file as defined in the ISO / IEC 23090-14 MPEG-I scene description standard.

[0222] In other words, for all 3D meshes in the scene description file (including 3D meshes in media files of type G-PCC and 3D meshes in other types of media files), a syntax element is added to the attributes of the corresponding primitives in the mesh description module, based on the syntax element in the same set of syntax elements.

[0223] For example, a certain 3D mesh includes geometric coordinates and color data, the index value of the accessor description module corresponding to the accessor for accessing the geometric coordinates is 1, and the index value of the accessor description module corresponding to the accessor for accessing the color data is 2. After adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the target extension array based on the first set of syntax elements, the mesh description module corresponding to the 3D mesh can be shown as follows: JPEG0007872437000032.jpg76165

[0224] In some embodiments, adding syntax elements to the attributes of the primitives of the mesh description module that correspond to each type of data contained in the 3D mesh corresponding to the mesh description module includes adding syntax elements to the attributes of the primitives of the first mesh description module that correspond to each type of data contained in the corresponding 3D mesh, based on syntax elements in a first set of syntax elements, and adding syntax elements to the attributes of the primitives of the second mesh description module that correspond to each type of data contained in the corresponding 3D mesh, based on syntax elements in a second set of syntax elements.

[0225] Here, the first mesh description module is a mesh description module corresponding to a 3D mesh in a media file whose type is a G-PCC encoded point cloud, and the second mesh description module is not a mesh description module corresponding to a 3D mesh in a media file whose type is a G-PCC encoded point cloud.

[0226] In some embodiments, the first set of syntax elements is a set formed by syntax elements supported in the attributes of primitives of the mesh description module of a scene description file as defined in the ISO / IEC 23090-14 MPEG-I scene description standard, and the second set of syntax elements is a set formed by syntax elements corresponding to a pre-configured G-PCC coded point cloud.

[0227] In other words, when adding syntax elements corresponding to each type of data contained in a corresponding 3D mesh to the attributes of a primitive in a mesh description module, it is necessary to divide the 3D mesh in the scene description file into two types depending on whether the 3D mesh belongs to a 3D mesh in a G-PCC type media file or not. For 3D meshes in media files that are not of type G-PCC encoded point clouds, syntax elements corresponding to each type of data contained therein are added to the attributes of the primitive in the corresponding mesh description module based on the syntax elements in the first set of syntax elements. For 3D meshes in media files that are of type G-PCC encoded point clouds, syntax elements corresponding to each type of data contained therein are added to the attributes of the primitive in the corresponding mesh description module based on the syntax elements in the second set of syntax elements.

[0228] For example, the scene description file contains two 3D meshes, named GPCCexample_mesh1 and GPCCexample_mesh2, respectively. Here, GPCCexample_mesh1 is a 3D mesh in a media file of type G-PCC, contains geometric coordinates and color data, the index value of the accessor description module corresponding to the accessor for accessing the geometric coordinates of GPCCexample_mesh1 is 0, the index value of the accessor description module corresponding to the accessor for accessing the color data of GPCCexample_mesh1 is 1, GPCCexample_mesh2 is a 3D mesh in a media file of type G-PCC, contains geometric coordinates and color data, the index value of the accessor description module corresponding to the accessor for accessing the geometric coordinates of GPCCexample_mesh2 is 2, and the index value of the accessor description module corresponding to the accessor for accessing the color data of GPCCexample_mesh2 is 3. After adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the target extension array based on the above embodiment, the mesh list (meshes) in the scene description file can be shown as follows. JPEG0007872437000033.jpg140164

[0229] In some embodiments, the method for generating the scene description file further includes the following steps:

[0230] Based on the name of the 3D mesh, the value of the mesh name syntax element (name) in the mesh description module corresponding to the 3D mesh is set.

[0231] In some embodiments, the method for generating the scene description file further includes the following steps:

[0232] Depending on the type of data contained in the 3D mesh, the syntax elements included in the attributes of the primitives in the mesh description module corresponding to the 3D mesh are set.

[0233] In some embodiments, the method for generating the scene description file further includes the following steps:

[0234] Based on the type of topology structure of the 3D mesh, the values ​​of the syntax elements used to describe the topology type of the 3D mesh in the mesh description module corresponding to the 3D mesh are set.

[0235] In some embodiments, the syntax element for describing the topology type of a 3D mesh in a mesh description module corresponding to a 3D mesh is "mode".

[0236] In some embodiments, the method for generating the scene description file further includes the following steps:

[0237] An accessor description module (accessor) corresponding to the target accessor is added to the buffer list (accessor) of the scene description file. Here, the target accessor is an accessor for accessing the decoded data of the target media file.

[0238] In some embodiments, the method for generating the scene description file further includes adding a buffer description module (buffer) corresponding to a target buffer to the buffer list (buffers) of the scene description file. Here, the target buffer is a buffer for storing the decoded data of the target media file.

[0239] In some embodiments, adding a buffer description module (buffer) corresponding to the target buffer to the buffer list (buffers) of the scene description file includes at least one of the following steps a1-a5.

[0240] In step a1, a byte length syntax element (byteLength) is added to the buffer description module corresponding to the target buffer, and the value of the byte length syntax element is set to the byte length of the target media file.

[0241] For example, if the amount of data in the G-PCC encoded point cloud is 15,000, the value of "byteLenth" in the buffer description module is set to "15,000".

[0242] In step a2, an MPEG ring buffer (MPEG_buffer_circular) is added to the buffer description module corresponding to the target buffer.

[0243] In step a3, a segment count syntax element (count) is added to the MPEG ring buffer, and the corresponding value of the segment count syntax element (count) is set based on the number of segments stored in the target buffer.

[0244] For example, if the number of storage segments in the ring buffer is 8, the "count" and its value in the ring buffer are set to "count":8.

[0245] In step a4, the media index syntax element (media) is added to the MPEG ring buffer, and the value of the media index syntax element (media) is set based on the index value of the target media description module.

[0246] For example, when the index value of the target media description module is 0, "media" and its value in the description module of the ring buffer are set to "media":0.

[0247] In step a5, add a second track index syntax element (tracks) to the MPEG ring buffer, and set the value of the second track index syntax element (tracks) according to the track index value of the source data of the data stored in the target buffer.

[0248] For example, when the index value of the code stream track to which the data stored in the ring buffer belongs is 1, "tracks" and its value in the description module of the ring buffer can be set to "tracks":"#trackIndex=1".

[0249] Exemplarily, adding a buffer description module corresponding to the target buffer to the buffer list of the scene description file includes any of the above steps a1 - a5, and the byte length of the target media file is 9000, the number of storage segments of a certain target buffer is 8, the index value of the media description module corresponding to the target media file is 1, and the track index value of the source data of the data stored in the MPEG ring buffer is 1. The buffer description module corresponding to the target buffer added to the buffer list of the scene description file can be shown as follows. JPEG0007872437000034.jpg55164

[0250] In some embodiments, the method for generating the scene description file further includes adding a buffer slice description module corresponding to the buffer slice of the target buffer to the buffer slice list (buffer Views) of the scene description file.

[0251] In some embodiments, adding a buffer slice description module corresponding to the buffer slice of the target buffer to the buffer slice list of the scene description file includes at least one of the following steps b1-b3.

[0252] In step b1, a buffer index syntax element (buffer) is added to the buffer slice description module corresponding to the buffer slice of the target buffer, and the value of the buffer index syntax element (buffer) is set based on the index value of the buffer description module corresponding to the target buffer to which the buffer slice belongs.

[0253] For example, if the index value of the buffer description module corresponding to a certain buffer is 2, then the "buffer" and its value in that buffer slice description module will be set to "buffer":2.

[0254] In step b2, a second byte length syntax element is added to the buffer slice description module corresponding to the buffer slice of the target buffer, and the value of the second byte length syntax element is set based on the capacity of the buffer slice.

[0255] In step b3, an offset amount syntax element (byte Offset) is added to the buffer slice description module corresponding to the buffer slice of the target buffer, and the value of the offset amount syntax element is set based on the offset of the stored data of the corresponding buffer slice.

[0256] For example, if the data range of a buffer slice containing the buffer is [1,12000], then based on steps b2 and b3 above, the "byteLenth" and its value in the buffer slice description module corresponding to the buffer slice are set to "byteLenth":12000, and the "byteOffset" and its value in the buffer slice description module corresponding to the buffer slice are set to "byteOffset":0. If the data range of a buffer slice containing the buffer is [12001,15000], then based on steps b2 and b3 above, the "byteLenth" and its value in the buffer slice description module corresponding to the buffer slice are set to "byteLenth":3000, and the "byteOffset" and its value in the buffer slice description module corresponding to the buffer slice are set to "byteOffset":12000.

[0257] For example, adding a buffer slice description module corresponding to the buffer slice of the target buffer to the buffer slice list (buffer Views) of the scene description file includes any of steps b1-b3, and if the index value of the buffer description module corresponding to a certain target buffer is 1, the capacity of the target buffer is 8000, and the target buffer contains two buffer slices, the capacity of the first buffer slice is 6000 and the offset is 0, and the capacity of the second buffer slice is 2000 and the offset is 6001, then adding a buffer slice description module corresponding to the buffer slice of the target buffer to the buffer slice list of the scene description file is as follows: JPEG0007872437000035.jpg63165

[0258] In some embodiments, the method for generating the scene description file further includes adding an accessor description module corresponding to a target accessor to the accessors of the scene description file, where the target accessor is an accessor for accessing the decoded data of the target media file.

[0259] In some embodiments, adding an accessor description module corresponding to a target accessor to the accessors in the scene description file includes at least one of the following steps c1-c6.

[0260] In step c1, a data type syntax element (component Type) is added to the accessor description module corresponding to the target accessor, and the value of the corresponding data type syntax element is set based on the type of data accessed by the target accessor.

[0261] For example, if the type of data accessed by a certain accessor is 5126, the data type syntax element and its value in the accessor description module corresponding to that accessor are set to "componentType":5126.

[0262] In step c2, an accessor type syntax element (type) is added to the accessor description module corresponding to the target accessor, and the value of the accessor type syntax element is set based on the pre-configured accessor type.

[0263] For example, if an accessor accesses an accessor type of "VEC3", the accessor type syntax element (type) and its value in the accessor description module corresponding to that accessor are set to "type":"VEC3".

[0264] In step c3, a data count syntax element (count) is added to the accessor description module corresponding to the target accessor, and the value of the corresponding accessor type syntax element is set based on the type of the target accessor.

[0265] In step c4, the MPEG time-varying accessor (MPEG_accessor_timed) is added to the accessor description module corresponding to the target accessor.

[0266] In step c5, a buffer slice index syntax element (bufferView) is added to the MPEG time-varying accessor, and the value of the corresponding slice index syntax element is set based on the index value of the buffer slice description module corresponding to the buffer slice that stores the data accessed by the target accessor.

[0267] For example, if the index value of the buffer slice description module corresponding to the buffer slice to which the data accessed by a certain accessor belongs is 3, the buffer slice index syntax element and its value in the MPEG time-varying accessor of the accessor description module corresponding to that target accessor are set to "bufferView":3.

[0268] In step c6, a time-varying syntax element (immutable) is added to the MPEG time-varying accessor, and the value of the time-varying syntax element is set based on whether or not the value of the syntax element in the corresponding target accessor changes over time.

[0269] In some embodiments, when the value of the syntax element in a target accessor does not change over time, set the time-varying syntax element and its value in the MPEG time-varying accessor of the accessor description module corresponding to the target accessor to "immutable": true. When the value of the syntax element in a target accessor changes over time, set the time-varying syntax element and its value in the MPEG time-varying accessor of the accessor description module corresponding to the target accessor to "immutable": false.

[0270] Exemplarily, adding the accessor description module corresponding to the target accessor for accessing the data in the buffer slice of the target buffer to the accessors of the scene description file includes any of the above steps c1 - c6. The type of data accessed by a target accessor is 5121, the accessor type of the target accessor is VEC2, the number of data accessed by the target accessor is 4000, the index value of the buffer slice description module corresponding to the buffer slice storing the data that the target accessor needs to access is 1, and when the value of the syntax element in the corresponding accessor does not change over time, the accessor description module corresponding to the target accessor added to the accessors of the scene description file can be shown as follows. JPEG0007872437000036.jpg58164

[0271] In some embodiments, the method for generating the scene description file further includes the following steps.

[0272] If a digital asset description module (asset) is added to the scene description file, and a version syntax element (version) is added to the digital asset description module, and the scene description file creates a scene description statement based on the glTF 2.0 version, the value of the version syntax element is set to 2.0.

[0273] For example, a digital asset description module added to the scene description file can be shown as follows: JPEG0007872437000037.jpg30164

[0274] In some embodiments, the method for generating the scene description file further includes the following steps:

[0275] The extensionsUsed module is added to the scene description file, and extensions for the glTF2.0 version of the MPEG used by the scene description file are added to the extensionsUsed module.

[0276] Exemplary, the MPEG extensions used in a scene description file include MPEG media (MPEG_media), MPEG ring buffer (MPEG_buffer_circular), and MPEG timed accessor (MPEG_accessor_timed), and the extension usage description module added to the scene description file can be shown as follows: JPEG0007872437000038.jpg41164

[0277] In some embodiments, the method for generating the scene description file further includes the following steps:

[0278] A scene statement (scene) is added to the aforementioned scene description file, and the value of the scene statement is set to the index value of the scene description module corresponding to the scene to be rendered.

[0279] For example, if the index value of the scene description module corresponding to the scene to be rendered is 0, adding a scene statement to the scene description file can be shown as follows: JPEG0007872437000039.jpg18164

[0280] Some embodiments of the present invention further provide a method for analyzing a scene description file, as shown in Figure 13, which includes the following steps S131-S133.

[0281] In S131, the scene description file for the 3D scene to be rendered is obtained.

[0282] Here, the 3D scene to be rendered includes a target media file whose type is a G-PCC encoded point cloud.

[0283] In the embodiments of the present application, the rendering target 3D scene may include one or more media files, and when the rendering target 3D scene includes multiple media files, the type of one or more of the multiple media files may be a G-PCC encoded point cloud. When the rendering target 3D scene includes multiple target media files of the type G-PCC encoded point cloud, the analysis method according to the embodiments of the present application can be performed on each of the target media files whose type is a G-PCC encoded point cloud.

[0284] In S132, the target media description module corresponding to the target media file is obtained from the media list (media) of the MPEG media (MPEG_media) in the scene description file.

[0285] For example, a target media description module corresponding to the target media file can be shown as follows: JPEG0007872437000040.jpg78164

[0286] In S133, the description information of the target media file is obtained based on the target media description module.

[0287] In some embodiments, step S133 (obtaining description information of the target media file based on the target media description module) includes at least one of the following steps 1331-1337.

[0288] In step 1331, the name of the target media file is obtained based on the value of the media name syntax element (name) in the target media description module.

[0289] For example, if the media name syntax element and its value in the target media description module are "name":"GPCC example", it can be determined that the name of the target media file is GPCC example.

[0290] In step 1332, it is determined whether the target media file should be autoplayed based on the value of the autoplay syntax element (autoplay) in the target media description module.

[0291] In some embodiments, determining whether a target media file should be autoplayed based on the value of the autoplay syntax element (autoplay) in the target media description module includes determining that the target media file should be autoplayed if the autoplay syntax element (autoplay) and its value in the target media description module are "autoplay": true, and determining that the target media file does not need to be autoplayed if the autoplay syntax element (autoplay) and its value in the target media description module are "autoplay": false.

[0292] In step 1333, it is determined whether the target media file needs to be played back in a loop based on the value of the loop playback syntax element (loop) in the target media description module.

[0293] In some embodiments, determining whether the target media file needs to be played back in a loop based on the value of the loop playback syntax element (loop) in the target media description module includes determining that the target media file needs to be played back in a loop if the loop playback syntax element (loop) and its value in the target media description module are "loop": true, and determining that the target media file does not need to be played back in a loop if the loop playback syntax element (loop) and its value in the target media description module are "loop": false.

[0294] In step 1334, the encapsulation format of the target media file is obtained based on the value of the media type syntax element (mime Type) in the options (alternatives) of the target media description module.

[0295] If the media file type is a G-PCC encoded point cloud, the value of the media type syntax element (mime Type) in the media description module corresponding to the media file is set to the capsule format value corresponding to the G-PCC encoded point cloud. Furthermore, the capsule format value corresponding to the G-PCC encoded point cloud may be "application / mp4". Therefore, if the capsule format value corresponding to the G-PCC encoded point cloud is "application / mp4", it can be determined that the capsule format of the target media file is MP4.

[0296] In step 1335, the access address of the target media file is obtained based on the value of the only address identifier syntax element (uri) in the alternatives of the target media description module.

[0297] For example, if the only address identifier syntax element (uri) in the options (alternatives) of the target media description module and its value is "uri": "http: / www.example.com / GPCCexample.mp4", then it can be determined that the access address of the target media file is http: / www.example.com / GPCCexample.mp4.

[0298] In step 1336, track information of the target media file is obtained based on the value of the first track index syntax element (track) in the track array (track) of the options (alternatives) of the target media description module.

[0299] In some embodiments, obtaining track information for the target media file based on the value of a first track index syntax element (track) in the track array (tracks) of the alternatives (alternatives) of the target media description module includes determining the value of the first track index syntax element as the index value of the code stream track of the target media file if the capsule file of the target media file is a single-track capsule file, and determining the value of the first track index syntax element as the index value of the geometric code stream track of the target media file if the target media file is a multi-track capsule file.

[0300] In step 1337, the code stream type and decoding parameters of the target media file are determined based on the values ​​of the codec parameter syntax elements (codecs) in the track array (tracks) of the alternatives (alternatives) of the target media description module and the ISO / IEC 23090-18G-PCC data transmission standard.

[0301] In some embodiments, step 1337 (determining the code stream type and decoding parameters of the target media file based on the values ​​of the codec parameter syntax elements (codecs) in the track arrays (tracks) of the alternatives (alternatives) of the target media description module and the ISO / IEC 23090-18G-PCC data transmission standard) includes the following steps 13371 and 13372.

[0302] In step 13371, the type of code stream and encoding parameters of the target media file are determined based on the values ​​of the codec parameter syntax elements (codecs) in the track array (tracks) of the alternatives (alternatives) of the target media description module and the ISO / IEC 23090-18G-PCC data transmission standard.

[0303] As specified in the ISO / IEC 23090-18 G-PCC data transmission standard, when the G-PCC encoded point cloud employs DASH capsules, and G-PCC pre-selection signaling is used in the MPD file, the "codecs" attribute of the pre-selection signaling should be set to 'gpc1' to indicate that the pre-selection medium is based on geometric point clouds, and when there are multiple G-PCC Tile tracks in the G-PCC container, the "codecs" attribute of the Main G-PCC Adaptation Set should be set to 'gpcb' or 'gpeb' to indicate that the adaptive set contains G-PCC Tile base track data. If Tile Component Adaptation Sets signal only a single G-PCC component data, the "codecs" attribute of the Main G-PCC adaptivesset should be set to 'gpcb'. If Tile Component Adaptation Sets signal all G-PCC component data, the "codecs" attribute of the Main G-PCC Adaptation Set should be set to 'gpeb'. When using G-PCC Tile pre-selection signaling in an MPD file, the "codecs" attribute of the pre-selection signaling should be set to 'gpt1' to indicate that the pre-selection media is based on geometric point cloud fragments. When the G-PCC encoded point cloud employs DASH capsules and G-PCC pre-selection signaling is used in an MPD file, the value of "codecs" in "tracks" of "alternatives" in the target media description module can be set to 'gpc1'. Therefore, the capsule scheme and encoding parameters of the target media file can be determined based on the value of the codec parameter syntax element (codecs) in the track array (tracks) of the options (alternatives) of the target media description module and the ISO / IEC23090-18 G-PCC data transmission standard.

[0304] In step 13372, the decoding parameters for the target media file are determined based on the encoding parameters of the target media file.

[0305] Since the decoding process and encoding process of the target media file are inverse operations of each other, the decoding parameters of the target media file can be determined based on the encoding parameters of the target media file.

[0306] For example, a target media description module corresponding to the target media file can be shown as follows: JPEG0007872437000041.jpg80164

[0307] In this case, the description information of the target media file obtained by the target media description module includes that the name of the target media file is AAAA, that the target media file does not need to be autoplayed but needs to be played in a loop, that the capsule format of the target media file is MP4, that the access address of the target media file is http: / www.bbb.com / AAAA.mp4, that the reference track of the target media file is a code stream track with an index value of 0, that the capsule / decapsulation scheme of the target media file is MP4, and that the codec parameter of the target media file is gpc1.

[0308] The method for analyzing a scene description file according to an embodiment of the present invention can obtain a scene description file of a rendering target 3D scene that includes a target media file of type G-PCC encoded point cloud, obtain a target media description module corresponding to the target media file from the media list of the MPEG media in the scene description file, and obtain description information of the target media file based on the target media description module. The method for analyzing a scene description file according to an embodiment of the present invention can obtain description information of the target media file based on the target media description module, and further render and display a rendering target 3D scene that includes a target media file of type G-PCC encoded point cloud based on the description information of the target media file. Therefore, the embodiment of the present invention provides a method for analyzing a scene description file of a 3D scene that includes a media file of type G-PCC encoded point cloud, and realizes the analysis of a scene description file of a 3D scene that includes a G-PCC encoded point cloud.

[0309] In some embodiments, the method for analyzing the scene description file according to the above embodiment further includes the following:

[0310] The target scene description module (scene) corresponding to the rendering target 3D scene is obtained from the scene list (scenes) of the scene description file, and description information of the rendering target 3D scene is obtained based on the target scene description module.

[0311] In some embodiments, a scene statement and its index value can be obtained from the scene description file, and a target scene description module corresponding to the rendering target 3D scene can be obtained from the scene list of the scene description file based on the scene statement and its index value.

[0312] For example, if the scene statement and its index value are "scene":0, the first scene description module in the scene list of the scene description file can be obtained as the target scene description module corresponding to the rendering target 3D scene based on the scene statement and its index value.

[0313] In some embodiments, obtaining description information of the rendering target 3D scene based on the target scene description module includes determining the index value of the node description module corresponding to a node in the rendering target 3D scene based on the index value declared in the node index list (nodes) of the target scene description module.

[0314] For example, the target scene description module is as follows: JPEG0007872437000042.jpg39165

[0315] In this case, based on the index values ​​declared in the node index list (nodes) of the target scene description module, it can be determined that the rendering target 3D scene contains two nodes, where the index value of the node description module corresponding to one node is 0 (the first node description module in the node list), and the index value of the node description module corresponding to the other node is 1 (the second node description module in the node list).

[0316] In some embodiments, after determining the index value of the node description module corresponding to a node in the 3D scene to be rendered based on the index value declared in the node index list (nodes) of the target scene description module, the method for parsing the scene description file provided in the above embodiments includes the following:

[0317] Based on the index value of the node description module corresponding to the node in the rendering target 3D scene, the node description module corresponding to the node in the rendering target 3D scene is obtained from the node list (nodes) of the scene description file, and the description information of the node in the rendering target 3D scene is obtained based on the node description module corresponding to the node in the rendering target 3D scene.

[0318] For example, if the node index list of the target scene description module contains only index values ​​of 0, the first node description module in the node list of the scene description file is obtained as the node description module corresponding to the node in the rendering target 3D scene.

[0319] Furthermore, for example, if the node index list of the target scene description module contains index values ​​0 and 1, the first node description module and the second node description module are obtained from the node list of the scene description file and used as node description modules corresponding to the nodes in the rendering target 3D scene.

[0320] In some embodiments, obtaining description information of nodes in the rendering target 3D scene based on node description modules corresponding to nodes in the rendering target 3D scene includes at least one of the following steps a1 and a2.

[0321] In step a1, the name of the node in the 3D scene to be rendered is obtained based on the value of the node name syntax element (name) in the node description module corresponding to the node in the 3D scene to be rendered.

[0322] In step a2, the index value of the mesh description module corresponding to the 3D mesh mounted on the node in the 3D scene to be rendered is determined based on the index value declared in the mesh index list within the node description module corresponding to the node in the 3D scene to be rendered.

[0323] As an example, the node description module corresponding to a certain node is shown below. JPEG0007872437000043.jpg35165

[0324] In this case, based on step a1 above, it can be determined that the name of the node is GPCC example_node, and based on step a2 above, it can be determined that the index values ​​of the mesh description modules corresponding to the 3D mesh mounted on the node are 0 and 1, respectively.

[0325] In some embodiments, after determining the index value of the mesh description module corresponding to the 3D mesh mounted on the node in the rendering target 3D scene, the method for analyzing the scene description file according to the above embodiment further includes obtaining the mesh description module corresponding to the 3D mesh mounted on the node in the rendering target 3D scene from the mesh list (meshes) of the scene description file based on the index value of the mesh description module corresponding to the 3D mesh mounted on the node in the rendering target 3D scene, and obtaining description information of the 3D mesh mounted on the node in the rendering target 3D scene based on the mesh description module corresponding to the 3D mesh mounted on the node in the rendering target 3D scene.

[0326] For example, if the mesh index list of a node description module contains only index values ​​of 0, the first mesh description module in the mesh list of the scene description file is obtained as the mesh description module corresponding to the 3D mesh mounted on the node corresponding to that node description module.

[0327] Furthermore, for example, if the declared index values ​​in the mesh index list of a node description module include 1 and 2, the second and third mesh description modules are obtained from the mesh list of the scene description file as mesh description modules corresponding to the 3D mesh mounted on the node corresponding to the node description module.

[0328] In some embodiments, obtaining descriptive information of a 3D mesh mounted on a node in the 3D scene to be rendered, based on a mesh description module corresponding to the 3D mesh mounted on the node in the 3D scene to be rendered, includes at least one of the following steps b1 to b4.

[0329] In step b1, the name of the 3D mesh is obtained based on the mesh name syntax element (name) in the mesh description module corresponding to the 3D mesh.

[0330] In step b2, the data types included in the 3D mesh are obtained based on the data type syntax element in the mesh description module corresponding to the 3D mesh.

[0331] In some embodiments, step b2 (obtaining data types included in the 3D mesh based on the data type syntax element in the mesh description module corresponding to the 3D mesh) includes obtaining data types included in the 3D mesh based on the data type syntax element in the target extension array of the primitives extensions in the mesh description module corresponding to the 3D mesh.

[0332] In some embodiments, the target extension sequence may be MPEG_primitve_GPCC.

[0333] For example, the extensions list of primitives in a mesh description module corresponding to a certain 3D mesh is as follows: JPEG0007872437000044.jpg44164

[0334] In this case, it is possible to determine that the 3D mesh includes position coordinates based on the position coordinate syntax element (position) in the target extension array (MPEG_primitives_GPCC) of the primitives extension list (primitives) of the mesh description module corresponding to the 3D mesh; to determine that the 3D mesh includes color values ​​based on the color value syntax element (color_0) in the target extension array (MPEG_primitives_GPCC) of the primitives extension list (primitives) of the mesh description module corresponding to the 3D mesh; and to determine that the 3D mesh includes normal vectors based on the normal vector syntax element (normal) in the target extension array (MPEG_primitives_GPCC) of the primitives extension list (primitives) of the mesh description module corresponding to the 3D mesh.

[0335] Furthermore, for example, the extensions list of primitives for a mesh description module corresponding to a certain 3D mesh is shown as follows: JPEG0007872437000045.jpg44164

[0336] In this case, it is possible to determine that the 3D mesh includes position coordinates based on the position coordinate syntax element (G-PCC_position) in the target extension array (MPEG_primitives_GPCC) of the primitives extension list (primitives) of the mesh description module corresponding to the 3D mesh; to determine that the 3D mesh includes color values ​​based on the color value syntax element (G-PCC_color_0) in the target extension array (MPEG_primitives_GPCC) of the primitives extension list (primitives) of the mesh description module corresponding to the 3D mesh; and to determine that the 3D mesh includes normal vectors based on the normal vector syntax element (G-PCC_normal) in the target extension array (MPEG_primitives_GPCC) of the primitives extension list (primitives) of the mesh description module corresponding to the 3D mesh.

[0337] In some embodiments, step b2 (obtaining the data types included in the 3D mesh based on the data type syntax element in the mesh description module corresponding to the 3D mesh) includes obtaining the data types included in the 3D mesh based on the data type syntax element in the attributes of the primitives in the mesh description module corresponding to the 3D mesh.

[0338] For example, the attributes of the primitives in a mesh description module corresponding to a certain 3D mesh are as follows: JPEG0007872437000046.jpg39165

[0339] In this case, it is determined that the 3D mesh contains position coordinates based on the position coordinate syntax element in the attributes of the primitives in the mesh description module corresponding to the 3D mesh; it is determined that the 3D mesh contains color values ​​based on the color value syntax element in the attributes of the primitives in the mesh description module corresponding to the 3D mesh; and it is determined that the 3D mesh contains normal vectors based on the normal vector syntax element in the attributes of the primitives in the mesh description module corresponding to the 3D mesh.

[0340] Furthermore, for example, the attributes of the primitives in a mesh description module corresponding to a certain 3D mesh are as follows: JPEG0007872437000047.jpg40164

[0341] In this case, it can be determined that the 3D mesh includes position coordinates based on the position coordinate syntax element (G-PCC_position) in the attributes of the primitives in the mesh description module corresponding to the 3D mesh; that the 3D mesh includes color values ​​based on the color value syntax element (G-PCC_color_0) in the attributes of the primitives in the mesh description module corresponding to the 3D mesh; and that the 3D mesh includes normal vectors based on the normal vector syntax element (G-PCC_normal) in the attributes of the primitives in the mesh description module corresponding to the 3D mesh.

[0342] In step b3, based on the value of the data type syntax element, the index value of the accessor description module corresponding to the accessor for accessing the data of the 3D mesh type is obtained.

[0343] As explained in the example above, the value of the position coordinate syntax element (G-PCC_position) is 0, so the index value of the accessor description module corresponding to the accessor for accessing the position coordinates of the 3D mesh is determined to be 0 (the first accessor in the Accessist); the value of the color value syntax element (G-PCC_color_0) is 1, so the index value of the accessor description module corresponding to the accessor for accessing the color value of the 3D mesh is determined to be 1 (the second accessor in the Accessist); and the value of the normal vector syntax element (G-PCC_normal) is 2, so the index value of the accessor description module corresponding to the accessor for accessing the normal vector of the 3D mesh is determined to be 2 (the third accessor in the Accessist).

[0344] In step b4, the type of topology structure of the 3D mesh is obtained based on the value of the mode syntax element (mode) in the mesh description module corresponding to the 3D mesh.

[0345] For example, if the value of the mode syntax element is 0, the type of topological structure of the 3D mesh can be determined to be scattered; if the value of the mode syntax element is 1, the type of topological structure of the 3D mesh can be determined to be linear; and if the value of the mode syntax element is 4, the type of topological structure of the 3D mesh can be determined to be triangular.

[0346] As an example, the mesh description module corresponding to a certain 3D mesh is as follows: JPEG0007872437000048.jpg78164

[0347] In this case, the description information of the 3D mesh obtained based on the mesh description module corresponding to the 3D mesh is that the name of the 3D mesh is G-PCC example_mesh, the topology type of the 3D mesh is scattered, the 3D mesh contains three types of data, each being position coordinates, color values, and normal vectors, the index value of the accessor description module corresponding to the accessor for accessing the position coordinates of the 3D mesh is 0, the index value of the accessor description module corresponding to the accessor for accessing the color values ​​of the 3D mesh is 1, and the index value of the accessor description module corresponding to the accessor for accessing the normal vectors of the 3D mesh is 2.

[0348] In some embodiments, after obtaining the index value of the accessor description module corresponding to the accessor for accessing the data of the 3D mesh type based on the value of the data type syntax element, the method further includes the following:

[0349] Based on the index value of the accessor description module corresponding to the accessor for accessing data of each type of 3D mesh, the accessor description module corresponding to the accessor for accessing each type of data of the 3D mesh is obtained from the accessory list of the scene description file, and the description information of the accessor for accessing data of each type of 3D mesh is obtained based on the accessor description module corresponding to the accessor for accessing data of each type of 3D mesh.

[0350] For example, if the index value of the accessor description module corresponding to the accessor for accessing the color values ​​of the 3D mesh is 1, the second accessor description module is obtained from the accessory list of the scene description file as the accessor description module corresponding to the accessor for accessing the color values ​​of the 3D mesh.

[0351] In some embodiments, obtaining descriptive information for accessors to access each type of data in a 3D mesh, based on accessor description modules corresponding to accessors for accessing each type of data in a 3D mesh, includes at least one of the following steps c1-c6.

[0352] In step c1, the type of data accessed by the accessor is determined based on the value of the data type syntax element (component Type) in the accessor description module.

[0353] For example, if the data type syntax element in an accessor description module corresponding to an accessor for accessing the normal vector of a certain 3D mesh is "componentType": 5126, then it can be determined that the type of data accessed by the accessor corresponding to that accessor description module (the normal vector of the 3D mesh) is a 32-bit floating-point number (float).

[0354] In step c2, the accessor type is determined based on the value of the accessor type syntax element (type) in the accessor description module.

[0355] For example, if the accessor type syntax element in an accessor description module corresponding to an accessor for accessing the position coordinates of a certain 3D mesh is "type":VEC3, then it can be determined that the type of the accessor corresponding to that accessor description module is a 3D vector.

[0356] In step c3, the quantity of data accessed by the accessor is determined based on the value of the data quantity syntax element (count) in the accessor description module.

[0357] For example, if the data count syntax element in the accessor description module corresponding to an accessor for accessing the color values ​​of a certain 3D mesh is "count":1000, then it can be determined that the number of data (color values ​​of the 3D mesh) accessed by the accessor corresponding to that accessor description module is 1000.

[0358] In step c4, it is determined whether the accessor is a time-varying accessor based on MPEG extension modifications, based on whether or not the accessor description module includes an MPEG time-varying accessor (MPEG_accessor_timed).

[0359] In some embodiments, determining whether an accessor is a time-varying accessor based on MPEG extension modifications, based on whether or not the accessor description module includes an MPEG time-varying accessor, includes determining that the accessor is a time-varying accessor based on MPEG extension modifications if the accessor description module includes an MPEG time-varying accessor, and determining that the accessor is not a time-varying accessor based on MPEG extension modifications if the accessor description module does not include an MPEG time-varying accessor.

[0360] In step c5, the index value of the buffer slice description module corresponding to the buffer slice that stores the data accessed by the accessor is determined based on the value of the buffer slice index syntax element (buffer View) in the MPEG time-varying accessor (MPEG_accessor_timed) of the accessor description module.

[0361] For example, if the buffer slice index syntax element in the MPEG time-varying accessor of an accessor description module corresponding to an accessor for accessing the normal vector of a certain 3D mesh is "bufferView":0, then it can be determined that the data accessed by the accessor corresponding to the accessor description module (the normal vector of the 3D mesh) is stored in the buffer slice corresponding to the first buffer slice description module in the buffer slice list.

[0362] In step c6, based on the value of the time-varying syntax element (immutable) in the MPEG time-varying accessor of the accessor description module, it is determined whether or not the value of the syntax element within the accessor changes over time.

[0363] In some embodiments, determining whether the value of a syntax element within an accessor changes over time based on the value of a time-varying syntax element (immutable) in the MPEG time-varying accessor of the accessor description module includes determining that the value of the syntax element within the accessor does not change over time if the time-varying syntax element and its value in the MPEG time-varying accessor of the accessor description module are "immutable":true, and determining that the value of the syntax element within the accessor changes over time if the time-varying syntax element and its value in the MPEG time-varying accessor of the accessor description module are "immutable":false.

[0364] For example, the accessor description module corresponding to a certain accessor is as follows: JPEG0007872437000049.jpg53164

[0365] In this case, the description information of the accessor obtained based on the accessor description module corresponding to the accessor includes that the type of data accessed by the accessor is 5123, the type of the accessor is SCALAR, the number of data accessed by the accessor is 1000, the accessor is a time-varying accessor based on MPEG extension modifications, the data accessed by the accessor is buffered in a buffer slice corresponding to the second buffer slice description module in the buffer slice list, and the values ​​of syntax elements within the accessor do not change over time.

[0366] In some embodiments, the scene description file analysis method according to the above embodiment further includes the following steps d-g.

[0367] In step d, the buffer description module in the buffer list (buffers) of the scene description file is obtained.

[0368] In step e, the value of the media index syntax element (media) in the buffer description module is obtained.

[0369] In step f, a buffer description module whose media index syntax element value is the same as the index value of the target media description module is determined to be the target buffer description module corresponding to the target buffer for buffering the decoded data of the target media file.

[0370] For example, if the index value of the target media description module is 0, the buffer description module whose media index syntax element value is 0 is determined to be the target buffer description module corresponding to the target buffer for buffering the decoded data of the target media file.

[0371] The number of target buffers for buffering the decoded data of the target media file may be one or more, and the embodiments of this application are not limited to this.

[0372] In Step g, the description information of the target buffer is obtained based on the target buffer description module.

[0373] In some embodiments, obtaining the description information of the target buffer based on the target buffer description module includes at least one of the following steps g1 to g4.

[0374] In step g1, the capacity of the target buffer is obtained based on the value of the first byte length syntax element in the target buffer description module.

[0375] For example, if the first byte length syntax element and its value in the target buffer description module are "byteLength":15000, then the capacity of the target buffer can be determined to be 15000 bytes.

[0376] In step g2, it is determined whether the target buffer is a ring buffer based on MPEG extension modifications, based on whether the target buffer description module includes an MPEG ring buffer (MPEG_buffer_circular).

[0377] In some embodiments, determining whether a target buffer is a ring buffer based on MPEG extension modifications based on whether the target buffer description module includes an MPEG ring buffer includes determining that the target buffer is a ring buffer based on MPEG extension modifications if the target buffer description module includes an MPEG ring buffer, and determining that the target buffer is not a ring buffer based on MPEG extension modifications if the target buffer description module does not include an MPEG ring buffer.

[0378] In step g3, the number of stored segments in the MPEG ring buffer is obtained based on the value of the segment count syntax element (count) in the MPEG ring buffer of the target buffer description module.

[0379] For example, if the segment count syntax element and its value in the target buffer description module for the MPEG ring buffer are "count":8, it can be determined that the MPEG ring buffer contains 5 storage segments.

[0380] In step g4, the track index value of the source data of the data buffered in the MPEG ring buffer is obtained based on the value of the second track index syntax element (tracks) in the MPEG ring buffer of the target buffer description module.

[0381] As an example, the buffer description module corresponding to a certain buffer is as follows: JPEG0007872437000050.jpg53164

[0382] In this case, obtaining the description information of the buffer based on the buffer description module corresponding to the buffer includes: the buffer has a capacity of 8000 bytes; the buffer is a ring buffer based on the MPEG extension modification; the number of storage segments of the ring buffer is 5; the media file stored in the ring buffer is the second media file declared on the MPEG media; and the track index value of the source data of the data buffered in the ring buffer is 1.

[0383] In some embodiments, the scene description file analysis method according to the above embodiment further includes the following steps h-k.

[0384] In step h, the buffer slice description module in the buffer slice list (buffer Views) of the scene description file is obtained.

[0385] In step i, the value of the buffer index syntax element (buffer) in the buffer slice description module is obtained.

[0386] In step j, a buffer slice description module whose buffer index syntax element value is the same as the index value of the target buffer description module is determined to be the buffer slice description module corresponding to the buffer slice of the target buffer.

[0387] For example, when the index value of the target media description module is 1, the buffer slice description module whose buffer index syntax element value is 1 is determined to be the buffer slice description module corresponding to the buffer slice of the target buffer.

[0388] The number of buffer slices in the target buffer may be one or more, and the embodiments of this application are not limited thereto.

[0389] In Step k, the description information of the buffer slice of the target buffer is obtained based on the buffer slice description module corresponding to the buffer slice of the target buffer.

[0390] In some embodiments, obtaining description information of the buffer slices of the target buffer based on a buffer slice description module corresponding to the buffer slices of the target buffer includes at least one of the following steps k1 and k2.

[0391] In step k1, the capacity of the buffer slice of the target buffer is obtained based on the value of the second byte length syntax element (byte Length) in the buffer slice description module corresponding to the buffer slice of the target buffer.

[0392] For example, if the second byte length syntax element and its value in the buffer slice description module corresponding to a buffer slice of the target buffer are "byteLength":12000, then the capacity of the buffer slice of the target buffer can be determined to be 12000 bytes.

[0393] In step k2, the offset of the buffer slice of the target buffer is obtained based on the value of the offset amount syntax element (byte Offset) in the buffer slice description module corresponding to the buffer slice of the target buffer.

[0394] For example, if the offset amount syntax element and its value in the buffer slice description module corresponding to a buffer slice of the target buffer are "byteOffset":0, then the offset amount of the buffer slice of the target buffer can be determined to be 0 bytes.

[0395] As an example, a buffer slice description module corresponding to a certain buffer slice is as follows: JPEG0007872437000051.jpg39165

[0396] In this case, obtaining the description information of the buffer slice based on the buffer slice description module corresponding to the buffer slice includes determining that the buffer slice is a buffer slice of the buffer corresponding to the second buffer description module in the buffer list, that the capacity of the buffer slice is 8000 bytes, and that the offset amount of the buffer slice is 0, i.e., the data range buffered in the buffer slice is the previous 8000 bytes.

[0397] In some embodiments, the scene description file analysis method according to the above embodiment further includes the following steps l-o.

[0398] In step l, the accessor description module in the accessor of the scene description file is obtained.

[0399] In step m, the value of the buffer slice index syntax element (buffer View) in the accessor description module is obtained.

[0400] In step n, an accessor description module whose buffer slice index syntax element value is the same as the index value of the buffer slice description module corresponding to the buffer slice of the target buffer is determined to be the accessor description module corresponding to the accessor for accessing data in the buffer slice of the target buffer.

[0401] For example, if the index value of the buffer slice description module corresponding to a buffer slice of the target buffer is 2, then the accessor description module whose buffer slice index syntax element value is 2 is determined to be the accessor description module corresponding to the accessor for accessing the data in that buffer slice of the target buffer.

[0402] In step o, description information for an accessor to access data in the buffer slice of the target buffer is obtained based on the accessor description module corresponding to the accessor for accessing data in the buffer slice of the target buffer.

[0403] In some embodiments, obtaining description information for an accessor to access data in a buffer slice of the target buffer, based on an accessor description module corresponding to an accessor to access data in a buffer slice of the target buffer, includes at least one of the following steps o1 to o6.

[0404] In step o1, the type of data accessed by the accessor is determined based on the value of the data type syntax element (component Type) in the accessor description module.

[0405] In step o2, the accessor type is determined based on the value of the accessor type syntax element (type) in the accessor description module.

[0406] In step o3, the quantity of data accessed by the accessor is determined based on the value of the data quantity syntax element (count) in the accessor description module.

[0407] In step o4, it is determined whether the accessor is a time-varying accessor based on MPEG extension modifications, based on whether or not the accessor description module includes an MPEG time-varying accessor (MPEG_accessor_timed).

[0408] In step o5, the index value of the buffer slice description module corresponding to the buffer slice that stores the data accessed by the accessor is determined based on the value of the buffer slice index syntax element (buffer View) in the MPEG time-varying accessor of the accessor description module.

[0409] In step o6, it is determined whether the value of the syntax element within the accessor changes over time, based on the value of the time-varying syntax element (immutable) in the MPEG time-varying accessor of the accessor description module.

[0410] The implementations of steps o1-o6 described above can be found by referring to the implementations of steps c1-c6 described above. To avoid redundant explanations, a detailed explanation is omitted here.

[0411] Some embodiments of the present invention further provide a method for rendering a three-dimensional scene, wherein the entity executing the method is a display engine in an immersive media description framework, and the method for rendering the three-dimensional scene includes the following steps, as shown in Figure 14.

[0412] In S141, the scene description file for the 3D scene to be rendered is obtained.

[0413] Here, the 3D scene to be rendered includes a target media file whose type is a G-PCC encoded point cloud.

[0414] In some embodiments, the implementation method for obtaining a scene description file of a 3D scene to be rendered includes sending request information to a media resource server to request the scene description file of the 3D scene to be rendered, and receiving a request response from the media resource server that includes the scene description file of the 3D scene to be rendered.

[0415] In S142, the description information of the target media file is obtained based on the media description module corresponding to the target media file in the media list (media) of the MPEG media (MPEG_media) of the scene description file.

[0416] In some embodiments, the description information for the target media file includes one or more of the following: the name of the target media file, whether or not the target media file needs to be played automatically, whether or not the target media file needs to be played in a cycle, the capsule format of the target media file, the type of code stream of the target media file, and the encoding parameters of the target media file.

[0417] An implementation method for obtaining description information of the target media file based on the media description module corresponding to the target media file can refer to the implementation method of the media description module that analyzes the target media file in the scene description file analysis method described above. To avoid redundant explanation, a detailed explanation is omitted here.

[0418] In S143, the description information of the target media file is sent to the media access function.

[0419] After the display engine sends description information of the target media file to the media access function, the media access function can retrieve the target media file based on the description information of the target media file, process the target media file to obtain decoded data of the target media file, and write the decoded data of the target media file to the target buffer.

[0420] In some embodiments, the display engine transmitting the description information of the target media file to the media access function includes the display engine transmitting the description information of the target media file to the media access function via the media access function API.

[0421] In some embodiments, the display engine transmitting the description information of the target media file to the media access function includes the display engine transmitting a media file processing instruction containing the description information of the target media file to the media access function.

[0422] In S144, the decoded data of the target media file is read from the target buffer.

[0423] In other words, data that has been fully processed by the media access function is read from the target buffer and can be used directly for rendering the 3D scene to be rendered.

[0424] In S145, the 3D scene to be rendered is rendered based on the decoded data of the target media file.

[0425] The method for rendering a 3D scene according to an embodiment of the present invention involves first obtaining a scene description file of a 3D scene to be rendered, which includes a target media file of type G-PCC encoded point cloud; first obtaining description information of the target media file based on the media description module corresponding to the target media file in the media list of the MPEG media in the scene description file; and sending the description information of the target media file to a media access function, which then obtains the target media file based on the description information of the target media file, processes the target media file to obtain decoded data of the target media file, writes the decoded data of the target media file to a target buffer; further reading the decoded data of the target media file from the target buffer; and rendering the 3D scene to be rendered based on the decoded data of the target media file. In the 3D scene rendering method according to the embodiment of the present application, the display engine obtains description information of the target media file based on the target media description module, transmits the description information of the target media file to the media access function, reads the decoded data of the target media file whose type is a G-PCC encoded point cloud, and renders the rendering target 3D scene based on the decoded data of the target media file. Thus, the embodiment of the present application provides a rendering method for a rendering target 3D scene including a media file whose type is a G-PCC encoded point cloud, and realizes rendering a media file whose type is a G-PCC encoded point cloud based on a scene description file.

[0426] Some embodiments of the present invention further provide a method for processing media files, the entity executing the media file processing method being a media access function in an immersive media description framework, and referring to Figure 15, the media file processing method includes the following steps:

[0427] In S151, the system receives the description information of the target media file, the description information of the target buffer, and the description information of the buffer slice of the target buffer, which have been transmitted by the display engine.

[0428] Here, the target media file is a media file of type G-PCC encoded point cloud, and the target buffer is a buffer for buffering the decoded data of the target media file.

[0429] In some embodiments, the description information for the target media file may include at least one of the following: the name of the target media file, whether or not the target media file needs to be played automatically, whether or not the target media file needs to be played in a cycle, the capsule format of the target media file, the type of code stream of the target media file, and the encoding parameters of the target media file.

[0430] In some embodiments, the description information of the target buffer may include at least one of the following: the buffer capacity, whether or not it is an MPEG ring buffer, the number of storage segments of the ring buffer, the index value of the media description module corresponding to the target media file, and the track index value of the source data of the data buffered in the ring buffer.

[0431] In some embodiments, the descriptive information of the buffer slice of the target buffer may include at least one of the buffer to which the buffer slice belongs, the capacity of the buffer slice, and the offset amount of the buffer slice.

[0432] In some embodiments, receiving description information of a target media file, description information of a target buffer, and description information of a buffer slice of the target buffer transmitted from a display engine includes receiving the description information of the target media file, description information of the target buffer, and description information of a buffer slice of the target buffer transmitted by the display engine via a media access function API.

[0433] In S152, decryption data for the target media file is obtained based on the description information of the target media file.

[0434] In some embodiments, the media access function obtaining decoded data of the target media file based on the description information of the target media file includes creating a target pipeline for processing the target media file based on the description information of the target media file, obtaining the target media file by the target pipeline, decapsulating and decrypting the target media file to obtain decoded data of the target media file.

[0435] In some embodiments, obtaining the target media file via the target pipeline, decapsulating and decoding the target media file to obtain the decoded data of the target media file includes obtaining the target media file via the input module of the target pipeline, inputting the target media file into the decapsulation module of the target pipeline, decoding the target media file via the decapsulation module to obtain the geometric code stream and attribute code stream of the target media file, decoding the geometric code stream via the geometric decoder of the target pipeline to obtain the geometric decoded data of the target media file, and decoding the attribute code stream via the attribute decoder of the target pipeline to obtain the attribute decoded data of the target media file.

[0436] In some embodiments, obtaining the target media file via the target pipeline, decapsulating and decrypting the target media file to obtain decrypted data of the target media file further includes obtaining geometric decrypted data of the target media file, processing the geometric decrypted data by a first post-processing module of the target pipeline, and obtaining attribute decrypted data of the target media file, processing the attribute decrypted data by a second post-processing module of the target pipeline.

[0437] For example, processing the geometrically decoded data by a first post-processing module of the target pipeline may include performing a format conversion on the geometrically decoded data by the first post-processing module of the target pipeline, and processing the attribute-decoded data by a second post-processing module of the target pipeline may include performing a format conversion on the attribute-decoded data by the second post-processing module of the target pipeline.

[0438] In S153, the decoded data of the target media file is written to the target buffer based on the description information of the target buffer and the description information of the buffer slice of the target buffer.

[0439] After writing the decoded data of the target media file to the target buffer, the display engine reads the decoded data of the target media file from the target buffer based on the description information of the target buffer and the description information of the buffer slice of the target buffer, and can render the 3D scene to be rendered, which includes the target media file, based on the decoded data of the target media file.

[0440] The media file processing method according to the embodiment of the present application receives description information of a target media file whose type is a G-PCC encoded point cloud, description information of a loose target buffer for buffering the decoded data of the target media file, and description information of a buffer slice of the target buffer from the display engine. Based on the description information of the target media file, the decoded data corresponding to the target media file is obtained, and based on the description information of the target buffer and the description information of the buffer slice of the target buffer, the decoded data of the target media file is written to the target buffer. Therefore, the display engine can read the decoded data of the target media file from the target buffer based on the description information of the target buffer and the description information of the buffer slice of the target buffer, and render a rendering target 3D scene including the target media file based on the decoded data of the target media file. Thus, the embodiment of the present application can support rendering a media file whose type is a G-PCC encoded point cloud in a scene description framework.

[0441] Some embodiments of the present invention further provide a buffer management method in which the entity executing the buffer management method is a buffer management module in an immersive media description framework, and referring to Figure 16, the buffer management method includes the following steps.

[0442] In S161, the description information of the target buffer and the description information of the buffer slice of the target buffer are received.

[0443] Here, the target buffer is a buffer for buffering the target media file, and the target media file is a media file of type G-PCC encoded point cloud.

[0444] In some embodiments, the description information of the target buffer may include at least one of the following: the buffer capacity, whether or not it is an MPEG ring buffer, the number of storage segments of the ring buffer, the index value of the media description module corresponding to the media file (the target media file) buffered in the ring buffer, and the track index value of the source data of the data buffered in the ring buffer.

[0445] In some embodiments, the description information of the buffer slice of the target buffer may include at least one of the buffer to which the buffer slice belongs, the capacity of the buffer slice, and the offset amount of the buffer slice.

[0446] In S162, the target buffer is created based on the description information of the target buffer.

[0447] For example, the description information for the target buffer includes that the target buffer has a capacity of 8000 bytes, that the target buffer is a ring buffer based on the MPEG extension modification, that the number of storage segments in the ring buffer is 3, that the media file stored in the ring buffer is the first media file declared on the MPEG media, and that the track index value of the source data of the data buffered in the ring buffer is 1, and the buffer management module creates a ring buffer with a capacity of 8000 bytes and containing 3 storage segments as the target buffer.

[0448] In S163, the buffer slice is divided into parts based on the description information of the buffer slice of the target buffer.

[0449] As described in the above embodiment, if the ring buffer includes two buffer slices, the description information of the first buffer slice includes a capacity of 6000 bytes and an offset of 0, and the description information of the second buffer slice includes a capacity of 2000 bytes and an offset of 6001, then the target buffer is divided into two buffer slices, the first buffer slice having a capacity of 6000 bytes and buffering the first 6000 bytes of data before the decoded data of the target media file, and the second buffer slice having a capacity of 2000 bytes and buffering the 6001-8000 bytes of data before the decoded data of the target media file.

[0450] The buffer management module divides the target buffer into buffer slices based on the buffer slice description information of the target buffer. Then, the media access function writes the decoded data of the target media file to the target buffer. The display engine reads the decoded data of the target media file from the target buffer and, based on the decoded data of the target media file, can render the 3D scene to be rendered, which includes the target media file.

[0451] The buffer management method according to the embodiment of the present application can, after receiving description information for a target buffer and description information for a buffer slice of the target buffer, create the target buffer based on the description information for the target buffer and divide the buffer slice of the target buffer based on the description information for the buffer slice of the target buffer. Therefore, the media access function can write decoded data of the target media file, which is decoded data of a media file of type G-PCC encoded point cloud, to the target buffer. The display engine can read the decoded data of the target media file from the target buffer and render a rendering target 3D scene including the target media file based on the decoded data of the target media file. Therefore, the embodiment of the present application can support rendering of a media file of type G-PCC encoded point cloud in a scene description framework.

[0452] Some embodiments of the present invention further provide a method for rendering a 3D scene, which includes a method for parsing a scene description file and rendering a 3D scene performed by a display engine, a method for processing a media file performed by a media access function, and a buffer management method performed by a buffer management module. As shown in Figure 17, the method includes the following steps.

[0453] In S1701, the display engine obtains the scene description file for the 3D scene to be rendered.

[0454] Here, the 3D scene to be rendered includes a target media file whose type is a G-PCC encoded point cloud.

[0455] In some embodiments, the display engine obtaining a scene description file for a scene to be rendered includes the display engine downloading the scene description file from a server using a network transmission service.

[0456] In some embodiments, the display engine obtaining a scene description file for a scene to be rendered includes reading the scene description file from local storage.

[0457] In S1702, the display engine obtains a media description module corresponding to each media file from the media list (media) of the MPEG media (MPEG_ media) in the scene description file (including obtaining a media description module corresponding to the target media file from the media list of the MPEG media in the scene description file).

[0458] In S1703, the display engine obtains description information for each media file based on the media description module corresponding to each media file (including obtaining description information for the target media file based on the media description module corresponding to the target media file).

[0459] In some embodiments, the media file description information includes at least one of the following: the name of the media file, whether the media file is autoplayable, whether the media file is cyclically played, the capsule format of the media file, the access address of the media file, the track information of the capsule file of the media file, and the codec parameters of the media file.

[0460] The implementation method by which the display engine obtains description information of the target media file based on the media description module corresponding to the target media file can refer to the implementation method of the media description module that analyzes the target media file in the scene description analysis method described above, and a detailed explanation is omitted here to avoid redundant explanation.

[0461] In S1704, the display engine transmits the description information of each media file to the media access function (including transmitting the description information of the target media file to the media access function).

[0462] Accordingly, the media access function receives description information for each media file transmitted from the display engine (including receiving description information for the target media file transmitted from the display engine).

[0463] In some embodiments, the display engine sending description information for each media file to a media access function includes the display engine sending description information for each media file to a media access function via a media access function API.

[0464] In some embodiments, the media access function receiving description information for each media file transmitted from the display engine includes the media access function receiving description information for each media file transmitted from the display engine via the media access function API.

[0465] In S1705, the media access function creates a corresponding pipeline for processing each media file based on the description information of each media file (including creating a target pipeline for processing the target media file based on the description information of the target media file).

[0466] In some embodiments, the target pipeline includes an input module, a decapsulation module, and a decoding module, wherein the input module is configured to acquire the target media file (capsule file), the decapsulation module is configured to decapsulate the target media file to acquire the code stream of the target media file (which may be a single-track encapsulated G-PCC code stream, or a multi-track encapsulated G-PCC geometric code stream and a G-PCC attribute code stream), and the decoding module includes a decoder, a geometric decoder, and an attribute decoder, wherein if the code stream of the target media file is a single-track encapsulated G-PCC code stream, the decoding module decodes the G-PCC code stream via the decoder to acquire the decoded data of the target media file, and if the code stream of the target media file is a multi-track encapsulated G-PCC geometric code stream and a G-PCC attribute code stream, the decoding module decodes the G-PCC geometric code stream and the G-PCC attribute code stream via the geometric decoder and the attribute decoder, respectively, to acquire the geometric data and attribute data of the target media file and acquire the decoded data of the target media file.

[0467] In some embodiments, the target pipeline further includes a first post-processing module and a second post-processing module, wherein the first post-processing module performs post-processing such as format conversion on geometric data obtained by decoding a G-PCC geometric code stream, and the second post-processing module performs post-processing such as format conversion on attribute data obtained by decoding a G-PCC attribute code stream.

[0468] In S1706, the media access function acquires each media file through pipeline processing corresponding to each media file, decapsulates and decrypts each media file, and obtains decrypted data corresponding to each media file. (This includes acquiring the target media file through the target pipeline, decapsulating and decrypting the target media file, and obtaining decrypted data corresponding to the target media file.)

[0469] In some embodiments, the description information of the target media file includes the access address of the target media file, and the media access function obtaining decoded data of the target media file based on the description information of the target media file includes the media access function obtaining the target media file based on the access address of the target media file.

[0470] In some embodiments, the media access function obtaining the target media file based on the access address of the target media file includes the media access function sending a media resource request to a media resource server based on the access address of the target media file and receiving a media resource response from the media server that includes the target media file.

[0471] In some embodiments, the media access function obtaining the target media file based on the access address of the target media file includes the media access function reading the target media file from a pre-configured storage space based on the access address of the target media file.

[0472] In some embodiments, the description information of the target media file further includes index values ​​for each code stream track of the target media file, and the media access function obtaining decoded data of the target media file based on the description information of the target media file includes the media access function decapsulating the target media file based on the encapsulation format of the target media file and obtaining the code streams for each code stream track of the target media file.

[0473] In some embodiments, the description information of the target media file further includes the type of code stream and codec parameters of the target media file, and the media access function obtaining decoded data of the target media file based on the description information of the target media file includes the media access function decodes the code stream of each code stream track of the target media file based on the type of code stream and codec parameters of the target media file, and obtaining decoded data of the target media file.

[0474] In S1707, the display engine obtains each buffer description module in the buffer list (buffers) of the scene description file (including obtaining the buffer description module from the buffer list of the scene description file that corresponds to the target buffer used to buffer the decoded data of the target media file).

[0475] In S1708, the display engine obtains description information for each buffer based on the buffer description module corresponding to each buffer (including obtaining description information for the target buffer based on the buffer description module corresponding to the target buffer).

[0476] In some embodiments, the buffer description information may include at least one of the following: the buffer capacity (byte length), the access address of the data buffered in the buffer, whether or not it is an MPEG ring buffer, the number of storage segments of the ring buffer, the index value of the media description module corresponding to the media file buffered in the ring buffer, and the track index value of the source data of the data buffered in the ring buffer.

[0477] In S1709, the display engine obtains each buffer slice description module in the buffer slice list (buffer Views) of the scene description file (including obtaining the buffer slice description module corresponding to the buffer slice description of the target buffer from the buffer slice list of the scene description file).

[0478] In S1710, the display engine obtains description information for each buffer's buffer slice based on the buffer slice description module corresponding to the buffer slice of each buffer (including obtaining description information for the target buffer's buffer slice based on the buffer slice description module corresponding to the buffer slice of the target buffer).

[0479] In some embodiments, the buffer description information may include at least one of the buffer to which the buffer slice belongs, the capacity of the buffer slice, and the offset amount of the buffer slice.

[0480] In S1711, the display engine obtains each accessor description module in the accessors of the scene description file (including obtaining the accessor description module corresponding to the target accessor for accessing the decoded data of the target media file from the accessors of the scene description file).

[0481] In S1712, the display engine obtains description information for each accessor based on the accessor description module corresponding to each accessor (including obtaining description information for the target accessor for accessing the decoded data of the target media file based on the accessor description module corresponding to the target accessor).

[0482] In some embodiments, the accessor description information may include at least one of the following: the buffer slice accessed by the accessor, the data type of the data accessed by the accessor, the type of accessor, the number of data accessed by the accessor, whether or not it is an MPEG time-varying accessor, the buffer slice accessed by the time-varying accessor, and whether or not the accessor parameters change over time.

[0483] In some embodiments, after steps S1707-S1712 described above, the embodiments of the present invention may transmit the description information of each buffer, the description information of each buffer's buffer slice, and the description information of each access unit to the media access function and the buffer management module according to the following solution 1.

[0484] In some embodiments, the implementation of Solution 1 (sending description information for each buffer, description information for each buffer's buffer slice, and description information for each accessor to the media access function and the buffer management module) includes the following steps a and b.

[0485] In step a, the display engine sends the description information for each buffer, the description information for each buffer's buffer slice, and the description information for each accessor to the media access function (including the display engine sending the description information for the target buffer, the description information for the target buffer's buffer slice, and the description information for the target accessor to the media access function).

[0486] In response, the media access function receives description information for each buffer and description information for each buffer's buffer slice, transmitted by the display engine (the media access function includes receiving description information for the target buffer, description information for the target buffer's buffer slice, and description information for the accessor, transmitted by the display engine).

[0487] In some embodiments, the implementation of step a (the display engine sending description information for each buffer, description information for each buffer's buffer slice, and description information for each accessor to the media access function) may be such that the display engine sends description information for each buffer, description information for each buffer's buffer slice, and description information for each accessor to the media access function via the media access function API.

[0488] In response to this, the implementation method for a media access function to receive description information for each buffer sent from the display engine may involve the media access function receiving description information for each buffer, description information for each buffer's buffer slice, and description information for each accessor, all sent from the display engine via the media access function API.

[0489] In step b, the media access function transmits description information for each buffer, description information for each buffer's buffer slice, and description information for each accessor to the buffer management module (the media access function also transmits description information for the target buffer, description information for the target buffer's buffer slice, and description information for the target accessor to the buffer management module).

[0490] In response, the buffer management module receives description information for each buffer, description information for each buffer's buffer slice, and description information for each access, transmitted by the media access function (including the media access function receiving description information for the target buffer, description information for the target buffer's buffer slice, and description information for the target accessor, transmitted by the display engine).

[0491] In some embodiments, the implementation of step b (the media access function sending description information for each buffer, description information for each buffer's buffer slice, and description information for each access to the buffer management module) may include the media access function sending description information for each buffer, description information for each buffer's buffer slice, and description information for each access to the buffer management module via the buffer API. Correspondingly, the implementation of the buffer management module receiving the description information for each buffer, description information for each buffer's buffer slice, and description information for each access sent by the media access function may include the buffer management module receiving the description information for each buffer, description information for each buffer's buffer slice, and description information for each access sent by the media access function via the buffer API.

[0492] In some embodiments, the implementation of Solution 1 (sending description information for each buffer, description information for each buffer's buffer slice, and description information for each accessor to the media access function and the buffer management module) includes the following steps c and d.

[0493] In step c, the display engine sends the description information for each buffer, the description information for each buffer's buffer slice, and the description information for each accessor to the media access function (including sending the description information for the target buffer, the description information for the target buffer's buffer slice, and the description information for the target accessor to the media access function).

[0494] In response, the media access function receives description information for each buffer, description information for each buffer's buffer slice, and description information for each accessor, transmitted by the display engine (the media access function includes receiving description information for the target buffer, description information for the target buffer's buffer slice, and description information for the accessor, transmitted by the display engine).

[0495] In step d, the display engine transmits the description information for each buffer, the description information for each buffer's buffer slice, and the description information for each accessor to the buffer management module (including the display engine transmitting the description information for the target buffer, the description information for the target buffer's buffer slice, and the description information for the target accessor to the buffer management module).

[0496] In response, the buffer management module receives description information for each buffer, description information for each buffer's buffer slice, and description information for each access, all transmitted by the display engine.

[0497] In some embodiments, the implementation of step d (the display engine sending description information for each buffer, description information for each buffer's buffer slice, and description information for each access to the buffer management module) may include the display engine sending description information for each buffer, description information for each buffer's buffer slice, and description information for each access to the buffer management module via a buffer API.

[0498] Accordingly, the implementation method for a buffer management module to receive description information for each buffer, description information for each buffer's buffer slice, and description information for each access transmitted by the display engine may include the buffer management module receiving description information for each buffer, description information for each buffer's buffer slice, and description information for each access transmitted from the display engine via the buffer API.

[0499] In some embodiments, after steps S1707-S1712 described above, the embodiments of the present invention may transmit the description information of each buffer, the description information of each buffer's buffer slice, and the description information of each accessor to the media access function, and the description information of each buffer and the description information of each buffer's buffer slice to the buffer management module, according to the following solution 2.

[0500] In some embodiments, the implementation method of Solution 2 (sending description information for each buffer, description information for each buffer's buffer slice, and description information for each accessor to the media access function, and sending description information for each buffer and description information for each buffer's buffer slice to the buffer management module) includes the following steps e and f.

[0501] In step e, the display engine transmits the description information for each buffer, the description information for each buffer's buffer slice, and the description information for each accessor to the media access function (including the display engine transmitting the description information for the target buffer, the description information for the target buffer's buffer slice, and the description information for the target accessor to the media access function).

[0502] In response, the media access function receives description information for each buffer and description information for each buffer's buffer slice, transmitted by the display engine (the media access function includes receiving description information for the target buffer, description information for the target buffer's buffer slice, and description information for the accessor, transmitted by the display engine).

[0503] In step f, the display engine transmits the description information for each buffer and the description information for each buffer's buffer slice to the buffer management module (including the display engine transmitting the description information for the target buffer and the description information for the target buffer's buffer slice to the media access function).

[0504] In response, the buffer management module receives description information for each buffer and description information for each buffer's buffer slice transmitted by the display engine (including the media access function receiving description information for the target buffer and description information for the target buffer's buffer slice transmitted by the display engine).

[0505] In some embodiments, the implementation method of Solution 2 (sending description information for each buffer, description information for each buffer's buffer slice, and description information for each accessor to the media access function, and sending description information for each buffer and description information for each buffer's buffer slice to the buffer management module) includes the following steps g and h.

[0506] In step g, the display engine transmits the description information for each buffer, the description information for each buffer's buffer slice, and the description information for each accessor to the media access function (including the display engine transmitting the description information for the target buffer, the description information for the target buffer's buffer slice, and the description information for the target accessor to the media access function).

[0507] In response, the media access function receives description information for each buffer and description information for each buffer's buffer slice, transmitted by the display engine (the media access function includes receiving description information for the target buffer, description information for the target buffer's buffer slice, and description information for the accessor, transmitted by the display engine).

[0508] In step f, the media access function sends the description information for each buffer and the description information for each buffer's buffer slice to the buffer management module (including sending the description information for the target buffer and the description information for the target buffer's buffer slice to the media access function).

[0509] In response, the buffer management module receives description information for each buffer and description information for each buffer's buffer slice sent by the media access function (including the media access function receiving description information for the target buffer and description information for the target buffer's buffer slice sent by the display engine).

[0510] According to the above-mentioned method 1, the description information of each buffer, the description information of each buffer's buffer slice, and the description information of each access are sent to the media access function and the buffer management module, or according to the above-mentioned method 2, the description information of each buffer, the description information of each buffer's buffer slice, and the description information of each access are sent to the media access function, and after the description information of each buffer and the description information of each buffer's buffer slice are sent to the buffer management module, the following steps are continued.

[0511] In S1713, the buffer management module creates each buffer based on the description information of each buffer (including creating the target buffer based on the description information of the target buffer).

[0512] In S1714, the buffer management module divides buffer slices for each buffer based on the buffer slice description information for each buffer (including dividing buffer slices for the target buffer based on the buffer slice description information for the target buffer).

[0513] In S1715, the media access function writes the decrypted data corresponding to each media file to the buffer corresponding to each media file based on the description information of each buffer, the description information of the buffer slice of each buffer, and the description information of each accessor (the media access function also includes writing the decrypted data of the target media file to the target buffer based on the description information of the target buffer, the description information of the buffer slice of the target buffer, and the description information of the target accessor).

[0514] In other words, information such as the buffer capacity in the buffer description of the media access function, the buffer slice capacity in the buffer's buffer slice description, the accessor type in the accessor description, and the data type in the accessor description are used to write the decoded data corresponding to the media file into the buffer in an accurate array format.

[0515] In S1716, the display engine obtains a scene description module corresponding to the 3D scene to be rendered from the scene list of the scene description file.

[0516] In S1717, the display engine obtains description information of the 3D scene to be rendered based on the scene description module corresponding to the 3D scene to be rendered.

[0517] Here, the description information of the 3D scene to be rendered includes the index value of the node description module corresponding to each node in the 3D scene to be rendered.

[0518] In S1718, the display engine obtains node description modules corresponding to each node in the 3D scene to be rendered from the node list of the scene description file, based on the index value of the node description module corresponding to each node in the 3D scene to be rendered.

[0519] In S1719, the display engine obtains description information for each node in the rendering target 3D scene based on the node description module corresponding to each node in the rendering target 3D scene.

[0520] Here, the description information for any node includes the index value of the mesh description module corresponding to the 3D mesh mounted by that node.

[0521] In some embodiments, the description information of any node further includes the name of that node.

[0522] In S1720, the display engine obtains a mesh description module corresponding to a 3D mesh in the 3D scene to be rendered from the mesh list in the scene description file, based on the index value of the mesh description module corresponding to the 3D mesh mounted on each node in the 3D scene to be rendered.

[0523] In S1721, the display engine obtains, based on the mesh description module corresponding to the 3D mesh in the 3D scene to be rendered, the types of data included in the 3D mesh in the 3D scene to be rendered, and accessors for accessing each type of data for each 3D mesh in the 3D scene to be rendered.

[0524] In some embodiments, the method further includes obtaining the name and topology type of the 3D mesh in the rendering target 3D scene based on a mesh description module corresponding to the 3D mesh in the rendering target 3D scene.

[0525] In S1722, the display engine creates each accessor based on the description information of each accessor (including creating accessors for accessing each type of data for each 3D mesh in the rendering target 3D scene based on the description information of the accessors for accessing each type of data for each 3D mesh in the rendering target 3D scene).

[0526] In S1723, the display engine reads the decoded data for each media file from the buffer corresponding to each media file using each accessor (including reading each type of data for each 3D mesh in the rendering target 3D scene from the target buffer memory using accessors for accessing each type of data for each 3D mesh in the rendering target 3D scene).

[0527] In S1724, the display engine renders the 3D scene to be rendered based on the decoded data of each media file.

[0528] In some embodiments, some embodiments of the present application are scene description file generation devices, Memory configured to store computer programs, A processor configured to cause the scene description file generation device to implement the scene description file generation method described in any of the above embodiments when a computer program is called, This provides a device for generating scene description files that include this functionality.

[0529] In some embodiments, the present invention provides a computer-readable storage medium in which a computer program is stored, and which, when the computer program is executed by a computer, causes the computer to implement the method for generating a scene description file described in any of the above embodiments.

[0530] In some embodiments, some embodiments of the present application provide a computer program product that, when executed on a computer, causes the computer to implement the method for generating a scene description file described in any of the above embodiments.

[0531] Finally, it should be noted that the above embodiments are merely for illustrating the technical proposal of the present application and do not limit it. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they may modify the technical proposal described in the above embodiments or make equivalent substitutions for some or all of the technical features therein, and that such modifications or substitutions do not cause the essence of the corresponding technical proposal to deviate from the scope of the technical proposal of each embodiment of the present application.

[0532] For the sake of clarity, the above description has been presented by combining specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit embodiments to the specific forms of the above disclosure. Various modifications and variations are possible according to the above teachings. The above selection and description of embodiments are for the purpose of better illustrating the principles and practical applications, and those skilled in the art will be able to better utilize the above embodiments and the various different variations considered in specific applications.

[0533] The claims may include the following items: (Item 1) A method for generating a scene description file, Determine the type of media file in the 3D scene to be rendered. If the type of the target media file in the rendering target 3D scene is a geometric point cloud compression G-PCC encoded point cloud, a target description module corresponding to the target media file is generated based on the description information of the target media file, and Add the target media description module to the media list of the MPEG media in the scene description file of the 3D scene to be rendered. How to generate a scene description file that includes this. (Item 2) Generating a target description module corresponding to the target media file based on the description information of the target media file is: The method according to item 1, which includes adding a media type syntax element to the options of the target media description module and setting the value of the media type syntax element to a capsule format value corresponding to a G-PCC encoded point cloud. (Item 3) Generating a target description module corresponding to the target media file based on the description information of the target media file is: The method according to item 1, comprising adding a first track index syntax element to an optional track array of the target media description module and setting the value of the first track index syntax element based on the encapsulation scheme of the target media file. (Item 4) Setting the value of the first track index syntax element based on the aforementioned encapsulation method of the target media file is: If the target media file is a single-track capsule file, the value of the first track index syntax element is set to the index value of the code stream track of the target media file. The method of item 3, which includes setting the value of the first track index syntax element to the index value of the geometric code stream track of the target media file, if the target media file is a multitrack capsule file. (Item 5) Generating a target description module corresponding to the target media file based on the description information of the target media file is: The method according to item 1, further comprising adding a codec parameter syntax element to an optional track array of the target media description module, and setting the value of the codec parameter syntax element based on the encoding parameters of the target media file, the type of code stream of the target media file, and the ISO / IEC 23090-18G-PCC data transmission standard. (Item 6) Generating a target description module corresponding to the target media file based on the description information of the target media file is: The method according to item 1, which includes adding a uniform resource identifier syntax element to the options of the target media description module and setting the value of the uniform resource identifier syntax element as the access address of the target media file. (Item 7) Add a target scene description module corresponding to the 3D scene to be rendered to the scene list of the aforementioned scene description file, and Add the index value of the node description module corresponding to the node in the rendering target scene to the node index list of the target scene description module. The method described in item 1, which further includes the method described in item 1. (Item 8) Adding node description modules corresponding to the nodes in the rendering target scene to the node list of the scene description file, and Add the index value of the mesh description module corresponding to the 3D mesh mounted on the node to the mesh index list of the node description module. The method described in item 1, which further includes the method described in item 1. (Item 9) Add a mesh description module corresponding to the 3D mesh in the rendering target scene to the mesh list of the scene description file. Adding syntax elements corresponding to each type of data included in the 3D mesh corresponding to the mesh description module to the mesh description module, and The values ​​of the syntax elements corresponding to each type of data are set as the index values ​​of the accessor description modules corresponding to the accessors for accessing each type of data. The method described in item 1, which further includes the method described in item 1. (Item 10) Adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the mesh description module is: Add an extended list to the primitive of the mesh description module corresponding to the 3D mesh in the target media file. Adding the target extension array to the aforementioned extension list, and Adding syntax elements corresponding to each type of data contained in the corresponding 3D mesh to the aforementioned target extension array, The method described in item 9, which includes the method described in item 9. (Item 11) Adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the mesh description module is: The method according to item 9, which includes adding syntax elements to the attributes of the primitives of the mesh description module that correspond to each type of data contained in the 3D mesh corresponding to the mesh description module. (Item 12) Adding syntax elements corresponding to each type of data contained in the 3D mesh corresponding to the mesh description module to the attributes of the primitives of the mesh description module is: Based on the syntax elements in the first set of syntax elements, syntax elements corresponding to each type of data contained in the corresponding 3D mesh are added to the attributes of the primitives of the first mesh description module, and the first mesh description module is a mesh description module corresponding to a 3D mesh in a media file whose type is a G-PCC encoded point cloud, and Based on the syntax elements in the second set of syntax elements, the attributes of the primitives in the second mesh description module are modified to include syntax elements corresponding to each type of data contained in the corresponding 3D mesh, and the second mesh description module is a mesh description module corresponding to a 3D mesh in a media file whose type is not a G-PCC encoded point cloud. The method described in item 11, which includes the method described in item 11. (Item 13) This further includes adding an accessor description module corresponding to the target accessor to the accessory list of the scene description file, The method according to item 1, wherein the target accessor is an accessor for accessing the decoded data of the target media file. (Item 14) Adding an accessor description module corresponding to the target accessor to the accessory list of the aforementioned scene description file is: Add a data type syntax element to the accessor description module corresponding to the target accessor, and set the value of the data type syntax element based on the type of data accessed by the target accessor. Add an accessor type syntax element to the accessor description module corresponding to the target accessor, and set the value of the accessor type syntax element based on the type of the target accessor. Add a data count syntax element to the accessor description module corresponding to the target accessor, and set the value of the data count syntax element based on the number of data accessed by the target accessor. Add an MPEG time-varying accessor to the accessor description module corresponding to the aforementioned target accessor. The MPEG time-varying accessor is to add a buffer slice index syntax element, and the value of the slice index syntax element is to be set based on the index value of the buffer slice description module corresponding to the buffer slice that stores the data accessed by the target accessor, and The method according to item 13, which includes at least one of the following: adding a time-varying syntax element to the MPEG time-varying accessor, and setting the value of the time-varying syntax element based on whether or not the value of the syntax element in the corresponding target accessor changes over time. (Item 15) This further includes adding a buffer description module corresponding to the target buffer to the buffer list of the scene description file, The method according to item 1, wherein the target buffer is a buffer for storing the decoded data of the target media file. (Item 16) Adding a buffer description module corresponding to the target buffer to the buffer list of the aforementioned scene description file is: Add a first byte length syntax element to the buffer description module and set the corresponding value of the first byte length syntax element based on the capacity of the target buffer. Adding an MPEG ring buffer to the aforementioned buffer description module, Add a segment count syntax element to the MPEG ring buffer and set the value of the corresponding segment count syntax element based on the number of segments stored in the target buffer. Adding a media index syntax element to the MPEG ring buffer, setting the value of the media index syntax element based on the index value of the target media description module, and The method according to item 15, further comprising adding a second track index syntax element to the MPEG ring buffer and setting the value of the second track index syntax element based on the track index value of the source data of the data stored in the target buffer. (Item 17) The method of item 15, further comprising adding a buffer slice description module corresponding to the buffer slice of the target buffer to the buffer slice list of the scene description file. (Item 18) Adding a buffer slice description module corresponding to the buffer slice of the target buffer to the buffer slice list of the aforementioned scene description file is: A buffer index syntax element is added to the buffer slice description module corresponding to the buffer slice of the target buffer, and the value of the buffer index syntax element is set based on the index value of the buffer description module corresponding to the target buffer to which the buffer slice belongs. Add a second byte length syntax element to the buffer slice description module corresponding to the buffer slice of the target buffer, set the value of the second byte length syntax element based on the capacity of the buffer slice, and The method according to item 17, which includes adding an offset amount syntax element to a buffer slice description module corresponding to a buffer slice of the target buffer, and setting the value of the offset amount syntax element based on the offset amount corresponding to the stored data of the buffer slice. (Item 19) A device for generating scene description files, Memory configured to store computer programs, A processor configured to cause the scene description file generation device to implement the scene description file generation method described in any one of items 1-18 when a computer program is called, A generator for scene description files that include this content. (Item 20) A method for rendering a 3D scene, Obtain a scene description file for a 3D scene to be rendered, which contains a target media file whose type is a geometrically-based point cloud compression G-PCC encoded point cloud. Based on the media description module corresponding to the target media file in the media list of the dynamic image expert group MPEG media in the scene description file, the description information of the target media file is obtained. Sending the description information of the target media file to the media access function so that the media access function obtains the target media file based on the description information of the target media file, processes the target media file to obtain decoded data of the target media file, and writes the decoded data of the target media file to the target buffer. Reading the decoded data of the target media file from the target buffer, and Rendering the 3D scene to be rendered based on the decoded data of the target media file. A method for rendering a 3D scene that includes [something]. (Item 21) Sending the description information of the target media file to the aforementioned media access function is: The method of item 20, which includes sending description information of the target media file to the media access function via the media access function application programming interface API. (Item 22) Before reading the decoded data of the target media file from the target buffer, the method: Obtain the buffer description module corresponding to the target buffer from the buffer list of the scene description file. Based on the buffer description module corresponding to the target buffer, the description information of the target buffer is obtained. Obtaining a buffer slice description module corresponding to the buffer slice of the target buffer from the buffer slice list of the scene description file, Based on the buffer slice description module corresponding to the buffer slice of the target buffer, the description information of the buffer slice of the target buffer is obtained, and The method for item 20, further comprising transmitting the description information of the target buffer and the description information of the buffer slices of the target buffer to the buffer management module so that the buffer management module creates the target buffer based on the description information of the target buffer and divides the target buffer into buffer slices based on the description information of the buffer slices of the target buffer. (Item 23) Sending the description information of the target buffer and the description information of the buffer slice of the target buffer to the buffer management module is: The method according to item 22, which includes transmitting description information of the target buffer and description information of the buffer slice of the target buffer to the buffer management module via the buffer API. (Item 24) Before reading the decoded data of the target media file from the target buffer, the method: The method according to item 23, further comprising transmitting the description information of the target buffer and the description information of the buffer slice of the target buffer to the media access function so that the media access function writes the decoded data of the target media file to the target buffer based on the description information of the target buffer and the description information of the buffer slice of the target buffer. (Item 25) Sending the description information of the target buffer and the description information of the buffer slice of the target buffer to the media access function mentioned above means The method according to item 24, which includes sending description information of the target buffer and description information of the buffer slice of the target buffer to the media access function via the media access function API. (Item 26) Sending the description information of the target buffer and the description information of the buffer slice of the target buffer to the buffer management module is: The method of item 22, comprising transmitting the description information of the target buffer and the description information of the buffer slices of the target buffer to the media access function via the media access function so that the media access function transfers the description information of the target buffer and the description information of the buffer slices of the target buffer to the buffer management module. (Item 27) Before reading the decoded data of the target media file from the target buffer, the method: Obtain an accessor description module corresponding to a target accessor, which is an accessor for accessing the decoded data of the target media file, from the accessory list of the scene description file. Based on the accessor description module corresponding to the target accessor, the description information of the target accessor is obtained, and The method of item 22, further comprising transmitting the target access description information to the media access function so that the media access function writes the decoded data of the target media file to the target buffer based on the target access description information. (Item 28) Reading the decoded data of the target media file from the aforementioned target buffer is: The method according to any one of items 20-27, which includes reading each type of data for each 3D mesh in the rendering target 3D scene from the decoded data of the target media file stored in the target buffer, by an accessor for accessing each type of data for each 3D mesh in the rendering target 3D scene. (Item 29) Before the accessor reads the data of each type of 3D mesh in the rendering target 3D scene from the decoded data of the target media file stored in the target buffer, the method, Based on the index value of the mesh description module corresponding to the 3D mesh mounted on each node of the 3D scene to be rendered, the mesh description module corresponding to each 3D mesh in the 3D scene to be rendered is obtained from the mesh list of the scene description file, and The method of item 28, further comprising obtaining the data types contained in each 3D mesh in the rendering target 3D scene and accessors for accessing each type of data in each 3D mesh in the rendering target 3D scene, based on a mesh description module corresponding to each 3D mesh in the rendering target 3D scene. (Item 30) Before obtaining the mesh description module corresponding to each 3D mesh in the rendering target 3D scene from the mesh list of the scene description file based on the index value of the mesh description module corresponding to each 3D mesh mounted on each node in the rendering target 3D scene, the method: Based on the index value of the node description module corresponding to each node in the 3D scene to be rendered, obtain the node description module corresponding to each node in the 3D scene to be rendered from the node list of the scene description file, and Based on the node description module corresponding to each node in the rendering target 3D scene, the description information of each node in the rendering target 3D scene is obtained, and the description information of any node includes the index value of the mesh description module corresponding to the 3D mesh mounted on that node. The method described in item 29, which further includes the method described in item 29. (Item 31) Before obtaining the node description module corresponding to each node in the 3D scene to be rendered from the node list of the scene description file based on the index value of the node description module corresponding to each node in the 3D scene to be rendered, the method Obtain the scene description module corresponding to the 3D scene to be rendered from the scene list of the aforementioned scene description file. Based on the scene description module corresponding to the rendering target 3D scene, the description information of the rendering target 3D scene is obtained, and the description information of the rendering target 3D scene includes the index value of the node description module corresponding to each node in the rendering target 3D scene. The method described in item 30, which further includes the method described in item 30. (Item 32) The description information of any 3D mesh further includes an index value of an accessor description module corresponding to an accessor for accessing each type of data of the 3D mesh, and obtaining an accessor for accessing each type of data of each 3D mesh in the rendering target 3D scene based on the mesh description module corresponding to each 3D mesh in the rendering target 3D scene is: Based on the index value of the accessor description module corresponding to the accessor for accessing each type of data for each 3D mesh in the rendering target 3D scene, the accessor description module corresponding to the accessor for accessing each type of data for each 3D mesh in the rendering target 3D scene is obtained from the accessory list of the scene description file. Based on the accessor description module corresponding to the accessor for accessing each type of data for each 3D mesh in the rendering target 3D scene, the description information of the accessor for accessing each type of data for each 3D mesh in the rendering target 3D scene is obtained, and Based on the description information of the accessors for accessing each type of data of each 3D mesh in the rendering target 3D scene, an accessor for accessing each type of data of each 3D mesh in the rendering target 3D scene is created. The method described in item 31, which includes the method described in item 31. (Item 33) A rendering device for 3D scenes, Memory configured to store computer programs, A processor configured to cause the rendering device for the 3D scene to implement the rendering method for the 3D scene described in any one of items 20-32 when a computer program is called, A rendering device for 3D scenes, including [specific elements].

Claims

1. A method for parsing scene description files, Obtain a scene description file for a 3D scene to be rendered that contains a target media file whose type is a G-PCC encoded point cloud. Obtaining a target media description module corresponding to the target media file from the media list of the dynamic image expert group MPEG media in the scene description file, and To obtain the description information of the target media file based on the target media description module, Includes, Here, the optional track array of the target media description module includes a first track index syntax element, If the target media file is a single-track capsule file, the value of the first track index syntax element is the index value of the code stream track of the target media file. A method for parsing a scene description file in which, if the target media file is a multi-track capsule file, the value of the first track index syntax element is the index value of the geometric code stream track of the target media file.

2. Obtaining description information of the target media file based on the target media description module is: Based on the value of the media name syntax element in the target media description module, the name of the target media file is obtained. To determine whether the target media file should be automatically played based on the value of the autoplay syntax element in the target media description module. Based on the value of the circular playback syntax element in the target media description module, it is determined whether or not the target media file needs to be played back in a circular fashion. Based on the value of the media type syntax element in the options of the target media description module, the capsule format of the target media file is obtained. Obtain the access address of the target media file based on the value of the only address identifier syntax element in the options of the target media description module. Based on the value of the first track index syntax element in the optional track array of the target media description module, track information of the target media file is obtained, and The method according to claim 1, further comprising determining the type of code stream and decoding parameters of the target media file based on the values ​​of the codec parameter syntax elements in the optional track array of the target media description module and the G-PCC data transmission standard.

3. The method according to claim 2, wherein the value of the codec parameter syntax element is one of gpcb, gpeb, and gpc1.

4. Obtaining a target scene description module corresponding to the rendering target 3D scene from the scene list of the aforementioned scene description file, and The method according to claim 1, further comprising obtaining description information of the rendering target 3D scene based on the target scene description module.

5. Obtaining description information of the rendering target 3D scene based on the target scene description module is: The method according to claim 4, further comprising determining the index value of the node description module corresponding to each node in the three-dimensional scene to be rendered, based on the index value declared in the node index list of the target scene description module.

6. After determining the index value of the node description module corresponding to each node in the rendering target 3D scene based on the index value declared in the node index list of the target scene description module, the method then: Based on the index value of the node description module corresponding to each node in the 3D scene to be rendered, obtain the node description module corresponding to each node in the 3D scene to be rendered from the node list of the scene description file, and The method according to claim 5, further comprising obtaining description information for each node in the rendering target 3D scene based on a node description module corresponding to each node in the rendering target 3D scene.

7. Obtaining description information for each node in the rendering target 3D scene based on the node description module corresponding to each node in the rendering target 3D scene is: Based on the value of the node name syntax element in the node description module corresponding to each node in the 3D scene to be rendered, the name of each node in the 3D scene to be rendered is obtained, and The method according to claim 6, comprising at least one of: determining the index value of a mesh description module corresponding to a 3D mesh mounted on each node in the 3D scene to be rendered, based on the index value declared in the mesh index list of a node description module corresponding to each node in the 3D scene to be rendered.

8. After determining the index value of the mesh description module corresponding to the 3D mesh mounted on each node in the 3D scene to be rendered, the method proceeds as follows: Based on the index value of the mesh description module corresponding to the 3D mesh mounted on each node in the 3D scene to be rendered, the mesh description module corresponding to the 3D mesh mounted on each node in the 3D scene to be rendered is obtained from the mesh list of the scene description file, and The method according to claim 7, further comprising obtaining description information of a three-dimensional mesh mounted on each node in the three-dimensional scene to be rendered, based on a mesh description module corresponding to a three-dimensional mesh mounted on each node in the three-dimensional scene to be rendered.

9. Obtaining description information of a 3D mesh mounted on each node in the 3D scene to be rendered, based on a mesh description module corresponding to a 3D mesh mounted on each node in the 3D scene to be rendered, Obtain the name of each 3D mesh based on the mesh name syntax element in the mesh description module corresponding to each 3D mesh. Based on the data type syntax element in the mesh description module corresponding to each 3D mesh, the data types contained in each 3D mesh are obtained. Based on the values ​​of the syntax elements for each data type, obtain the index value of the accessor description module corresponding to the accessor for accessing each type of data for each 3D mesh, and The method according to claim 8, comprising at least one of: obtaining the type of topological structure of each three-dimensional mesh based on the value of a mode syntax element in a mesh description module corresponding to each three-dimensional mesh.

10. The method according to claim 7, wherein the attributes of the primitives of the mesh description module include syntax elements corresponding to each type of data contained in the corresponding three-dimensional mesh, the syntax elements corresponding to each type of data are set up based on a first syntax element set and a second syntax element set, the first syntax element set includes syntax elements supported in the attributes of the primitives of the mesh description module of the scene description file, and the second syntax element set includes syntax elements pre-set for the G-PCC encoded point cloud.

11. The method according to claim 7, wherein the attributes of the primitives of the mesh description module include syntax elements corresponding to each type of data contained in the corresponding three-dimensional mesh, the syntax elements corresponding to each type of data are set up based on a first set of syntax elements, and the first set of syntax elements includes syntax elements supported by the attributes of the primitives of the mesh description module in the scene description file.

12. The method according to claim 7, wherein the attributes of the primitives of the mesh description module include syntax elements corresponding to each type of data contained in the corresponding three-dimensional mesh, the syntax elements corresponding to each type of data are set up based on a second set of syntax elements, and the second set of syntax elements includes syntax elements pre-set for the G-PCC coded point cloud.

13. The method according to claim 11, wherein the first set of syntax elements includes geometric coordinate syntax elements, color value syntax elements, normal vector syntax elements, tangent vector syntax elements, texture coordinate syntax elements, articulation syntax elements, and weight syntax elements.

14. The method according to claim 12, wherein the second set of syntax elements includes G-PCC geometric coordinate syntax elements, G-PCC color value syntax elements, G-PCC normal vector syntax elements, G-PCC tangent vector syntax elements, G-PCC texture coordinate syntax elements, G-PCC joint syntax elements, and G-PCC weight syntax elements.

15. After obtaining the index value of the accessor description module corresponding to the accessor for accessing each type of data for each 3D mesh based on the value of the syntax element for each data type, the method then: Based on the index value of the accessor description module corresponding to the accessor for accessing each type of data for each 3D mesh, the accessor description module corresponding to the accessor for accessing each type of data for each 3D mesh is obtained from the accessory list of the scene description file, and The method according to claim 9, further comprising obtaining descriptive information for accessors to access each type of data for each three-dimensional mesh, based on accessor description modules corresponding to accessors for accessing each type of data for each three-dimensional mesh.

16. To obtain each buffer description module in the buffer list of the aforementioned scene description file, To obtain the value of the media index syntax element in each buffer description module, Determine a buffer description module whose media index syntax element value is the same as the index value of the target media description module as the target buffer description module corresponding to the target buffer for buffering the decoded data of the target media file, and The method according to claim 1, further comprising obtaining description information of the target buffer based on the target buffer description module.

17. Obtaining the description information of the target buffer based on the target buffer description module is: The capacity of the target buffer is obtained based on the value of the first byte length syntax element in the target buffer description module. Based on whether or not the target buffer description module includes an MPEG ring buffer, it is determined whether or not the target buffer is a ring buffer based on MPEG extension modifications. Based on the value of the segment count syntax element in the MPEG ring buffer of the target buffer description module, the number of storage segments in the MPEG ring buffer is obtained, and The method according to claim 16, comprising at least one of obtaining the track index value of the source data of the data buffered in the MPEG ring buffer based on the value of a second track index syntax element in the MPEG ring buffer of the target buffer description module.

18. To obtain each buffer slice description module in the buffer slice list of the aforementioned scene description file, To obtain the value of the buffer index syntax element in each buffer slice description module, Determine a buffer slice description module whose buffer index syntax element value is the same as the index value of the target buffer description module as the buffer slice description module corresponding to the target buffer, and The method according to claim 16, further comprising obtaining description information of the buffer slice of the target buffer based on a buffer slice description module corresponding to the buffer slice of the target buffer.

19. Obtaining description information of a buffer slice of the target buffer based on a buffer slice description module corresponding to a buffer slice of the target buffer is: Based on the value of the second byte length syntax element in the buffer slice description module corresponding to the buffer slice of the target buffer, the capacity of the buffer slice of the target buffer is obtained, and The method according to claim 18, comprising at least one of obtaining the offset amount of the buffer slice of the target buffer based on the value of the offset amount syntax element in the buffer slice description module corresponding to the buffer slice of the target buffer.

20. To obtain each accessor description module in the accessory list of the aforementioned scene description file, To obtain the value of the buffer slice index syntax element in each accessor description module, Determine an accessor description module whose buffer slice index syntax element value is the same as the index value of the buffer slice description module corresponding to the buffer slice of the target buffer as the accessor description module for accessing data in the buffer slice of the target buffer, and The method according to claim 18, further comprising obtaining description information of an accessor for accessing data in a buffer slice of the target buffer, based on an accessor description module corresponding to an accessor for accessing data in a buffer slice of the target buffer.

21. Obtaining accessor description information based on the accessor description module corresponding to the accessor is possible. The type of data an accessor accesses is determined based on the value of the data type syntax element in the accessor description module. Determining the accessor type based on the value of the accessor type syntax element in the accessor description module. The number of data items accessed by an accessor is determined based on the value of the data count syntax element in the accessor description module. To determine whether an accessor is a time-varying accessor based on MPEG extension modifications, based on whether or not the accessor description module contains MPEG time-varying accessors. Based on the value of the buffer slice index syntax element in the MPEG time-varying accessor of the accessor description module, the index value of the buffer slice description module corresponding to the buffer slice that stores the data accessed by the accessor is determined, and The method according to claim 15, comprising at least one of determining whether the value of a syntax element in an accessor changes over time based on the value of a time-varying syntax element in an MPEG time-varying accessor of an accessor description module.

22. Obtaining accessor description information based on an accessor description module corresponding to an accessor is: The type of data an accessor accesses is determined based on the value of the data type syntax element in the accessor description module. Determining the accessor type based on the value of the accessor type syntax element in the accessor description module. The number of data items accessed by an accessor is determined based on the value of the data count syntax element in the accessor description module. To determine whether an accessor is a time-varying accessor based on MPEG extension modifications, based on whether or not the accessor description module contains MPEG time-varying accessors. Based on the value of the buffer slice index syntax element in the MPEG time-varying accessor of the accessor description module, the index value of the buffer slice description module corresponding to the buffer slice that stores the data accessed by the accessor is determined, and The method according to claim 20, comprising at least one of determining whether the value of a syntax element in an accessor changes over time based on the value of a time-varying syntax element in an MPEG time-varying accessor of an accessor description module.

23. A scene description file parser, Memory configured to store computer programs, A scene description file analysis device, which includes a processor configured to cause the rendering device for the three-dimensional scene to implement the scene description file analysis method described in any one of claims 1 to 22 when a computer program is called.

Images