System and method for semantic coding and decoding of grid primitives

Abstract

A method includes obtaining a scene description file including a data structure representing objects of an immersive video system, where the data structure includes a segment extension indicating that a mesh object is segmented and including data for each mesh segment; and rendering the scene from the scene description file according to the data structure such that the object is at least partially rendered in the scene. Associated apparatus, a computer-readable medium comprising instructions for causing one or more processors to perform the method, a computer-readable medium computer program product comprising instructions for causing one or more processors to perform the method, and a signal comprising a scene description file for a 3D scene are also described.

Description

Systems and methods for mesh primitive semantic encoding and decoding

[0001] Cross-reference: This application claims priority to European Patent Application No. 23306719.8, filed on October 6, 2023, entitled “System and Method of Mesh Primitive Semantical Coding,” which is incorporated herein by reference in its entirety. Background Technology

[0002] This disclosure relates to 3D scene and object interaction in immersive environments, including but not limited to mesh extensions that provide semantic descriptions of primitives created under mesh objects.

[0003] Various technologies can be used to generate, process, and render virtual 3D scenes. The information characterizing a 3D scene (called a scene description) can be time-dependent, allowing the 3D scene to change in a manner similar to video playback. This behavior can be achieved by relying on the framework defined in the scene description of the MPEG media document (Information Technology – Codec Representation of Immersive Media – Part 14: Scene Description of MPEG Media, ISO / IEC DIS 23090-14:2021(E)). A scene update mechanism based on the JSON patching protocol defined in IETF RFC 6902 can be used to synchronize virtual content with MPEG media streams.

[0004] Extended Reality (XR) is a technology that enables interactive experiences where real-world environments and / or video content are enhanced by virtual content. This virtual content can be utilized across multiple sensory modalities, including visual, auditory, and tactile. During application runtime, virtual content (e.g., 3D content or audio / video files) is rendered in real-time in a manner consistent with the user's context (environment, viewpoint, device, etc.). A scene graph (such as glTF (Graphics Language Transfer Format) and its extensions described in MPEG Scene Description Format or USDZ) is a representation of the content to be rendered. The scene graph combines a declarative description of the scene structure linking real-world objects and virtual objects with a binary representation of the virtual content. The scene description framework ensures that timed media and corresponding associated virtual content are available at any time during application rendering. The scene description can also carry scene-level data describing how the user interacts with scene objects at runtime to achieve an immersive XR experience. Summary of the Invention

[0005] A method includes: obtaining a scene description file containing a data structure representing objects of an immersive video system, wherein the data structure includes segmentation extensions indicating that mesh objects are segmented and including data for each mesh segment; and rendering a scene from the scene description file according to the data structure, such that objects are at least partially rendered in the scene. Associated means are also described, including a computer-readable medium including instructions for causing one or more processors to perform the method, a computer program product including the computer-readable medium including instructions for causing one or more processors to perform the method, and a signal including a scene description file for a 3D scene. Attached Figure Description

[0006] Figure 1 shows a flowchart of a method for rendering objects in a scene according to at least one embodiment.

[0007] Figure 2 shows a flowchart of a method for parsing semantic arrays according to at least one embodiment.

[0008] Figure 3 shows a flowchart of a method for parsing a segmented array according to at least one embodiment.

[0009] Figure 4 shows a flowchart of a method for parsing primitives according to at least one embodiment.

[0010] Figure 5 shows a grid consisting of seven primitives with corresponding semantics according to at least one embodiment.

[0011] Figure 6 shows the front of each cube / primitive of a grid consisting of seven primitives with corresponding semantics, according to at least one embodiment.

[0012] Figure 7 shows a block diagram of a device for displaying and / or rendering 3D scene information according to at least one embodiment. Detailed Implementation

[0013] In some 3D applications, scene descriptions are used to combine an explicit and easily parsed description of the scene structure with some binary representation of the media content.

[0014] In time-based media streaming, the scene description itself can evolve over time to provide relevant virtual content for each sequence of the media stream. For example, for advertising purposes, a virtual bottle could be displayed during a video sequence of people drinking from it.

[0015] This behavior can be achieved by relying on the framework defined in the scene description of the MPEG media document (Information Technology – Coded and Decoded Representation of Immersive Media – Part 14: Scene Description of MPEG Media, ISO / IEC DIS 23090-14:2021(E)). A scene update mechanism based on the JSON patch protocol defined in IETF RFC 6902 can be used to synchronize virtual content with the MPEG media stream.

[0016] Generally, animation composition can be performed before creating a glTF file or other runtime asset delivery file. During this preprocessing, the composition is calculated according to any custom rules, and the resulting new animation is stored in the runtime asset delivery file, which allows for the transfer and rendering of the final scene.

[0017] This disclosure relates to 3D scene and object interaction in immersive environments. Segmented expansion of the mesh provides a semantic description of primitives created under the mesh object. This mechanism allows the use of sub-components of mesh primitives without requiring mesh fragmentation or consideration of how the mesh is rendered in the scene.

[0018] This disclosure introduces node or segmentation extensions for MPEG-I Scene Description (SD) to support mesh partitioning / segmentation in interactive 3D environments. The current node level in the glTF format does not allow 3D mesh primitives to be partitioned / segmented, nor does it allow the definition or description of high-level semantic descriptors. In cases with several sub-parts, generating submeshes and associating nodes with each sub-part becomes cumbersome; these sub-parts may eventually intersect, and the scene description is over-filled with repetitive geometric information without contextual information. Using primitive arrays in object meshes facilitates mesh subdivision, although subdivided mesh parts lose local context at a finer level, making it difficult for engines and applications to infer / use interactive models within mesh sub-parts.

[0019] This disclosure describes several mechanisms for referencing mesh sub-components and including contextual information about these regions. This solution advantageously reduces the number of sub-mesh to a single mesh, improves the contextual information of mesh primitives, and provides several methods for segmenting meshes and primitives using, for example, the glTF format.

[0020] In the data structures used by the glTF format and scene graphics architecture, nodes point to meshes and child nodes, assuming that nodes exist in a similar mesh hierarchy. While not always the case, manually performing, organizing, and labeling or identifying such parts of 3D data can be difficult and laborious when a 3D mesh is subdivided into parts of an entire object. The described solution facilitates meshing without copying 3D geometry or buffer information.

[0021] Table 1 includes node object properties in glTF format. In this table and throughout all other grammatical descriptions herein, the indication that specific data is "required" indicates only that a particular embodiment requires that specific data to be present; this should not be construed as an indication that the information is required in all embodiments. Table 1.

[0022] Table 2 includes the properties of mesh objects in glTF format; Table 2.

[0023] Mesh primitive objects facilitate the partitioning of mesh objects, although semantic information present at the mesh level is lost. Current solutions for mesh-based representation formats do not provide contextual meaning for each primitive. Table 3 illustrates the lack of contextual information about primitives. The segmentation extension described in this paper provides a mechanism for semantically labeling or identifying mesh primitives. Segmentation extension is applied at the "mesh.primitive" level, for example, at the node level. Table 3 includes the properties of mesh.primitive objects in glTF format; Table 3.

[0024] Mesh segments are parts of a mesh that, when combined with other mesh segments, reconstruct the complete mesh. A single mesh segment can also represent a complete mesh. Mesh segments provide information such as geometric and texture information for the mesh portion, as well as additional metadata that provides relevant information, such as the semantic relationship between the mesh segment and the mesh, and any non-geometric and non-texture information used for the mesh segment.

[0025] Figure 1 illustrates a flowchart of a method for rendering objects in a scene. A scene description file containing data structures representing objects in an immersive video system is obtained at step 102. This scene description file can be obtained by receiving a transmitted file (including a streaming file or a file received via wireless or wired communication resources), reading data from a storage medium, etc. The schema of the scene description file can advantageously be an MPEG-I scene description file. The scene description file can be a JSON file. It can also be a glTF file. The data structures include segmentation extensions, such as MPEG_mesh_segmentation extensions, which indicate that mesh objects are segmented and include data for each mesh segment. The method may optionally include parsing, reading, or extracting the data structure from the scene description file at step 104, and processing the data structure at step 106 for use, for example, in a synthetic 3D model. For example, such processing may include reading and displaying or visualizing the synthetic 3D model as a mesh, converting the mesh to a point cloud, removing hidden points from the point cloud, converting the point cloud to a data frame including coordinates and vectors, and storing the resulting data. The scene is then rendered from the scene description file at step 108, according to the data structure, such that objects are at least partially rendered in the scene.

[0026] Data structures can advantageously include static and formatted representations. Static representations can contain metadata, such as metadata representing a user's identity, including at least one of the user's name, age, and input device. Formatted representations can point to a 3D avatar formatted representation containing at least one of geometry, texture mapping, and animation parameters. For example, based on user-provided data, data structures can facilitate appearance generalization and stylization.

[0027] The data structure advantageously includes a representation of an object that is compatible with any immersive, augmented, and / or streaming technology that takes into account the transmission of object data or human data, such as 2D videos or images containing bodies or faces, or 3D technologies for rendering, animate, or manipulating human assets. The data structure may include, for example, a digital representation of a human, comprising at least one of a skeletal structure, an object-specific model, a statistical shape model, or metadata representing an individual's social attributes or parameters. The data structure may be a synthetic 3D representation of a digital human.

[0028] The format of primitive semantics can conform to the glTF format and is compatible with current MPEG efforts to extend glTF with MPEG extensions. Although this document uses the glTF format to describe example embodiments, it should be understood that the principles described herein are not limited to using glTF. For example, different embodiments may use different formats, such as XML, USD, etc.

[0029] Object "semantics" are advantageously added at the root level of the mesh object. Segment extensions for mesh objects in glTF 2.0 (e.g., MPEG_mesh_segmentation extension) are included in data structures in the scene description file, for example, to indicate that a mesh with segmentation extensions is segmented, and that the geometry and texture data for each mesh segment exists in or is included in the scene description file. This information is stored in the mesh.primitive attribute of the mesh object. The presence of segmentation extensions (such as MPEG_mesh_segmentation) in the scene description file can be indicated by mechanisms such as the extensionRequired and extensionUsed attributes in glTF 2.0. .

[0030] Segmentation extensions provide information such as semantic attributes that associate grid primitives with grid segments. Grid-level segmentation extensions (such as MPEG_grid_segmentation) have the following description: .

[0031] The object "semantic" is an array of strings that allows content creators to add context to the meaning of each primitive in the grid. The array "semantic" is index-aligned with the array of primitives, so the number of elements in the "semantic" array matches the number of elements in the "primitive" array. This index alignment facilitates grid segmentation and labeling, enabling immersive applications to act appropriately relative to the primitives.

[0032] Grid-level segmentation extensions (such as MPEG_Grid_Segmentation) have the following description: .

[0033] The "Segment" object contains an array of "Segment" objects. A "Segment" is an object containing the string ("Semantic" property) and index ("Index" property) of a primitive in the grid. The "Segment" object facilitates the semantic description of primitives and gives content creators the freedom not to describe each primitive if it is unnecessary or undesirable.

[0034] Object "semantics" is advantageously added at the root level of "mesh.primitive" objects. Primitive semantic extensions for "mesh.primitive" objects in glTF 2.0 (e.g., "MPEG_primitive_semantic" extensions) are included in data structures within scene description files (e.g., glTF) to indicate that "mesh.primitives" with "MPEG_primitive_semantic" extensions have associated semantic information. The presence of MPEG_primitive_semantic extensions in scene description files can be indicated by mechanisms such as the `extensionRequired` and `extensionUsed` attributes in glTF 2.0. .

[0035] For example, primitive semantic extensions such as MPEG_primitive_semantics provide information, such as semantic attributes that associate "mesh.primitive" with mesh segments. The primitive semantic extensions at the mesh.primitive level have the following descriptions: .

[0036] Object "semantics" are strings that enable content creators to add context to the meaning of each primitive in a grid. Object "semantics" facilitate grid segmentation and labeling, allowing immersive applications to act appropriately relative to the primitives.

[0037] The primitive semantic extension at the grid primitive level (e.g., MPEG primitive semantics) has the following description: .

[0038] Segmented objects in an array may have the following description: .

[0039] In the examples provided herein, the indication that a syntax element is optional ("O") or mandatory ("M") merely indicates that, according to the syntax of the particular embodiment shown, the syntax element is optional or mandatory. A syntax element marked as mandatory for a particular embodiment is not necessarily mandatory (or even not used) in different embodiments. The object "Segment" contains an array of "Segment" objects. A "Segment" is an object containing a string ("Semantic" attribute) and an index ("Index" attribute) of primitives in the mesh. "Semantic" describes the meaning or context of this segmentation. "Index" points to an accessor to segment "mesh.primitive". The object "Segment" contains vertex indices associated with the semantics. The object "Segment" facilitates the semantic description of primitives and gives content creators the freedom to segment primitives with relevant contexts.

[0040] Figure 2 illustrates a flowchart of the method for parsing the semantic array. This method parses the naming convention of each primitive and eliminates ambiguity for identical naming conventions. The process begins with processing the grid from node objects (202). When a segmentation extension, such as “MPEG_Grid_Segmentation”, exists or is included (204) in the scene description file, the process checks (206) whether the “Primitive” array and the “Semantic” array have the same or equal dimensions. If the dimensions differ (206), a warning or error is reported (208). If the dimensions are the same or equal (206), the semantic array is parsed (210). Each string in the semantic array is separated by the character “ / ”, and a path is created and stored for each string to represent a path or hierarchy. Once all strings have been parsed, identical pathnames are disambiguated to facilitate accurate string processing. When all pathnames are unique (212), data structure processing continues. When any two or more pathnames are identical (212), different indices are added (214) to each identical path to eliminate ambiguity for identical paths until each pathname is unique, and processing continues. The index numbers do not need to be provided in any particular order. For example, the semantic strings “left_arm / hand” and “left_arm / hand” have the same path / string, so the parsing mechanism can generate “left_arm / hand / 0” and “left_arm / hand / 1” to eliminate the ambiguity of the same name.

[0041] Figure 3 illustrates a flowchart of the method for parsing the segment array. This method parses the naming convention of each primitive and eliminates ambiguity for identical naming conventions. The process begins with processing the node object 302 (grid). When a segment extension, such as “MPEG_grid_segment”, exists or is included 304 in the scene description file, the process checks 306 whether the “primitive” array and the “segment” array have the same or equal dimensions. If the dimensions differ 306, a warning or error is reported 308. If the dimensions are the same or equal 306, the segment array is parsed 310. Each string in the segment array is separated by the character “ / ”, and a path is created and stored for each string to represent a path or hierarchy. Once all strings have been parsed, identical path names are eliminated to facilitate accurate string processing. When all path names are unique 312, data structure processing continues. When any two or more path names are identical 312, different indices are added 314 to each identical path to eliminate ambiguity for identical paths until each path name is unique, and processing continues. The index numbers do not need to be provided in any particular order. For example, the semantic strings “left_arm / hand” and “left_arm / hand” have the same path / string, so the parsing mechanism can generate “left_arm / hand / 0” and “left_arm / hand / 1” to eliminate the ambiguity of the same name.

[0042] Figure 4 illustrates a flowchart of the method for parsing primitives. This method parses the naming convention of each primitive and eliminates ambiguity for identical naming conventions. The mesh is processed from node objects, and each primitive is parsed 402. When a segmentation extension, such as “MPEG_mesh_segmentation”, exists or is included 404 in the scene description file, a semantic array is parsed 406, and each string in the semantic array is separated by the character “ / ” 406, and a path is created and stored for each string to represent a path or hierarchy. If not all primitives are parsed 408, the process continues at 402, processing another primitive. When all primitives have been parsed 408, identical pathnames are disambiguated to facilitate accurate string processing. When all pathnames are unique 410, data structure processing continues. When any two or more pathnames are identical 410, different indices are added 412 to each identical path to eliminate ambiguity for identical paths until each pathname is unique, and processing continues. The index numbers do not need to be provided in any particular order. For example, the semantic strings “left_leg / foot” and “left_leg / foot” have the same path / string, so the parsing mechanism can generate “left_leg / foot / 0” and “left_leg / foot / 1” to eliminate the ambiguity of the same name.

[0043] The glTF files below are examples of segmented expansion instantiations (not exhaustive). Numerous instantiations are possible, depending on the application. These examples only show portions of glTF files relevant to various implementations. Data (buffers, accessors, etc.) is included but not shown for constructing mesh primitives.

[0044] Figure 5 shows a mesh composed of seven primitives with corresponding semantic meanings. In this example, the mesh consists of seven cubes. Semantic meanings are added to each of these primitives. For example, the top cube represents the character's head, where 0 represents white. One implementation is as follows: .

[0045] The glTF loader first decodes the regular content (buffers, meshes, nodes, etc.) excluding the segmented extension data. A single mesh has seven primitives, each with the same topology (same "index") and specific vertex coordinates and colors (different "POSITION" and "COLOR_0" attributes). Once the default loading is complete, the segmented extension data is parsed. The segmented extension contains a single "semantic" attribute with seven strings. Each string describes the semantics of the primitives in the same order, as shown in Figure 5. For example, the first value of "semantic" is "full_body / upper_body / head" and corresponds to the first primitive, and so on.

[0046] The following example solves the same problem as the previous example, except that the mesh includes multiple mesh segments: .

[0047] The glTF loader first decodes the regular content (buffers, meshes, nodes, etc.) excluding the segmented extension data. A single mesh has seven primitives, each with the same topology (same "index") and specific vertex coordinates and colors (different "POSITION" and "COLOR_0" attributes). Once the default loading is complete, the segmented extension data is parsed. The segmented extension contains a single "segment" attribute, and there are seven objects. As shown in Figure 5, each object describes the semantics of the primitives in the same order. For example, the first object contains a "semantic" attribute with "full_body / upper_body / head" and an "index" attribute with a value of 0, corresponding to the first primitive.

[0048] This example solves the same problem as the previous example, except that it includes a single semantic object: .

[0049] The glTF loader first decodes the ordinary content (buffers, meshes, nodes, etc.) excluding primitive semantic extension data. A single mesh has seven primitives, each with the same topology (same "index") and specific vertex coordinates and colors (different "POSITION" and "COLOR_0" attributes). Once the default loading is complete, the primitive semantic extension data is parsed. Each primitive contains a single "semantic" attribute describing the primitive's semantics. For example, the first primitive contains a primitive semantic extension like MPEG_primitive_semantic, where the "semantic" attribute is "full_body / upper_body / head".

[0050] Figure 6 shows the front face of each cube / primitive in a mesh composed of seven primitives with corresponding semantics. In the example below, the same mesh and primitives as in the previous example are used, except that semantics are assigned to the front face of each cube / primitive. One possible implementation is as follows: .

[0051] The glTF loader first decodes the ordinary content (buffers, meshes, nodes, etc.) excluding primitive semantic extension data. A single mesh has seven primitives, each with the same topology (identical "index") and specific vertex coordinates and colors (different "POSITION" and "COLOR_0" attributes). Once the default loading is complete, the primitive semantic extension data is parsed. Each primitive contains a single "segment" attribute describing the primitive's semantics. For example, the first primitive contains a primitive semantic extension like MPEG_primitive_semantic, where the "segment" attribute has a single item where "semantic" equals "full_body / upper_body / head / face," and the "index" references an accessor with six values ​​corresponding to the vertices of the front triangle of the cube.

[0052] Human digital representations can take many different forms, such as grids, volumes, point clouds, images, videos, sounds, and so on. These representations facilitate the creation and formalization of digital media content. To allow for interoperable representations, templates are formalized that capture one or more different representations of humans. These templates can be general human models with attached skeletal structures, object-specific models, statistical shape models, or metadata representing individual social attributes or parameters. Any of these approaches can precisely provide a human image as an initial stage or basic model capable of statistically representing different human representations.

[0053] Synthetic 3D models are representations of digital humans. Synthetic representations, or models, are easier to manipulate and create to match specific human anatomical, visual, and social parameters or visual cues, and can be animated in immersive reality. Synthetic representations also facilitate the generalization and stylization of appearance, which can be performed by professionals and used for animation and streaming.

[0054] In this specification, some embodiments utilize the glTF file format and are compatible with recent MPEG-I Scene Description (SD) efforts to extend glTF with MPEG extensions, where MPEG stands for Moving Picture Experts Group. However, the concepts described herein are not limited to use with glTF and can be encoded and decoded in other formats, such as XML, USD, or other modes for delivering runtime assets.

[0055] The above solution applies to the encoding of scene description files for a scene. One approach involves dividing a mesh of objects in the scene into mesh segments and encoding a data structure representing the objects in an immersive video system in the scene description file. This data structure includes segmentation extensions that indicate the mesh object is segmented and include data for each mesh segment. The data may include at least one of geometric data and texture data for each mesh segment. The data may be stored in the mesh primitive attributes of the mesh object. The data structure may also include primitive semantic extensions to indicate that the primitives have associated semantic information. The segmentation extensions can provide information including semantic attributes that associate mesh primitives with mesh segments. Semantic objects may be added at one or more root levels of the mesh object and mesh primitive objects. The scene description file may include extensions to at least one of the mesh object and mesh primitive objects. A semantic array can be resolved by dividing each string of the semantic array with the character " / " and creating a path for each string. A segmented array can be resolved by dividing each string of the segmented array with the character " / " and creating a path for each string. The same path name can be disambiguated by adding different indices to each identical path name.

[0056] Devices for displaying and / or rendering 3D scene information and their control electronics can be implemented using systems such as the system shown in Figure 7. Figure 7 is a block diagram of an example system implementing various embodiments. System 700 can be implemented as a device including the various components described below and configured to perform one or more processes described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptops, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 700 can be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 700 are distributed across multiple ICs and / or discrete components. In various embodiments, system 700 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 700 is configured to provide one or more features described in this document.

[0057] System 700 includes at least one processor 702 configured to execute instructions loaded therein for performing various features or methods, such as those described in this document, including the method of FIG. 2. Processor 702 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 700 includes at least one memory 704, such as a volatile memory device and / or a non-volatile memory device. System 700 includes a storage device 706, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 706 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0058] System 700 includes an encoder / decoder module 708, which is configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 708 may include its own processor and memory. The encoder / decoder module 708 may represent a module that can be included in a device to perform encoding and / or decoding functions. It is well known that a device may include one or both encoding and decoding modules. Furthermore, the encoder / decoder module 708 may be implemented as a separate element of system 700, or it may be incorporated within processor 702 as a combination of hardware and software known to those skilled in the art.

[0059] Program code to be loaded onto processor 702 or encoder / decoder 708 to execute the various aspects described in this document may be stored in storage device 706 and subsequently loaded into memory 704 for execution by processor 702. According to various embodiments, one or more of processor 702, memory 704, storage device 706, and encoder / decoder module 708 may store one or more items of various types during the execution of the processes described in this document. Such stored items may include, but are not limited to, a scene description file containing data structures including user attribute object 302, input video, decoded video, or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing of equations, formulas, operations, and operational logic.

[0060] In some embodiments, the memory within processor 702 and / or encoder / decoder module 708 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., processor 702 or encoder / decoder module 708) is used for one or more of these functions. External memory may be memory 704 and / or storage device 706, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, for example for MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, while 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Codec; also known as H.265 and MPEG-H Part 2), or VVC (Various Video Codec, a new standard developed by the Joint Video Experts Group, JVET).

[0061] Inputs to the components of system 700 can be provided via various input devices as shown in input box 710. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcaster, (ii) component (COMP) input terminals (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Other examples include composite video.

[0062] In various embodiments, the input device of input box 710 has associated respective input processing elements, as known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting the signal band to a band), (ii) down-converting the selected signal, (iii) further band-limiting to a narrower band to select, for example, a signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners that perform various of these functions, including, for example, down-converting received signals to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0063] Furthermore, the USB and / or HDMI terminals may include their respective interface processors for connecting system 700 to other electronic devices across USB and / or HDMI connections. Various aspects of input processing (e.g., Reed-Solomon error correction) can be implemented as needed, for example, within a separate input processing IC or within processor 702. Similarly, if necessary, various aspects of USB or HDMI interface processing can be implemented within a separate interface IC or within processor 702. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 702 and encoder / decoder 708, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on the output device.

[0064] Various components of system 700 can be housed within an integrated housing. Within the integrated housing, various components can be interconnected and data can be transferred between them using suitable connection devices 712, such as internal buses known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards.

[0065] System 700 includes a communication interface 714, which enables communication with other devices via a communication channel 716. The communication interface 714 may include, but is not limited to, a transceiver configured to send and receive data via the communication channel 716. The communication interface 714 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 716 may be implemented, for example, in a wired and / or wireless medium.

[0066] In various embodiments, data is streamed or otherwise provided to system 700 using a wireless network, such as a Wi-Fi network (e.g., IEEE 802.11 (IEEE stands for Institute of Electrical and Electronics Engineers)). In these embodiments, the Wi-Fi signal is received via a communication channel 716 and a communication interface 714 adapted for Wi-Fi communication. The communication channel 716 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to system 700, with the set-top box transmitting data via an HDMI connection to input box 710. Other embodiments use an RF connection to input box 710 to provide streaming data to system 700. As described above, various embodiments provide data in a non-streaming manner. Furthermore, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0067] System 700 can provide output signals to various output devices, including a display 718, a speaker 720, and other peripheral devices 722. The display 718 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a flexible display, and / or a foldable display. The display 718 can be used in a television, tablet computer, laptop computer, cellular phone (mobile phone), immersive headset or head-mounted display, or other device. The display 718 can also be integrated into other components, such as a smartphone, or can be standalone, such as an external monitor for a laptop or desktop computer. In various examples of embodiments, other peripheral devices 722 include one or more of a standalone digital video disc (or digital universal disc) (DVR, for both terms), disc player, stereo system, and / or lighting system. Various embodiments use one or more peripheral devices 722 that provide functionality based on the output of system 700. For example, a disc player can perform the function of playing the output of system 700.

[0068] In various examples, signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols capable of enabling device-to-device control with or without user intervention is used to transmit control signals between system 700 and display 718, speaker 720, or other peripheral devices 722. Output devices can be communicatively coupled to system 700 via dedicated connections through their respective interfaces 724, 726, and 728. Alternatively, output devices can be connected to system 700 via communication interface 714 using communication channel 716. Display 718 and speaker 720 can be integrated into a single unit with other components of system 700 in electronic devices such as televisions. In various examples, display interface 724 includes display drivers, such as, for example, a timing controller (TCon) chip.

[0069] For example, when the RF portion of input box 710 is part of a standalone set-top box, display 718 and speaker 720 can alternatively be separated from one or more other components. In various embodiments where display 718 and speaker 720 are external components, output signals can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0070] System 700 may include one or more sensor devices 730. Examples of sensor devices that may be used include one or more GPS sensors, gyroscope sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors can be used to determine information such as user position and orientation. When system 700 is used as a control module for an extended reality display, the user's position and orientation can be used to determine how image data is rendered so that the user perceives the correct portion of a virtual object or scene from the correct perspective. In the case of a head-mounted display device, the device's own position and orientation can be used to determine the user's position and orientation for the purpose of rendering virtual content. In the case of other display devices such as telephones, tablets, computer monitors, or televisions, other inputs can be used to determine the user's position and orientation for the purpose of rendering content. For example, the user can use a touchscreen, keypad or keyboard, trackball, joystick, or other input devices to select and / or adjust the desired viewpoint and / or viewing direction to interact with the video scene. When the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation for the purpose of rendering content can be selected and / or adjusted based on the movement of the display device.

[0071] These embodiments can be implemented using computer software executed by processor 702, or by hardware, or a combination of hardware and software. As a non-limiting example, the embodiments can be implemented using one or more integrated circuits. Memory 704 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as, as a non-limiting example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 702 can be of any type suitable for the technical environment and can include one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0072] A method includes: obtaining a scene description file containing a data structure representing objects of an immersive video system, wherein the data structure includes segmentation extensions that indicate that mesh objects are segmented and include data for each mesh segment; and rendering a scene from the scene description file according to the data structure, such that objects are at least partially rendered in the scene. The data may include geometric data for each mesh segment. The data may include texture data for each mesh segment. The data may be stored in the mesh primitive attributes of the mesh objects. The data structure may also include primitive semantic extensions to indicate that primitives have associated semantic information. The presence of primitive semantic extensions or segmentation extensions in the scene description file may be indicated by one of the extensionRequired and extensionUsed attributes in glTF 2.0. Segmentation extensions can provide information including semantic attributes that associate mesh primitives with mesh segments. Semantic objects may be added at the root level of the mesh objects. Semantic objects may be added at the root level of the mesh primitive objects. The scene description file includes extensions to the mesh objects. The scene description file includes extensions to the mesh primitive objects. The schema of the scene description file may be an MPEG-I scene description file. The data structure may include a synthetic 3D representation of a digital human. Data structures can facilitate appearance generalization and stylization. The scene description file can be a JSON file. The scene description file can also be a glTF file. The method may further include parsing, reading, or extracting data structures from the scene description file, and optionally processing those data structures. The method may further include parsing the semantic array by dividing each string of the semantic array with the character " / " and creating a path for each string. The method may further include eliminating ambiguity for identical pathnames in the semantic array by adding a different index to each identical pathname. The method may further include parsing the segmented array by dividing each string of the segmented array with the character " / " and creating a path for each string. The method may further include eliminating ambiguity for identical pathnames in the segmented array by adding a different index to each identical pathname.

[0073] An apparatus may include one or more processors configured to perform any of the methods described herein. A computer-readable medium includes instructions for causing one or more processors to perform any of the methods described herein. The computer-readable medium may be a non-transitory storage medium. A computer program product may include instructions that, when executed by one or more processors, cause one or more processors to perform any of the methods described herein.

[0074] The signal may include a scene description file for a 3D scene, wherein the scene description data includes data structures representing objects in an immersive video system, wherein the data structures include segmentation extensions that indicate that mesh objects are segmented and include data for each mesh segment. The data may include geometric data for each mesh segment. The data may also include texture data for each mesh segment. The data structure may further include primitive semantic extensions to indicate that primitives have associated semantic information. The scene description file may include extensions to mesh objects. The scene description file may include extensions to mesh primitive objects.

[0075] The decoding method according to some embodiments includes obtaining a scene description file, which includes at least: a mesh object; an array of mesh primitive objects associated with the mesh object; and a plurality of semantic objects, each semantic object being associated with a corresponding one of the mesh primitive objects. Rendering the scene described by the scene description file includes rendering at least a portion of at least one mesh primitive object based on the corresponding semantic object.

[0076] The encoding method according to some embodiments includes dividing a mesh in a scene into multiple mesh segments; and encoding a scene description file, the scene description file including at least: mesh objects associated with the mesh; an array of mesh primitive objects associated with the corresponding mesh segments; and multiple semantic objects, each semantic object being associated with a corresponding one of the mesh primitive objects.

[0077] In some embodiments, multiple semantic objects are included in a grid object as an array of semantic objects, and the index of each semantic object corresponds to the index of the associated grid primitive object.

[0078] In some embodiments, multiple semantic objects are included in a grid object as an array of semantic objects, and each semantic object includes information identifying the index of the corresponding associated grid primitive object.

[0079] In some embodiments, each semantic object is included in a corresponding associated mesh primitive object.

[0080] In some embodiments, each semantic object includes information identifying a set of vertices in a mesh primitive object associated with the corresponding semantic object.

[0081] In some embodiments, each semantic object includes a string describing the corresponding mesh primitive object.

[0082] In some embodiments, multiple semantic objects define the hierarchy of graph segments.

[0083] In some embodiments, at least one mesh primitive object is associated with two or more semantic objects.

[0084] According to some embodiments, the signal includes a scene description file, which includes at least: a mesh object associated with a mesh; an array of mesh primitive objects associated with various segments of the mesh; and a plurality of semantic objects, each semantic object being associated with a corresponding one of the mesh primitive objects.

[0085] This disclosure describes various aspects, including tools, features, embodiments, models, methods, and so on. Many of these features are described in detail and are generally described in a manner that may appear restrictive, at least for the purpose of illustrating each aspect. However, this particularity is for the purpose of clarity and does not limit the disclosure or scope of these aspects. Different aspects may be combined and interchanged to provide more aspects. Furthermore, these aspects may also be combined and interchanged with aspects described in earlier applications.

[0086] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are specifically shown, other embodiments are also contemplated, and the discussion of particular embodiments does not limit the breadth of implementation. At least one aspect generally relates to the encoding, decoding, and rendering of scene description information, and at least one other aspect generally relates to transmitting the generated or encoded file and / or bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions thereon stored on them for encoding, decoding, or rendering scene description data according to any of the described methods, and / or computer-readable storage media having bitstreams thereon stored on them generated according to any of the described methods.

[0087] This document describes various methods, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Furthermore, terms such as "first," "second," etc., may be used in various embodiments to modify elements, components, steps, operations, etc., such as "first decoding" and "second decoding," for example. Unless specifically required, the use of these terms does not imply an ordering of the modified operations. In this example, the first decoding does not need to be performed before the second decoding and can occur, for example, before, during, or in a time period overlapping with the second decoding.

[0088] For example, various numerical values ​​may be used in this disclosure. Specific values ​​are for illustrative purposes, and the aspects described are not limited to these specific values.

[0089] The embodiments described herein can be implemented by computer software implemented by a processor or other hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented in one or more integrated circuits. As a non-limiting example, the processor can be of any type suitable for the technical environment and can include one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0090] When a diagram is presented as a flowchart, it also provides a block diagram of the corresponding apparatus. Similarly, when a diagram is presented as a block diagram, it also provides a flowchart of the corresponding method / process.

[0091] The implementations and aspects described herein can be implemented, for example, in methods or processes, apparatuses, software programs, data flows, or signals. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), the implementation of the discussed features can be implemented in other forms (e.g., apparatuses or programs). Apparatuses can be implemented, for example, in suitable hardware, software, and firmware. Methods can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0092] References to “an embodiment” or “an embodiment” or “an implementation” or “implementation”, and other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the appearance of the phrases “in an embodiment” or “in an embodiment” or “in an implementation” or “in an implementation”, and any other variations appearing throughout this disclosure, do not necessarily refer to the same embodiment.

[0093] Furthermore, this disclosure may relate to "determining" various pieces of information. Determining information may include, for example, one or more of estimated information, calculated information, predicted information, or information retrieved from memory.

[0094] Furthermore, this disclosure may relate to “accessing” various information pieces. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or one or more of these.

[0095] Furthermore, this disclosure may relate to "receiving" various pieces of information. Like "access," receiving is intended to be a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) one or more. Moreover, "receiving" is generally referred to in one way or another during operations such as, for example, storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0096] Any use of " / ", "and / or", and "at least one" (e.g., in the cases of "A / B", "A and / or B", and "at least one of A and B") is intended to include selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended to as many items as listed.

[0097] As used herein, the word “signaling” specifically refers to instructing the corresponding decoder to do something. For example, in some embodiments, encoder signaling is used to select a specific one of several parameters for region-based filter parameter selection in artifact removal filtering. Thus, in embodiments, the same parameter is used at both the encoder and decoder sides. Therefore, for example, the encoder can send a specific parameter to the decoder (explicit signaling) so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter and others, signaling can be used without sending them (implicit signaling) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various examples by avoiding the transmission of any actual functionality. Signaling can be accomplished in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. Although the verb form of the word “signaling” has been referred to above, the word “signaling” can also be used as a noun herein.

[0098] The implementation can generate various signals that are formatted to carry information, such as information that can be stored or transmitted. The information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, the signal may be formatted to carry a bitstream of the described embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via various wired or wireless links. The signal may be stored on a processor-readable medium.

[0099] Numerous embodiments are described. Features of these embodiments may be provided individually or in any combination across various claim classes and types. Furthermore, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination across various claim classes and types: • a bitstream or signal comprising one or more described syntax elements or variations thereof; • a bitstream or signal comprising a syntax conveying information generated according to any described embodiment; • creating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more described syntax elements or variations thereof; • creating and / or transmitting and / or receiving and / or decoding according to any described embodiment; • a method, process, apparatus, medium having stored instructions, medium having stored data, computer program product comprising instructions, or signal according to any described embodiment. Various hardware elements of one or more described embodiments are referred to as “modules,” the execution of which (i.e., implementation, performance, etc.) includes the various functions described herein in conjunction with the respective module. As used herein, a module may include hardware deemed suitable for a given implementation, such as one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and one or more memory devices. Each described module may also include executable instructions for performing one or more functions described as being performed by the corresponding module, and those instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., or include them, and may be stored in any suitable one or more non-transitory computer-readable media, such as commonly referred to as RAM, ROM, etc.

[0100] Although the features and elements have been described above in specific combinations, each feature or element can be used alone or in any combination with other features and elements. Furthermore, the methods described herein can be implemented in a computer program, software, or firmware contained in a computer-readable medium for execution by a computer or processor. Examples of computer-readable media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROMs and digital versatile discs (DVDs). The processor associated with the software can be used to implement a radio frequency transceiver used in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A decoding method, comprising: Obtain a scene description file, the scene description file comprising at least: a mesh object; an array of mesh primitive objects associated with the mesh object; and a plurality of semantic objects, each semantic object being associated with a corresponding one of the mesh primitive objects; and rendering the scene described by the scene description file, including rendering at least a portion of at least one mesh primitive object based on the corresponding semantic object.

2. A decoding apparatus comprising one or more processors configured to at least perform: obtaining a scene description file, the scene description file comprising at least: Grid object; An array of mesh primitive objects associated with a mesh object; And multiple semantic objects, each semantic object is associated with a corresponding one in the mesh primitive object; And rendering a scene described by a scene description file, including rendering at least a portion of at least one mesh primitive object based on the corresponding semantic object.

3. An encoding method, comprising: Divide the mesh in the scene into multiple mesh segments; The scene description file is encoded, and the scene description file includes at least: a grid object associated with a grid; an array of grid primitive objects associated with corresponding grid segments; and a plurality of semantic objects, each semantic object being associated with a corresponding one of the grid primitive objects.

4. An encoding apparatus comprising one or more processors configured to at least perform: dividing a mesh in a scene into multiple mesh segments; The scene description file is encoded, and the scene description file includes at least: a grid object associated with a grid; an array of grid primitive objects associated with corresponding grid segments; and a plurality of semantic objects, each semantic object being associated with a corresponding one of the grid primitive objects.

5. The method according to claim 1 or 3, or the apparatus according to claim 2 or 4, wherein, Multiple semantic objects are included in the grid object as an array of semantic objects, and the index of each semantic object corresponds to the index of the associated grid primitive object.

6. The method according to claim 1 or 3, or the apparatus according to claim 2 or 4, wherein, Multiple semantic objects are included in the grid object as an array of semantic objects, and each semantic object includes information identifying the index of the corresponding associated grid primitive object.

7. The method according to claim 1 or 3, or the apparatus according to claim 2 or 4, wherein, Each semantic object is included in its corresponding associated mesh primitive object.

8. The method according to claim 1 or 3, or the apparatus according to claim 2 or 4, wherein, Each semantic object is included in a corresponding associated mesh primitive object, and each semantic object includes information identifying a set of vertices in the mesh primitive object associated with the corresponding semantic object.

9. The method according to claim 1 or 3, or claims 5-8 as dependent on claim 1 or 3, or the apparatus according to claim 2 or 4, or claims 5-8 as dependent on claim 2 or 4, wherein, Each semantic object includes a string describing the corresponding mesh primitive object.

10. The method according to claim 1 or 3, or claims 5-9 as dependent on claim 1 or 3, or the apparatus according to claim 2 or 4, or claims 5-9 as dependent on claim 2 or 4, wherein, Multiple semantic objects define the hierarchy of graph segments.

11. The method according to claim 1 or 3, or claims 5-10 as dependent on claim 1 or 3, or the apparatus according to claim 2 or 4, or claims 5-10 as dependent on claim 2 or 4, wherein, At least one mesh primitive object is associated with two or more semantic objects.

12. The method according to claim 1 or 3, or claims 5-11 as dependent on claim 1 or 3, or the apparatus according to claim 2 or 4, or claims 5-11 as dependent on claim 2 or 4, wherein, The architecture of the scene description file is the MPEG-I scene description file.

13. The method according to claim 1 or 3, or claims 5-12 as dependent on claim 1 or 3, or the apparatus according to claim 2 or 4, or claims 5-12 as dependent on claim 2 or 4, wherein, The scene description file is a JSON file.

14. The method according to claim 1 or 3, or claims 5-13 as dependent on claim 1 or 3, or the apparatus according to claim 2 or 4, or claims 5-13 as dependent on claim 2 or 4, wherein, The scene description file is a glTF file.

15. A signal including a scene description file, said scene description file including at least: A grid object associated with a grid; an array of grid primitive objects associated with the corresponding segments of the grid; And multiple semantic objects, each semantic object is associated with a corresponding one in the mesh primitive object.

Images