Method, device and computer program for decoding geometric patches of 3D meshes

Abstract

The present disclosure relates to a method and system for encoding or decoding a 3D mesh with a constrained geometry dynamic range. An exemplary decoding method includes receiving a coded bitstream including a geometry patch of a three-dimensional mesh, extracting from the coded bitstream a first syntax element indicating whether the geometry patch is partitioned, the geometry patch including one or more partitions, and obtaining, for each partition in the geometry patch, a dynamic range of pixel values ​​for points in each partition that correspond to vertices in the three-dimensional mesh, the dynamic range allowing the geometry patch to be coded within a predetermined bit depth.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 316,876, filed March 4, 2022, which is incorporated by reference in its entirety. This application is also based on and claims the benefit of priority to U.S. Provisional Patent Application No. 17 / 973,824, filed October 26, 2022, which is incorporated by reference in its entirety.

[0002] The present disclosure relates generally to mesh encoding (or compression) and mesh decoding (or decompression) processes, and more particularly to a method and system for mesh compression with constrained geometry dynamic range. [Background technology]

[0003] The discussion of the background art provided herein is intended to generally present the context of the present disclosure. The inventors' work is not admitted, expressly or impliedly, as prior art to the present disclosure to the extent that that work is described in this background section, together with aspects of the description that may not otherwise be admitted as prior art at the time of filing of this application.

[0004] Various techniques have been developed to capture, represent, and simulate real-world objects, environments, and the like in three-dimensional (3D) space. A 3D representation of the world may enable more immersive interactive communication. Exemplary 3D representations of objects and environments include, but are not limited to, point clouds and meshes. A series of 3D representations of objects and environments may form a video sequence. To compress and code such video sequences into a more compact digital format, redundancy and correlation within a sequence of 3D representations of objects and environments may be exploited. Summary of the Invention [Problem to be solved by the invention]

[0005] This disclosure relates generally to coding (compressing) and decoding (decompressing) 3D meshes, and more particularly to compressing meshes with constrained geometry and dynamic range. [Means for solving the problem]

[0006] This disclosure describes a method for decoding a geometry patch of a three-dimensional mesh. The method includes receiving, by a device, a coded bitstream including the geometry patch of the three-dimensional mesh. The device includes a memory for storing instructions and a processor in communication with the memory. The method further includes extracting, by the device, from the coded bitstream, a first syntax element indicating whether the geometry patch is partitioned, the geometry patch including one or more partitions, and for each partition in the geometry patch, obtaining, by the device, a dynamic range of pixel values ​​for points in each partition that correspond to vertices in the three-dimensional mesh, the dynamic range allowing the geometry patch to be coded within a predetermined bit depth.

[0007] According to another aspect, an embodiment of the present disclosure provides an apparatus for encoding or decoding a 3D mesh, the apparatus including a memory storing instructions and a processor in communication with the memory, the processor being configured, when executing the instructions, to cause the apparatus to perform the method described above.

[0008] In another aspect, an embodiment of the present disclosure provides a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform the above-described method for video decoding and / or encoding.

[0009] These and other aspects and their implementations are described in further detail in the drawings, specification, and claims.

[0010] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Diagram 2] FIG. 1 is a schematic diagram of a simplified block diagram of a streaming system according to one embodiment. [Diagram 3] 1 is a block diagram of an encoder for encoding a mesh frame according to some embodiments. [Figure 4] 1 is a block diagram of a decoder for decoding a compressed bitstream corresponding to a mesh frame according to some embodiments. [Diagram 5] FIG. 2 is a schematic diagram of a simplified block diagram of a video decoder according to one embodiment. [Figure 6] FIG. 2 is a schematic diagram of a simplified block diagram of a video encoder according to one embodiment. [Figure 7] 1 is a block diagram of an encoder for encoding a mesh frame according to some embodiments. [Figure 8] 1 shows a diagram of a framework for mesh compression according to some embodiments of the present disclosure. [Figure 9] 1 is a flowchart outlining an example process in accordance with some embodiments. [Figure 10] 1 shows a diagram of a framework for mesh compression according to some embodiments of the present disclosure. [Figure 11] FIG. 13 is another diagram of a framework for mesh compression according to some embodiments of the present disclosure. [Figure 12] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Throughout this specification and the claims, terms may have subtle meanings that are suggested or implied in the context beyond the meaning explicitly stated. The phrases "in one embodiment" or "in some embodiments" used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in other embodiments" used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" used herein do not necessarily refer to the same implementation, and the phrases "in another implementation" or "in other implementations" used herein do not necessarily refer to different implementations. For example, the claimed subject matter is intended to include all or part of the exemplary embodiments / implementations.

[0013] Generally, technical terms may be understood at least in part from their usage in context. For example, terms such as "and", "or", or "and / or" as used herein may include various meanings that may depend at least in part on the context in which such terms are used. Typically, "or" when used to relate a list such as A, B, or C is intended to mean A, B, and C, which are used herein in an inclusive sense, as well as A, B, or C, which are used herein in an exclusive sense. In addition, the terms "one or more" or "at least one" as used herein may be used to describe any feature, structure, or characteristic in a singular sense, or may be used to describe a combination of features, structures, or characteristics in a plural sense, depending at least in part on the context. Similarly, terms such as "a", "an", or "the" may also be understood to convey a singular usage or a plural usage, depending at least in part on the context. In addition, the terms "based on" or "determined by" may be understood as not necessarily intended to convey an exclusive set of factors, but instead may allow for the existence of additional factors not necessarily expressly recited, also depending at least in part on the context.

[0014] Technological developments in 3D media processing, such as advances in 3D capture, 3D modeling, and 3D rendering, are facilitating the ubiquitous generation of 3D content across several platforms and devices. Such 3D content includes information that can be processed to generate various forms of media that provide immersive viewing / rendering and interactive experiences. Applications of 3D content are abundant and include, but are not limited to, virtual reality, augmented reality, metaverse interaction, gaming, immersive video conferencing, robotics, computer-aided design (CAD), and the like. According to one aspect of the present disclosure, to improve the immersive experience, 3D models are becoming increasingly sophisticated, and the creation and consumption of 3D models demands a significant amount of data resources, such as data storage, data transmission resources, and data processing resources.

[0015] Compared with traditional two-dimensional (2D) content, which is generally represented by a dataset (such as an image) in the form of a 2D pixel array, 3D content including a full resolution pixilation in three dimensions may be very resource intensive and is nevertheless not required in many, if not most, practical applications. In most 3D immersive applications, a less data intensive representation of the 3D content may be used according to some aspects of the present disclosure. For example, most applications may only require topographical information, not volumetric information, of objects in a 3D scene (either a real-world scene captured by a sensor such as a LIDAR device, or an animated 3D scene generated by a software tool). In this way, a more efficient form of dataset may be used to represent 3D objects and 3D scenes. For example, a 3D mesh may be used as a type of 3D model to represent immersive 3D content, such as a 3D object in a 3D scene.

[0016] A mesh (also referred to as a mesh model) of one or more objects may include a collection of vertices. The vertices may connect to each other to form edges. The edges may further connect to form faces. The faces may further form polygons. 3D surfaces of various objects may be decomposed into faces, polygons, and the like. Each vertex, edge, face, polygon, or surface may be associated with various attributes such as color, normal, texture, and the like. The normal of a surface may be referred to as a surface normal, and / or the normal of a vertex may be referred to as a vertex normal. Information about how vertices are connected to become an edge, face, or polygon may be referred to as connectivity information. The same set of vertices can form different faces, surfaces, and polygons, so connectivity information is important to uniquely define the components of a mesh. In general, the location of a vertex in 3D space may be represented by its 3D coordinates. A face may be represented by a set of sequentially connected vertices, each associated with a set of 3D coordinates. Similarly, an edge may be represented by two vertices, each associated with its 3D coordinates. The vertices, edges, and faces may be indexed into a 3D mesh data set.

[0017] A mesh may be defined and described by a collection of one or more such basic element types. However, not all of the above types of elements are required to completely describe a mesh. For example, a mesh may be completely described using only vertices and their connectivity. In another example, a mesh may be completely described using only a list of faces and their common vertices. Thus, meshes may be of a variety of alternative types, described by alternative dataset configurations and formats. Exemplary mesh types include, but are not limited to, face-vertex meshes, winged edge meshes, half-edge meshes, quad edge meshes, corner table meshes, vertex-vertex meshes, etc. Correspondingly, the mesh dataset may be stored with information conforming to alternative file formats, including, but not limited to, file extensions such as .raw, .blend, .fbx, .3ds, .dae, .dng, 3dm, .dsf, .dwg, .obj, .ply, .pmd, .stl, amf, .wrl, .wrz, .x3d, .x3db, .x3dv, .x3dz, .x3dbz, .x3dvz, ​​.c4d, .lwo, .smb, .msh, .mesh, .veg, .z3d, .vtk, .l4d, etc. Attributes of such elements, such as color, normals, textures, etc., may be included in the mesh dataset in various manners.

[0018] In some implementations, the vertices of a mesh may be mapped into a pixelated 2D space, referred to as UV space. Thus, each vertex of a mesh may be mapped to a pixel in UV space. In some implementations, a vertex may be mapped to more than one pixel in UV space, for example, a vertex on a boundary may be mapped to two or three pixels in UV space. Similarly, a face or surface in a mesh may be sampled to multiple 3D points that may or may not be among the recorded vertices in the mesh, and these multiple 3D points may be mapped to pixels in the two-dimensional UV space. As described in more detail below, mapping the vertices in a mesh and the sampled 3D points of faces and surfaces into UV space, followed by data analysis and processing in UV space, may facilitate data storage, compression, and coding of a 3D data set of a mesh or a sequence of meshes. The mapped UV space data set may be referred to as a UV image, or a 2D map, or a 2D image of a mesh.

[0019] According to the mapping of the vertices and sampled surface points in the 3D mesh to the 2D UV space, some pixels may be mapped to the vertices and sampled surface points of the 3D mesh, while other pixels may not be mapped. Each of the mapped pixels in the 2D image of the mesh may be associated with information of the corresponding mapped vertex or surface point in the 3D mesh. Depending on the type of information to include for the pixel in the UV space, various 2D images or maps of the mesh may be constructed. A collection of multiple 2D maps may be used as an alternative and / or combined representation of the mesh.

[0020] For example, the simplest 2D map for a mesh may be constructed as an occupancy map. The occupancy map may indicate pixels in UV space that are mapped to the 3D vertices or sampled surface points of the mesh. The indication of occupancy may be represented by a binary indicator at each 2D pixel, for example, including a binary value "1" indicating mapping or occupancy, and a binary value "0" indicating non-mapping or non-occupancy. In this way, the occupancy map may be constructed as a 2D image. While a standard 2D image includes an array of three channels (RGB, YUV, YCrCb, etc.) with a color depth such as 8 bits, such a 2D occupancy map of a mesh only requires a 1-bit binary channel.

[0021] In another example, a 2D geometry map may be constructed for a mesh: rather than containing a single binary channel, the 2D geometry map becomes a full three-channel image, where the three color channels at each occupied pixel correspond to the three 3D coordinates of the corresponding mapped vertex, or sampled 3D point, in the mesh.

[0022] In some implementations, other 2D maps may be constructed for the mesh. For example, a set of attributes for each of the vertices and sampled 3D points of the mesh may be extracted from the mesh dataset and coded into three color channels of the 2D map image. Such a 2D map may be referred to as an attribute map of the mesh. A particular attribute map may include a three-channel color for each occupied pixel in UV space. In another example, a texture attribute associated with each of the mapped vertices or sampled 3D points of the mesh may be parameterized into a three-channel value and coded into the 2D attribute map. In another example, a normal attribute associated with each of the mapped vertices or sampled 3D points of the mesh may be parameterized into a three-channel value and coded into the 2D attribute map. In some example implementations, multiple 2D attribute maps may be constructed to hold all the necessary attribute information for the vertices and sampled surface points of the mesh.

[0023] The 2D maps described above are merely examples. Other types of 2D maps for a mesh may be constructed. Additionally, other datasets may be extracted from the 3D mesh to jointly represent the original 3D mesh with the 2D maps described above. For example, vertex-to-vertex connection or connectivity information may be grouped and organized separately from the 2D map in the form of a list, table, or the like. For example, the connectivity information may reference vertices using a vertex index. The vertex index may be mapped to its corresponding pixel location in the 2D map. In another example, surface texture, color, normal, displacement, and other information may be extracted and organized separately from the 2D map, rather than as a 2D map. Other metadata may be further extracted from the 3D mesh to represent the 3D mesh with the 2D map and other datasets described above.

[0024] While the above-described exemplary implementations focus on static meshes, according to aspects of the present disclosure, a 3D mesh may be dynamic. A dynamic mesh may refer to a mesh in which at least one of its components (geometry information, connectivity information, mapping information, vertex attributes, and attribute maps) changes over time, for example. In this manner, a dynamic mesh may be described by a sequence of meshes or meshes (also called mesh frames), similar to a timed sequence of 2D image frames that form a video.

[0025] In some example implementations, the dynamic mesh may have constant connectivity information, time-varying geometry, and time-varying vertex attributes. In some other examples, the dynamic mesh may have time-varying connectivity information. In some examples, digital 3D content creation tools may be used to generate the dynamic mesh with time-varying attribute maps and time-varying connectivity information. In some other examples, volumetric acquisition / detection / sensing techniques are used to generate the dynamic mesh. Volumetric acquisition techniques can generate dynamic meshes with time-varying connectivity information, especially under real-time constraints.

[0026] Dynamic meshes may require large amounts of data, as they may contain a significant amount of information that changes over time. However, to exploit redundancy, compression may be performed within mesh frames (intra-compression) and between mesh frames (inter-compression). Various mesh compression processes may be implemented to allow efficient storage and transmission of media content within mesh representations, particularly for mesh sequences.

[0027] Aspects of the present disclosure provide example architectures and techniques for mesh compression. These techniques may be used for various mesh compressions, such as, but not limited to, static mesh compression, dynamic mesh compression, compression of dynamic meshes with constant connectivity information, compression of dynamic meshes with time-varying connectivity information, compression of dynamic meshes with time-varying attribute maps, etc. These techniques may be used for lossy and lossless compression for various applications, such as real-time immersive communication, storage, free viewpoint video, augmented reality (AR), virtual reality (VR), etc. Applications may include features such as random access and scalable / progressive coding.

[0028] Although this disclosure explicitly describes techniques and implementations applicable to 3D meshes, the principles underlying the various implementations described herein are applicable to other types of 3D data structures, including, but not limited to, point cloud (PC) data structures. For simplicity, references below to 3D meshes are general and intended to include other types of 3D representations, such as point clouds and other 3D volumetric datasets.

[0029] Turning first to an exemplary architecture-level implementation, FIG. 1 illustrates a simplified block diagram of a communication system (100) according to an exemplary embodiment of the present disclosure. The communication system (100) may include multiple terminal devices that can communicate with each other, such as via a communication network (150) (also referred to as a network). For example, the communication system (100) may include a pair of terminal devices (110) and (120) interconnected via the network (150). In the example of FIG. 1, a first pair of terminal devices (110) and (120) may perform a unidirectional transmission of a 3D mesh. For example, the terminal device (110) may compress a 3D mesh or a sequence of 3D meshes, which may be generated by the terminal device (110) or retrieved from a storage device or captured by a 3D sensor (105) connected to the terminal device (110). The compressed 3D mesh or sequence of 3D meshes may be transmitted, for example in the form of a bitstream (also called a coded bitstream), via the network (150) to other terminal devices (120). The terminal devices (120) may receive the compressed 3D mesh or sequence of 3D meshes from the network (150), decompress the bitstream to reconstruct the original 3D mesh or sequence of 3D meshes, and process the reconstructed 3D mesh or sequence of 3D meshes appropriately for display or other purposes / applications. Unidirectional data transmission may be common, such as in media serving applications.

[0030] In the example of FIG. 1, any one or both of the terminal devices (110) and (120) may be implemented as a server, a stationary or mobile personal computer, a laptop computer, a tablet computer, a smart phone, a gaming console, a media player, and / or a dedicated three-dimensional (3D) appliance, etc., although the principles of the present disclosure may not be so limited. The network (150) may represent any type of network or combination of networks that transmits a compressed 3D mesh between the terminal device (110) and the terminal device (120). The network (150) may include, for example, a wireline and / or wireless communication network. The network (150) may exchange data over circuit-switched and / or packet-switched channels. Representative networks include long-haul telecommunications networks, local area networks, wide area networks, cellular networks, and / or the Internet. For purposes of the present disclosure, the architecture and topology of the network (150) may not be important to the operation of the present disclosure, unless otherwise described herein below.

[0031] 2 illustrates an exemplary simplified block diagram of a streaming system (200) according to one embodiment of the present disclosure. FIG. 2 illustrates an exemplary application of the disclosed implementations with respect to 3D meshes and compressed 3D meshes. The disclosed subject matter may be equally applicable to other 3D mesh or point cloud enabled applications, such as 3D telepresence applications, virtual reality applications, etc.

[0032] The streaming system (200) may include a capture or storage subsystem (213). The capture or storage subsystem (213) may include a 3D mesh generator or storage medium (201), such as a 3D mesh or point cloud generation tool / software, a graphics generation component, or a point cloud sensor such as a light detection and ranging (LIDAR) system, a 3D camera, a 3D scanner, a 3D mesh storage device, etc., that generates or provides an uncompressed 3D mesh (202) or point cloud (202). In some example implementations, the 3D mesh (202) includes vertices of a 3D mesh, or 3D points of a point cloud (both referred to as a 3D mesh). The 3D mesh (202) is shown with a thicker line to highlight the corresponding larger amount of data compared to the compressed 3D mesh (204) (compressed 3D mesh bitstream). The compressed 3D mesh (204) may be generated by an electronic device (220) that includes an encoder (203) coupled to the 3D mesh (202). The encoder (203) may include hardware, software, or a combination thereof for enabling or implementing aspects of the disclosed subject matter, as described in more detail below. The compressed 3D mesh (204) (or a bitstream of the compressed 3D mesh (204)), shown with thin lines to emphasize the smaller amount of data compared to the stream of the uncompressed 3D mesh (202), may be stored on the streaming server (205) for future use. One or more streaming client subsystems, such as the client subsystems (206) and (208) of FIG. 2, may access the streaming server (205) to retrieve copies (207) and (209) of the compressed 3D mesh (204). The client subsystem (206) may include a decoder (210), for example, in the electronic device (230). The decoder (210) may be configured to decode an input copy (207) of the compressed 3D mesh to create an output stream of a reconstructed 3D mesh (211) that can be rendered on a rendering device (212) or for other uses.

[0033] It should be noted that the electronic devices (220) and (230) may include other components (not shown). For example, the electronic device (220) may include a decoder (not shown), and the electronic device (230) may also include an encoder (not shown).

[0034] In some streaming systems, the compressed 3D meshes (204), (207), and (209) (e.g., a bitstream of the compressed 3D meshes) may be compressed according to some standards. In some examples, as described in more detail below, video coding standards are used to exploit redundancy and correlation in compressing the 3D meshes after they are first projected and mapped to a 2D representation suitable for video compression. Non-limiting examples of such standards include High Efficiency Video Coding (HEVC), Universal Video Coding (VVC), etc., as described in more detail below.

[0035] A compressed 3D mesh or sequence of 3D meshes may be generated by an encoder, while a decoder may be configured to decompress the compressed or coded 3D meshes. FIG. 3 illustrates a high-level example data flow of a 3D mesh in such an encoder (301) and decoder (303). As shown in FIG. 3, a raw input 3D mesh or sequence of 3D meshes (302) may be pre-processed by track remeshing, parameterization, and / or voxelization to generate input data to a mapping unit for mapping the 3D mesh to a 2D UV space (304), which may include a mesh with a UV atlas in some implementations. The 3D mesh may be sampled to include 3D surface points that may not be among the vertices, and these sampled 3D surface points may be added to the mapping to the UV space. Various 2D maps may be generated in the encoder 301, including, but not limited to, an occupancy map (310), a geometry map (312), and an attribute map (314). Such image types of maps may be compressed by the encoder 301, for example, using video coding / compression techniques. For example, the video coder may use intra prediction techniques and inter prediction with other 3D mesh reference frames to aid in the compression of the 3D mesh frames. Other non-image or non-map data, or metadata (316), may also be coded in various manners to remove redundancy and generate compressed non-map data, in a non-limiting example, via entropy coding. The encoder 301 may then combine or multiplex the compressed 2D map and non-map data to further code the combined data to generate an encoded bitstream (also referred to as a coded bitstream). The encoded bitstream may then be stored or transmitted for use by the decoder 303.The decoder may be configured to decode the bitstream, demultiplex the decoded bitstream to obtain the compressed 2D maps and non-map data, and perform decompression to generate a decoded occupancy map (320), a decoded geometry map (322), a decoded attribute map (324), and decoded non-map data and metadata (326). The decoder 303 may then be further configured to reconstruct a 3D mesh or sequence of 3D meshes (330) from the decoded 2D maps (320, 322, and 324) and the decoded non-map data (326).

[0036] More specifically, Figure 4 illustrates a block diagram of an exemplary 3D mesh encoder (400) for encoding a 3D mesh frame, in accordance with some embodiments of the present disclosure. In some example implementations, the mesh encoder (400) may be used in the communication system (100) and the streaming system (200). For example, the encoder (203) may be configured and operate similarly to the mesh encoder (400).

[0037] The mesh encoder (400) may receive a 3D mesh frame as an uncompressed input and generate a bitstream corresponding to a compressed 3D mesh. In some example implementations, the mesh encoder (400) may receive the 3D mesh frame from any source, such as the mesh or point cloud source (201) of FIG.

[0038] In the example of FIG. 4, the mesh encoder (400) may include a patch generation module (406) (alternatively referred to as a chart generation module), a patch packing module (408), a geometry image generation module (410), a texture image generation module (412), a patch information module (404), an occupancy map module (414), a smoothing module (436), image padding modules (416) and (418), a group expansion module (420), video compression modules (422), (423), and (432), an auxiliary patch information compression module (438), an entropy compression module (434), and a multiplexer (424).

[0039] In various embodiments of the present disclosure, a module may refer to a software module, a hardware module, or a combination thereof. A software module may include a computer program or a portion of a computer program that has a certain function, such as such functions described in the present disclosure, and works together with other related parts to achieve a certain purpose. A hardware module may be implemented using a processing circuit and / or memory configured to perform the functions described in the present disclosure. Each module may be implemented using one or more processors (or a processor and memory). Similarly, a processor (or a processor and memory) may be used to implement one or more modules. Furthermore, each module may be part of an entire module that includes the functionality of the module. The description herein may apply to the term module, as well as other equivalent terms (e.g., unit).

[0040] According to one aspect of the disclosure, and as described above, the mesh encoder (400) converts the 3D mesh frame into an image-based representation (e.g., a 2D map) along with some non-mapped metadata (e.g., patch or chart information) that is used to aid in converting the compressed 3D mesh back to a decompressed 3D mesh. In some examples, the mesh encoder (400) converts the 3D mesh frame into a 2D geometry map or image, a texture map or image, and an occupancy map or image, and then may encode the geometry image, texture image, and occupancy map along with the metadata and other compressed non-mapped data into a bitstream using video coding techniques. Generally, and as described above, a 2D geometry image is a 2D image that includes 2D pixels filled with geometry values ​​associated with 3D points projected (the term "projected" is used in the sense of "mapped") onto the 2D pixels, where the 2D pixels filled with geometry values ​​may be referred to as geometry samples. A texture image is a 2D image that includes pixels filled with texture values ​​associated with 3D points projected onto the 2D pixels, where the 2D pixels filled with texture values ​​may be referred to as texture samples. An occupancy map is a 2D image that contains 2D pixels filled with values ​​indicating occupancy or non-occupancy by 3D points.

[0041] The patch generation module (406) segments the 3D mesh into a set of charts or patches (e.g., a patch is defined as a contiguous subset of the surface described by the 3D mesh or point cloud), which may or may not overlap, so that each patch may be described by a depth field relative to a plane in 2D space (e.g., the deeper a 3D point on the surface, the further away it is from the center of the corresponding 2D map, making the surface flatter). In some embodiments, the patch generation module (406) aims to decompose the 3D mesh into a minimum number of patches with smooth boundaries while minimizing the reconstruction error.

[0042] The patch information module (404) may collect patch information indicating the size and shape of the patch. In some examples, the patch information may be packed into a data frame and then encoded by the auxiliary patch information compression module (438) to generate compressed auxiliary patch information. The auxiliary patch compression may be implemented in various forms, including, but not limited to, various types of arithmetic coding.

[0043] The patch or chart packing module (408) may be configured to map the extracted patches to a 2D grid in UV space while minimizing unused space. In some example implementations, pixels in the 2D UV space may be granulated into blocks of pixels for mapping of patches or charts. The block size may be predefined. For example, the block size may be M×M (e.g., 16×16). Such granularity may ensure that every M×M block of the 2D UV grid is associated with a unique patch. In other words, each patch is mapped to the 2D UV space with a 2D granularity of M×M. Efficient patch packing can directly affect compression efficiency by either minimizing unused space or ensuring temporal consistency. Example implementations of packing patches or charts into 2D UV space are shown in more detail below.

[0044] The geometry image generation module (410) can generate a 2D geometry image associated with the geometry of the 3D mesh at a given patch location in the 2D grid. The texture image generation module (412) can generate a 2D texture image associated with the texture of the 3D mesh at a given patch location in the 2D grid. As described above, the geometry image generation module (410) and the texture image generation module (412) essentially utilize the 3D to 2D mapping calculated during the packing process described above to store the geometry and texture of the 3D mesh as a 2D image.

[0045] In some implementations, the 2D images may be layered to better handle cases where multiple points are projected onto the same sample (e.g., overlapping patches in the 3D space of the mesh). In other words, each patch may be projected onto, for example, two images, called layers, such that multiple points may be projected onto the same point in different layers.

[0046] In some example implementations, the geometry image may be represented by a monochromatic frame of width x height (W x H). Thus, three geometry images of three luma or chroma channels may be used to represent the 3D coordinates. In some example implementations, the geometry image may be represented by a 2D image with three channels (RGB, YUV, YCrCb, etc.) with a particular color depth (e.g., 8 bits, 12 bits, 16 bits, etc.). Thus, one geometry image with three color channels may be used to represent the 3D coordinates.

[0047] To generate the texture image, the texture generation procedure utilizes the reconstructed / smoothed geometry to compute colors associated with points sampled from the original 3D mesh (see "Sampling" in Figure 3, this generates, for example, 3D surface points that are not among the vertices of the original 3D mesh).

[0048] The occupancy map module (414) may be configured to generate an occupancy map that describes the padding information in each unit. For example, as described above, the occupancy image may include a binary map that indicates, for each cell of a 2D grid, whether the cell belongs to free space or to a 3D mesh. In some example implementations, the occupancy map may use binary information to describe, for each pixel, whether the pixel is padded or not. In some other example implementations, the occupancy map may use binary information to describe, for each block of pixels (e.g., each M×M block), whether the block of pixels is padded or not.

[0049] The occupancy map generated by the occupancy map module (414) may be compressed using lossless or lossy coding. When lossless coding is used, an entropy compression module (434) may be used to compress the occupancy map. When lossy coding is used, a video compression module (432) may be used to compress the occupancy map.

[0050] It should be noted that the patch packing module (408) may leave some empty space between the 2D patches packed into the image frame. The image padding modules (416) and (418) may fill the empty space (referred to as padding) to generate an image frame that may be suitable for 2D video and image codecs. Image padding, also referred to as background filling, may fill the unused space with redundant information. In some instances, well-performed background filling provides a minimal increase in bitrate while avoiding significant coding artifacts around patch boundaries.

[0051] The video compression modules (422), (423), and (432) can encode 2D images, such as padded geometry images, padded texture images, and occupancy maps, based on an appropriate video coding standard, such as HEVC, VVC, etc. In some example implementations, the video compression modules (422), (423), and (432) are individual components that operate separately. It should be noted that the video compression modules (422), (423), and (432) may be implemented as a single component in some other example implementations.

[0052] In some example implementations, the smoothing module (436) may be configured to generate a smoothed image of the reconstructed geometry image. The smoothed image may be provided to the texture image generation (412). The texture image generation (412) may then adjust the generation of the texture image based on the reconstructed geometry image. For example, when a patch shape (e.g., geometry) is slightly distorted during encoding and decoding, the distortion may be taken into account when generating a texture image to correct for the distortion in the patch shape.

[0053] In some embodiments, the group expansion (420) is configured to pad pixels around object boundaries with redundant low-frequency content to improve the coding gain as well as the visual quality of the reconstructed 3D mesh.

[0054] The multiplexer (424) may be configured to multiplex the compressed geometry images, the compressed texture images, the compressed occupancy maps, and the compressed auxiliary patch information into a compressed bitstream (also referred to as a coded bitstream).

[0055] 5 illustrates a block diagram of an exemplary mesh decoder (500) for decoding a compressed bitstream corresponding to a 3D mesh frame according to some embodiments of the present disclosure. In some example implementations, the mesh decoder (500) may be used in a communication system (100) and a streaming system (200). For example, the decoder (210) may be configured to operate similarly to the mesh decoder (500). The mesh decoder (500) receives a compressed bitstream and generates a reconstructed 3D mesh based on the compressed bitstream, including compressed geometry images, compressed texture images, compressed occupancy maps, compressed auxiliary patch information, etc.

[0056] In the example of FIG. 5, the mesh decoder (500) may include a demultiplexer (532), video decompression modules (534) and (536), an occupancy map decompression module (538), an auxiliary patch information decompression module (542), a geometry reconstruction module (544), a smoothing module (546), a texture reconstruction module (548), and a color smoothing module (552).

[0057] A demultiplexer (532) may receive the compressed bitstream and separate it into compressed texture images, compressed geometry images, compressed occupancy maps, and compressed auxiliary patch information.

[0058] The video decompression modules (534) and (536) can decode compressed images according to an appropriate standard (e.g., HEVC, VVC, etc.) and output decompressed images. For example, the video decompression module (534) may decode compressed texture images and output decompressed texture images. The video decompression module (536) may also decode compressed geometry images and output decompressed geometry images.

[0059] The occupancy map decompression module (538) may be configured to decode the compressed occupancy map according to an appropriate standard (e.g., HEVC, VVC, etc.) and output a decompressed occupancy map.

[0060] The auxiliary patch information decompression module (542) may be configured to decode the compressed auxiliary patch information according to a suitable decoding algorithm and output the decompressed auxiliary patch information.

[0061] A geometry reconstruction module (544) may be configured to receive the decompressed geometry image and generate a reconstructed 3D mesh geometry based on the decompressed occupancy map and the decompressed auxiliary patch information.

[0062] The smoothing module (546) may be configured to smooth discrepancies at the edges of the patches. The smoothing procedure may be aimed at mitigating potential discontinuities that may occur at patch boundaries due to compression artifacts. In some example implementations, a smoothing filter may be applied to pixels located on patch boundaries to reduce distortions that may be caused by compression / decompression.

[0063] A texture reconstruction module (548) may be configured to determine texture information for points in the 3D mesh based on the decompressed texture image and the smoothed geometry.

[0064] A color smoothing module (552) may be configured to smooth out coloring discrepancies. Non-neighboring patches in 3D space are often packed adjacent to each other in 2D video. In some examples, pixel values ​​from non-neighboring patches may be blended by a block-based video codec. A goal of color smoothing may be to reduce visible artifacts that appear at patch boundaries.

[0065] 6 shows a block diagram of an exemplary video decoder (610) according to one embodiment of the present disclosure. The video decoder (610) may be used in the mesh decoder (500). For example, the video decompression modules (534) and (536) and the occupancy map decompression module (538) may be configured similarly to the video decoder (610).

[0066] The video decoder (610) may include a parser (620) for reconstructing symbols (621) from a compressed image, such as a coded video sequence. These symbol categories may include information used to manage the operation of the video decoder (610). The parser (620) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles including variable length coding with or without context dependency, Huffman coding, arithmetic coding, etc. The parser (620) may extract a set of subgroup parameters for at least one subgroup of pixels in the video decoder based on at least one parameter corresponding to the group from the coded video sequence. The subgroup may include a group of pictures (GOP), image, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The parser (620) may also extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0067] The parser (620) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory to produce symbols (621).

[0068] The reconstruction of the symbols (621) may involve a number of different units, depending on the type of coded video picture or portion thereof (inter-picture and intra-picture, inter-block and intra-block, etc.), as well as other factors. Which units are involved, and how, may be controlled by subgroup control information parsed from the coded video sequence by the parser (620). The flow of such subgroup control information between the parser (620) and the following units is not depicted for clarity.

[0069] Besides the functional blocks already mentioned, the video decoder (610) can be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of these units may closely interact with each other and may be at least partially integrated with each other. The following conceptual subdivision into functional units is made solely for purposes of describing the disclosed subject matter.

[0070] The video decoder (610) may include a scaler / inverse transform unit (651), which may receive quantized transform coefficients as well as control information including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. as symbol(s) (621) from the parser (620). The scaler / inverse transform unit (651) may output blocks including sample values ​​that may be input to an aggregator (655).

[0071] In some cases, the output samples of the scaler / inverse transform (651) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed image may use prediction information from a previously reconstructed portion of the current image. Such prediction information may be provided by an intra-picture prediction unit (652). In some cases, the intra-picture prediction unit (652) may generate a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from a current picture buffer (658). The current picture buffer (658) may, for example, buffer a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (655) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (652) to the output sample information as provided by the scaler / inverse transform unit (651).

[0072] In other cases, the output samples of the scalar / inverse transform unit (651) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion compensated prediction unit (653) may access the reference picture memory (657) to retrieve samples to use for prediction. After motion compensating the fetched samples according to the symbols (621) related to the block, these samples may be added by the aggregator (655) to the output of the scalar / inverse transform unit (651) (in this case referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (657) from which the motion compensated prediction unit (653) fetches prediction samples may be controlled by motion vectors available to the motion compensated prediction unit (653), for example, in the form of symbols (621) that may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (657) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0073] The output samples of the aggregator (655) may be subjected to various loop filtering techniques in a loop filter unit (656). Video compression techniques may include in-loop filter techniques that are controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (656) as symbols (621) from the parser (620), but may also be responsive to previously reconstructed and loop filtered sample values ​​as well as responsive to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence.

[0074] The output of the loop filter unit (656) may be a sample stream that can be output to a rendering device and also stored in a reference picture memory (657) for use in future inter-picture prediction.

[0075] Once fully reconstructed, a particular coded picture may be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (620)), the current picture buffer (658) may become part of the reference picture memory (657) and a new current picture buffer may be relocated before beginning reconstruction of a subsequent coded picture.

[0076] The video decoder (610) may perform decoding operations according to a given video compression technique in a standard such as ITU-T Rec. H.265. The coded video sequence may comply with the syntax specified by the video compression technique or standard being used in the sense that the coded video sequence adheres to both the syntax of the video compression technique or standard and to the profile documented in the video compression technique or standard. Specifically, the profile may select certain tools from among all tools available in the video compression technique or standard as those only available under that profile. Also, what is required for compliance may be that the complexity of the coded video sequence is within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further limited in some cases by the specification of a hypothetical reference decoder (HRD) and metadata for HRD buffer management signaled within the coded video sequence.

[0077] 7 shows a block diagram of a video encoder (703) according to one embodiment of the present disclosure. The video encoder (703) can be used in the mesh encoder (400) to compress a 3D mesh or point cloud. In some example implementations, the video compression modules (422) and (423) and the video compression module (432) are configured similarly to the encoder (703).

[0078] A video encoder (703) may receive 2D images, such as padded geometry images, padded texture images, etc., and generate compressed images.

[0079] According to an exemplary embodiment of the present disclosure, the video encoder (703) can code and compress pictures (images) of a source video sequence into a coded video sequence (compressed images) in real time or under any other time constraint required by the application. Providing an appropriate coding rate is one function of the controller (750). In some embodiments, the controller (750) controls and is functionally coupled to other functional units as described below. For clarity, coupling is not shown. Parameters set by the controller (750) can include rate control related parameters (e.g., picture skip, quantization, lambda values ​​for rate-distortion optimization techniques), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (750) can be configured with other appropriate functions for the video encoder (703) optimized for a particular system design.

[0080] In some example implementations, the video encoder (703) may be configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop may include a source coder (730) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture) and a (local) decoder (733) embedded in the video encoder (703). The decoder (733) may reconstruct the symbols to create sample data in a similar manner as the (remote) decoder would do (since any compression between the symbols and the coded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) may be input to a reference picture memory (734). Since the decoding of the symbol stream results in a bit-exact result regardless of the location of the decoder (local or remote), the contents of the reference picture memory (734) are also bit-exact between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, e.g., due to channel errors) is also used in several related technologies.

[0081] The operation of the "local" decoder (733) may be the same as the operation of a "remote" decoder, such as the video decoder (610), already described in detail above in conjunction with Figure 6. However, with brief reference also to Figure 6, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (745) and parser (620) may be lossless, the entropy decoding portion of the video decoder (610), including the parser (620), may not be fully implemented in the local decoder (733).

[0082] In various embodiments of the present disclosure, any decoder technology other than analysis / entropy decoding present in the decoder may necessarily also need to be present in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter in various embodiments may focus on the operation of the decoder. A description of the encoder technology may be omitted since the encoder technology is the opposite of the decoder technology, which is described generically. Only in certain areas are more detailed descriptions necessary, which are presented below.

[0083] In operation, in some examples, the source coder (730) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (732) codes differences between pixel blocks of the input picture and pixel blocks of reference picture(s) that may be selected as predictive reference(s) for the input picture.

[0084] The local video decoder (733) may decode coded video data of pictures that may be designated as reference pictures based on symbols created by the source coder (730). The operation of the coding engine (732) may advantageously be a lossy process. When the coded video data may be decoded at a video decoder (not shown in FIG. 7), the reconstructed video sequence may be a copy of the source video sequence, usually with some errors. The local video decoder (733) may replicate the decoding process that may be performed by the video decoder for the reference pictures, such that the reconstructed reference pictures are stored in a reference picture cache (734). In this manner, the video encoder (703) may locally store copies of reconstructed reference pictures that have common content with the reconstructed reference pictures obtained by the far-end video decoder (free of transmission errors).

[0085] The predictor (735) may perform the prediction search of the coding engine (732). That is, the predictor (735) may search for sample data (as candidate reference pixel blocks) from the reference picture memory (734) for a new picture to be coded, or some metadata such as reference picture motion vectors, block shapes, etc., which may serve as suitable prediction references for the new picture. The predictor (735) may operate on one sample block per pixel block to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (735), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (734).

[0086] The controller (750) may manage the coding operations of the source coder (730), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0087] The output of all the aforementioned functional units may be entropy coded in an entropy coder (745), which may convert the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, or the like.

[0088] The controller (750) may manage the operation of the video encoder (703). During coding, the controller (750) may assign several coding picture types to each of the coded pictures, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned as one of the following picture types:

[0089] An intra picture (I-picture) may be one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0090] A predictive picture (P picture) may be one that can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0091] A bidirectionally predicted picture (B-picture) may be one that can be coded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multiple predicted picture may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0092] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. A block may be predictively coded with reference to other (already coded) blocks determined by the coding assignment applied to the block's respective picture. For example, a block of an I picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). A pixel block of a P picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. A block of a B picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0093] The video encoder (703) may perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Recommendation H.265. In its operations, the video encoder (703) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0094] Video may be in the form of multiple source pictures (images) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In one example, a particular picture being coded / decoded, called the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a reference picture that has been previously coded and is still buffered in the video, the block in the current picture may be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0095] In some embodiments, bi-prediction techniques may be used for inter-picture prediction. According to bi-prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which are before the decoding order of the current picture in the video (but the display order may be past and future, respectively). A block in the current picture may be coded by a first motion vector pointing to a first reference block in the first reference picture and by a second motion vector pointing to a second reference block in the second reference picture. A block may be predicted by a combination of the first reference block and the second reference block.

[0096] Furthermore, to improve coding efficiency, merge mode techniques may be used for inter-picture prediction.

[0097] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed on a block-by-block basis. For example, according to the HEVC standard, a picture in a sequence of video pictures is divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quad-tree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels can be partitioned into one CU of 64×64 pixels, or into four CUs of 32×32 pixels, or into 16 CUs of 16×16 pixels. In one example, each CU is analyzed to determine the prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) according to temporal predictability and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Taking a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, and 16×8 pixels.

[0098] In various embodiments, the mesh encoder (400) and mesh decoder (500) described above may be implemented in hardware, software, or a combination thereof. For example, the mesh encoder (400) and mesh decoder (500) may be implemented using processing circuitry, such as one or more integrated circuits (ICs) operating with or without software, such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. In another example, the mesh encoder (400) and mesh decoder (500) may be implemented as software or firmware including instructions stored in a non-volatile (or non-transitory) computer-readable storage medium. The instructions, when executed by a processing circuitry, such as one or more processors, cause the processing circuitry to perform the functions of the mesh encoder (400) and / or the mesh decoder (500).

[0099] 8 illustrates an example of a mapping of a 3D patch (810) in a 3D xyz coordinate system (819) to a 2D UV plane (850) in a uv coordinate system (859). In some implementations, a 3D patch (or simply a patch) can generally refer to a contiguous subset of a surface described by a set of vertices that correspond to a mesh in 3D space. In a non-limiting example, a patch includes vertices with 3D coordinates, normal vectors, color, texture, and other information. In some implementations, a 2D geometry patch (or simply a geometry patch) can refer to a projected shape in a 2D UV plane, the projected shape corresponding to the patch, and the geometry patch corresponding to the patch.

[0100] In the projected shape in the UV plane, each of the mapped points (ui,vi) corresponds to a 3D vertex at position (xi,yi,zi), where i=1,2,3,4 etc. For example, a first vertex (811) with 3D coordinates (x1, y1, z1) is mapped to a first point (851) with 2D coordinates (u1, v1), a second vertex (812) with 3D coordinates (x2, y2, z2) is mapped to a second point (852) with 2D coordinates (u2, v2), a third vertex (813) with 3D coordinates (x3, y3, z3) is mapped to a third point (853) with 2D coordinates (u3, v3), and a fourth vertex (814) with 3D coordinates (x4, y4, z4) is mapped to a fourth point (854) with 2D coordinates (u4, v4).

[0101] The coding of the vertex geometry (i.e., the xi, yi, and zi values) is then converted into a coding of three-channel values ​​in the 2D plane, where each three-channel value at a (u,v) location corresponds to an associated 3D location. For example, the pixel value of a first location (851) includes a three-channel value (i.e., three channel values) that correspond to the 3D coordinates of a first vertex (811), where a first channel value of the three-channel value corresponds to an x1 value, a second channel value of the three-channel value corresponds to a y1 value, and a third channel value of the three-channel value corresponds to a z1 value.

[0102] The 2D plane onto which one or more patches are projected / mapped is thus referred to as a geometry image, and therefore may be coded using any image or video codec, such as a video codec that supports the 4:4:4 color format.

[0103] In some implementations, the pixel value of a point projected / mapped onto the UV plane may correspond to the distance from the 3D vertex to the UV plane. Thus, multiple planes with different orientations may be used for such projections to localize the vertex position information. In this case, each position (or called a point) in the UV plane is a one-channel value and records the distance. Such an image is called a depth image. The depth image may be coded using any image or video codec, such as any video codec that supports YUV4:2:0 or YUV4:0:0 color format.

[0104] In some implementations, there may be challenges / problems in using pixel values ​​of points in the uv plane to represent 3D coordinate values ​​of vertices in a 3D mesh. One challenge / problem may be that the dynamic range of one or more components of the 3D coordinate values ​​of a vertex may be larger than the range of pixel values ​​with a particular bit depth. For example, the bit depth of sample values ​​on the 2D UV plane may need to match the dynamic range of the geometric positions of the 3D patch. Such dynamic range may be very large, and thus the bit depth that needs to be supported from the video codec is larger than the maximum supported bit depth for some video codecs. In other words, not all video codecs support coding of high bit depth to match the very large dynamic range of the 3D coordinate values ​​of vertices in a mesh. Another challenge / problem may include that even if a particular video codec supports high bit depth, it is not necessary to assign high bit depth to all patches since not all patches have high dynamic range in the geometric position distribution.

[0105] This disclosure describes various embodiments for encoding or decoding 3D meshes having constrained geometry dynamic range, addressing at least one of the challenges / problems discussed above to improve efficiency in compressing geometry information and / or advance efficient 3D mesh compression techniques.

[0106] 9 shows a flowchart 900 of an exemplary method according to the principles underlying the above implementations for decoding a 3D mesh with a constrained geometry dynamic range. The flow of the exemplary decoding method starts at S901 and may include some or all of the following steps: receiving a coded bitstream including a geometry patch of a three-dimensional mesh at S910, extracting from the coded bitstream at S920 a first syntax element indicating whether the geometry patch is partitioned, the geometry patch including one or more partitions at S930, obtaining a dynamic range of pixel values ​​for points in each partition that correspond to vertices in the three-dimensional mesh, the dynamic range allowing the geometry patch to be coded within a predetermined bit depth. The exemplary method stops at S999.

[0107] In some implementations, another example method for decoding a 3D mesh with a constrained geometry dynamic range may include receiving a coded bitstream; extracting from the coded bitstream a first syntax element indicating whether a geometry patch is partitioned, where the geometry patch includes one or more partitions; for a partition in the geometry patch, obtaining from the coded bitstream extremal pixel values ​​of points in the partition, where the points in the partition correspond to vertices in the three-dimensional mesh; obtaining from the coded bitstream coded pixel values ​​of points in the partition; and determining pixel values ​​for the points in the partition based on the extremal pixel values ​​and the coded pixel values, where the pixel values ​​correspond to three-dimensional coordinates of the vertices in the three-dimensional mesh.

[0108] In some implementations, the coded bitstream includes at least one of a coded geometry map or coded metadata. For example, the coded bitstream may be the compressed bitstream of FIG. 4, which may include one or more compressed geometry images / maps, one or more compressed texture images / maps, one or more compressed occupancy maps, and / or compressed auxiliary patch information. Some coded bitstreams may not have an occupancy map, because the occupancy map information may be inferred from the decoder side when the bounding vertices of each patch are signaled. In some implementations, the geometry patch is one of the patches that corresponds to the coded geometry map.

[0109] In some implementations, when the first syntax element indicates that the geometry patch is partitioned, the geometry patch may include two or more partitions.

[0110] In some implementations, when the first syntax element indicates that the geometry patch is not divided, the geometry patch may contain a single partition, which is the geometry patch itself.

[0111] In some implementations, each partition, including any partition when the geometry patch is divided or a single partition when the geometry patch is not divided, includes one or more points in the uv-plane, where the points in the partition correspond to vertices in the three-dimensional mesh, and the coded pixel values ​​of the points in the partition are associated with the three-dimensional coordinates of the vertices in the three-dimensional mesh.

[0112] In some implementations, the extreme pixel value of a point in a partition may be an offset relative to the coded pixel value in the partition. A pixel value of a point in a partition that corresponds to the 3D coordinate of a vertex in the 3D mesh may be obtained based on the coded pixel value of the point and the extreme pixel value of the point in the partition.

[0113] In some implementations, the extremal pixel value of a point in a partition can be either the minimum pixel value of the point in the partition or the maximum pixel value of the point in the partition. In a non-limiting example, when the pixel values ​​have three channels (e.g., RGB), the minimum pixel value may include three minimum values ​​for each channel, where the minimum pixel value is (r_min, g_min, b_min), where r_min is the minimum red channel value for all points in the partition, g_min is the minimum green channel value for all points in the partition, and b_min is the minimum blue channel value for all points in the partition.

[0114] In some implementations, in response to a first syntax element indicating that the geometry patch is not split, the geometry patch includes a single partition, e.g., in response to a first syntax element indicating that the geometry patch is not split, the geometry patch always includes a single partition.

[0115] In some implementations, in response to a first syntax element indicating that the geometry patch is partitioned, the method further includes extracting, by the device, from the coded bitstream a second syntax element indicating a partition type of the geometry patch, and partitioning, by the device, based on the partition type to obtain a plurality of partitions.

[0116] In some implementations, the example decoding method further includes, for a partition within the plurality of partitions, extracting, by the device, from the coded bitstream, a third syntax element indicating whether the partition is further divided, and in response to the third syntax element indicating that the partition is further divided, extracting, by the device, from the coded bitstream, a fourth syntax element indicating a partition type of the partition, and further dividing the partition based on the partition type of the partition.

[0117] In some implementations, in response to a first syntax element indicating that the geometry patch is not divided, the geometry patch includes a single partition.

[0118] In some implementations, in response to the first syntax element indicating that the geometry patch is split, the method further includes extracting, by the device, a second syntax element from the coded bitstream indicating a partition depth, and / or splitting, by the device, the geometry patch based on the partition depth and a predefined partition type to obtain a plurality of partitions.

[0119] In some implementations, the extreme pixel value of a point in the partition is a minimum pixel value, and determining the pixel value of the point in the partition based on the extreme pixel value and the coded pixel value includes adding together components of the minimum pixel value and each of the coded pixel values ​​to obtain a pixel value.

[0120] In some implementations, the minimum pixel value includes {x_min, y_min, z_min}, each of the coded pixel values ​​includes {x_coded, y_coded, z_coded), and each pixel value includes {x, y, z} determined according to x=x_min+x_coded, y=y_min+y_coded, and z=z_min+z_coded, where x, y, z, x_min, y_min, z_min, x_coded, y_coded, z_coded are non-negative integers.

[0121] In some implementations, the example decoding method may further include obtaining, by the device, from the coded bitstream, an over-constraint value of the over-constraint point in the partition.

[0122] In some implementations, the step of adding a component of the minimum pixel value and a component of each of the coded pixel values ​​for the limit overrun point to obtain a pixel value includes the step of adding a component of the limit overrun value, a component of the minimum pixel value, and a component of the coded pixel value that corresponds to the limit overrun point to obtain a pixel value of the limit overrun point.

[0123] In some implementations, for an over-limit point, the components of the coded pixel value that correspond to the non-zero components of the over-limit value are equal to 2N-1, where N is the bit depth of the coded pixel value.

[0124] In some implementations, the extremal pixel value of a point in a partition is the maximum pixel value.

[0125] In some implementations, the step of determining a pixel value for a point in the partition based on the extreme pixel value and the coded pixel value includes subtracting each component of the coded pixel value from a component of the maximum pixel value to obtain a pixel value.

[0126] In some implementations, the maximum pixel value includes {x_max, y_max, z_max}, each of the coded pixel values ​​includes {x_coded, y_coded, z_coded), and each pixel value includes {x, y, z} determined according to x=x_max-x_coded, y=y_max-y_coded, and z=z_max-z_coded, where x, y, z, x_max, y_max, z_max, x_coded, y_coded, z_coded are non-negative integers.

[0127] In some implementations, the exemplary decoding method may further include obtaining, by the device, from the coded bitstream an over-limit value of the over-limit point in the partition, where for the over-limit point, subtracting a component of each of the coded pixel values ​​from a component of the maximum pixel value to obtain a pixel value includes subtracting a component of the over-limit value and a component of each of the coded pixel values ​​from a component of the maximum pixel value to obtain a pixel value.

[0128] In some implementations, for an over-limit point, the components of the coded pixel value that correspond to the non-zero components of the over-limit value are equal to 2^N-1, where N is the bit depth of the coded pixel value.

[0129] In some implementations, obtaining an extreme pixel value of a point in the partition includes obtaining an extreme pixel value difference corresponding to the extreme pixel value of the point in the partition, and determining the extreme pixel value of the point in the partition based on the extreme pixel value difference and a second extreme pixel value.

[0130] In some implementations, the example decoding method may further include obtaining a second extremum pixel value of the second partition or of the second geometry patch from the coded bitstream.

[0131] The various steps in one or more embodiments or implementations may be applied individually or in any combination. Various embodiments of the present disclosure may be applied to dynamic meshes or static meshes, where there may be only one frame of the mesh or the mesh content does not change over time. Various embodiments of the present disclosure may be extended to coding of depth images / attribute images / texture images, etc., where the dynamic range of sample values ​​may be above a certain threshold, such as 8 bits.

[0132] In some implementations, each chart / patch in the geometry image may be partitioned so that the dynamic range of each partition falls within a particular dynamic range for coding the geometry image with a particular bit depth, such as 8 bits, 10 bits, etc. The charts / patches may be partitioned in one or more predefined patterns.

[0133] In a non-limiting example, referring to FIG. 10, quadtree partitioning may be used to divide a 2D geometry patch / chart. The 2D geometry patch / chart (1011) has an arbitrary shape and a bounding box (1010). The dynamic range of this patch / chart may exceed the target dynamic range (e.g., 8 bits). Quadtree partitioning may be applied to the bounding box (1010) of the patch / chart, resulting in a partition structure (1020) where the entire patch / chart has four partitions. When the dynamic range of each partition is within the target dynamic range (e.g., 8 bits), the division may be stopped. For example, when one partition (or called sub-partition) (1021) is within the target dynamic range, the division of this partition (1021) may be stopped.

[0134] If the dynamic range of each partition is not within the target dynamic range, the patch / chart may be further divided into smaller partitions until the dynamic range of all partitions is within the target dynamic range (e.g., 8 bits). For example, a partition (1030) that has been quadtree divided twice is shown, which contains 16 partitions (or called sub-partitions).

[0135] In some implementations, the separate partition may be applied only to partitions that are not within the target dynamic range, and the separate partition may not be applied to partitions that are within the target dynamic range.

[0136] This recursive partitioning algorithm may converge because the dynamic range decreases with each partition. In the worst case, all or part of the patch / chart may be partitioned down to a single pixel (e.g., the smallest partition that has a size of 1x1 and contains a single pixel).

[0137] FIG. 11 illustrates some non-limiting examples, including predefined partitioning patterns that allow for recursive partitioning to form a partitioning tree. In one example, some or all of the 10 partitioning structures or patterns may be predefined. Exemplary partitioning structures may include various 2:1 / 1:2 (third row of FIG. 11) and 4:1 / 1:4 (first row of FIG. 11) rectangular partitioning. The partition types with three subpartitions shown at 1102, 1104, 1106, and 1108 in the second row may be referred to as "T-shaped" partitions: left T, top T, right T, and bottom T, respectively. In some exemplary implementations, none of the rectangular partitions in FIG. 11 can be further subdivided. A partition depth may be further defined to indicate the partitioning depth from the patch or chart. In some implementations, only the all-square partitions of 1110 may allow for recursive partitioning to the next level of the partitioning tree. In other words, recursive partitioning may not be possible with square partitions in a T-shaped pattern. In some other examples, a quadtree structure may be used to divide a patch or a partition of a patch into quadtree partitions. Such quadtree division may be applied hierarchically and recursively to any square partition.

[0138] In some implementations, various partition structures / methods in video encoding / decoding may be applicable to the division of patches / charts.

[0139] In some implementations for implementing a mesh codec, the method may include some or all of the following: first, some syntax elements need to be signaled in the coded bitstream to assist the decoding process; second, the pixel values ​​in each partition need to be subtracted from the minimum value of that partition to modify the geometry image so that the image can be processed / coded in a specific bit depth range.

[0140] In some implementations, the syntax elements for each patch / chart may include some or all of the following:

[0141] A binary flag is signaled to indicate whether the patch / chart is partitioned or not. When the binary flag indicates that the patch / chart is partitioned, several partition flags are signaled to indicate how the patch / chart is partitioned. Partitions may be applied in various forms and the syntax elements associated with partitions may vary accordingly. For example, quad-trees, binary trees and any other forms of partitions, symmetric or asymmetric, may be applied. Also, several combinations of different partition structures may be applied.

[0142] In a non-limiting example, with reference to FIG. 9, only quad-trees are applied, and the quad-tree partition depth may be signaled to indicate the partitioning pattern.

[0143] In another non-limiting example, splitting is applied only to regions where the dynamic range is beyond the limit, and no splitting is applied to regions where the dynamic range is within the limit. In this case, a flag may be signaled for each partition / region to indicate whether further splitting is required, and when the flag indicates that further splitting is required, another flag may be signaled to indicate the partition type.

[0144] In some implementations, the minimum of each partition may be signaled, including the minimum x, y, z corresponding to the geometry image. When the patch / chart is not divided, the minimum of the entire patch may be signaled. In this case, the entire patch / chart may be considered as a single partition, and the minimum of the entire patch may be the minimum of the single partition.

[0145] In some implementations, some partitions may be empty (i.e., have no sampling points within them) and the minimum of the empty patch / chart may be skipped. For example, the bottom left partition (or subpartition) (1039) in Figure 10 is an empty partition, and there is no need to signal the minimum of partition (1039).

[0146] In some implementations, the minimum value of each partition in one patch / chart (e.g., the minimum x, y, z of the geometry image, respectively) needs to be signaled in the coded bitstream. For each sample value of each channel in the UV plane, the signaled minimum value of the same patch or partition can be subtracted from the value of this channel to reduce its dynamic range, while still keeping the update value non-negative. For example, let xm, ym, zm be the minimum values ​​of x, y, and z coordinates in a partition, respectively. For any pixel value (x, y, z) in the patch, the coded update value can be (x-xm, y-ym, z-zm).

[0147] In some implementations, when there are multiple patches in a mesh, each patch may have a minimum value. Predictive coding may be used to signal these values, and the minimum value of a patch may be used as a predictor for another patch. The difference between such pairs of minimum values ​​is coded to reduce the number of bits required for coding into the bitstream and improve mesh compression / decompression efficiency.

[0148] In some implementations, the signaling of the minimum value is done for each defined region. The 2D geometry image is divided into several non-overlapping regions. For each region, the minimum value per channel is signaled. Such coding of the minimum value can be done by predictive coding from previously coded values, such as spatially neighboring regions. For example, the whole image can be divided into a 64x64 based grid. For each 64x64 region, the minimum (x,y,z) of this region is signaled. All samples of this region are updated by subtracting the minimum value before coding.

[0149] In some implementations, a patch may be further divided into smaller sub-patches, where the dynamic range of the sample values ​​within each sub-patch falls within a particular dynamic range that can be supported by the video codec. An offset value for each channel is signaled, which is the minimum value for that channel of the samples within that sub-patch.

[0150] In some implementations, a special coding process may be applied to some isolated 3D points in a patch, where one or more channels of these points have a dynamic range that exceeds the given bit depth: the position of this point in the UV plane, together with the overshoot, is recorded and coded separately.

[0151] In some implementations, at one or more points with a dynamic range that exceeds the given bit depth, the original value may be replaced with the maximum possible value at the given bit depth. For example, when the bit depth is 8, the original value may be replaced with 255 (=2^8-1). In another example, when the bit depth is 12, the original value may be replaced with 4095 (=2^12-1). In some implementations, the final decoded value may be equal to the maximum value plus the excess portion.

[0152] In a non-limiting example, xm, ym, zm may represent the minimum values ​​of x, y, and z coordinates in the partition, respectively, and the bit depth with which the samples can be coded may be 8 bits. The update value for a point (x1, y1, z1) in the patch is (x1-xm, y1-ym, z1-zm). If x1-xm>255(=2^8-1), y1-ym<=255, z1-zm<=255, and the projected position of this point in the UV plane is (u1, v1), then in addition to the normal UV plane coding with 8-bit depth for each position in the UV plane, the position (u1, v1) is coded in the bitstream with additional over-limit information (x1-xm-255,0,0). On the decoder side, after decoding the geometry image using the method described above, an additional value (x1-xm-255,0,0) must be decoded and added to the already decoded 3-channel value at position (u1,v1) in order to recover the true geometry information at this point. This results in (x1-xm-255,0,0)+(255,y1-ym,z1-zm)=(x1-xm,y1-ym,z1-zm). By further adjusting each position by adding the minimum value of each channel (xm,ym,zm), the original geometry (x1,y1,z1) may be recovered.

[0153] In some implementations, if the dynamic range of the entire geometry image is known, such as [0,2^m), and each patch has a bit depth of n, where m>n, then the lower n bits of dynamic range of this value may be coded with other regular values ​​using a regular m-bit video codec. The higher (mn) bits of the geometry information of this value may be coded separately. In a non-limiting example where the bit depth is 8 bits, if the dynamic range of the entire geometry image is 14 bits, then the dynamic range of the higher bit information may be limited to a 6-bit range, since 6=14-8.

[0154] In some implementations, instead of the minimum value being coded / decoded, the maximum value may be used for coding / decoding. The maximum value of each partition in one patch (e.g., maximum x, y, z of the geometry image, respectively) may be signaled in the bitstream. For each sample value of each channel in the UV plane, the distance from the value of this channel to the signaled maximum value of the same patch or partition may reduce its dynamic range, while still keeping the update value non-negative. For example, xm, ym, zm are the maximum values ​​of the partition in the three coordinate axes, respectively. For any point (x, y, z) in the patch, the update value coded may be (xm-x, ym-y, zm-z).

[0155] In some implementations, a minimum value may be signed with some patches and a maximum value may be signed with some other patches. In some implementations, for a single patch, a minimum value may be signed with some patches within the patches and a maximum value may be signed with some other patches within the patches.

[0156] In some implementations, the signaling of the maximum value may be done for each defined region.

[0157] In some implementations, a patch may be further divided into smaller sub-patches, where the dynamic range of the sample values ​​within each sub-patch falls within a particular dynamic range that may be supported by a video codec, and an offset value for each channel may be signaled, which may be the maximum value for that channel of the samples within that sub-patch.

[0158] In some implementations, a special coding process is applied to some isolated 3D points in a patch, where one or more channels of these points have a dynamic range that exceeds the given bit depth: the position of this point in the UV plane, together with the overshoot, is recorded and coded separately, and at the same time the original value can be replaced by the maximum possible value for the given bit depth.

[0159] In a non-limiting example, if xm, ym, zm are the maximum values ​​of the patch in the three coordinate axes, respectively, and the bit depth allowed for coding samples is 8 bits, then for a point (x1, y1, z1) in the patch, the updated value is (xm-x1, ym-y1, zm-z1). If xm-x1>255, ym-y1<=255, zm-z1<=255, and the projected position of this point in the UV plane is (u1,v1), then in addition to coding the normal UV plane with 8 bit depth for each position in the UV plane, the position (u1,v1) is coded in the bitstream with additional over-limit information (xm-x1-255,0,0). At the decoder side, after decoding the geometry image using the method described above, in order to correctly recover the true geometry information of this point, an additional value (xm-x1-255,0,0) can be decoded and added to the already decoded three channel value at position (u1,v1). This results in (xm-x1-255,0,0)+(255,ym-y1,zm-z1)=(xm-x1,ym-y1,zm-z1). This can be further adjusted by subtracting it from the maximum value of each channel (xm,ym,zm) for each position to recover the original geometry (x1,y1,z1)=(xm,ym,zm)-(xm-x1,ym-y1,zm-z1).

[0160] In this disclosure, the notation "m^n" refers to the exponentiation operation, i.e., m n , corresponding to the base m, and the exponent or power n.

[0161] The techniques disclosed in this disclosure may be used separately or combined in any order. Furthermore, each of the techniques (e.g., methods, embodiments), encoders, and decoders may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In some examples, the one or more processors execute a program stored in a non-transitory computer-readable medium.

[0162] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, FIG. 12 illustrates a computer system (1300) suitable for implementing certain embodiments of the disclosed subject matter.

[0163] The computer software may be coded using any suitable machine code or computer language that may undergo assembly, compilation, linking, or similar mechanisms to generate code including instructions that may be executed directly, or through interpretation and execution of microcode, such as by one or more computer central processing units (CPUs) and graphics processing units (GPUs).

[0164] The instructions may be executed in various types of computers or components thereof including, for example, a personal computer, a tablet computer, a server, a smart phone, a gaming device, an Internet of Things device, and the like.

[0165] 12 for computer system (1300) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (1300).

[0166] The computer system (1300) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, via tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), and olfactory input (not shown). The human interface devices may be used to capture certain media that are not necessarily directly related to conscious human input, such as sound (speech, music, ambient sounds, etc.), images (scanned images, photographic images obtained from a still image camera, etc.), and video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0167] The input human interface devices may include one or more of a keyboard (1301), a mouse (1302), a trackpad (1303), a touch screen (1310), a data glove (not shown), a joystick (1305), a microphone (1306), a scanner (1307), and a camera (1308) (only one of each is depicted).

[0168] The computer system (1300) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1310), data gloves (not shown), or joystick (1305), although there may be haptic feedback devices that do not function as input devices), audio output devices (such as speakers (1309), headphones (not depicted)), visual output devices (such as screens (1310), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or three-dimensional or higher output via means such as stereographic output, virtual reality glasses (not depicted), holographic displays, and smoke tanks (not depicted)), and printers (not depicted).

[0169] The computer system (1300) may also include human-accessible storage devices and their associated media, such as optical media, including CD / DVD ROM / RW (1320) with CD / DVD or similar media (1321), thumb drives (1322), removable hard drives or solid state drives (1323), legacy magnetic media (not depicted) such as tapes and floppy disks, and specialized ROM / ASIC / PLD based devices (not depicted) such as security dongles.

[0170] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0171] The computer system (1300) may also include an interface (1354) to one or more communication networks (1355). The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, and the like. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, and the like, television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicular and industrial including CANBus, and the like. Certain networks typically require an external network interface adapter connected to a particular general-purpose data port or peripheral bus (1349) (e.g., a USB port on the computer system (1300)), while others are generally integrated into the core of the computer system (1300) by connecting to a system bus, as described below (e.g., an Ethernet interface for a PC computer system, or a cellular network interface for a smartphone computer system). Using any of these networks, the computer system (1300) may communicate with other entities. Such communications may be one-way, receive-only (e.g., television broadcast), one-way transmit-only (e.g., from a CANbus to a particular CANbus device), or two-way, e.g., to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used in each of these networks and network interfaces, as described above.

[0172] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to a core (1340) of the computer system (1300).

[0173] The cores (1340) may include one or more central processing units (CPUs) (1341), graphics processing units (GPUs) (1342), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (1343), hardware accelerators for specific tasks (1344), graphics adapters (1350), and the like. These devices may be connected through a system bus (1348), along with read only memory (ROM) (1345), random access memory (1346), and internal mass storage (1347), such as an internal non-user accessible hard drive, SSD, and the like. In some computer systems, the system bus (1348) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and the like. Peripheral devices may be attached directly to the core's system bus (1348) or through a peripheral bus (1349). In one example, a screen (1310) may be connected to a graphics adapter (1350). Architectures for peripheral buses include PCI, USB, etc.

[0174] The CPU (1341), GPU (1342), FPGA (1343) and accelerator (1344) may combine to execute certain instructions that may constitute the aforementioned computer code. This computer code may be stored in ROM (1345) or RAM (1346). Transient data may also be stored in RAM (1346), while persistent data may be stored in, for example, internal mass storage (1347). Rapid storage and retrieval from any of the memory devices may be enabled using cache memory, which may be closely associated with one or more of the CPU (1341), GPU (1342), mass storage (1347), ROM (1345), RAM (1346), etc.

[0175] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the type well known and available to those skilled in the computer software arts.

[0176] By way of example and not limitation, a computer system (1300) having an architecture, and specifically a core (1340), may provide functionality as a result of a processor (one or more) (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage as previously described, as well as media associated with specific storage of the core (1340) that is non-transitory in nature, such as the core internal mass storage (1347) or ROM (1345). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (1340). The computer-readable media may include one or more memory devices or chips, depending on the particular needs. The software may cause the core (1340), and specifically the processors therein (including a CPU, GPU, FPGA, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (1346) and modifying such data structures according to processes defined by the software. Additionally, or alternatively, a computer system may provide functionality as a result of hardwired or otherwise embodied logic in circuitry (e.g., accelerator (1344)) that can operate in place of or in conjunction with software to perform particular processes or particular portions of particular processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0177] While this disclosure has described several exemplary embodiments, there are modifications, substitutions, and various substitute equivalents which are within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods which, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure. [Explanation of symbols]

[0178] 100 communication system, 105 3D sensor, 110 terminal device, 120 terminal device, 150 communication network, 200 streaming system, 201 storage medium, 202 uncompressed 3D mesh (point cloud), 203 encoder, 204 compressed 3D mesh, 205 streaming server, 206 client subsystem, 207 copy, 208 client subsystem, 209 copy, 210 decoder, 211 reconstructed 3D mesh, 212 rendering device, 213 storage subsystem, 220 electronic device, 230 electronic device, 301 encoder, 302 raw input 3D mesh or sequence of 3D meshes, 303 decoder, 304 mapping to UV space, 310 occupancy map, 312 geometry map, 314 attribute map, 316 other non-image or non-map data or metadata, 320 decoded occupancy map, 322 decoded geometry map, 324 decoded attribute map, 326 decoded non-mapped data and metadata, 330 reconstructed 3D mesh or sequence of 3D meshes, 400 mesh encoder, 404 patch information module, 406 patch generation module, 408 patch packing module, 410 geometry image generation module, 412 texture image generation module, 414 occupancy map module, 416 image padding module, 418 image padding module, 420 group expansion module, 422 video compression module, 423 video compression module, 424 multiplexer, 432 video compression module, 434 entropy compression module, 436 smoothing module, 438 auxiliary patch information compression module, 500 mesh decoder, 532 demultiplexer, 534 video decompression module, 536 video decompression module, 538 occupancy map decompression module, 542 auxiliary patch information decompression module, 544 geometry reconstruction module, 546 Smoothing module, 548 Texture reconstruction module, 552 Color smoothing module, 610 Video decoder, 620 Parser, 621 Symbol, 651 Scaler / inverse transform unit, 652 Intra-picture prediction unit, 653motion compensation prediction unit, 655 aggregator, 656 loop filter unit, 657 reference picture memory, 658 current picture buffer, 703 video encoder, 730 source coder, 732 coding engine, 733 local video decoder, 734 reference picture memory, 735 predictor, 743 compressed image, 745 entropy coder, 750 controller, 810 3D patch, 811 first vertex, 812 second vertex, 813 third vertex, 814 fourth vertex, 819 3D xyz coordinate system, 850 2D UV plane, 851 first point, 852 second point, 853 third point, 854 fourth point, 859 uv coordinate system, 900 flow chart, 1010 bounding box, 1011 2D geometry patch / chart, 1020 partition structure, 1021 partition, 1030 quadtree partition twice, 1039 empty partition, 1102 subpartition, 1104 subpartition, 1106 subpartition, 1108 subpartition, 1110 all square partition, 1300 computer system, 1301 keyboard, 1302 mouse, 1303 trackpad, 1305 joystick, 1306 microphone, 1307 scanner, 1308 camera, 1309 speaker, 1310 touch screen, 1320 optical medium, 1321 medium, 1322 thumb drive, 1323 solid state drive, 1340 core, 1341 central processing unit (CPU), 1342 graphics processing unit (GPU), 1343 field programmable gate area (FPGA), 1344 hardware accelerator, 1345 Read-only memory (ROM), 1346 Random Access Memory (RAM), 1347 Internal Mass Storage, 1348 System Bus, 1349 Peripheral Bus, 1350 Graphics Adapter, 1354 Interface, 1355 Communication Network

Claims

1. 1. A method for decoding a geometry patch of a three-dimensional mesh, comprising the steps of: receiving, by a device having a memory storing instructions and a processor in communication with the memory, a coded bitstream including a geometry patch of a three-dimensional mesh; extracting, by the device, from the coded bitstream a first syntax element indicating whether the geometry patch is partitioned, the geometry patch including one or more partitions; for each partition in the geometry patch, obtaining by the device a dynamic range of pixel values ​​for points in the partition that correspond to vertices of the 3D mesh, the dynamic range allowing the geometry patch to be coded within a predefined bit depth; A method comprising:

2. the coded bitstream comprising: A coded geometry map, or Coded Metadata The method of claim 1 , comprising at least one of:

3. in response to the first syntax element indicating that the geometry patch is not partitioned, the geometry patch includes a single partition; In response to the first syntax element indicating that the geometry patch is split, the method further comprises: extracting, by the device, from the coded bitstream a second syntax element indicating a partition type of the geometry patch; dividing, by the device, the geometry patch to obtain a plurality of partitions based on the partition type; The method of claim 1 further comprising:

4. For a partition within the plurality of partitions, extracting, by the device, from the coded bitstream a third syntax element indicating whether the partition is further divided; in response to the third syntax element indicating that the partition is further divided, extracting, by the device, from the coded bitstream a fourth syntax element indicating a partition type of the partition, and further dividing the partition based on the partition type of the partition; The method of claim 3 further comprising:

5. in response to the first syntax element indicating that the geometry patch is not partitioned, the geometry patch includes a single partition; In response to the first syntax element indicating that the geometry patch is split, the method further comprises: extracting, by the device, from the coded bitstream a second syntax element indicating a partition depth; dividing, by the device, the geometry patch based on the partition depth and a predefined partition type to obtain a plurality of partitions; The method of claim 1 further comprising:

6. obtaining the dynamic range of pixel values ​​for the points in each partition corresponding to the vertices in the three-dimensional mesh, comprising: obtaining extremal pixel values ​​of the points within each of the partitions; obtaining coded pixel values ​​for said points in each said partition; determining pixel values ​​of the points in each partition based on the extremal pixel values ​​and the coded pixel values, the pixel values ​​corresponding to three-dimensional coordinates of the vertices in the three-dimensional mesh; The method of claim 1 , comprising:

7. the extremal pixel value of the point in each partition is a minimum pixel value; determining the pixel value of the point in each partition based on the extremal pixel value and the coded pixel value, adding together the components of the minimum pixel value and each of the coded pixel values ​​to obtain the pixel value. The method of claim 6, comprising:

8. the minimum pixel value comprises {x_min, y_min, z_min}, and each of the coded pixel values ​​comprises {x_coded, y_coded, z_coded); comprising {x, y, z}, where each pixel value is determined according to x=x_min+x_coded, y=y_min+y_coded, and z=z_min+z_coded; x, y, z, x_min, y_min, z_min, x_coded, y_coded, z_coded are non-negative integers. The method of claim 7.

9. obtaining, by the device, an over-constraint value of an over-constraint point in each of the partitions from the coded bitstream. Further comprising: said step of adding, for said limit exceeding point, a component of said minimum pixel value and a component of each of said coded pixel values ​​to obtain said pixel value, adding together the over-limit value component, the minimum pixel value component, and a coded pixel value component corresponding to the over-limit point to obtain the pixel value of the over-limit point. The method of claim 7, comprising:

10. For the over-limit point, the components of the coded pixel value that correspond to the non-zero components of the over-limit value are Ｎ −1, and N is the predetermined bit depth of the coded pixel values.

10. The method of claim 9.

11. the extremal pixel value of a point in each partition is a maximum pixel value; determining the pixel value of the point in each partition based on the extremal pixel value and the coded pixel value, subtracting each component of the coded pixel value from the component of the maximum pixel value to obtain the pixel value. The method of claim 6, comprising:

12. the maximum pixel value comprises {x_max, y_max, z_max}, and each of the coded pixel values ​​comprises {x_coded, y_coded, z_coded); comprising {x, y, z}, where each pixel value is determined according to x=x_max-x_coded, y=y_max-y_coded, and z=z_max-z_coded; x, y, z, x_max, y_max, z_max, x_coded, y_coded, z_coded are non-negative integers. The method of claim 11.

13. obtaining, by the device, from the coded bitstream, an over-constraint value of an over-constraint point in each of the partitions. Further comprising: for said limit exceeding point, subtracting a component of each of said coded pixel values ​​from a component of said maximum pixel value to obtain said pixel values; subtracting the limit overshoot component and each of the coded pixel values ​​from the maximum pixel value component to obtain the pixel value. The method of claim 11 , comprising:

14. For the over-limit point, the components of the coded pixel value that correspond to the non-zero components of the over-limit value are Ｎ −1, and N is the predetermined bit depth of the coded pixel values. The method of claim 13.

15. said step of obtaining the extremal pixel values ​​of said points in each partition further comprises: obtaining extreme pixel value differences corresponding to the extreme pixel values ​​of the points in each partition; determining the extremal pixel value of the point in each partition based on the extremal pixel value difference and a second extremal pixel value; The method of claim 6, comprising:

16. obtaining the second extremal pixel value of a second partition or of a second geometry patch from the coded bitstream. The method of claim 15 further comprising:

17. 1. An apparatus for decoding a geometry patch of a three-dimensional mesh, the apparatus comprising: A memory for storing instructions; A processor in communication with the memory, the processor being configured, when the processor executes the instructions, to cause the apparatus to perform a method according to any one of claims 1 to 16. An apparatus comprising:

18. A computer program comprising instructions, which when executed by a processor are configured to cause the processor to perform a method according to any one of claims 1 to 16.

Images