Predictive coding of boundary geometry information for mesh compression.

Abstract

The present disclosure relates to a method and system for encoding or decoding a 3D mesh using predictive coding of boundary geometry information. An exemplary decoding method includes receiving a coded bitstream including a geometry patch for the three-dimensional mesh, extracting from the coded bitstream a reconstructed geometry image for the geometry patch including a set of points corresponding to a set of geometry coordinates for boundary vertices in the three-dimensional mesh, deriving 3D geometry coordinates for the boundary vertices based on the set of geometry coordinates, obtaining prediction residuals for the boundary vertices, and reconstructing, by a device, geometry information of the boundary vertices based on the derived 3D geometry coordinates and the prediction residuals.

Description

[Technical field]

[0001] [Incorporated by reference] This application is based on and claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 319,976, filed March 15, 2022, the entire contents of which are incorporated by reference. This application also is based on and claims the benefit of priority to U.S. Provisional Patent Application No. 17 / 973,792, filed October 26, 2022, the entire contents of which are incorporated by reference.

[0002] [Technical field] The present disclosure relates generally to mesh encoding (or compression) and mesh decoding (or decompression) processes, and more particularly to methods and systems for predictive coding of boundary geometry information for mesh compression. [Background technology]

[0003] This background discussion provided herein is intended to generally present the context of the present disclosure. The work of the presently named inventors is not admitted explicitly or implicitly as prior art to the present disclosure, to the extent that that work is described in this background, along with aspects of the description that would not ordinarily be considered prior art at the time of filing.

[0004] Various techniques have been developed to capture, represent, and simulate real-world objects, environments, and the like in three-dimensional (3D) space. A 3D representation of the world can enable a more immersive form of interactive communication. Exemplary 3D representations of objects and environments include, but are not limited to, point clouds and meshes. A series of 3D representations of objects and environments may form a video sequence. Redundancies and correlations within a sequence of 3D representations of objects and environments may be exploited to compress and code such video sequences into a more compact digital format. Summary of the Invention

[0005] This disclosure relates generally to coding (compression) and decoding (decompression) of 3D meshes, and more particularly to predictive coding of boundary geometry information for mesh compression.

[0006] This disclosure describes an embodiment of a method for decoding a geometry patch for a three-dimensional mesh. The method includes receiving, by a device, a coded bitstream including the geometry patch for the three-dimensional mesh. The device includes a memory that stores instructions and a processor in communication with the memory. The method also includes extracting, by the device, from the coded bitstream a reconstructed geometry image for the geometry patch including a set of points corresponding to a set of geometry coordinates for bound vertices in the three-dimensional mesh, deriving, by the device, 3D geometry coordinates for the bound vertices based on the set of geometry coordinates, obtaining, by the device, prediction residuals for the bound vertices, and reconstructing, by the device, geometry information for the bound vertices based on the derived 3D geometry coordinates and the prediction residuals.

[0007] This disclosure describes another embodiment of a method for decoding a geometry patch for a three-dimensional mesh. The method includes receiving, by a device, a coded bitstream. The device includes a memory that stores instructions and a processor in communication with the memory. The method also includes extracting, by the device, a plurality of geometry predictions for bound vertices from the coded bitstream, each geometry prediction including a set of predicted geometry coordinates; determining, by the device, predicted geometry values ​​for the bound vertices based on the plurality of geometry predictions; obtaining, by the device, prediction residuals for the bound vertices; and reconstructing, by the device, geometry information for the bound vertices based on the predicted geometry values ​​and the prediction residuals.

[0008] According to another aspect, an embodiment of the present disclosure provides an apparatus for encoding or decoding a 3D mesh, the apparatus including a memory storing instructions and a processor in communication with the memory, the instructions being configured, when executed by the processor, to cause the apparatus to perform the method described above.

[0009] In another aspect, embodiments of the present disclosure provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform the above-described method for video decoding and / or encoding.

[0010] These and other aspects and implementations thereof are described in more detail in the drawings, detailed description, and claims. [Brief description of the drawings]

[0011] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Diagram 2] FIG. 1 is a schematic diagram of a simplified block diagram of a streaming system according to one embodiment. [Diagram 3] 1 shows a block diagram of an encoder for encoding a mesh frame according to some embodiments. [Figure 4] 1 shows a block diagram of a decoder for decoding a compressed bitstream corresponding to a mesh frame according to some embodiments. [Diagram 5] FIG. 2 is a schematic diagram of a simplified block diagram of a video decoder according to one embodiment. [Figure 6] FIG. 2 is a schematic diagram of a simplified block diagram of a video encoder according to one embodiment. [Figure 7] 1 shows a block diagram of an encoder for encoding a mesh frame according to some embodiments. [Figure 8]1 shows a diagram of a framework for mesh compression according to some embodiments of the present disclosure. [Figure 9] FIG. 1 illustrates another diagram of a framework for mesh compression according to some embodiments of the present disclosure. [Figure 10] FIG. 13 is another diagram of a framework for mesh compression according to some embodiments of the present disclosure. [Figure 11] 1 shows a flowchart outlining an example process according to some embodiments. [Figure 12] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Throughout this specification and the claims, terms may have subtle meanings that are suggested or implied in the context beyond the explicitly stated meaning. The phrases "in one embodiment" or "in some embodiments" used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in other embodiments" used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" used herein do not necessarily refer to the same implementation, and the phrases "in another implementation" or "in other implementations" used herein do not necessarily refer to different implementations. For example, the subject matter of the claims is intended to include a combination of example embodiments / implementations in whole or in part.

[0013] Generally, terms may be understood at least in part from their usage in context. For example, terms such as "and", "or", or "and / or" as used herein may include various meanings that may depend at least in part on the context in which such terms are used. Typically, when "or" is used to relate a list such as A, B, or C, it is intended to mean A, B, and C, which are used herein in an inclusive sense, and A, B, or C, which are used herein in an exclusive sense. Furthermore, the terms "one or more" or "at least one" as used herein may be used to describe any feature, structure, or characteristic in the singular, or may be used to describe a combination of features, structures, or characteristics in the plural, depending at least in part on the context. Similarly, terms such as "a", "an", or "the" may be understood to convey a singular usage or a plural usage, depending at least in part on the context. Additionally, the terms "based on" or "determined by" may be understood as not necessarily intended to convey an exclusive set of elements, but instead may allow for the presence of additional elements not necessarily explicitly recited, depending at least in part on the context.

[0014] Technological developments in 3D media processing, such as advances in 3D capture, 3D modeling, and 3D rendering, are facilitating the ubiquitous creation of 3D content across several platforms and devices. Such 3D content includes information that can be processed to generate various forms of media, for example, to provide immersive viewing / rendering and interactive experiences. Applications of 3D content are abundant and include, but are not limited to, virtual reality, augmented reality, metaverse interactions, gaming, immersive video conferencing, robotics, computer-aided design (CAD), and the like. According to aspects of the present disclosure, to improve the immersive experience, 3D models are becoming increasingly sophisticated, and the creation and consumption of 3D models requires a significant amount of data resources, such as data storage, data transmission resources, and data processing resources.

[0015] Compared to traditional 2D, 2-dimensional content, which is typically represented by a dataset in the form of a 2D pixel array (such as an image), 3D content with three-dimensional full resolution pixelation can be quite resource intensive and is nevertheless unnecessary in many, if not most, practical applications. In most 3D immersive applications, a less data-intensive representation of the 3D content may be used, according to some aspects of the present disclosure. For example, most applications may only require topographical information, not volumetric information, of objects in a 3D scene (either a real-world scene captured by a sensor such as a LIDAR device, or an animated 3D scene generated by a software tool). Thus, more efficient forms of datasets may be used to represent 3D objects and 3D scenes. For example, a 3D mesh may be used as a type of 3D model to represent immersive 3D content, such as a 3D object in a 3D scene.

[0016] A mesh (alternatively called a mesh model) of one or more objects may include a collection of vertices. The vertices may connect to each other to form edges. The edges may further connect to form faces. The faces may further form polygons. 3D surfaces of various objects may be decomposed into faces and polygons, for example. Each of the vertices, edges, faces, polygons or surfaces may be associated with various attributes such as color, normal, texture, etc. The normal of a surface may be referred to as a surface normal and / or the normal of a vertex may be referred to as a vertex normal. Information of how the vertices are connected to the edges, faces or polygons may be referred to as connectivity information. Connectivity information is important for uniquely defining components of a mesh, since the same set of vertices can form different faces, surfaces and polygons. In general, the position of a vertex in 3D space may be represented by its 3D coordinates. A face may be represented by a set of contiguously connected vertices, each associated with a set of 3D coordinates. Similarly, an edge may be represented by two vertices each associated with its 3D coordinates. The vertices, edges and faces may be indexed in the 3D mesh dataset.

[0017] A mesh may be defined and described by a collection of one or more of these basic element types. However, not all of the above types of elements are necessary to completely describe a mesh. For example, a mesh may be completely described by simply using vertices and their connectivity. In another example, a mesh may be completely described by simply using a list of faces and their common vertices. Thus, meshes can be of a variety of alternative types described by alternative dataset configurations and formats. Exemplary mesh types include, but are not limited to, face-vertex meshes, winged-edge meshes, half-edge meshes, quad-edge meshes, corner-table meshes, vertex-vertex meshes, etc. Correspondingly, the mesh dataset may be stored with information conforming to alternative file formats having file extensions including, but not limited to, .raw, .blend, .fbx, .3ds, .dae, .dng, 3dm, .dsf, .dwg, .obj, .ply, .pmd, .stl, amf, .wrl, .wrz, .x3d, .x3db, .x3dv, .x3dz, .x3dbz, .x3dvz, ​​.c4d, .lwo, .smb, .msh, .mesh, .veg, .z3d, .vtk, .l4d, etc. Attributes of these elements, such as color, normals, textures, etc., may be included in the mesh dataset in various manners.

[0018] In some implementations, the vertices of a mesh may be mapped to a pixelated 2D space called UV space. Thus, each vertex of a mesh may be mapped to a pixel in UV space. In some implementations, a vertex may be mapped to more than one pixel in UV space, for example, a vertex at a boundary may be mapped to two or three pixels in UV space. Similarly, a face or surface in a mesh may be sampled to multiple 3D points that may or may not be among the recorded vertices in the mesh, and these multiple 3D points may also be mapped to pixels in the two-dimensional UV space. Mapping the vertices and sampled 3D points of a face or surface in a mesh to UV space and performing subsequent data analysis and processing in UV space may facilitate data storage, compression, and coding of a 3D data set of a mesh or a sequence of meshes, as described in more detail below. The mapped UV space data set may be referred to as a UV image or a 2D map or a 2D image of a mesh.

[0019] Following mapping of the vertices and sampled surface points in the 3D mesh to the 2D UV space, some pixels may be mapped to the vertices and sampled surface points of the 3D mesh, while other pixels may not be mapped. Each mapped pixel in the 2D image of the mesh may be associated with information of the corresponding mapped vertex or surface point in the 3D mesh. Depending on the type of information contained in the pixel in UV space, various 2D images or maps of the mesh may be constructed. A collection of multiple 2D maps may be used as alternative and / or joint representations of the mesh.

[0020] For example, the simplest 2D map for a mesh may be constructed as an occupancy map. The occupancy map may indicate which pixels in UV space are mapped to the 3D vertices or sampled surface points of the mesh. The indication of occupancy may be represented by a binary indicator in each of the 2D pixels, for example with a binary value "1" indicating mapping or occupancy and a binary value "0" indicating non-mapping or non-occupancy. Thus, the occupancy map may be constructed as a 2D image. While a typical 2D image includes an array of three channels (RGB, YUV, YCrCb, etc.) with, for example, 8-bit color depth, a 2D occupancy map for such a mesh only requires a single-bit binary channel.

[0021] In another example, a 2D geometry map may be constructed for the mesh that, instead of containing a single binary channel, is a full three-channel image, where the three color channels at each occupied pixel correspond to the three 3D coordinates of the corresponding mapped vertex or sampled 3D point in the mesh.

[0022] In some implementations, other 2D maps may be constructed for the mesh. For example, a set of attributes for each of the vertices and sampled 3D points of the mesh may be extracted from the mesh dataset and coded into three color channels of a 2D map image. Such a 2D map may be referred to as an attribute map of the mesh. A particular attribute map may include three channels of color for each occupied pixel in UV space. In another example, texture attributes associated with each mapped vertex or sampled 3D point of the mesh may be parameterized into a three-channel value and coded into a 2D attribute map. In another example, normal attributes associated with each mapped vertex or sampled 3D point of the mesh may be parameterized into a three-channel value and coded into a 2D attribute map. In some example implementations, multiple 2D attribute maps may be constructed to hold all the necessary attribute information of the vertices and sampled surface points of the mesh.

[0023] The above 2D maps are merely examples. Other types of 2D maps for a mesh can be constructed. Additionally, other data sets may be extracted from the 3D mesh and carried forward with the above 2D maps to jointly represent the original 3D mesh. For example, the connection or connectivity information between vertices may be grouped and organized separately from the 2D map in the form of a list, table, etc. The connectivity information may, for example, refer to the vertices using vertex indices. The vertex indices may be mapped to their corresponding pixel locations in the 2D map. In another example, surface texture, color, normals, displacements, and other information may be extracted and organized separately from the 2D map, rather than as a 2D map. Other metadata may be further extracted from the 3D mesh to represent the 3D mesh together with the 2D map and the other data sets described above.

[0024] Although the above example implementations focus on meshes that are static, according to aspects of the present disclosure, a 3D mesh may be dynamic. A dynamic mesh may, for example, refer to a mesh in which at least one of its components (geometry information, connectivity information, mapping information, vertex attributes, and attribute maps) changes over time. Thus, a dynamic mesh can be described by a sequence of one or more meshes (also called mesh frames), similar to a timed sequence of 2D image frames that form a video.

[0025] In some example implementations, the dynamic mesh may have constant connectivity information, time-varying geometry, and time-varying vertex attributes. In some other examples, the dynamic mesh can have time-varying connectivity information. In some examples, digital 3D content creation tools may be used to generate the dynamic mesh with time-varying attribute maps and time-varying connectivity information. In some other examples, volumetric acquisition / detection / sensing techniques are used to generate the dynamic mesh. Volumetric acquisition techniques can generate dynamic meshes with time-varying connectivity information, especially under real-time constraints.

[0026] Dynamic meshes may require large amounts of data, as they may contain a significant amount of information that changes over time. However, compression may be performed to exploit redundancy within mesh frames (intra-compression) and between mesh frames (inter-compression). Various mesh compression processes may be implemented to allow efficient storage and transmission of media content in mesh representations, especially for mesh sequences.

[0027] Aspects of the present disclosure provide example architectures and techniques for mesh compression. The techniques may be used for various mesh compressions, including but not limited to static mesh compression, dynamic mesh compression, compression of dynamic meshes with constant connectivity information, compression of dynamic meshes with time-varying connectivity information, compression of dynamic meshes with time-varying attribute maps, etc. The techniques may be used in lossy and lossless compression for various applications, such as real-time immersive communication, storage, free viewpoint video, augmented reality (AR), virtual reality (VR), etc. Applications may include features such as random access and scalable / progressive coding.

[0028] Although this disclosure explicitly describes techniques and implementations applicable to 3D meshes, the principles underlying the various implementations described herein are applicable to other types of 3D data structures, including, but not limited to, point cloud (PC) data structures. For brevity, references below to 3D meshes are intended to be general and include other types of 3D representations such as point clouds and other 3D volumetric datasets.

[0029] Referring first to an exemplary architecture-level implementation, FIG. 1 illustrates a simplified block diagram of a communication system (100) according to an exemplary embodiment of the present disclosure. The communication system (100) includes a plurality of terminal devices that can communicate with each other, for example, via a communication network (150) (alternatively referred to as a network). For example, the communication system (100) may include a pair of terminal devices (110) and (120) interconnected via the network (150). In the example of FIG. 1, a first pair of terminal devices (110) and (120) may perform a unidirectional transmission of a 3D mesh. For example, the terminal device (110) may compress a 3D mesh or a sequence of 3D meshes, which may be generated by the terminal device (110) or may be acquired for storage or may be captured by a 3D sensor (105) connected to the terminal device (110). The compressed 3D mesh or sequence of 3D meshes may be transmitted, for example in the form of a bitstream (also called a coded bitstream), via the network (150) to another terminal device (120). The terminal device (120) may receive the compressed 3D mesh or sequence of 3D meshes from the network (150), decompress the bitstream to reconstruct the original 3D mesh or sequence of 3D meshes, and appropriately process the reconstructed 3D mesh or sequence of 3D meshes for display or other purposes / uses. One-way data transmission may be typical, for example, in media serving applications.

[0030] In the example of FIG. 1, one or both of the terminal devices (110) and (120) may be implemented as a server, a fixed or mobile personal computer, a laptop computer, a tablet computer, a smart phone, a gaming console, a media player, and / or a dedicated three-dimensional (3D) appliance, etc., and the principles of the present disclosure may not be limited thereto. The network (150) may represent any type of network or combination of networks that transmits compressed 3D images between the terminal devices (110) and (120). The network (150) may include, for example, wired and / or wireless communication networks. The network (150) may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include backhaul telecommunications networks, local area networks, wide area networks, cellular networks, and / or the Internet. For purposes of the present disclosure, the architecture and topology of the network (150) may not be important to the operation of the present disclosure, unless otherwise described herein below.

[0031] 2 illustrates an example simplified block diagram of a streaming system (200) according to an embodiment of the present disclosure. FIG. 2 illustrates an example application for implementation of the disclosure related to 3D meshes and compressed 3D meshes. The disclosed subject matter may be similarly applicable to other 3D mesh or point cloud enabled applications, such as 3D telepresence applications, virtual reality applications, etc.

[0032] The streaming system (200) may include a capture or storage subsystem (213). The capture or storage subsystem (213) may include a 3D mesh generator or storage medium (201), such as a 3D mesh or point cloud generation tool / software that generates or provides an uncompressed 3D mesh (202) or point cloud (202), a graphics generation component, or a point cloud sensor, such as a light detection and ranging (LIDAR) system, a 3D camera, a 3D scanner, a 3D mesh store, etc. In some example implementations, the 3D mesh (202) includes vertices of a 3D mesh or 3D points of a point cloud (both referred to as a 3D mesh). The 3D mesh (202) is shown in bold to highlight the high amount of data compared to the compressed 3D mesh (204) (compressed 3D mesh bitstream). The compressed 3D mesh (204) may be generated by an electronic device (220) that includes an encoder (203) coupled to the 3D mesh (202). The encoder (203) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The compressed 3D mesh (204) (or a bitstream of the compressed 3D mesh (204)), depicted as a thin line to emphasize the low amount of data compared to the stream of the uncompressed 3D mesh (202), may be stored on the streaming server (205) for future use. One or more streaming client subsystems, such as the client subsystems (206) and (208) of FIG. 2, may access the streaming server (205) to retrieve copies (207) and (209) of the compressed 3D mesh (204). The client subsystem (206) may include a decoder (210), for example within the electronic device (230). The decoder (210) decodes the input copy (207) of the compressed 3D mesh and creates an output stream of a reconstructed 3D mesh (211) that can be rendered on a rendering device (212) or for other uses.

[0033] It is noted that electronic devices 220 and 230 may include other components (not shown). For example, electronic device 220 may include a decoder (not shown), and similarly, electronic device 230 may include an encoder (not shown).

[0034] In some streaming systems, the compressed 3D meshes 204, 207, and 209 (e.g., a bitstream of the compressed 3D meshes) can be compressed according to a certain standard. In some examples, a video coding standard is used to exploit redundancy and correlation in compressing the 3D meshes after they are first projected to be mapped to a 2D representation suitable for video compression, as described in more detail below. Non-limiting examples of these standards include High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), etc., as described in more detail below.

[0035] A compressed 3D mesh or sequence of 3D meshes may be generated by an encoder, and a decoder may be configured to decompress the compressed or coded 3D mesh. Figure 3 shows a high-level example data flow of a 3D mesh in such an encoder (301) and decoder (303). As shown in Figure 3, a raw input 3D mesh or sequence of 3D meshes (302) may be pre-processed by track remeshing, parameterization and / or voxelization to generate input data to a mapping unit for mapping the 3D mesh into a 2D UV space (304), which may in some implementations include a mesh with a UV atlas. The 3D mesh may be sampled to include 3D surface points that may not be among the vertices, and these sampled 3D surface points may be added to the mapping into the UV space. Various 2D maps may be generated in the encoder 301, including, but not limited to, an occupancy map (310), a geometry map (312), and an attribute map (314). These image type maps may be compressed by the encoder 301 using, for example, video coding / compression techniques. For example, the video coder may help compress the 3D mesh frames using intra prediction techniques and inter prediction with other 3D mesh reference frames. Other non-picture or non-mapped data or metadata (316) may also be coded in various manners to remove redundancy and generate compressed non-mapped data, such as, by way of non-limiting example, via entropy coding. The encoder 301 may then combine or multiplex the compressed 2D map and non-mapped data and further code the combined data to generate an encoded bitstream (also referred to as a coded bitstream). The encoded bitstream may then be stored or transmitted for use by the decoder 303.The decoder may be configured to decode the bitstream, demultiplex the decoded bitstream to obtain the compressed 2D map and non-map data, and perform decompression to generate a decoded occupancy map (320), a decoded geometry map (322), a decoded attribute map (324), and decoded non-map data and metadata (326). The decoder 303 may then be further configured to reconstruct a 3D mesh or a sequence of 3D meshes (330) from the decoded 2D maps (320, 322, and 324) and the decoded non-map data (326).

[0036] More specifically, Figure 4 illustrates a block diagram of an example 3D mesh encoder (400) for encoding a 3D mesh frame, in accordance with some embodiments of the present disclosure. In some example implementations, the mesh encoder (400) may be used in the communication system (100) and the streaming system (200). For example, the encoder (203) may be configured and operate in a manner similar to the mesh encoder (400).

[0037] The mesh encoder (400) may receive a 3D mesh frame as an uncompressed input and generate a bitstream corresponding to the compressed 3D mesh frame. In some example implementations, the mesh encoder (400) may receive the 3D mesh frame from any source, such as the mesh or point cloud source (201) of FIG. 2.

[0038] In the example of FIG. 4, the mesh encoder (400) may include a patch generation module (406) (alternatively referred to as a chart generation module), a patch packing module (408), a geometry image generation module (410), a texture image generation module (412), a patch information module (404), an occupancy map module (414), a smoothing module (436), image padding modules (416) and (418), a group expansion module (420), video compression modules (422), (423), and (432), an auxiliary patch information compression module (438), an entropy compression module (434), and a multiplexer (424).

[0039] In various embodiments of the present disclosure, a module may refer to a software module, a hardware module, or a combination thereof. A software module may include a computer program or a portion of a computer program having a certain function and working with other related parts to achieve a certain goal (such as the functions described in this disclosure). A hardware module may be implemented using a processing circuit and / or memory configured to perform the functions described in this disclosure. Each module may be implemented using one or more processors (or a processor and memory). Similarly, a processor (or a processor and memory) may be used to implement one or more modules. Furthermore, each module may be part of an entire module that includes the functionality of the module. The description herein may also apply to the term module and other equivalent terms (e.g., unit).

[0040] According to aspects of the disclosure, as described above, the mesh encoder (400) converts the 3D mesh frame into an image-based representation (e.g., a 2D map) along with some non-map metadata (e.g., patch and chart information) used to assist in converting the compressed 3D mesh into a decompressed 3D mesh. In some examples, the mesh encoder (400) may convert the 3D mesh frame into a geometry map or image, a texture map or image, and an occupancy map or image, and then encode the geometry image, texture image, and occupancy map along with the metadata and other compressed non-map data into a bitstream using video coding techniques. In general, as described above, a 2D geometry image is a 2D image having 2D pixels that are filled with geometry values ​​associated with 3D points projected (the term "project" is used to mean "map") onto the 2D pixels, where the 2D pixels that are filled with geometry values ​​may be referred to as geometry samples. A texture image is a 2D image having pixels that are filled with texture values ​​associated with 3D points projected onto the 2D pixels, where the 2D pixels that are filled with texture values ​​may be referred to as texture samples. An occupancy map is a 2D image with 2D pixels filled with values ​​that indicate whether they are occupied or unoccupied by a 3D point.

[0041] The patch generation module (406) segments the 3D mesh into a set of charts or patches (e.g., a patch is defined as a contiguous subset of the surface described by the 3D mesh or point cloud), which may or may not overlap, whereby each patch may be described by a depth field relative to a plane in 2D space (e.g., flattening the surface such that 3D points deeper on the surface are further away from the center of the corresponding 2D map). In some embodiments, the patch generation module (406) aims to decompose the 3D mesh into a minimum number of patches with smooth boundaries while minimizing the reconstruction error.

[0042] The patch information module (404) can collect patch information indicating the size and shape of the patch. In some examples, the patch information can be packed into a data frame and then encoded by the auxiliary patch information compression module (438) to generate compressed auxiliary patch information. The auxiliary patch compression can be implemented in a variety of forms, including, but not limited to, various types of arithmetic coding.

[0043] The patch or chart packing module (408) may be configured to map the extracted patches to a 2D grid in UV space while minimizing unused space. In some exemplary implementations, pixels in the 2D UV space may be granularized into blocks of pixels for patch or chart mapping. The block size may be predefined. For example, the block size may be M×M (e.g., 16×16). With such granularity, every M×M block of the 2D UV grid is associated with a unique patch. In other words, each patch is mapped to the 2D UV space with a 2D granularity of 2. Efficient patch packing can directly impact compression efficiency by either minimizing unused space or ensuring temporal consistency. An exemplary implementation of packing patches or charts into 2D UV space is given in more detail below.

[0044] The geometry image generation module (410) can generate a 2D geometry image associated with the geometry of the 3D mesh at a given patch location in the 2D grid. The texture image generation module (412) can generate a 2D texture image associated with the texture of the 3D mesh at a given patch location in the 2D grid. The geometry image generation module (410) and the texture image generation module (412) essentially utilize the 3D to 2D mapping calculated during the packing process described above to store the geometry and texture of the 3D mesh as 2D images, as described above.

[0045] In some implementations, the 2D images may be layered to better handle the case where multiple points project onto the same sample (e.g., patches overlap in the 3D space of the mesh). In other words, each patch may be projected onto two images, called layers, so that, for example, multiple points can project onto the same point in different layers.

[0046] In some exemplary implementations, the geometry image may be represented by a monochromatic frame of width x height (W x H). Thus, three geometry images of three luma or chroma channels may be used to represent the 3D coordinates. In some exemplary implementations, the geometry image may be represented by a 2D image with three channels (RGB, YUV, YCrCb, etc.) at a particular color depth (e.g., 8 bits, 12 bits, 16 bits, etc.). Thus, one geometry image with three color channels may be used to represent the 3D coordinates.

[0047] To generate a texture image, the texture generation procedure utilizes the reconstructed / smoothed geometry to calculate the colors to be associated with points sampled from the original 3D mesh (e.g., see "Sampling" in Figure 3, which generates 3D surface points that are not among the vertices of the original 3D mesh).

[0048] The occupancy map module (414) may be configured to generate an occupancy map that describes the padding information at each unit. For example, as described above, the occupancy image may include a binary map that indicates for each cell of a 2D grid whether the cell belongs to empty space or to a 3D mesh. In some example implementations, the occupancy map may use binary information to describe for each pixel whether the pixel is padded or not. In some other example implementations, the occupancy map may use binary information to describe for each block of pixels (e.g., each M×M block) whether the block of pixels is padded or not.

[0049] The occupancy map generated by the occupancy map module (414) may be compressed using lossless or lossy coding. When lossless coding is used, an entropy compression module (434) may be used to compress the occupancy map. When lossy coding is used, a video compression module (432) may be used to compress the occupancy map.

[0050] Note that the patch packing module (408) may leave some empty space between the 2D patches packed into the image frame. The image padding modules (316) and (318) may fill the empty space (called padding) to generate an image frame that may be suitable for 2D video and image codecs. Image padding is also called background filling, which may fill unused space with redundant information. In some examples, well-performed background filling increases bitrate minimally, but avoids introducing significant coding artifacts near patch boundaries.

[0051] The video compression modules (422), (423), and (432) can encode 2D images, such as padded geometry images, padded texture images, and occupancy maps, based on an appropriate video coding standard, such as HEVC, VVC, etc. In some exemplary implementations, the video compression modules (422), (423), and (432) are individual components that operate separately. Note that in some other exemplary implementations, the video compression modules (422), (423), and (432) can be implemented as a single component.

[0052] In some example implementations, the smoothing module (436) may be configured to generate a smoothed image of the reconstructed geometry image. The smoothed image may be provided to the texture image generation (412). The texture image generation (412) may then adjust the generation of the texture image based on the reconstructed geometry image. For example, when the patch shape (e.g., geometry) is slightly distorted during encoding and decoding, the distortion may be taken into account when generating the texture image to correct the distortion of the patch shape.

[0053] In some embodiments, the group expansion (420) is configured to pad pixels around object boundaries with redundant low-frequency content to improve coding gain as well as the visual quality of the reconstructed 3D mesh.

[0054] A multiplexer (424) may be configured to multiplex the compressed geometry images, the compressed texture images, the compressed occupancy maps, and the compressed auxiliary patch information into a compressed bitstream (also referred to as a coded bitstream).

[0055] 5 illustrates a block diagram of an example mesh decoder (500) for decoding a compressed bitstream corresponding to a 3D mesh frame, according to some embodiments of the present disclosure. In some example implementations, the mesh decoder (500) can be used in the communication system (100) and the streaming system (200). For example, the decoder (210) can be configured to operate in a manner similar to the mesh decoder (500). The mesh decoder (500) receives a compressed bitstream and generates a reconstructed 3D mesh based on the compressed bitstream, including, for example, compressed geometry images, compressed texture images, compressed occupancy maps, and compressed auxiliary patch information.

[0056] In the example of FIG. 5, the mesh decoder (500) includes a demultiplexer (532), video decompression modules (534) and (536), an occupancy map decompression module (538), an auxiliary patch information decompression module (542), a geometry reconstruction module (544), a smoothing module (546), a texture reconstruction module (548), and a color smoothing module (552).

[0057] A demultiplexer (532) may receive the compressed bitstream and separate it into compressed texture images, compressed geometry images, compressed occupancy maps, and compressed auxiliary patch information.

[0058] The video decompression modules (534) and (536) can decode compressed images and output decompressed images according to an appropriate standard (e.g., HEVC, VVC, etc.). For example, the video decompression module (534) may decode compressed texture images and output decompressed texture images. The video decompression module (536) may further decode compressed geometry images and output decompressed geometry images.

[0059] An occupancy map decompression module (538) may decode the compressed occupancy map according to an appropriate standard (e.g., HEVC, VVC, etc.) and output a decompressed occupancy map.

[0060] The auxiliary patch information decompression module (542) may be configured to decode the compressed auxiliary patch information according to a suitable decoding algorithm and output decompressed auxiliary patch information.

[0061] A geometry reconstruction module (544) may be configured to receive the decompressed geometry image and generate a reconstructed 3D mesh geometry based on the decompressed occupancy map and the decompressed auxiliary patch information.

[0062] The smoothing module (546) may be configured to smooth incongruences at the edges of the patches. The smoothing procedure may aim to mitigate potential discontinuities that may occur at patch boundaries due to compression artifacts. In some example implementations, a smoothing filter may be applied to pixels located at patch boundaries to mitigate distortions that may be caused by compression / decompression.

[0063] A texture reconstruction module (548) may be configured to determine texture information for points in the 3D mesh based on the decompressed texture image and the smoothed geometry.

[0064] A color smoothing module (552) may be configured to smooth out coloring discrepancies. Non-adjacent patches in 3D space are often packed next to each other in 2D video. In some instances, pixel values ​​from non-adjacent patches may be confused by block-based video codecs. The purpose of color smoothing may be to reduce visible artifacts that appear at patch boundaries.

[0065] 6 illustrates a block diagram of an exemplary video decoder (610) according to an embodiment of the present disclosure. The video decoder (610) may be used in the mesh decoder (500). For example, the video decompression modules (534) and (536) and the occupancy map decompression module (538) may be configured similarly to the video decoder (610).

[0066] The video decoder (610) may include a parser (620) for reconstructing symbols (621) from a compressed image, such as a coded video sequence. These symbol categories include information used to manage the operation of the video decoder (610). The parser (620) may parse / entropy decode a received coded video sequence. The coding of the coded video sequence may be according to a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (620) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroups may include Groups of Pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (620) may also extract from the coded video sequence information such as transform coefficients, quantization parameter values, motion vectors, and so on.

[0067] The parser (620) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory to generate symbols (621).

[0068] The reconstruction of the symbols (621) can involve several different units, depending on the type of coded video picture or part thereof (e.g., inter and intra picture, inter and intra block) and other factors. Which units are involved and how may be controlled by subgroup control information parsed by the parser (620) from the coded video sequence. The flow of such subgroup control information between the parser (620) and the following units is not shown for clarity.

[0069] In addition to the functional blocks described above, the video decoder (610) can be conceptually subdivided into multiple functional units, as described below. In a practical implementation operating under commercial constraints, many of these units will interact closely with each other and may be at least partially integrated with each other. The following conceptual subdivision into functional units is made solely for purposes of describing the disclosed subject matter.

[0070] The video decoder (610) may include a scalar / inverse transform unit (651), which receives quantized transform coefficients as symbols (621) from the parser (620), as well as control information including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. The scalar / inverse transform unit (651) may output blocks containing sample values ​​that can be input to an aggregator (655).

[0071] In some cases, the output samples of the scaler / inverse transform (651) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (652). In some cases, the intra-picture prediction unit (652) may generate blocks of the same size and shape of the block being reconstructed using surrounding already reconstructed information fetched from the current picture buffer (658). The current picture buffer (558) may buffer, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (655) may add, in some cases, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (651).

[0072] In other cases, the output samples of the scalar / inverse transform unit (651) may relate to a block that is inter-coded and potentially motion compensated. In such cases, the motion compensated prediction unit (653) may access the reference picture memory (657) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (621) associated with the block, these samples may be added by the aggregator (655) to the output of the scalar / inverse transform unit (651) (in this case referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (657) from which the motion compensated prediction unit (653) fetches prediction samples may be controlled by motion vectors available to the motion compensated prediction unit (653), e.g., in the form of symbols (521) that may have X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (657) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0073] The output samples of the aggregator (655) may be subject to various loop filtering techniques in a loop filter unit (656). The video compression techniques are controlled by parameters contained in the coded video sequence (also called a coded video bitstream) and made available to the loop filter unit (656) as symbols (621) from the parser (620), but may include in-loop filter techniques that may be responsive to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, as well as to previously reconstructed loop filtered sample values.

[0074] The output of the loop filter unit (656) may be a sample stream that can be output to a render device or stored in a reference picture memory (657) for use in future inter-picture prediction.

[0075] A particular coded picture, once fully reconstructed, may be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (620)), the current picture buffer (658) may become part of the reference picture memory (657), and a fresh current picture buffer may be reallocated before beginning reconstruction of a subsequent coded picture.

[0076] The video decoder (610) may perform decoding operations according to a given video compression technique in a standard, such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence adheres to both the syntax of the video compression technique or standard and to a profile documented in the video compression technique or standard. In particular, a profile may select a particular tool as the only tool available for use under that profile. Compliance may also require that the complexity of the coded video sequence be within a range defined by a level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further restricted in some cases through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled with the coded video sequence.

[0077] 7 shows a block diagram of a video encoder (703) according to one embodiment of the present disclosure. The video encoder (703) may be used in the mesh encoder (400) to compress a 3D mesh or point cloud. In some example implementations, the video compression modules (422) and (423) and the video compression module (432) are configured similarly to the encoder (703).

[0078] The video encoder (703) may receive 2D images, such as padded geometry images, padded texture images, etc., and generate compressed images.

[0079] According to an exemplary embodiment of the present disclosure, the video encoder (703) may code and compress pictures of a source video sequence into a coded video sequence (compressed images) in real time or under any other time constraint required by the application. Enforcing the appropriate coding rate is one function of the controller (750). In some embodiments, the controller (750) controls and is operatively coupled to other functional units, as described below, which coupling is not shown for clarity. Parameters set by the controller (750) may include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (750) may be configured with other appropriate functions related to the video encoder (703) that are optimized for a particular system design.

[0080] In some example implementations, the video encoder (703) may be configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop may include a source coder (730) (responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and reference pictures, for example) and a (local) decoder (733) embedded in the video encoder (703). The decoder (733) may reconstruct the symbols to create sample data in a manner similar to that created by the (remote) decoder (since any compression between the symbols and the coded video bitstream is lossless with the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) may be input to a reference picture memory (734). Since the decoding of the symbol stream results in bit-accurate results independent of the decoder location (local or remote), the content in the reference picture memory (734) is also bit-accurate between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder "sees" when using the prediction during decoding. This basic principle of reference picture synchronicity (and the resulting drift when synchronicity cannot be maintained, e.g., due to channel error) is used in several related techniques as well.

[0081] The operation of the "local" decoder (733) may be the same as a "remote" decoder, such as the video decoder (610), already described above in connection with Figure 6. However, with brief reference also to Figure 6, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (745) and parser (620) may be lossless, the entropy decoding portion of the video decoder (610), including the parser (620), may not be fully implemented in the local decoder (733).

[0082] In various embodiments of the present disclosure, any decoder technology, except for parsing / entropy decoding, present in a decoder may necessarily be present in a corresponding encoder in substantially the same functional form. For this reason, the subject matter disclosed in various embodiments may focus on the decoder operation. Descriptions of the encoder technology may be omitted, since they are the opposite of the decoder technology, which is described generically. Only in certain areas are more detailed descriptions required and are provided below.

[0083] In some examples, during operation, the source coder (730) may perform motion-compensated predictive coding, which predictively codes an input picture with respect to one or more previously coded pictures from the video sequence designated as “reference pictures.” In this manner, the coding engine (732) may code differences between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture.

[0084] The local video decoder (733) may decode the coded video data of pictures that may be designated as reference pictures based on symbols created by the source coder (730). The operation of the coding engine (732) may advantageously be a lossy process. When the coded video data may be decoded in a video decoder (not shown in FIG. 7), the reconstructed video sequence may be a replica of the source video sequence, typically with some errors. The local video decoder (733) may replicate the decoding process that may be performed by the video decoder on the reference pictures and store the reconstructed reference pictures in a reference picture cache (734). In this way, the video encoder (703) may locally store copies of reconstructed reference pictures that have common content as reconstructed reference pictures obtained by the far-end video decoder (without transmission errors).

[0085] The predictor (735) may perform a prediction search for the coding engine (732). That is, for a new picture to be coded, the predictor (735) may search the reference picture memory (734) for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors, block shapes, etc., which may serve as suitable prediction references for the new picture. The predictor (735) may operate on a sample block-by-pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (735), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (734).

[0086] The controller (750) may manage the coding operations of the source coder (730), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0087] The output of all the above functional units may be subject to entropy coding in an entropy coder (745), which may convert the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0088] A controller (750) may manage the operation of the video encoder (703). During coding, the controller (750) may assign each coded picture a particular coding picture type, which may affect the coding that may be applied to the respective picture. For example, pictures may often be assigned as one of the following picture types:

[0089] An intra picture (I-picture) may be one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art will recognize these variations of I-pictures and their respective uses and characteristics.

[0090] A predictive picture (P picture) may be coded and decoded using intra- or inter-prediction, using at most one motion vector and reference index to predict the sample values ​​of each block.

[0091] Bidirectionally predicted pictures (B-pictures) may be coded and decoded using intra- or inter-prediction, using up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0092] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8 or 16x16 samples each) and coded block by block. Blocks may be predictively coded with respect to other (already coded) blocks as determined by the coding assignment applied to the respective picture of the block. For example, blocks of an I-picture may be non-predictively coded or they may be predictively coded with respect to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P-picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0093] The video encoder (703) may perform coding operations according to a given video coding technique or standard, such as ITU-T Rec. H.265. In doing so, the video encoder (703) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0094] Video may be in the form of multiple source pictures (images) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) uses spatial correlation in a given picture, while inter-picture prediction uses correlation (temporal or other) between pictures. In one example, a particular picture being coded / decoded is called a current picture and is divided into blocks. When a block in a current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture can be coded by a vector called a motion vector. A motion vector points to a reference block in a reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0095] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture, both of which precede the current picture in the video in decoding order (but may be past and future, respectively, in display order). A block in the current picture can be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. A block can be predicted by a combination of the first and second reference blocks.

[0096] Furthermore, to improve coding efficiency, merge mode techniques can be used in inter-picture prediction.

[0097] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed on a block-by-block basis. For example, according to the HEVC standard, a picture in a sequence of video pictures is divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quad-tree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels can be partitioned into one CU of 64×64 pixels, four CUs of 32×32 pixels, or 16 CUs of 16×16 pixels. In one example, a CU is analyzed to determine a prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) depending on temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PU) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, a prediction block includes a matrix of values ​​(e.g., luma values) for pixels, such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0098] In various embodiments, the mesh encoder (400) and mesh decoder (500) described above can be implemented in hardware, software, or a combination thereof. For example, the mesh encoder (400) and mesh decoder (500) can be implemented using processing circuitry, such as one or more integrated circuits (ICs) operating with or without software, such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. In another example, the mesh encoder (400) and mesh decoder (500) can be implemented as software or firmware including instructions stored on a non-volatile (or non-transitory) computer-readable storage medium. The instructions, when executed by a processing circuitry, such as one or more processors, cause the processing circuitry to perform the functions of the mesh encoder (400) and / or mesh decoder (500).

[0099] FIG. 8 illustrates an example of mapping a 3D patch (810) in a 3D xyz coordinate system (819) to a 2D UV plane (850) in a uv coordinate system (859). In some implementations, a 3D patch (or simply a patch) may generally represent a contiguous subset of a surface described by a set of vertices that correspond to a mesh in 3D space. In a non-limiting example, a patch includes vertices with 3D coordinates, normal vectors, color, texture, and other information. In some implementations, a 2D geometry patch (or simply a geometry patch or patch) may represent a projected shape in the 2D UV plane, where the projected shape corresponds to the patch and the geometry patch corresponds to the patch.

[0100] In the projected shape in the UV plane, each mapped point (ui, vi) corresponds to a 3D vertex with position (xi, yi, zii), where i=1, 2, 3, 4, etc. For example, a first vertex (811) with 3D coordinates (x1, y1, z1) is mapped to a first point (851) with 2D coordinates (u1, v1), a second vertex (812) with 3D coordinates (x2, y2, z2) is mapped to a second point (852) with 2D coordinates (u2, v2), a third vertex (813) with 3D coordinates (x3, y3, z3) is mapped to a third point (853) with 2D coordinates (u3, v3), and a fourth vertex (814) with 3D coordinates (x4, y4, z4) is mapped to a fourth point (854) with 2D coordinates (u4, v4).

[0101] The coding of the vertex geometry (i.e., the xi, yi, and zi values) is then converted into a coding of three-channel values ​​in the 2D plane, where each three-channel value at a (u,v) location corresponds to an associated 3D location. For example, the pixel value of a first location (851) includes a three-channel value (or three channel values) that correspond to the 3D coordinates of a first vertex (811), where a first channel value of the three-channel value corresponds to an x1 value, a second channel value of the three-channel value corresponds to a y1 value, and a third channel value of the three-channel value corresponds to a z1 value.

[0102] The projected / mapped 2D plane with one or more patches is therefore called a geometry image and may therefore be coded using any image or video codec, for example a video codec that supports the 4:4:4 color format.

[0103] In some implementations, the pixel value of the projected / mapped point on the UV plane may correspond to the distance from the 3D vertex to the UV plane. Therefore, multiple planes of different orientations may be used for such projections to find the vertex position information. In this case, each position (or called point) in the UV plane is a one-channel value, recording the distance. Such an image is called a depth image. Depth images may be coded using any image or video codec, for example a video codec that supports the YUV4:2:0 or YUV4:0:0 color format.

[0104] Figures 9 and 10 show a bound vertex. A bound vertex may be a single vertex in 3D space, or it may be mapped to more than one point in 2D UV space.

[0105] 9, the border vertices (911, 912, 913, 914, 915 and 916) may be defined in 2D UV space. The border vertices lie on the border edges (e.g. 921 and 922) of the connected components (e.g. patches / charts) and the non-border vertices (e.g. 931) lie inside the connected components (patches / charts).

[0106] In some implementations, a border edge may be determined by checking whether an edge appears in only one triangle; when the edge appears in only one triangle, the edge is a border edge, and when the edge appears in more than two triangles, the edge is not a border edge.

[0107] In some implementations, some or all of the following information of the bounding vertices is important and may be coded and signaled in the bitstream: geometry information, i.e. 3D xyz coordinates, and / or 2D UV coordinates.

[0108] In some implementations, a vertex may be shared by more than one mesh surface polygon, and when a mesh is divided into multiple patches, such a vertex may appear on the boundaries of multiple patches as a common vertex.

[0109] As a non-limiting example, referring to FIG. 10, two adjacent 3D patches (1010 and 1020) may be mapped to a 2D UV plane (1050). In the projected shape in the UV plane, each point (ui, vi) corresponds to a 3D vertex having a position in 3D space represented by 3D coordinates (xi, yi, zi), where i is a positive integer such as 1, 2, 3, 4, etc. The coding of the vertex geometry is then converted to coding of three-channel values ​​in the 2D plane, where each three-channel value at a (u, v) location corresponds to an associated 3D location (x, y, z). Thus, the projected 2D plane with one or more patches is called a geometry image and may thus be coded using any image or video codec, for example a video codec that supports the 4:4:4 color format.

[0110] In particular, a vertex on the boundary of two 3D patches becomes a common vertex shared by both patches. When such a vertex is projected onto a 2D plane, it has more than one UV coordinate. For example, a first vertex (1011) in 3D space may be mapped to a first point (1051) in 2D UV space, a second vertex (1012) in 3D space may be mapped to a second point (1052) and a fifth point (1062) in 2D UV space, a third vertex (1013) in 3D space may be mapped to a third point (1053) and a sixth point (1063) in 2D UV space, and a fourth vertex (1014) in 3D space may be mapped to a fourth point (1054) in 2D UV space. In other words, multiple points on the UV plane with their own boundary geometry values ​​may correspond to the same 3D vertex, and therefore their own boundary geometry values ​​should be equal to have the same geometric information. For example, (1012)(x2,y2,z2) and (1013)(x3,y3,z3) are on the boundary of both patch 1010 and patch 1020. When these two patches are projected onto a 2D plane, for patch 1010, the corresponding UV coordinates for these two boundary vertices are (1052)(u2,v2) and (1053)(u3,v3), respectively, and for patch 1020, the corresponding UV coordinates for these two boundary vertices are (1062)(u2',v2') and (1063)(u3',v3'), respectively. Note that in this case, position (1062)(u'',v2') is different from (1052)(u2,v2), and position (1053)(u3,v3) is different from (1063)(u3',v3').

[0111] Considering that the geometry information of boundary vertices is important for high-quality 3D mesh reconstruction during decoding, some challenges / issues exist in encoding, signaling and decoding the geometry information of boundary vertices in some implementations. When one boundary vertex appears in two patches, it corresponds to two points in the 2D UV plane, and the boundary geometry values ​​of the two points should be the same when no compression is applied. However, after compression is applied to different patches, the geometry information of the same pair of vertices may become different, which may be due to the non-reversible nature of some compression methods, e.g., the quantization process. This may affect the mesh reconstruction after decoding. Typically, the distorted geometry for boundary vertices may cause some cracks to become visible in the reconstructed mesh surface, since the surface is no longer smoothly connected from one patch to another.

[0112] In some implementations, the boundary geometry information may be coded using a lossless or near lossless process. In some implementations, the boundary geometry information may be coded separately from other vertices in each patch. Some of the above implementations may impact the encoding / decoding efficiency of the 3D mesh compression.

[0113] The present disclosure describes various embodiments for encoding or decoding 3D meshes using predictive coding of boundary geometry information to address at least one of the challenges / problems discussed above and therefore improve efficiency in compressing geometry information and / or advance efficient 3D mesh compression techniques.

[0114] FIG. 11 shows a flowchart 1100 of an exemplary method according to the principles underlying the above implementation for decoding a 3D mesh using predictive coding of boundary geometry information. The flow of the exemplary decoding method starts at S1101 and may include some or all of the following steps: At S1110, receive a coded bitstream including a geometry patch for a 3D mesh. At S1120, extract from the coded bitstream a reconstructed geometry image for the geometry patch including a set of points corresponding to a set of geometry coordinates for boundary vertices in the 3D mesh. At S1130, derive 3D geometry coordinates for the boundary vertices based on the set of geometry coordinates. At S1140, obtain prediction residuals for the boundary vertices. At S1150, reconstruct geometry information of the boundary vertices based on the derived 3D geometry coordinates and the prediction residuals. The exemplary method ends at S1199.

[0115] In some implementations, a set of geometric coordinates associated with a set of points may include multiple 3D geometric coordinates (x,y,z), and each point in the set of points includes 2D coordinates (u,v) in the UV plane and a pixel value (e.g., RGB channels) for each point corresponding to each geometric coordinate (x,y,z) in the set of geometric coordinates.

[0116] In some implementations, the coded bitstream includes at least one of a coded geometry map or coded metadata. For example, the coded bitstream may be the compressed bitstream in FIG. 4, which may include one or more compressed geometry images / maps, one or more compressed texture images / maps, one or more compressed occupancy maps, and / or compressed auxiliary patch information. Some coded bitstreams may not have an occupancy map, since the occupancy map information can be inferred from the decoder side when the bounding vertices of each patch are signaled. In some implementations, the geometry patch is one of the patches corresponding to the coded geometry map.

[0117] In some implementations, reconstructing the geometry information of the bounding vertices based on the derived 3D geometry coordinates and the prediction residual includes reconstructing the geometry information of the bounding vertices according to P+Q*QS, where P is the derived 3D geometry coordinate, Q is the prediction residual, and QS is a quantization step size.

[0118] In some implementations, the quantization step size is signaled in a high-level syntax (HLS), including at least one of a sequence header, a frame header, or a slice header.

[0119] In some implementations, extracting from the coded bitstream a reconstructed geometry image for a geometry patch including a set of points corresponding to a set of geometric coordinates for bound vertices in the three-dimensional mesh includes reconstructing a geometry image from the coded bitstream and obtaining a set of geometric coordinates for the bound vertices based on pixel values ​​of the set of points in the reconstructed geometry image.

[0120] In some implementations, deriving 3D geometric coordinates for the bound vertices based on the set of geometric coordinates includes calculating an average of the set of geometric coordinates to obtain averaged 3D geometric coordinates, and determining the averaged 3D geometric coordinates as the derived 3D geometric coordinates for the bound vertices.

[0121] In some implementations, deriving 3D geometric coordinates for the bound vertices based on the set of geometric coordinates includes calculating a weighted average of the set of geometric coordinates to obtain weighted average 3D geometric coordinates, and determining the weighted average 3D geometric coordinates as the derived 3D geometric coordinates for the bound vertices.

[0122] In some implementations, the set of geometric coordinates includes a first geometric coordinate for a first patch and a second geometric coordinate corresponding to a second patch, where the first patch has a greater sampling rate than the second patch, and where for the weighted average, a first weight for the first geometric coordinate is greater than a second weight for the second geometric coordinate.

[0123] In some implementations, the set of geometric coordinates includes a first geometric coordinate for a first patch and a second geometric coordinate corresponding to a second patch, where the first patch has a larger patch size than the second patch, and where for the weighted average, the first weight of the first geometric coordinate is larger than the second weight of the second geometric coordinate.

[0124] In some implementations, the set of geometric coordinates includes a first geometric coordinate for a first patch and a second geometric coordinate corresponding to a second patch, where the first patch has a higher quantization level than the second patch, and where for the weighted average, the first weight of the first geometric coordinate is smaller than the second weight of the second geometric coordinate.

[0125] In some implementations, deriving 3D geometric coordinates for the bound vertices based on the set of geometric coordinates includes establishing a candidate list based on the set of geometric coordinates, obtaining an index from the coded bitstream that corresponds to the candidate list, and determining derived 3D geometric coordinates for the bound vertices based on the index and the candidate list.

[0126] In some implementations, the order of the geometry coordinates in the candidate list is determined according to their decoding order.

[0127] In some implementations, extracting from the coded bitstream a reconstructed geometry image for a geometry patch including a set of points corresponding to a set of geometric coordinates for bounding vertices in the three-dimensional mesh includes reconstructing geometric information of a second bounding vertex, the second bounding vertex being reconstructed before the bounding vertex is reconstructed, and determining the geometric information of the second bounding vertex as a geometric coordinate within the set of geometric coordinates.

[0128] In some implementations, the second bound vertex is an adjacent vertex of the bound vertex.

[0129] In some implementations, another example method for decoding a 3D mesh using predictive coding of boundary geometry information may include some or all of the following steps: Extract from the coded bitstream a plurality of geometry predictions for the boundary vertices, each geometry prediction including a set of predicted geometry coordinates; Determine predicted geometry values ​​for the boundary vertices based on the plurality of geometry predictions; Obtain prediction residuals for the boundary vertices; Reconstruct geometry information for the boundary vertices based on the predicted geometry values ​​and the prediction residuals.

[0130] In some implementations, the coded bitstream includes at least one of a coded geometry map or coded metadata. For example, the coded bitstream may be the compressed bitstream in FIG. 4, which may include one or more compressed geometry images / maps, one or more compressed texture images / maps, one or more compressed occupancy maps, and / or compressed auxiliary patch information. Some coded bitstreams may not have an occupancy map, since the occupancy map information can be inferred from the decoder side when the bounding vertices of each patch are signaled. In some implementations, the geometry patch is one of the patches corresponding to the coded geometry map.

[0131] In some implementations, reconstructing the geometry information of the bound vertices based on the predicted geometry values ​​and the prediction residuals includes reconstructing the geometry information of the bound vertices according to P+Q*QS, where P is the predicted geometry value, Q is the prediction residual, and QS is a quantization step size. In some implementations, the predicted geometry value (P) includes a set of 3D coordinates (x_P, y_P, z_P) and / or the prediction residual (Q) includes a set of 3D coordinates (x_Q, y_Q, z_Q). The quantization step size (QS) may be a single value (QS_common) that is the same for all three 3D coordinates, or may be a set of three values ​​(QS_x, QS_y, QS_z), where each of the three values ​​corresponds to one coordinate in the 3D coordinates. The reconstructed geometry information may be (x_P+x_Q*QS_common, y_P+y_Q*QS_common, z_P+z_Q*QS_common) or (x_P+x_Q*QS_x, y_P+y_Q*QS_y, z_P+z_Q*QS_z).

[0132] In some implementations, the quantization step size is signaled in a high-level syntax (HLS), including at least one of a sequence header, a frame header, or a slice header.

[0133] In some implementations, extracting multiple geometry predictions for the bound vertices from the coded bitstream includes reconstructing a geometry image from the coded bitstream and obtaining multiple geometry predictions for the bound vertices based respectively on multiple pixels in the geometry image that correspond to the bound vertices.

[0134] In some implementations, determining a predicted geometry value for the bound vertex based on the multiple geometry predictions includes calculating an average of the multiple geometry predictions to obtain an average geometry prediction and determining the average geometry prediction as the predicted geometry value for the bound vertex.

[0135] In some implementations, determining a predicted geometry value for the bound vertex based on the multiple geometry predictions includes calculating a weighted average of the multiple geometry predictions to obtain a weighted average geometry prediction, and determining the weighted average geometry prediction as the predicted geometry value for the bound vertex.

[0136] In some implementations, the multiple geometry predictions include a first geometry prediction for a first patch and a second geometry prediction corresponding to a second patch, where the first patch has a greater sampling rate than the second patch, and where for a weighted average, a first weight of the first geometry prediction is greater than a second weight of the second geometry prediction.

[0137] In some implementations, the multiple geometry predictions include a first geometry prediction for a first patch and a second geometry prediction corresponding to a second patch, where the first patch has a larger patch size than the second patch, and where for a weighted average, a first weight of the first geometry prediction is larger than a second weight of the second geometry prediction.

[0138] In some implementations, the multiple geometry predictions include a first geometry prediction for a first patch and a second geometry prediction corresponding to a second patch, where the first patch has a larger quantization level than the second patch, and for a weighted average, a first weight of the first geometry prediction is smaller than a second weight of the second geometry prediction.

[0139] In some implementations, determining a predicted geometry value for a bounding vertex based on a plurality of geometry predictions includes establishing a prediction candidate list based on the plurality of geometry predictions, obtaining an index from the coded bitstream corresponding to the predicted candidate list, and determining a predicted geometry value based on the index and the predicted candidate list.

[0140] In some implementations, the order of the geometry predictions in the prediction candidate list is determined according to the decoding order of the geometry predictions.

[0141] In some implementations, extracting multiple geometry predictions for the bound vertex from the coded bitstream includes reconstructing geometry information of the second bound vertex, the second bound vertex being reconstructed before the bound vertex is reconstructed, and / or determining the geometry information of the second bound vertex as a geometry prediction in the multiple geometry predictions.

[0142] In some implementations, the second bound vertex is an adjacent vertex of the bound vertex.

[0143] The various steps in one or more embodiments or implementations may be applied individually or in any form of combination. Various embodiments in the present disclosure may be applied to dynamic meshes or static meshes, where there may be only one frame of the mesh or the mesh content does not change over time. Various embodiments in the present disclosure may be extended to coding of depth images / attribute images / texture images / etc.

[0144] In some implementations, the geometry information of a bound vertex, including its xyz coordinates (or referred to as bound geometry information), may be predicted from a reconstructed geometry image (given the corresponding uv coordinates) or from previously reconstructed bound information for another bound vertex.

[0145] In some implementations, at the encoder side, the prediction residual of the boundary geometry may be derived by subtracting the original geometry from the prediction, i.e., R i =O i -P i and O i are the original xyz coordinates of the i-th boundary vertex, and P i are the predicted xyz coordinates of the i-th boundary vertex, and R i is the prediction residual of the x, y, and z coordinates of the i-th boundary vertex. The prediction residual may then be quantized and entropy coded.

[0146] In some implementations, the prediction residual may be quantized by a scalar value, i.e., Q i =round(R i / QS), where QS represents the quantization step size. QS may be signaled in a high-level syntax (HLS), for example, in a sequence header, a frame header, a slice header, etc.

[0147] In some implementations, at the decoder side, the reconstructed geometry information is i =P i +Q i *May be derived by QS.

[0148] In some implementations, the prediction of the boundary geometry information may be performed according to at least one of the following example methods.

[0149] In one exemplary method, a prediction of the boundary geometry information may be derived from the reconstructed geometry image, i.e., P i=GIM_REC(u i ,v i ), GIM_REC denotes the reconstructed geometry image, and P i are the predicted geometric xyz coordinates of the i-th vertex boundary, and (u i ,v i ) are the UV coordinates in the geometry image of the i-th bound vertex.

[0150] Considering that one boundary vertex (xyz) may correspond to multiple uv coordinates, one or more of the multiple hypothesis predictions may be multiple P i =GIM_REC(u i ,v i ) may be used to obtain a prediction based on

[0151] In some implementations, the average of multiple predictions may be used as the prediction for the i-th boundary vertex, i.e.,

number

[0152] In some implementations, a prediction candidate list may be established and one item may be selected from the prediction candidate list as a prediction for the boundary vertex. One such predictor vertex may come from one of the multiple predictions. In a non-limiting example, the encoding / decoding order of the multiple predictions may be used to establish the order of the multiple candidates in the list. When more than one predictor is allowed to predict the geometry information of the current vertex, an index is signaled in the bitstream to indicate which one in the list is selected.

[0153] In some implementations, a weighted average of multiple predictions may be used as the prediction for the i-th boundary vertex, i.e.,

number

[0154] In some implementations, the prediction of the boundary geometry information may be derived from previously reconstructed boundary vertices.

[0155] As a non-limiting example, on the encoder side, the geometry information of the boundary vertices may be encoded as being predicted by the geometry information of another boundary vertex that is available and has been encoded when coding the current boundary vertex, that is, P i =Rec j where j < i, i corresponds to the current boundary vertex, j corresponds to the boundary vertex being coded, and is available when coding the current boundary vertex. On the decoder side, the geometry information of the boundary vertices may be decoded as being predicted by the geometry information of another boundary vertex that is available and has been decoded (or reconstructed) when decoding the current boundary vertex, that is, P i =Rec j where j < i, i corresponds to the current boundary vertex, and j corresponds to the reconstructed boundary vertex.

[0156] In another non-limiting example, the geometry information of the boundary vertices may be predicted based on its adjacent boundary vertices of the same patch. In 3D space, a set of boundary vertices forms the boundary of a patch. Among these vertices, two adjacent vertices are close to each other in position, so their geometry information is also close to each other, resulting in high prediction efficiency.

[0157] In another non-limiting example, the geometry information of a bound vertex may be predicted by the geometry information of multiple bound vertices that are coded and available when coding the geometry information of the current bound vertex. When multiple corresponding bound vertices are available to predict the current bound vertex (through multiple UV coordinates), the following prediction mechanism may be applied.

[0158] In one prediction mechanism, a prediction candidate list may be established. One such predictor vertex may come from one previously coded patch. In a non-limiting example, the encoding / decoding order of the patch may be used to establish the order of the candidates in the list. When more than one predictor is allowed to predict the geometry information of the current vertex, an index is signaled in the bitstream to indicate which one in the list is selected.

[0159] In another prediction mechanism, an average or weighted average of multiple candidates (if applicable) may be generated to predict the current boundary vertex. The weighting parameters for each candidate may be determined according to one or more factors. In a non-limiting example, the weighting factor may be determined based on the quantization level (such as QS) at which the candidate predictor is coded. For example, a heavily quantized candidate corresponding to a larger quantization step size (QS) may correspond to a smaller weight.

[0160] The techniques disclosed in this disclosure may be used separately or in combination in any order. Furthermore, each of the techniques (e.g., methods, embodiments), the encoder, and the decoder may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In some examples, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0161] The techniques described above may be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 12 illustrates a computer system (1300) suitable for implementing certain embodiments of the disclosed subject matter.

[0162] Computer software may be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to create code that includes instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., either directly or through interpretation, microcode execution, etc.

[0163] The instructions may be executed in various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0164] 12 for computer system (1300) are exemplary in nature and are not intended to suggest any limitations as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (1300).

[0165] The computer system (1300) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, through tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), and olfactory input (not shown). Human interface input devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (voice, music, ambient sounds, etc.), images (scanned images, photographic images obtained from a still image camera, etc.), and video (2D video, 3D video including stereoscopic vision, etc.).

[0166] The human interface input devices may include one or more of a keyboard (1301), a mouse (1302), a trackpad (1303), a touch screen (1310), a data glove (not shown), a joystick (1305), a microphone (1306), a scanner (1307), and a camera (1308) (only one of each is shown).

[0167] The computer system (1300) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1310), data gloves (not shown), or joystick (1305), although there may be haptic feedback devices that do not function as input devices), audio output devices (speakers (1309), headphones (not shown), etc.), visual output devices (e.g., screens (1310), including CRT, LCD, plasma, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may output two-dimensional visual output or greater than three-dimensional output through such means as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0168] The computer system (1300) may also include human-accessible storage devices and their associated media, such as optical media or similar media (1321), including CD / DVD ROM / RW (1320) with CDs / DVDs, thumb drives (1322), removable hard drives or solid state drives (133), legacy magnetic media such as tapes and floppy disks (not shown), and special ROM / ASIC / PLD based devices such as security dongles (not shown).

[0169] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0170] The computer system (1300) may also include interfaces (1354) to one or more communication networks (1355). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, and the like. Examples of networks include local area networks such as Ethernet, WLAN, and the like, cellular networks including GSM, 3G, 4G, 5G, LTE, and the like, TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial TV, vehicular and industrial networks including CANBus, and the like. Certain networks typically require an external network interface adapter (e.g., a USB port on the computer system (1300)) attached to a particular general-purpose data port or peripheral bus (1349), while others are typically integrated into the core of the computer system (1300) by attachment to a system bus (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system) as described below. Using any of these networks, the computer system (1300) can communicate with other entities. Such communications can be one-way receive-only (e.g., broadcast TV), one-way transmit-only (e.g., from a particular CANbus to a particular CANbus device), or bidirectional to other computer systems, for example, using local or wide area digital networks. As noted above, specific protocols and protocol stacks can be used in each of these networks and network interfaces.

[0171] The above human interface devices, human accessible storage devices and network interfaces may be attached to the core (1340) of the computer system (1300).

[0172] The core (1340) may include one or more central processing units (CPUs) (1341), graphics processing units (GPUs) (1342), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (1343), hardware accelerators for specific tasks (1344), graphics adapters (1350), etc. These devices may be connected through a system bus (1348), along with read only memory (ROM) (1345), random access memory (RAM) (1346), and internal mass storage (1347), such as an internal non-user accessible hard drive, SSD, etc. In some computer systems, the system bus (1348) is accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1348) or through a peripheral bus (1349). In one example, a screen (1310) may be connected to the graphics adapter (1350). Peripheral bus architectures include PCI, USB, and the like.

[0173] The CPU (1341), GPU (1342), FPGA (1343) and accelerator (1344) can execute certain instructions, which in combination can constitute the above-mentioned computer code. The computer code can be stored in ROM (1345) or RAM (1346). Also, temporary data can be stored in RAM (1346) while permanent data can be stored, for example, in internal mass storage (1347). Rapid storage and retrieval from any of the memory devices can be enabled through the use of cache memory, which can be closely associated with one or more of the CPU (1341), GPU (1342), mass storage (1347), ROM (1345), RAM (1346), etc.

[0174] The computer-readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0175] By way of example and not limitation, a computer system having the architecture (1300) and in particular the cores (1340) can provide functionality as a result of the processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as introduced above, as well as media associated with the specific storage of the cores (1340) of a non-transitory nature, such as the core internal mass storage (1347) or ROM (1345). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the cores (1340). The computer-readable media can include one or more memory devices or chips according to particular needs. The software can cause the cores (1340) and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.) to execute particular processes or particular parts of particular processes described herein, including defining data structures stored in RAM (1346) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (1344)) that may operate in place of or in conjunction with software to perform particular processes, or particular portions of particular processes, described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry embodying logic for execution, or both, as appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0176] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents which are within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise various systems and methods which, although not explicitly shown or described herein, embody the principles of this disclosure and are therefore within the spirit and scope of this disclosure.

Claims

1. 1. A method for decoding a geometry patch for a three-dimensional mesh, comprising: receiving, by a device including a memory storing instructions and a processor in communication with the memory, a coded bitstream including geometry patches for a three-dimensional mesh; extracting, by the device, from the coded bitstream a reconstructed geometry image for the geometry patch that includes a set of points corresponding to a set of geometry coordinates for bounding vertices in the 3D mesh; deriving, by the device, 3D geometric coordinates for the bounding vertices based on the set of geometric coordinates; obtaining, by the device, a prediction residual for the boundary vertex; reconstructing, by the device, geometry information of the bounding vertices based on the derived 3D geometry coordinates and the prediction residuals. Including, deriving the 3D geometric coordinates for the bounding vertices based on the set of geometric coordinates, comprising: calculating a weighted average of the set of geometry coordinates to obtain a weighted average 3D geometry coordinate; determining the weighted average 3D geometric coordinates as the derived 3D geometric coordinates for the bounding vertices; Including, the set of geometric coordinates includes first geometric coordinates for a first patch and second geometric coordinates corresponding to a second patch; the first patch has a greater sampling rate than the second patch; The method, wherein for the weighted average, a first weight for the first geometric coordinate is greater than a second weight for the second geometric coordinate.

2. The coded bitstream comprises: Coded Geometry Map or Coded Metadata The method of claim 1 , comprising at least one of:

3. reconstructing the geometry information of the bounding vertices based on the derived 3D geometry coordinates and the prediction residuals, 2. The method of claim 1 , comprising: reconstructing the geometry information of the bounding vertices according to P+Q*QS, where P is the derived 3D geometry coordinate, Q is the prediction residual, and QS is a quantization step size.

4. The method of claim 3 , wherein the quantization step size is signaled in a high level syntax (HLS) including at least one of a sequence header, a frame header, or a slice header.

5. extracting from the coded bitstream the reconstructed geometry image for the geometry patch including the set of points corresponding to the set of geometry coordinates for the bounding vertices in the 3D mesh, comprising: reconstructing a geometry image from the coded bitstream; obtaining the set of geometric coordinates for the bounding vertices based on pixel values ​​of the set of points in the reconstructed geometric image; The method of claim 1 , comprising:

6. A method for decoding a geometry patch for a three-dimensional mesh, comprising: receiving, by a device including a memory storing instructions and a processor in communication with the memory, a coded bitstream including geometry patches for a three-dimensional mesh; extracting, by the device, from the coded bitstream a reconstructed geometry image for the geometry patch that includes a set of points corresponding to a set of geometry coordinates for bounding vertices in the 3D mesh; deriving, by the device, 3D geometric coordinates for the bounding vertices based on the set of geometric coordinates; obtaining, by the device, a prediction residual for the boundary vertex; reconstructing, by the device, geometry information of the bounding vertices based on the derived 3D geometry coordinates and the prediction residuals. Including, deriving the 3D geometric coordinates for the bounding vertices based on the set of geometric coordinates, comprising: calculating a weighted average of the set of geometry coordinates to obtain a weighted average 3D geometry coordinate; determining the weighted average 3D geometric coordinates as the derived 3D geometric coordinates for the bounding vertices; Including, the set of geometric coordinates includes first geometric coordinates for a first patch and second geometric coordinates corresponding to a second patch; The first patch has a larger patch size than the second patch; The method, wherein for the weighted average, a first weight for the first geometric coordinate is greater than a second weight for the second geometric coordinate.

7. A method for decoding a geometry patch for a three-dimensional mesh, comprising: receiving, by a device including a memory storing instructions and a processor in communication with the memory, a coded bitstream including geometry patches for a three-dimensional mesh; extracting, by the device, from the coded bitstream a reconstructed geometry image for the geometry patch that includes a set of points corresponding to a set of geometry coordinates for bounding vertices in the 3D mesh; deriving, by the device, 3D geometric coordinates for the bounding vertices based on the set of geometric coordinates; obtaining, by the device, a prediction residual for the boundary vertex; reconstructing, by the device, geometry information of the bounding vertices based on the derived 3D geometry coordinates and the prediction residuals. Including, deriving the 3D geometric coordinates for the bounding vertices based on the set of geometric coordinates, comprising: calculating a weighted average of the set of geometry coordinates to obtain a weighted average 3D geometry coordinate; determining the weighted average 3D geometric coordinates as the derived 3D geometric coordinates for the bounding vertices; Including, the set of geometric coordinates includes first geometric coordinates for a first patch and second geometric coordinates corresponding to a second patch; the first patch has a greater quantization level than the second patch; The method, wherein for the weighted average, a first weight for the first geometric coordinate is less than a second weight for the second geometric coordinate.

8. extracting from the coded bitstream the reconstructed geometry image for the geometry patch including the set of points corresponding to the set of geometry coordinates for the bounding vertices in the 3D mesh, comprising: reconstructing geometry information of a second bounding vertex, the second bounding vertex being reconstructed before the bounding vertex is reconstructed; and determining the geometry information of the second bounding vertex as geometry coordinates within the set of geometry coordinates; The method of claim 1 , comprising:

9. The method of claim 8 , wherein the second bounding vertex is an adjacent vertex of the bounding vertex.

10. 1. An apparatus for decoding a geometry patch for a three-dimensional mesh, comprising: A memory for storing instructions; a processor in communication with the memory; 10. An apparatus comprising: a processor configured to execute the instructions, the processor being configured to cause the apparatus to perform a method according to any one of claims 1 to 9.

11. A computer program comprising instructions, A computer program product, the instructions being configured, when executed by a processor, to cause the processor to perform a method according to any one of claims 1 to 9.

12. 1. A method for encoding a geometry patch for a three-dimensional mesh, comprising the steps of: transmitting, by a device including a memory storing instructions and a processor in communication with the memory, a coded bitstream including geometry patches for a three-dimensional mesh; the coded bitstream includes a reconstructed geometry image for the geometry patch that includes a set of points corresponding to a set of geometry coordinates for bounding vertices in the 3D mesh; deriving 3D geometric coordinates for the bounding vertices by computing a weighted average of the set of geometric coordinates and determining the weighted average geometric coordinates as the 3D geometric coordinates for the bounding vertices; and reconstructing, by the device, geometry information of the bounding vertices based on the derived 3D geometry coordinates and prediction residuals for the bounding vertices; the set of geometric coordinates includes first geometric coordinates for a first patch and second geometric coordinates corresponding to a second patch; the first patch has a greater sampling rate than the second patch; The method, wherein for the weighted average, a first weight for the first geometric coordinate is greater than a second weight for the second geometric coordinate.

13. A method for encoding a geometry patch for a three-dimensional mesh, comprising the steps of: transmitting, by a device including a memory storing instructions and a processor in communication with the memory, a coded bitstream including geometry patches for a three-dimensional mesh; the coded bitstream includes a reconstructed geometry image for the geometry patch that includes a set of points corresponding to a set of geometry coordinates for bounding vertices in the 3D mesh; deriving 3D geometric coordinates for the bounding vertices by computing a weighted average of the set of geometric coordinates and determining the weighted average geometric coordinates as the 3D geometric coordinates for the bounding vertices; and reconstructing, by the device, geometry information of the bounding vertices based on the derived 3D geometry coordinates and prediction residuals for the bounding vertices; the set of geometric coordinates includes first geometric coordinates for a first patch and second geometric coordinates corresponding to a second patch; The first patch has a larger patch size than the second patch; The method, wherein for the weighted average, a first weight for the first geometric coordinate is greater than a second weight for the second geometric coordinate.

14. A method for encoding a geometry patch for a three-dimensional mesh, comprising the steps of: transmitting, by a device including a memory storing instructions and a processor in communication with the memory, a coded bitstream including geometry patches for a three-dimensional mesh; the coded bitstream includes a reconstructed geometry image for the geometry patch that includes a set of points corresponding to a set of geometry coordinates for bounding vertices in the 3D mesh; deriving 3D geometric coordinates for the bounding vertices by computing a weighted average of the set of geometric coordinates and determining the weighted average geometric coordinates as the 3D geometric coordinates for the bounding vertices; and reconstructing, by the device, geometry information of the bounding vertices based on the derived 3D geometry coordinates and prediction residuals for the bounding vertices; the set of geometric coordinates includes first geometric coordinates for a first patch and second geometric coordinates corresponding to a second patch; the first patch has a greater quantization level than the second patch; The method, wherein for the weighted average, a first weight for the first geometric coordinate is less than a second weight for the second geometric coordinate.

Images