Base graph based mesh compression

Abstract

In one implementation, we present a first base-graph based framework that enables a proper disentanglement of mesh geometry and mesh connectivity for the purpose of mesh compression. In particular, we propose an end-to-end learning-based mesh compression network framework WrappingNet, which is able to operate on meshes of different sizes and connectivity while producing useful fixed-length latent representations. Furthermore, when best reconstruction performance is desired (at cost of lower compression) rather than fixed-length codeword compression summarizing the global topology of the mesh, our proposed encoder and decoder modules can be adapted to produce and digest (respectively) a latent feature map living on a low-resolution base mesh.

Description

BASE GRAPH BASED MESH COMPRESSIONTECHNICAL FIELD[1] The present embodiments generally relate to a method and an apparatus for 3D mesh compression.BACKGROUND[2] The Point Cloud (PC) data format is a universal data format across several business domains, e.g., from autonomous driving, robotics, augmented reality Virtual reality (AR / VR), civil engineering, computer graphics, to the animation / movie industry. 3D LiDAR (Light Detection and Ranging) sensors have been deployed in self-driving cars, and affordable LiDAR sensors are released from Velodyne Velabit, Apple iPad Pro 2020 and Intel RealSense LiDAR camera L515. With advances in sensing technologies, 3D point cloud data becomes more practical than ever and is expected to be an ultimate enabler in the applications discussed herein.SUMMARY[3] According to an embodiment, a method for decoding 3D mesh is presented, comprising: obtaining a template grid; obtaining a base graph containing connectivity information of said 3D mesh; forming a first 3D mesh based on said base graph and geometry of said template grid; obtaining a codeword representing geometry information of said 3D mesh; and reconstructing said 3D mesh from said first 3D mesh and said codeword using a learning-based module.[4] According to another embodiment, a method for decoding 3D mesh is presented, comprising: obtaining a base mesh associated with said 3D mesh; obtaining a codeword representing geometry information of said 3D mesh; and reconstructing said 3D mesh using a learning-based module, based on said base mesh and said codeword.[5] According to another embodiment, a method for encoding 3D mesh is presented, comprising: obtaining a base mesh corresponding to said 3D mesh and face features associated with said base mesh; generating a fixed-length codeword from said face features associated with said base mesh, using a first learning-based module; forming a base graph indicating connectivity of said base mesh, based on said base mesh and a template grid, using a second learning-based module; and encoding said fixed-length codeword and said based graph. i[6] According to another embodiment, a method for encoding 3D mesh is presented, comprising: obtaining a base mesh corresponding to said 3D mesh and face features associated with said base mesh; and encoding said face features and said based mesh.[7] According to another embodiment, an apparatus for decoding 3D mesh is presented, comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to obtain a template grid; obtain a base graph containing connectivity information of said 3D mesh; form a first 3D mesh based on said base graph and geometry of said template grid; obtain a codeword representing geometry information of said 3D mesh; and reconstruct said 3D mesh from said first 3D mesh and said codeword using a learning-based module.[8] According to another embodiment, an apparatus for decoding 3D mesh is presented, comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to obtain a base mesh associated with said 3D mesh; obtain a codeword representing geometry information of said 3D mesh; and reconstruct said 3D mesh using a learning-based module, based on said base mesh and said codeword.[9] According to another embodiment, an apparatus for encoding 3D mesh is presented, comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to obtain a base mesh corresponding to said 3D mesh and face features associated with said base mesh; generate a fixed-length codeword from said face features associated with said base mesh, using a first learning-based module; form a base graph indicating connectivity of said base mesh, based on said base mesh and a template grid, using a second learning-based module; and encode said fixed-length codeword and said based graph.

[0010] According to another embodiment, an apparatus for encoding 3D mesh is presented, comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to obtain a base mesh corresponding to said 3D mesh and face features associated with said base mesh; and encode said face features and said based mesh.

[0011] One or more embodiments also provide a computer program comprising instructions which when executed by one or more processors cause the one or more processors to perform the encoding method or decoding method according to any of the embodiments described herein. One or more of the present embodiments also provide a computer readable storage medium having stored thereon instructions for encoding or decoding point cloud data according to the methods described herein.

[0012] One or more embodiments also provide a computer readable storage medium having stored thereon point cloud data generated according to the methods described above. One or more embodiments also provide a method and apparatus for transmitting or receiving the point cloud data generated according to the methods described herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] FIG. 1 illustrates a block diagram of a system within which aspects of the present embodiments may be implemented.

[0014] FIG. 2 illustrate a proposed fixed-length WrappingNet mesh compression network, according to an embodiment.

[0015] FIG. 3 illustrates a feature map WrappingNet mesh compression network, according to an embodiment.

[0016] FIG. 4A illustrates the UnWrapping module, and FIG. 4B illustrates the Enhanced Wrapping module, according to an embodiment.

[0017] FIG. 5A illustrates a feature extractor, FIG. 5B illustrates DownFaceConv, and FIG. 5C illustrates UpFaceConv, according to an embodiment.

[0018] FIG. 6 illsutrates Face2Node, according to an embodiment.

[0019] FIG. 7 illustrates a proposed GNN based fixed-length compression network, according to an embodiment.

[0020] FIG. 8 illustrates fixed-length coding of the base graph, according to an embodiment.

[0021] FIG. 9 illustrates a block diagram of a ResFaceConv, according to an embodiment.

[0022] FIG. 10 illustrates a block diagram of an Inception-ResFaceConv block, according to anembodiment.

[0023] FIG. 11 illustrates a block diagram of partition-based encoding, according to an embodiment.

[0024] FIG. 12 illustrates a block diagram of partition-based decoding, according to an embodiment.

[0025] FIG. 13 illustrates a mesh classification based on proposed fixed-length compression network, according to an embodiment.

[0026] FIG. 14 illustrates a diagram of a learning-based mesh compression system, according to an embodiment.

[0027] FIG. 15 illustrates a diagram of a learning-based mesh compression system using base graph, according to an embodiment.DETAILED DESCRIPTION

[0028] FIG. 1 illustrates a block diagram of an example of a system in which various aspects and embodiments can be implemented. System 100 may be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this application. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 100, singly or in combination, may be embodied in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, the system 100 is communicatively coupled to other systems, or to other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various embodiments, the system 100 is configured to implement one or more of the aspects described in this application.

[0029] The system 100 includes at least one processor 110 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this application. Processor 110 may include embedded memory, input output interface, and various other circuitries as known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memorydevice, and / or a non-volatile memory device). System 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drive, and / or optical disk drive. The storage device 140 may include an internal storage device, an attached storage device, and / or a network accessible storage device, as non-limiting examples.

[0030] System 100 includes an encoder / decoder module 130 configured, for example, to process data to provide an encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents module(s) that may be included in a device to perform the encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. Additionally, encoder / decoder module 130 may be implemented as a separate element of system 100 or may be incorporated within processor 110 as a combination of hardware and software as known to those skilled in the art.

[0031] Program code to be loaded onto processor 110 or encoder / decoder 130 to perform the various aspects described in this application may be stored in storage device 140 and subsequently loaded onto memory 120 for execution by processor 110. In accordance with various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more of various items during the performance of the processes described in this application. Such stored items may include, but are not limited to, the input video, the decoded video or portions of the decoded video, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0032] In several embodiments, memory inside of the processor 110 and / or the encoder / decoder module 130 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other embodiments, however, a memory external to the processing device (for example, the processing device may be either the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be the memory 120 and / or the storage device 140, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for video codingand decoding operations, such as for MPEG-2, JPEG Pleno, MPEG-I, V-DMC, HEVC, or VVC.

[0033] The input to the elements of system 100 may be provided through various input devices as indicated in block 105. Such input devices include, but are not limited to, (i) an RF portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0034] In various embodiments, the input devices of block 105 have associated respective input processing elements as known in the art. For example, the RF portion may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) down converting the selected signal, (iii) bandlimiting again to a narrower band of frequencies to select (for example) a signal frequency band which may be referred to as a channel in certain embodiments, (iv) demodulating the down converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, bandlimiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion may include a tuner that performs various of these functions, including, for example, down converting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, down converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements may include inserting elements in between existing elements, for example, inserting amplifiers and an analog- to-digital converter. In various embodiments, the RF portion includes an antenna.

[0035] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 100 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within processor 110 as necessary. Similarly, aspects of USB or HDMI interface processing may beimplemented within separate interface ICs or within processor 110 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 110, and encoder / decoder 130 operating in combination with the memory and storage elements to process the datastream as necessary for presentation on an output device.

[0036] Various elements of system 100 may be provided within an integrated housing, Within the integrated housing, the various elements may be interconnected and transmit data therebetween using suitable connection arrangement 115, for example, an internal bus as known in the art, including the 12C bus, wiring, and printed circuit boards.

[0037] The system 100 includes communication interface 150 that enables communication with other devices via communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 190. The communication interface 150 may include, but is not limited to, a modem or network card and the communication channel 190 may be implemented, for example, within a wired and / or a wireless medium.

[0038] Data is streamed to the system 100, in various embodiments, using a Wi-Fi network such as IEEE 802. 11. The Wi-Fi signal of these embodiments is received over the communications channel 190 and the communications interface 150 which are adapted for Wi-Fi communications. The communications channel 190 of these embodiments is typically connected to an access point or router that provides access to outside networks including the Internet for allowing streaming applications and other over-the-top communications. Other embodiments provide streamed data to the system 100 using a set-top box that delivers the data over the HDMI connection of the input block 105. Still other embodiments provide streamed data to the system 100 using the RF connection of the input block 105.

[0039] The system 100 may provide an output signal to various output devices, including a display 165, speakers 175, and other peripheral devices 185. The other peripheral devices 185 include, in various examples of embodiments, one or more of a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide a function based on the output of the system 100. In various embodiments, control signals are communicated between the system 100 and the display 165, speakers 175, or other peripheral devices 185 using signaling such as AV. Link, CEC, or other communications protocols that enable device-to-device control with orwithout user intervention. The output devices may be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, the output devices may be connected to system 100 using the communications channel 190 via the communications interface 150. The display 165 and speakers 175 may be integrated in a single unit with the other components of system 100 in an electronic device, for example, a television. In various embodiments, the display interface 160 includes a display driver, for example, a timing controller (T Con) chip.[40J The display 165 and speaker 175 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments in which the display 165 and speakers 175 are external components, the output signal may be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0041] It is contemplated that point cloud data may consume a large portion of network traffic, e.g., among connected cars over 5G network, and immersive communications (VR / AR). Point cloud understanding and communication would essentially lead to efficient representation formats. In particular, raw point cloud data need to be properly organized and processed for the purposes of world modeling and sensing.

[0042] Furthermore, point clouds may represent a sequential scan of the same scene, which contains multiple moving objects. They are called dynamic point clouds as compared to static point clouds captured from a static scene or static objects. Dynamic point clouds are typically organized into frames, with different frames being captured at different times.

[0043] The automotive industry and autonomous car are domains in which point clouds may be used. Autonomous cars should be able to “probe” their environment to make good driving decisions based on the reality of their immediate surroundings. Typical sensors like LiDARs produce (dynamic) point clouds that are used by the perception engine. These point clouds are not intended to be viewed by human eyes and they are typically sparse, not necessarily colored, and dynamic with a high frequency of capture. They may have other attributes like the reflectance ratio provided by the LiDAR as this attribute is indicative of the material of the sensed object and may help in making a decision.

[0044] Virtual Reality (VR) and immersive worlds are foreseen by many as the future of 2D flat video. For VR and immersive worlds, a viewer is immersed in an environment all around the viewer, as opposed to standard TV where the viewer can only look at the virtual world in front of the viewer. There are several gradations in the immersivity depending on the freedom of the viewer in the environment. Point cloud is a good format candidate to distribute VR worlds. The point cloud for use in VR may be static or dynamic and are typically of average size, for example, no more than millions of points at a time.

[0045] Point clouds may also be used for various purposes such as culture heritage / buildings in which objects like statues or buildings are scanned in 3D in order to share the spatial configuration of the object without sending or visiting the object. Also, point clouds may also be used to ensure preservation of the knowledge of the object in case the object may be destroyed, for instance, a temple by an earthquake. Such point clouds are typically static, colored, and huge.

[0046] Another use case is in topography and cartography in which using 3D representations, maps are not limited to the plane and may include the relief. Google Maps is a good example of 3D maps but uses meshes instead of point clouds. Nevertheless, point clouds may be a suitable data format for 3D maps and such point clouds are typically static, colored, and huge.

[0047] World modeling and sensing via point clouds could be a useful technology to allow machines to gain knowledge about the 3D world around them for the applications discussed herein.

[0048] 3D point cloud data are essentially discrete samples on the surfaces of objects or scenes. To fully represent the real world with point samples, in practice it requires a huge number of points. For instance, a typical VR immersive scene contains millions of points, while point clouds typically contain hundreds of millions of points. Therefore, the processing of such large-scale point clouds is computationally expensive, especially for consumer devices, e.g., smartphone, tablet, and automotive navigation system, that have limited computational power. Additionally, the discrete samples comprising the 3D point cloud data still contain incomplete information about the underlying surfaces of objects and scenes. Hence, recently efforts are being made to also explore mesh representation for 3D scene / surface representation. Meshes can be considered as a 3D point cloud along with the connectivity information between the points. Thus, the mesh representation bridges the gap between point clouds and the underlying continuous surfaces through local 2D polygonal patches (called face) approximation of the underlying surface.

[0049] The first step for any kind of processing or inference on the mesh data is to have efficient storage methodologies. To store and process the input mesh with affordable computational cost, one solution is to down-sample it first, where the down-sampled mesh summarizes the geometry of the input point cloud while having much fewer (but bigger) faces. The down-sampled mesh is then fed to the subsequent machine task for further consumption. However, further reduction in storage space can be achieved by converting the raw mesh data (original or downsampled) into a fixed-length codeword or a feature map living on a very low-resolution mesh. This codeword or the feature map can be converted to a bitstream through existing entropy coding techniques. Moreover, the codeword or feature map is also useful by itself as it represents global or local surface information (respectively) of the underlying scene / object and can be paired with subsequent downstream (machine vision) tasks.

[0050] Various attempts to design compression networks on meshes have been made in recent years. The first attempts can be traced back to the compression network described in an article by Litany, Or, et al., (entitled “Deformable shape completion with graph convolutional compression network,” Proceedings of the IEEE conference on computer vision and pattern recognition, 2018) and CoMA (Ranjan, Anurag, et al., “Generating 3D faces using Convolutional Mesh Compression networks,” European Conference on Computer Vision (ECCV), 2018). The former work treats the mesh purely as a graph and applies a variational graph compression network, using the mesh geometry as input features. This method does not have hierarchical pooling and does not apply any mesh-specific operations. The latter work defines fixed up- and down- sampling operations in a hierarchical fashion, based on quadric error simplification, combined with spectral convolution layers, which must operate on meshes of the same size and connectivity. This is because the pooling and unpooling operations are predefined and dependent on the connectivity represented as an adjacency matrix.

[0051] Follow-up works improve the convolution layers but are still limited to the fixed size and connectivity constraint. A separate work of note is MeshCNN, which defines learnable up and downsampling modules that can adapt to different meshes of variable size. However, these layers have not been demonstrated to construct a good compression network, but rather for mesh classification and segmentation.

[0052] Another line of works has investigated subdivision-based mesh processing where theoriginal mesh is converted into a new mesh that well approximates the original mesh but exhibits subdivision connectivity, i.e., a semi-regular mesh. Broadly speaking, a semi-regular mesh is imbued with a hierarchical face structure where every face has three neighboring faces (corresponding to its three edges), and a face and its three neighbors can be combined to form a single face. This property makes the semi-regular mesh amenable to fixed up- and down- sampling operations, which is a cornerstone of convolution-based architectures.

[0053] Notable works that define learning-based modules on subdivision meshes include a work by Hu, Shi-Min, et al., entitled “Subdivision-based mesh convolution networks,” ACM Transactions on Graphics (TOG) 41.3 (2022): 1-16, which is a general framework that defines face-based convolution layers (treating faces almost like pixels in images), and a work by Liu, Hsueh-Ti Derek, et al., entitled “Neural subdivision,” arXiv preprint arXiv:2005.01819 (2020), which implements coarse-to-fine mesh super-resolution capabilities. For compression networks, only one work (Hahner, Sara, et al., “Mesh Convolutional Compression network for semi-regular meshes of different sizes,” Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision, 2022) has attempted to implement a compression network using subdivision meshes, achieving autoencoding capabilities on meshes of different sizes. However, their method is unable to generate fixed-length latent representations from different sized meshes and is only able to generate latent feature maps on the base mesh which can only be compared across meshes with the same base mesh connectivity and face ordering. This precludes meaningful latent space comparisons across heterogeneous meshes, which may differ in size, connectivity, or ordering. Having a fix-length latent representation is preferable because it is critical to sub-sequent analysis / understanding about the input mesh geometry.

[0054] This document is aimed at heterogeneous semi-regular meshes and how an efficient compression architecture can be designed for these heterogeneous meshes.

[0055] In image compression network systems, the encoder and decoder typically alternate convolution and up / down sampling operations. Due to the fixed grid support of the images, these down- and up- sampling layers can be easily defined with a fixed ratio (e.g., 2x pooling). Moreover, since images can be easily resized to the same size via interpolation techniques, one can define hard-coded layer sizes that will always map images to a fixed-size latent representation and back to the original image size. In contrast, triangle mesh data, which are composed of geometry (listof points) and connectivity (list of triangles with indexing corresponding to the points) are variable in size and have highly irregular support. This prevents easily defining convolution neighborhood structure, up- and down- sampling structure, as well as extracting fixed-length latent representations from variable size meshes. While previous mesh compression networks have attempted to resolve some of these issues, there is no compression network method that can process heterogeneous meshes and extract meaningful fixed-length latent representations that generalize across meshes of different sizes and connectivity in a fashion similar to image compression networks.

[0056] One could also draw comparisons with compression networks for point cloud data, since point clouds also suffer from having irregular structure and variable size. While meshes have included connectivity information which carries more topological information about the underlying surface compared to point clouds, the connectivity can also bring additional challenges. State-of-the-art point cloud compression networks such as FoldingNet and TearingNet which are able to extract fixed-length latent representation on point clouds of different sizes, reconstruct a point cloud in some canonical ordering which may not be the same as the original ordering of the input point cloud. This prevents mesh reconstruction with the original connectivity since connectivity may no longer be aligned with the output point ordering. It is also not obvious how to integrate the connectivity information into such learning pipelines.

[0057] In the follows, we describe the proposed WrappingNet, a first learning-based mesh compression system that is able to work with heterogeneous meshes of different sizes. To overcome issues regarding defining a convolution on irregularly structured data, we utilize methods that first remesh the meshes to a subdivision or semi-regular structure. This alleviates the irregularity challenges and allows one to define more image-like convolutions. Doing so also removes the need to explicitly construct or transmit connectivity information at every upsampling step in the decoder. Furthermore, the method has the ability to either output a latent feature map on the base mesh, or to learn a fixed-length latent representation of the mesh, both of which are then converted into a compressed bitstream. In the latter case, this is achieved by applying global pooling at the end of the encoder along with a novel module which disassociates latent representation from the connectivity information in the base mesh, called the UnWrapping module.

[0058] FIG. 2 illustrates the overall process for the proposed mesh compression network,according to an embodiment. We represent a single input subdivision mesh (205) with n vertices (points) and m faces using a list of positions X E IR’IX3, and a list of triangles T E Hmx3which contain indices of the corresponding points. Note that due to the structure of subdivision meshes, the base mesh is immediately known, with corresponding positions and triangles Xb, Tb. At a high level, the proposed mesh encoder UnWrappingEnc (UnWrapping-based Encoder) consumes the input subdivided mesh through any face-based mesh feature extractor (210) (e g., consisting of face convolution layers followed by reverse loop subdivision pooling), which produces an initial feature map (220) over the faces of the base mesh (225). The feature map is a matrix of dimension mbx 1024, where mbis the number of faces in the base mesh and 1024 is the feature dimension. Following that, AdaptMaxPool (230) (essentially shared MLP layers followed by global pooling) is applied across the faces to generate a single latent vector (i.e., the fixed-length codeword). In FIG. 2, the fixed length would be 1024. However, depending on the dimension of the features output from the feature extraction module (210), the fixed length can vary.

[0059] A learnable module called the UnWrapping module (240), consisting of a series of face convolutions and a novel mesh processing layer we call Face2Node, deforms the base mesh geometry into an approximate canonical geometry (245) and then matches (250) to the actual canonical grid geometry (251, e.g., a fixed sphere grid). In FIG. 2, the canonical grid is a sphere grid, and more generally, the canonical grid can be a pre-defined template grid with a pre-defined number of vertices and pre-defined vertex positions (the canonical grid contains only geometry but not connectivity). The output of the matching (250) is the base graph G (255) which is a matrix representing the connectivity of the base mesh but between the matched vertices (i.e., nearest neighbor in one example) of the canonical grid.

[0060] Moreover, the latent vector is quantized and converted into a bitstream (235), for example, using learnable bottleneck layers. The base graph is also encoded.

[0061] For decoding, we propose WrappingDec (Wrapping-based Decoder). First a quantized latent vector and based graph are recovered from the bitstream. Then, from the canonical grid (261) and the base graph (255), the decoder generates (260) an initial base mesh (275). Note that the geometry of this initial base mesh is different from the approximate canonical grid geometry (245) at the encoder side. Then based on the initial base mesh (275) and the latent vector repeated mbtimes (280, to serve as repeated face features) are first “wrapped” back into the base mesh (271)using another learnable module called Wrapping module (270) (with the exact same architecture as UnWrapping module). With an estimate of the base mesh along with the codeword (latent vector) repeated matimes (285), the decoder further uses additional Wrapping modules (290) enhanced with loop subdivision based unpooling layers to produce a final reconstructed mesh (295) at the same resolution as the input subdivision mesh (205). Note that the final reconstructed mesh (295) is an upsampled and refined version of the base mesh (271).

[0062] For generating a feature map rather than a codeword, the AdaptMaxPool is skipped, and thus UnWrappingNet (and the first WrappingNet) are not needed as the base mesh itself is transmitted to the decoder. FIG. 3 presents a diagram for this procedure according to an embodiment.

[0063] The codeword version of our compression network is useful for extracting fixed-length representations from heterogeneous mesh datasets, whereas the feature map version is more suited for high fidelity mesh reconstruction.

[0064] Both versions of our compression network are trained end-to-end with MSE loss at each reconstruction stage with the ground truth remeshed mesh at that stage.

[0065] Face Feature Initialization

[0066] Before moving further, it is important to discuss the use of the face-centric features that are propagated throughout the model. We wish to ensure that the input features are invariant to ordering of nodes and faces, and global position or orientation of the face. Hence, we choose the input face features to be the normal vector of the face, the face area, and a vector containing curvature information of the face, which is defined as follows. For face i, let j0, jj, and j2denote the face indices of its 3 neighbors. The curvature vector is simply c , wherec£, CjQ, Cj Cj2are the centroids of the faces, respectively. Thus, we have a total of 7 input features which are used for any module that consumes a mesh directly but requires some input face features. The list of features for the input subdivided mesh is denoted by F e Rmx7.

[0067] We now provide some details of the proposed modules in the following.

[0068] UnWrapping based Heterogeneous Mesh Encoder

[0069] As shown in FIG. 2, our proposed encoding module named UnWrappingEnc consists of aface-centric mesh feature extraction module (210) followed by an UnWrapping module (240). In one embodiment, the face-centric mesh feature extraction consists of K repetitions of DownFaceConv layers (520, 521, 522) as shown in FIG. 5 A. The feature extraction module accepts the connectivity (T) and initial features (F) as input, and outputs the geometry (Xb), connectivity (Tb) and features (Fb) of the base mesh. The dimension of the output features Fb is denoted by I. Each DownFaceConv layer is a pair of FaceConv (550) and SubDivPool (560) as shown in FIG. 5B. The FaceConv layer is an existing state-of-the-art proposal for mesh face feature extraction given a subdivision mesh while the loop subdivision-based pooling / downsampling SubDivPool is also a known traditional method. The purpose of DownFaceConv layer is to downsample the number of faces by a factor of 4.

[0070] As shown in FIG. 4A, the UnWrapping module consists of pairs of FaceConv and Face2Node layers (420, 430, 421, 431, 422, 432). Note that FIG. 4A illustrates three pairs of FaceConv and Face2Node, but there can be more or fewer pairs. The Face2Node, shown in FIG. 6, is our novel proposal to convert a set of face features directly into associated node position updates for the node and updated set of face features. The UnWrapping module accepts the base mesh (vertices and faces) as input and outputs a deformed base mesh whose geometry resembles a sphere mesh.

[0071] Wrapping-based Heterogeneous Mesh Decoder

[0072] Our proposed decoding module shown in FIG. 2 named WrappingDec consists of several repetitions of Wrapping modules. Each Wrapping module in turn consists of pairs of FaceConv (or UpFaceConv) and Face2Node layers (460, 470, 461, 471, 462, 472) as shown in FIG. 4B. Each UpFaceConv layer, shown in FIG. 4B, is again a pair of FaceConv (590) and SubDivUnpool (LoopUnpool) (585), as shown in FIG. 5C. The FaceConv is the same as in our encoder while the loop subdivision based unpooling / upsampling SubDivUnpool is a known traditional method.

[0073] FaceiNode

[0074] The loop subdivision based unpooling / upsampling performs upsampling on an input mesh in a deterministic manner and is akin to the naive upsampling in the 2D image domain. Thus, the output node locations in the upsampled mesh are fixed given the input mesh node positions. However, as our aim is to output the best reconstruction of the input mesh given the codeword, tofacilitate this reconstruction it is beneficial to supervise the intermediate lower resolution approximate reconstructions as well. This also enables us to perform scalable decoding depending on the desired decoded resolution and depending on the decoder resources, rather than being restricted to always output a reconstruction matching the resolution of the input mesh.

[0075] Hence, as illustrated in FIG. 6, our proposed Face2Node layer takes as input (610) the face list (T), face features (F), node locations (X), and face connectivity and outputs the updated node locations (X’) corresponding to an intermediate approximation of the input mesh along with the associated updated face features (F’). Consider the set of face features as F 6and the set of node locations as X G IRnx3. Then, as the first step of Face2Node we construct (620) a set of augmented node-specific face features 6, which can be considered as the face features from the point of view of specific nodes that are a part of those faces.

[0076] Let’s say that node j is the A: -th node of the i-th mesh face, where k can be 0, 1 or 2. Then, the face feature f according to node ; is gtJo / o3] , whererepresents the index of node j in the i-th mesh face, %3 represents modulus with respect to 3, and etrepresents the Z-th edge vector. The edge vectors are defined in a predefined way as: e0= xt— x0, e1= x2— x1;and e2= x0— x2; and are concatenated in a cyclic manner depending on the index of the reference node index in the face(hence the modulus). The set of augmented nodespecific face features G is then updated using a shared MLP (MultiLayer Perceptron) module (630) that operates on eachin parallel and outputs the updated node-specific feature set G' (640).

[0077] Finally, the updated face feature / )' is the average of the updated node-specific features as fi = ~ ,j 9ijl^‘] (650), while the updated node locations are obtained from neighborhood averaging as x- = Xj (660), where the neighborhood Nj is defined by all thefaces that contain node j as a vertex. In another example, the updated node locations can be obtained from neighborhood averaging as Xj [0:3] (660). Here, [0:3] represents thefirst three entries of the vector while [3:] represents the 4thto the last entries of the vector.

[0078] Latent Representation and Wrapping Modules

[0079] In our fixed-length codeword compression network there is a need to inject the geometry (mesh vertex positions) information at different scales into the fixed-length codeword, especiallythe base mesh geometry. Without forcing this information into the codeword, the quality of the codeword is severely diminished and the codeword just contains a summary of local face-specific information which degrades the performance of the codeword when paired with a downstream task like classification, segmentation, etc.

[0080] As shown in FIG. 2, a module named UnWrapping module (240) is used to achieve disentanglement of this geometry information. The UnWrapping module (240) aims to match the base mesh geometry to a predefined canonical geometry (e.g., a set of points sampled on the unit sphere), for example, by matching the vertices of the canonical geometry and the vertices of the base mesh. This is done by deforming the base mesh geometry into an approximate canonical geometry and then matching to the actual canonical geometric shape, for example, using either EMD or Sinkhom algorithm (250). During training, only the output of this matching (matched points on the canonical grid) is transmitted to the decoder, and thus the codeword from the encoder is forced to learn the geometry information. The UnWrapping module can be trained separately or in tandem with our overall compression network in an end-to-end fashion supervised by the well-known Chamfer distance. UnWrapping module architecture consists of three pairs of FaceConv and Face2Node layers to output the canonical-shape-mapped base mesh vertex positions using the face features of the base mesh.

[0081] At the decoder side, a complementary Wrapping module is used, having the same architecture (but with different parameters) as UnWrapping module, to reconstruct the base mesh geometry from the matched points to the canonical shape. The decoder starts by taking the canonical geometry (261) and the base graph G (255, the connectivity part of the base mesh) to form (260) an initial base mesh (275), which is then the initial input to the first Wrapping module.

[0082] We must note that the use of the matching indices to align the input mesh and reconstructed mesh is only needed to enforce the loss during training. During inference, all that is needed is to use the matching to re-order the base graph before sending it to the decoder and there is no need to transmit the matching indices.

[0083] Latent Representation and GNNs

[0084] The aforementioned architecture still uses the connectivity information through fixed connectivity dependent operations via the FaceConv layers. However, this connectivityinformation can be better exploited through an architecture employing graph based learnable operations. For this purpose, the UnWrapping module (in encoder) is removed and Wrapping first stage module (in decoder) get replaced by a single graph neural network based module that we refer to as Base Mesh Reconstruction GNN (BaseConGNN). As shown in an updated architecture in FIG. 7, BaseConGNN (760) takes the quantized codeword c from the bitstream and first converts (745) it into a set of local face-specific codewords as Cb— Gbl. These local codewords along with the connectivity information present as a weighted graph Gbfrom the base mesh connectivity are then fed to a standard GNN architecture that makes graph-aware updates through shared MLPs to transform the local codewords into estimated base mesh face features and geometry. The rest of the decoding pipeline remains the same as before.

[0085] Base graph coding

[0086] One important aspect of the proposed architecture is to encode the base graph G into the bitstream. There are several ways of doing this in traditional vs learning based methods. In the following we outline a few of these methods.

[0087] In one embodiment the base graph can be encoded in a fixed-length coding manner where for each triangle in the base mesh, connectivity is represented by the list of nodes connected by each edge. The entries of this ordered matrix can be encoded in a sequential manner using fixed- length coding, as shown in FIG. 8. The entries of the matrix representing the base graph (810) are first row-wise vectorized (820) and each entry is compressed using a fixed-length encoder (830). On the decoder side, the fixed-length decoder (835) is used to first obtain the vectorized entries (840) which can be easily unvectorized to obtain the base graph.

[0088] In one embodiment, existing traditional methods like EdgeBreaker algorithm can be used to encode the base graph. Edgebreaker’s compression traverses the mesh from one triangle to an adjacent one, while at each stage, compression produces an opcode (operation code) describing the topological relation between the current triangle and the boundary of the remaining part of the mesh. The Decompression uses these opcodes to reconstruct the entire incidence graph.

[0089] In one embodiment, a learning-based method like Partition and Code (PnC) can be used which compresses the mesh in three steps. The first step is to decompose the original graph into several component subgraphs, followed by a second step of constructing a dictionary out ofcomponent graphs and learning a distribution of each dictionary element. The final step is to use an entropy coder to translate this representation into bits.

[0090] Advanced micro-architectures

[0091] In one implementation, the feature aggregation module takes inspiration for the well- known ResNet architecture, as shown in FIG. 9. In this example, it shows the architecture of a ResFaceConv (RFC) block to aggregate features with D channels. FIG. 9 introduces a residual connection from the input and added with the output of the series of FaceConv layers. All FaceConv can be replaced with ResFaceConv.

[0092] In another implementation, the feature aggregation module takes inspiration from Inception-ResNet architecture, as shown in FIG. 10. In this example, it shows the architecture of an Inception-ResFaceConv (IRFC) block to aggregate features with D channels.

[0093] Partition-based Coding

[0094] Earlier we have described the architecture when our architecture encodes / decodes mesh as a whole. However, this procedure becomes increasingly time consuming and computationally expensive as the geometry data precision and the density of points in the mesh increases. Moreover, the process of converting the raw mesh data into a remeshed mesh takes longer as well. To deal with this issue, as shown in FIG. 11, this embodiment proposes to first convert the raw mesh (1110) into partitions via a shallow octree (1120), bringing the data points in each partition from the original coordinates to the local partition coordinates (by shifting the origin for each partition), and finally encoding / decoding each partition mesh separately using our proposed UnWrapping-based encoder (1150). Other partitioning schemes such as object-based or part-based partitioning can also be employed, and one only needs the origins of each partition in the original coordinates to construct the shallow octree.

[0095] With this procedure, each partition contains a smaller part of the mesh which can be remeshed (1130) faster and in parallel for each partition. After compression and decompression, the recovered meshes from all partitions are combined and brought back into the original coordinates. The auxiliary information from the shallow octree regarding the partitioning is also compressed using uniform entropy coding (1140) and added (1160) to the bitstream (1170).

[0096] As shown in FIG. 12, the decoding procedure follows an exact opposite process of theencoding procedure. First, the transmitted bitstream (1210) is split into component bitstreams: partitioning bitstream (1220) and partition bitstreams (1230). The partitioning bitstream is fed to the entropy decoder (1240) to decode the shallow partitioning octree (1260), while the partition bitstreams are fed (separately) to the Wrapping-based decoder (1250) to produce the reconstructed partition meshes (1270). Finally, the shallow partitioning octree and the reconstructed mesh partitions are combined to produce the final reconstructed mesh (1280) by adding the leaf node positions of shallow octree to the vertex positions of the associated mesh partitions.[97J Classification

[0098] The proposed WrappingNet architecture can also be used to perform classification on the input meshes. For this purpose, as illustrated in FIG. 13 which shows the classification architecture containing the (pre-trained) UnWrappingNet, the UnWrapping encoder part (1310) of the overall (pre-trained) WrappingNet architecture can be followed by a few MLP layers followed by a softmax function (1320). The output dimension of the softmax would be equal to the number of classes that need to be differentiated and each output value will provide a score for a respective class. This architecture can be trained with a cross-entropy loss to finally enable mesh classification.

[0099] In FIG. 14, we provide a general methodology for mesh compression using a base mesh, according to an embodiment. To enable this, we can use any general-purpose mesh feature extractor (1410) that takes a mesh topology as input and produces a lower resolution base mesh (M) along with a feature map (F) associated with each vertex of the base mesh. The feature map (F) summarizes the details of the original mesh input and is compressed (1430) in a lossy manner using an entropy bottleneck layer or existing lossy compression techniques. The base mesh (M) is compressed (1420) losslessly to preserve the coarse representation of the original mesh, using any existing lossless mesh coding techniques like Google Draco, etc. On the decoder side, the base mesh is decoded (1440). The feature map F decoded by an entropy decoder (1450) is sent to a feature decoder (1460) to reconstruct the mesh.

[0100] FIG. 15 illustrates a general methodology for mesh compression using a base graph, according to an embodiment. This is done to achieve higher compression rates by disentangling the vertex position information from the connectivity information. Instead of a feature map and a base mesh, a single codeword along with a base graph are transmitted to succinctly describe themesh topology. To enable this, as before, we can use any general-purpose mesh feature extractor (1510) that takes a mesh topology as input and produces a lower resolution base mesh (M) along with some base mesh features (F) associated with each vertex of the base mesh. However, all features in the feature map are adaptively pooled into a single codeword (feature) to be transmitted in a lossy manner using existing lossy compression techniques (1520). The base mesh (M) also gets “unwrapped” (1530) into a template mesh to remove the vertex position information and only the connectivity information is compressed (1540) losslessly using any existing lossless coding techniques like Google Draco, etc.

[0101] On the decoder side, the base graph is decoded (1550) and converted into a modified template mesh using original template mesh vertex positions and the base graph connectivity. Then the modified template mesh is “wrapped” (1560) to produce a first base mesh reconstruction using the decoded codeword (1570). Further refinement of the base mesh is performed via the wrapping-based feature decoder (1580) using the decoded codeword to produce a final reconstructed mesh.

[0102] Various numeric values are used in the present application, for example, the number of repetitions of FaceConv and Face2Node layers in the UnWrapping module, or the size of the feature map of the base mesh. The specific values are for example purposes and the aspects described are not limited to these specific values.

[0103] Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as “first”, “second”, etc. may be used in various embodiments to modify an element, component, step, operation, etc., such as, for example, a “first decoding” and a “second decoding”. Use of such terms does not imply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or in an overlapping time period with the second decoding.

[0104] The implementations and aspects described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as amethod), the implementation of features discussed may also be implemented in other forms (for example, an apparatus or program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, for example, computers, cell phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.

[0105] Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0106] Additionally, this application may refer to “determining” various pieces of information. Determining the information may include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0107] Further, this application may refer to “accessing” various pieces of information. Accessing the information may include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0108] Additionally, this application may refer to “receiving” various pieces of information. Receiving is, as with “accessing”, intended to be a broad term. Receiving the information may include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0109] It is to be appreciated that the use of any of the following “ / ”, “and / or”, and “at least one of’, for example, in the cases of “A / B”, “A and / or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as is clear to one of ordinary skill in this and related arts, for as many items as are listed.[HO] As will be evident to one of ordinary skill in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal may be formatted to carry the bitstream of a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

Claims

CLAIMS1. A method for decoding 3D mesh, comprising: obtaining a template grid; obtaining a base graph containing connectivity information of said 3D mesh; forming a first 3D mesh based on said base graph and geometry of said template grid; obtaining a codeword representing geometry information of said 3D mesh; and reconstructing said 3D mesh from said first 3D mesh and said codeword using a learningbased module.

2. A method for decoding 3D mesh, comprising: obtaining a base mesh associated with said 3D mesh; obtaining a codeword representing geometry information of said 3D mesh; and reconstructing said 3D mesh using a learning-based module, based on said base mesh and said codeword.

3. The method of claim 1 or 2, further comprising converting said reconstructed 3D mesh to another version of said 3D mesh using another learning-based module, based on said codeword, wherein said another version of said 3D mesh is a refined version of said reconstructed 3D mesh.

4. The method of any one of claims 1-3, wherein said learning-based module is a single graph neural network based module.

5. The method of any one of claims 1-3, wherein said learning-based module includes at least a module to extract face-based features and a module to update node positions and face features.

6. The method of any one of claims 1-5, wherein said another learning-based module includes at least a module to extract face-based features, at least a module to update node positions and face features, and at least a loop subdivision based upsampling layer.

7. The method of claim 5 or 6, wherein said module to update node positions and face features performs: constructing a set of augmented node-specific face features; and updating said augmented node-specific face features by a shared MLP (MultiLayer Perceptron) module.

8. The method of any one of claims 1-7, wherein said reconstructing said 3D mesh further generates face features associated with said 3D mesh.

9. A method for encoding 3D mesh, comprising: obtaining a base mesh corresponding to said 3D mesh and face features associated with said base mesh; generating a fixed-length codeword from said face features associated with said base mesh, using a first learning-based module; forming a base graph indicating connectivity of said base mesh, based on said base mesh and a template grid, using a second learning-based module; and encoding said fixed-length codeword and said based graph.

10. A method for encoding 3D mesh, comprising: obtaining a base mesh corresponding to said 3D mesh and face features associated with said base mesh; and encoding said face features and said based mesh.

11. The method of claim 9, wherein said forming a base graph comprises: matching vertices of said based mesh and vertices of said template grid.

12. The method of claim 11, wherein said matching comprises: deforming geometry information of said base mesh to an approximate canonical geometry; and matching vertices of said approximate canonical geometry to vertices of said template grid.

13. The method of any one of claims 9-12, further comprising: converting said 3D mesh to said base mesh, said base mesh having fewer points than said3D mesh.

14. The method of any one of claims 9 and 11-13, wherein said forming a base graph comprises: mapping said base mesh to a first 3D mesh, wherein a shape of said first 3D mesh is similar to a shape of said template grid; and matching vertices of said first 3D mesh to vertices of said template grid.

15. The method of any one of claims 9 and 11-14, further comprising: performing classification on said 3D mesh.

16. The method of any one of claims 9 and 11-15, wherein said first learning-based module includes at least a module to extract face-based features and a module to update node positions and face features.

17. The method of claim 16, wherein said module to update node positions and face features performs: constructing a set of augmented node-specific face features; and updating said augmented node-specific face features by a shared MLP (MultiLayer Perceptron) module.

18. An apparatus, comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to perform the method of any of claims 1-17.

19. A non-transitory computer readable medium comprising instructions which, when the instructions are executed by a computer, cause the computer to perform the method of any of claims 1-17.

Images