Learning-based grid compression using latent grids

Abstract

3D meshes rely heavily on connectivity information, and therefore on face information within the mesh data format. This connectivity or face information presents challenges in designing efficient architectures (traditional or learning-based) for mesh processing and compression. The base mesh can be viewed as a downsampled version of the input mesh and depicts a coarse shape of an object or scene. Providing the base mesh offers a way to address the challenges arising from connectivity and faces. In one implementation, we propose a learning-based mesh compression system based on latent mesh generation and encoding. The latent mesh allows for a tradeoff between bitrate and distortion. A predictive encoding strategy is applied to encode a list of points in the latent mesh. Furthermore, an adaptive arithmetic encoding of the point list is applied based on a set of distribution parameters identified from training.

Description

Technical Field

[0001] This embodiment generally relates to methods and apparatus for 3D mesh compression. Background Technology

[0002] 3D point clouds are a common data format spanning several business areas, from autonomous driving, robotics, AR / VR, civil engineering, and computer graphics to animation / film. 3D LiDAR sensors are already widely deployed in self-driving cars. Affordable LiDAR sensors have been released by companies such as Velodyne Velabit, Apple iPad Pro 2020, and Intel RealSense LiDAR camera L515. With significant advancements in sensing technology, 3D point cloud data has become more practical than ever before and is expected to be the ultimate enabler for the aforementioned applications.

[0003] Point cloud data is also considered to consume a significant portion of network traffic, for example in cars connected via 5G networks and in immersive communications (VR / AR). Understanding and communicating point clouds will inherently lead to efficient representation formats. In particular, raw point cloud data needs to be properly organized and processed for use in world modeling and sensing purposes.

[0004] Furthermore, point clouds can represent continuous scans of the same scene containing multiple moving objects. They are called dynamic point clouds in contrast to static point clouds captured from static scenes or static objects. Dynamic point clouds are typically organized into frames, with different frames captured at different times.

[0005] A mesh is a data format composed of both points (also called vertices) and faces. Compared to a point cloud, a mesh is considered a complete model of a 3D surface, while the points in a point cloud are just a set of discrete samples of the surface.

[0006] For computer graphics (CG), meshes are generally available because the data is generated by the content provider. For data acquired by sensors (including LiDAR), the raw data is typically in point cloud format. For point cloud data, meshes can be created via a meshing process. Summary of the Invention

[0007] According to an embodiment, a method for decoding a mesh is proposed, comprising: decoding a feature map; decoding a latent mesh, including: decoding a face list of the latent mesh to obtain a predicted point list of the point list of the latent mesh, decoding residual information associated with the point list of the latent mesh, and reconstructing the point list of the latent mesh based on the residual information and the predicted point list of the point list of the latent mesh; and decoding the mesh based on the decoded feature map, and decoding the decoded latent mesh based on a first neural network.

[0008] According to another embodiment, a method for encoding a mesh is proposed, comprising: obtaining a feature map and a base mesh corresponding to the mesh based on a fourth neural network; encoding the feature map; transforming the base mesh into a latent mesh; and encoding the latent mesh, comprising: encoding a face list of the latent mesh to obtain a predicted point list of a point list of the latent mesh, obtaining residual information associated with the point list, and encoding the residual information of the point list.

[0009] According to another embodiment, an apparatus for decoding a mesh is proposed, comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to: decode a feature map; decode a latent mesh, wherein the one or more processors are configured to: decode a face list of the latent mesh to obtain a predicted point list of a point list of the latent mesh, decode residual information associated with the point list of the latent mesh, and reconstruct the point list of the latent mesh based on the residual information and the predicted point list of the point list of the latent mesh; and decode the mesh based on the decoded feature map, and decode the decoded latent mesh based on a first neural network.

[0010] According to another embodiment, an apparatus for encoding a mesh is proposed, comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to: obtain a feature map and a base mesh corresponding to the mesh based on a fourth neural network; encode the feature map; transform the base mesh into a latent mesh; and encode the latent mesh, wherein the one or more processors are configured to: encode a face list of the latent mesh, obtain a predicted point list of a point list of the latent mesh, obtain residual information associated with the point list, and encode the residual information of the point list.

[0011] One or more embodiments also provide a computer program including instructions that, when executed by one or more processors, cause the one or more processors to perform an encoding or decoding method according to any embodiment described herein. One or more of these embodiments also provide a computer-readable storage medium storing instructions for encoding or decoding a mesh according to the methods described herein.

[0012] One or more embodiments also provide a computer-readable storage medium storing data generated according to the described method. One or more embodiments also provide a method and apparatus for transmitting or receiving data generated according to the methods described herein. Attached Figure Description

[0013] Figure 1 The diagram illustrates a block diagram of a system in which various aspects of this embodiment can be implemented.

[0014] Figure 2 The diagram illustrates a learning-based grid compression system.

[0015] Figure 3 The diagram illustrates a learning-based grid compression system that uses a base graph.

[0016] Figure 4 The diagram illustrates a wrapping block.

[0017] Figure 5 The diagram illustrates a surface convolution kernel.

[0018] Figure 6 The diagram illustrates the residual network diagram for updating point positions in the potential grid.

[0019] Figure 7 The diagram illustrates a learning-based grid compression system using a latent grid according to an embodiment.

[0020] Figure 8 The diagram shows the package block.

[0021] Figure 9 The diagram illustrates the reverse wrapping block.

[0022] Figure 10 The diagram illustrates the encoding of a potential mesh according to an embodiment.

[0023] Figure 11 The illustration shows the generation of predicted values ​​based on parallelograms.

[0024] Figure 12 The diagram illustrates the decoding of a potential mesh according to an embodiment.

[0025] Figure 13 The diagram illustrates the encoding of a latent mesh using a super-prior model according to an embodiment.

[0026] Figure 14 The diagram illustrates the decoding of a latent mesh using a super-prior model according to an embodiment.

[0027] Figure 15 The diagram illustrates the encoding of a latent grid using an autoregressive super-prior model according to an embodiment.

[0028] Figure 16 The diagram illustrates the decoding of a latent grid using an autoregressive super-prior model according to an embodiment.

[0029] Figure 17 The diagram illustrates the feature encoder.

[0030] Figure 18 The diagram illustrates subdivision-downsampling.

[0031] Figure 19 The diagram illustrates the feature decoder.

[0032] Figure 20 The diagram illustrates subdivision-upsampling.

[0033] Figure 21 The diagram illustrates a non-manifold mesh. Detailed Implementation

[0034] Figure 1 The diagram illustrates an example of a system in which various aspects and embodiments can be implemented. System 100 can be implemented as a device including the various components described below and configured to perform one or more aspects of the aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 100 can be implemented individually or in combination in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices via, for example, a communication bus or through dedicated input ports and / or output ports. In various embodiments, system 100 is configured to implement one or more aspects of the aspects described in this application.

[0035] System 100 includes at least one processor 110 configured to execute instructions loaded thereon for implementing various aspects as described in this application. Processor 110 may include embedded memory, input / output interfaces, and various other circuitry as known in the art. System 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). System 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 140 may include internal storage devices, attached storage devices, and / or network-accessible storage devices.

[0036] System 100 includes an encoder / decoder module 130 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents one or more modules that can be included in a device to perform encoding and / or decoding functions. It is well known that a device may include one or both encoding and decoding modules. Furthermore, the encoder / decoder module 130 may be implemented as a separate element of system 100, or it may be incorporated into processor 110 as a combination of hardware and software as known to those skilled in the art.

[0037] Program code to be loaded onto processor 110 or encoder / decoder 130 to execute the various aspects described in this application may be stored in storage device 140 and subsequently loaded onto memory 120 for execution by processor 110. According to various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more items during the execution of the processes described in this application. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing equations, formulas, operations, and operational logic.

[0038] In several embodiments, the internal memory of processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing during encoding or decoding. However, in other embodiments, external memory of the processing device (e.g., processor 110 or encoder / decoder module 130) is used for one or more of these functions. The external memory may be memory 120 and / or storage device 140, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, fast external volatile memory (such as RAM) is used as working memory for video encoding and decoding operations, such as for MPEG-2, JPEG Pleno, MPEG-I, V-DMC, HEVC, or VVC.

[0039] Input to the components of system 100 can be provided through various input devices as indicated in block 105. Such input devices include, but are not limited to: (i) an RF section that receives, for example, RF signals transmitted over the air by a broadcaster; (ii) a composite input terminal; (iii) a USB input terminal; and / or (iv) an HDMI input terminal.

[0040] In various embodiments, the input device of block 105 has associated corresponding input processing elements as known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting a signal band to a band), (ii) down-converting the selected signal, (iii) further band-limiting to a narrower band to select (e.g.,) a signal band that may be referred to as a channel in some embodiments), (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements for performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners that perform various functions among these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or down-converting it to baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the components described above (and others), remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0041] Furthermore, USB and / or HDMI terminals may include corresponding interface processors for connecting system 100 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented as needed, for example, within a separate input processing IC or within processor 110. Similarly, various aspects of USB or HDMI interface processing may be implemented as needed, either within a separate interface IC or within processor 110. Demodulation, error correction, and demultiplexing streams are provided to various processing elements, including, for example, processor 110 and encoder / decoder 130, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on an output device.

[0042] Various components of system 100 can be provided within an integrated housing in which various components can be interconnected and transmit data therebetween using a suitable connection arrangement 115 (e.g., internal buses as known in the art, including I2C buses, wiring and printed circuit boards).

[0043] System 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 190 may be implemented, for example, within a wired and / or wireless medium.

[0044] In various embodiments, a Wi-Fi network (such as IEEE 802.11) is used to stream data to system 100. The Wi-Fi signals in these embodiments are received via a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 in these embodiments is typically connected to an access point or router that provides access to external networks (including the Internet) to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to system 100, delivering data via an HDMI connection to input block 105. Still other embodiments use an RF connection to input block 105 to provide streaming data to system 100.

[0045] System 100 can provide output signals to various output devices, including display 165, speaker 175, and other peripheral devices 185. In various examples of embodiments, other peripheral devices 185 include one or more of a standalone DVR, disk player, stereo system, lighting system, and other devices that provide functionality based on the output of system 100. In various embodiments, control signals are transmitted between system 100 and display 165, speaker 175, or other peripheral devices 185 using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to system 100 via dedicated connections through corresponding interfaces 160, 170, and 180. Alternatively, output devices can be connected to system 100 via communication interface 150 using communication channel 190. In electronic devices (e.g., televisions), display 165 and speaker 175 can be integrated into a single unit with other components of system 100. In various embodiments, display interface 160 includes a display driver, such as a timing controller (TCon) chip.

[0046] For example, if the RF portion of input 105 is part of a separate set-top box, then display 165 and speaker 175 can alternatively be separated from one or more other components. In various embodiments where display 165 and speaker 175 are external components, output signals can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0047] The automotive industry and autonomous vehicles are among the sectors where point clouds can be used. Autonomous vehicles should be able to "detect" their environment to make sound driving decisions based on the realities of their immediate surroundings. For example, typical LiDAR sensors generate (dynamic) point clouds used by perception engines. These point clouds are not intended to be seen by the human eye, and they are typically sparse, not necessarily colored, and dynamic with a high capture frequency. They can possess other properties, such as reflectivity provided by LiDAR, as this property indicates the material of the sensed object and can aid in decision-making.

[0048] Virtual reality (VR) and immersive worlds are envisioned by many as the future of 2D video. The basic idea is to immerse viewers in an environment surrounding them, rather than as with standard TV where they can only see a virtual world in front of them. The level of immersion varies depending on the viewer's degree of freedom within the environment. Point clouds and meshes are both good candidate formats for distributing VR content. They can be static or dynamic and are typically moderately sized, containing no more than a few million points at a time.

[0049] Point clouds and meshes can also be used for various purposes, such as cultural heritage / buildings, where objects like statues or buildings are 3D scanned to share their spatial configurations without needing to send or access them. Furthermore, this is a way to ensure knowledge of an object is preserved in the event of potential damage; for example, a temple destroyed by an earthquake. Such point clouds and meshes are typically static, colored, and massive.

[0050] World modeling and sensing via point clouds / mesh can be a key technology for allowing machines to gain knowledge about the 3D world around them, which is essential for the applications discussed above.

[0051] 3D point cloud data is essentially discrete samples of the surface of an object or scene. To fully represent the real world using these point samples, a massive number of points are actually required. For example, a typical VR immersive scene contains millions of points, while a point cloud typically contains hundreds of millions. Therefore, processing such large-scale point clouds is computationally expensive, especially for consumer devices with limited computing power, such as smartphones, tablets, and car navigation systems. Furthermore, the discrete samples comprising 3D point cloud data still contain incomplete information about the underlying surfaces of objects and scenes. Therefore, efforts are currently underway to explore mesh representations for 3D scene / surface representation.

[0052] A mesh can be viewed as a 3D point cloud and the connectivity information between the points. Therefore, a mesh representation bridges the gap between the point cloud and the underlying continuous surface through local 2D polygonal patches (called faces) approximated by the underlying surface. From another perspective, a mesh can simplify a point cloud by reducing the number of points. For example, faces in a mesh might be suitable for representing flat areas, while a point cloud might require dense patches of points to represent the same flat area.

[0053] The first step in any processing or inference of mesh data is having an efficient storage method. To store and process meshes at an affordable computational cost, one solution is to first downsample them, where the downsampled mesh summarizes the geometry of the input mesh while having fewer but more (or much larger) faces. The downsampled mesh is then fed into subsequent machine tasks for further use. However, further reductions in storage space can be achieved by converting the raw mesh data (raw or downsampled) into fixed-length codewords or feature maps located on a very low-resolution mesh. These codewords or feature maps can be converted into a bitstream using existing entropy coding techniques. Furthermore, the codewords or feature maps themselves are useful because they (respectively) represent global or local surface information of the underlying scene / object and can be paired with subsequent downstream (machine vision) tasks.

[0054] Figure 2 A typical diagram of a learning-based grid compression system consisting of two branches is illustrated. The main branch is used for feature encoding. The main branch encoder uses a feature extractor (210) to extract and aggregate features from the input grid. The extracted feature map F is sent to the entropy encoder (230, ENC). F To form a bitstream BS F By entropy decoder (250, DEC) F Feature maps of decoding The input mesh is sent to the feature decoder (260) to reconstruct the mesh. Utilizing feature extraction / aggregation, the input mesh is typically downsampled to a base mesh M with a lower resolution for compression benefits. The feature map F is associated with the base mesh M.

[0055] Figure 2 The side branch of the medium-grid compression is used to encode the base grid M. It consists of the grids used for encoding to form the bitstream BS. M ENC M Block (220) and DEC for decoding M The grid consists of blocks (240). Encoding and decoding the base mesh involves encoding the list of faces. In the case of a triangular mesh, each face is represented by a list of three connected points. Each point is described by 3D coordinates. In one example, we let M = (X, T), where X is the list of points and T is the list of faces. Each point in a face is represented by its index in X. To encode the base mesh, the list of points X and the list of faces T need to be encoded.

[0056] Figure 3 A high-level graph of another related work on learning-based mesh compression is shown, which is a graph-based mesh compression method, as described in the co-owned patent application entitled "Base Graph based Mesh Compression" (U.S. Provisional Application No. 63 / 533,303, Attorney General's File No. 2023PF00502). Figure 2 Compared to the graph in the code, the base grid is transformed into a new representation called the base graph before being encoded. On the encoder side, this is done via the wrapping block WN. G (310, WrappingNet) performs the transformation. It attempts to decouple the topological information from the geometric information in the base mesh. Therefore, the base graph aims to convey only the topological information. On the decoder side, inverse wrapping (IWN) is performed. G (320) to transform the base graph back to the base mesh. Entropy-coded block ENC G (330) and entropy decoding block DEC G (340) is updated accordingly to process the base graph and the corresponding bitstream BS. G Instead of the base mesh, we let G = (N, T), where N is the list of graph nodes and T is the list of faces. The face list is the same as defined in the base mesh. The graph node list is obtained via the wrapping block WN. G Calculated.

[0057] WN Packet G It consists of two steps, such as Figure 4 As shown in the diagram. It first uses the "deform" block (410) to deform the input mesh M into a spherical mesh M. SIn one embodiment, the shape of the spherical grid is set to deform to approximate a sphere. Then, points in the spherical grid are matched with spherical grid points (S) via a sphere matching block (420). The spherical grid points are presampled on the sphere and kept fixed. Finally, the wrapper block outputs a list of graph nodes G.

[0058] Figure 4 The "deformation" block in the middle is further composed of two structures to generate the spherical mesh M. S One is a CNN network (411, 413) used to aggregate features on the surface. Then, the point positions of the input grid are deformed by the position update block (PU, 412, 414).

[0059] The first CNN architecture takes a grid M as its input. In such a way... Figure 4 In one embodiment shown, the CNN+PU structure is repeated twice to enhance the deformation. After the first CNN+PU structure (411, 412), the grid M is downsampled to M1 while simultaneously computing the feature map F1 associated with the downsampled grid M1. In one example, feature aggregation is performed on surfaces rather than points. Therefore, the CNN is chosen to run convolutions on adjacent surfaces, also known as surface convolutional layers. CNNs can be utilized in the case of manifold grids with regular spatial neighborhoods. Figure 5 The conventional faceted convolutional (CNN) kernels shown are in use. For example, 3-neighborhood, 6-neighborhood, or 9-neighborhood kernels.

[0060] like Figure 6 The design position update PU block shown is mainly composed of a residual update block (RU, 610) and an addition operation (620) between the input position and the displacement. The RU block (610) first performs a feature transformation from surface to node via a learnable network, and then calculates the displacement to update the point position.

[0061] Sphere matching loss is used to supervise the training of deformable blocks. In one embodiment, sphere matching is defined as the chamfer distance between a point in the deformable mesh and a cloud of randomly sampled points from a unit sphere.

[0062] In the following section, we propose a method for processing the encoding of the underlying mesh in a learning-based mesh compression system.

[0063] Figure 7 The diagram illustrates the overall structure of the proposed learning-based mesh compression according to an embodiment. The proposed learning-based mesh codec consists of two branches. In the main branch, features are extracted / aggregated based on the input mesh. The feature map F is generated (710) and ENC is applied. F (760) Entropy coding to form bitstream BS F And by DEC F(770) Decoding. The feature map is associated with the underlying mesh M. The processing of the underlying mesh is the second branch of the proposed codec.

[0064] Specifically, in the second branch, the base mesh M is connected via the wrapping block WN. L (720, WrappingNet) is transformed into a latent mesh M L Potential grid M L By ENC L (730) Entropy coding to form bitstream BS L And by DEC L (740) Decoding. On the decoder side, perform inverse wrapping IWN. L (750) to decode the potential grid Transform back to the reconstructed base mesh Decoded feature map and the base grid for reconstruction It is sent to the feature decoder (780) to reconstruct the mesh. .

[0065] and Figure 2 Compared to typical learning-based mesh compression shown, the proposed method will not directly encode the underlying mesh. Furthermore, it differs from... Figure 3 The system shown is not transformed into a base graph because the base mesh is not transformed into a base graph. Note that the intention of transforming from the base mesh to the base graph is to completely decouple the topological information. Instead, it is proposed here to transform the base mesh into a novel format, represented in this document as a latent mesh.

[0066] On the encoder side, the base mesh is processed via a novel WN L Block (720) deformation. From a high-level perspective, the latent mesh can be viewed as a trade-off format between the base mesh and the base graph. We want to address the following issues: Direct encoding of the base grid is beneficial for reconstruction quality, but may result in undesirable high bit rate consumption.

[0067] For encoding the base graph, it may be advantageous for the encoding bit rate of the base grid; however, it may sacrifice too much in terms of reconstruction quality.

[0068] For the proposed potential grid M L We propose a novel approach to control the trade-offs and use latent grids for optimized coding with respect to both bitrate and distortion.

[0069] Potential Mesh Generation As mentioned earlier, the base mesh is represented as M=(X,T), where X is a list of points and T is a list of faces. The base graph is represented as G=(N,T), where N is a list of graph nodes and T is a list of identical faces in the base mesh. To decouple the topology from the base mesh, the list of points is replaced by a list of graph nodes. Graph nodes N are represented by the indices of the grid points.

[0070] Similar to the basic mesh, the proposed potential mesh M L M consists of a list of points and a list of faces. L =(X L ,T). Potential mesh M L The face list T in the mesh is the same as the face list defined in the base mesh M. However, the point list X... L It is a novel transformation of the point list X.

[0071] In one embodiment, the deformable block WN L (720) can be designed as follows Figure 8 As shown, it is similar to Figure 4 WN in G Note that in Figure 8 middle, Figure 4 The sphere matching block (420) in the model is removed. It (420) was designed to match points in the sphere grid with points in the sphere grid. Deformation block WN L The output is a deformed mesh, called the (novel) latent mesh M. L Potential grid M L Deformation and spherical mesh M S The deformation is performed differently. The deformation of the spherical mesh (410) aims to approximate the shape of a sphere. It (410) is achieved through training based on sphere loss. In the case of the latent mesh, a trade-off is proposed for the deformation.

[0072] In order to make the potential grid M L Located between the base mesh and the base graph, a specific training loss is proposed to replace the chamfer distance in the ball loss. Specifically, the proposed loss function consists of two terms. The first term reflects the distortion measured between the base mesh and the latent mesh. The second term is the bitrate required to encode the latent mesh. This bitrate can be estimated using an entropy bottleneck layer, as proposed in our co-owned patent application entitled "Entropy Bottleneck Layer for Learning-Based Mesh Compression" (U.S. Provisional Application No. 63 / 595,559, Our Attorney General's File No. 2023PF00974). The total loss function is a weighted combination of the distortion and bitrate terms: Loss=Distortion+λ ∙ Rate.

[0073] The degree of distortion is determined by a trade-off between bitrate and distortion. In other words, the hyperparameter λ used to determine the target bitrate and distortion implicitly guides the degree of distortion.

[0074] Deformed network structure picture Figure 4 Like the deformable blocks in the game, Figure 8 The proposed package (deformation) block WN L It consists of two structures. One is a CNN network (810, 830) used to aggregate features about the surface. After feature aggregation, the point positions of the input grid are updated by the PU block (820, 840).

[0075] Besides the grid M, the first CNN structure (810) takes the initial features F as its input. In such a case... Figure 8 In one embodiment shown, the two structures (CNN and PU) are repeated twice to enhance the wrapping (deformation). In another embodiment, the CNN and PU can be repeated more than twice. In one example, feature aggregation is performed on surfaces rather than points. Therefore, the CNN is chosen to run convolutions on adjacent surfaces, and this layer is also called a surface convolutional layer. CNNs can be utilized in the case of manifold meshes with regular spatial adjacency relationships. In other words, a structure such as... Figure 5 The diagram shows a regular faceted convolutional (CNN) kernel with 3-neighborhood, 6-neighborhood, or 9-neighborhood kernels. The first feature input is a 7-dimensional feature containing the area (1D), normal vector (3D), and curvature vector (3D) of each face. The first faceted convolutional layer maps to a hidden dimension of 128 channels, while the second faceted convolutional layer maintains 128 dimensions. The PU block uses a faceted, 2-layer MLP (Multilayer Perceptron) with input-output dimensions (128, 128, 128) to generate the displacement vector output.

[0076] The position update PU block can be designed as follows: Figure 6 As shown, it mainly consists of RU blocks and addition operations between input positions and displacements. The RU block first performs a surface-to-node feature transformation via a learnable network, and then computes displacements to update point positions. For example, the RU block applies a facet-by-face 2-layer MLP with dimensions (128, 128, 131) to an output size of 131. The first 3 entries are used as displacement vectors, and the last 128 entries are used as updated feature outputs. 3D displacements are added to the positions of the point list X of the input mesh.

[0077] The corresponding reverse wrapping (deformation, 750) is in Figure 9 As shown in the figure. This figure is only... Figure 8 The inverse graph. Except for the input of the module, the dimensions are... Figure 8Similar to the modules described earlier, we concatenate 7 fixed features—area (1D), normal vector (3D), and curvature vector (3D)—with a learned feature map of dimension 128. Therefore, the first surface convolutional layer (910) is (135, 128). The remaining surface convolutional layers (930) and 2 MLP layers in the PU block (920, 940) have 128 input, hidden, and output dimensions.

[0078] It should be noted that, in addition to the hyperparameter λ given above, the size of the latent grid (the number of faces) plays a crucial role in determining the bitrate point of the learning-based grid codec. The size of the latent grid is typically determined by the number of faces. Furthermore, the latent grid size can be configured as an input parameter for running the codec. With a smaller latent grid size, the codec tends to operate at a lower bitrate. If higher quality reconstruction is desired, the latent grid size (and coarser faces) should be increased. With a larger latent grid size (and finer faces), more geometric (shape) details are preserved losslessly.

[0079] Encoding of latent grids According to the embodiments, it is proposed to use, such as Figure 10 The novel encoder (730) shown encodes the potential mesh.

[0080] First, based on the input quantization parameter QP, the potential grid M is quantized using a Q-block (1010). L Quantization is performed on a list of points. The reconstructed point locations from quantization / dequantization are determined by... The face list T is losslessly encoded (1020) into a bitstream. The quantized point list... The following process will be followed for encoding. Note that we do not distinguish between quantization points and dequantization points in the document. When referenced for encoding, This represents the quantized value. It is used as a reference when referring to their location. This represents the dequantized value.

[0081] First, the input potential mesh is serialized (1040). Serialization organizes the mesh into... T is a sequence of 1D points, representing the encoded order of the points. Serialization is achieved through a process called edgebreaker. This edgebreaker serialization process operates as a state machine. In each state, it moves in a spiral order from one triangle to a neighboring triangle. Once a neighboring triangle is traversed, the next triangle will be either to the left or right, as the remaining triangles are those traversed in the current state. During this traversal, all visited triangles and boundary vertices are sequentially labeled. Assuming T is lossless encoded, the decoder can reproduce the exact same traversal order. A prediction table is also maintained, indicating which points are used as references for encoding the next point. For a grid, each point in the grid is associated with a position and an index, and each face is defined by three points (e.g., using their indices). In one example, using serialization (e.g., edgebreaker), point indices are stored in an array indicating their encoding order, while the prediction relationships are also stored.

[0082] It is worth noting that the "prediction table" can be virtual and can be generated on the fly when the edgebreaker runs on the fact list T. The exact same prediction table can also be reproduced on the fly by the decoder.

[0083] For organizational sequence For each point in the table (when reference points are available), a prediction code (1030) is applied to encode the location of the next point. First, three reference points are accessed according to the prediction table. In one embodiment, the predicted value X (1030) is calculated based on the following parallelogram equation. P : X P =X0+(X2-X1) This is Figure 11 Further illustration is provided. X0, X1, and X2 are reference points, while X... P These are the prediction points from which we want to calculate the residuals. Once the predictions are calculated, the residual E (1060) is calculated. N =X N -X P , where X N It is the current point, and the current triangle moves from X0X1X2 to the next triangle X0X2X. P .

[0084] Finally, Figure 10 In the middle, for residual E N Perform arithmetic encoding (1050). The arithmetic encoding is guided by the probability distribution parameter set P. The distribution parameters are obtained from the training phase. When the encoding process is invoked, it is assumed that they are available.

[0085] During the training phase, the distribution parameter set P is initialized with random values. When backpropagation is performed during training, the distribution parameter set P is updated in the same way as the other neural network parameters.

[0086] Decoding of latent grids According to an embodiment, the latent grid decoding (740) is in Figure 12 As shown in the figure. From the bitstream BS LIn the first step, we run the arithmetic decoder (1240) to reconstruct the face list T. From the face list, serialization (1250) is performed in a manner similar to that in the encoder. Through serialization, a prediction table is created, and indices are assigned to the points to be decoded.

[0087] Next, the list of points is decoded. The set of distribution parameters P estimated during training is also available at the decoder. It will guide the residual E. N Arithmetic decoding (1210). Then, based on the prediction table, three reference points (X0, X1, X2) are retrieved to calculate (1230) the predicted value X of the current point. P =X0+(X2-X1). Calculate (1220) the current point X. N =X P +E N To generate a quantized list of points Based on the decoded surface list T and the quantized point list. Perform inverse quantization (1260) to obtain the mesh for reconstruction. Dequantized point list (Note that we use the same notation for both quantized and dequantized point lists.) ).

[0088] Utilizing advanced prior latent grid coding exist Figure 13 and 14 In this embodiment, we propose a latent lattice encoder and decoder that utilizes a priori information. Similar to previous implementations, the latent lattice encoder / decoder uses arithmetic encoding guided by a set of probability distribution parameters. However, the parameters of the distribution used in previously proposed encoders / decoders are estimated offline during the training phase. In this embodiment, we propose an advanced method to estimate the distribution parameters on the fly during the inference phase (i.e., during encoding / decoding time).

[0089] exist Figure 13 and 14 An EDP block (1310, 1410) is inserted to estimate the distribution parameter set on the fly. On the encoder side, the EDP (1310) takes residual information as its input and outputs the estimated distribution parameter set. In one embodiment, for a low-latency configuration, the residual information extends up to the current point. In another embodiment, the residual information is used for the entire list of points in a high-coding-efficiency configuration that allows for longer latency. Simultaneously, a bitstream is encoded to represent the distribution parameter set, also known as the hyperprior parameters. A dedicated bitstream is used to encode the hyperprior parameters. The estimated distribution parameters are used to guide the arithmetic encoding of the residual information.

[0090] On the decoder side, the hyperprior parameters (1410) are decoded from the bitstream via DP blocks. The hyperprior parameters (distributed parameter set) are then used to aid in the arithmetic decoding of the residual information.

[0091] Utilizing autoregressive super-prior latent grid coding exist Figure 15 and 16 In the embodiments, we propose a latent mesh encoder and decoder using predictive super-prior. (Compared to...) Figure 13 and 14 Compared to the codec shown, in this embodiment we do not use a bitstream to transmit the prior. Instead, both the encoder and decoder use EDP blocks (1510, 1610) to predict the distribution parameter set. The predictions made by the EDP here use previously encoded points, rather than... Figure 13 The next point to be encoded in the code.

[0092] In one embodiment, we use residual information from three reference points identified in the prediction table to predict the distribution parameters. In the encoder ( Figure 15 ) and decoder ( Figure 16 Both share the EDP design.

[0093] Feature encoder and feature decoder In this document, we provide some example designs of feature encoders and decoders to make the overall codec description complete.

[0094] Figure 17 An example feature encoder design is shown. A feature encoder is essentially a feature extraction and aggregation network. It generates feature maps associated with a base mesh. In this example, we assume the input mesh is a semi-regular mesh, meaning each face in the input mesh has three adjacent faces.

[0095] The feature encoder consists of two basic blocks. One is a surface convolutional block (1710, 1730), and the other is a subdivision-based pooling block (downsampling, 1720, 1740). The surface convolution is defined by the chosen surface convolutional kernel, for example, ... Figure 5 The diagram shows 3-neighborhood kernels, 6-neighborhood kernels, or 9-neighborhood kernels. Subdivision pooling merges the input concatenations into a single concatenation when the first three concatenations are combined. Figure 18 The diagram is shown in the figure. For example, the first surface convolutional layer has dimensions (7, 128), where the input features are the area, normal vector, and curvature vector of M. Subsequent surface convolutional layers all have a size of (128, 128).

[0096] Figure 19An example feature decoder design is shown. The feature decoder is the inverse process of the feature encoder. It takes a feature map and the corresponding underlying mesh as input and outputs a reconstructed mesh. In one embodiment, it consists of two basic blocks. One is a face convolutional block (1910, 1930), and the other is a subdivision-based upsampling block (1920, 1940). Subdivision upsampling is achieved by dividing each existing edge in two and placing a new node at the midpoint of the edge, splitting a single face from the input into four faces as the output, such as... Figure 20 As illustrated in the diagram. For example, the input feature F is again concatenated with the fixed 7-dimensional features (area, normal, curvature) of M. The first surface convolutional layer has a size of (135, 128), and subsequent surface convolutional layers have a size of (128, 128).

[0097] Graph Convolutional Networks (GCNs) When choosing a CNN for feature encoding / decoding and latent grid encoding / decoding, we propose how to use a regular CNN for surface convolution. In another embodiment, a graph convolutional network (GCN) is used to implement the CNN.

[0098] The implementation of CNNs requires a mesh to maintain regular spatial neighborhoods. That is, the mesh must follow a regular pattern in which faces are connected to each other. For example, in the case of a triangular mesh, each face usually always has three adjacent faces, unless they are along the boundaries of the surface.

[0099] Furthermore, running a regular CNN on a manifold mesh is easy, but running it on a non-manifold mesh is not. On a manifold mesh, an edge can appear on at most two faces. However, on a non-manifold mesh, an edge can appear on more than two faces. For example, as... Figure 21 As illustrated, an edge from the intersection of two manifolds can be on at least three faces. The mesh has two manifolds. One manifold has faces (X0, X1, X2) and (X2, X3, X0), while the other manifold has face (X0, X2, X4). Therefore, the edge (X0, X2) is used on all three faces.

[0100] When conventional CNNs struggle in such situations, we propose using Graph Convolutional Networks (GCNs) to perform convolutional computations. Such GCNs can support more flexible spatial relationships between adjacent faces (or points). It first constructs a graph structure describing the spatial topology of the faces. Then, convolutions are computed along the edges of the graph. GCNs are a known alternative to conventional CNNs.

[0101] Various numerical values ​​are used in this application. Specific values ​​are for illustrative purposes, and the aspects described are not limited to these specific values.

[0102] Various methods are described herein, and each method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Furthermore, terms such as "first," "second," etc., may be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless specifically required, the use of such terms does not imply a sequence of modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding and can occur, for example, before, during, or within a time period overlapping with the second decoding.

[0103] The implementations and aspects described herein can be implemented, for example, in a method or process, apparatus, software program, data stream, or signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the discussed features can also be implemented in other forms (e.g., apparatus or program). Apparatus can be implemented, for example, in suitable hardware, software, and firmware. Methods can be implemented, for example, in an apparatus, such as a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate the transfer of information between end users.

[0104] References to “an embodiment” or “an embodiment” or “an implementation” or “implementation”, and other variations thereof, mean that the specific features, structures, characteristics, etc., described in connection with the embodiment are included in at least one embodiment. Therefore, the phrases “in an embodiment” or “in an embodiment” or “in an implementation” or “in an implementation” appearing throughout this application, and any other variations thereof, do not necessarily refer to the same embodiment.

[0105] Furthermore, this application may involve "determining" fragments of various information. Determining information may include one or more of, for example, estimation information, calculation information, prediction information, or information retrieved from memory.

[0106] Furthermore, this application may relate to “accessing” fragments of various information. Accessing information may include one or more of the following: receiving information, retrieving information (e.g., retrieving information from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0107] Furthermore, this application may relate to "receiving" fragments of various information. As with "access," receiving is intended to be a broad term. Receiving information may include one or more of, for example, accessing information or retrieving information (e.g., retrieving information from memory). Moreover, "receiving" is generally referred to in one or more ways during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0108] To be understood, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one of…” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). As will be clear to those skilled in the art and related fields, this can be extended to as many items as possible listed.

[0109] As will be apparent to those skilled in the art, implementations can generate various signals that are formatted to carry, for example, information that can be stored or transmitted. The information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, the signal may be formatted to carry a bitstream of the described embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding the data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that signals can be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

Claims

1. A method for decoding a grid, comprising: Decode the feature map; Decoding the potential grid includes: Decode the face list of the potential mesh. Obtain the predicted point list of the potential grid's point list. Decode the residual information associated with the list of points of the potential grid, and Based on the residual information and the predicted point list of the point list of the potential grid, the point list of the potential grid is reconstructed; as well as The grid is decoded based on the decoded feature map, and the latent grid is decoded based on the first neural network.

2. The method according to claim 1, further comprising: The decoded latent mesh is transformed into a reconstructed base mesh, wherein the mesh is decoded based on the decoded feature map and the reconstructed base mesh.

3. The method of claim 2, wherein the face list of the potential mesh is used as the face list of the base mesh of the reconstruction.

4. The method according to claim 2 or 3, wherein the transformation of the decoded potential grid comprises: The second neural network is used to aggregate features on the surface of the decoded latent mesh; as well as A third neural network is used to update the point positions of the decoded latent grid.

5. The method of claim 4, wherein the third neural network includes a residual update block, the residual update block performing the following operations: Transforming surfaces into node features via a learnable network; and Displacement information is obtained to update the point positions of the decoded potential mesh.

6. The method of claim 5, wherein the residual update block comprises a face-by-face MLP (multilayer perceptron).

7. The method according to any one of claims 4-6, wherein the second neural network comprises a convolutional neural network or a graph convolutional network.

8. The method according to any one of claims 1-7, wherein obtaining the list of predicted points comprises: The decoded face list of the potential grid is serialized to obtain an ordered list of points and a prediction table is generated, wherein points are predicted based on previous points in the ordered list of points indicated by the prediction table.

9. The method according to any one of claims 1-8, wherein decoding the residual information associated with the list of points of the potential mesh comprises: The probability distribution parameters are decoded, wherein the residual information associated with the list of points of the potential grid is decoded based on the probability distribution parameters.

10. The method according to any one of claims 1-8, wherein decoding the residual information associated with the list of points of the potential mesh comprises: The probability distribution parameters are obtained based on the previously decoded residual information, wherein the residual information of the point list of the potential grid is decoded based on the probability distribution parameters.

11. The method according to any one of claims 1-10, wherein the list of points for reconstructing the potential mesh comprises: The point list of the potential grid is inverse quantized.

12. A method for encoding a grid, comprising: The feature map and the basic grid corresponding to the grid are obtained based on the fourth neural network; The feature map is encoded; Transform the basic mesh into a potential mesh; as well as Encoding the potential mesh includes: The face list of the potential mesh is encoded. Obtain the predicted point list of the potential grid's point list. Obtain residual information associated with the point list, and The residual information of the point list is encoded.

13. The method of claim 12, wherein the face list of the base mesh is used as the face list of the potential mesh.

14. The method of claim 12 or 13, wherein transforming the base mesh comprises: Features on the surface of the base mesh are aggregated based on the fifth neural network; as well as The point positions of the base grid are updated based on the sixth neural network to form the potential grid.

15. The method of claim 14, wherein the sixth neural network includes a residual update block, the residual update block performing the following operations: Transforming surfaces into node features via a learnable network; and Displacement information is obtained to update the point positions of the base mesh.

16. The method of claim 15, wherein the residual update block comprises a face-by-face MLP (multilayer perceptron).

17. The method according to any one of claims 14-16, wherein the fifth neural network comprises a convolutional neural network or a graph convolutional network.

18. The method according to any one of claims 12-17, further comprising obtaining the size of the potential mesh.

19. The method according to any one of claims 12-18, wherein obtaining the list of predicted points comprises: Quantize the list of points in the potential grid; as well as The list of faces of the potential mesh is serialized to obtain an ordered list of points and a prediction table is generated, wherein the points are predicted based on previous points in the ordered list of points indicated by the prediction table.

20. The method according to any one of claims 12-19, wherein encoding the residual information comprises: The probability distribution parameters are encoded, wherein the residual information of the point list is encoded based on the probability distribution parameters.

21. The method of claim 20, wherein the probability distribution parameter is obtained based on the residual information.

22. The method according to any one of claims 12-19, wherein encoding the residual information associated with the list of points comprises: The probability distribution parameters are obtained based on previously encoded residual information, wherein the residual information of the point list is encoded based on the probability distribution parameters, and wherein the probability distribution parameters are not transmitted to the decoder.

23. The method according to any one of claims 12-22, further comprising: Obtain parameters that control the code rate used to encode the potential grid and the trade-off between the distortion of the potential grid and the base grid; The bitrate is estimated based on an entropy bottleneck layer; as well as The wrapping network is trained based on the parameters controlling the tradeoffs, the bit rate, and the distortion, wherein the wrapping network is used to transform the base grid into the latent grid.

24. The method of claim 23, further comprising: During the training, probability distribution parameters are obtained, wherein the residual information of the point list is encoded based on the probability distribution parameters.

25. An apparatus comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to perform the method according to any one of claims 1-24.

26. A non-transitory computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1-24.

Images