Entropy bottleneck layer for learning-based mesh compression

Abstract

As a type of information, connectivity in a mesh poses challenges not only in designing an efficient AI architecture, but also in training a learning-based compression system for the mesh. In one implementation, we present the entropy bottleneck layer as an enabler by allowing training of an end-to-end learning-based compression system for images / video / point clouds. In particular, we present ways of modeling the distortion introduced by compression. We also present ways of modeling the distribution of mesh points and estimating the rate required to encode the mesh.

Description

Technical Field

[0001] This embodiment generally relates to methods and apparatus for 3D mesh compression. Background Technology

[0002] 3D point clouds are a versatile data format that spans several business areas, including autonomous driving, robotics, augmented reality / virtual reality (AR / VR), civil engineering, computer graphics, and animation / film. 3D LiDAR (light detection and ranging) sensors have been widely deployed in autonomous vehicles. Affordable LiDAR sensors have been released by companies such as Velodyne Velabit, Apple iPad Pro 2020, and Intel RealSense LiDAR Camera L515. With significant advancements in sensing technology, 3D point cloud data has become more practical than ever before and is expected to be the ultimate enabler for the aforementioned applications.

[0003] Point cloud data is also considered to consume a significant portion of network traffic, for example, in connected cars and immersive communications (VR / AR) over 5G networks. Understanding and communicating point clouds will inherently lead to efficient representation formats. In particular, raw point cloud data needs to be properly organized and processed for world modeling and sensing purposes. Furthermore, point clouds can represent continuous scans of the same scene containing multiple moving objects. These are called dynamic point clouds compared to static point clouds captured from static scenes or objects. Dynamic point clouds are typically organized into frames, where different frames are captured at different times.

[0004] Meshes are another data format consisting of both points (also called vertices) and faces. Compared to point clouds, meshes are assumed to be a complete model of a 3D surface, while the points in a point cloud are merely a set of discretized samples of the surface.

[0005] For computer graphics (CG), data is typically available in a mesh format because it is generated by the content provider. For data acquired by sensors (including LiDAR), the raw data is usually in point cloud format. For point cloud data, a mesh can be created via a meshing procedure. Summary of the Invention

[0006] According to one embodiment, a method for training a learning-based grid compression system with an entropy bottleneck layer is proposed, the method comprising: obtaining a grid; adjusting the grid to obtain an adjusted grid; performing serialization on the grid or the adjusted grid to obtain an ordered list of vertices; for vertices in the ordered list of vertices: predicting the vertices to obtain predicted positions, obtaining a prediction error between the position of the vertex and the predicted position, and obtaining a bitrate for encoding the prediction error based on probability distribution parameters of the prediction error; obtaining a loss value based on the bitrate; and updating the parameters of a neural network used for encoding and decoding the grid based on the loss value.

[0007] According to another embodiment, an apparatus for training a learning-based mesh compression system with an entropy bottleneck layer is proposed. The apparatus includes one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to: obtain a mesh; adjust the mesh to obtain an adjusted mesh; perform serialization on the mesh or the adjusted mesh to obtain an ordered list of vertices; for vertices in the ordered list of vertices: predict the vertices to obtain predicted positions, obtain a prediction error between the position of the vertex and the predicted position, and obtain a bitrate for encoding the prediction error based on a probability distribution parameter of the prediction error; obtain a loss value based on the bitrate; and update the parameters of a neural network used for encoding and decoding the mesh based on the loss value.

[0008] In one embodiment, the loss value is further based on the distortion between the mesh and the reconstructed version of the adjusted mesh.

[0009] In one embodiment, the acquisition of the bitrate is performed for each of the plurality of vertices in the ordered vertex list.

[0010] In one embodiment, the prediction of the vertex is based on previous vertices in the ordered vertex list.

[0011] In one embodiment, the prediction of the vertex is based on a prediction table of the ordered list of vertices, wherein the prediction table indicates which vertices are used as references to the vertex.

[0012] In one embodiment, the serialization is based on an edgebreaker.

[0013] In one embodiment, the adjustment of the mesh is performed by adding noise to the mesh.

[0014] In one embodiment, uniform noise with variance is added as said noise, and the bit rate of said prediction error is further obtained based on said variance.

[0015] In one embodiment, the adjustment of the mesh is performed by quantizing the mesh.

[0016] In one embodiment, a quantization step size is used to quantize the grid, and the bit rate of the prediction error is further obtained based on the quantization step size.

[0017] In one embodiment, quantization is applied during the forward computation of training and bypassed during the backpropagation of training.

[0018] In one embodiment, the method further includes: decomposing an input grid into a base grid and a feature map, wherein the base grid corresponds to the grid. The method may further include extracting the feature map and the base grid from the input grid; generating another version of the base grid from the entropy bottleneck layer; and decoding the other version of the grid to form a decoded grid.

[0019] In one embodiment, the loss value is further based on the distortion between the input grid and the decoded grid.

[0020] In one embodiment, the method includes: obtaining a probability distribution parameter set based on the prediction error, wherein the probability distribution parameter set is used to obtain the bitrate. The method may further include obtaining a hyperparameter set describing the probability distribution parameter set; and obtaining another bitrate based on the hyperparameter set.

[0021] In one embodiment, obtaining the probability distribution parameter set includes: extracting features from the prediction error; quantizing the output from the extracted features to form the hyperparameter set; and applying a feature decoder to the hyperparameter set to obtain the parameter set.

[0022] One or more embodiments also provide a computer program including instructions that, when executed by one or more processors, cause the one or more processors to perform an encoding or decoding method according to any embodiment described herein. One or more embodiments in this document also provide a computer-readable storage medium having instructions stored thereon for encoding or decoding point cloud data according to the methods described herein.

[0023] One or more embodiments also provide a computer-readable storage medium storing point cloud data generated according to the method described above. One or more embodiments also provide a method and apparatus for transmitting or receiving point cloud data generated according to the method described herein. Attached Figure Description

[0024] Figure 1 The diagram illustrates a block diagram of a system in which various aspects of this embodiment can be implemented.

[0025] Figure 2 The diagram illustrates a learning-based compression system.

[0026] Figure 3 The diagram illustrates the training of a learning-based compression system.

[0027] Figure 4 The diagram illustrates the entropy bottleneck layer.

[0028] Figure 5 The diagram illustrates a learning-based grid compression system.

[0029] Figure 6 The diagram illustrates the encoding of the basic grid.

[0030] Figure 7 The diagram illustrates a learning-based grid compression system trained using the proposed grid entropy bottleneck layer according to one embodiment.

[0031] Figure 8 The diagram illustrates an entropy bottleneck layer for a grid compression system (EN) according to one embodiment.

[0032] Figure 9 An illustration is presented showing the generation of a parallelogram-based predictor according to one embodiment.

[0033] Figure 10 The diagram illustrates an entropy bottleneck layer for a grid compression system (EQ) according to one embodiment.

[0034] Figure 11 The diagram illustrates an entropy bottleneck layer for a grid compression system (LN) according to one embodiment.

[0035] Figure 12 The diagram illustrates an entropy bottleneck layer for a grid compression system (LQ) according to one embodiment.

[0036] Figure 13 The diagram illustrates an entropy bottleneck layer for a grid compression system with distributed parameter estimation according to one embodiment.

[0037] Figure 14 The diagram illustrates an entropy bottleneck layer for a grid compression system with distributed parameter estimation according to another embodiment.

[0038] Figure 15 The diagram illustrates the design of an EDP according to one embodiment. Detailed Implementation

[0039] Figure 1 The diagram illustrates an example of a system in which various aspects and embodiments may be implemented. System 100 may be embodied as a device including the various components described below and configured to perform one or more of the aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 100 may be embodied individually or in combination in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more of the aspects described in this application.

[0040] System 100 includes at least one processor 110 configured to execute instructions loaded thereon to implement various aspects, such as those described in this application. Processor 110 may include embedded memory, input / output interfaces, and various other circuitry as known in the art. System 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). System 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 140 may include internal storage devices, attached storage devices, and / or network-accessible storage devices.

[0041] System 100 includes an encoder / decoder module 130 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents one or more modules that can be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Furthermore, as is known to those skilled in the art, the encoder / decoder module 130 may be implemented as a separate element of system 100, or it may be incorporated within processor 110 as a combination of hardware and software.

[0042] Program code to be loaded onto processor 110 or encoder / decoder 130 to execute the various aspects described in this application may be stored in storage device 140 and subsequently loaded onto memory 120 for execution by processor 110. According to various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more items of various kinds during the execution of the processes described in this application. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0043] In several embodiments, memory within processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be processor 110 or encoder / decoder module 130) is used for one or more of these functions. External memory may be memory 120 and / or storage device 140, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, fast external volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2, JPEG Pleno, MPEG-I, V-DMC, HEVC, or VVC.

[0044] Inputs to the components of system 100 can be provided by various input devices, as indicated in block 105. Such input devices include, but are not limited to: (i) an RF section that receives, for example, RF signals transmitted over the air by a broadcasting device, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0045] In various embodiments, as known in the art, the input device of block 105 has associated respective input processing elements. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting the signal band to a band), (ii) down-converting the selected signal, (iii) further band-limiting to a narrower band to select, for example, a signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements to perform these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0046] Additionally, the USB and / or HDMI terminals may include their respective interface processors for connecting system 100 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) can be implemented as needed, for example, within a separate input processing IC or within processor 110. Similarly, various aspects of USB or HDMI interface processing can be implemented as needed, either within a separate interface IC or within processor 110. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 110 and encoder / decoder 130 operating in conjunction with memory and storage elements, to process the data streams as needed for presentation on the output device.

[0047] Various components of system 100 can be housed within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data between them using a suitable connection arrangement 115 (e.g., internal buses known in the art, including I2C buses, wiring, and printed circuit boards).

[0048] System 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 190 may be implemented, for example, within a wired and / or wireless medium.

[0049] In various embodiments, data is streamed to system 100 using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signals in these embodiments are received via a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that delivers data via an HDMI connection to input block 105 to provide streaming data to system 100. Still other embodiments use an RF connection to input block 105 to provide streaming data to system 100.

[0050] System 100 can provide output signals to various output devices, including display 165, speaker 175, and other peripheral devices 185. In various examples of embodiments, other peripheral devices 185 include one or more of a standalone DVR, disc player, stereo system, lighting system, and other devices that provide functionality based on the output of system 100. In various embodiments, signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention is used to communicate control signals between system 100 and display 165, speaker 175, or other peripheral devices 185. Output devices can be communicatively coupled to system 100 via dedicated connections through their respective interfaces 160, 170, and 180. Alternatively, output devices can be connected to system 100 via communication interface 150 using communication channel 190. Display 165 and speaker 175 can be integrated into a single unit with other components of system 100 in electronic devices, such as televisions. In various embodiments, display interface 160 includes a display driver, such as a timing controller (TCon) chip.

[0051] For example, if the RF portion of input 105 is part of a separate set-top box, then display 165 and speaker 175 may alternatively be separate from one or more other components. In various embodiments where display 165 and speaker 175 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0052] Point cloud / mesh use cases The automotive industry and autonomous vehicles are areas where point clouds can be used. Autonomous vehicles should be able to "detect" their environment to make sound driving decisions based on the realities of their surroundings. Typical sensors like LiDAR generate (dynamic) point clouds used by perception engines. These point clouds are not intended to be seen by the human eye, and they are typically sparse, not necessarily colored, and dynamic with a high capture frequency. They can possess other properties, such as reflectivity, as provided by LiDAR, which indicates the material of the sensed object and can aid in decision-making.

[0053] Virtual reality (VR) and immersive worlds are envisioned by many as the future of 2D video. The basic idea is to immerse viewers in all of their surroundings—as opposed to standard TV, where viewers can only see the virtual world in front of them. Immersion exists in several levels, depending on the viewer's degree of freedom within the environment. Point clouds and meshes are both good format candidates for distributing VR content. They can be static or dynamic and are typically moderately sized, containing no more than a few million points at a time.

[0054] Point clouds and meshes can also be used for various purposes, such as cultural heritage / civil engineering, where objects like statues or buildings are 3D scanned to share their spatial configuration without moving or accessing them. Furthermore, this is a way to ensure the preservation of knowledge about an object in case it might be damaged (e.g., in case a temple is destroyed by an earthquake). Such point clouds and meshes are typically static, colored, and massive.

[0055] World modeling and sensing via point clouds / meshes may be a fundamental technique that allows machines to gain knowledge about the 3D world around them, which is crucial for the applications discussed in this article.

[0056] 3D point cloud data is essentially a discrete sample of the surface of an object or scene. To fully represent the real world using point samples, a massive number of points are actually needed. For example, a typical VR immersive scene contains millions of points, while a point cloud typically contains hundreds of millions. Therefore, processing such large-scale point clouds is computationally expensive, especially for consumer devices with limited computing power, such as smartphones, tablets, and car navigation systems. Furthermore, the discrete samples comprising 3D point cloud data still contain incomplete information about the underlying surfaces of objects and scenes. Therefore, efforts are currently underway to explore mesh representations for 3D scene / surface representation.

[0057] A mesh can be thought of as a 3D point cloud, accompanied by information about the connectivity between points. Therefore, a mesh representation bridges the gap between the point cloud and the underlying continuous surface through approximations of local 2D polygon patches (called faces) on the underlying surface. In another perspective, a mesh can simplify a point cloud by reducing the number of points. For example, faces in a mesh might be suitable for representing flat areas, while a point cloud might require dense patches of points to represent the same flat area.

[0058] The first step in any processing or inference of mesh data is having an efficient storage method. To store and process meshes at an affordable computational cost, one solution is to first downsample them, where the downsampled mesh summarizes the geometry of the input mesh while having far fewer (but larger) faces. The downsampled mesh is then fed into subsequent machine tasks for further consumption. However, further reductions in storage space can be achieved by converting the raw mesh data (original or downsampled) into fixed-length codewords or feature maps existing on a very low-resolution mesh. This codeword or feature map can be converted into a bitstream using existing entropy coding techniques. Furthermore, the codeword or feature map itself is useful, as it represents global or local surface information of the underlying scene / object, respectively, and can be paired with subsequent downstream (machine vision) tasks.

[0059] Learning-based methods have been used in encoding several types of data, such as images, videos, and point clouds. Figure 2 The paper describes a typical learning-based compression system. Input signal PC The data (whether it's an image, video, or point cloud) is first sent to a feature extraction block (210) consisting of learnable neural network layers. Then, it is processed via ENC. F Feature maps generated from block pairs F Perform entropy encoding (220). In ENC F Within the block, it is entropy-encoded into a bit stream. BS Previously, feature maps F First, it is rounded to an integer or quantized to an integer. On the decoder side, it is handled by DEC. F Entropy decoding (2^30) is performed on the block-to-block bitstream. Decoded feature map. The image, video, or point cloud is sent to the feature decoder (240) to reconstruct the input image, video, or point cloud. Such an end-to-end compression system is called a deep feature-based codec.

[0060] For a neural network-based system to be trained, all blocks essentially need to be differentiable. Differentiation capability is required during the process known as backpropagation to update the network parameters. However, the rounding or quantization operations during entropy encoding are not differentiable.

[0061] To solve the differentiability problem of entropy coding, such as Figure 3 As shown, an entropy bottleneck layer is introduced to estimate the entropy of the feature map to be encoded. The entropy bottleneck layer (320, EB block) will replace the ENC in the compression system during the training phase. F and DEC F Of course, during the inference phase, ENC F and DEC F The feature maps are brought back to achieve the actual encoding purpose. The introduced EB block takes the feature maps generated by the feature extractor (310) as input. It will first output the distorted feature maps. It has some noise added to it. The EB block also outputs an estimated code rate (entropy) of the feature map. The entropy bottleneck layer is differentiable by its design. Finally, by adjusting the input signal... PC The reconstructed signal output by the feature decoder (330) The training of the compression system is supervised by jointly counting the distortion between the data points and the estimated bit rate required to encode the feature maps.

[0062] In one example, the entropy bottleneck layer is designed as follows: Figure 4 As shown in the diagram, some uniform noise is injected (410) into the input feature map. F The noise level is controlled by input parameters (e.g., variance σ). The feature map output from the EB block is distorted due to noise. In parallel, feature maps F It is sent to the DR (distribution to rate) block (420). Except F In addition, the DR block will also distribute the parameter set. P The variance σ is taken as input, and the output is the feature map. F The estimated code rate required for encoding. Note the distribution parameter set. P For input features F The probability distribution is modeled. It will be iteratively updated to minimize the bitrate during training. Note that the two blocks (410, 420) are not constructed using neural network layers, but rather using a deterministic procedure. However, the distribution parameter set... P It contains parameters to be learned during training and is updated via backpropagation during the training phase.

[0063] In the following section, we propose an entropy bottleneck to enable the training of a learning-based grid compression system.

[0064] A typical learning-based grid compression system.

[0065] Figure 5The diagram illustrates a typical learning-based grid compression system consisting of two branches. The main branch of the feature encoding function and... Figure 2 This is similar to typical learning-based compression. In the main branch, the encoder extracts and aggregates features from the input grid using a feature extraction block (510). The extracted feature maps F Sent to entropy encoder ENC F (530). By entropy decoder DEC F (550) Decoded feature map The input mesh is sent to the feature decoder (560) to reconstruct the mesh. Using feature extraction / aggregation, the input mesh is typically downsampled to a base mesh with a lower resolution. M This is beneficial for compression. Feature map F With the base grid M Related.

[0066] Figure 5 The second branch of medium-grid compression is based on the base grid. M Encoding is performed. It is handled by the ENC used for encoding. M Block (520) and DEC for decoding M The grid consists of blocks (540). Encoding and decoding the basic mesh involves encoding a list of faces. In the case of a triangular mesh, each face is represented by a list of three connected points. Each point is described by 3D coordinates. In one example, we let... ,in X It is a list of points, and T It is a list of faces. Each point in a face is determined by its position on the face. X The points are represented by indices. To encode the basic grid, the point list needs to be... X Dough Kneading List T Encode it.

[0067] like Figure 6 The diagram illustrates the encoding of a base mesh (or the original mesh in another embodiment). The pipeline consists of two sub-branches. The first sub-branch is a face list. T The encoding. This is done by the arithmetic encoder AE on the encoder side. T Block (630) and the arithmetic decoder AD on the decoder side T The block (650) is simply completed. The encoded surface is the bitstream BS. M Part of it.

[0068] The second subbranch is a list of dots. X The encoding. The position of each point in the list is determined by AE. X Before block (620) is arithmetic encoded, by Q X Block (610) is quantized. The quantized points are... This is represented by the decoder. The decoder first uses AD... X Block (640) performs arithmetic decoding on the bitstream, and then passes it through Q. -1 X Block (660) is dequantized. The encoded points are bitstream BS. M Part of it.

[0069] Entropy bottleneck for grid coding like Figure 3 As shown, an entropy bottleneck layer is needed for training. Figure 2 A learning-based compression system. In one embodiment, in order to train... Figure 5 In the learning-based grid compression system, we propose... Figure 7 The diagram shown.

[0070] Corresponding to Figure 5 The two encoding branches insert two entropy bottleneck layers between the feature extractor (710) and the feature decoder (740). The first entropy bottleneck layer is EB. F (730) functions and Figure 3 It is the same as the entropy bottleneck layer EB in [the context of the text]. It outputs reconstructed features. and the estimated code rate used to encode the feature map R F A second grid entropy bottleneck layer, EB, is proposed. M (720) is responsible for encoding the basic mesh. The mesh entropy bottleneck layer takes the mesh as input and outputs the reconstructed mesh. and the estimated bitrate of the grid R M At the estimated bitrate R F and R M When available, the loss function consists of a weighted sum of the total bitrates ( R F + R M The loss is defined by the rate-distortion loss, which is measured between the input grid and the decoded grid. The training of the learning-based grid codec is then supervised by a joint rate-distortion loss. Because the underlying grid has a data structure unique to the feature map, existing entropy bottleneck layers cannot be applied.

[0071] Figure 8 A diagram depicts the proposed mesh entropy bottleneck layer according to one embodiment. Note the face list. T The encoding is typically lossless and consumes a fixed bit rate given the same input base grid. Therefore, there is no need to count the bit rate required to encode the list of faces. In the proposed graph, we only need to be concerned with encoding the list of points.

[0072] First, the noise adder block (810) is used to add uniform noise to the point list of the input mesh. It outputs a mesh that is distorted by noise. ,in It is a list of points whose positions can be adjusted.

[0073] Second, the input mesh is serialized (820). Serialization organizes the mesh into a 1D point sequence. This represents the encoding order of the points. Serialization is achieved through a process called edge shredder. This edge shredder serialization process works as a state machine. In each state, it moves from one triangle to an adjacent triangle. Once an adjacent triangle has been traversed, the next triangle will be either the left or right triangle, because the remaining triangle is one that has been traversed in the current state. During this traversal, all visited triangles and boundary vertices are marked sequentially. Assume... T Lossless encoding allows the decoder to recover the exact same sequence. Organized sequences. It also maintains a "prediction table," which indicates which points are used as references for encoding the next point. It's worth noting that when the edge trimmer is in the fact list... T When running, the "prediction table" can be virtual and generated on the fly.

[0074] For organized sequences For each point in the prediction table (with available references), the required code rate for encoding is estimated as follows: Three reference points are accessed according to the prediction table. In one embodiment, the prediction block (830) calculates the predictor according to the following parallelogram equation. X P : .

[0075] This is Figure 9 Further illustrations are provided. X 0、 X 1 and X 2 is the reference point, and X P The prediction point is where we want to calculate the error (840). Deformation from the predicted point, where X N This is the current point. Once predicted, the current triangle... X 0 X 1 X 2. Move to the triangle X 0 X 2 X NThe error is then sent to the DR block (850). This is based on the estimated set of probability distribution parameters. P The variance σ used to generate noise is estimated by the DR block (850). E N Encoding bitrate R N By estimating and counting the bitrate of all points, a list of points was output. R Total bitrate required.

[0076] In this document, the proposed grid entropy bottleneck layer is referred to as Instance EN (Early Split of Branch with Noise Adder).

[0077] DR block The purpose of the DR block is to estimate the bit rate / entropy based on distribution parameters. (Assumption error) The bitrate consists of errors encoded along the x-, y-, and z- coordinates. R N It can be estimated as the total entropy of the error. , in It has the following characteristics: e i The probability mass function for estimating the error at the center. It can be calculated as the following integral: , Here, Let represent the probability density function, which is parameterized by a parameter set P (learned via backpropagation during the training process). In one embodiment, the probability distribution... It can be a Gaussian distribution. In this case, the parameter set consists of the following two numbers: mean and variance.

[0078] It is important to note that, according to Figure 4 When the noise σ increases, the output feature map The quality is degraded. However, the interval of the integral in equation (3) (i.e., It also got bigger. Therefore, Becoming larger results in a smaller bitrate. R N Therefore, the entropy bottleneck layer can successfully model the trade-off between bit rate and distortion.

[0079] More embodiments of the proposed entropy bottleneck layer In the following sections, we describe several variations of the grid entropy bottleneck layer proposed above.

[0080] Figure 10 The first embodiment is shown. It is referred to as embodiment EQ (early split with quantized branches). With Figure 8 Compared to the embodiment EN shown, the noise adder (810) is replaced by a quantization block (1010) that performs both forward vectorization and inverse quantization. Due to this change, when estimating the bit rate, the DR module takes the quantization step size QP instead of the noise variance σ as input. Like the noise variance σ, QP is provided as another type of hyperparameter indicating the level of compression; a larger quantization step size (or noise variance σ) results in greater compression and distortion. Clearly, the bit rate depends on the quantization step size (or noise variance σ).

[0081] Figure 11 The second embodiment is shown. It is referred to as embodiment LN (late split of a branch with a noise adder). With Figure 8 Compared to the embodiment EN shown, the starting position of the rate estimation branch is moved from before the noise adder (1110) to after the noise adder (1110). Except that the distribution to the rate estimation DR block does not take the noise variance σ as input, the rest of the processing graph remains essentially unchanged. However, the derivation of the DR block still follows the previous reference... Figure 8 The basic principles described.

[0082] Figure 12 The third embodiment is shown. It is called embodiment LQ (late split with quantized branch). The difference from embodiment LN is that the noise adder (1110) is replaced by a quantization block (1210). This embodiment exhibits good performance with a simpler design.

[0083] Using the EQ and LQ examples, a quantization block is introduced that is non-differentiable and poses a challenge to backpropagation during training. In the diagram, we use dashed lines to bypass the quantization block Q. This means that quantization is applied during the forward computation of training. However, the quantization step is bypassed during the backpropagation of training.

[0084] Entropy bottleneck layer with distributed parameter estimation block In the following sections, we describe an additional embodiment with additional blocks to estimate the distribution parameters (EDP) within the proposed entropy bottleneck layer.

[0085] Figure 13 The first embodiment is illustrated in the figure. (Compared to...) Figure 12 different, Figure 13 parameter set in P This is the output of the EDP block (1310). Specifically, the EDP block will output the error. E N The parameter set P is taken as input and output as an estimated set of parameters. In one embodiment, the EDP block consists of a neural network layer similar to the feature extraction (FE) block, which digests the prediction error. EN And generate parameter set P .

[0086] exist Figure 14 The second embodiment is provided, which is related to Figure 13 Compared to more advanced embodiments, in this embodiment, EDP (1410) will... E N Take as input and output two sets of parameters. P and P ',in P 'is a description P The hyperparameter set. Utilizing P ', can reproduce parameter set P Then, the hyperparameter set. P 'Additionally fed to another rate estimation block DR' (1420) to estimate P The bitrate of ' is expressed as R P ).here P "is a description" P The distribution parameter set of ', and it will be learned via backpropagation during training. In this case, the output bitrate R It is all R N and R P Total bitrate.

[0087] It should be noted that, although in Figure 13 and Figure 14 The quantization block is used to distort the input grid, but a noise adder can be used instead of the quantization block.

[0088] In one embodiment, an EDP block consists of three steps, such as... Figure 15 As shown in the diagram. The first step applies a neural network layer, such as from... E N The feature extractor (1510) is then fed into the quantizer Q (1520), resulting in the hyperparameter set. P Then, in the final step, another set of neural network layers (such as the feature decoder (1530)) will... P 'Take as input and output parameter set' P .

[0089] Various numerical values ​​are used in this application. Specific values ​​are for illustrative purposes only, and the aspects described are not limited to these specific values.

[0090] This document describes various methods, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Furthermore, terms such as "first," "second," etc., can be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding." Unless specifically required, the use of such terms does not imply an ordering of the modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding and can occur, for example, before, during, or in a time period overlapping with the second decoding.

[0091] The implementations and aspects described herein can be implemented, for example, in a method or process, apparatus, software program, data stream, or signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the features discussed can be implemented in other forms (e.g., apparatus or program). An apparatus can be implemented, for example, in suitable hardware, software, and firmware. For example, a method can be implemented, for example, in an apparatus, such as a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0092] References to “an embodiment” or “an embodiment” or “an implementation” or “an implementation”, and other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the appearance of the phrase “in an embodiment” or “in an embodiment” or “in an implementation” or “in an implementation”, and any other variations appearing throughout the various places of this application, do not necessarily refer to the same embodiment.

[0093] Additionally, this application may refer to "determining" various information pieces. Determining information may include, for example, one or more of the following: estimation information, calculation information, prediction information, or information retrieved from memory.

[0094] Furthermore, this application may refer to "accessing" various information fragments. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or one or more of these.

[0095] Additionally, this application may refer to "receiving" various pieces of information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) one or more. Furthermore, "receiving" is generally referred to in one way or another during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0096] It will be understood that the use of any of the following “ / ”, “and / or”, and “at least one” (e.g., in the cases of “A / B”, “A and / or B”, and “at least one of A and B”) is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many items as listed.

[0097] As will be apparent to those skilled in the art, implementations can generate various signals, which are formatted to carry information, for example, that can be stored or transmitted. This information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of a spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, signals can be transmitted via various wired or wireless links. Signals may be stored on a processor-readable medium.

Claims

1. A method for training a learning-based grid compression system with an entropy bottleneck layer, the method comprising: Obtain the grid; Adjust the grid to obtain the adjusted grid; Perform serialization on the mesh or the adjusted mesh to obtain an ordered list of vertices; For the vertices in the ordered vertex list: Predict the vertex to obtain the predicted position. Obtain the prediction error between the position of the vertex and the predicted position, and The code rate for encoding the prediction error is obtained based on the probability distribution parameters of the prediction error. The loss value is obtained based on the bitrate. as well as Based on the loss value, the parameters of the neural network used to encode and decode the grid are updated.

2. The method according to claim 1, wherein, The loss value is further based on the distortion between the mesh and the reconstructed version of the adjusted mesh.

3. The method according to claim 1 or 2, wherein, The acquisition of the bitrate is performed for each of the multiple vertices in the ordered vertex list.

4. The method according to any one of claims 1-3, wherein, The prediction of the vertex is based on previous vertices in the ordered vertex list.

5. The method according to any one of claims 1-4, wherein, The prediction of the vertex is based on a prediction table of the ordered vertex list, wherein the prediction table indicates which vertices are used as references for the vertex.

6. The method according to any one of claims 1-5, wherein, The serialization is based on an edge trimmer.

7. The method according to any one of claims 1-6, wherein, The adjustment of the mesh is performed by adding noise to the mesh.

8. The method according to claim 7, wherein, Uniform noise with variance is added as said noise, and wherein the bit rate of said prediction error is further obtained based on said variance.

9. The method according to any one of claims 1-6, wherein, The adjustment of the mesh is performed by quantizing the mesh.

10. The method according to claim 9, wherein, A quantization step size is used to quantize the grid, and the bit rate of the prediction error is further obtained based on the quantization step size.

11. The method according to claim 9 or 10, wherein, Quantization is applied during the forward computation of training, and quantization is bypassed during the backpropagation of training.

12. The method according to any one of claims 1-11, further comprising: The input mesh is decomposed into a base mesh and a feature map, wherein the base mesh corresponds to the mesh.

13. The method of claim 12, further comprising: Extract the feature map and the base grid from the input grid; Another version of the base mesh is generated from the entropy bottleneck layer; as well as The other version of the mesh is decoded to form a decoded mesh.

14. The method according to claim 13, wherein, The loss value is further based on the distortion between the input grid and the decoded grid.

15. The method according to any one of claims 2-14, further comprising: A probability distribution parameter set is obtained based on the prediction error, wherein the probability distribution parameter set is used to obtain the bit rate.

16. The method of claim 15, further comprising: Obtain the hyperparameter set describing the probability distribution parameter set; as well as Another bitrate is obtained based on the set of hyperparameters.

17. The method according to claim 15 or 16, wherein, The obtained probability distribution parameter set includes: Extract features from the prediction error; The output from the extracted features is quantized to form the hyperparameter set; and The feature decoder is applied to the hyperparameter set to obtain the probability distribution parameter set.

18. An apparatus comprising one or more processors and at least one memory coupled to said one or more processors, wherein, The one or more processors are configured to perform the method of any one of claims 1-17.

19. A non-transitory computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1-17.

Images