Systems and methods for compressing three-dimensional volume representations

Abstract

Systems and methods relate to encoding and / or decoding of texture / geometry for three-dimensional volumetric representations. An encoding computing system can obtain a voxel block from a three-dimensional volumetric representation of an object. The encoding computing system can encode the voxel block with a machine-learned voxel encoding model to obtain an encoded voxel block. The encoding computing system can decode the encoded voxel block with a machine-learned voxel decoding model to obtain a reconstructed voxel block. The encoding computing system can generate a reconstructed mesh representation of the object based at least in part on the one or more reconstructed voxel blocks. The encoding computing system can encode a texture associated with the voxel block according to an encoding scheme and based at least in part on the reconstructed mesh representation of the object to obtain an encoded texture.

Description

Technical Field

[0001] This disclosure generally relates to compressing three-dimensional volume representations. More specifically, this disclosure relates to compressing the geometry and texture of a three-dimensional volume representation using machine learning models and corresponding texture compression techniques, respectively. Background Technology

[0002] Three-dimensional volumetric representation has been popular in 3D and 4D reconstruction techniques. However, transmitting high-quality 3D and 4D sequences remains challenging due to their large memory coverage. For example, the memory coverage associated with mesh representations of compressed volumes (e.g., objects, people, etc.) is relatively high. If the data to be transmitted is 4D (e.g., a series of 3D frames), then the large memory coverage can hinder the high-quality transmission of the data. Attached Figure Description

[0003] This specification provides a detailed discussion of embodiments that are suitable for those skilled in the art, and the specification refers to the accompanying drawings, in which:

[0004] Figure 1A A block diagram of an example computing system for encoding and decoding three-dimensional volume representation according to an example embodiment of the present disclosure is depicted.

[0005] Figure 1B A block diagram depicts an example computing device operating according to an example embodiment of the present disclosure.

[0006] Figure 1C A block diagram depicts an example computing device operating according to an example embodiment of the present disclosure.

[0007] Figure 2 A block diagram of an example machine learning encoding and decoding pipeline according to an example embodiment of the present disclosure is depicted.

[0008] Figure 3 A flowchart depicting the encoding and decoding of a three-dimensional volume representation according to an example embodiment of the present disclosure is provided.

[0009] Figure 4 A flowchart is depicted illustrating an encoding scheme for encoding a plurality of textures correspondingly associated with a geometry representing a three-dimensional volume, according to an example embodiment of the present disclosure.

[0010] Figure 5 A block diagram depicts an example training architecture for training a machine learning model according to an example embodiment of the present disclosure.

[0011] Figure 6A A flowchart depicts an example method for encoding a three-dimensional volume representation according to an example embodiment of the present disclosure.

[0012] Figure 6B A flow diagram depicting an example method for decoding a three-dimensional volumetric representation in accordance with example embodiments of the present disclosure is depicted.

[0013] Figure 7 A flow diagram depicting an example method for encoding a plurality of textures associated with a geometry of a three-dimensional volumetric representation in accordance with example embodiments of the present disclosure is depicted.

[0014] Reference numerals that repeat across multiple figures are intended to identify corresponding features in various embodiments. DETAILED DESCRIPTION

[0015] OVERVIEW

[0016] Generally, the present disclosure relates to compressing three-dimensional volumetric representations using a compression pipeline. More specifically, machine learning encoding and decoding models can be used to encode and decode the geometry of a volumetric representation (e.g., voxels containing truncated signed distance fields, etc.) in a lossy manner, while texture atlases can be used to encode and store textures associated with the volumetric representation in a lossless manner. Subsets of the volumetric representation geometry (e.g., blocks of voxels of the representation, etc.) can be selectively encoded to reduce the overall amount of data used for encoding (e.g., using machine learning voxel encoding models, etc.). The geometry and associated textures can be encoded at an encoding computing system and transmitted or otherwise provided to a decoding computing system that can be the same or different from the encoding computing system. The decoding computing system can reconstruct the voxels of the geometry with the encoded blocks of voxels (e.g., using machine learning voxel decoding models, etc.), and then extract a mesh representation of the volumetric representation from the reconstructed voxels with an extraction algorithm (e.g., a marching cubes algorithm, etc.). The decoding computing system can decode the encoded texture atlas and apply the decoded textures to the mesh representation to generate a reconstructed three-dimensional volumetric representation. In this way, a three-dimensional volumetric representation can be efficiently compressed at a sending computing system and subsequently reconstructed at a receiving computing system while still preserving a high degree of representative quality.

[0017] More specifically, three-dimensional volumetric representations have been popular in three- and four-dimensional reconstruction techniques. However, transmitting high quality three- and four-dimensional sequences remains challenging due to their large memory footprints. As an example, the memory footprint associated with compressing a mesh representation of a volume (e.g., an object, a person, etc.) is relatively high. If the data to be transmitted is four-dimensional (e.g., a sequence of three-dimensional frames, etc.), the memory footprint can hinder establishing a high quality transmission of the data. Additionally, texture mapping associated with these volumetric representations can require a stream of coordinates that interrelate the texture mapping with spatial locations on the mesh representation of the volume.

[0018] In response to these issues, the present disclosure proposes a block-based end-to-end trainable geometry compression model based on signed distance fields (e.g., truncated signed distance fields (TSDFs), etc.) stored in voxels that make up a volumetric representation. The model can include lossy TSDF compression, lossy parameter-free compression of textures, and lossless compression of the topology of a surface using conditional distributions of TSDF signs. More specifically, the present disclosure proposes obtaining a block of voxels (e.g., a subset of voxels) from a plurality of voxels that make up a three-dimensional representation of an object. The voxels can include a magnitude value and a sign value, and the block of voxels can include a subset of the plurality of voxels. A machine-learned voxel encoding model can be used to encode the block of voxels. A machine-learned decoding model can be used to decode the encoded block of voxels to obtain a reconstructed block of voxels. A reconstructed mesh representation can be generated from the reconstructed block of voxels, and a texture can be encoded based on the reconstructed mesh representation. The encoded texture (e.g., a texture atlas) can be decoded and applied to the reconstructed mesh representation to generate a reconstructed three-dimensional volumetric representation.

[0019] More specifically, an encoding computing system can obtain one or more blocks of voxels from a three-dimensional volumetric representation of an object (e.g., a person, a bowling ball, etc.). The three-dimensional volumetric representation can include a plurality of voxels and a corresponding plurality of textures associated therewith. Each of the plurality of voxels can include a magnitude value and a sign value. Each of the one or more blocks of voxels can include a subset of the plurality of voxels.

[0020] In some implementations, the blocks of voxels can include and / or can be an implicit representation (e.g., a signed distance field, a truncated signed distance field, etc.) of a surface of an object represented by the volumetric representation. As an example, each voxel can include a signed distance field that implicitly represents a distance (e.g., a magnitude) to the surface and a sign value associated with a spatial location of the voxel with respect to the surface. For example, a voxel located “inside” the surface of the volume can have a negative sign value, while a voxel located “outside” the surface of the volume can have a positive sign value. As another example, a voxel that implicitly contains the surface can have a sign and magnitude value of zero (e.g., a “zero-crossing”). In some implementations, the voxels can contain a truncated signed distance field. More specifically, the magnitude values of the voxels can be truncated at a certain threshold. For example, a voxel located relatively far from the surface of the volume can have a magnitude value of 15. If the magnitude truncation threshold is set to 1, then any magnitude value greater than 1 can be truncated to 1. In this way, less “relevant” data (e.g., voxels far from the surface of the volume) can be truncated, thus allowing for a more efficient representation of the surface of the volume.

[0021] In some implementations, one or more blocks of voxels can be obtained based at least in part on the sign values of the voxels. More specifically, a block of voxels can have a size of k x k x k voxels (e.g., a three-dimensional block of voxels). In some implementations, each of the one or more blocks of voxels can be a non-overlapping block of voxels containing a zero-crossing (e.g., an occupied block). As an example, the plurality of voxels can be iterated in a block-wise manner (e.g., k x k x k voxels at a time, etc.) to determine whether a current block contains a voxel with a zero-crossing. If the block does contain a voxel (e.g., a zero-crossing voxel, etc.), the block can be extracted and encoded. If the block does not contain a voxel (e.g., a zero-crossing voxel, etc.), the block can be skipped from encoding. In this way, the encoding computing system can extract blocks related to the surface of the object and encode them, while ignoring blocks that are less relevant, thus greatly reducing the number of blocks of voxels that are encoded and transmitted. It should be noted that in some implementations, the size of the object and / or the volumetric size of the blocks can require that only one block be selected from the plurality of voxels (e.g., all voxels with zero-crossings are contained in one block, etc.).

[0022] In some implementations, the blocks of voxels can be indexed with a block of voxels index. The block of voxels index can store the spatial location of the block of voxels after the block of voxels is extracted. It should be noted that in some implementations, extracting blocks of voxels containing zero-crossings can increase the importance of maintaining spatial coherence between the extracted blocks to properly reconstruct the surface of the volume. As such, the block of voxels index can be used by the decoding computing system to reconstruct the block of voxels at the original spatial location of the block of voxels, as will be discussed in greater detail with respect to the various figures.

[0023] The encoding computing system can encode the one or more blocks of voxels with a first instance of a machine-learned voxel encoding model (e.g., a trained convolutional neural network, recurrent neural network, etc.) to obtain one or more encoded blocks of voxels. More specifically, given a block x to be transmitted, the encoding computing system (e.g., a sending / transmitting computing system, etc.) can compute a lossy quantized latent representation e using the machine-learned encoding model E with parameters The encoding computing system can use to compute the conditional probability distribution over the sign values of the voxels as where s is the true value sign configuration of the block, and s are the learnable parameters of the distribution. In some implementations, the encoding computing system can use an entropy encoder to encode the latent code and ​and the symbol s to compute a bitstream and s bits As an example, can be parameterized by the φ parameter learning prior stereo pixel distribution over s. It should be noted that in some implementations, the prior distribution can be trained prior to encoding in a training phase, as will be discussed in more detail with respect to various figures.

[0024] The encoding computing system can decode the one or more encoded stereo pixel blocks with a first instance of a machine-learned stereo pixel decoding model (e.g., a trained convolutional neural network, recurrent neural network, etc.) to obtain a first instance of one or more reconstructed stereo pixel blocks. The reconstructed stereo pixel blocks can be a lossy reconstruction of the one or more stereo pixel blocks. In some implementations, the reconstructed stereo pixel blocks can be reconstructed based at least in part on stereo pixel indices that describe spatial locations of the one or more stereo pixel blocks.

[0025] The encoding computing system can generate a first instance of a reconstructed mesh representation of the object based at least in part on the first instance of the one or more reconstructed stereo pixel blocks. The reconstructed mesh (e.g., a triangular mesh, etc.) can be a polygonal mesh representation of a surface of the object in the volume. In some implementations, the reconstructed mesh representation can include an associated plurality of textures of the three-dimensional volume representation. As an example, the reconstructed mesh representation can be a textured mesh representation. Alternatively, in some implementations, the reconstructed mesh representation can be reconstructed from the plurality of textures individually.

[0026] The encoding computing system can encode the plurality of textures according to an encoding scheme to obtain a plurality of encoded textures. In some implementations, the encoding can be based at least in part on the first instance of the reconstructed mesh representation of the object. More specifically, the encoding scheme can be a traceless UV parameterization method that can be combined with the block-level geometry encoding of the one or more stereo pixel blocks. In some implementations, the polygons of each stereo pixel block can be extracted and grouped based at least in part on one or more characteristics of the polygons (e.g., a normal of the polygon, a texture characteristic associated with a texture of the polygon, etc.). As an example, the polygons of a stereo pixel block can be grouped into one group. As another example, the polygons of a stereo pixel block can be mapped to a plurality of groups.

[0027] In some implementations, the encoding computation system can generate a polygon graph comprising one or more groups of polygons. The polygon graph can be configured to maintain spatial coherence between each group within the polygon group. More specifically, the average value of the polygon features of the group (e.g., the average value of the normals) can be used to determine the tangent space on which the vertices of the polygons in the group are projected. The projections can be rotated until they fit into an axis-aligned rectangle with the smallest area (e.g., using a rotation caliper). In some implementations, this can result in deterministic UV coordinates of each vertex in the group relative to the bounding box of the vertex projection. As an example, the bounding boxes of the groups can have a size and can be ordered by size. In some implementations, based on the order, the encoding computation system can pack the groups into a square polygon graph using a packing algorithm (e.g., a quadtree algorithm). Additionally, the UV coordinates of the polygons of a block can be offset relative to the graph. In this way, a two-dimensional graph can be generated for each three-dimensional pixel block.

[0028] In some implementations, the encoding computation system can map (e.g., pack) polygon graphs into a texture atlas. The spatial location of the polygon graph in the texture atlas can correspond to the spatial location of the reconstructed stereo pixel block (e.g., specified by the stereo pixel index in the original 3D volume representation). The polygon graph can be mapped to the texture atlas in a manner that maximizes spatiotemporal coherence. More specifically, as previously described, each of the stereo pixel blocks can be indexed by integer triples because they originally exist in the 3D volume representation (e.g., x, y, z coordinates, etc.). The binary representation of these integers can be converted into representative Morton code (e.g., 2D Morton code, 3D Morton code, etc.). As an example, the integer triples describing the 3D spatial location of the stereo pixel block can be converted into 2D Morton code. For example, each texture graph can be represented by a pair of integers. Indexing, the 2D Morton code for this pair of integers is represented in binary as v B-1 v B-1 ...integers of u0v0 This type of transformation provides compatibility with the two-dimensional regular grid structure of polygon graphs (e.g., converting x, y, z variables into u, v variables, etc.). As another example, it is possible to generate three-dimensional Morton codes for texture maps, which include and rank the two-dimensional Morton codes of the map. For instance, it is possible to map a polygon graph to a texture map location for a 3D pixel block at a volume position (x, y, z). where rank is the rank of the 3D Morton code in a list of 3D Morton codes. In this way, the 3D Morton codes of the atlas are able to preserve the three-dimensionality of the packed texture using a ranking scheme, and thus are able to be easily extracted back into three-dimensional space by the decoder (e.g., by de-multiplexing the bits of the Morton codes, etc.).

[0029] In some implementations, the encoded and / or entropy encoded stereoscopic pixel blocks can be transmitted to a decoding computing system (e.g., a remote or local decoding computing system, etc.). The stereoscopic pixel blocks can be transmitted with the stereoscopic pixel block indices and the plurality of encoded textures. The transmission can be facilitated via a network (e.g., a local area network, a wireless network, etc.) or a storage medium (e.g., a flash drive, a hard drive, etc.).

[0030] In some implementations, the decoding computing system can receive the encoded stereoscopic pixel blocks and decode the encoded stereoscopic pixel blocks with a machine-learned stereoscopic pixel decoding model to obtain a second instance of the one or more reconstructed stereoscopic pixel blocks. It should be noted that the machine-learned stereoscopic pixel decoding model can be a pre-trained deterministic model. As such, the outputs of the first instance of the machine-learned stereoscopic pixel decoding model (e.g., on the encoding computing system) and the second instance of the model (e.g., on the decoding computing system) can produce identical deterministic outputs given the same input. As such, the first instance of the reconstructed blocks on the encoding computing system will be the same as or substantially similar to the second instance of the reconstructed blocks on the decoding computing system (e.g., due to transmission loss, etc.). In this way, the textures encoded based on the first instance of the reconstructed blocks will be more compatible with the second instance of the reconstructed blocks (e.g., for application of the blocks and / or multiplication of the blocks, etc.).

[0031] More specifically, in some implementations, the machine-learned decoding model of the decoding computing system can receive the latent representation and the associated symbols. Alternatively, in some implementations, the decoding computing system can receive the bitstream and s bits from the encoding computing system and recover using an entropy decoder with the learned stereoscopic pixel distribution (e.g., of the encoding computing system) to recover to recompute to recover the lossless coded real-valued symbols s. The decoding computing system can recover the lossy TSDF values (e.g., amplitudes, etc.) by using the machine-learned stereoscopic pixel decoding model D along with the real-valued symbols s as where is the element-wise multiplication operator, | · | is the element-wise absolute value operator, and dTo parameterize the machine-learned decoding model. In this way, one or more encoded stereoscopic pixel blocks can be decoded by the decoding computing system using the machine-learned decoding model to obtain one or more reconstructed stereoscopic pixel blocks.

[0032] In some embodiments, the decoding computing system can decode the received encoded textures to obtain a plurality of decoded textures. The textures can be decoded according to the encoding scheme used by the encoding computing system. More specifically, the bits (e.g., Morton codes) of the texture atlas containing the textures can be de-muxed and used to decode the textures in a manner that preserves the spatial coherence of the textures.

[0033] In some embodiments, the plurality of decoded textures can be applied to the second instance of the reconstructed stereoscopic pixel blocks to obtain a reconstructed three-dimensional volumetric representation of the object. More specifically, the decoding computing system can first generate the second instance of the reconstructed mesh representation in the same manner as the encoding computing system (e.g., using a marching cubes algorithm with the reconstructed stereoscopic pixel blocks, etc.). Note that the second instance of the reconstructed mesh representation can be identical or substantially similar to the first instance on the encoding computing system. The plurality of decoded textures can be applied to the reconstructed mesh representation (e.g., polygon mesh, etc.) in the manner specified by the de-muxed Morton codes of the texture atlas. Since the plurality of decoded textures are packed into the texture atlas in a manner that preserves spatial and positional coherence, the textures can be easily unpacked and iteratively applied to the mesh in the same order in which the textures were extracted.

[0034] In some embodiments, the computing system can train the models and distributions of the present embodiments in an end-to-end manner. More specifically, a loss function can be evaluated, and based on the loss function, one or more parameters can be adjusted for at least one of the machine-learned stereoscopic pixel encoding model, the machine-learned stereoscopic pixel decoding model, the learned stereoscopic pixel distribution, or the learned symbol distribution. Note that the computing system can include and use all of the above models. Thus, it is unnecessary for the transmission to include an entropy encoder and an entropy decoder. Instead, in some embodiments, uniform noise can be added during the training step to simulate the entropy encoding and decoding (e.g., using the encoding and / or decoding computing system, etc.) that occurs during the inference step. Additionally or alternatively, in some embodiments, uniform noise can be added during the training step to simulate the quantization noise that results from quantization prior to the encoding step.

[0035] In some embodiments, the loss function can evaluate the difference between the one or more stereoscopic pixel blocks and the one or more reconstructed stereoscopic pixel blocks. The loss function can further evaluate one or more bitrate terms that each evaluate a measure of the number of bits used to encode a portion of the encoded signal. More specifically, the loss function can evaluate distortion, potential bitrate, and symbol bitrate, as described as

[0036]

[0037] In some implementations, the distortion parameter Reconstruction error between the ground truth and predicted stereo pixel values (e.g., TSDF values, magnitudes, etc.) can be evaluated. Masks can be used to focus the training of the model on more “relevant” stereo pixels of the stereo pixel blocks (e.g., stereo pixels with neighboring stereo pixels of opposite signed values). As an example, for each dimension, masks can be created for important stereo pixels (i.e., m x , m y , and m z ). Stereo pixels with more than one neighbor of opposite sign can appear in multiple masks, further increasing the weight of these stereo pixels. The masks can be used to compute the squared difference for important stereo pixels, as specified by the following equation for B blocks:

[0038]

[0039] In some implementations, the potential bit rate parameter can evaluate an estimate of the differential entropy of the noisy code . As an example, the potential bit rate measure can provide an estimate of the entropy caused by noise in the code as coded by the entropy coder. In this way, the potential bit rate parameter can reduce the bit rate of the compressed code, enabling more efficient compression.

[0040] In some implementations, the sign bit rate parameter R s (s; 0 s ) can evaluate the lossless compression rate of the sign value of each of the plurality of stereo pixels of the three-dimensional volumetric representation. Since S only contains the discrete values {-1, +1}, it can be losslessly compressed using entropy coding. As previously mentioned, a conditional probability distribution rather than a prior distribution p s (s) can be used. Note that in some implementations, the conditional distribution can have much lower entropy than the prior, since s is designed to depend on This allows for more efficient compression of the sign value.

[0041] To make this dependency explicit, in some implementations, an additional head can be added to the architecture of the machine learning decoding model. More specifically, in some implementations, the machine learning decoding model can be or otherwise include a convolutional neural network with a final convolutional layer that includes two convolutional heads. The convolutional heads can respectively generate reconstructions and s (e.g., the sign value), such that and Symbol rate loss R s The cross-entropy between the real-valued symbols s (where -1 is remapped to 0) and their conditional predictions In this way, minimizing the symbol bit rate parameter R s (s; 0 s ) can train the network to make more accurate symbol predictions while also minimizing the bit rate of the compressed symbol values.

[0042] The present disclosure provides several technical effects and benefits. As one example technical effect and benefit, the systems and methods of the present disclosure enable significant progress in three-dimensional volume compression as compared to other approaches. While other approaches generally involve compression of mesh representations of three-dimensional objects, the present embodiments provide for three-dimensional representations using implicit representations (e.g., signed distance fields, etc.) and novel texture compression algorithms corresponding to implicit volume representations. This functionality provides significant improvements in compressed size of the volume representation, while still maintaining a very high degree of reconstruction accuracy. By reducing the size of the compressed representation, the present embodiments substantially reduce network resources for transmitting and receiving volume representations. In turn, this reduces the processing resources, memory resources, and power resources required to send and transmit three-dimensional representations.

[0043] Reference will now be made to the drawings wherein illustrated, example embodiments of the present disclosure will be discussed in further detail.

[0044] Example devices and systems

[0045] Figure 1A A block diagram of an example computing system 100 to perform compression of a three-dimensional volume representation in accordance with example embodiments of the present disclosure is depicted. The system 100 includes an encoding computing system 102, a decoding computing system 130, and a training computing system 150 communicatively coupled over a network 180.

[0046] The encoding computing system 102 can be any type of computing system or device such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, a computing server (e.g., a cloud-based server platform, etc.), a virtualized computing server, or any other type of computing device or system.

[0047] The encoding computing system 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc. and combinations thereof. The memory 114 can store data 116 and instructions 118 that are executed by the processor 112. The instructions 118 can include instructions for performing the operations described herein.

[0048] In some implementations, the operations performed by the processor 112 can include texture encoding operations. More specifically, the operations can include generating a texture atlas to encode and store textures associated with a volumetric representation in a lossless manner. The texture atlas can be packed with corresponding textures using a particular algorithmic operation (e.g., a marching cubes algorithm, etc.). In some implementations, the texture encoding operations can correspond to operations and / or outputs of instances of machine learning encoding and decoding models 120 (e.g., encoding textures based on deterministic lossy outputs of machine learning encoding and decoding models, etc.). See Figure 3 Example implementations of texture encoding operations and / or their relationship to model instances 120 and 140 are discussed.

[0049] In some implementations, the encoding computing system 102 can store or include instances 120 of machine learning encoding models and machine learning decoding models. For example, the model instances 120 can be or can otherwise include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including non-linear models and / or linear models. The neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. See Figures 2-3 Example model instances 120 are discussed.

[0050] In some implementations, one or more model instances 120 can be received from the decoding computing system 130 over the network 180, stored in the encoding computing system memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the model instances 120 and 140 can be instances of the same machine learning encoding model and machine learning decoding model. More specifically, the instances can be parallel instances of the same machine learning encoding model and machine learning decoding model trained in the same way (e.g., at the training computing system 150, etc.) and distributed to both the decoding computing system 130 and the encoding computing system 102 (e.g., over the network 180, etc.). In this way, in some implementations, the parallel instances (e.g., instance 120 and instance 140) can perform the same operations (e.g., encoding and decoding operations) in a deterministic manner. For example, the instances 120 and 140 can receive the same input and generate the same output, respectively.

[0051] More specifically, the instance 120 of the machine learning encoding and decoding model can be used to encode and decode the geometry of a three-dimensional volumetric representation (e.g., a voxel representing a truncated signed distance field, etc.) in a lossy manner. In some implementations, the three-dimensional volumetric representation can be generated at the encoding computing system 102. For example, volumetric data can be captured using an associated and / or connected imaging device (e.g., a plurality of cameras configured to capture the three-dimensional volumetric representation, etc.). In some implementations, the encoding computing system 102 can receive data describing the three-dimensional volumetric representation (e.g., via the network 180, a flash memory, a physical storage medium, etc.).

[0052] The instance 120 of the machine learning encoding model can be used to generate one or more encoded voxel blocks. The one or more encoded voxel blocks can be decoded by the instance 120 of the machine learning decoding model to generate one or more reconstructed voxel blocks. In some implementations, the reconstructed voxel blocks can be used, in part, by the encoding computing system 102 to encode a texture associated with the three-dimensional volumetric representation. The encoding computing system 102 can transmit the encoded texture to the decoding computing system 130 (e.g., via the network 180). Alternatively or additionally, in some implementations, the encoded voxel blocks can be transmitted to the decoding computing system 130 (e.g., via the network 180).

[0053] Additionally or alternatively, the instance 140 of the machine-learned encoding and decoding model can be included in or otherwise stored and implemented by a decoding computing system 130 that is in communication with the encoding computing system 102 according to a sender-receiver relationship, or vice versa. For example, the instance 140 of the machine-learned encoding and decoding model can be implemented by the decoding computing system 130 to decode the encoded texture and / or geometry of the three-dimensional representation. As an example, the decoding computing system 130 can utilize the instance 140 of the machine-learned decoding model to decode an encoded stereopixel block transmitted from the encoding computing system 102 to the decoding computing system 130 (e.g., via the network 180). By decoding the encoded stereopixel block, the decoding computing system 130 can generate a reconstructed stereopixel block. The decoding computing system 130 can generate a reconstructed mesh representation from the reconstructed stereopixel block. Additionally, the decoding computing system 130 can decode the encoded texture to generate a decoded texture (e.g., based on the reconstructed stereopixel block and / or the reconstructed mesh representation, etc.). The decoding computing system 130 can apply the decoded texture to the reconstructed mesh representation to generate a reconstructed three-dimensional volumetric representation. Thus, in this manner, the instance 120 of the machine-learned encoding and decoding model can be stored and implemented at the encoding computing system 102, and the same instance 140 of the machine-learned encoding and decoding model can be stored and implemented at the decoding computing system 130.

[0054] The decoding computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 134 can include one or more non-transitory computer-readable storage media, such as

[0055] In some implementations, the encoding computing system 102 includes or is otherwise implemented by one or more encoding computing devices. In the case that the encoding computing system 102 includes multiple encoding computing devices, such encoding computing devices can operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0056] In some implementations, the decoding computing system 130 includes or is otherwise implemented by one or more decoding computing devices. In cases where the decoding computing system 130 includes multiple decoding computing devices, such decoding computing devices can operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0057] It should be noted that in some implementations, the terms "encoding computing system" and "decoding computing system" are used merely to more easily describe the functionality of the present embodiments. More specifically, in some implementations, the decoding computing system 130 can be used as an encoding computing system, and the encoding computing system 102 can be used as a decoding computing system. As an example, operations as previously described with respect to the encoding computing system 102 can be performed at the decoding computing system 130, and vice versa. In this manner, both computing systems (e.g., systems 102 and 130) can be utilized to encode and / or decode three-dimensional volumetric representations or any components thereof (e.g., voxels, blocks of encoded voxels, textures, encoded textures, etc.).

[0058] The encoding computing system 102 and / or the decoding computing system 130 can train the base model of the model instances 120 and / or 140 via interaction with the training computing system 150 communicatively coupled over the network 180. The training computing system 150 can be separate from the encoding computing system 102 and the decoding computing system 130. Alternatively or additionally, the training computing system 150 can be part of the decoding computing system 130 and / or the encoding computing system 102 (e.g., as multiple instances of a training computing system in one or more of the other systems and / or devices, etc.).

[0059] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as

[0060] The training computing system 150 can include a model trainer 160 that trains the machine learning models of the model instances 120 and 140 stored at the encoding computing system 102 and the decoding computing system 130 using various training or learning techniques such as, for example, backpropagation of errors. For example, a loss function can be backpropagated through a model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over multiple training iterations.

[0061] In some implementations, performing backpropagation of errors can include performing truncated backpropagation through time. The model trainer 160 can perform several regularization techniques (e.g., weight decay, dropout, etc.) to improve the regularization of the trained models.

[0062] In particular, the model trainer 160 can train the OVERALL model 120 and / or 140 based on a set of training data 162. The training data 162 can include, for example, a three-dimensional volumetric representation including voxels storing a truncated signed distance field (TSDF). The voxels can include a known sign value and a known magnitude value (e.g., ground truth value). Implementations of the training (e.g., loss function, evaluation of loss function parameters, etc.) performed by the model trainer 160 will be discussed in greater detail with respect to Figure 5 In particular, the model trainer 160 can train the OVERALL model 120 and / or 140 based on a set of training data 162. The training data 162 can include, for example, a three-dimensional volumetric representation including voxels storing a truncated signed distance field (TSDF). The voxels can include a known sign value and a known magnitude value (e.g., ground truth value). Implementations of the training (e.g., loss function, evaluation of loss function parameters, etc.) performed by the model trainer 160 will be discussed in greater detail with respect to

[0063] In some implementations, the training examples can be provided by the encoding computing system 102 and / or the decoding computing system 130 if the user has consented. Thus, in such implementations, the instances 120 and 140 of the models provided to the encoding computing system 102 and the decoding computing system 130 can be trained by the training computing system 150 based on data obtained by one of the computing systems (e.g., systems 102 and / or 130). In some instances, this process can be referred to as personalizing the models.

[0064] The model trainer 160 includes computer logic in the form of software or firmware instructions. The software or firmware instructions configure the central processing unit 202 to operate as the model trainer 160. The software or firmware instructions can be stored in the memory 204 and executed by the central processing unit 202. The software or firmware instructions can be written in any suitable programming language such as, for example, C, C++, Java, Visual Basic, or any other suitable programming language. The software or firmware instructions can be compiled or interpreted from source code to generate modular or non-modular executable instructions.

[0065] The network 180 can be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communications over the network 180 can be carried using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL), via any type of wired and / or wireless connection.

[0066] Figure 1A An example computing system that can be used to implement the present disclosure is illustrated. Other computing systems can also be used. For example, in some implementations, the encoding computing system 102 can include the model trainer 160 and the training data set 162. In such implementations, the models of the instance 120 and 140 can be trained and used locally at the encoding computing system 102 and / or the decoding computing system 130. In some of such implementations, the encoding computing system 102 and / or the decoding computing system 130 can implement the model trainer 160 to personalize the models of the instance 120 and 140 based on data obtained by one or both of the computing systems (e.g., systems 102 and 130).

[0067] Figure 1B A block diagram of an example computing system 10 is depicted in accordance with example embodiments of the present disclosure. The computing system 10 can be an encoding computing system and / or a decoding computing system.

[0068] The computing system 10 includes several applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine learning models. For example, each application can include a machine learning model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0069] As Figure 1B illustrated in FIG. 1, each application can communicate with several other components of the computing system, such as (for example) one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0070] Figure 1C A block diagram of an example computing system 50 is depicted in accordance with example embodiments of the present disclosure. The computing system 50 can be an encoding computing system and / or a decoding computing system.

[0071] The computing system 50 includes several applications (e.g., applications 1 through N). Each application communicates with the central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and the like. In some implementations, each application is able to communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0072] The central intelligence layer includes several machine learning models. For example, as illustrated in Figure 1C each application. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included within or otherwise implemented by the operating system of the computing system 50. In some implementations, a model can be an instance of one or more models. As an example, model 1 and model 2 can be two parallel instances of a single machine learning model.

[0073] The central intelligence layer can communicate with a central system data layer. The central system data layer can be a centralized repository for data for the computing system 50. As illustrated in Figure 1C the central system data layer can communicate with several other components of the computing system, such as (for example) one or more sensors, a context manager, a system state component, and / or additional components. In some implementations, the central system data layer can communicate with each device component using an API (e.g., a private API).

[0074] Example model arrangement

[0075] Figure 2 A block diagram of an example compression pipeline 200 according to example embodiments of the present disclosure is depicted. In some implementations, the pipeline 200 includes two or more machine learning models and / or more than two instances of a machine learning model (e.g., a machine learning encoding model 202 and a machine learning decoding model 203). The models of the pipeline 200 are trained to receive a set of input data 204 describing a three-dimensional volumetric representation and, as a result of receiving the input data 204, provide output data 206 describing a lossy reconstruction of the three-dimensional volumetric representation. Thus, in some implementations, the pipeline 200 can include an instance of a machine learning encoding model 202 operable to lossy encode input data and an instance of a machine learning decoding model 203 operable to decode the input data and generate output data (e.g., a lossy reconstruction of a three-dimensional volumetric representation).

[0076] Figure 3A flow diagram depicting encoding and decoding of a three-dimensional volumetric representation in accordance with example embodiments of the present disclosure is depicted. Compression pipeline 300 is similar to compression pipeline 200 in Figure 2 , except that pipeline 300 further includes a second instance of a machine learning decoding model. More specifically, compression pipeline 300 is depicted as being distributed across multiple computing systems (e.g., an encoding computing system and a decoding computing system). It should be noted that this distribution does not necessarily change the core functionality (e.g., inputs, outputs, operations, etc.) of pipeline 300 as compared to pipeline 200 of Figure 2 .

[0077] More specifically, the encoding computing system is able to obtain a three-dimensional volumetric representation 302. The three-dimensional volumetric representation 302 can include one or more voxel blocks 302A (e.g., a geometry of the three-dimensional volumetric representation 302, etc.) and an associated plurality of textures 302B. The voxel block 302A can include and / or otherwise be a voxel block (e.g., a subset of voxels) obtained by the encoding computing system from a plurality of voxels that make up the three-dimensional volumetric representation 302. The voxels of the voxel block 302A can include a magnitude value and a sign value, and the voxel block 302A can include a subset of the plurality of voxels.

[0078] In some implementations, the voxel block 302A can include and / or can be an implicit representation (e.g., a signed distance field, a truncated signed distance field, etc.) of a surface of an object represented by the three-dimensional volumetric representation 302. As an example, each voxel can include a signed distance field that implicitly represents a distance (e.g., a magnitude) to the surface and a sign value associated with a voxel spatial location with respect to the surface. For example, voxels located “inside” the surface of the volume can have a negative sign value, while voxels located “outside” the surface of the volume can have a positive sign value. As another example, voxels that implicitly contain the surface can have a sign value and a magnitude value of zero (e.g., a “zero-crossing”). In some implementations, the voxels can contain a truncated signed distance field. More specifically, the magnitude values of the voxels can be truncated at some threshold value. For example, voxels located relatively far away from the surface of the volume can have a magnitude value of 15. If the magnitude truncation threshold is set to 1, then any magnitude value greater than 1 can be truncated to 1. In this way, less “relevant” data (e.g., voxels far away from the surface of the volume) can be truncated, thus allowing for a more efficient representation of the surface of the volume.

[0079] In some implementations, the stereo pixel blocks 302A can be obtained based at least in part on the sign values of the stereo pixels. More specifically, the stereo pixel blocks 302A can have a size of k x k x k stereo pixels (e.g., a three-dimensional stereo pixel block). In some implementations, each of the stereo pixel blocks 302A can be a non-overlapping stereo pixel block containing a zero-crossing (e.g., an occupied block). As an example, the plurality of stereo pixels can be iterated in a block-wise manner (e.g., k x k x k stereo pixels at a time, etc.) to determine whether a current block contains a stereo pixel with a zero-crossing. If the block does contain a stereo pixel, the block can be extracted and encoded. If the block does not contain a stereo pixel, the block can be skipped for encoding. In this way, the encoding computing system can extract blocks related to the surface of the object and encode them, while ignoring blocks that are less relevant, thus greatly reducing the number of stereo pixel blocks to be encoded and transmitted. The stereo pixels of the stereo pixel blocks 32A can include a magnitude value x 306 and a sign value s 304. It should be noted that in some implementations, the size of the object and / or the volumetric size of the blocks can require that only one block be selected from the plurality of stereo pixels that make up the three-dimensional volumetric representation 302 (e.g., all stereo pixels with zero-crossings are contained in one block, etc.).

[0080] In some implementations, the stereo pixel blocks 302A can be indexed with a stereo pixel block index. The stereo pixel block index can store the spatial location of the stereo pixel blocks 302A after the stereo pixel blocks are extracted. It should be noted that in some implementations, extracting the stereo pixel blocks 302A containing zero-crossings can increase the importance of maintaining spatial coherence between the extracted blocks to properly reconstruct the surface of the volume. As such, the stereo pixel block index can be used by the decoding computing system to reconstruct the stereo pixel blocks at their original spatial locations.

[0081] The encoding computing system can encode the stereo pixel blocks 302A with a first instance of a machine-learned stereo pixel encoding model 308 (e.g., a trained convolutional neural network, recurrent neural network, etc.) to obtain one or more encoded stereo pixel blocks 310. More specifically, given the blocks x 306 to be transmitted, the encoding computing system (e.g., a sending / transmitting computing system, etc.) can compute the lossy quantized latent representation e using the machine-learned encoding model E with parameters The encoding computing system can use the encoded stereo pixel blocks 310 to compute the learned sign distribution over the sign values 304 of the stereo pixels 302A as where the signs 304 are the true value sign configuration of the blocks, and s are the learnable parameters of the learned sign distribution.

[0082] The encoding computing system can decode the encoded stereo-pixel blocks 310 with a first instance of a machine-learned stereo-pixel decoding model 312 (e.g., a trained convolutional neural network, recurrent neural network, etc.) to obtain one or more first instances of reconstructed stereo-pixel blocks 313. The reconstructed stereo-pixel blocks 313 can be lossy reconstructions of the stereo-pixel blocks 302A. In some implementations, the reconstructed stereo-pixel blocks 313 can be reconstructed based at least in part on stereo-pixel indices that describe spatial locations of the one or more stereo-pixel blocks 302A.

[0083] The encoding computing system can generate a first instance of a reconstructed mesh representation of the object based at least in part on the one or more first instances of reconstructed stereo-pixel blocks 313. The reconstructed mesh (e.g., a triangular mesh, etc.) can be a polygonal mesh representation of a surface of the object in the three-dimensional volumetric representation 302. In some implementations, the reconstructed mesh representation can include the associated plurality of textures 302B of the three-dimensional volumetric representation 302. As an example, the reconstructed mesh representation can be a textured mesh representation (e.g., with textures 302B). Alternatively, in some implementations, the reconstructed mesh representation can be reconstructed separately from the plurality of textures 302B and can be used to extract the plurality of textures 302B from the three-dimensional volumetric representation 302.

[0084] The encoding computing system can encode the plurality of textures 302B according to an encoding scheme to obtain a plurality of encoded textures 314. In some implementations, the encoding can be based at least in part on the first instance of the reconstructed mesh representation of the object. More specifically, the encoding scheme can be a traceless UV parameterization method that can be combined with the block-level geometry encoding of the one or more stereo-pixel blocks 302A. In some implementations, the polygons of each stereo-pixel block can be extracted and grouped based at least in part on one or more characteristics of the polygons (e.g., a normal of the polygon, texture properties associated with the texture of the polygon, etc.). As an example, the polygons of one stereo-pixel block can be grouped into one group. As another example, the polygons of one stereo-pixel block can be mapped to multiple groups.

[0085] In some implementations, the encoding computing system can use an entropy encoder / decoder 316 to calculate a bitstream and from the latent code (e.g., the encoded stereo-pixel blocks 310) and the symbol s by respectively using the distributions and s bits . As an example, can be the latent code learning a prior stereo pixel distribution on the (e.g., encoded stereo pixel blocks 310). It should be noted that in some embodiments, learning the stereo pixel distribution and learning the symbol distribution can be trained prior to a training phase, as will be described with respect to Figure 5 in more detail.

[0086] In some embodiments, the encoded and / or entropy encoded stereo pixel blocks 310 can be transmitted to a decoding computing system (e.g., a remote or local decoding computing system, etc.). The encoded stereo pixel blocks 310 (e.g., and symbols 304) can be transmitted with the stereo pixel block indices and the plurality of encoded textures 314. The transmission can be facilitated via a network (e.g., a local area network, a wireless network, etc.) or a storage medium (e.g., a flash drive, a hard drive, etc.) as depicted in FIG. 3B. Figure 1A

[0087] The decoding computing system can receive the encoded stereo pixel blocks 310 and decode the encoded stereo pixel blocks 310 with a second instance of the machine-learned stereo pixel decoding model 318 to obtain a second instance of one or more reconstructed stereo pixel blocks 319. It should be noted that the second instance of the machine-learned stereo pixel decoding model 318 can be a pre-trained deterministic model. As such, the outputs of the first instance of the machine-learned stereo pixel decoding model 312 and the second instance of the model 318 can produce identical deterministic outputs 313 and 319. Thus, the first instance of the reconstructed blocks 313 on the encoding computing system can be identical or substantially similar to the second instance of the reconstructed blocks 319 on the decoding computing system. In this way, textures encoded based on the first instance of the reconstructed blocks 313 will be more compatible (e.g., for application of blocks and / or products of blocks, etc.) with the second instance of the reconstructed blocks 319.

[0088] More specifically, in some embodiments, the machine-learned decoding model 319 of the decoding computing system can receive the latent representation (e.g., the encoded stereo pixel blocks 310) and the associated symbols 304. Alternatively, in some embodiments, the decoding computing system can receive a bitstream and s bits from the encoding computing system and use the entropy encoder / decoder 316 to recover 310 using the learned stereo pixel distribution (e.g., the learned stereo pixel distribution of the encoding computing system). The decoding computing system can use 310 to recompute (e.g., the learned symbol distribution) to recover the lossless coded real-valued symbols s 304. The decoding computing system can recover the real-valued symbols 304 by using the second instance of the machine-learned stereo pixel decoding model 318 along with the real-valued symbols 304 as ​to recover the lossy TSDF values (e.g., magnitude 306, etc.), where is an element-wise multiplication operator, | · | is an element-wise absolute value operator, and d parameters of the machine learning decoding model. In this way, one or more of the encoded voxel blocks 310 can be decoded by the decoding computing system using a second instance of the machine learning decoding model 318 to obtain one or more reconstructed voxel blocks 319.

[0089] In some implementations, the decoding computing system can decode the received encoded textures 314 to obtain a plurality of decoded textures 315. The textures can be decoded according to the encoding scheme used by the encoding computing system. More specifically, the bits (e.g., Morton codes) of the texture atlas containing the encoded textures 314 of the textures can be de-muxed and used to decode the textures in a manner that preserves the spatial coherence of the textures.

[0090] In some implementations, the plurality of decoded textures 315 can be applied to a second instance of the reconstructed voxel blocks 319 to obtain a reconstructed three-dimensional volumetric representation of the object 322. More specifically, the decoding computing system can first generate a second instance of the reconstructed mesh representation 320 in the same manner as the encoding computing system (e.g., using a marching cubes algorithm with the reconstructed voxel blocks, etc.). Note that the second instance of the reconstructed mesh representation 320 can be identical or substantially similar to the first instance on the encoding computing system. The plurality of decoded textures 315 can be applied to the reconstructed mesh representation 320 (e.g., a polygon mesh, etc.) in the manner specified by the de-muxed Morton codes of the texture atlas. Since the plurality of decoded textures 315 were packed into the texture atlas in a manner that preserves spatial and positional coherence, the textures can be easily unpacked and iteratively applied to the reconstructed mesh 320 in the same order in which they were extracted.

[0091] Figure 4 A flowchart depicting an encoding scheme for encoding a plurality of textures that are respectively associated with a geometry of a three-dimensional volumetric representation in accordance with example embodiments of the present disclosure is depicted. More specifically, an encoding computing system can obtain a plurality of voxel blocks 402 from a three-dimensional volumetric representation. The three-dimensional volumetric representation can include a plurality of voxels (e.g., voxels of the voxel blocks 402) and a plurality of textures that are respectively associated. Each voxel block can have a three-dimensional location. As an example, the three-dimensional locations (e.g., as integer x, y, z coordinates, etc.) of the voxel blocks from block group 402A will have a higher z coordinate relative to the voxel blocks from block group 402B. A plurality of texture atlases 408 can be respectively associated with the plurality of voxel blocks 402.

[0092] The encoding computing system generates a respective plurality of Morton codes 406 for the plurality of voxel blocks 402. The Morton codes can be based on three-dimensional locations (e.g., as integer x, y, z coordinates, etc.). More specifically, a binary representation of an integer that indexes a spatial location (e.g., x, y, z coordinates) of a voxel block can be converted into a representative Morton code (e.g., 2D Morton code, 3D Morton code, etc.). As an example, an integer triple that describes a three-dimensional spatial location of a voxel block can be converted into a 2D Morton code. For example, each texture atlas 408 can be indexed by a pair of integers The pair of integers 2D Morton codes is binary represented as v B-1 v B-1 … u0v0integer The conversion performed in this manner can provide compatibility with the two-dimensional regular grid structure of the texture atlas 404 (e.g., converting x, y, z variables to u, v variables, etc.).

[0093] The encoding computing system can determine a code rank 410 that ranks each of the plurality of Morton codes 406. More specifically, a three-dimensional Morton code can be generated for the texture atlas 404 that includes and ranks the Morton codes of the texture atlas 404. The texture atlas 408 can be mapped to a texture atlas location where rank is the rank of the 3D Morton code in the list of 3D Morton codes. As an example, the ranked Morton codes 410 of the voxel blocks can be used to rank their corresponding texture atlas (e.g., ranked texture atlas 412). For example, the voxel blocks from the voxel group 402A will be ranked based on their determined Morton codes in the code rank 410, and the texture atlas corresponding to the voxel blocks will be ranked in the respective locations in the ranked texture atlas 412.

[0094] The encoding computing system can determine a respective position of each texture atlas 408 in the texture map 404 based at least in part on the rank of the corresponding stereoscopic pixel block's Morton code in the code rank 410. More specifically, the respective position of a texture atlas in the ranked texture atlases 412 can correspond to the position of the respective associated stereoscopic pixel block in the code rank 410, and the ranked texture atlases can be packed into the texture map 404 based on their rank. As an example, a Morton code (e.g., a code from the Morton codes 406) can be generated for the stereoscopic pixel blocks of the block group 402A (e.g., blocks having a relatively higher z-position integer toward the "head" of the person). The stereoscopic pixel blocks can be ranked in the code rank 410 based on the relative position of the blocks (e.g., the x, y, z values of the blocks). The texture atlases (e.g., in the texture atlases 408) associated with the stereoscopic pixel blocks can be ranked in the ranked texture atlases 412 based on the stereoscopic pixel block codes ranked in the code rank 410. The ranked texture atlases 412 can be or otherwise describe the respective position of each of the texture atlases in the texture map.

[0095] Based on the texture map position of each of the texture atlases, the encoding computing system can generate a texture map including each of the plurality of texture atlases. In this way, the initial spatiotemporal location of the stereoscopic pixel blocks (e.g., 402A) can correspond to the location of the texture atlases (e.g., 404A) in the texture map 404. Similarly, blocks from different locations, such as the block group 402B, can correspond to different locations in the texture map, such as locations 404B and 404C. In some implementations, the generation of the Morton codes can be based on one spatial coordinate, but not another spatial coordinate (e.g., prefer the x coordinate, but not the y coordinate). As depicted, the generated Morton codes for the block group 402B can preferentially use the x coordinate, represented by the horizontal positioning variance between the texture atlases at locations 404B and 404C. In this way, the 3D Morton codes of the texture map 404 can preserve the three-dimensionality of the packed texture atlases using a ranking scheme, and thus can be easily extracted back into three-dimensional space by a decoder (e.g., by de-multiplexing the bits of the Morton codes, etc.).

[0096] Figure 5 A block diagram of an example training architecture for training a machine learning model is depicted in accordance with example embodiments of the present disclosure. It should be noted that although Figure 5 End-to-end training of the model is depicted, the model does not necessarily need to be trained end-to-end. More specifically, Figure 5The machine learning encoder model 501, the machine learning decoder model 503, the learned stereo pixel distribution 505, and the learned symbol distribution 507 can be trained end-to-end simultaneously, and after training, can be utilized as Figure 3 depicted in the middle. The stereo pixel block 502 can be input into the machine learning encoding model 501. As depicted, the machine learning encoding model 501 can include several convolutional layers (e.g., layer 504). However, it should be noted that any other kind of machine learning model and / or machine learning layer (e.g., neural network, recurrent neural network, LTSM (long-term short memory) layer, etc.) can be utilized.

[0097] The machine learning encoding model 501 can generate an encoded stereo pixel block (e.g., an encoded representation of the stereo pixel block 502). In some implementations, the encoded stereo pixel block can be a latent space representation of the stereo pixel block 502. More specifically, given a stereo pixel block 502 to be transmitted, the computing system can compute a lossy quantized latent representation e using the machine learning encoding model 501E with parameters θ The computing system can use the learned symbol distribution 507 on the symbol values of the stereo pixel to compute where s is the true value symbol configuration of the stereo pixel block 502, and θ s are the learnable parameters of the learned symbol distribution 505. In some implementations, the computing system can compute the bitstream and the symbol s using an entropy encoder by losslessly coding the latent code (e.g., the learned stereo pixel distribution 505) and (e.g., the learned symbol distribution 507) of the stereo pixel block 502. and s bits As an example, can be the learned stereo pixel distribution on φ parameterized by φ.

[0098] Since the quantization operation of the inference steps (e.g., 316 in Figure 3 the entropy encoding and decoding of the latent space representation is not differentiable, the quantization noise can be modeled during training, rather than explicitly discretizing the output of the machine learning encoding model 501. More specifically, the encoded stereo pixel block can be quantized by rounding to the nearest integer which can be modeled by adding uniform noise 506. The uniform noise 506 can be represented with to model the quantization error.

[0099] The encoded stereopixel blocks, along with uniform noise, can be received by the machine learning decoding model 503. As depicted, the machine learning decoding model 503 can include several convolutional layers (e.g., layer 508). However, it should be noted that any other kind of machine learning model and / or machine learning layer (e.g., neural networks, recurrent neural networks, LTSM (long-term short memory) layers, etc.) can be utilized. Additionally, as depicted, in some implementations, the machine learning decoding model 503 can utilize two convolutional heads (e.g., 510A and 510B). More specifically, in some implementations, the machine learning decoding model 503 can be or otherwise include a convolutional neural network having a final convolutional layer that includes two convolutional heads 510A and 510B. The first convolutional head 510A and the second convolutional head 510B can respectively generate the stereopixel block magnitude values 514 and the sign values 512, such that and The stereopixel block magnitude values 512 and the sign values 514 can be decoded and / or summed to generate the reconstructed stereopixel blocks 516.

[0100] The loss function 518 can evaluate the difference between the reconstructed stereopixel blocks 516 or one or more components of the reconstructed stereopixel blocks 516 (e.g., the sign values 512 and / or the magnitude values 514) and the stereopixel blocks 502. Based on the loss function 518, one or more parameters of the machine learning encoding model 501, the machine learning decoding model 503, the learned stereopixel distribution 505, and / or the learned sign distribution 507 can be adjusted. More specifically, in some implementations, the loss function 518 can evaluate distortion, potential bit rate, and sign bit rate, as described as

[0101]

[0102] In some implementations, the distortion parameter can evaluate the reconstruction error between the true values (e.g., the sign values of the stereopixel blocks 502) and the predicted stereopixel values (e.g., the predicted sign values 512 and magnitude values 514). Masks can be used to focus the training of the model on more “relevant” stereopixels of the stereopixel blocks 502 (e.g., stereopixels that have neighboring stereopixels with opposite sign values). As an example, for each dimension, a mask can be created for the important stereopixels (i.e., m x , m y , and m z ) of the stereopixel blocks 502. Stereopixels that have more than one neighbor with opposite signs can appear in multiple masks, further increasing the weight of these stereopixels. The masks can be used to compute the squared differences for the important stereopixels, as specified by the following equation for B blocks: In some implementations, the potential bit rate parameter Able to evaluate noisy code The differential entropy is estimated. As an example, the potential bit rate metric can provide an estimate of the entropy caused by noise in the code encoded by an entropy coder (e.g., simulated by uniform noise 506). In this way, the potential bit rate parameter can reduce the bit rate of the compressed code, thereby achieving more efficient compression.

[0103] In some implementations, the symbol bit rate parameter R s (s;θ s This allows for the evaluation of the lossless compression ratio of the sign value (e.g., sign value 512) of each of the multiple stereo pixels in stereo pixel block 502. Since S (e.g., the sign of stereo pixel block 502) contains only discrete values ​​{-1, +1}, it can be losslessly compressed using entropy coding. As previously mentioned, a learned sign distribution can be used. 507, instead of the prior distribution p s (s). It should be noted that in some implementations, the learned symbol distribution can have much lower entropy than the prior, because s512 is designed to rely on (e.g., the latent spatial representation of 3D pixel block 502). This allows for more efficient compression of symbol values.

[0104] Symbol rate loss R s Then it can be the cross-entropy between the real value symbol s (where -1 is remapped to 0) and symbol 512. In this way, the symbol bit rate parameter R is made... s (s;θ s Minimizing the bit rate of compressed symbol values ​​enables training models (e.g., 501 and 503) and distributions (e.g., 505 and 507) to make more accurate symbol predictions.

[0105] Example Method

[0106] Figure 6A A flowchart depicts an example method for encoding a three-dimensional volume representation according to an exemplary embodiment of this disclosure. Although for illustrative and discussion purposes, Figure 6A The steps are described in a specific order, but the method of this disclosure is not limited to the specific order or arrangement described. Various steps of method 600 can be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of this disclosure.

[0107] At 602, the computing system can obtain one or more voxel blocks from the three-dimensional representation of the object. The three-dimensional volume representation can include a plurality of voxels and a plurality of textures respectively associated therewith. Each of the plurality of voxels can include a magnitude value and a sign value. Each of the one or more voxel blocks can include a subset of the plurality of voxels.

[0108] In some implementations, the voxel blocks can include and / or can be an implicit representation (e.g., signed distance field, truncated signed distance field, etc.) of a surface of the object represented by the volume. As an example, each voxel can include a signed distance field that implicitly represents a distance (e.g., magnitude) to the surface and a sign value associated with a voxel spatial location with respect to the surface. For example, voxels located "inside" the surface of the volume can have a negative sign value, while voxels located "outside" the surface of the volume can have a positive sign value. As another example, voxels that implicitly contain the surface can have a sign and magnitude value of zero (e.g., "zero-crossings"). In some implementations, the voxels can contain a truncated signed distance field. More specifically, the magnitude values of the voxels can be truncated at a certain threshold. For example, voxels located relatively far from the surface of the volume can have a magnitude value of 15. If the magnitude truncation threshold is set to 1, then any magnitude value greater than 1 can be truncated to 1. In this way, less "relevant" data (e.g., voxels far from the surface of the volume) can be truncated, thus allowing for a more efficient representation of the volume surface.

[0109] In some implementations, the one or more voxel blocks can be obtained based at least in part on the sign values of the voxels. More specifically, the voxel blocks can have a size of k x k x k voxels (e.g., three-dimensional voxel blocks). In some implementations, each of the one or more voxel blocks can be a non-overlapping voxel block (e.g., occupied block) that contains zero-crossings. As an example, the plurality of voxels can be iterated in a block-wise manner (e.g., k x k x k voxels at a time, etc.) to determine whether a current block contains voxels with zero-crossings. If the block does contain voxels, then the block can be extracted and encoded. If the block does not contain voxels, then the block can be skipped for encoding. In this way, the computing system can extract and encode blocks relevant to the surface of the object while ignoring less relevant blocks, thus greatly reducing the number of voxel blocks that are encoded and transmitted. It should be noted that in some implementations, the size of the object and / or the volume size of the blocks can require that only one block be selected from the plurality of voxels (e.g., all voxels with zero-crossings contained in one block, etc.).

[0110] In some implementations, stereo pixel blocks can be indexed using a stereo pixel block index. The stereo pixel block index can store the spatial location of the stereo pixel blocks after they have been extracted. It should be noted that in some implementations, extracting stereo pixel blocks containing zero intersections increases the importance of maintaining spatial coherence between the extracted blocks to properly reconstruct the volumetric surface. Therefore, the stereo pixel block index can be used by a separate computing system (e.g., a decoding computing system) to reconstruct the stereo pixel blocks at their original spatial locations.

[0111] At position 604, the computing system can encode one or more stereo pixel blocks using a machine learning stereo pixel encoding model to obtain one or more encoded stereo pixel blocks. More specifically, given a block x to be transmitted, the computing system (e.g., a sending / transmitting computing system, etc.) can encode one or more stereo pixel blocks with parameter θ. e In this case, a machine learning encoding model E is used to compute the lossy quantization latent representation. The computing system is able to use The conditional probability distribution of the sign values ​​of the 3D pixels is calculated as follows: Where s is the actual value symbol configuration of the block, and θ s These are learnable parameters of the symbol distribution. In some implementations, the computational system can use an entropy encoder to learn the distribution by using the distribution separately. and For potential code Lossless encoding of symbol s is used to compute the bitstream. and s bits As an example, It can be parameterized by φ The prior learned stereo pixel distribution is used. It should be noted that in some implementations, the prior learned stereo pixel distribution and the learned symbol distribution can be trained before the training phase, as will be discussed in more detail with respect to the individual figures.

[0112] At point 606, the computing system can decode one or more coded stereo pixel blocks using a first instance of a machine learning stereo pixel decoding model (e.g., a trained convolutional neural network, recurrent neural network, etc.) to obtain a first instance of one or more reconstructed stereo pixel blocks. The reconstructed stereo pixel blocks can be lossy reconstructions of one or more stereo pixel blocks. In some implementations, the reconstructed stereo pixel blocks can be reconstructed at least partially based on stereo pixel indices describing the spatial locations of one or more stereo pixel blocks.

[0113] At 608, the computing system can generate a first instance of a reconstructed mesh representation of the object based at least in part on the one or more first instances of the reconstructed voxel blocks. The reconstructed mesh (e.g., a triangular mesh, etc.) can be a polygonal mesh representation of the surface of the object in the volume. In some implementations, the reconstructed mesh representation can include an associated plurality of textures of the three-dimensional volumetric representation. As an example, the reconstructed mesh representation can be a textured mesh representation. Alternatively, in some implementations, the reconstructed mesh representation can be reconstructed from the plurality of textures individually.

[0114] At 610, the computing system can encode the plurality of textures according to an encoding scheme to obtain a plurality of encoded textures. In some implementations, the encoding can be based at least in part on the first instance of the reconstructed mesh representation of the object. More specifically, the encoding scheme can be a traceless UV parameterization method that can be combined with the block-level geometry encoding of the one or more voxel blocks. In some implementations, the polygons of each voxel block can be extracted and grouped based at least in part on one or more characteristics of the polygons (e.g., a normal of the polygon, a texture characteristic associated with a texture of the polygon, etc.). As an example, the polygons of one voxel block can be grouped into one group. As another example, the polygons of one voxel block can be mapped to a plurality of groups.

[0115] Figure 6B A flow diagram depicting an example method for decoding a three-dimensional volumetric representation in accordance with example embodiments of the present disclosure is depicted. Although the method is depicted in a particular order for ease of discussion and explanation, Figure 6B steps are depicted in a particular order, the methods of the present disclosure are not limited to the particular illustrated order or arrangement. The various steps of the method 600 can be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure. Additionally, although Figure 6B is depicted as a continuation of the method Figure 6A , it should be noted that Figure 6B the steps of the method Figure 6A can be performed separately and independently of the steps of the method

[0116] At 612, the computing system can receive the encoded stereo pixel blocks and decode the encoded stereo pixel blocks with a second instance of the machine-learned stereo pixel decoding model to obtain a second instance of one or more reconstructed stereo pixel blocks. In some implementations, the computing system of steps 612 and subsequent steps (e.g., steps 614-618) can be a different computing system than the computing system of steps 602-610. Alternatively, in some implementations, the computing system of steps 612-618 can be the same computing system as the computing system of steps 602-610. It should be noted that the machine-learned stereo pixel decoding model can be a pre-trained deterministic model. Thus, given the same input, the output of the first instance of the machine-learned stereo pixel decoding model (e.g., on the encoding computing system) and the second instance of the model (e.g., on the decoding computing system) can produce the same deterministic output. Thus, the first instance of the reconstructed blocks on the computing system will be the same or substantially similar to the second instance of the reconstructed blocks on the separate computing system (e.g., due to transmission loss, etc.). In this way, the texture encoded based on the first instance of the reconstructed blocks will be more compatible with the second instance of the reconstructed blocks (e.g., for application of the blocks and / or multiplication of the blocks, etc.).

[0117] At 614, the computing system can decode the received encoded texture to obtain a plurality of decoded textures. The textures can be decoded according to the encoding scheme previously utilized. More specifically, the bits (e.g., Morton codes) of the texture atlas containing the textures can be de-muxed and used to decode the textures in a manner that preserves the spatial coherency of the textures.

[0118] At 616, the computing system can generate a second instance of the reconstructed mesh representation based at least in part on the second instance of the one or more reconstructed stereo pixel blocks. The reconstructed mesh representation can be generated in the same manner as the first instance of the reconstructed mesh representation (e.g., using a marching cubes algorithm with the reconstructed stereo pixel blocks, etc.). It should be noted that the second instance of the reconstructed mesh representation can be the same or substantially similar to the first instance.

[0119] At 618, the computing system can apply the second instance of the one or more reconstructed stereo pixel blocks to obtain a reconstructed three-dimensional volumetric representation of the object. In some implementations, the plurality of decoded textures can be applied to the reconstructed mesh representation (e.g., a polygon mesh, etc.) in a manner specified by the de-muxed Morton codes of the texture atlas. Since the plurality of decoded textures were packed into the texture atlas in a manner that preserves spatial and positional coherency, the textures can be easily unpacked and iteratively applied to the mesh in the same order in which the textures were extracted.

[0120] Figure 7A flowchart depicting an example method for decoding a three-dimensional volumetric representation according to example embodiments of the present disclosure is shown. Although for the purposes of illustration and discussion, Figure 7 Steps are depicted in a particular order, but the methods of the present disclosure are not limited to the particular illustrated order or arrangement. Various steps of the method 700 can be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0121] At 700, the computing system can obtain a plurality of voxel blocks from the three-dimensional volumetric representation. In some implementations, one or more voxel blocks can be obtained based at least in part on the sign values of the voxels. More specifically, a voxel block can have a size of k x k x k voxels (e.g., a three-dimensional voxel block). In some implementations, each of the one or more voxel blocks can be a non-overlapping voxel block containing a zero-crossing (e.g., an occupied block). As an example, the plurality of voxels can be iterated in a block-wise manner (e.g., k x k x k voxels at a time, etc.) to determine whether a current block contains voxels with a zero-crossing. If the block does contain voxels, then the block can be selected. If the block does not contain voxels, then the block can be skipped. In this way, voxel blocks that are relevant to the surface of an object in the three-dimensional volumetric representation can be selected while blocks that are less relevant are ignored, thus greatly reducing the number of voxel blocks used.

[0122] The three-dimensional volumetric representation can include a plurality of voxels and a plurality of textures associated therewith, respectively. Each voxel block can have a three-dimensional location. As an example, the three-dimensional locations (e.g., as integer x, y, z coordinates, etc.) of the voxel blocks from a first group of blocks (e.g., around the head level of a person) can have a higher z coordinate relative to the voxel blocks from a second group of blocks (e.g., around the feet level of the person).

[0123] A plurality of texture atlases can be respectively associated with a plurality of solid pixel blocks. In some embodiments, a plurality of texture atlases can be respectively extracted from each of the solid pixel blocks. As an example, for each block, the textures (e.g., polygons, triangles, etc.) in the block can be extracted and grouped by their normals. Most blocks have only one group, while blocks in more complex regions (e.g., a human finger) can have more than one respectively associated group. The vertices of the textures (e.g., polygons, etc.) in each texture atlas can be mapped to UV space as follows: (1) the average normal in the group is used to determine tangent space, onto which the vertices in the group are projected. (2) the projections are rotated using a rotation gauge until they fit into an axis-aligned rectangle with the smallest area. Note that in some embodiments, this can result in a deterministic UV coordinate for each vertex in the group relative to the bounding box of the vertex projection. (3) the bounding boxes of the groups respectively associated with the solid pixel blocks are then sorted by size, and packed into texture atlases using a quadtree-like algorithm. In some embodiments, there can be one 2D atlas for each solid pixel block. The UV coordinates of the vertices in the block can be offset relative to the texture atlas.

[0124] At 704, the computing system can generate a respective plurality of Morton codes for the plurality of solid pixel blocks. The Morton codes can be based on three-dimensional locations (e.g., as integer x, y, z coordinates, etc.). More specifically, a binary representation of an integer that indexes a spatial location (e.g., x, y, z coordinates) of a solid pixel block can be converted into a representative Morton code (e.g., 2D Morton code, 3D Morton code, etc.). As an example, an integer triple that describes a three-dimensional spatial location of a solid pixel block can be converted into a 2D Morton code. For example, each texture atlas can be indexed by a pair of integers indexed, the 2D Morton code of the pair of integers is the binary representation v B-1 v B-1 ... u0v0 The conversion made in this way can provide compatibility with the two-dimensional regular grid structure of the texture atlas (e.g., converting x, y, z variables to u, v variables, etc.).

[0125] At 706, the computing system can determine a respective position of each texture atlas in the texture atlas based at least in part on a rank of the Morton code of the corresponding solid pixel block in the code ranking. More specifically, a three-dimensional Morton code can be generated for the texture atlas, which includes and ranks the Morton codes of the texture atlas. The texture atlas can be mapped to a texture atlas position where rank is the rank of the 3D Morton code in the list of 3D Morton codes. As an example, the ranked Morton codes of the stereoscopic pixel blocks can be used to rank their corresponding texture atlas. For example, stereoscopic pixel blocks from a first stereoscopic pixel group can be ranked based on their determined Morton codes in the code rank, while the texture atlas corresponding to the stereoscopic pixel blocks can have a determined position in the texture atlas that corresponds to the rank of the stereoscopic pixel blocks in the code rank.

[0126] At 708, the computing system can generate the texture atlas. The texture atlas can include each of the plurality of texture atlases. The position of the texture atlas in the texture atlas can be based at least in part on the corresponding determined texture atlas position. The atlas of blocks can then be packed into the atlas. After the atlas packing, the UV coordinates of the vertices can be offset again with respect to the texture atlas (e.g., offset to or otherwise include global UV mapping, etc.). After the UV parameterization, color information can be obtained from each vertex color in the geometry, the previously generated atlas, or even the original RGB capture. In this way, the initial spatiotemporal location of the stereoscopic pixel blocks can correspond to the location of the texture atlas in the texture atlas. Similarly, blocks from different locations can correspond to different locations in the texture atlas. In some implementations, the generation of the Morton code can be based on one spatial coordinate, but not another spatial coordinate (e.g., prefer the x coordinate over the y coordinate, etc.). In this way, the 3D Morton code of the texture atlas can use a ranking scheme to preserve the three-dimensionality of the packed texture atlas, and thus can be easily extracted back into three-dimensional space by a decoder (e.g., by de-multiplexing the bits of the Morton code, etc.).

[0127] Additional disclosure

[0128] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, and actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or a plurality of devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0129] While the present subject matter has been described in detail with respect to various specific example embodiments of the disclosure, it should be kept in mind that those specific example embodiments are not intended to limit the present disclosure in any way. Those skilled in the art will readily appreciate that various modifications, changes, and adaptations can be made to such embodiments without departing from the essence of the present disclosure, which is to provide a method and system for providing a user interface for a virtual reality environment. Accordingly, the disclosure is intended to encompass all such modifications, changes, and adaptations as fall within the scope of the present subject matter. For example, features described or illustrated as part of one embodiment can be used with another embodiment to yield still a further embodiment. Thus, the present disclosure is not intended to be limited to the embodiments described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A computer-implemented method for performing compression of a three-dimensional volumetric representation, the method comprising: obtaining, by an encoding computing system comprising one or more computing devices, one or more blocks of voxels from a three-dimensional volumetric representation of an object, wherein the three-dimensional volumetric representation comprises a plurality of voxels and a corresponding plurality of textures associated therewith, wherein each of the plurality of voxels comprises a magnitude value and a sign value, and wherein each of the one or more blocks of voxels comprises a subset of the plurality of voxels; encoding, by the encoding computing system, the one or more blocks of voxels with a machine-learned voxel encoding model to obtain one or more encoded blocks of voxels; decoding, by the encoding computing system, the one or more encoded blocks of voxels with a first instance of a machine-learned voxel decoding model to obtain a first instance of one or more reconstructed blocks of voxels; generating, by the encoding computing system, a first instance of a reconstructed mesh representation of the object based at least in part on the first instance of the one or more reconstructed blocks of voxels; and encoding, by the encoding computing system, at least a portion of the plurality of textures according to an encoding scheme and based at least in part on the first instance of the reconstructed mesh representation of the object to obtain a plurality of encoded textures.

2. The computer-implemented method of claim 1, further comprising: encoding, by the encoding computing system, the one or more encoded blocks of voxels using an entropy encoder to obtain one or more entropy-encoded blocks of voxels; and transmitting, by the encoding computing system, the one or more entropy-encoded blocks of voxels, a block of voxels index, and the plurality of encoded textures to a decoding computing system located remotely from the encoding computing system, wherein the block of voxels index describes a spatial location of each of the one or more blocks of voxels.

3. The computer-implemented method of claim 2, further comprising: decoding, by a decoding computing system, the one or more encoded blocks of voxels with a second instance of the machine-learned voxel decoding model to obtain a second instance of the one or more reconstructed blocks of voxels; decoding, by the decoding computing system, the plurality of encoded textures according to the encoding scheme to obtain a plurality of decoded textures; and applying, by the decoding computing system, the plurality of decoded textures to the second instance of the one or more reconstructed blocks of voxels to obtain a reconstructed three-dimensional volumetric representation of the object.

4. The computer-implemented method of claim 3, wherein, applying, by the decoding computing system, the plurality of decoded textures to the second instance of the one or more reconstructed blocks of voxels to obtain the reconstructed three-dimensional volumetric representation of the object comprises: generating, by the decoding computing system, a second instance of the reconstructed mesh representation of the object based at least in part on the second instance of the one or more reconstructed blocks of voxels; and applying, by the decoding computing system, the decoded plurality of textures to the second instance of the reconstructed mesh representation of the object to obtain the reconstructed three-dimensional volumetric representation.

5. The computer-implemented method of claim 4, wherein, obtaining, by the encoding computing system, the one or more encoded stereopixel blocks further comprises: encoding, by the encoding computing system, the one or more encoded stereopixel blocks using the entropy encoder to obtain one or more entropy encoded stereopixel blocks.

6. The computer-implemented method of claim 1, wherein, Each stereopixel comprises a truncated signed distance field.

7. The computer-implemented method of claim 1, wherein, At least one of the machine-learned stereopixel encoding model or the machine-learned stereopixel decoding model comprises a convolutional neural network configured to convolve tensors across the one or more stereopixel blocks.

8. The computer-implemented method of claim 3, wherein, Encoding, by the encoding computing system, the one or more stereopixel blocks with the machine-learned stereopixel encoding model to obtain the one or more encoded stereopixel blocks further comprises: encoding, by the encoding computing system, the sign values of the one or more stereopixel blocks with the machine-learned stereopixel encoding model to obtain encoded sign values associated with the one or more encoded stereopixel blocks, respectively, the encoded sign values being based on a learned sign distribution, the learned sign distribution being conditioned on the one or more encoded stereopixel blocks.

9. The computer-implemented method of claim 2, wherein: Encoding, by the encoding computing system, the one or more encoded stereopixel blocks using the entropy encoder to obtain one or more entropy encoded stereopixel blocks further comprises: encoding, by the encoding computing system, the one or more encoded sign values using the entropy encoder to obtain entropy encoded sign values; and Transmitting, by the encoding computing system, the one or more entropy encoded stereopixel blocks to the decoding computing system further comprises: transmitting, by the encoding computing system, the entropy encoded sign values to the decoding computing system.

10. The computer-implemented method of claim 8, wherein, Decoding, by the decoding computing system, the one or more encoded stereopixel blocks and the one or more encoded sign values with the second instance of the machine-learned stereopixel decoding model to obtain the second instance of the one or more reconstructed stereopixel blocks.

11. The computer-implemented method of claim 9, wherein, The entropy encoded sign values comprise lossless encoding of the sign values of the plurality of stereopixels.

12. The computer-implemented method of claim 8, wherein: Encoding the one or more encoded stereopixel blocks based at least in part on a learned stereopixel distribution; Training at least one of the learned stereopixel distribution, the learned sign distribution, the machine-learned stereopixel encoding model, or the machine-learned stereopixel decoding model based on a loss function; and The loss function evaluates a difference between the three-dimensional volumetric representation and the reconstructed three-dimensional volumetric representation.

13. The computer-implemented method of any one of claims 1-12, encoding, by the encoding computing system, at least a portion of the plurality of textures according to the encoding scheme and based at least in part on the first instance of the reconstructed mesh representation of the object to obtain a plurality of encoded textures comprises: extracting, by the encoding computing system, a plurality of polygons from the reconstructed mesh representation; grouping, by the encoding computing system, the plurality of polygons into one or more polygon groups based at least in part on one or more polygon characteristics of each of the plurality of polygons; generating, by the encoding computing system, a polygon chart comprising the one or more polygon groups, the polygon chart configured to maintain spatial coherence between each of the one or more polygon groups; and mapping, by the encoding computing system, each of the one or more polygon charts to a texture atlas, wherein spatial locations of the one or more polygon charts in the texture atlas correspond to the spatial locations of the one or more reconstructed voxel blocks.

14. A computing system comprising: one or more processors; and one or more non-transitory computer-readable media collectively storing a first set of instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: obtaining one or more voxel blocks from a three-dimensional volumetric representation of an object, wherein the three-dimensional volumetric representation comprises a plurality of voxels and a correspondingly associated plurality of textures, wherein each of the plurality of voxels comprises a magnitude value and a sign value, and wherein each of the one or more voxel blocks comprises a subset of the plurality of voxels; encoding, with a machine-learned voxel encoding model, the one or more voxel blocks to obtain one or more encoded voxel blocks and correspondingly associated encoded sign values, the one or more encoded voxel blocks based at least in part on a learned voxel distribution, the encoded sign values based at least in part on a learned sign distribution conditioned on the one or more encoded voxel blocks; decoding, with a machine-learned voxel decoding model, the one or more encoded voxel blocks to obtain one or more reconstructed voxel blocks; decoding, with the machine-learned voxel decoding model, the encoded sign values to obtain decoded sign values correspondingly associated with the one or more reconstructed voxel blocks; and evaluating a loss function that evaluates a difference between the one or more voxel blocks and the one or more reconstructed voxel blocks.

15. The computing system of claim 14, wherein, the operations further comprising adjusting, based at least in part on the loss function, one or more parameters of at least one of: the machine-learned voxel encoding model; the machine-learned voxel decoding model; the learned voxel distribution; or the learned sign distribution.

16. The computing system of claim 14, wherein, the loss function evaluates a difference between a subset of voxels in each of the one or more voxel blocks and the one or more reconstructed voxel blocks, the subset specified by a mask based at least in part on the sign value of each voxel in the subset of voxels.

17. The computing system of claim 14, wherein, the loss function further evaluates a difference between the decoded sign values and the sign values of the plurality of voxels of the three-dimensional volumetric representation.

18. The computing system of claim 14, wherein, adding uniform noise to the machine-learned voxel encoding model during encoding.

19. The computing system of claim 14, wherein, the loss function further evaluates a size of the one or more encoded voxel blocks.

20. The computing system of any one of claims 14 to 19, wherein, The machine-learned stereo pixel decoding model includes a convolutional neural network having a final convolutional layer including two convolutional heads, a first convolutional head of the two convolutional heads configured to generate the one or more reconstructed stereo pixel blocks, and a second convolutional head of the two convolutional heads configured to generate the decoded symbol values.

21. A computer-implemented method for performing compression of textures associated with a three-dimensional volumetric representation, the method comprising: obtaining, by an encoding computing system comprising one or more computing devices, a plurality of stereo pixel blocks from a three-dimensional volumetric representation, wherein the three-dimensional volumetric representation includes a plurality of stereo pixels and a corresponding plurality of textures associated therewith, wherein each stereo pixel block has a three-dimensional location, and wherein a plurality of texture maps are respectively associated with the plurality of stereo pixel blocks; generating, by the encoding computing system, a respective plurality of Morton codes for the plurality of stereo pixel blocks; determining, by the encoding computing system, a code rank that ranks each of the plurality of Morton codes; determining, by the encoding computing system, for each texture map, a respective location in a texture atlas based at least in part on the rank of the Morton code of the corresponding stereo pixel block in the code rank; and generating, by the encoding computing system, the texture atlas including the respective location of each of the plurality of texture maps based at least in part on the respective texture atlas location.

22. The computer-implemented method of claim 21, wherein, each of the locations of the texture maps is described by a Morton code.

23. The computer-implemented method of any one of claims 21 to 22, wherein, generating, by the computing system, the respective plurality of Morton codes for the plurality of stereo pixel blocks includes converting, by the computing system, two-dimensional coordinates of each of the plurality of stereo pixel blocks into a corresponding Morton code.

24. A computer-implemented method for performing decompression of a three-dimensional volumetric representation, the method comprising: receiving, by a decoding computing system comprising one or more computing devices, one or more encoded stereo pixel blocks and a corresponding plurality of encoded textures corresponding to the three-dimensional volumetric representation, wherein the plurality of encoded textures are encoded based at least in part on a reconstructed mesh representation generated from one or more reconstructed stereo pixel blocks produced at a different computing system by a first instance of a machine-learned stereo pixel decoding model; decoding, by the decoding computing system, the one or more encoded stereo pixel blocks with a second instance of the machine-learned stereo pixel decoding model to obtain the one or more reconstructed stereo pixel blocks; decoding, by the decoding computing system, the plurality of encoded textures according to an encoding scheme to obtain a plurality of decoded textures, wherein the plurality of encoded textures are encoded according to the encoding scheme at the different computing system; generating, by the decoding computing system, the reconstructed mesh representation based at least in part on the one or more reconstructed stereo pixel blocks; and applying, by the decoding computing system, the decoded plurality of textures to the reconstructed mesh representation to obtain a reconstruction of the three-dimensional volumetric representation.

Images