Graphical texture reconstruction

Abstract

Certain aspects of the present disclosure provide techniques for reconstructing texels of a texture. Such techniques can include receiving a plurality of feature sets corresponding to a texture, where the plurality of feature sets includes a respective feature set for each respective grid point of a grid; receiving coordinate information corresponding to a texel of the texture; receiving detail level information; selecting a subset of grid points of the grid based on a second resolution being lower than a first resolution; sampling one or more grid points from the subset of grid points based on the coordinate information to obtain sampled features associated with the one or more grid points; inputting the sampled features to a machine learning model; and receiving, from the machine learning model, a reconstruction of the texel of the texture at the second resolution based on the sampled features.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Patent Application No. 63 / 557,998, filed February 26, 2024; U.S. Provisional Patent Application No. 63 / 562,148, filed March 6, 2024; and U.S. Provisional Application No. 18 / 649,690, filed April 29, 2024, the entire contents of which are incorporated herein by reference. Background Technology Technical Field

[0004] Various aspects of this disclosure relate to computer vision, and more specifically to techniques for performing graphical texture reconstruction.

[0005] Related technical descriptions

[0006] Techniques for compressing and reconstructing graphics textures are used in various fields, including video games, virtual reality, and special effects rendering. Specifically, in the field of graphics reconstruction, the goal is to efficiently compress textures while still retaining the ability to reconstruct textures at multiple resolutions and to provide random access to texture data. This allows for the efficient storage and streaming of more textures while still providing the access patterns needed to render complex 3D scenes.

[0007] For example, techniques have been introduced to store pre-computed texture pyramids containing multiple texture resolutions. While this provides direct access to different resolutions, these techniques require significant storage overhead. Alternative techniques have relied on parameterized “mesh” optimized during training to contain potential texture representations across resolutions. However, these mesh-based techniques require storing the mesh at multiple resolutions to explicitly handle texture decoding at the desired resolution.

[0008] To provide random access capabilities for rendering, mesh-based techniques typically rely on dedicated decoding architectures that can randomly sample texture data from the mesh. However, these decoder architectures are limited to operating on a mesh at a predetermined resolution. The techniques have not yet adequately decoupled the decoding resolution from the set mesh resolution to enable more flexible decoding from a compressed latent space.

[0009] Furthermore, existing learning-based methods require joint optimization of decoder parameters and the data representation encoded within the grid. These methods can be computationally intensive. Improved techniques that offer lower optimization overhead while still providing the necessary decoding flexibility remain desirable. Summary of the Invention

[0010] One aspect provides a method for reconstructing texels of a texture. In some aspects, the method may include: receiving a plurality of feature sets corresponding to a texture, wherein the plurality of feature sets include a corresponding feature set for each corresponding grid point of a mesh, wherein each corresponding grid point of the mesh is associated with a corresponding portion of the texture, wherein the mesh has a first resolution; receiving coordinate information corresponding to the texels of the texture; receiving level-of-detail information indicating a second resolution for reconstructing the texture, wherein the second resolution is lower than the first resolution; selecting a subset of grid points of the mesh based on the second resolution being lower than the first resolution; sampling one or more grid points from the subset of grid points based on the coordinate information to obtain sampled features associated with the one or more grid points; inputting the sampled features into a machine learning model; and receiving a reconstruction of the texels of the texture at the second resolution from the machine learning model based on the sampled features.

[0011] Other aspects provide: an apparatus capable of operating to, being configured to, or otherwise adapted to perform one or more of the methods described herein and / or those elsewhere in the document; a non-transitory computer-readable medium comprising instructions that, when executed by a processor of the apparatus, cause the apparatus to perform the methods described herein and those elsewhere in the document; a computer program product embodied on a computer-readable storage medium comprising: code for performing the methods described herein and those elsewhere in the document; and / or an apparatus comprising components for performing the methods described herein and those elsewhere in the document. By way of example, an apparatus may include a processing system, a device having a processing system, or a processing system cooperating via one or more networks.

[0012] For illustrative purposes, the following description and figures illustrate certain features. Attached Figure Description

[0013] The accompanying drawings depict certain features of the various aspects described herein and should not be considered as limiting the scope of this disclosure.

[0014] Figure 1 Details of a graphical texture reconstruction system according to various examples of this disclosure are depicted.

[0015] Figure 2A A block diagram depicts an exemplary training process for a graphical texture reconstruction system according to various aspects of this disclosure.

[0016] Figure 2B Example grids are depicted according to various aspects of this disclosure.

[0017] Figure 3Additional details of the encoder of the graphic texture reconstruction system according to various aspects of this disclosure are described.

[0018] Figure 4 Additional details of the decoder for the graphics texture reconstruction system according to various aspects of this disclosure are described.

[0019] Figure 5 An example AI architecture is shown that can be used for artificial intelligence (AI) enhanced wireless communication.

[0020] Figure 6 An example AI architecture of a first wireless device communicating with a second wireless device is illustrated.

[0021] Figure 7 An example artificial neural network is shown.

[0022] Figure 8 An example method for performing graphical texture reconstruction is described.

[0023] Figure 9 Various aspects of the example device are described. Detailed Implementation

[0024] This disclosure provides apparatus, methods, processing systems, and computer-readable media for performing graphics texture reconstruction. In some aspects, graphics reconstruction can be based on machine learning (ML), such as neural networks.

[0025] More specifically, some aspects involve methods for compressing textures and / or using compressed textures to reconstruct textures as part of graphics rendering, such as for applications involving rendering in graphic arts, games, and virtual reality environments. A texture can be a representation of material rendered on objects in an environment. For example, an environment may include a wall, and that wall might represent a brick wall. Therefore, a brick texture can be rendered on the surface of the wall to provide the appearance of a brick wall.

[0026] Compared to digital image compression and reconstruction, texture compression and reconstruction presents several additional challenges. Specifically, digital image compression and reconstruction involves compressing an image at one resolution and then reconstructing the entire image at the same resolution using the compressed image. In contrast, graphic textures may need to be rendered at different resolutions (e.g., MIP or MIPMAP levels, where the letters in MIP represent Latin phrases). multum in parvo(This implies finding the big picture in the small details) and further, only a portion of the texture needs to be rendered, thus requiring random access to the texture. For example, it might be necessary to render different parts / views of the environment based on changes in the view to be rendered, such as due to changes in the part of the environment to be displayed and / or user input at the zoom level. Based on the part / view of the environment to be displayed and / or the zoom level, different parts of the texture might need to be rendered at different resolutions on the surface of the object. Additionally, depending on the material, a given texture can be formed from several different layers / components of the texture, thus requiring the rendering of several layers for a single texture. Therefore, textures can be formed from texture sets, where a texture set is a collection of texture components / layers, such as diffuse components, normal components, roughness components, subsurface components, etc.

[0027] As discussed, a technical problem exists in graphics texture compression and reconstruction where current techniques require storing texture data across multiple resolutions for explicit use in handling texture decoding at other desired resolutions. This storage incurs high storage and computational overhead.

[0028] The aspects described in this paper provide graphics reconstruction techniques that may not require storing texture data across multiple resolutions. Such techniques offer technical solutions to the problems discussed and provide technical benefits in reducing storage requirements and computational overhead. For example, the aspects described in this paper provide techniques in which an encoder generates a compressed latent representation of a texture as a feature set “mesh.” This mesh can be sampled during decoding to reconstruct textures at multiple resolutions without having a predetermined texture pyramid level.

[0029] In some respects, the encoder can be configured to output a feature set at a fixed mesh resolution, regardless of the texture content or the target reconstruction resolution. For example, the mesh resolution might match the highest resolution of the texture to be compressed. In other respects, during decoding, striding techniques can be used to sample this fixed mesh at different rates to produce varying output resolutions. This provides flexibility without explicitly using multiple mesh resolutions to handle different texture resolutions.

[0030] Example Graphics Texture Reconstruction System

[0031] Figure 1 Details of a graphic texture reconstruction system 100 based on various aspects of this disclosure are depicted. For example... Figure 1The depicted graphics texture reconstruction system 100 may include a rendering engine 102 configured to render graphics using one or more textures from various images that can be displayed on a display device (103) and / or integrated into a processing system. In some aspects, the rendering engine 102 may render portions of the texture at varying resolutions (e.g., at different MIP levels). In other aspects, it may be necessary to render only specific portions of the texture, thus requiring random access to the texture data needed for graphics rendering. For example, based on user input that changes the view or zoom level of the displayed or rendered environment, various segments or perspectives of the environment may be rendered to reflect changes in the scene being displayed or rendered. Therefore, different portions of the texture may need to be rendered at different resolutions on the surface of an object based on the portion / view and / or zoom level of the environment to be displayed or rendered.

[0032] According to various aspects of this disclosure, rendering engine 102 may include texture engine 104, which is configured to provide portions of a texture required for rendering to rendering engine 102. In an example, the portion of the texture provided by texture engine 104 may correspond to a texel or basic unit of the texture. In some aspects, the texture may include texture set 106, wherein texture set 106 includes a group of associated texture components / layers 108 representing different material properties of the texture. A texel is a specific value at a given location within each component / layer of the component / layer 108. For example, an example texture set 106 may correspond to a brick texture mapped by rendering engine 102 onto the surface of a wall in a rendering environment. Brick texture set 106 may include components / layers 108 representing different properties of the brick used by rendering engine 102 to render the appearance of the brick. These components / layers 108 may include, but are not limited to, a diffuse component 108A containing color / albedo information, a roughness component 108B containing spatial roughness variation information, a bump or normal component 108C containing simulated bump / normal details for lighting calculations, a displacement component 108D containing depth details for mesh displacement, a subsurface scattering component 108E related to simulated light diffusion beneath the surface, an ambient occlusion component 108F containing pre-calculated ambient occlusion shadows, and / or additional components 108G related to other aspects of the brick material's appearance and / or behavior. The different components / layers 108 of the texture set 106 enable the rendering engine 102 to render more realistic and visually rich 3D objects. That is, the different components / layers 108 enable the rendering engine 102 to render various material properties and behaviors corresponding to different lighting conditions of real-world brick materials.

[0033] Since storing and accessing texture set 106 at different resolutions can be resource-intensive (e.g., requiring significant storage resources to store the components / layers 108 of each texture set 106 in rendering), in some aspects, texture engine 104 is configured to reconstruct texture information from compressed texture information stored in one or more meshes 112. In some aspects, to facilitate compressed storage of texture set 106, graphics texture reconstruction system 100 utilizes encoder 110, which is configured to generate a compressed representation of texture set 106 such that the compressed representation of texture set 106 can be stored as one or more meshes 112. In some aspects, encoder 110 utilizes machine learning architectures, such as neural network architectures, such as, but not limited to, convolutional autoencoder architectures including convolutional layers and downsampling layers. In some aspects, encoder 110 can output a reduced latent space representation of texture set 106 in the form of meshes 112. More specifically, encoder 110 can be configured to receive texture set 106 and output one or more feature meshes 112 corresponding to the components / layers 108 of texture set 106. Mesh 112 allows texture set 106 to be reconstructed at different resolutions while utilizing less storage space than is required to store multiple resolutions of texture set 106.

[0034] A mesh may include a set of mesh points corresponding to a mesh resolution, such as 2×2 mesh points. Each mesh point may represent a portion of the original texture (referred to herein as a “tile”). For example, the original texture may be divided into a set of tiles corresponding to the number of mesh points in the mesh. In one example, the original 1024×1024 texture may be divided into 2×2 tiles, such that each tile corresponds to 512×512 texels of the original texture. Each mesh point may represent one texel tile out of the 512×512 texel tiles. Each mesh point may correspond to a set of features associated with the tile associated with the mesh point, and the position of the mesh point within the mesh (e.g., index). Each feature in the feature set of the mesh point may represent a combination of one or more different types of texture components / layers 108 of texture set 106. The number of features per mesh point may be, for example, 32, and each feature may be referred to as a channel. Each feature may represent a tile as a whole of the original texture, not just a specific texel of the tile. Therefore, a mesh may include multiple sets of features, with each mesh point of the mesh including a corresponding set of features. As used herein, the term “set” may refer to one or more, unless otherwise specified for a particular case. As used in this article, the term "subset" can refer to less than the entirety of a "set". Therefore, when referring to a subset of a set, the set must necessarily include multiple elements, and the subset is less than the total of those multiple elements.

[0035] As another example, grid 112 may correspond to the storage location of multiple grids of multiple texture sets 106. Example grid 114 may include a first grid G0 116A corresponding to multiple components / layers 108 of texture set 106. In one example, the first grid G0 116A may be a 128×128 grid (e.g., l = 128, w = 128) comprising 128×128 grid points 118. Assuming the source texture set 106 has a resolution of 1024×1024, each grid point 118 represents an 8×8 texel tile of the source texture set 106, such that the complete first grid G0 116A represents the entire texture set 106 in a compressed 128×128 representation. In some examples, each grid point 118 of the first grid G0 116A may be represented as a d-dimensional tensor, where d equals the number of features per grid point. In some aspects, texture set 106 may be represented using a single grid (e.g., the first grid G0 116A).

[0036] In some aspects, the output features from encoder 110 can be separated into multiple grids (e.g., 116A and 116B) associated with texture set 106. That is, each grid (e.g., 116A, 116B) may include different features from the feature set. For example, each feature set at each grid point may include 48 features. In some aspects, grid G0 116A may include 118 channels per grid point, while grid G1 116B may include 32 channels per grid point. Multiple grids 116A and 116B may have the same resolution (e.g., share their corresponding tile regions), such that grid G0 116A and grid G1 116B together represent the complete texture set 106 in a compressed form. In some aspects, although for clarity... Figure 1 A two-dimensional mesh is depicted, but one or more meshes 114 may include tensors of three or higher dimensions, wherein mesh points 118 are arranged in a matching multidimensional manner. Figure 1 The depicted mesh structure allows for random access during decoding to reconstruct the texels corresponding to the texture set.

[0037] Texture engine 104 may include sampler 120 and decoder 122. In some aspects, sampler 120 is configured to sample one or more meshes 112 to extract features from mesh points 118 corresponding to regions of interest in texture set 106 for decoding. In some aspects, sampler 120 may receive location information indicating the region of interest in texture set 106 for decoding. Furthermore, sampler 120 may receive level of detail information indicating the desired resolution for reconstructing texels. In some examples, rendering engine 102 may provide location and level of detail information to sampler 120. Based on the level of detail and location information, sampler 120 may select a subset of mesh points from one or more meshes 114 (e.g., 116A and / or 116B) and sample the subset of mesh points using a sampling algorithm (e.g., nearest neighbor, bilinear sampling, bicubic sampling, etc.) to obtain sampled features associated with one or more mesh points. In some aspects, the sampled features associated with one or more mesh points correspond to interpolated mesh point features of the desired texture region.

[0038] In some aspects, decoder 122 is configured to reconstruct texels from compressed mesh representation 114 using sampled features associated with one or more mesh points sampled by sampler 120. In some aspects, decoder 122 utilizes a machine learning model to reconstruct texels based on sampled features, level-of-detail information, and positional information associated with one or more mesh points sampled by sampler 120. Decoder 122 may be configured to reconstruct texels at multiple resolutions 124. Thus, in some aspects, decoder 122 may reconstruct texel 126A at a first resolution (e.g., MIP0). Alternatively or additionally, decoder 122 may reconstruct texel output 126B at a second resolution (e.g., MIP1). As another example, decoder 122 may reconstruct texel 126C at another resolution (e.g., MIPn). One or more of these resolutions (e.g., MIP0, MIP1, MIPn) may be greater than, equal to, or less than the resolution of texture set 106. Furthermore, one or more of these resolutions (e.g., MIP0, MIP1, MIPn) may be smaller than another of these resolutions (e.g., MIP1 may be smaller than MIP0, and MIPn may be smaller than MIP1).

[0039] Rendering engine 102 may receive reconstructed texels (e.g., 126A) from decoder 122 and use the reconstructed texels (e.g., 126A) to render an image, such as an image displayed at display device 103. In some aspects, rendering engine 102 may include texture engine 104. Alternatively or additionally, texture engine 104 may be separate from rendering engine 102. Although in Figure 1Resolutions MIP0, MIP1, and MIPn are depicted, but it should be understood that decoder 122 may reconstruct texels at resolutions different from those shown. Furthermore, texture set 106 may include resolutions greater than those shown. Figure 1 The example illustrates more or fewer components / layers 108. Additionally, grids 112 and 116A / 116B may have different resolutions than previously described.

[0040] Example Graphics Texture Reconstruction System Training

[0041] Figure 2A A block diagram depicts an exemplary training process 200 of a graphics texture reconstruction system 100 according to various aspects of this disclosure. As relative to... Figure 1 As described, in some respects, encoder 110 and decoder 122 utilize machine learning components to decode the texture set and decode the sampled grid points to reconstruct texels. Figure 2A An exemplary training process 200 for training encoder 110 and decoder 122 is described to improve texture set compression and texel reconstruction accuracy when processing texture set 106.

[0042] In some respects and such Figure 2A As depicted, the training process 200 includes providing one or more source texture sets 202 comprising multiple components / layers 204 to train the encoder 110 and decoder 122. Such as relative to Figure 1 As described, encoder 110 receives input texture set 202 and outputs a latent space representation in the form of one or more grids 116A, 116B, each grid comprising a corresponding set of features for each corresponding grid point of the one or more grids 116A, 116B. In some respects, each corresponding grid point is associated with a corresponding portion of the input texture set 202. Encoder 110 is trained under the guidance of loss function 218 to output grids (e.g., 116A, 116B) representing a compressed version of the input texture set 106.

[0043] Training process 200 enables decoder 122 to reconstruct texels from meshes 116A, 116B at multiple resolution levels (e.g., 220), including full-resolution MIP 0 (e.g., 216A) down to lower resolution MIP n levels (e.g., 216C). Therefore, in some respects, sampling of each of one or more meshes (116A, 116B) is based on the (e.g., predicted) resolution used to render the texels. For example, when the resolution used to render the texels is lower than the mesh resolution, the mesh can be sampled by excluding (also known as striding) certain mesh points during sampling. For example, when the resolution used to render the texture is 64×64 and the mesh resolution is 128×128, every other mesh point in every other row of the mesh can be used for sampling, instead of sampling using all mesh points of the mesh. For example, the four mesh points closest to a specific coordinate in every other row of the mesh can be sampled.

[0044] When the resolution used to render the texture is greater than or equal to the resolution of the mesh, the entire mesh can be used for sampling. When the resolution used to render the texture is less than the resolution of the mesh, a subset of mesh points can be selected for sampling. The subset of mesh points selected for sampling can be based on a stride level, where the stride level can be based on the ratio between the resolution used to render the texture and the resolution of the mesh. For example, if the resolution used to render the texture is 2×2 and the resolution of the mesh is 6×6, then the stride level could be 3 (e.g., 6 / 2).

[0045] As an example of striding Figure 2B Example grid 116A is shown, where numbers represent grid points. As shown, such grid 116A is illustrated as having 5×5 grid points, numbered from 0 to 24. Assuming a stride level of 2, the first grid point included in the subset could be grid point 0, and each additional grid point included in the subset is determined by adding the stride level to the previously selected grid points in the subset until the end of the grid, such that the additional grid points included in the subset would be 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, and 24. A stride level of 3 could give the following grid points in the subset: 0, 3, 6, 9, 12, 15, 18, 21, 24. A stride level of 5 could give the following grid points in the subset: 0, 5, 10, 15, 20.

[0046] Therefore, as part of the training, the texture training input 210 may indicate a level of detail, which provides a resolution level input indicating the target MIP level for each training iteration. This MIP level guides the stride selector 206 to configure the sampling stride for the sampler 120. Additionally, the texture training input 210 may indicate location information corresponding to the regions reconstructed from the texels. Thus, the sampler 120 provides the sampled mesh features, along with optional location information, to the decoder 122. In some respects, the decoder 122 utilizes a machine learning model 208 (e.g., a multilayer perceptron) to reconstruct the texels 220 using the sampled mesh features against the location and resolution indicated by the level of detail and resolution level in the texture training input 210.

[0047] As training progresses, loss function 218 calculates the difference between the reconstructed texels 220 of decoder 122 and the original samples of texture set 202, driving adjustments to the weights of the machine learning model within decoder 122 to iteratively reduce reconstruction errors at one or more MIP levels 216A, 216B, 216C. Additionally, the weights of encoder 110 can be updated via backpropagation of decoder 122 errors to further optimize the mesh generated by encoder 110. In this way, coordinated training of encoder 110 and decoder 122 enables texture compression and reconstruction at multiple resolutions while storing textures in a compact mesh representation (e.g., 116A, 116B).

[0048] Here is an example of a loss function:

[0049] Here, Indicates the weights of the decoder. Indicates the encoder weights, Represents coordinates relative to the texture (Using texture as input to the encoder, utilizing) The decoder's texel output (using the operation) (to perform the operation), T Representing coordinates The actual texels at the location. During training, encoder 110 and decoder 122 run multiple times with different textures as training input 210 to output reconstructed texels at different resolutions and coordinates of the textures, and use loss function 218 to adjust the weights of the machine learning model implemented by encoder 110 and the machine learning model implemented by decoder 122. Therefore, the weights of encoder 110 and decoder 122 can be specifically trained for a given texture set 202, such that different texture sets are associated with different weights. In other respects, encoder 110 and decoder 122 can be trained across multiple textures, such that multiple textures are associated with the same weights.

[0050] In some respects, encoder 110 and decoder 122 can be trained to support multiple sampling algorithms, thereby using the sampler configuration of sampler 120 to construct localized mesh feature inputs. For example, as training progresses, sampler 120 can change sampling techniques, such as nearest neighbor sampling, bilinear sampling, and bicubic sampling, to account for different sampling patterns. In some examples, different sampling patterns may be based on texture set 202, desired compression level, mesh resolution, etc.

[0051] In some respects, the encoder 110 and decoder 122 are trained in multiple stages. For example, when using texture sets... At this time, the initial portion of the texture set may be randomly cropped to a size of 256×256. Subsequently, the cropped texture set can be passed to encoder 110, followed by one or more mesh constructors (e.g., as described below). Figure 4 (further described in the text), which helps in constructing the corresponding mesh pairs. (For example, Figure 1 (116A and 116B).

[0052] In some respects, the encoder 110 and decoder 122 are trained jointly using an alternating process. For example, in the first stage, the weights of the encoder 110 may remain fixed, while the weights of the decoder 122 may be updated based on the gradient of the loss function 218 relative to the decoder weights. In some respects, this allows the decoder 122 to adapt the current output of the encoder 110 and improve its reconstruction of the input texture. In some respects, and during the second stage, the weights of the decoder 122 may remain fixed, while the weights of the encoder 110 may be updated based on the gradient of the loss function 218 relative to the encoder weights, which can be obtained by backpropagating the gradient through the decoder 122. In some respects, this allows the encoder 110 to adapt its output (e.g., intermediate grids and feature information) to improve the reconstruction quality of the fixed decoder 122. Such an example training process may alternate between the two stages in multiple iterations. In each iteration, the encoder 110 may process the input texture to generate intermediate grids and feature information, which the decoder 122 can then use to reconstruct the texture. The loss function 218 can compare the reconstructed texture with the original input texture and can compute the gradient relative to the decoder weights (in the first stage) and the encoder weights (in the second stage).

[0053] For texel reconstruction, MIP levels proportional to the area of ​​randomly cropped MIP levels can be randomly selected. In the example, this random selection is achieved by sampling from an exponential distribution with a rate parameter λ = log₄. To address the issue associated with undersampling of low-resolution MIP levels, a certain proportion (e.g., 10%) of the training batches can be randomly selected from a uniform distribution spanning the entire MIP level chain, thus helping to ensure a more balanced representation across different resolution levels during training. In one or more subsequent training phases, the cropping size C... s The number of steps can be increased by a factor (e.g., factor 2) until the decoder 122 is able to reconstruct the complete MIP-level chain. Therefore, in some examples, training can utilize batch sizes of, for example, 4 and, for example, 10. -4 The learning rate (LR), which increases with the pruning size at each stage, can be reduced by a factor (e.g., 2). In the final stage, 10 can be used. -5 The LR is used to train the model (e.g., 20,000 steps).

[0054] In some respects and during the early training phase, Additive uniform noise within the range can be replaced by quantization, where This is the desired number of bits allocated to store each element of the grid. However, during the final training phase, the feature values ​​can be quantized using a pass-through estimator (STE) to optimize the grid pairs. All eigenvalues ​​are subject to a fixed quantization rate. .

[0055] In some aspects, bit-per-pixel per-channel (BPPC) can be measured, which encompasses both the bits required to allocate grid pairs and decoder-specific parameters that are uniquely trained for each texture set. Used to store each grid pair (in The total number of bits required for the storage grid is determined by... Given, among which Represents a grid The number of channels, and This represents the resolution of the texture set. Furthermore, the total number of bits required to store the decoder corresponds to the total number of parameters in the decoder multiplied by the parameter precision. To control the bit rate, the mesh pairs... The number of channels can be changed, and the size and number of hidden layers in one or more of the encoder 110 or decoder 122 can also be changed.

[0056] In some aspects, encoder 110 may be an autoencoder configured to compress an input texture into a latent representation, which may include multiple sets of features corresponding to grid points. Decoder 122 may be configured to reconstruct texels from the latent representation. The training process of the autoencoder may include performing multiple iterations across multiple MIP levels of the input texture. In some aspects, during each iteration, the input texture encoder 110 at a specific MIP level may be processed to generate a latent representation. Decoder 122 may then use the latent representation to reconstruct texels at the same MIP level. A loss function (e.g., loss function 218) may then be used to compare the reconstructed texels with the original texels, and the gradient of the loss may be backpropagated through decoder 122 and encoder 110 to update their respective weights.

[0057] In some respects, a progressive approach can be used to train the autoencoder, starting with the highest MIP level (i.e., the lowest resolution) and then moving to lower MIP levels (i.e., higher resolution). This allows the autoencoder to first learn coarse-grained features of the low-resolution texture and then progressively refine the details at higher resolutions. In some respects, the training process may involve additional techniques (such as adversarial training), where a discriminator model can be used to provide additional feedback to the autoencoder to improve the realism of the reconstructed texels. In some respects, a perceptual loss function can be used to train the autoencoder, which uses a pre-trained neural network to compare high-level features extracted from the reconstructed texels and the original texels.

[0058] Example encoder of a graphics texture reconstruction system

[0059] Figure 3 An example encoder is described (e.g., Figure 1 Details of the encoder 110. In some respects, the encoder may implement an autoencoder neural network architecture that includes jointly trained encoder and decoder sub-components to generate a compressed intermediate mesh representation of the input texture set.

[0060] like Figure 1 As described in Figure 2, the trained encoder 302 (e.g., corresponding to...) Figure 1The encoder 110 (components) may include one or more machine learning components that undergo training process 200 to generate features stored as grid points corresponding to portions of the texture set. In some aspects, the trained encoder 302 includes optional residual blocks 304A to 304B, optional convolutional blocks 306, and optional feature scaling blocks 308 to construct a compact latent spatial grid representation (e.g., 116A, 116B) corresponding to the input texture set 106. In some aspects, residual blocks 304A and 304B are characterized by skip connections that enhance the training of the deep neural network. That is, residual connections can send information more directly across different layers, thereby improving information flow and accelerating training. In some aspects, residual blocks 304A and 304B may include one or more Conv layers and one or more LeakyReLU layers, such as... Figure 3 As depicted. In an additional aspect, encoder 302 applies an activation function, such as feature scaling using the hyperbolic tangent function, after one or more convolutional blocks 306 prior to pooling / downsampling. This scaling and normalization of the feature maps provides boundaries to the range of feature values ​​to aid subsequent optional linear quantization during compression (e.g., 310A / 312A and 310B / 312B, which in some respects may be...). Figure 1 The features of encoder 110 are scaled to a discrete level. For example, feature scaling block 308 can limit values ​​to a predefined range, such as between -0.5 and 0.5. This scaling and normalization of the feature map enables subsequent scalar quantization blocks 312A to 312B to quantize the mesh feature values ​​to a discrete level during compression. The combined use of residual blocks 304A, 304B and feature scaling block 308 improves compression efficiency when compressing texture set 106 into one or more mesh representations. Each of these meshes (e.g., meshes 116A and 116B) can be stored in a storage location that stores one or more meshes 112.

[0061] In some respects, encoder 302 (e.g., Texture sets Mapping to bottleneck latent representation , making Encoder 302 can generate potential bottlenecks. It has a resolution reduced by, for example, a factor of 8, e.g., h z =h / 8 and w z =w / 8, and by applying at the last layer of encoder 302 Constrained to subspace In some examples, the encoder input is a texture set T with a resolution of h×w, because the resolution is reduced to, for example, 4×4 (…). ∈ , 0 ≤ m ≤ M = The texture set corresponding to the MIP level is obtained by scaling down T, where such information is included in the texture set. T middle.

[0062] In some respects, blocks 310A / 310B can be referred to as mesh constructors. It will potentially create bottlenecks. Mapped to mesh pairs G0 116A and G1 116B, where In some respects, the grid It is the size of A tensor that stores quantized features to reconstruct texture sets at various MIP levels. In some examples, the mesh constructor... It can be a linear projection scalar quantizer , where 312A / 312B can correspond to this type of scalar quantizer. Therefore, In order to quantize the mesh Features Indicates range Asymmetric scalar quantization, where It is allocated for storage The expected number of bits for each element.

[0063] Example decoder for a graphics texture reconstruction system

[0064] Figure 4 Details of the selection, sampling, and decoding aspects of the graphics texture reconstruction system 100 according to various aspects of this disclosure are described. For example, relative to... Figure 1 As described in Figure 2, the texture engine 104 of the graphics texture reconstruction system 100 includes a trained decoder that can reconstruct texels from a latent spatial grid representation (e.g., grid 114) with varying target resolutions.

[0065] In some respects, sampler 120 first receives texture request 404 from rendering engine 102 (such as...). Figure 1 (As depicted). Rendering engine 102 ( Figure 1 The sampler 120 can output a request to sample and reconstruct portions of a stored texture set (e.g., mesh 112) at desired locations and resolutions during graphics rendering. Texture request 404 indicates parameters guiding the appropriate reconstruction, including coordinate information of a specific texel at a specific coordinate of the texture to be reconstructed, and desired level of detail information indicating the resolution (e.g., MIP level) used to reconstruct the texels (e.g., 126A, 126B, 126C). The coordinate information and the desired level of detail information can be provided to sampler 120.

[0066] In some respects, it can be based on rendering engine 102 ( Figure 1 The initial texture request 404 guides the sampling process to obtain sampled mesh features based on the information indicated therein. For example, based on the indicated level of detail 416, the selector (e.g., Figure 2A The stride selector 206 can select a subset of grid points in one or more grids 112 to be sampled using the stride as discussed. For example, when reconstructing a lower resolution output, Figure 2A The step selector 206 can use stepping to sparsely identify grid points at wider intervals, instead of densely selecting grid points at each point.

[0067] Sampler 120 can map coordinate information to a grid space of grid 112 (such as grids 116A and 116B). For example, it can identify tiles including texels (e.g., 405A, 405B) and corresponding grid points indicated by coordinate information.

[0068] In some aspects, grid points of grid 112 (such as a selected subset of grid points) may be sampled. In some aspects, sampler 120 performs sampling (such as nearest neighbor sampling and / or bilinear sampling) using grid points (such as a subset of grid points) to obtain one or more features or inputs (e.g., nearest neighbor samples 406 and / or bilinear samples 408) to a trained decoder 402 (e.g., Figure 1 Examples of decoder 122 are provided, thereby reconstructing texels. For instance, in nearest neighbor sampling, sampler 120 may identify (e.g., a subset of grid points) the four nearest grid points in grid 112 for a texel (e.g., 405A, 405B) indicated by coordinate information and obtain a feature set of such identified grid points for input into trained decoder 402. In bilinear sampling, sampler 120 may identify (e.g., a subset of grid points) the four nearest grid points in grid 112 for a texel (e.g., 405A, 405B) indicated by coordinate information and generate a weighted average of the feature sets of such identified grid points, which are weighted based on the distance between the texel and each of the identified grid points to obtain a weighted average feature set for input into trained decoder 402.

[0069] In some respects, sampler 120 can output sampled grid point features as a grid feature vector from one or more grids 112, as a localized representation of a texture set focused on a region of interest, such as in the texels to be reconstructed. Decoder 402 can use the input features to reconstruct the texels.

[0070] In some respects, given the coordinates (x, y) of MIP level m, the mesh sampler (For example, implemented as 120) locate the surrounding voxels and cascade them at each corner of the voxels. Dimensional features and output Grid sampler (For example, implemented as 120) Look up surrounding voxels in a similar way, but store the information at each corner of the voxel based on the relative position of (x, y) within the voxel. Perform linear interpolation on dimensional features and output In this way, It can capture more detailed features of the texture set, and Capture more abstract information.

[0071] In the example, and as discussed earlier, a single resolution grid pair can be used instead of multiple grid pairs with different resolutions for each subset of the MIP level. That is, to account for resolutions lower than the grid pair... MIP level > 3) Can take a step Sampling is performed; therefore, after locating the top-left corner of the surrounding voxels, a step relative to the located top-left corner is used. To select additional voxel angles.

[0072] In some aspects, optionally, one or more additional inputs are fed into the trained decoder 402, such as level of detail 416, texel coordinates 414 corresponding to the coordinate information of texture request 404, or localization encoding information 412, which can enhance texel reconstruction. Localization encoding information 412 may represent the localization of a specific texel coordinate within a tile represented by grid points corresponding to sampled features. Localization encoding information 412 may be the value of a function using specific coordinates as input, representing the localization of the specific coordinate within the tile represented by grid points. In some examples, localization encoding information 412 may correspond to a specific (x, y) coordinate position in texture space, which is mapped to or encoded as a localization vector representing the target reconstructed position of the texel in grid space. Decoder 402 may be further trained with such additional one or more inputs.

[0073] In some respects and such Figure 4As depicted, the trained decoder 402 utilizes sampled mesh features from mesh sampler 120, optionally incorporating one or more of detail level 416, localization encoding information 412, and / or coordinates 414, to generate reconstructed texel output 126 (e.g., 126A to 126C). While the architecture of decoder 402 may vary, in some respects, decoder 402 includes a neural network structure (e.g., 420) to infer the desired texture details from finite mesh input features. As an example, the decoded input features are first processed by an initial dense neural network layer 422A to expand the finite features to higher dimensions. Next, one or more residual linear layers 424A and 424B allow the signal to be propagated deeper into the decoder network to improve reconstruction accuracy. In some examples, residual linear layer 424 may include one or more linear layers and one or more leakyReLU layers, such as... Figure 4 As depicted, skip connections combine feature outputs from multiple decoder layers to preserve both local and global texture features. Finally, the linear output layer 422B compresses the higher-dimensional temporary decoder output to the desired texel feature dimension that matches the texel properties (e.g., the size of the reconstructed texels). Figure 4 The depicted architecture allows decoder 402 to transform sampled mesh feature inputs into fully reconstructed texel outputs 126A, 126B, or 126C based on the MIP level received in texture request 404.

[0074] In some respects, the trained decoder 402 will process grid samples and Cascaded, normalized MIP level and positioning code Mapping to MIP level texels at coordinates (x, y) In some respects, the trained decoder 402 includes fully connected layers with skip connections.

[0075] In some aspects, reconstructing texels of a texture may involve determining whether a second resolution (e.g., the resolution at which texels are to be reconstructed) is lower than, equal to, or greater than a first resolution (e.g., the resolution of the mesh). This determination may be based on received level-of-detail information indicating the second resolution. For example, when it is determined that the second resolution is lower than the first resolution, a subset of mesh points may be selected from the mesh using sampler 120. The selection of the subset may be performed in response to determining that the second resolution is lower than the first resolution. The subset of mesh points may be selected such that the mesh point density in the subset corresponds to the lower second resolution. For example, if the second resolution is half the first resolution in every dimension, the subset may include every other mesh point in each dimension. For example, the subset of mesh points selected for sampling may be based on a stride level, where the stride level may be based on the ratio between the resolution used to render the texture and the resolution of the mesh.

[0076] In some respects, when it is determined that the second resolution is equal to the first resolution, subset selection can be omitted and grid points can be used directly for sampling. This is because the grid point density in the original grid already corresponds to the desired output resolution. In other respects, when it is determined that the second resolution is greater than the first resolution, the entire grid can be used for sampling, making grid points available for sampling.

[0077] Example AI system for performing graphic texture reconstruction

[0078] Some aspects described in this paper can be implemented, at least in part, using some form of artificial intelligence (AI), such as the process of using a machine learning (ML) model to infer or predict output data based on input data. Example ML models may include mathematical representations of one or more relationships between various objects to provide outputs representing one or more predictions or inferences. Once an ML model has been trained, it can be deployed to process data that is wholly or partially similar to or related to the training data, and to provide outputs representing one or more predictions or inferences based on the input data.

[0079] ML is typically characterized by a learning type that generates a specific type of learning model that performs a particular type of task. For example, different types of machine learning include supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

[0080] Supervised learning algorithms typically model the relationships and dependencies between input features (e.g., feature vectors) and one or more target outputs. Supervised learning uses labeled training data, which consists of data including one or more inputs and the desired output. Supervised learning can be used to train models to perform tasks such as classification (where the goal is to predict discrete values) or regression (where the goal is to predict continuous values). Some example supervised learning algorithms include nearest neighbor, Naive Bayes, decision trees, linear regression, support vector machines (SVM), and artificial neural networks (ANN).

[0081] Unsupervised learning algorithms process unlabeled input data and train models that take the input and transform it into output to solve real-world problems. Examples of unsupervised learning tasks are clustering (where the model's output might be cluster labels), dimensionality reduction (where the model's output is an output feature vector with fewer features than the input feature vector), and outlier detection (where the model's output is a value indicating how the input differs from typical examples in the dataset). An example unsupervised learning algorithm is k-means.

[0082] Semi-supervised learning algorithms process datasets containing both labeled and unlabeled examples, where the number of unlabeled examples is typically much greater than the number of labeled examples. However, the goal of semi-supervised learning is to achieve the goals of supervised learning. Typically, a semi-supervised model involves a model trained to generate pseudo-labels for unlabeled data, which are then combined with labeled data to train a second classifier that leverages a larger volume of overall training data to improve task performance.

[0083] Reinforcement learning algorithms use observations gathered by an agent from its interactions with the environment to take actions that maximize reward or minimize risk. Reinforcement learning is a continuous and iterative process in which the agent learns from its experience with the environment until it has explored, for example, the entire range of possible states. An example type of reinforcement learning algorithm is adversarial networks. Reinforcement learning can be particularly beneficial when used to improve or attempt to optimize the behavior of models deployed in dynamically changing environments, such as wireless communication networks.

[0084] ML models can be deployed in one or more devices (e.g., network entities such as base stations and / or user equipment) to support various wired and / or wireless communication aspects of a communication system. For example, ML models can be trained to identify patterns and relationships in data corresponding to networks, devices, air interfaces, etc. ML models can improve operations associated with one or more aspects, such as transceiver circuitry control, frequency synchronization, timing synchronization, channel state estimation, channel equalization, channel state feedback, modulation, demodulation, device location, transceiver tuning, beamforming, signal decoding / decoding, network routing, load balancing, and energy saving (to name just a few). AI-enhanced transceiver circuitry control may include, for example, filter tuning, transmit power control, gain control (including automatic gain control), phase control, power management, etc.

[0085] The aspects described herein can be used to describe technical solutions for performing certain tasks and various technical problems by applying specific types of ML models, such as ANNs. However, it should be understood that other types of AI models can be used as supplements or alternatives to ANNs. ML models can be examples of AI models, and any suitable AI model can be used as a supplement or alternative to any ML model described herein. Therefore, unless explicitly stated otherwise, the topic of ML models is not necessarily intended to be limited to ANN solutions or machine learning. Furthermore, it should be understood that, unless specifically stated otherwise, terms such as “AI model,” “ML model,” “AI / ML model,” or “trained ML model” are intended to be used interchangeably.

[0086] Figure 5 This is an illustration of an example AI architecture 500 that can be used to perform graphics texture reconstruction, as shown above relative to... Figures 1 to 4 As described. Figure 5 As illustrated, architecture 500 includes multiple logical entities such as model training host 502, model inference host 504, data source 506, and agent 508. The AI ​​architecture can be used in any of the various use cases of wireless communication, such as those listed above.

[0087] The model inference host 504 in architecture 500 is configured to run an ML model based on inference data 512 provided by data source 506. The model inference host 504 may produce an output 514 (e.g., a prediction or inference, such as discrete or continuous values) based on the inference data 512, and then provide it as input to agent 508.

[0088] The intelligent agent 508 can be a component or entity of a wireless communication system, including, for example, a radio access network (RAN), a wireless local area network, a device-to-device (D2D) communication system, etc. As an example, the intelligent agent 508 can be a user equipment (UE), a base station, or any of its decomposed network entities (including centralized units (CUs), distributed units (DUs), and / or radio units (RUs)), an access point, a radio station, a RAN intelligent controller (RIC) in a cloud-based RAN, etc. Additionally, the type of the intelligent agent 508 may also depend on the type of task performed by the model inference host 504, the type of inference data 512 provided to the model inference host 504, and / or the type of output 514 generated by the model inference host 504.

[0089] For example, if the output 514 from the model inference host 504 is associated with texture reconstruction of a 3D scene, then the agent 508 could be a user device including a game console GPU or a dedicated graphics card. As another example, if the output 514 from the model inference host 504 is associated with reconstructed texels, then the agent 508 could be a rendering engine or a texture management module.

[0090] After agent 508 receives output 514 from model inference host 504, agent 508 can determine whether to take action based on the output. For example, if agent 508 is a rendering engine or texture management module and output 514 from model inference host 504 is associated with reconstructed texels, then if the location or region associated with the reconstructed texel is occluded, agent 508 can determine to discard the reconstructed texel. As another example, agent 508 decides to render a texture, and then agent 508 can provide texel data to the shading pipeline stage.

[0091] Data source 506 can be configured to collect data that can be used as training data 516 for training an ML model or as inference data 512 for feeding ML model inference operations. Specifically, data source 506 can collect data from any of various entities (e.g., texture maps designed by an artist or procedural generation methods), which may include action agents 510 and provide the collected data to model training host 502 for ML model training. For example, after action agent 510 (e.g., a shading pipeline) receives reconstructed texels from agent 508, action agent 510 can provide performance feedback associated with the reconstructed texels to data source 506, where the performance feedback can be used by model training host 502 to monitor and / or evaluate ML model performance, such as whether the output 514 (e.g., a prediction) provided to agent 508 is accurate. In some examples, if the output 514 provided to agent 508 is inaccurate (or its accuracy is below an accuracy threshold), model training host 502 can determine, for example, to modify or retrain the ML model used by model inference host 504 via ML model deployment / update.

[0092] For example, after a new stone wall texture is compressed by the encoder and decompressed by the decoder, a graphic designer can provide subjective quality feedback to the data source. If the perceived quality is below a threshold, excessive blurring or block artifacts can indicate that the decoder model needs to be retrained to improve the reconstruction. In some examples, if the decoder output is inaccurate compared to the source texture, the model training host 502 can determine to fine-tune the model parameters or switch to an augmentation architecture via model updates.

[0093] In some respects, the model training host 502 may be deployed at the same or a different entity as the entity that deploys the model inference host 504, or together with that entity. For example, to offload model training processing that might affect the performance of the model inference host 504, the model training host 502 may be deployed at a model server, as further described herein. Furthermore, in some cases, training and / or inference may be distributed across devices in a decentralized or federated manner.

[0094] Figure 6 An example AI architecture is illustrated for a first wireless device 602 communicating with a second wireless device 604. The first wireless device 602 can be used to perform graphical texture reconstruction, as described herein relative to... Figures 1 to 5 As described herein. Similarly, the second wireless device 604 can be used to perform graphics texture reconstruction, as described herein relative to... Figures 1 to 5 As described. Note that the AI ​​architecture of the first wireless device 602 can be applied to the second wireless device 604.

[0095] The first wireless device 602 may be or may include a chip, a system-on-a-chip (SoC), a system-in-package (SiP), a chipset, a package, or a device, which includes one or more processors, processing blocks, or processing elements (collectively, “processor 610”) and one or more memory blocks or elements (collectively, “memory 620”).

[0096] As an example, in transmission mode, processor 610 can transform information (e.g., packets or data blocks) into modulated symbols. As digital baseband signals (e.g., digital in-phase (I) and / or quadrature (Q) baseband signals representing corresponding symbols), processor 610 can output the modulated symbols to transceiver 640. Processor 610 can be coupled to transceiver 640 for transmitting and / or receiving signals via one or more antennas 646. In this example, transceiver 640 includes radio frequency (RF) circuitry 642, which can be coupled to antenna 646 via interface 644. As an example, interface 644 can include a switch, duplexer, double-ended converter, multiplexer, etc. RF circuitry 642 can, for example, use a digital-to-analog converter to convert digital signals into analog baseband signals. RF circuitry 642 can include any circuitry of various types, including, for example, baseband filters, mixers, frequency synthesizers, power amplifiers, and / or low-noise amplifiers. In some cases, RF circuitry 642 can up-convert baseband signals to one or more carrier frequencies for transmission. Antenna 646 can transmit RF signals, which can be received at the second wireless device 604.

[0097] In receive mode, RF signals received via antenna 646 (e.g., from a second wireless device 604) can be amplified and converted to a baseband frequency (e.g., down-conversion). The received baseband signal can be filtered and converted into digital I or Q signals for digital signal processing. Processor 610 can receive the digital I or Q signals and further process them, for example, demodulate them.

[0098] One or more ML models 630 may be stored in memory 620 and accessible by processor 610. In some cases, different ML models 630 with different characteristics may be stored in memory 620, and a particular ML model 630 may be selected based on its characteristics and / or application and the characteristics and / or conditions of the first wireless device 602 (e.g., power state, mobility state, battery reserve, temperature, etc.). For example, ML models 630 may have different inference data and output pairings (e.g., different types of inference data produce different types of outputs), and predictions (e.g., Figure 5The output of 514 is associated with different accuracy levels (e.g., 80%, 90%, or 95% accuracy), different latency associated with generating predictions (e.g., processing time less than 10ms, 100ms, or 1 second), different ML model sizes (e.g., file size), different coefficients or weights, etc.

[0099] Processor 610 can use ML model 630 based on input data (e.g., Figure 5 The inferred data 512) is used to generate output data (e.g., Figure 5 The output is 514), for example, as in this paper relative to Figure 5 The inference host 504 is described. The ML model 630 can be used to perform any AI augmentation task in a variety of AI augmentation tasks, such as those listed above.

[0100] As an example, the ML model 630 can generate texel reconstructions corresponding to the requested level of detail and coordinate information. Input data may include, for example, one or more meshes, level of detail, and coordinate location information. Output data may include, for example, reconstructed texel values ​​corresponding to regions of interest in the texture and / or texture set, as previously described. It should be noted that other input and / or output data may be used as a supplement to or alternative to the examples described herein.

[0101] In some respects, model server 650 can perform any ML model lifecycle management (LCM) task for the first wireless device 602 and / or the second wireless device 604. Model server 650 can serve as... Figure 5 The model training host 502 operates and updates the ML model 630 using the training data. In some cases, the model server 650 can operate as... Figure 5 The data source 506 operates to collect and host training data, inference data, and / or performance feedback associated with the ML model 630. In some respects, the model server 650 can host various types and / or versions of the ML model 630 for download by the first wireless device 602 and / or the second wireless device 604.

[0102] In some cases, model server 650 can monitor and evaluate the performance of ML model 630 to trigger one or more LCM tasks. For example, model server 650 can determine whether to activate or deactivate the use of a specific ML model at first wireless device 602 and / or second wireless device 604, and model server 650 can provide such instructions to the respective first wireless device 602 and / or second wireless device 604. In some cases, model server 650 can determine whether to switch to a different ML model 630 used at first wireless device 602 and / or second wireless device 604, and model server 650 can provide such instructions to the respective first wireless device 602 and / or second wireless device 604. In yet other examples, model server 650 can also act as a central server for decentralized machine learning tasks such as federated learning.

[0103] Example Artificial Intelligence Model

[0104] Figure 7 This is an exemplary block diagram of an example artificial neural network (ANN) 700.

[0105] ANN 700 may receive input data 706, which may include one or more bits of data 702, preprocessed data output from preprocessor 704 (optionally), or some combination thereof. Here, for example, depending on the development and / or deployment phase of ANN 700, data 702 may include training data, validation data, application-related data, etc. In some other embodiments, preprocessor 704 may be included within ANN 700. Preprocessor 704 may, for example, process all or part of data 702, which may result in some parts of data 702 being altered, replaced, deleted, etc. In some embodiments, preprocessor 704 may add additional data to data 702.

[0106] The ANN 700 includes at least one first layer 708 of artificial neurons 710 (e.g., perceptrons) to process input data 706, and provides the resulting first layer output data to at least a portion of at least one second layer 714 via edge 712. The second layer 714 processes the data received via edge 712 and provides the second layer output data to at least a portion of at least one third layer 718 via edge 716. The third layer 718 processes the data received via edge 716 and provides the third layer output data to at least a portion of a final layer 722 comprising one or more neurons via edge 720 to provide output data 724. All or part of the output data 724 may be further processed in some way by an (optional) post-processor 726. Thus, in some embodiments, the ANN 700 can provide output data 728 based on the output data 724, post-processed data output from the post-processor 726, or some combination thereof. In some other embodiments, the post-processor 726 may be included within the ANN 700. Postprocessor 726 may process all or part of output data 724, which may result in output data 728 being at least partially different from output data 724, for example, due to data being changed, replaced, deleted, etc. In some embodiments, postprocessor 726 may be configured to add additional data to output data 724. In this example, the second layer 714 and the third layer 718 represent intermediate or hidden layers that may be arranged in a hierarchical or other similar structure. Although not explicitly shown, one or more additional intermediate layers may exist between the second layer 714 and the third layer 718.

[0107] The structure and training of the artificial neurons 710 in each layer can be customized according to the specific requirements of the application. Within a given layer of an ANN, some or all of the neurons can be configured to process the information provided to that layer and output corresponding transformed information from that layer. For example, the transformed information from a layer can represent a weighted sum of input information associated with a nonlinear activation function or another activation function used to “activate” the artificial neurons in the next layer, or otherwise based on that nonlinear activation function or the other activation function used to “activate” the artificial neurons in the next layer. Artificial neurons in such layers can be activated by or in response to weights and biases that can be adjusted during the training process. The weights of various artificial neurons can act as parameters controlling the connection strength between layers or between artificial neurons, while the biases can act as parameters controlling the connection direction between layers or between artificial neurons. Activation functions can select or determine whether an artificial neuron sends its output to the next layer in response to the data received by the artificial neuron. Different activation functions can be used to model different types of nonlinear relationships. By introducing nonlinearity into the ML model, activation functions allow the ML model to “learn” the input data (e.g., Figure 5Complex patterns and relationships in (506). Some non-exhaustive example activation functions include linear functions, binary step functions, sigmoid, hyperbolic tangent (tanh), rectified linear unit (ReLU) and its variants, exponential linear unit (ELU), Swish, Softmax, etc.

[0108] Design tools (such as computer applications, programs, etc.) can be used to select the appropriate structure and number of layers for the ANN 700, as well as the number of artificial neurons in each layer, and to select activation functions, loss functions, training procedures, etc. Once the initial model is designed, it can be trained using training data. Training data may include one or more datasets within which the ANN 700 can detect, determine, identify, or discover patterns. Training data can represent various types of information, including written, visual, audio, environmental context, operational attributes, etc. During training, the parameters of the artificial neurons 710 can be changed, such as to minimize or otherwise reduce the loss function or cost function. The training process can be repeated multiple times to fine-tune the ANN 700 with each iteration.

[0109] Various ANN model architectures are available for consideration. For example, in a feedforward ANN architecture, each artificial neuron 710 in a layer receives information from the previous layer and similarly generates information for the next layer. In a convolutional ANN architecture, some layers can be organized as filters that extract features from data (e.g., training data and / or input data). In a recursive ANN architecture, some layers may have connections that allow data to be processed across time, such as for processing information with temporal structure (e.g., time series data prediction).

[0110] In the autoencoder ANN architecture, compact representations of data can be processed, and models can be trained to make predictions or potentially reconstruct the original data from a reduced set of features. The autoencoder ANN architecture can be used for tasks related to dimensionality reduction and data compression.

[0111] Generative adversarial network (GAN) architectures can include generator ANNs and discriminator ANNs trained to compete against each other. GANs are ANN architectures that can be used for tasks related to generating synthetic data or improving the performance of other models.

[0112] Transformer ANN structures utilize an attention mechanism that enables the model to process input sequences in a parallel and efficient manner. The attention mechanism allows the model to focus on different parts of the input sequence at different times. This attention mechanism can be implemented using a series of layers called attention layers to compute, compute, determine, or select a weighted sum of input features based on the similarity between different elements of the input sequence. A transformer ANN structure may include a series of feedforward ANN layers that "learn" the non-linear relationship between the input and output sequences. The output of the transformer ANN structure can be obtained by applying a linear transformation to the output of the final attention layer. Transformer ANN structures are particularly useful for tasks involving sequence modeling or other similar processing.

[0113] Another example type of ANN structure is a model with one or more invertible layers. This type of model can be inverted or "unfolded" to reveal the input data used to generate the output of the layers.

[0114] Other examples of ANN model architectures include fully connected neural networks (FCNN) and long short-term memory (LSTM) networks.

[0115] ANN 700 or other ML models can be implemented in various types of processing circuits, as well as their memory and applicable instructions, for example, as described in this paper relative to... Figure 6 and Figure 7 As described. For example, the model can be implemented using general-purpose hardware circuitry such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs). One or more ML accelerators, such as tensor processing units (TPUs), embedded neural processing units (eNPUs) or other dedicated processors, and / or field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc., can also be used to develop ANN models. Various programming tools can be used to develop ANN models.

[0116] Various aspects of artificial intelligence model training

[0117] There are deployable ML models (such as...) Figure 7 The various model training techniques and processes used at some point before or after the ANN 700.

[0118] As part of the model development process, information may be collected or otherwise created in a suitable training data format for training the ML model accordingly. For example, training data may be collected or otherwise created relating to data on received / transmitted signal strength, interference, and resource usage, as well as any other relevant data that can be used to train the model to solve one or more problems or challenges in a communication system. In some instances, all or part of the training data may originate from one or more user equipment (UEs), one or more network entities, or one or more other devices in a wireless communication system. In some cases, all or part of the training data may be aggregated from multiple sources, such as one or more UEs, one or more network entities, the Internet, etc. For example, wireless network architectures (such as self-organizing networks (SONs) or mobile-driven test (MDT) networks) may be adapted to support data collection for ML model applications. In another example, training data may be generated or collected online, offline, or both online and offline by UEs, network entities, or other devices, and all or part of such training data may be transmitted or shared (real-time or near real-time), such as through store-and-forward functions. Offline training may refer to, for example, creating and using a static training dataset in a batch manner, while online training may refer to collecting and using training data in real-time or near real-time. For example, online or offline training can be used to train and / or fine-tune the ML model at a network device (e.g., a UE). For offline training, data collection and training can occur offline at the network side (e.g., at a base station or other network entity) or at the UE side. For online training, training of the ML model at the UE side can be performed locally at the UE or by a server device (e.g., a server hosted by the UE vendor) based on data provided from the UE to the server device, in a real-time or near real-time manner.

[0119] In some instances, all or part of the training data may be shared within the wireless communication system, or even shared (or obtained from outside) the wireless communication system.

[0120] Once the ML model has been trained on the training data, its performance can be evaluated. In some scenarios, evaluation / validation tests can be conducted using a validation dataset, which may include data not present in the training data, to compare the model's performance against a baseline or other benchmark information. If the model's performance is deemed unsatisfactory, fine-tuning the model may be beneficial, for example, by changing its architecture, retraining it on the data, or using different optimization techniques. Once the model's performance is deemed satisfactory, it can be deployed accordingly. In some instances, the model may be updated in some way, such as by changing or replacing all or part of the model, or by undergoing further training, to name just a few examples.

[0121] As for use in ANNs (such as Figure 7 As part of the training process of an ANN (Advanced Neural Network 700), parameters affecting the operation of artificial neurons and layers can be tuned. For example, backpropagation can be used to train an ANN by iteratively adjusting the weights and / or biases of certain artificial neurons associated with the error between the model's predicted output and the expected output, which may be known or otherwise considered acceptable. Backpropagation may include forward propagation, loss function, backward propagation, and parameter updates that can be performed during training iterations. This process can be repeated a certain number of times for each training dataset until the weights of the artificial neurons / layers are properly tuned.

[0122] Backpropagation, associated with the loss function, measures how well a model can predict the desired output for a given input. Optimization algorithms can be used during training to adjust weights and / or biases to reduce or minimize the loss function, which improves model performance. Various optimization algorithms exist that can be used with backpropagation or other training techniques. Some initial examples include gradient descent-based and stochastic gradient descent-based optimization algorithms. Stochastic gradient descent (or ascent) can be used to adjust weights / biases to minimize or otherwise reduce the loss function. Mini-batch gradient descent, a variant of gradient descent, involves updating weights / biases using mini-batch training data instead of the entire dataset. Momentum techniques can accelerate the optimization process by adding momentum terms to update or otherwise influence certain weights / biases.

[0123] Adaptive learning rate techniques adjust the learning rate of an optimization algorithm that is associated with one or more characteristics of the training data. Batch normalization techniques can be used to normalize the input to a model in order to stabilize the training process and potentially improve the model's performance.

[0124] The "drop-out" technique can be used to randomly discard some of the artificial neurons from the model during the training process, for example, to reduce overfitting and potentially improve the model's generalization.

[0125] The “early stopping” technique can be used to stop an ongoing training process early, such as when the performance of a model using a validation dataset begins to degrade.

[0126] Another example technique includes data augmentation, which generates additional training data by applying transformations to all or part of the training information.

[0127] Transfer learning techniques can be used, which involve using a pre-trained model as a starting point for training a new model. This can be useful when training data is limited or when there are multiple tasks that are related to each other.

[0128] Multi-task learning techniques can be used, which involve training a model to perform multiple tasks simultaneously to potentially improve the model's performance on one or more of these tasks. In some instances, hyperparameters can be input and applied during the training process.

[0129] Another example technique that can be useful for ML models is some form of "pruning." Pruning techniques, which can be performed during the training process or after the model has been trained, involve removing unnecessary (e.g., because they have no effect on the output), less necessary (e.g., because their effect on the output is negligible), or potentially redundant features from the model. In some instances, pruning techniques can reduce the complexity of the model or improve its efficiency without compromising its expected performance.

[0130] Pruning techniques can be particularly useful in the context of wireless communication, where available resources, such as power and bandwidth, may be limited. Some example pruning techniques include weight pruning, neuron pruning, layer pruning, structural pruning, and dynamic pruning. Pruning techniques can, for example, reduce the amount of data corresponding to a model that may need to be transmitted or stored.

[0131] Weight pruning techniques may involve removing some weights from a model. Neuron pruning techniques may involve removing some neurons from a model. Layer pruning techniques may involve removing some layers from a model. Structural pruning techniques may involve removing some connections between neurons in a model. Dynamic pruning techniques may involve adapting a pruning strategy for a model to one or more characteristics associated with the data or environment. For example, in some wireless communication devices, dynamic pruning techniques may be more aggressive in pruning models used in low-power or low-bandwidth environments and less aggressive in pruning models used in high-power or high-bandwidth environments. In some aspects, pruning techniques may also be applied to training data, such as to remove outliers. In some specific implementations, preprocessing techniques on all or part of the training dataset can improve model performance or facilitate faster model convergence. For example, training data may be preprocessed to alter or remove unnecessary, irrelevant, incorrect, or otherwise identifiable data. Such preprocessing of training data can, for example, lead to a reduction in potential overfitting or otherwise improve the performance of the trained model.

[0132] One or more of the example training techniques presented above can be used as part of the training process. As mentioned above, some example training processes that can be used to train ML models include supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning techniques.

[0133] Decentralized, distributed, or shared learning, such as federated learning, enables training on data distributed across multiple devices or organizations without the need for centralized data or training. Federated learning can be particularly useful in data-sensitive or privacy-constrained situations, or when centralized data is impractical, inefficient, or expensive. For example, in the context of wireless communications, federated learning can be used to improve performance by allowing ML models to be trained on data collected from a wide range of devices and environments. For instance, ML models can be trained on data collected from a large number of wireless devices in a network, such as distributed wireless communication nodes, smartphones, or Internet of Things (IoT) devices, to improve network performance and efficiency. With federated learning, a user equipment (UE) or other device can receive a full or partial copy of the model and perform local training on such a copy of the model using locally available training data. Such devices can provide updated information about the locally trained model (e.g., trainable parameter gradients) to one or more other devices, such as network entities or servers, where updates from other similar devices (such as other UEs) can be aggregated and used to provide updates to shared models, etc. The federated learning process can be iteratively repeated until all or part of the model achieves a satisfactory level of performance. Federated learning enables devices to protect the privacy and security of local data while supporting collaboration on training and updating shared models, in whole or in part.

[0134] In some implementations, one or more devices or services may support processes related to the use, maintenance, activation, or reporting of ML models. In some instances, all or part of a dataset or model may be shared across multiple devices, for example, to provide or otherwise enhance or improve processing. In some examples, signaling mechanisms may be used at various nodes in a wireless network to signal capabilities for performing specific functions related to the ML model, support for a particular ML model, capabilities for collecting, creating, and transmitting training data, or other ML-related capabilities. ML models in wireless communication systems may, for example, support decisions related to wireless resource allocation or selection, wireless channel condition estimation, interference mitigation, beam management, positioning accuracy, energy saving, or modulation or decoding schemes. In some implementations, model deployment may be conducted jointly or individually at various network levels, such as central units (CUs), distributed units (DUs), radio units (RUs), etc.

[0135] Example methods for performing 3D scene reconstruction

[0136] Figure 8 A method 800 for performing graphic texture reconstruction is shown. In one aspect, method 800, or any aspect thereof, may be provided by means of a device (such as...) Figure 9The processing system 900 is used to execute the method, which includes various components that are operable to, configured to, or adapted to execute the method 800.

[0137] Method 800 begins at 802: receiving a plurality of feature sets corresponding to a texture. In an implementation, the plurality of feature sets include a corresponding feature set for each corresponding grid point of a mesh, wherein each corresponding grid point of the mesh is associated with a corresponding portion of the texture, and wherein the mesh has a first resolution.

[0138] Then, method 800 can proceed to 804: receiving coordinate information corresponding to the texels of the texture.

[0139] Then, method 800 may proceed to 806: receiving detail level information indicating a second resolution for reconstructing the texture, wherein the second resolution is lower than the first resolution.

[0140] Then, method 800 can proceed to 808: selecting a subset of grid points based on the second resolution being lower than the first resolution.

[0141] Then, the method can continue to 810: sampling one or more grid points from a subset of grid points based on coordinate information to obtain sampled features associated with one or more grid points.

[0142] Then, the method can continue to step 812: inputting the sampled features into the machine learning model.

[0143] Then, method 800 can end at 814: the reconstruction of the texture at a second resolution is received from the machine learning model based on the sampled features.

[0144] In some implementations, method 800 further includes: using a loss function to train an encoder and a machine learning model to adjust the weights of the encoder and the machine learning model; inputting textures into the encoder; and receiving multiple feature sets from the encoder as outputs.

[0145] In some implementations of method 800, the encoder includes a convolutional layer.

[0146] In some implementations of method 800, training the encoder and the machine learning model includes: generating a first candidate set of multiple features by the encoder; reconstructing one or more texels at one or more resolutions by the machine learning model based on the first candidate set of multiple features; and adjusting the weights of the encoder and the machine learning model based on a loss function.

[0147] In some implementations of method 800, multiple feature sets are quantized into discrete levels.

[0148] In some implementations of method 800, sampling one or more grid points includes performing one or more of four nearest neighbor sampling or bilinear sampling.

[0149] In some implementations of method 800, sampling one or more grid points includes nearest neighbor interpolation of four nearest neighbor sampling.

[0150] In some implementations, method 800 further includes: receiving a second plurality of feature sets corresponding to a texture, wherein the second plurality of feature sets include a corresponding feature set for each corresponding grid point of a second grid, wherein each corresponding grid point of the second grid is associated with a corresponding portion of the texture, wherein the second grid has a first resolution; sampling the second grid at one or more second grid points to obtain a second feature associated with one or more second grid points; and inputting the second feature into a machine learning model, wherein the reconstruction of the texels of the texture received from the machine learning model is also based on the second feature.

[0151] In some implementations of method 800, each of the multiple feature sets includes a multi-channel feature vector.

[0152] In some implementations, method 800 further includes: selecting a stride level based on the ratio between a first resolution and a second resolution, wherein selecting a subset of grid points based on the second resolution being lower than the first resolution includes selecting a subset of grid points based on the stride level.

[0153] In some implementations of method 800, the machine learning model includes a multilayer perceptron architecture with skip connections.

[0154] In some implementations of method 800, the level of detail information indicates the mipmap level of the texture.

[0155] In some embodiments of method 800, the reconstruction of texels includes texture properties corresponding to material properties.

[0156] In some implementations of method 800, coordinate information is encoded into a positioning encoding vector based on the values ​​of a pair of coordinate variables.

[0157] In some implementations, method 800 also includes a modem coupled to one or more antennas and to one or more processors, wherein the modem and the one or more antennas are configured to receive textures.

[0158] In some embodiments of method 800, the modem and one or more antennas are integrated into one of a vehicle, a surreal device, or a mobile device.

[0159] In some implementations, method 800 further includes: inputting a texture into an encoder; and receiving from the encoder a plurality of feature sets corresponding to the texture as output.

[0160] It should be noted that Figure 8 This is merely one example of a method, and other methods that include fewer, additional, or alternative steps may also be consistent with this disclosure.

[0161] Example processing system for performing 3D scene reconstruction

[0162] Figure 9 Various aspects of the example processing system 900 are described.

[0163] Processing system 900 includes processing system 902, which includes one or more processors 920. The one or more processors 920 are coupled to computer-readable medium / memory 930 via bus 906. In some aspects, the computer-readable medium / memory 930 is configured to store instructions (e.g., computer-executable code) that, when executed by the one or more processors 920, cause the one or more processors 920 to perform relative to… Figure 8 The described method 800 or any aspect thereof, including relative to Figure 8 Any additional steps or sub-steps described.

[0164] In the depicted example, computer-readable medium / memory 930 stores: code 931 (e.g., executable instructions) for receiving a plurality of feature sets corresponding to a texture, wherein the plurality of feature sets include a corresponding feature set for each corresponding grid point of a mesh, wherein each corresponding grid point of the mesh is associated with a corresponding portion of the texture, wherein the mesh has a first resolution; code 932 for receiving coordinate information corresponding to texels of the texture; code 933 for receiving detail level information indicating a second resolution for reconstructing the texture, wherein the second resolution is lower than the first resolution; code 934 for selecting a subset of grid points of the mesh based on the second resolution being lower than the first resolution; code 935 for sampling one or more grid points from the subset of grid points based on the coordinate information to obtain sampled features associated with one or more grid points; code 936 for inputting the sampled features into a machine learning model; and code 937 for receiving, based on the sampled features, a reconstruction of the texels of the texture at the second resolution from the machine learning model. Processing of codes 931 to 937 enables processing system 900 to execute and perform relative to Figure 8 The method described 800 or any aspect thereof.

[0165] One or more processors 920 include circuitry configured to implement (e.g., execute) code stored in a computer-readable medium / memory 930, the circuitry including: circuitry 921 for receiving a plurality of feature sets corresponding to a texture, wherein the plurality of feature sets include a corresponding feature set for each corresponding grid point of a mesh, wherein each corresponding grid point of the mesh is associated with a corresponding portion of the texture, wherein the mesh has a first resolution; circuitry 922 for receiving coordinate information corresponding to texels of the texture; circuitry 923 for receiving detail level information indicating a second resolution for reconstructing the texture, wherein the second resolution is lower than the first resolution; and circuitry 924 for selecting a subset of grid points of the mesh based on the second resolution being lower than the first resolution; circuitry 925 for sampling one or more grid points from the subset of grid points based on the coordinate information to obtain sampled features associated with one or more grid points; circuitry 926 for inputting the sampled features into a machine learning model; and circuitry 927 for receiving a reconstruction of the texture's texels at the second resolution from the machine learning model based on the sampled features. Processing using circuitry 921 to 927 enables processing system 900 to execute and perform relative to... Figure 8 The method described 800 or any aspect thereof.

[0166] Example Terms

[0167] Specific implementation examples are described in the following numbered clauses: Clause 1: A method for reconstructing texels of a texture, the method comprising: receiving a plurality of feature sets corresponding to the texture, wherein the plurality of feature sets include a corresponding feature set for each corresponding grid point of a mesh, wherein each corresponding grid point of the mesh is associated with a corresponding portion of the texture, wherein the mesh has a first resolution; receiving coordinate information corresponding to the texels of the texture; receiving level-of-detail information indicating a second resolution for reconstructing the texture, wherein the second resolution is lower than the first resolution; selecting a subset of grid points of the mesh based on the second resolution being lower than the first resolution; sampling one or more grid points from the subset of grid points based on the coordinate information to obtain sampled features associated with the one or more grid points; inputting the sampled features into a machine learning model; and receiving, based on the sampled features, a reconstruction of the texels of the texture at the second resolution from the machine learning model.

[0168] Clause 2: The method according to Clause 1 further includes: using a loss function to train an encoder and the machine learning model to adjust the weights of the encoder and the machine learning model; inputting the texture into the encoder; and receiving the plurality of feature sets from the encoder as output.

[0169] Clause 3: The method according to Clause 2, wherein the loss function is based on the difference between the output of the machine learning model and the input to the encoder.

[0170] Clause 4: The encoder comprising a convolutional layer according to the method of Clause 2.

[0171] Clause 5: The method according to Clause 2, wherein training the encoder and the machine learning model comprises: generating a first candidate plurality of feature sets by the encoder; reconstructing one or more texels at one or more resolutions by the machine learning model based on the first candidate plurality of feature sets; and adjusting the weights of the encoder and the machine learning model based on the loss function.

[0172] Clause 6: The method according to any one of Clauses 1 to 5, wherein the plurality of feature sets are quantized into discrete levels.

[0173] Clause 7: The method according to any one of Clauses 1 to 6, wherein sampling of the one or more grid points includes performing one or more of four nearest neighbor sampling or bilinear sampling.

[0174] Clause 8: The method according to Clause 7, wherein sampling of the one or more grid points includes performing nearest neighbor interpolation of the four nearest neighbor sampling.

[0175] Clause 9: The method according to any one of Clauses 1 to 8, the method further comprising: receiving a second plurality of feature sets corresponding to the texture, wherein the second plurality of feature sets includes a corresponding feature set for each corresponding grid point of a second grid, wherein each corresponding grid point of the second grid is associated with a corresponding portion of the texture, wherein the second grid has the first resolution; sampling the second grid at one or more second grid points to obtain a second feature associated with the one or more second grid points; and inputting the second feature into the machine learning model, wherein the reconstruction of the texels of the texture received from the machine learning model is also based on the second feature.

[0176] Clause 10: The method according to any one of Clauses 1 to 9, wherein each of the plurality of feature sets comprises a multi-channel feature vector.

[0177] Clause 11: The method according to any one of Clauses 1 to 10, the method further comprising: selecting a stride level based on the ratio between the first resolution and the second resolution, wherein selecting the subset of grid points of the grid based on the second resolution being lower than the first resolution includes selecting the subset of grid points of the grid based on the stride level.

[0178] Clause 12: The method according to any one of Clauses 1 to 11, wherein the machine learning model comprises a multilayer perceptron architecture with skip connections.

[0179] Clause 13: The method according to any one of Clauses 1 to 12, wherein the level of detail information indicates the mipmap level of the texture.

[0180] Clause 14: The method according to any one of Clauses 1 to 13, wherein the reconstruction of the texel includes texture properties corresponding to the material properties.

[0181] Clause 15: The method according to any one of Clauses 1 to 14, wherein the coordinate information is encoded as a positioning encoding vector based on the values ​​of a pair of coordinate variables.

[0182] Clause 16: The method according to any one of Clauses 1 to 15, wherein the method is performed by an apparatus including a modem coupled to one or more antennas and coupled to one or more processors, wherein the modem and the one or more antennas are configured to receive the texture.

[0183] Clause 17: The method described in Clause 16, wherein the modem and the one or more antennas are integrated into one of a vehicle, a surreal device, or a mobile device.

[0184] Clause 18: The method according to any one of Clauses 1 to 17, the method further comprising: inputting the texture into an encoder; and receiving from the encoder the plurality of feature sets corresponding to the texture as output.

[0185] Clause 19: The method according to Clause 1, wherein an encoder and a loss function are used to train the machine learning model to adjust the weights of the encoder and the machine learning model.

[0186] Clause 20: The method according to Clause 19, wherein the loss function is based on the difference between the output of the machine learning model and the input to the encoder.

[0187] Clause 21: The method according to Clause 19, wherein the encoder comprises a convolutional layer.

[0188] Clause 22: The method according to Clause 19, wherein the encoder and the machine learning model are trained using a first candidate plurality of feature sets, wherein one or more texels are reconstructed at one or more resolutions based on the first candidate plurality of feature sets, and wherein the weights of the encoder and the machine learning model are adjusted based on the loss function.

[0189] Clause 23: The method according to Clause 1, wherein the machine learning model is trained using an encoder that performs multiple iterations across multiple resolutions.

[0190] Clause 24: The method according to Clause 1, wherein an autoencoder is used to train the machine learning model, the autoencoder comprising an encoder model and a decoder model, the autoencoder being trained by: compressing an input texture into a latent representation using the encoder model, the latent representation comprising a plurality of feature sets corresponding to grid points; reconstructing texels of the input texture from the latent representation using the decoder model; comparing the reconstructed texels of the input texture with the original texels using a loss function; and updating the weights of the encoder model and the decoder model based on the gradient of the loss function.

[0191] Clause 25: The method according to Clause 1, further comprising: receiving a second plurality of feature sets corresponding to the texture, wherein the second plurality of feature sets includes a corresponding feature set for each corresponding grid point of a second grid, wherein each corresponding grid point of the second grid is associated with a corresponding portion of the texture; receiving level-of-detail information indicating a second resolution for reconstructing the texels; determining whether the second resolution is lower than, equal to, or greater than a first resolution; in response to determining that the second resolution is lower than the first resolution, selecting a subset of grid points of the grid; sampling one or more grid points in the subset of grid points based on the coordinate information to obtain features associated with the one or more grid points, wherein if the second resolution is lower than the first resolution, the one or more grid points are selected from the subset of grid points; sampling the second grid at the one or more second grid points to obtain a second feature associated with the one or more second grid points; and inputting the second feature into the machine learning model, wherein the reconstruction of the texels of the texture received from the machine learning model is also based on the second feature.

[0192] Clause 26: The method according to Clause 1, the method further comprising: receiving a second plurality of feature sets corresponding to the texture, wherein the second plurality of feature sets includes a corresponding feature set for each corresponding grid point of a second grid, wherein each corresponding grid point of the second grid is associated with a corresponding portion of the texture, wherein the second grid has a third resolution; sampling the second grid at one or more second grid points to obtain a second feature associated with the one or more second grid points; and inputting the second feature into the machine learning model, wherein the reconstruction of the texels of the texture received from the machine learning model is also based on the second feature.

[0193] Clause 27: The method described in Clause 26, wherein the third resolution is the same as the first resolution.

[0194] Clause 28: The method described in Clause 26, wherein the third resolution is different from the first resolution.

[0195] Clause 29: One or more means comprising: one or more memories including executable instructions; and one or more processors configured to execute the executable instructions and cause the one or more means to perform the method according to any one of Clauses 1 to 28.

[0196] Clause 30: One or more means comprising: one or more memories; and one or more processors coupled to the one or more memories, the one or more processors being configured to cause the one or more means to perform the method according to any one of Clauses 1 to 28.

[0197] Clause 31: One or more means comprising: one or more memories; and one or more processors coupled to the one or more memories, the one or more processors being configured to perform the method according to any one of Clauses 1 to 28.

[0198] Clause 32: One or more apparatuses, said apparatuses comprising components for performing the method according to any one of Clauses 1 to 28.

[0199] Clause 33: One or more non-transitory computer-readable media, the one or more non-transitory computer-readable media comprising executable instructions that, when executed by one or more processors of one or more devices, cause the one or more devices to perform the method according to any one of Clauses 1 to 28.

[0200] Clause 34: One or more computer program products embodied on one or more computer-readable storage media, the one or more computer-readable storage media including code for performing the method according to any one of Clauses 1 to 28.

[0201] Additional Notes

[0202] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein do not limit the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, the function and arrangement of the elements discussed may be changed without departing from the scope of this disclosure. Various processes or components may be omitted, substituted, or added as appropriate in various examples. For example, the described methods may be performed in a different order than described, and various actions may be added, omitted, or combined. Furthermore, features described in some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement an apparatus or practice. Moreover, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or replace the various aspects of this disclosure set forth herein. It should be understood that any aspect of the disclosure herein may be embodied by one or more elements of the claims.

[0203] The various exemplary logic blocks, modules, and circuits described in this disclosure can be implemented or executed using a general-purpose processor, digital signal processor (DSP), ASIC, field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic device, discrete hardware component, or any combination thereof designed to perform the functions described herein. While the general-purpose processor may be a microprocessor, in alternative embodiments, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors working in conjunction with a DSP core, a system-on-a-chip (SoC), or any other such configuration.

[0204] As used in this article, the phrase “at least one of” in a list of items refers to any combination of those items, including a single member. For example, “at least one of a, b, or c” is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0205] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, operation, processing, deduction, investigation, lookup (e.g., searching in a table, database, or other data structure), assertion, etc. Additionally, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "determine" can include parsing, selecting, picking, building, etc.

[0206] As used herein, unless otherwise stated, “coupled to” and “coupled with” generally encompass both direct and indirect coupling (e.g., including intermediate aspects of coupling). For example, stating that a processor is coupled to memory allows for direct coupling or coupling via an intermediate aspect such as a bus.

[0207] The methods disclosed herein include one or more actions for implementing the methods. These method actions may be interchanged without departing from the scope of the claims. In other words, unless a specified order of actions is given, the order and / or use of a particular action may be modified without departing from the scope of the claims. Furthermore, the various operations of the methods described above may be performed by any suitable component capable of performing the corresponding function. This component may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors.

[0208] The following claims are not intended to be limited to the aspects shown herein, but should be given the full scope consistent with the language of the claims. References to singular elements are not intended to mean “only one” (unless specifically stated as “only one”), but rather “one or more”. Unless otherwise specified, definite articles (e.g., “the” or “described”) subsequently used with an element (e.g., “processor”) are not intended to give that element a singular meaning (e.g., “only one”). For example, unless otherwise specified, references to elements (e.g., “processor”, “controller”, “memory”, “transceiver”, “antenna”, “the processor”, “the controller”, “the memory”, “the transceiver”, “the antenna”, etc.) should be understood to refer to one or more elements (e.g., “one or more processors”, “one or more controllers”, “one or more memories”, “a plurality of transceivers”, etc.). The terms “set” and “group” are intended to include one or more elements and may be used interchangeably with “one or more”. In the case of references to one or more elements performing a function (e.g., steps of a method), one element may perform all the functions, or more than one element may collectively perform those functions. When more than one element performs these functions together, each function does not need to be performed by every single element (e.g., different functions can be performed by different elements), and / or each function does not need to be performed by only one element as a whole (e.g., different elements can perform different sub-functions of a function). Similarly, when referring to one or more elements configured to cause another element (e.g., a device) to perform a function, one element may be configured to cause another element to perform all functions, or more than one element may be jointly configured to cause another element to perform these functions. Unless otherwise specifically stated, the term "some" means one or more. All structural and functional equivalents of the elements of the various aspects described throughout this disclosure that are currently or hereafter known to those skilled in the art are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is explicitly recited in the claims.

Claims

1. An apparatus, the apparatus comprising: One or more memory units; and One or more processors, said one or more processors coupled to said one or more memories, said one or more processors being configured to: Receive a plurality of feature sets corresponding to a texture, wherein the plurality of feature sets include a corresponding feature set for each corresponding grid point of a mesh, wherein each corresponding grid point of the mesh is associated with a corresponding portion of the texture, wherein the mesh has a first resolution; Receive coordinate information corresponding to the texels of the texture; Receive detail level information indicating a second resolution for reconstructing the texture, wherein the second resolution is lower than the first resolution; A subset of grid points is selected based on the fact that the second resolution is lower than the first resolution; Based on the coordinate information, sample one or more grid points from the subset of grid points to obtain sampled features associated with the one or more grid points; The sampled features are then input into the machine learning model; as well as Based on the sampled features, the machine learning model receives a reconstruction of the texels of the texture at the second resolution.

2. The apparatus of claim 1, wherein the one or more processors are configured to: A loss function is used to train the encoder and the machine learning model to adjust the weights of the encoder and the machine learning model; The texture is input into the encoder; as well as The encoder receives the plurality of feature sets as output.

3. The apparatus of claim 2, wherein the loss function is based on the difference between the output of the machine learning model and the input to the encoder.

4. The apparatus of claim 2, wherein the encoder comprises a convolutional layer.

5. The apparatus of claim 2, wherein training the encoder and the machine learning model comprises: The encoder generates a first set of multiple candidate features; The machine learning model reconstructs one or more texels at one or more resolutions based on the first candidate feature set; and The weights of the encoder and the machine learning model are adjusted based on the loss function.

6. The apparatus of claim 1, wherein the plurality of feature sets are quantized to discrete levels.

7. The apparatus of claim 1, wherein sampling the one or more grid points comprises performing one or more of four nearest neighbor sampling or bilinear sampling.

8. The apparatus of claim 7, wherein sampling the one or more grid points includes performing nearest neighbor interpolation of the four nearest neighbor sampling.

9. The apparatus of claim 1, wherein the one or more processors are configured to: Receive a second plurality of feature sets corresponding to the texture, wherein the second plurality of feature sets include a corresponding feature set for each corresponding grid point of a second grid, wherein each corresponding grid point of the second grid is associated with a corresponding portion of the texture, wherein the second grid has the first resolution; Sampling the second grid at one or more second grid points to obtain a second feature associated with the one or more second grid points; and The second feature is input into the machine learning model, wherein the reconstruction of the texels of the texture received from the machine learning model is also based on the second feature.

10. The apparatus of claim 1, wherein each of the plurality of feature sets comprises a multi-channel feature vector.

11. The apparatus of claim 1, wherein the one or more processors are configured to: The step level is selected based on the ratio between the first resolution and the second resolution, wherein selecting the subset of grid points of the grid based on the second resolution being lower than the first resolution includes selecting the subset of grid points of the grid based on the step level.

12. The apparatus of claim 1, wherein the machine learning model comprises a multilayer perceptron architecture with skip connections.

13. The apparatus of claim 1, wherein the level of detail information indicates the mipmap level of the texture.

14. The apparatus of claim 1, wherein the reconstruction of the texels includes texture properties corresponding to material properties.

15. The apparatus of claim 1, wherein the coordinate information is encoded as a positioning encoding vector based on the values ​​of a pair of coordinate variables.

16. The apparatus of claim 1, further comprising a modem coupled to one or more antennas and coupled to the one or more processors, wherein the modem and the one or more antennas are configured to receive the texture.

17. The apparatus of claim 16, wherein the modem and the one or more antennas are integrated into one of a vehicle, a surreal device, or a mobile device.

18. The apparatus of claim 1, wherein the one or more processors are configured to: The texture is input into the encoder; and The encoder receives the plurality of feature sets corresponding to the texture as output.

19. A method for reconstructing texels for texture, the method comprising: Receive a plurality of feature sets corresponding to the texture, wherein the plurality of feature sets include a corresponding feature set for each corresponding grid point of the mesh, wherein each corresponding grid point of the mesh is associated with a corresponding portion of the texture, wherein the mesh has a first resolution; Receive coordinate information corresponding to the texels of the texture; Receive detail level information indicating a second resolution for reconstructing the texture, wherein the second resolution is lower than the first resolution; A subset of grid points is selected based on the fact that the second resolution is lower than the first resolution; Based on the coordinate information, sample one or more grid points from the subset of grid points to obtain sampled features associated with the one or more grid points; The sampled features are then input into the machine learning model; as well as Based on the sampled features, the machine learning model receives a reconstruction of the texels of the texture at the second resolution.

20. The method according to claim 19, further comprising: A loss function is used to train the encoder and the machine learning model to adjust the weights of the encoder and the machine learning model; The texture is input into the encoder; as well as The encoder receives the plurality of feature sets as output.

Images