System and procedure for reducing and reconstructing the size of a signal for storage and transmission

By using a feature extractor to identify common features and a signal reconstruction neural network for multimedia data, the system addresses the inefficiencies of current neural field representations, achieving a compact and scalable solution for storing and transmitting multimedia content.

WO2024246374A1PCT designated stage expired Publication Date: 2024-12-05VOLUMETRIC TECHNOLOGIES SL
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
PCT/ES2023/070356
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-30
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Current neural field representations are not memory-efficient and fail to scale well with increasing content, leading to degraded quality and increased parameters, making them inefficient for storing and transmitting large multimedia data like images, videos, and 3D scenes.

Method used

The system employs a feature extractor to identify common features across different signals, using pre-trained neural networks, implicit coordinate encoders, or direct gradient descent, and a common feature space to reduce signal size, followed by a signal reconstruction neural network that uses shared features to reconstruct the signal efficiently.

Benefits of technology

This approach significantly reduces the number of parameters required, enabling a compact representation that improves scalability, quality, and reduces memory usage, facilitating efficient storage and transmission of multimedia content, including volumetric video and virtual production scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 000016
    Figure 000016
  • Figure 000017
    Figure 000017
  • Figure 00000015_0000
    Figure 00000015_0000
Patent Text Reader

Abstract

System that allows different types of information (images, videos, 3D scenes, etc.) to be represented efficiently, that is, where the memory necessary to store this information is low and the recovery of this information is fast, wherein the system comprises processing means comprising: - a feature extractor: which is responsible for extracting the features common to all the signals; - a space of common features: the data structure where the feature vectors are stored, which can be a continuous grid or discrete grid space; - and a signal reconstruction neural network, which may be carried out by: specific neural networks for each signal, or a single common network for all signals having embedded means specific to each type of signal.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] SYSTEM AND PROCEDURE FOR REDUCING AND RECONSTRUCTING THE SIZE OF A SIGNAL FOR ITS STORAGE AND TRANSMISSION

[0002] DESCRIPTION

[0003] OBJECT OF THE INVENTION

[0004] The object of the present invention, as the title of the invention establishes, is, on the one hand, the system necessary for the reduction and reconstruction of the size of a signal, as well as the procedure carried out that allows the reduction of the size of a signal for its storage and transmission, as well as its subsequent reconstruction.

[0005] This invention allows different types of information (images, videos, 3D scenes, etc.) to be represented efficiently, that is, the memory required to store this information is low and the retrieval of this information is fast.

[0006] The present invention is characterized by the special design and configuration of each and every one of the means, as well as the stages employed in the development of the invention, which is achieved using neural networks that learn the relationship between their value and the spatial and temporal coordinates. To improve the quality, learning, and speed of signal reconstruction, feature vectors are often used as inputs along with the signal coordinates.

[0007] This invention leverages a common feature space between different signals; that is, different neural networks use the same features to represent different signals, drastically reducing the total number of features needed to represent them all. This can also be applied to representing large-scale signals, as a large signal can be broken down into smaller ones.

[0008] Therefore, the present invention is limited to the scope of signal processing, preferably images, videos, 3D representations, in order to reduce their size to facilitate their storage, transmission and subsequent recovery.

[0009] BACKGROUND OF THE INVENTION

[0010] A field is a quantity defined for all spatial and / or temporal coordinates. Examples of fields include gravity, which describes how the force of attraction of an object varies in space, and the electromagnetic fields used for telecommunications, which vary not only with distance but also with time. However, not all fields are physical. For example, images are essentially an RGB color that changes at each pixel, while videos extend this color field into the temporal dimension. The 3D scenes that will power the future metaverse can be represented by a combination of density and RGB color. The key difference between physical fields and these types of fields is that the latter are not governed by a universal physical law, meaning they cannot be described by simple mathematical equations.As a result, each image, video, or 3D scene is represented by highly complex equations, making it difficult to find an adequate representation.

[0011] To address this challenge, scientists and engineers have developed automatic approximation methods, such as wavelets for images or video encodings and meshes and textures for 3D scenes. Despite being well-established, these methods have several drawbacks. First, each approximation method must be designed independently for each field type, resulting in a lack of generality. Furthermore, the resulting representations often require a large amount of digital memory, as they are stored as digital files (such as PNG, JPG, MP4, MOV, PLY, OBJ).

[0012] A promising approach in recent years has been neural fields, which use a neural network to approximate the underlying function of each field. This technique is versatile and can be applied to various types of fields, producing higher-quality results than traditional methods. However, it also has a major drawback: it is not memory-efficient. We define an efficient representation as one that requires a small amount of digital memory and can be retrieved quickly. For example, PNG files can be displayed quickly, but their memory usage is high, whereas using a zip file to compress a group of images reduces memory usage, but retrieval is slow.Similarly, neural fields require a balance between retrieval speed and memory usage, and are unable to perform well in both areas, resulting in inefficient representations of information.

[0013] This invention addresses this problem by proposing an efficient representation for complex fields through the use of shared features.

[0014] Current methods apply two main strategies to reduce the number of parameters of a signal, i.e., the size, if models need to represent a given signal: the first one is based on an architecture search problem, where the most suitable architecture is searched for each of the given signals. Signals, as shown in the publications [1], [2], [3]; on the other hand, other works have addressed this problem by applying quantization techniques on a set of feature vectors, as described in the publications [4], [5]. However, all these methods treat each of the signals as completely unrelated. The currently employed neural fields do not scale correctly as the amount of content to be represented scales: the quality degrades and the necessary number of parameters increases without a proportional increase in the representation capacity.

[0015] Therefore, the purpose of this application is to overcome the drawbacks of the state of the art by developing a system and method that significantly improves the scalability of neural fields by representing a large amount of content, improving its quality and reducing the number of necessary parameters (creating a much more compact representation).

[0016] DESCRIPTION OF THE INVENTION

[0017] The object of the present invention is set out in its essence in the independent claim and the different embodiments are set out in the dependent claims.

[0018] The present invention relates to a method for reducing and reconstructing the size of a signal for storage and transmission.

[0019] The necessary elements are the following:

[0020] - Feature extractor: which is responsible for extracting the characteristics common to all signals, which can be carried out using any of the following three features: o Using a pre-trained neural network o Using an implicit coordinate encoder o Using means that carry out a direct gradient descent process on the characteristics of the signals

[0021] - A common feature space: It is the data structure where the feature vectors are stored, which in one possible embodiment is a memory, and can be a continuous grid space or a discrete grid space.

[0022] - A signal reconstruction neural network, which can be carried out by: o Specific neural networks for each signal, o A single common network for all signals having embedded media specific to each type of signal.

[0023] Our method is the first to exploit common redundancies between different signals, reconstructed using neural fields, to reduce the representation size. Furthermore, by employing an implicit coordinate encoder as a common feature extractor, it is the first to use a neural field to map signal coordinates to a feature space that encodes these redundancies.

[0024] This invention can be applied to the storage and transmission of multimedia content, such as video or volumetric video. Our invention reduces the signal size, which relaxes the requirements for video transmission (bandwidth, heavy usage, etc.) and enables the transmission of volumetric video (which is currently impossible due to the size of these videos).

[0025] This invention can also be applied to virtual production. Our method allows for the capture of unlimited large scenes, enabling their use in virtual production scenarios. This application also extends to virtual or augmented reality (VR / AR).

[0026] Unless otherwise indicated, all technical and scientific elements used herein have the meaning commonly understood by one of ordinary skill in the art to which this invention pertains. In the practice of the present invention, methods and materials similar or equivalent to those described herein can be used. Throughout the description and claims, the word "comprises" and its variants are not intended to exclude other technical features, additives, components, or steps. For those skilled in the art, other objects, advantages, and features of the invention will be apparent partly from the description and partly from the practice of the invention.

[0027] EXPLANATION OF THE FIGURES

[0028] To complement the description being made and in order to help better understand the characteristics of the invention, in accordance with a preferred example of practical implementation thereof, a set of drawings is attached as an integral part of said description, in which the following has been represented for illustrative and non-limiting purposes.

[0029] In Figure 1, we can observe the flow diagram that shows the procedure followed for the reduction and reconstruction of a signal without optimizing the common feature extractor.

[0030] In Figure 2, we can see the flowchart showing the procedure followed for the reduction and reconstruction of a signal where the common feature extractor has been previously optimized.

[0031] PREFERRED EMBODIMENT OF THE INVENTION

[0032] A preferred embodiment of the proposed invention is described below in view of the figures.

[0033] In a preferred embodiment, the system for reducing and reconstructing the size of a signal comprises processing means comprising: - A feature extractor: which is responsible for extracting the characteristics common to all signals, which can be carried out in one of the following three ways: o By means of a pre-trained neural network: In this case, the different signals to be represented will be processed by this neural network that will generate the feature space. o By means of an implicit coordinate encoder, which is a neural network that parameterizes an additional neural field. In this case, the neural field relates the coordinates of the signal (which are shared in all signals) with the characteristics of the given coordinates. This neural network is trained together with the signal-specific networks to obtain the most appropriate characteristics for the given signals.In this case, the model has an autoencoder architecture, as shown in Figure 2. It is achieved by means of performing a direct gradient descent process on the signal features, where the features can be considered as directly optimizable vectors. In this case, the features are directly optimized using gradient descent, so this method is used as a feature extractor to obtain the features, applying it during the training of signal-specific neural networks. In this case, the model has an autodecoder architecture. It should be noted that this method and the implicit coordinate encoder are not exclusive, and both can be used at different training stages.

[0034] - A common feature space: This is the data structure where the feature vectors are stored, which in one possible embodiment is a memory. This common feature space can be: a. Continuous grid space: With this type of grid, we store a feature vector for each point of the signal to be evaluated. In this case, the resolution at which the signal will be reconstructed is fixed. For example, it can be applied to images that store a feature vector per pixel. b. Discrete grid space: In this case, the features are stored at the corners of a grid that divides the signal domain or at specific points. Therefore, to obtain the feature vector at a given point, the features closest to the point are linearly interpolated.

[0035] - A signal reconstruction neural network, which can be implemented using: o Signal-specific neural networks, which are the different neural networks that parameterize the neural field of each signal. They take common characteristics and, optionally, the signal coordinates as input and output the signal values, which in turn can be implemented using any of the following alternatives, but are not limited to:

[0036] ■ ReLU (Rectified Linear Unit)-activated multilayer perceptron: To obtain the signal value, a dense neural network or a multilayer perceptron that uses a rectified linear unit (ReLU) to activate its hidden layers can be used as a specific signal decoder.

[0037] ■ Sinusoidal representation networks (SIREN): The use of a multilayer perceptron with sines or cosines as the activation function for its hidden layers can be applied, obtaining better performance for certain types of signals [7], or A single common network for all signals with details for each signal embedded within the neural network. In this case, a single neural network will be used to reconstruct all these signals, which will use, in addition to the common feature space, an identifier with information for each signal. These identifiers can be vectors of additional independent features for each signal, but with a reduced dimension so that they do not increase the size of the representation.

[0038] In Figure 1 we can see the flow diagram of the stages followed for the reduction and reconstruction of the size of a signal, which comprises the following stages or phases:

[0039] - start of the procedure (1 )

[0040] - Application of a first neural network for extracting common features (2)

[0041] - Apply reconstruction neural networks (3) that can be specific neural networks for each signal and obtain the reconstructed value for each signal or apply a common neural network provided with specific means for each type of signal.

[0042] - Application of a mean square error calculation stage (4) for each signal.

[0043] - Application of a stochastic gradient descent stage (5), where an adjustment of all the parameters of the neural network is produced.

[0044] - Repeat the process until training is complete (6), i.e., a certain number of iterations is performed or a specific quality metric is achieved. If training is complete, the process ends (7).

[0045] Depending on the available hardware, this process can be performed in mini-batches, meaning each iteration is composed of a subset of random samples rather than the entire signals. It should be noted that, when using a pre-trained feature extractor, it is not necessary to perform feature extraction, so feature extraction can be left out of the loop. In this case, the process will follow the flowchart shown in Figure 2.

[0046] Having sufficiently described the nature of the present invention, as well as the manner of putting it into practice, it is noted that, within its essence, it may be put into practice in other embodiments that differ in detail from the one indicated as an example, and to which the protection sought will also be achieved, provided that it does not alter, change or modify its fundamental principle.

[0047] REFERENCES

[0048] [1] T. Davies, D. Nowrouzezahrai, and A. Jacobson, “ON THE EFFECTIVENESS OF WEIGHT-ENCODED NEURAL IMPLICIT 3D SHAPES,” Accessed: Sep. 16, 2022. [Online], Available: https: / / github.com / u2ni / ICLR2021.

[0049] [2] E. Dupont, A. Goliñski, M. Alizadeh, Y. W. Teh, and A. Doucet, “COIN:

[0050] Compression with Implicit Neural representations,” Neural Compression Work. @ ICLR 2021 , pp. 1-7, 2021 , [Online], Available: http: / / arxiv.org / abs / 2103.03123.

[0051] [3] Y. Zhang, T. Van Rozendaal, J. Brehmer, M. Nagel, T. S. Cohen Qualcomm, and A. I. Research, “Implicit Neural Video Compression.”

[0052] [4] T. Takikawa et al., “Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes.”

[0053] [5] T. Takikawa et al., “Variable Bitrate Neural Fields,” pp. 1-9, Aug. 2022, doi: 10.1145 / 3528233.3530727.

[0054] [6] Y. Xie et al., “Neural Fields in Visual Computing and Beyond,” CoRR, vol. abs / 2111 .1 , 2021 , [Online], Available: https: / / arxiv.org / abs / 2111.11426.

[0055] [7] V. Sitzmann, J. N. P. Martel, A. W. Bergman, D. B. Lindell, G. Wetzstein, and S. University, “Implicit Neural Representations with Periodic Activation Functions,” Adv. Neural Inf. Process. Syst. 33 (NeurlPS 2020), 2020.

Claims

CLAIMS 1.- System for the reduction and reconstruction of the size of a signal for its storage and transmission, characterized in that it comprises processing means provided with: - A feature extractor responsible for extracting the features common to all signals, which is done in one of the following three ways: o Using a pre-trained neural network: in this case, the different signals to be represented will be processed by this neural network that will generate the feature space. o Using an implicit coordinate encoder, which is a neural network that parameterizes an additional neural field.In this case, the neural field relates the signal coordinates (which are shared across all signals) to the features of the given coordinates, or By means that perform a direct gradient descent process on the signals features, where features can be considered as directly optimizable vectors, the features are directly optimized by gradient descent, so this method is used as a feature extractor to obtain the features by applying it during the training of signal-specific neural networks, having an autodecoder architecture. - A common feature space, which is the data structure where the feature vectors are stored, this common feature space is one of the following alternatives: c. Continuous grid space: With this type of grid we store a feature vector for each point of the signal to be evaluated. d. Discrete grid space: In this case, features are stored at the corners of a grid that divides the signal domain or at given points, so to obtain the feature vector at a given point the features closest to the point are linearly interpolated. - A signal reconstruction neural network, which is implemented using one of the following alternatives: o Signal-specific neural networks, which are the different neural networks that parameterize the neural field of each signal. They take common characteristics and, optionally, signal coordinates as input and output the signal values; or a single common network for all signals, with details for each signal embedded within the neural network. 2.- System for the reduction and reconstruction of the size of a signal for its storage and transmission according to claim 1, characterized in that the specific neural networks for each value can be through some of the following alternatives: ■ ReLU (Rectified Linear Unit)-activated multilayer perceptron: To obtain the signal value, a dense neural network or a multilayer perceptron that uses a rectified linear unit (ReLU) to activate its hidden layers can be used as a specific signal decoder. ■ Sinusoidal representation networks (SIREN): The use of a multilayer perceptron with sines or cosines as the activation function of its hidden layers can be applied, obtaining better performance for certain types of signals [7], 3.- Procedure implemented in a processing system for the reduction and reconstruction of the size of a signal for its storage and transmission carried out by means of the system claimed according to any of the preceding claims, characterized in that it comprises the steps of: - start of the procedure (1 ) - Application of a first neural network for extracting common features (2) - Apply reconstruction neural networks (3) that can be specific neural networks for each signal and obtain the reconstructed value for each signal or apply a common neural network provided with specific means for each type of signal. - Application of a mean square error calculation stage (4) for each signal. - Application of a stochastic gradient descent stage (5), where an adjustment of all the parameters of the neural network is produced. - Repeat the process until training is complete (6), i.e., a certain number of iterations is performed or a specific quality metric is achieved. If training is complete, the process ends (7). 4.- Procedure implemented in a processing system for the reduction and reconstruction of the size of a signal for its storage and transmission according to claim 3, characterized in that the feature extractor in charge of extracting the characteristics common to all the signals is carried out in one of the three following ways: o By means of a pre-trained neural network: in this case, the different signals to be represented will be processed by this neural network that will generate the characteristic space. o By means of an implicit coordinate encoder, which is a neural network that parameters an additional neural field, in this case, the neural field relates the coordinates of the signal (which are shared in all the signals) with the characteristics of the given coordinates, o By means that carry out a direct gradient descent process on the characteristics of the signals, where the characteristics can be considered as directly optimizable vectors, the characteristics are directly optimized by gradient descent, so this method is used as a feature extractor to obtain the characteristics by applying it during the training of the signal-specific neural networks, having an autodecoder architecture. 5.- Procedure implemented in a processing system for the reduction and reconstruction of the size of a signal for its storage and transmission according to claim 3, characterized in that in the case of using a pre-trained feature extractor, it is not necessary to apply the feature extraction, so the feature extraction is left out of the cycle. 6.- Procedure implemented in a processing system for the reduction and reconstruction of the size of a signal for its storage and transmission according to any of the previous claims, characterized in that in the case of using a single common network for all the signals having details for each signal embedded within the neural network, it will use, in addition to the common feature space, an identifier with information for each signal where these identifiers are vectors of additional independent features for each signal, but of a reduced dimension so that it does not entail an increase in the size of the representation.

Citation Information

Patent Citations

  • Image processing method and device and computer readable storage medium

    CN114445633A

  • Image processing method and apparatus, and computer-readable storage medium

    US20230326173A1