Data compression and reconstruction using sparse meta-learning neural networks

Abstract

A method, system, and apparatus including a computer program encoded on a computer storage medium for compressing and restoring a data signal using a sparse meta-learning neural network.

Description

[Technical Field]

[0001] Cross-Reference to Related Applications This application claims priority to U.S. Provisional Application No. 63 / 338,018, filed on May 3, 2022. The disclosure of the prior application is considered part of the disclosure of the present application and is incorporated herein by reference.

[0002] This specification relates to compressing and reconstructing input signals using a machine learning model. [Background Art]

[0003] As an example, a neural network is a machine learning model that uses one or more layers of non-linear units to predict an output from a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to another layer in the network, for example, a subsequent hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of weights. [Summary of Invention] [Means for Solving the Problems]

[0004] This specification describes a system, implemented as a computer program on one or more computers, that compresses an input signal using a data reconstruction neural network having network parameters.

[0005] In particular, the system uses the input signal to determine an update to shared values of network parameters for a subset of the network parameters.

[0006] The system then generates a compressed representation of the input signal that identifies the respective updates for the subset of network parameters.

[0007] The system or another system can use a data reconstruction neural network to reconstruct the input signal by determining updated values ​​for network parameters using shared values ​​and compressed representations, and then using the updated values ​​to reconstruct the input signal.

[0008] In one embodiment, the method includes the steps of: maintaining data specifying shared values ​​for the network parameters of a data reconstruction neural network, wherein the data reconstruction neural network is configured to receive an input specifying coordinates from the coordinate space of an input data signal, process the input according to the network parameters, and generate one or more predicted values ​​of the input data signal at the specified coordinates as an output; receiving a new input signal containing one or more new values ​​at each of a plurality of new coordinates; determining an update for each subset of the network parameters, determining one or more sets of current values ​​for the network parameters of the data reconstruction neural network in each of one or more internal iterations, determining a gate value for each network parameter in the subset, specifying for each set of current values ​​whether the update for each subset is set to zero according to a set of distribution parameters; and determining a set of current values, which is arbitrary if not in the subset. A step to determine a set of current values ​​for a network parameter, including setting the current value based on the shared value of the network parameter, setting the current value based on the shared value of the network parameter for any network parameter in a subset where each gate value specifies that the update for each subset of the subset is set to zero, and setting the current value based on the shared value of the network parameter and the update for each network parameter for any network parameter in a subset where each gate value specifies that the update for each subset of the subset is not set to zero, and a step to process an input specifying a new coordinate according to the set of current values ​​of the network parameter, using a data reconstruction neural network to generate one or more current predicted values ​​for each new coordinate, for each of the sets of current values ​​of the network parameter, and an update for each network parameter in the subset, and for each set of current values,The process includes: (i) determining the respective gradients for each new coordinate with respect to a reconstruction quality term that measures the error between one or more current predicted values ​​of the new coordinate and one or more new values ​​of the new coordinate in the new input signal, and (ii) the distribution parameter of the internal loss function which includes a differentiable sparseness term that penalizes non-zero updates for a subset of network parameters; updating each update using the respective gradients for each subset of network parameters and distribution parameter; and generating a compressed representation of the new input signal which identifies each update for the subset of network parameters.

[0009] In some implementations, the method includes the step of storing a compressed representation associated with data that identifies a new input signal.

[0010] In some implementations, the method includes the step of transmitting the compressed representation over a data communication network.

[0011] In some implementations, the step of determining each update for each subset of network parameters further includes: determining each final gate value for each network parameter in the subset according to the distribution parameters after one or more internal iterations; setting the final update of any network parameter in the subset to zero for which the final gate value specifies that each update of the subset is set to zero; and setting the final update of any network parameter in the subset based on each update of the network parameter after one or more internal iterations for which the final gate value specifies that each update of the subset is not set to zero.

[0012] In some implementations, a subset of network parameters is a suitable subset of network parameters.

[0013] In some implementations, the first neural network layer within the neural network has network parameters including (i) weight tensors and (ii) modulation tensors, where the modulation tensors are in a subset and the weight tensors are not.

[0014] In some implementations, the first neural network layer is configured to perform operations that include computing an affine transformation between the weight tensor and the layer input to the layer, and applying a modulation tensor to the output of the affine transformation.

[0015] In some implementations, the network parameters of the first neural network layer further include (iii) a bias tensor, the bias tensor is not in a subset, and applying the modulation tensor to the output of the affine transform includes applying the modulation tensor and the bias tensor to the output of the affine transform.

[0016] In some implementations, the subset includes all of the neural network parameters.

[0017] In some implementations, this method includes the steps of maintaining data that specifies the shared distribution parameters and setting the distribution parameters to be equal to the shared distribution parameters before the first iteration of one or more internal iterations.

[0018] In some implementations, this method involves training the neural network with multiple training signals to determine shared values ​​for network parameters.

[0019] In some implementations, the step of training a neural network with multiple training signals includes a step of training the neural network to minimize an internal loss function, which is evaluated after performing a certain number of internal training steps starting from a given set of shared values ​​for the network parameters, for a given set of shared values.

[0020] In some implementations, the step of training a neural network with multiple training signals to determine the shared values ​​of network parameters includes the step of training a neural network with multiple training signals to determine the shared values ​​and shared distribution parameters of network parameters.

[0021] In some implementations, the step of training a neural network with multiple training signals includes training the neural network to minimize an internal loss function, which is evaluated after performing a certain number of internal training steps, starting from a given set of shared values ​​and a given set of shared distribution parameters for the network parameters.

[0022] In some implementations, the step of determining the respective gate value of each network parameter in a subset, which specifies whether each update in the subset is set to zero according to a set of distribution parameters, includes the step of sampling noise from a noise distribution and the step of mapping the set of distribution parameters and the sampled noise to the respective gate values ​​of the network parameters in the subset.

[0023] In some implementations, the step of mapping a set of distribution parameters and sampled noise to the respective gate values ​​of network parameters in a subset includes the step of applying hard rectification to the values ​​determined from the distribution parameters and sampled noise.

[0024] In some implementations, the compressed representation of the new input signal identifies only non-zero updates of network parameters within the subset of network parameters.

[0025] In some implementations, each network parameter in the subset has a respective gate value different from each other network parameter in the subset.

[0026] In some implementations, two or more network parameters in the subset share the same respective gate value.

[0027] In some implementations, the differentiable sparsity term measures a sum of respective probabilities for respective gate values, wherein each respective probability for a gate value is defined by a distribution parameter and specifies a likelihood that a respective update of one or more network parameters corresponding to the gate value is set to a non-zero value.

[0028] In some implementations, the new input signal is an image, each coordinate corresponds to a respective pixel of the image in a two-dimensional coordinate space, and the one or more respective values include one or more intensity values of the pixel.

[0029] In some implementations, the new input signal is a three-dimensional image, each coordinate corresponds to a respective voxel of the image in a three-dimensional coordinate space, and the one or more respective values include one or more intensity values of the voxel.

[0030] In some implementations, the new input signal is a point cloud, each coordinate corresponds to a respective point in a three-dimensional coordinate space, and the one or more respective values include a respective intensity of the respective point.

[0031] In some implementations, the new input signal is video, and each coordinate is a three-dimensional coordinate that identifies the spatial position of a pixel within a video frame from the video, with one or more values ​​for each containing one or more intensity values ​​for the pixel.

[0032] In some implementations, the new input signal is an audio signal, each coordinate is a different time point in the audio signal, and each of the one or more values ​​contains one or more values ​​that define the amplitude of the audio signal at that time point.

[0033] In some implementations, the new input signal represents a signed distance function, where one or more values ​​each contain the signed distance from the boundary of the object at the corresponding coordinates.

[0034] In some implementations, a new input signal represents the rendered scene.

[0035] In another embodiment, the method includes the steps of receiving a request to reconstruct an input data signal, and obtaining (i) data specifying shared values ​​of the network parameters of the data reconstruction neural network, and (ii) data specifying updates to each of the subsets of network parameters determined for the input data signal, by training a data reconstruction neural network to reconstruct the input signal while applying a differentiable sparseness term that penalizes updates to a subset of non-zero network parameters, and generating a reconstructed input signal, which includes processing an input specifying coordinates using the data reconstruction neural network according to the shared values ​​and the values ​​of the network parameters defined by the respective updates, in order to generate one or more values ​​of the reconstructed input signal at each of a plurality of coordinates from the coordinate space of the input data signal.

[0036] In some implementations, the reconstructed input signal has values ​​for more coordinates than the input signal.

[0037] In another aspect, this specification describes one or more computer-readable storage media for storing a compressed representation of a new data signal, wherein the compressed representation of the data signal maintains data specifying shared values ​​of network parameters of a data reconstruction neural network, the data reconstruction neural network is configured to receive an input specifying coordinates from the coordinate space of an input data signal, process the input according to the network parameters to produce one or more predicted values ​​of the input data signal at the specified coordinates as outputs, maintains, receives a new input signal containing one or more respective new values ​​at each of a plurality of new coordinates, and determines each update for each subset of network parameters, the determination of each update being, in each of one or more internal iterations, determines one or more sets of current values ​​of the network parameters of the data reconstruction neural network, and for each set of current values, whether each update of the subset is set to zero according to a set of distribution parameters. Determining one or more sets of current values ​​for network parameters, including determining the gate value for each network parameter in a subset, setting the current value for any network parameter not in the subset based on the shared value of the network parameter, setting the current value for any network parameter in the subset based on the shared value of the network parameter for any network parameter in the subset where the respective gate value specifies that each update in the subset is set to zero, and setting the current value for any network parameter in the subset based on the shared value of the network parameter and each update in the network parameter for any network parameter in the subset where the respective gate value specifies that each update in the subset is not set to zero, and for each of the one or more sets of current values ​​for network parameters, using a data reconstruction neural network to generate one or more current predicted values ​​for each new coordinate according to the set of current values ​​for the network parameter.The process is carried out by determining each update, which includes processing an input specifying new coordinates, determining each update of the network parameters in a subset, and for each set of current values, determining the respective gradients with respect to (i) a reconstruction quality term that measures the error between one or more current predicted values ​​of the new coordinate and one or more new values ​​of the new coordinate in the new input signal for each new coordinate, and (ii) a distribution parameter of the internal loss function that includes a differentiable sparseness term that penalizes non-zero updates for the subset of network parameters, updating each update using the respective gradients for each subset of network parameters and distribution parameter, and generating a compressed representation of the new input signal that identifies each update for the subset of network parameters.

[0038] In another aspect, this specification describes a compressed representation of a data signal, such as a bitstream, where the compressed representation of the data signal maintains data specifying shared values ​​of network parameters of a data reconstruction neural network, the data reconstruction neural network is configured to receive an input specifying coordinates from the coordinate space of an input data signal, process the input according to the network parameters to produce one or more predicted values ​​of the input data signal at the specified coordinates as an output, maintains data, receives a new input signal containing one or more new values ​​at each of a plurality of new coordinates, and determines an update for each subset of network parameters, where determining an update means determining one or more sets of current values ​​of the network parameters of the data reconstruction neural network in each of one or more internal iterations, where for each set of current values, specifies whether the update for each subset is set to zero according to a set of distribution parameters, sub Determining one or more sets of current values ​​for network parameters, which includes determining the gate value for each network parameter in the set, setting the current value for any network parameter not in the subset based on the shared value of the network parameter, setting the current value for any network parameter in the subset based on the shared value of the network parameter for any network parameter in the subset where the gate value specifies that each update in the subset is set to zero, and setting the current value for any network parameter in the subset based on the shared value of the network parameter and each update in the network parameter for any network parameter in the subset where the gate value specifies that each update in the subset is not set to zero, and for each of the one or more sets of current values ​​for network parameters, using a data reconstruction neural network to generate one or more current predicted values ​​for each new coordinate according to the set of current values ​​for the network parameter.The process is carried out by determining each update, which includes processing an input specifying new coordinates, determining each update of the network parameters in a subset, and for each set of current values, determining the respective gradients with respect to (i) a reconstruction quality term that measures the error between one or more current predicted values ​​of the new coordinate and one or more new values ​​of the new coordinate in the new input signal for each new coordinate, and (ii) a distribution parameter of the internal loss function that includes a differentiable sparseness term that penalizes non-zero updates for the subset of network parameters, updating each update using the respective gradients for each subset of network parameters and distribution parameter, and generating a compressed representation of the new input signal that identifies each update for the subset of network parameters.

[0039] Certain embodiments of the subject matter described herein can be implemented to achieve one or more of the following advantages:

[0040] Some data compression techniques use neural networks to map data from coordinate space to underlying continuous signals in order to compress it. This type of approach is called implicit neural representation (INR), and after careful architectural searches, INR has been shown to perform better than other established compression methods for smaller dimensional data or when a small compression ratio is required.

[0041] However, these approaches require that for each data signal being compressed, the entire set of dense network parameters of a neural network that maps coordinates to underlying signal values ​​("data reconstruction neural network") be sent from the compression system to the reconstruction system. Therefore, INR has not been shown to scale to real-world compression scenarios and large-scale real-world data signals.

[0042] This specification describes techniques for significantly improving the compression ratio of INR (and thus significantly reducing compression costs) while maintaining high reconstruction quality (e.g., low reconstruction error). In particular, this specification describes techniques for significantly reducing the amount of data that needs to be communicated between the compression and reconstruction systems for each signal within the INR framework, while maintaining high reconstruction quality. More specifically, this improvement in compression ratio is achieved by learning per-signal parameter value updates for a subset of network parameters in a way that encourages sparse updates, thereby maintaining a high compression ratio because only these sparse updates need to be transmitted to the reconstruction system. Furthermore, the techniques described can be applied to achieve high reconstruction quality in a wide variety of data modalities, including images, manifolds, signed distance functions, 3D shapes, and scenes.

[0043] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0044] [Figure 1] This is a diagram illustrating an exemplary compression and decompression system. [Figure 2] This figure shows an example of compressing and restoring a new data signal using a data reconstruction neural network. [Figure 3] This is a flowchart illustrating an exemplary process for compressing a new signal. [Figure 4] This is a flowchart illustrating an exemplary process for restoring a new signal. [Figure 5] This is a flowchart for training a data reconstruction neural network using meta-learning. [Figure 6] This figure shows an example of the compression results achieved using the described techniques compared to other approaches for three different image datasets. [Figure 7] This figure shows an example of the compression results achieved using the described techniques for various types of data signals, compared to other approaches. [Figure 8] This figure shows an exemplary reconstruction of an exemplary image generated by the described technology. [Figure 9] This figure shows an example of compression results achieved using the described technique when the network parameter values ​​are sparse. [Modes for carrying out the invention]

[0045] Similar reference numbers and names in various drawings refer to the same elements.

[0046] Figure 1 shows an exemplary compression system 100 and an exemplary restoration system 150.

[0047] System 100 and System 150 are examples of systems implemented as computer programs on one or more computers in one or more locations where the systems, components, and technologies described below can be implemented.

[0048] The compression system 100 is a system that uses a data reconstruction neural network 110 to compress an input signal 102 and generate a compressed representation 112 of the input signal 102, for example, a bitstream that encodes signal-specific information necessary to reconstruct the input signal 102.

[0049] The restoration system 150 is a system that restores the input signal 102 from a compressed representation 112, for example, by generating a reconstruction 152 of the input signal 102 using a data reconstruction neural network 110.

[0050] In general, the compression system 100 and the decompression system 150 may be located in the same place or in separate locations. That is, the compression system 100 may be implemented on the same set of computers as the decompression system 150, or on a different set of computers located in a different location from the decompression system 150.

[0051] The compressed representation generated by the compression system 100 can be provided to the decompression system 150 by any of the following methods.

[0052] For example, compressed data may be stored (e.g., in a physical data storage device or logical data storage area) associated with data that identifies the corresponding input signal, and then retrieved from storage and provided to the recovery system 150.

[0053] As another example, compressed data may be sent to a destination via a communication network (e.g., the Internet), where it is subsequently extracted and provided to a decompression system 150.

[0054] The input signal 102 is a data signal having one or more values ​​for each of a plurality of coordinates in coordinate space.

[0055] For example, the input signal can be an image in a two-dimensional coordinate space, such as a two-dimensional grid, where each of the multiple pixels has one or more intensity values ​​for each channel. Thus, each coordinate corresponds to each pixel of the image in the two-dimensional coordinate space, and each of the one or more values ​​is one or more intensity values ​​of the pixel. For example, in the case of an RGB image, each of the one or more values ​​may include the R, G, and B values ​​of the pixel.

[0056] As another example, the input signal can be a three-dimensional image, where each coordinate corresponds to a voxel in a three-dimensional coordinate space, such as a three-dimensional grid, and one or more values ​​are one or more intensity values ​​of the voxel. Examples of such images include computed tomography (CT) images, magnetic resonance imaging (MRI) images, ultrasound images, X-ray images, mammograms, fluoroscopic images, or positron emission tomography (PET) images.

[0057] As another example, the input signal can be a point cloud, where each coordinate can correspond to a point in 3D coordinate space, and one or more values ​​can include the respective intensity of each point. These values ​​can optionally include other components of the points within the point cloud, such as return, second return, extension, etc. For example, the input signal can be a point cloud captured by a laser sensor, such as a LiDAR sensor.

[0058] As another example, the input signal could be video, where each coordinate is a three-dimensional coordinate that identifies the spatial position of a pixel within the video frame from the video, for example, the x, y, and t coordinates of the pixel, and one or more of each value is one or more intensity values ​​of the pixel.

[0059] As another example, the input signal could be an audio signal, each coordinate being a point in time within the audio signal, and each of the one or more values ​​being one or more values ​​that define the amplitude of the audio signal at that point in time, for example, a raw amplitude value, a compressed amplitude value, a compounded amplitude value, or a compressed and compounded amplitude value.

[0060] As can be seen from the example above, the input signal can be any suitable signal sensed by one or more sensors, for example, one or more sensors configured to sense the real-world environment.

[0061] As another example, the input signal could represent, for example, a signed distance function of an object or a set of one or more objects, where one or more values ​​each contain the signed distance from the boundary of the object (or set of objects) at the corresponding coordinates.

[0062] As another example, the input signal can represent a rendered scene, such as a 3D rendered scene, where the coordinates represent points in the scene's coordinate space, such as a 3D coordinate system, and each value may include the scene's density value and one or more color values ​​at the point. The 3D rendered scene may be a 3D scene rendered based on one or more 2D images, such as one or more 2D images captured by a camera. That is, the input signal can represent the information needed to generate a 2D image of the 3D rendered scene from an arbitrary new viewpoint different from the 2D image captured by the camera. In other words, the input signal can represent a scene in a neural radiance field (NeRF) framework.

[0063] The data reconstruction neural network 110 is a neural network configured to receive an input specifying a coordinate from the coordinate space of an input data signal, process the input according to network parameters, and generate one or more predicted values ​​of the input data signal at the specified coordinate, i.e., predicted values ​​of one or more values ​​of the input data signal at the specified coordinate, as an output. Therefore, the neural network 110 can be used to generate or reconstruct a data signal by providing the coordinate as an input to the neural network 110 for each coordinate of the data signal and obtaining one or more predicted values ​​of the data signal at the coordinate as an output.

[0064] The data reconstruction neural network 110 can generally have any suitable architecture that enables the neural network 110 to map an input specifying coordinates to one or more predicted values ​​of those coordinates.

[0065] For example, the neural network 110 can be a multilayer perceptron (MLP). Optionally, the MLP can be extended with position encoding, a sinusoidal activation function, or both.

[0066] To compress and restore the data signal using the neural network 110, the compression system 100 and the restoration system 150 each maintain data that specifies the shared values ​​120 of the network parameters of the data reconstruction neural network 110.

[0067] These values ​​are called "shared" values ​​because they are determined before any given data item to be compressed is received and are used as part of the subsequent compression and decompression process for each data signal.

[0068] For example, the shared value 120 can be determined, for instance, by training a data reconstruction neural network 110 with a set of training data containing multiple different data signals, using meta-learning. An example of training the data reconstruction neural network 110 is described in more detail below with reference to Figure 5.

[0069] When the system 100 receives a new input signal 102 containing one or more values ​​("new values") at each of multiple coordinates ("new coordinates") in coordinate space, the system 100 determines an update 122 for each of the subsets of network parameters.

[0070] In some implementations, the subset includes all network parameters.

[0071] In some other implementations, the subset is a suitable subset containing fewer parameters than all network parameters. These implementations are described in more detail below with reference to Figure 3.

[0072] At high levels, system 100 generates updates 122 in such a way that the reconstructed neural network 110 produces a more accurate reconstruction of the new input signal 102, while encouraging updates 122 to be sparse, i.e., updates 122 to be zero for most of the network parameters in a subset.

[0073] The decision regarding the update 122 to the subset of network parameters will be explained in more detail below, with reference to Figures 2 and 3.

[0074] After generating update 122, system 100 generates a compressed representation 112 of a new input signal 102 that identifies each update 122 for a subset of network parameters. Since system 100 determines each update 122, many of the updates 122 are zero, and the compressed representation 112 only needs to identify which parameters have non-zero updates and each update 122 for the identified parameters. This significantly reduces compression costs while maintaining high-quality compression.

[0075] For example, system 100 can apply a compression technique to data that identifies which parameters have non-zero updates and each update 122 of the identified parameters in order to generate a compressed representation 112. The compression technique can be any suitable technique, such as an entropy coding technique such as arithmetic coding or Huffman coding, or a learned deep entropy coding technique, or any other suitable technique.

[0076] As a specific example, system 100 can apply a quantization scheme, such as a uniform quantization scheme, to a non-zero update to generate a quantized update, and then entropy-code the quantized update to generate a compressed representation 112.

[0077] Since the compressed representation only needs to identify the non-zero parameter update 122, it will be understood that the amount of data required to encode update 122 is already significantly reduced (i.e., without further compression) compared to the data required to represent all parameter updates. Therefore, in the alternative example, the compressed representation 112 can represent update 122 without applying any further compression techniques. In other words, update 122 and the compressed representation 112 may be identical.

[0078] Once generated, the system 100 can store the compressed representation 112 in association with data that identifies the new signal 102 for later use, or it can transmit the compressed representation 112 over a data communication network.

[0079] When the restoration system 150 receives a request to reconstruct the input signal 102, it obtains data specifying updates 122 for each subset of network parameters determined for the input signal 102.

[0080] In particular, the system 150 can restore the compressed representation 112 using a decompression technique corresponding to the compression technique used to generate the compressed representation 112, and determine the respective updates 122.

[0081] Next, system 150 generates a reconstructed input signal 152 using the shared value 120 and the respective updates 122. If no further compression is applied to the updates 122, system 150 can simply receive the updates 122.

[0082] In particular, system 150 can determine a new value 160 for a network parameter defined by a shared value 120 and its respective update value 122, for example, by combining the update 122 and the shared value 120 of the network parameter for each network parameter having a non-zero update 122. System 150 can also determine a new value 160 for a given network parameter, for example, by adding the update 122 and the shared value 120, or by subtracting the update 122 from the shared value 120.

[0083] To generate the reconstructed input signal 152, the system 150 processes the input specifying the coordinates for each of the multiple coordinates from the coordinate space of the input data signal 102 using the data reconstruction neural network 110 according to a new value 160, thereby generating the value of the reconstructed input signal 152 in the coordinates.

[0084] In some implementations, the restoration system 150 generates a "faithful" reconstruction that includes the reconstructed values ​​of each coordinate of the input signal 102.

[0085] In some other implementations, the reconstruction system 150 can generate a higher-resolution reconstruction that has more coordinate values ​​than the input signal 102. That is, as part of generating the reconstruction 152, the system 150 can perform super-resolution or infilling to increase the resolution of the input signal 102.

[0086] Figure 2 shows an example of compressing and restoring a new data signal using a data reconstruction neural network 110.

[0087] As shown in Figure 2, the compression of the new data signal begins with dense initialization values ​​210 of the network parameters of the data reconstruction neural network 110, which are also referred to herein as the "shared values" of the parameters.

[0088] These dense initialization values ​​are called dense because there are no constraints imposed on the values ​​that would encourage a significant proportion of the values ​​to be zero. In a specific example, these values ​​may be determined by training a data reconstruction neural network 110 through meta-learning. This example is described in more detail below with reference to Figure 5.

[0089] Next, the compression system 100 performs a sparse adaptation 220 to determine sparse updates for a subset of network parameters. The adaptation 220 generates dense updates to the initial values ​​210 of the network parameters. The adaptation 220 is called "sparse" because, during the adaptation, the system 100 is penalized for parameters in the subset that have non-zero updates. As a result, the adaptation 220 generally ends with updates that set a large number of parameters in the subset to zero.

[0090] The process of performing this sparse adaptation 220 to determine each update is described in more detail below, with reference to Figure 3.

[0091] Next, the compression system 100 performs encoding 230 to generate a compressed representation of the new data signal, for example, a bitstream 232. The compressed representation contains enough information for the restoration system 150 to accurately reconstruct the new data signal.

[0092] In particular, since dense initialization values ​​210 are shared across all data signals, and the reconstruction system 150 also maintains values ​​210 and instances of the reconstructed neural network 110, the only information required by the reconstruction system 150 to reconstruct the data signals is the update determined by the sparse adaptation 220 by performing the adaptation 220.

[0093] Therefore, the compression system 100 encodes the updates by applying compression techniques to them, generating a bitstream 232 that can be decoded by the decompression system 150. Since the adaptation 220 is "sparse", only a small portion of the updates are non-zero and need to be encoded into the bitstream 232, further improving the compression ratio of the compression performed by system 100.

[0094] To reconstruct the data signal, the recovery system 150 performs decoding 240 by recovering the bitstream 232 to retrieve the updates, and then combining the updates with dense initialization values ​​210 (shared values) to generate new values ​​for the network parameters.

[0095] Next, system 150 performs a dense reconstruction 250 according to the new values. The reconstruction is dense, which enables high reconstruction quality, but this density is achieved by dense initialization values ​​210 (shared values) that do not affect the compression cost per signal. Therefore, signal-specific information is only updated signal-specifically, and these are sparse, enabling a high compression ratio.

[0096] Figure 3 is a flowchart of an exemplary process 300 for compressing a new data signal. For convenience, process 300 is described as being performed by a system of one or more computers located in one or more locations. For example, a well-programmed compression system, such as the compression system 100 in Figure 1, can perform process 300.

[0097] The system maintains data that specifies the shared values ​​for the network parameters of the data-reconstructed neural network (step 302).

[0098] Next, the system receives a new input signal containing one or more values ​​("new values") at each of several coordinates ("new coordinates") in coordinate space (step 304).

[0099] As described above, the system determines the update for each subset of network parameters.

[0100] In some implementations, the subset includes all network parameters.

[0101] In some other implementations, the subset is a suitable subset containing fewer parameters than all network parameters.

[0102] As a specific example, one or more layers in a neural network may have network parameters including (i) weight tensors and (ii) modulation tensors, where the modulation tensors are in a subset and the weight tensors are not.

[0103] For example, when processing an input, a layer having this architecture can be configured to perform operations that include computing an affine transformation between a weight tensor and the layer input to the layer, and applying a modulation tensor to the output of the affine transformation. For example, the affine transformation can be a multiplication or convolution between matrices or between matrix vectors. The layer can apply modulation coefficients by adding a modulation tensor to the output of the affine transformation.

[0104] Optionally, layers with this architecture may also have bias tensors that are not present in the subset, and applying a modulation tensor to the output of an affine transform may include applying both a modulation tensor and a bias tensor to the output of an affine transform. For example, a layer may add both a modulation tensor and a bias tensor to the output of an affine transform.

[0105] Therefore, layers with this architecture can adapt to each new input signal by changing only a small subset of parameters, and parameters not in the subset cannot be updated and thus do not need to be represented in a compressed representation, thus reducing the compression cost.

[0106] To decide whether to update the network parameters, the system performs one or more internal optimization iterations ("internal iterations"). In each internal iteration, the system optimizes the internal loss function so that the update encourages improved reconstruction quality while also encouraging sparseness among the network parameters within a subset.

[0107] In particular, in each internal iteration, the system determines one or more sets of current values ​​for network parameters (step 306).

[0108] To generate a given set of current values ​​in a given internal iteration, the system determines the respective gate values ​​for each network parameter within a subset, according to a set of distribution parameters, which specifies whether each update in the subset is set to zero.

[0109] For example, the distribution parameters could be parameters of a continuous distribution that assigns a probability to each gate. For instance, the continuous distribution could be a hard concrete distribution, or any other continuous distribution that allows for the reparameterization trick. The system can then use a suitable estimator to sample from the continuous distribution during testing.

[0110] In the first internal iteration, the parameter can be a shared distribution parameter common to all new data signals. That is, the system can maintain data specifying the shared distribution parameter and set the distribution parameter to be equal to the shared distribution parameter before the first iteration of one or more internal iterations.

[0111] As an example, these shared distribution parameters can be predetermined and fixed for all data signals.

[0112] As another example, shared distribution parameters can be learned along with shared values ​​of network parameters, for example, during the training of a data reconstruction neural network using meta-learning. That is, as part of meta-learning, the system effectively learns an initial sparse update configuration that is determined to work well for the training data signal used for meta-learning.

[0113] In any subsequent internal iteration, the distribution parameter can be the distribution parameter after it has been updated in the preceding internal iteration.

[0114] In this example, the system can determine each gate value by using a reparameterization trick, i.e., by sampling noise from a noise distribution, then performing a deterministic mapping of the distribution parameters and noise to the respective probabilities of each gate, and then performing hard rectification, which maps each gate to zero (indicating that the update is not set to zero) or 1 (indicating that the update is not set to zero). Performing hard rectification means mapping the value s sampled from the distribution (or determined using the deterministic mapping) to 0 or 1 by applying a function g such as g(s) = min(1, max(0, s)).

[0115] Next, the system determines the set of current values.

[0116] In particular, for any network parameter not included in a subset, the system sets its current value based on the shared value of the network parameter, for example, to be equal to the shared value of the network parameter.

[0117] For any network parameter within a subset whose respective gate value specifies that each update of the subset should be set to zero, the system sets the current value based on the shared value of the network parameter, for example, to be equal to the shared value.

[0118] For any network parameter in a subset whose respective gate value specifies that each update in the subset is not set to zero, the system sets a current value based on the shared value of the network parameter and each update of the network parameter, for example, to be equal to the sum or difference of the shared value of the network parameter and each update at the time of the internal iteration, or to be equal to the sum or difference of the moving average of the shared value of the network parameter and each update at the time of the internal iteration and any previous internal iteration.

[0119] In some implementations, each network parameter within a subset has a different gate value than each other network parameter within the subset; that is, the distribution parameter assigns a different probability to each network parameter for each separate gate value.

[0120] In some other implementations, two or more network parameters within a subset share the same respective gate values. That is, this system imposes a sparse pattern on updates by requiring specific network parameters within a subset to be jointly set to zero or jointly updated.

[0121] When multiple sets of current values ​​are generated, the system can sample noise independently for each set of current values, meaning that different sets of current values ​​can assign zero updates to different network parameters within the subset.

[0122] For each of one or more sets of current values ​​of the network parameters, the system uses the set of current values ​​of the network parameters to generate a current reconstruction of the new input signal (step 308). That is, for each set of current values ​​and each of the new coordinates, the system uses a data-compressed neural network to process the input specifying the new coordinates according to the set of current values ​​of the network parameters in order to generate a current predicted value for the new coordinates.

[0123] In each internal iteration, the system determines the respective gradients with respect to each update of the network parameters within the subset and to the distribution parameters of the internal loss function (step 310).

[0124] In particular, the internal loss function includes, for each set of current values, (i) a reconstruction quality term that measures the error between one or more current predicted values ​​of the new coordinate and one or more new values ​​of the new coordinate in the new input signal for each new coordinate, and (ii) a differentiable sparseness term that penalizes non-zero updates for a subset of network parameters. For example, the internal loss function can be the sum or weighted sum of the reconstruction quality term and the differentiable sparseness term. Thus, as described above, the internal loss function encourages improvements in reconstruction quality and encourages sparse updates.

[0125] For example, the reconstruction term may be a squared error term that measures the error between a vector of one or more current predicted values ​​for the new coordinate and a vector of one or more new values ​​for the new coordinate, such as the sum or average of the squares of the L2 distances.

[0126] This system can use one of several differentiable sparseness terms that penalize non-zero updates, the larger the number of non-zero updates expected given the current distribution parameters.

[0127] For example, a differentiable sparseness term can measure the sum of the probabilities for each gate value.

[0128] As described above, the probability for each gate value is defined by the distribution parameter, which specifies the likelihood that each update of one or more network parameters corresponding to the gate value will be set to a non-zero value.

[0129] For example, the probability of a given gate value can be expressed using the cumulative probability density function (CDF) of a continuous probability distribution. For instance, given the current distribution parameters, it can be expressed as 1 minus the probability that the value s is less than or equal to zero, according to the CDF.

[0130] Next, the system updates each of the network parameters and distribution parameters using each gradient, for example by applying an optimizer to the gradient (step 312).

[0131] After the final internal iteration, the system determines the (final) update for each subset of network parameters (step 314).

[0132] In particular, after the last inner iteration, the system determines the final gate value for each network parameter in the subset, as described above, according to the distribution parameters after the last inner iteration.

[0133] For any network parameter within a subset whose final gate value specifies that each update of the subset should be set to zero, the system sets the final update of the network parameter to zero.

[0134] For any network parameter in a subset whose respective final gate value specifies that each update in the subset is not set to zero, the system sets each final update of the network parameter based on each update of the network parameter after the last inner iteration. For example, the system may set the final update of a network parameter to be equal to each update of the network parameter after the last inner iteration, or it may set the final update of a network parameter to be equal to the moving average of each update after each of one or more inner iterations.

[0135] Next, the system generates a compressed representation of the new input signal that identifies each update for a subset of the network parameters (step 316).

[0136] Due to the differentiable sparseness term, many updates are zero, and in the compressed representation, it is only necessary to identify which parameters have non-zero updates and each update for the identified parameters. This significantly reduces compression costs while maintaining high-quality compression.

[0137] For example, as described above, the system can apply compression techniques to data that identifies which parameters have non-zero updates and each update of the identified parameters in order to generate a compressed representation.

[0138] Figure 4 is a flowchart of an exemplary process 400 for generating a reconstruction of a new data signal. For convenience, process 400 is described as being performed by a system of one or more computers located in one or more locations. For example, a properly programmed recovery system, such as the recovery system 150 in Figure 1, can perform process 400.

[0139] The system receives a request to reconstruct the input data signal (step 402).

[0140] The system obtains (i) data specifying the shared values ​​of the network parameters of the data-reconstructed neural network, and (ii) data specifying the updates for each subset of the network parameters determined for the input data signal (step 404).

[0141] For example, data specifying shared values ​​for network parameters can be maintained by the system and used when reconstructing all data signals.

[0142] The data specifying the update of each subset of network parameters may be determined by the compression system by training the data reconstruction neural network to reconstruct the input signal while applying a differentiable sparseness term that penalizes updates to non-zero subsets of network parameters, i.e., by training the data reconstruction neural network with the internal loss function described above.

[0143] The system generates a reconstructed input signal (step 406).

[0144] In particular, the system processes the input specifying coordinates using a data reconstruction neural network, according to shared values ​​and the values ​​of network parameters defined by their respective updates, in order to generate one or more reconstructed values ​​of the input signal at each of multiple coordinates from the coordinate space of the input data signal.

[0145] Therefore, the reconstructed input signal contains, for each of the multiple coordinates, one or more values ​​generated by the coordinate data reconstruction neural network.

[0146] Figure 5 is a flowchart of an exemplary process 500 for training a data reconstruction neural network through meta-learning to obtain shared values ​​for network parameters. For convenience, process 400 is described as being performed by one or more computer systems located in one or more locations. For example, a compression system such as the compression system 100 in Figure 1, a restoration system such as the restoration system 150 in Figure 1, or different appropriately programmed training systems can perform process 500.

[0147] The system acquires multiple training signals (step 502).

[0148] The system trains a data reconstruction neural network with multiple training signals to determine the shared values ​​of network parameters (step 504). Optionally, the system also jointly learns the shared values ​​of distribution parameters; i.e., the system trains a neural network with multiple training signals to determine the shared values ​​of network parameters and shared distribution parameters.

[0149] In particular, the system trains the neural network to minimize the external loss function.

[0150] The external loss function measures the value of the internal loss function, which is evaluated after performing a certain number of internal training steps on a given training signal, starting with a given set of shared values ​​for the network parameters, for a given set of shared values ​​and a given training signal.

[0151] When the system has also determined the shared distribution parameters, the external loss function measures the value of the internal loss function, which is evaluated after performing a certain number of internal training steps on a training signal starting from a given set of shared values ​​and a given set of shared distribution parameters for a given set of shared values ​​and a given training signal for a given set of shared distribution parameters for the network parameters.

[0152] The system can use any of the various meta-learning techniques to minimize the external loss function. For example, the system can use model-independent meta-learning (MAML) techniques, such as updating shared values ​​of network parameters and optionally shared distribution parameters, by directly optimizing the second-order objective, for example, by differentiating through the learning process, or by using a first-order approximation of this optimization.

[0153] Because the differentiable sparseness term in the internal loss function enhances sparseness, the computational cost of minimizing the external loss function is reduced compared to meta-learning approaches that do not employ a sparseness term in the internal loss. This reduction is a result of computational savings achieved, for example, when the imposed sparseness causes the gradient of the internal loss function to be zero for many network parameters.

[0154] Figure 6 shows an example of compression results achieved using the described technique (MSCN) compared to other approaches for three different image datasets: CelebA, SDF, and ImageNette. The results are shown in terms of peak signal-to-noise ratio (PSNR), a metric commonly used to quantify reconstruction quality. As can be seen from Figure 6, at a given compression ratio (residual parameters, i.e., the percentage of parameters with non-zero updates), the described technique generally yields higher quality reconstructions than other approaches.

[0155] Figure 7 shows an example of compression results achieved using the described technique (MSCN) compared to another approach (Functa) for various types of data signals, including CelebA (images), ERA5 (manifold), ShapeNet (voxel grid), and SRN Cars (rendered 3D scenes). In the example in Figure 7, the subset of network parameters is a suitable subset and does not include all of the neural network parameters. The different columns in Figure 7 correspond to different sizes of the suitable subset. As can be seen from Figure 7, for a given size of subset, the described technique generally yields higher quality reconstructions than other approaches across various types of data signals.

[0156] Figure 8 shows an exemplary reconstruction 800 of exemplary image 802 generated by the described technique. In particular, Figure 8 shows an exemplary reconstruction 804 generated using the described technique, compared to reconstruction 806 generated using the existing technique Meta-Sparse-INR at the same sparseness, with different percentages of network parameter updates in the subset set to zero. In particular, Figure 8 shows reconstructions 804 and 806 at (from left to right) 90% sparseness, 95% sparseness, 97% sparseness, 98% sparseness, and 99% sparseness. As can be seen from Figure 8, the described technique provides a consistent improvement in reconstruction quality across all sparseness levels and can provide a consistent reconstruction of the original image 802 even at 99% sparseness.

[0157] Figure 9 shows an example of a compression result achieved using the described technique when the network parameter values ​​are sparse. That is, as described above, the system does not impose any sparse constraints on the shared values ​​of the network parameters. Therefore, the current values ​​of the network parameters used in any given internal iteration, and the new values ​​of the network parameters used by the reconstruction system to reconstruct the data signal, are likely to be dense, i.e., unlikely to have a significant number of zero values. However, a sparse final network, i.e., a neural network with a substantial number of parameters having zero values, may have desirable applications (e.g., for a fast forward pass during inference when performing internal iterations or reconstructing the data signal).

[0158] In such cases, the system can apply the gate value to the sum of the shared value and the update during the internal iteration, meaning that not only the updates become sparse, but the current values ​​of the network parameters also become sparse. That is, if the gate value indicates that the network parameter value should be set to zero for a subset of parameters, the system sets the new value of the parameter (not just the update) to zero. Thus, instead of sparse updates, the system achieves sparse current values ​​for the network parameters in the subset.

[0159] The example in Figure 9 shows the PSNR of three different approaches on the SDF dataset: the existing Meta-Sparse INR technique, the aforementioned approach with sparse updates (MSCN-δθ sparse), and the aforementioned approach with sparse current values ​​(MSCN-(θ0+δθ)). As can be seen from the example in Figure 9, the approach with sparse updates improves over both alternatives for each proportion of residual parameters, while the approach with sparse current values ​​nevertheless consistently improves over the existing technique. Note that in the case of the (MSCN-(θ0+δθ)) technique, residual parameters mean that the parameters have non-zero values, not just that the parameter updates are sparse. Therefore, the (MSCN-(θ0+δθ)) technique can reduce inference time and inference cost because there are many zero-value parameters in the neural network by setting the current values ​​of many network parameters in a subset to zero.

[0160] This specification uses the term “configured” with respect to systems and computer program components. One or more computer systems are configured to perform a particular operation or action to mean that the systems have software, firmware, hardware, or a combination thereof installed that causes the system to perform the operation or action while it is running. One or more computer programs are configured to perform a particular operation or action to mean that the programs, when executed by a data processing device, contain instructions that cause the device to perform the operation or action.

[0161] The subject matter and functional operating embodiments described herein can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-temporary storage medium for execution by or control of the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage board, a random or serial access memory device, or one or more combinations thereof. Alternatively, or additionally, the program instructions can be encoded on artificially generated propagating signals, such as mechanically generated electrical, optical, or electromagnetic signals generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0162] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, machines, and equipment for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. A device may also be, or further include, a dedicated logic circuit such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). Optionally, in addition to hardware, a device may include code that creates an execution environment for computer programs, such as processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0163] Computer programs, also called or sometimes written as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, such as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, though not necessarily, correspond to a file in a file system. A program can be stored in a single file dedicated to that program, or in multiple coordinated files, such as a file storing one or more modules, subprograms, or parts of code, or in part of a file that holds other programs or data, such as one or more scripts stored in a markup language document. A computer program can be deployed to run on one computer, located in one site, or distributed across multiple sites and interconnected by a data communication network.

[0164] In this specification, the term “database” is used broadly to refer to any collection of data, which does not need to be structured in any particular way, or does not need to be structured at all, and can be stored in a storage device in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed differently.

[0165] Similarly, in this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines can be installed and run on the same one or more computers.

[0166] The processes and logic flows described herein can be executed by one or more programmable computers running one or more computer programs to perform their functions by operating on input data and producing outputs. The processes and logic flows can also be executed by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0167] A computer suitable for running computer programs can be based on a general-purpose microprocessor, a dedicated microprocessor, or both, or other types of central processing units. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are a central processing unit for executing or running instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory can be supplemented by or integrated into dedicated logic circuits. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or is operablely coupled to one or more mass storage devices for receiving data from, transferring data to, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, for example, a portable storage device such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, Global Positioning System (GPS) receiver, or Universal Serial Bus (USB) flash drive, to name just a few examples.

[0168] Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROMs, EEPROMs, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and all forms of non-volatile memory, media, and memory devices, including CD-ROMs and DVD-ROM disks.

[0169] To provide user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user, as well as a keyboard and a pointing device such as a mouse or trackball to which the user can provide input to the computer. User interaction can also be provided using other types of devices, for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user can be received in any form, including acoustic, voice, or haptic input. Furthermore, the computer can interact with the user by sending and receiving documents to and from devices used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages from the user in return.

[0170] Data processing devices for implementing machine learning models may also include, for example, dedicated hardware accelerator units for handling machine learning training or fabrication, i.e., inference, the common and computationally intensive parts of the workload.

[0171] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework or the Jax framework.

[0172] Embodiments of the subject matter described herein can be implemented in a computing system that includes, for example, a backend component as a data server, or a middleware component such as an application server, or a frontend component such as a client computer having a graphical user interface, a web browser, or an application on which a user can interact with the implementation of the subject matter described herein, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0173] A computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact via a communication network. The relationship between a client and a server arises from computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device to display data to a user interacting with a device acting as a client and to receive user input from the user. Data generated on the user device, such as the results of user interaction, can be received from the device to the server.

[0174] This specification includes many specific details of implementation, which should not be construed as limitations on the scope of any invention or potentially claimed scope, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any suitable partial combination in multiple embodiments. Furthermore, features are described above as acting in several combinations and are initially claimed as such, but in some cases one or more features from a claimed combination may be removed from the combination, and the claimed combination may be subject to a partial combination or a variation of a partial combination.

[0175] Similarly, although operations are shown in the drawings and described in the claims in a specific order, this should not be understood as requiring such operations to be performed in a specific order or sequence as shown, or that all illustrated operations must be performed to achieve a desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be combined into a single software product or packaged into multiple software products.

[0176] A specific embodiment of the subject matter is described. Other embodiments are within the scope of the claims below. For example, the actions enumerated in the claims may be performed in a different order and still achieve the desired result. As an example, the process shown in the accompanying drawings does not necessarily require the specific order or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0177] 100 Compression Systems 102 Input Signal 110 Data Reconstruction Neural Networks 112 Compressed representation 120 Shared value 122 Update 150 recovery systems 152 Reconstructed input signal 160 New value 210 Dense initialization values 220 Sparse adaptation 230 encoding 232 bitstream 240 Decryption 250 Dense Reconstruction 300 processes 400 processes 500 processes 800 Reconfiguration 802 images 804 Reconfiguration 806 Reconfiguration

Claims

1. A method performed by one or more computers, A step of maintaining data that specifies shared values ​​for the network parameters of a data reconstruction neural network, wherein the data reconstruction neural network is configured to receive an input that specifies coordinates from the coordinate space of an input data signal, process the input according to the network parameters, and generate one or more predicted values ​​of the input data signal at the specified coordinates as an output. The steps include receiving a new input signal that includes one or more new values ​​for each of several new coordinates, This step involves determining the update for each of the subsets of the aforementioned network parameters. In each of the one or more internal iterations, The step is to determine one or more sets of current values ​​of the network parameters of the data reconstruction neural network, and for each set of current values, The steps include determining the respective gate values ​​for each network parameter in the subset, specifying whether each update of the subset is set to zero according to a set of distribution parameters, This is a step in determining the set of current values. For any network parameter not included in the subset, the current value is set based on the shared value of the network parameter. For any network parameter in the subset whose respective gate value specifies that each update of the subset is set to zero, the current value is set based on the shared value of the network parameter. For any network parameter in the subset whose respective gate value specifies that the respective updates of the subset are not set to zero, the current value is set based on the shared value of the network parameter and the respective updates of the network parameter. The steps of determining the set of current values ​​and Steps including, For each of the one or more sets of current values ​​of the network parameters, For each of the new coordinates, the data reconstruction neural network is used to process the input specifying the new coordinates according to the set of current values ​​of the network parameters in order to generate one or more current predicted values ​​for the new coordinates. Steps including, The steps include determining the respective gradients for each update of the network parameters in the subset and, for each set of current values, with respect to (i) a reconstruction quality term that measures the error between one or more current predicted values ​​of the new coordinate and one or more new values ​​of the new coordinate in the new input signal, and (ii) the distribution parameter of an internal loss function that penalizes non-zero updates for the subset of network parameters. A step of updating each of the updates using the respective gradients for each of the subset of network parameters and the distribution parameters, A step of generating a compressed representation of the new input signal that identifies each of the updates for the subset of network parameters. Methods that include...

2. Steps to store the compressed representation in association with data that identifies the new input signal. The method according to claim 1, further comprising:

3. The step of transmitting the compressed representation via a data communication network. The method according to claim 1, further comprising:

4. For each of the subsets of the aforementioned network parameters, the step of determining the respective update is: After the aforementioned one or more internal iterations, The steps include determining the final gate value for each network parameter in the subset according to the distribution parameters after one or more internal iterations, For any network parameter in the subset whose respective final gate value specifies that the respective updates of the subset are set to zero, the step of setting the final update of the network parameter to zero; For any network parameter in the subset whose respective final gate value specifies that the respective updates of the subset are not set to zero, the step of setting the respective final update of the network parameter based on the respective updates of the network parameter after one or more internal iterations. The method according to claim 1, further comprising:

5. The method according to claim 1, wherein the subset of network parameters is a suitable subset of the network parameters.

6. The method according to claim 5, wherein a first neural network layer in the data reconstruction neural network has network parameters including (i) weight tensors and (ii) modulation tensors, wherein the modulation tensors are in the subset and the weight tensors are not in the subset.

7. The method according to claim 6, wherein the first neural network layer is configured to perform an operation including computing an affine transform between the weight tensor and the layer input to the first neural network layer, and applying the modulation tensor to the output of the affine transform.

8. The method according to claim 7, wherein the network parameters of the first neural network layer further include (iii) a bias tensor, the bias tensor is not in the subset, and applying the modulation tensor to the output of the affine transform includes applying the modulation tensor and the bias tensor to the output of the affine transform.

9. The method according to claim 1, wherein the subset includes all of the network parameters of the data reconstruction neural network.

10. A step of maintaining data that specifies the shared distribution parameters, Before the first iteration of the one or more internal iterations, the step of setting the distribution parameter to be equal to the shared distribution parameter. The method according to claim 1, further comprising:

11. The step of training the data reconstruction neural network with multiple training signals in order to determine the shared values ​​of the network parameters. The method according to claim 1, further comprising:

12. The step of training the data reconstruction neural network with the aforementioned multiple training signals is: A step of training the data reconstruction neural network to minimize the internal loss function, which is evaluated after performing a certain number of internal training steps starting from the given set of shared values ​​of the network parameters, for a given set of shared values. The method according to claim 11, including the method described in claim 11.

13. The step of training the data reconstruction neural network with multiple training signals in order to determine the shared values ​​of the network parameters is as follows: The step of training the data reconstruction neural network with the plurality of training signals in order to determine the shared values ​​and shared distribution parameters of the network parameters. The method according to claim 12, as referenced to claim 10, including the method described in claim 12.

14. The step of training the data reconstruction neural network with the aforementioned multiple training signals is: A step of training the data reconstruction neural network to minimize the internal loss function, which is evaluated after performing a certain number of internal training steps, starting from the given set of shared values ​​and the given set of shared distribution parameters of the network parameters, on a given set of shared values ​​and a given set of shared distribution parameters. The method according to claim 13, including the method described in claim 13.

15. The step of determining the respective gate values ​​for each network parameter in the subset, specifying whether each update of the subset is set to zero according to the set of distribution parameters, The steps involve sampling noise from the noise distribution, A step of mapping the set of distribution parameters and the sampled noise to the respective gate values ​​of the network parameters in the subset. The method according to claim 1, including the method described in claim 1.

16. The step of mapping the set of distribution parameters and the sampled noise to the respective gate values ​​of the network parameters in the subset is the step of applying hard rectification to the values ​​determined from the distribution parameters and the sampled noise. The method according to claim 15, including the method described in claim 15.

17. The method according to claim 1, wherein the compressed representation of the new input signal identifies only non-zero updates of network parameters within the subset of network parameters.

18. The method according to claim 1, wherein each network parameter in the subset has a gate value different from each other network parameter in the subset.

19. The method according to claim 1, wherein two or more network parameters within the subset share the same respective gate values.

20. The method according to claim 1, wherein the differentiable sparseness term measures the sum of the respective probabilities for each gate value, and the respective probability for each gate value specifies the likelihood that the update of each of the one or more network parameters corresponding to the gate value is set to a non-zero value, defined by the distribution parameter.

21. The method according to claim 1, wherein the new input signal is an image, each coordinate corresponds to each pixel of the image in a two-dimensional coordinate space, and each of the one or more values ​​includes one or more intensity values ​​of the pixel.

22. The method according to claim 1, wherein the new input signal is a three-dimensional image, each coordinate corresponds to a voxel of the image in three-dimensional coordinate space, and each of the one or more values ​​includes one or more intensity values ​​of the voxel.

23. The method according to claim 1, wherein the new input signal is a point cloud, where each coordinate corresponds to a point in a three-dimensional coordinate space, and each of the one or more values ​​includes the respective intensity of each of the points.

24. The method according to claim 1, wherein the new input signal is a video, and each coordinate is a three-dimensional coordinate that identifies the spatial position of a pixel in a video frame from the video, and the one or more values ​​each include one or more intensity values ​​of the pixel.

25. The method according to claim 1, wherein the new input signal is an audio signal, each coordinate is a respective time in the audio signal, and each of the one or more values ​​includes one or more values ​​that define the amplitude of the audio signal at each of the respective time points.

26. The method according to claim 1, wherein the new input signal represents a signed distance function, and each of the one or more values ​​includes the signed distance from the boundary of the object of the corresponding coordinates.

27. The method according to claim 1, wherein the new input signal represents a rendered scene.

28. A method performed by one or more computers, The steps include receiving a request to reconstruct an input data signal, The steps include: training a data reconstruction neural network to reconstruct an input signal while applying a differentiable sparseness term that penalizes the updating of a subset of non-zero network parameters, thereby obtaining (i) data specifying the shared values ​​of the network parameters of the data reconstruction neural network, and (ii) data specifying the updates of each of the subsets of the network parameters determined for the input data signal; The step of generating a reconstructed input signal, wherein for each of the multiple coordinates from the coordinate space of the input data signal, The step of processing the input specifying the coordinates using the data reconstruction neural network, according to the shared values ​​and the values ​​of the network parameters defined by the respective updates, in order to generate one or more values ​​of the reconstructed input signal at the coordinates: Steps and Methods that include...

29. The method according to claim 28, wherein the reconstructed input signal has values ​​for more coordinates than the input signal.

30. One or more computers, When executed by the one or more computers, one or more storage devices store instructions that cause the one or more computers to perform each of the operations of the method according to any one of claims 1 to 29. A system equipped with these features.

31. One or more computer-readable storage media that, when executed by one or more computers, store instructions causing the one or more computers to perform the method according to any one of claims 1 to 29.

Images