Method and device for fine tuning a set of selected parameters in a deep coding system - Patents.com

Abstract

A deep neural network-based coding system for images determines update parameters for a deep neural network model for decoding an image. These parameters are determined by an encoder and provided to a decoder to update the decoder's model before decoding the image. This provides structural sparsity by fine-tuning only a portion of the neural decoder's parameters. The updates are performed on a set of parameters selected based on an embedding representing the coded image, thereby eliminating the need to transmit information related to the selection of the updated parameters. Applications to a more general optimizer / inference engine and audio upsampling are also described.

Description

[Technical Field]

[0001] This application claims priority to European Patent Application No. 22306599.6, filed October 21, 2022, which is incorporated herein by reference in its entirety.

[0002] Technical Field At least one of the present embodiments relates generally to neural networks, and more particularly to fine-tuning a selected set of parameters of a deep neural network. [Background technology]

[0003] background A deep neural network consists of multiple neural layers, such as convolutional layers. Each neural layer can be described as a function that first multiplies the input by a tensor, adds a vector called a bias, and then applies a nonlinear function to the resulting value. The shape (and other properties) of the tensor and the type of nonlinear function are called the "architecture" of the network. In the following, the values ​​of the tensor and bias are called "weights." The weights and, if applicable, the parameters of the nonlinear function are called "parameters." The architecture and parameters define a "model."

[0004] The model M can be trained on a database of images D to learn its weights. In supervised learning, this database contains input / output pairs (i, o), and the model M is a function that attempts to predict the output from the input.

number

number

[0005] Training is successful when the resulting loss value is small. The trained model performs well on average for all inputs, but is likely suboptimal for any single input. In some applications, such as compression, inference is part of a two-stage system, where the input is first prepared or verified by an optimizer (the encoder in compression), and then processed in a second step by an inference engine (in the decoder in compression), often on a separate device. In such systems, inference results can be improved by fine-tuning (i.e., retraining) the model weights for each input individually in the optimizer. By retraining M specifically for this input, sending a weight update δ to the inference engine in addition to the input, and adding δ to θ before inference, the reconstructed output

number

number

[0006] Image and video compression is a fundamental task in image processing and has become crucial with the widespread and increasing use of video streaming. Thanks to decades of community efforts, traditional methods have reached the current state-of-the-art rate / distortion performance and dominate current industrial codec solutions. Recently, end-to-end trainable deep models have emerged as an alternative and have shown promising results. They currently outperform the best traditional compression method (VVC, generalized video coding) in terms of peak signal-to-noise ratio even for the compression of a single image. Summary of the Invention

[0007] overview In at least one embodiment, a deep neural network-based coding system for images determines update parameters for a deep neural network model for decoding the image. These parameters are determined by an encoder and provided to a decoder to update the decoder's model before decoding the image. This provides structural sparsity by fine-tuning only a subset of the neural decoder's parameters. The updates are performed on a set of parameters selected based on an embedding representing the coded image, thereby eliminating the need to transmit information related to the selection of the updated parameters. A more general optimizer / inference engine enabling data transformation and applications to audio upsampling are also described.

[0008] According to a first aspect, a method includes obtaining input data, selecting a subset of parameters for fine-tuning a model of a neural network, determining parameter updates for the selected subset of parameters based on a loss function, and packaging the input data and the parameter updates.

[0009] According to a second aspect, a method includes obtaining input data and parameter updates for a selected subset of parameters; selecting a subset of parameters for fine-tuning a model of a neural network based on the parameter optimization and the input data; updating the model of the neural network based on the parameter updates for the selected subset of parameters; and determining output data by processing the input data with the updated neural network.

[0010] According to a third aspect, a device includes a processor configured to obtain input data, select a subset of parameters for fine-tuning a model of a neural network, determine parameter updates for the selected subset of parameters based on a loss function, and package the input data and the parameter updates.

[0011] According to a fourth aspect, a device includes a processor configured to obtain input data and parameter updates for a selected subset of parameters; select a subset of parameters for fine-tuning a model of a neural network based on the parameter optimization and the input data; update the model of the neural network based on the parameter updates for the selected subset of parameters; and determine output data by processing the input data with the updated neural network.

[0012] In a first variation of the first and third aspects adapted to encoding an image, the input data is an image, a first neural network is used for encoding, and a second neural network is used for decoding, the second neural network being updated using parameter updates for a selected subset of parameters, and the method further includes determining an embedding representing the input by encoding the image using the first neural network, quantizing the embedding, and selecting the subset of parameters based on the quantized embedding.

[0013] In a second variant of the first and third aspects adapted to compress sound, the input data is an audio signal, and the method further comprises compressing the audio signal, decompressing the compressed audio signal, selecting a subset of parameters based on the decompressed compressed audio signal, and packaging the compressed audio signal and the parameter updates.

[0014] In a first variant of the second and fourth aspects adapted for decoding an image, the selection of the subset of parameters is further based on the obtained quantized embedding.

[0015] In a second variant of the second and fourth aspects adapted to decoding sound, the selection of the subset of parameters is further based on the obtained reconstructed audio signal.

[0016] According to a fifth aspect of at least one embodiment, there is provided a computer program comprising program code instructions executable by a processor, the computer program, when executed on the processor, performing at least the steps of a method according to the first or second aspect.

[0017] According to a sixth aspect of at least one embodiment, there is presented a non-transitory computer readable medium comprising program code instructions executable by a processor, the instructions, when executed on the processor, performing at least the steps of a method according to the first or second aspect.

[0018] In variations of the first, second, third and fourth embodiments, the parameters are selected from a set including biases, weights, parameters of nonlinear functions of the model, a subset of layers of the model, a specific layer of the model, biases of a specific layer of the model and a subset of neurons of the model. [Brief explanation of the drawings]

[0019] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] 1 illustrates an example of an end-to-end neural network-based compression system for encoding images using deep neural networks. [Figure 2] 1 illustrates an optimizer process according to at least one embodiment of a generic data transformation system. [Figure 3] 1 illustrates the process of an inference engine according to at least one embodiment of a universal data transformation system. [Figure 4] FIG. 1 illustrates an architecture diagram of an optimizer and inference engine according to at least one embodiment. [Figure 5A] 1 illustrates an encoder process associated with an end-to-end image compression system, according to at least one embodiment. [Figure 5B] 1 illustrates a decoder process associated with an end-to-end image compression system according to at least one embodiment. [Figure 6] 1 illustrates an encoder architecture associated with an end-to-end image compression system, according to at least one embodiment. [Figure 7] 1 illustrates a decoder architecture associated with an end-to-end image compression system, according to at least one embodiment. [Figure 8] 10 shows an example of size information comparing a bitstream generated in accordance with at least one embodiment with a bitstream of the same input generated without using any of the presented embodiments. [Figure 9] 1 illustrates an example application of an optimizer and inference engine in connection with sound enhancement, according to at least one embodiment. [Figure 10] 1 illustrates a block diagram of an example system in which various aspects and embodiments may be implemented. [Figure 11] 1 illustrates a process for an image encoder according to at least one embodiment. [Figure 12] 1 illustrates a process for an image decoder according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] Detailed Description FIG. 1 shows an example of an end-to-end neural network-based compression system 100 for encoding an image using a deep neural network. An input image x to be compressed is first processed by a device 110 including a deep neural network encoder (hereafter identified as a deep encoder or encoder). The output of the encoder y is called the embedding of the image. This embedding is converted into a bitstream 120 via a quantizer Q and then via an arithmetic encoder AE. The resulting bitstream thus contains the encoded quantized embedding of the input image. This bitstream is provided to a device 130 including a deep neural network decoder 130 (hereafter identified as a deep decoder or decoder). The bitstream is decoded via an arithmetic decoder AD to decode the quantized embedding.

number

number

[0021] Deep encoders and decoders consist of multiple neural layers. Typically, the encoder and decoder are fixed based on a predefined model that is assumed to be known during encoding and decoding. The encoder and decoder models are, for example, trained simultaneously to be compatible. Together, they can be called an "autoencoder," a model that encodes an input and then reconstructs it. The decoder architecture is typically mostly the inverse of the encoder, although some layers and their order may be slightly different. In the following, the set of decoder parameters is denoted by Ω.

[0022] Many end-to-end architectures have been proposed that may be more complex than the one shown in Figure 1, but retain deep encoders and decoders. Prior art models may be competitive with traditional video codecs such as Versatile Video Coding (VVC) in terms of rate / distortion tradeoff.

[0023] The model M needs to be trained on a large image database D to learn the encoder and decoder weights. Typically, the weights are, for example,

number

number

[0024] Typically, the architecture consists of a set of models {M i The model is trained several times using different values ​​of λ to yield a}. Typically, different architectures yield models with different r / d points. To compare these architectures, the r / d points of each architecture are interpolated, resulting in a function d(r) for each architecture, providing a distortion estimate for any rate value.

[0025] The deep decoder proposed in Figure 1 can decode any type of image. In other words, it performs well on average for all images, but is likely suboptimal for a single image. It is possible to improve the rate / distortion tradeoff for a single video by specifically retraining the decoder for this video and sending a weight update δ to the decoder in addition to the quantized embeddings for the intra frames of this video. δ is added to θ before decoding the quantized embeddings. Such a technique is called fine-tuning. The weight update δ can be calculated, for example, as follows:

number

number

number

[0026] However, this approach does not achieve rate / distortion improvement for a single image due to the increased code size caused by the included weight updates. In an exemplary solution, an additional term can be added to the loss to impose a global sparsity constraint on δ, which causes many weight updates to have the same value (0), making the coding more efficient.

[0027] Current techniques for fine-tuning decoders using a global sparsity constraint lead to improved performance in terms of rate / distortion when encoding video. However, even with a global sparsity constraint, this technique is not suitable for single images due to the increased code size caused by the weight updates involved.

[0028] A second solution proposes to fine-tune the decoder for a single image by either updating a fixed subset of weights for all images, or by updating a subset of weights specific to each image. In the latter case, the weights to be updated must be identified in the bitstream.

[0029] Previous approaches to instance-specific weight overfitting inevitably face certain suboptimality problems. If the same subset of weights is optimized for all inputs, the weight selection is not optimal for all inputs. If a subset of weights is selected specifically for each input, those weights must be identified in the bitstream, thus increasing the bit length. To limit this extra cost, weights are typically selected in chunks (e.g., layer by layer), thus limiting distortion reduction.

[0030] The embodiments described below are designed with the above in mind and are based on a novel fine-tuning procedure that proposes that the inference engine use a reproducible procedure to implicitly select a subset of weights that is optimized for a particular input. In other words, the embodiments are based on selecting and optimizing a subset of these weights without requiring transmission of input-specific weight identifiers (or locations). This weight selection is therefore more suited to a particular input (such as an image or frame, Group of Pictures, patches, or other inputs), but does not increase bit length because the location of the selected weights does not need to be transmitted. Updates still need to be transmitted. The trade-off is increased computational cost in the inference engine.

[0031] One embodiment relates to a general-purpose data transformation system that includes fine-tuning capabilities. In an end-to-end compression embodiment, the inference engine is an end-to-end decoder. In at least one embodiment, the principles are applied to a video compression system that includes a video encoder and a video decoder, where the encoded video bitstream generated by the encoder can be reduced in size because it does not include information identifying the weights to be updated.

[0032] A general-purpose system for data transformation based on selecting and optimizing a subset of these weights without transmitting an input-specific weight identifier can be implemented through the optimizer and inference engine shown in Figures 2 and 3, according to at least one embodiment.

[0033] FIG. 2 illustrates an optimizer process 200 according to at least one embodiment of a generic data transformation system. The optimizer includes a model M identical to the model of the inference engine. The model implements any function that maps an input domain to an output domain. Process 200 may be implemented, for example, by device 1000 of FIG. 10, or more specifically, by processor 1010 of such a device. In step 210, the optimizer obtains data representing inputs i and, optionally, data representing a target output o, which is the desired output of the inference engine. If a target output exists, this output is the goal of model M, and the optimizer attempts to optimize the parameters of the model so that the output approaches the target output. If a target output does not exist, the optimizer may attempt to optimize metrics related to the output that do not consider the target. These are called "no-reference" metrics. For example, if the output domain of the model is an image, metrics such as the Blind / No-Reference Image Space Quality Evaluator (BRISQUE) can be used, which attempts to maximize the probability of the label if the domain is associated with a probability distribution over labels, or if the output domain is an audio signal, the optimization may attempt to clip the signal or limit Gaussianity.

[0034] In step 220, the optimizer selects a subset of weights of fixed size s based on the input i.

number

number

[0035] This operation is replicated by the inference engine, so this step does not depend on the target output (because the inference engine does not have access to it) and only on the quantized embedding.

[0036] The ideal choice is to solve the following optimization problem:

number

[0037] The idea of ​​the first approach is to select the weights that, when modified, have the greatest impact on the target output of the model M. This property can be estimated through the gradient of the target output of the model. This can be easily computed by using the backpropagation algorithm (typically used to train models). Thus, in this first approach, the subset of weights is

number

[0038] The second approach proposes to use machine learning algorithms to infer the subsets directly from the input. A possible option is to use a supervised learning algorithm, where we have an input i and an optimal subset ω for this input. * A second machine learning model N is trained using a database containing pairs of (i). * (i) is the output of model N. Such model N can then be used as a function f(M,i) in the inference engine to determine the subset of parameters to be updated based on the quantized embedding. This function is known to both the encoder and decoder, so there is no need to transmit the location / identifier of the updated weights.

[0039] A third approach is to use reinforcement learning algorithms, which can, for example, improve ω by adding or removing elements of Ω. * We can gradually build up r, fine-tune these weight updates, and use the resulting r / d tradeoff as the algorithm's reward.

[0040] Many variations of these approaches are envisioned. In a first variation, the optimization is performed over

number

[0041] In the second variant, additional constraints are imposed on the allowable values ​​of ω, e.g., Ω (or Ω′) is a set of non-overlapping subsets

number

[0042] The weights may be selected using a no-reference loss, in which case the optimization may be performed to maximize a function that quantifies the quality of the output of M(i). As an example, the BRISQUE metric may be used to evaluate the quality of an image. Other no-reference losses have also been discussed above.

[0043] In embodiments, this procedure is not limited to deep neural networks and can use any machine learning model.

[0044] In step 230, the fine-tuning algorithm adjusts the parameter ω * The corresponding update δω * The fine-tuning loss is, for example,

number

[0045] This loss may be the same as or different from the loss used in the second step. The loss may also include additional terms, for example, terms that induce some constraints on the weights, such as a sparsity constraint. In a variation of this embodiment, the input i is

number

[0046] If no output is provided, the update can be computed by optimizing any loss that does not require a reference signal, such as the no-reference loss mentioned above.

[0047] In step 240, the inputs and weight updates are prepared for the inference engine and packaged, for example, as a data set that is stored together.

[0048] FIG. 3 illustrates a process for an inference engine according to at least one embodiment of a generic data conversion system. Process 300 may be implemented, for example, by device 1000 of FIG. 10, or more specifically, by processor 1010 or decoder 1030 of such a device. In step 310, the device obtains an input and associated parameter updates for a selected subset of parameters. In step 320, the input and parameter subset selection from model M are recomputed using the same procedure as the optimizer (or a procedure that gives the same results). In step 330, model M is updated based on the recomputed subset of parameters and parameter updates. In step 340, the updated model M processes the input and determines an output.

[0049] In at least one embodiment, an example of a parameter used in process 200 of FIG. 2 and process 300 of FIG. 3 is a weight, as shown in FIG.

[0050] 4 shows an architecture diagram of an optimizer and inference engine according to at least one embodiment. The optimizer (400) receives an input i (411) and optionally a target output o (412). In step 420, the optimizer selects a subset of weights of fixed size s based on the input i.

number

number

[0051] The description and figures refer to updating weights for ease of reading, but the same techniques can be used to update any other parameter of a neural network. In other words, the embodiments described below as applying to weights also apply more generally to any parameter of a neural network model, i.e., parameters selected from a set including biases, weights, parameters of nonlinear functions of the model, a subset of layers of the model, specific layers of the model, biases of specific layers of the model, and a subset of neurons of the model.

[0052] According to at least one embodiment, a generic data transformation system (as described in Figures 2, 3, and 4) based on selecting and optimizing a subset of input-specific weights without transmitting their identifiers is applied to an image compression system and implemented through an image encoder and an image decoder. The processes of these devices are shown in Figures 5A and 5B, respectively. The architectures of these devices are shown in Figures 6 and 7, respectively. In the context of this end-to-end compression, the inference engine is the decoder, the optimizer is the encoder, and the weight selection procedure is performed in both the encoder and the decoder. Herein, the input i and output o of the generic approach are the image / frame x to be encoded and the corresponding quantized embedding vector, respectively.

number

[0053] Figure 5A shows a process of an encoder associated with an end-to-end image compression system, according to at least one embodiment. Process 500A may be implemented, for example, by device 1000 of Figure 10, more specifically, by processor 1010 or encoder 1030 of such a device. Figure 6 shows an example architecture of such an encoder 600, which is based on the same principles as optimizer 400 of Figure 4, but adapted in the context of end-to-end image compression.

[0054] In step 510, the encoder obtains an image x. In step 515, the encoder determines an embedding vector y by using a deep encoder (610). In step 520, the embedding vector y is quantized using a quantizer (611). In step 525, the encoder calculates the quantized embedding vector

number

[0055] This corresponds to step 220 of the general method of Figure 2 above. This selection is made by a selection element 620, which is also present in the decoder to enable it to perform the same operation. In the first approach proposed above, i.e., when selecting parameters that have a large influence on the output at the decoder, a subset of weights may be selected, e.g.

number

[0056] The second approach (using machine learning model N) is similar to the one described above. However, the training loss for these models may consider the bit length of the parameter updates in addition to the improvement in model prediction. In other words, these models are trained to generate a subset of weights that achieves the best r / d tradeoff, rather than just distortion.

[0057] In step 530 (in relation to architectural element 630), the fine-tuning algorithm calculates the parameters ω * The corresponding update δω * For end-to-end coding, the fine-tuning loss is:

number

[0058] The loss may also include additional terms, e.g., terms that induce some additional constraints on the weights, such as a sparsity constraint.

[0059] In a variation of this embodiment, the quantized embedding

number

number

[0060] In step 535, these weight updates are typically quantized (631) and encoded (632). These quantized weight updates are

number

[0061] In step 540, for example, the following data: For example, a quantized embedding encoded by the arithmetic encoder (612) or another encoder.

number

number

[0062] Optionally, the quantization and encoding of the weight updates may depend on several parameters. These parameters may be the same for all images, or some or all of them may be tweaked for each image. In the latter case, the bitstream also includes the values ​​of these parameters (denoted by C) that are inserted as coding information (644).

[0063] The quantization and coding of the embedding may also depend on additional parameters. These elements may be placed in any order within the bitstream, or even interleaved.

[0064] Figure 5B illustrates a decoder process associated with an end-to-end image compression system, according to at least one embodiment. Process 500B may be implemented, for example, by device 1000 of Figure 10, and more specifically, by processor 1010 or decoder 1030 of such a device. Figure 7 illustrates the architecture of such a decoder 700, which is based on the same principles as inference engine 410 of Figure 4, but adapted in the context of end-to-end image compression.

[0065] In step 550, the quantized embedding and weight updates are extracted from the bitstream (640). Optionally, both the quantized embedding and the quantized weight updates are decoded (711 and 713) using parameters carried by the coding information (also extracted from the bitstream).

[0066] In step 560, a subset of weights ω * is determined (620). This step should produce the same result as the corresponding step 540 in the encoder. This can be achieved by using the same procedure (620). An advantage of the embodiments described herein is that the subset ω is selected at the cost of an extra operation in the decoder to make the selection (620). * The advantage of this approach is that it is not necessary to include it in the bitstream.

[0067] In step 570, the deep decoder is updated (720) based on the subset of weights and the quantized weight updates. Then, in step 580, the image is decoded from the quantized embedding by the updated decoder (730).

[0068] The above-described embodiment is based on a system in which reversible operations related to quantization of weight updates are also inverted within the AD block. The same system can be described using additional blocks called, for example, "dequantization" or "inverse quantization" to perform these operations. An example of such a reversible operation is scaling weight updates before quantization to change the quantization resolution.

[0069] FIG. 8 shows an example of size information comparing a bitstream generated according to at least one embodiment with a bitstream of the same input generated without using any of the presented embodiments. Bitstream 800 is generated based on an embodiment related to end-to-end image compression according to an exemplary implementation of process 500A of FIG. 5A. It includes coded quantization embedding 801, coded weight updates 802, and optional coding information 803. The sizes of these different elements are 28160, 1440, and 40 bits, respectively. These specific values ​​were obtained using the "cheng2020#anchor" end-to-end encoder from the compressAI library as model M. Bitstream 810 is generated based on the same input and with the same settings based on a prior art fine-tunable end-to-end neural network-based compression system. Contrary to the proposed embodiment, such a system requires information 811 identifying the weights to be updated (i.e., their location, e.g., based on exponents) to be communicated from the encoder to the decoder. The other elements of bitstream 810 have the same size as bitstream 800, but the addition of data 811 increases the size of the encoded message. Alternative implementations may generate data of other sizes, but still achieve the same benefits of reducing the size of the generated bitstream and thus improving the performance of the end-to-end video compression system.

[0070] According to at least one embodiment, a generic data conversion system (such as those described in Figures 2, 3, and 4) based on selecting and optimizing a subset of these weights without transmitting an input-specific weight identifier is applied to a sound enhancement or compression system.

[0071] 9 shows an example application of an optimizer (900) and an inference engine (901) in connection with sound enhancement, according to at least one embodiment. Deep sound upsampling is a sound improvement method in which an audio signal is transformed by an upsampling neural network that increases the number of sample points. One possible application of sound upsampling is to improve the quality of low-frequency or downsampled audio signals. In this context, a method for improving the sound quality of a downsampled audio signal sent to a device can be applied as follows: The downsampled original audio signal (or rather the audio received by the inference engine) is the input i (911). The original high-frequency audio signal is the target output o (912). The upsampling neural network is the model M.

[0072] The original audio signal may be pre-processed by an optimizer before being sent to an audio playback device, such as a mobile device or a computing device that reads an audio file streamed by a server or any other device adapted to play audio. In such a scenario, the optimizer is implemented in the audio server and the inference engine is implemented in the audio playback device.

[0073] The server first obtains the audio file with the high-frequency target signal. The server then prepares the original audio file for transmission (if not already done) by, for example, compressing (915) it to obtain a signal 918 and processing it, for example, decoding or decompressing (916) it to recover a signal 919 used by the inference engine. As an example, this may mean encoding (i.e., compressing) and decoding (i.e., decompressing) the signal. This modified signal (919) is then used to select (920) a subset of weights for the model M to be modified. The weight update δω *is optimized or fine-tuned (930), prepared for sending to the inference engine, and concatenated with the audio file ready for transmission (940), both of which can be sent to a device with the inference engine or stored for later use.

[0074] In the audio playback device, the audio signal (941) and weight updates (943) are received and recovered by an inference engine. This includes any processing (916) (e.g., decoding or reconstruction) performed in the inference engine. The recovered audio signal (944) is then used to determine (950) a subset of weights, which, along with the weight updates, are used to update (960) model M to an updated model M'. In other words, the selection of the parameter subset is independent of the information representing the parameter updates (940). Finally, the received audio signal (944) is used as an input for updated model M' (970), generating a resulting upsampled audio signal (980). This audio signal can be played to a user or used for any other purpose.

[0075] The optimizer 900 and the inference engine 901 are implemented, for example, by the device 1000 of Figure 10, more particularly by the processor 1010 or the decoder 1030 of such a device. The optimizer 900 and the inference engine 901 are based on the same principles as the optimizer 400 of Figure 4 and the inference engine 410 of Figure 4, respectively, but adapted in the context of end-to-end image compression.

[0076] Figure 10 shows a block diagram of an example system in which various aspects and embodiments may be implemented. System 1000 may be embodied as a device including various components described below and configured to implement one or more of the aspects described herein, such as optimizer 400 of Figure 4, inference engine 410 of Figure 4, encoder 600 of Figure 6, decoder 700 of Figure 7, optimizer 900 of Figure 9, or inference engine of Figure 9. Such a system may implement optimizer process 200 of Figure 2, inference process 300 of Figure 3, encoding process 500A of Figure 5A, decoding process 500B of Figure 5B, encoding process 1101 of Figure 11, or decoding process 1201 of Figure 12. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected consumer electronics appliances, encoders, transcoders, and servers. The elements of system 1000, singly or in combination, may be embodied in a single integrated circuit, multiple ICs and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to other similar systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the aspects described herein.

[0077] The system 1000 includes at least one processor 1010 configured to execute loaded instructions, for example, to implement various aspects described herein. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits as known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes storage 1040, which may include non-volatile memory and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives. The storage 1040 may include, by way of non-limiting example, internal storage, attached storage, and / or network-accessible storage.

[0078] System 1000 includes, for example, an encoder / decoder module 1030 configured to process data to provide encoded or decoded video, which may include its own processor and memory. Encoder / decoder module 1030 represents a module that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding and a decoding module. Additionally, encoder / decoder module 1030 may be implemented as a separate element of system 1000 or may be incorporated within processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0079] Program code loaded into the processor 1010 or the encoder / decoder 1030 to implement various aspects described herein may be stored in the storage device 1040 and then loaded into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items while performing the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and arithmetic logic.

[0080] In some embodiments, memory internal to the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage 1040, such as dynamic volatile memory and / or non-volatile flash memory. In some embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, high-speed external dynamic volatile memory, such as RAM, is used as working memory for video coding and decoding operations, such as MPEG-2, HEVC, or VVC (Versatile Video Coding).

[0081] Input to the elements of system 1000 may be provided through various input devices as shown in block 1130. Such input devices include, but are not limited to, (i) an RF section for receiving RF signals transmitted over the air by, for example, a broadcast station, (ii) a composite input, (iii) a USB input, and / or (iv) an HDMI input.

[0082] In various embodiments, the input devices of block 1130 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements necessary to (i) select a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) downconvert the selected signal, (iii) band-limit again to a narrower frequency band to select a signal frequency band, which may be referred to as a channel in a particular embodiment (for example), (iv) demodulate the downconverted, band-limited signal, (v) perform error correction, and (vi) demultiplex to select a desired stream of data packets. The RF section of various embodiments includes one or more elements for performing these functions, such as, for example, a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include, for example, a tuner to perform these various functions, including downconverting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency near baseband) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0083] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 1000 to other electronic devices over USB and / or HDMI connections. It should be understood that various aspects of the input processing, such as Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or processor 1010, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, as desired, in a separate interface IC or processor 1010. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 1010 and encoder / decoder 1030, operating in conjunction with memory and storage to process the data stream as desired for presentation on an output device.

[0084] The various elements of system 1000 may be provided within an integrated housing in which the various elements may be interconnected and transmit data therebetween using an internal bus 1140 known in the art, including suitable connection arrangements such as an I2C bus, wiring, and printed circuit boards.

[0085] System 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. Communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 1060. Communication interface 1050 may include, but is not limited to, a modem or a network card, and communication channel 1060 may be implemented within a wired and / or wireless medium, for example.

[0086] Data is streamed to system 1000 using a Wi-Fi network, such as IEEE 802.11, in various embodiments. The Wi-Fi signal in these embodiments is received over communication channel 1060 and communication interface 1050, which are adapted for Wi-Fi communication. Communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. Other embodiments provide streaming data to system 1000 using a set-top box that delivers data over the HDMI connection of input block 1130. Still other embodiments provide streaming data to system 1000 using the RF connection of input block 1130.

[0087] System 1000 can provide output signals to various output devices, including display 1100, speakers 1110, and other peripheral devices 1120. Other peripheral devices 1120, in various example embodiments, include one or more of a standalone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 1000. In various embodiments, control signals are communicated between system 1000 and display 1100, speakers 1110, or other peripheral devices 1120 using signaling such as AVLink, CEC, or other communication protocols that enable inter-device control with or without user intervention. Output devices may be communicatively coupled to system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to system 1000 using communication channel 1060 via communication interface 1050. The display 1100 and speakers 1110 may be integrated into a single unit along with other components of the system 1000, for example, in an electronic device such as a television. In various embodiments, the display interface 1070 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0088] The display 1100 and speakers 1110 may alternatively be separate from one or more of the other components, for example, if the RF portion of the input 1130 is part of a separate set-top box. In various embodiments in which the display 1100 and speakers 1110 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output. Implementations described herein may be implemented, for example, as a method or process, an apparatus, a software program, a data stream, or a signal. Even when discussed only in connection with a single implementation (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented, for example, in appropriate hardware, software, and firmware. A method may be implemented in an apparatus, such as a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end users.

[0089] FIG. 11 illustrates a process for an image encoder according to at least one embodiment. Process 1101 may be performed, for example, by device 1000 of FIG. 10, or more specifically, by processor 1010 or encoder 1030 of such a device. In step 1111, the processor obtains input data. In step 1112, the processor selects a subset of parameters based on the input data, the subset of parameters being used to fine-tune a first neural network model. In step 1113, the processor determines parameter updates for the selected subset of parameters based on a loss function. In step 1114, the processor packages the input data and information representing the parameter updates.

[0090] FIG. 12 illustrates an image decoder process according to at least one embodiment. Process 1201 may be performed, for example, by device 1000 of FIG. 10, or more specifically, by processor 1010 or decoder 1030 of such a device. In step 1211, the processor obtains input data and information representing parameter updates for a selected subset of parameters. In step 1212, the processor selects a subset of parameters based on the input data, which subset of parameters is used to fine-tune a model of a first neural network. In step 1213, the processor updates the model of the first neural network based on the parameter updates for the selected subset of parameters. In step 1214, the processor determines output data by processing the input data with the updated first neural network.

[0091] References to "one embodiment" or "embodiment" or "one implementation" or "implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment. Thus, appearances of "one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations thereof in various places throughout this specification are not necessarily all referring to the same embodiment.

[0092] Additionally, this application or its claims may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0093] Additionally, this application or its claims may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, predicting information, or estimating information.

[0094] Additionally, the application or claims thereof may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from memory or optical media storage). Furthermore, "receiving" typically involves, in operation, various ways, such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0095] It should be understood that the use of any of " / ", "and / or", "at least one of" is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B), for example, in the case of "A / B", "A and / or B", "at least one of A and B". As a further example, in the case of "A, B and / or C" and "at least one of A, B and C", such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, and B, and C). This may be expanded as many times as the number of items listed, as would be readily apparent to one of ordinary skill in this and related arts.

[0096] As will be apparent to one skilled in the art, implementations may generate a variety of signals formatted to carry information that may be stored or transmitted, for example. Information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

Claims

1. Obtaining input data; selecting a subset of parameters for fine-tuning a model of a first neural network based on the input data; determining parameter updates for the selected subset of parameters based on a loss function; packaging input data and information representing said parameter updates; A method comprising:

2. The method of claim 1 , wherein determining the parameter updates is further based on a target output.

3. The method of claim 1 or 2, wherein selecting the subset of parameters is based on the gradient of the model's output to select the parameters with the greatest influence.

4. The method of claim 1 or 2, wherein the selecting the subset of parameters is based on a loss function using a no-reference metric determined based on the input data.

5. 5. The method according to claim 1, wherein the parameters are selected from a set comprising biases of the model, weights of the model, parameters of nonlinear functions of the model, a subset of layers of the model, a specific layer of the model, the biases of a specific layer of the model, and a subset of neurons of the model.

6. 1. A method adapted to encode an image, wherein the first neural network is used for decoding and the second neural network is used for encoding, the method comprising: - determining an embedding by encoding the image using the second neural network; quantizing said embedding; - performing said selection of a subset of parameters based on said quantized embedding; 6. The method of claim 1, further comprising: updating the first neural network using parameter updates for the selected subset of parameters.

7. The method of claim 6 , wherein determining parameter updates is further based on the image.

8. and a method adapted to compress sound, wherein the input data is an audio signal, the method comprising: compressing the audio signal; decompressing the compressed audio signal; performing said selection of a subset of parameters based on said restored compressed audio signal; packaging the compressed audio signal and parameter updates; The method of any one of claims 1 to 5, further comprising:

9. obtaining input data and information representing updates to a selected subset of parameters; selecting a subset of parameters for fine-tuning a model of a first neural network based on the input data; updating the model of the first neural network based on the updates to the selected subset of parameters; determining output data by processing the input data with the updated first neural network; and A method comprising:

10. 10. The method of claim 9 adapted to decode an image, wherein the input data is a quantized embedding.

11. 10. The method of claim 9, adapted to decode sound, wherein the selection of the subset of parameters is further based on a obtained reconstructed audio signal.

12. The method according to any one of claims 1 to 11, wherein the selection of the subset of parameters is independent of information representative of the parameter update.

13. 1. A device including a processor, the processor comprising: Obtaining input data; selecting a subset of parameters for fine-tuning a model of a first neural network based on the input data; determining parameter updates for the selected subset of parameters based on a loss function; packaging input data and information representing said parameter updates; A device that is configured to:

14. The device of claim 13 , wherein determining the parameter updates is further based on a target output.

15. 15. The device of claim 13 or 14, wherein selecting the subset of parameters is based on a gradient of the output of the model, in order to select the parameters that have the greatest influence.

16. The device of claim 13 or 14, wherein selecting the subset of parameters is based on a loss function using a no-reference metric determined based on the input data.

17. 17. The device of claim 13, wherein the parameters are selected from a set comprising biases of the model, weights of the model, parameters of nonlinear functions of the model, a subset of layers of the model, a specific layer of the model, the biases of a specific layer of the model, and a subset of neurons of the model.

18. and a second neural network adapted to encode an image, the first neural network being used for decoding and the second neural network being used for encoding, the processor comprising: encoding the image using the second neural network to determine an embedding; quantizing the embedding; and performing said selection of a subset of parameters based on said quantized embedding; wherein the first neural network is updated using parameter updates for the selected subset of parameters.

19. The device of claim 18 , wherein determining parameter updates is further based on the image.

20. and a processor adapted to compress sound, the input data being an audio signal, the processor comprising: compressing the audio signal; decompressing the compressed audio signal; performing said selection of a subset of parameters based on said restored compressed audio signal; packaging the compressed audio signal and parameter updates; The device of any one of claims 13 to 17, further configured to:

21. 1. A device including a processor, the processor comprising: obtaining input data and information representing parameter updates for a selected subset of parameters; selecting a subset of parameters for fine-tuning a model of a first neural network based on the input data; updating the model of the first neural network based on parameter updates for the selected subset of parameters; determining output data by processing the input data with the updated first neural network; and A device that is configured to:

22. 22. The device of claim 21, adapted to decode an image, wherein the input data is a quantized embedding.

23. 22. The device of claim 21, adapted to decode sound, wherein the selection of the subset of parameters is further based on a obtained reconstructed audio signal.

24. A device according to any one of claims 13 to 23, wherein the information representative of the parameter update does not include information representative of the selection of the subset of parameters.

25. A computer program comprising program code instructions for performing the method according to any one of claims 1 to 12 when the computer program is executed by a processor.

26. A non-transitory computer readable medium comprising program code instructions for performing the method of any one of claims 1 to 12 when executed by a processor.

Images