Image processing method, device and storage medium

Abstract

Processing circuitry decodes an index from an encoded bitstream carrying compressed images, wherein the index points to a value in a set of values for a parameter, a change in the value of the parameter is used to adjust a compression rate of the compressed images, the compressed images are generated from the parameter by a neural network based encoder. The processing circuitry inputs the value of the parameter into a multi-rate compressed domain computer vision task decoder, the multi-rate compressed domain computer vision task decoder comprises one or more neural networks for performing a computer vision task from the compressed images according to respective values of the parameter used to generate the respective compressed images. The multi-rate compressed domain computer vision task decoder generates a computer vision task result from the compressed images in the encoded bitstream and the value of the parameter.

Description

[0001] Cross-reference of related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 124,828, filed March 22, 2023, entitled “MULTI-RATE OF COMPUTER VISIONTASK NEURAL NETWORKS IN COMPRESSION DOMAIN,” which claims priority to U.S. Provisional Application No. 63 / 331,168, filed April 14, 2022, entitled “Multi-rate of ComputerVision Task Neural Networks in Compression Domain.” The entire contents of the earlier application are incorporated herein by reference. Technical Field

[0003] This disclosure describes embodiments that are generally related to image / video processing. Background Technology

[0004] The background description provided herein is for the purpose of presenting the general context of this disclosure. Within the scope described in this background section, neither the work of the currently named inventor nor any aspect of this description that does not qualify as prior art at the time of submission is expressly or impliedly acknowledged as prior art to this disclosure.

[0005] Image / video compression facilitates the transfer of image / video files across different devices, storage systems, and networks with minimal quality degradation. Improving image / video compression tools can require significant expertise, effort, and time. Machine learning techniques can be applied to image / video compression to simplify and accelerate the improvement of compression tools. Summary of the Invention

[0006] This disclosure provides methods and apparatus for image / video processing (e.g., encoding and decoding). In some examples, an image processing apparatus includes processing circuitry. The processing circuitry decodes an index based on an encoded bitstream carrying a compressed image, wherein the index points to a value in a set of parameter values, the changes in which the parameter values ​​are used to adjust the compression ratio of the compressed image, and generates the compressed image using a neural network-based encoder based on the parameters. The processing circuitry inputs the parameter values ​​into a multi-rate compression domain computer vision task decoder in the compression domain computer vision task framework CDCVTF, the multi-rate compression domain computer vision task decoder including one or more neural networks for performing computer vision tasks from multiple compressed images based on corresponding values ​​of the parameters, the corresponding values ​​of which are used to generate multiple compressed images. The multi-rate compression domain computer vision task decoder generates computer vision task results based on the compressed images in the encoded bitstream, wherein the compressed images in the encoded bitstream are compressed at corresponding compression ratios from multiple different compression ratios based on the parameter values.

[0007] In some examples, a first neural network in a multi-rate compression domain computer vision task decoder converts the values ​​of parameters into tensors; the tensors are input into one or more layers in a second neural network in the multi-rate compression domain computer vision task decoder; and the second neural network generates computer vision task results based on the compressed image and the tensors.

[0008] In some examples, the first neural network includes one or more convolutional layers.

[0009] In some examples, the first neural network includes convolutional layers with activation functions.

[0010] In some examples, the second neural network is configured to generate results for computer vision tasks without generating reconstructed images from compressed images.

[0011] In some examples, the second neural network is configured to generate a reconstructed image from the compressed image, and then generate computer vision task results from the reconstructed image.

[0012] In some examples, the neural network-based encoder is an encoder model based on the Neural Image Compression (NIC) framework; the multi-rate compression domain computer vision task decoder is a decoder model based on the NIC framework; the NIC framework is trained end-to-end.

[0013] In some examples, the decoder model of the multi-rate compression domain computer vision task decoder is trained separately from the encoder model of the neural network-based encoder.

[0014] In some examples, the parameters are hyperparameters used to weight the distortion when calculating the rate distortion loss.

[0015] In some examples, computer vision tasks include at least one of the following: image classification, image denoising, object detection, and super-resolution.

[0016] This disclosure also provides a non-transitory computer-readable storage medium for storing a program executed by at least one processor to perform the encoding and / or decoding methods for the images / videos described above. Attached Figure Description

[0017] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which: Figure 1 The following examples illustrate neural image compression (NIC) frameworks; Figure 2 Examples of main encoder networks are shown in some examples; Figure 3 Examples of master decoder networks are shown in some examples; Figure 4 Examples of super encoder networks are shown in some examples; Figure 5 Examples of superdecoder networks are shown in some examples; Figure 6 Examples of context model neural networks are shown in some examples; Figure 7 Examples of entropy parameter neural networks are shown in some examples; Figure 8 Image encoders are shown in some examples; Figure 9 The image decoders shown are some examples; Figure 10 and Figure 11 The image encoder and corresponding image decoder are shown in some examples; Figure 12 Examples of systems for performing computer vision (CV) tasks in a compressed domain are shown; Figure 13 Examples of systems for performing CV tasks in a compressed domain are shown; Figure 14 Examples of systems for performing CV tasks in a multi-rate compression domain are shown; Figure 15Examples of systems for performing CV tasks in a multi-rate compression domain are shown; Figure 16 A flowchart outlining the process in some examples is shown; Figure 17 A flowchart outlining the process in some examples is shown; Figure 18 These are schematic diagrams of computer systems in some examples. Detailed Implementation

[0018] According to one aspect of this disclosure, some video codecs may be difficult to optimize as a whole. For example, improvements to individual modules (e.g., encoders) in a video codec may not result in coding gains for the overall performance. In contrast, in a video / image coding framework based on an Artificial Neural Network (ANN), a machine learning process can be performed; then, different modules of the ANN-based video / image coding framework can be jointly optimized from input to output to improve the final objective (e.g., rate-distortion performance, such as the rate-distortion loss L described in this disclosure). For example, a learning or training process (e.g., a machine learning process) can be performed on an ANN-based video / image coding framework to jointly optimize the modules of the ANN-based video / image coding framework, thereby achieving overall optimized rate-distortion performance. Thus, the optimization result can be an end-to-end (E2E) optimized neural image compression (NIC).

[0019] The following description uses the NIC framework to illustrate an ANN-based video / image coding framework. While image compression (e.g., encoding and decoding) is described below, it should be noted that image compression techniques can be appropriately applied to video compression.

[0020] According to some aspects of this disclosure, the NIC framework can be trained during offline training and / or online training. During offline training, a previously collected set of training images can be used to train and optimize the NIC framework. In some examples, the parameters of the NIC framework determined through the offline training process can be referred to as pre-trained parameters, and the NIC framework with pre-trained parameters can be referred to as a pre-trained NIC framework. The pre-trained NIC framework can be used for image compression operations.

[0021] In some examples, when one or more images (also referred to as one or more target images) are available for image compression operations, a pre-trained NIC framework can be further trained based on the one or more target images during online training to adjust the parameters of the NIC framework. The parameters of the NIC framework adjusted through the online training process can be referred to as online training parameters, and the NIC framework with online training parameters can be referred to as the online-trained NIC framework. The online-trained NIC framework can then perform image compression operations on the one or more target images. Some aspects of this disclosure provide techniques for adjusting an encoder based on online training in neural image compression.

[0022] A neural network is a computational architecture that simulates the biological brain. Neural networks can be models implemented in software or hardware that mimic the computational capabilities of biological systems by connecting a large number of artificial neurons with interconnects. These artificial neurons, called nodes, are interconnected and work together to process input data. Neural networks (NNs) are also known as artificial neural networks (ANNs).

[0023] Nodes in an ANN can be organized into any suitable architecture. In some embodiments, nodes in an ANN are organized into layers, including an input layer that receives input signals into the ANN and an output layer that outputs output signals from the ANN. In one embodiment, the ANN also includes layers between the input and output layers, which may be referred to as hidden layers. Different layers can perform different types of transformations on their respective inputs. Signals can be transmitted from the input layer to the output layer.

[0024] An ANN with multiple layers between the input and output layers can be called a Deep Neural Network (DNN). A DNN can have any suitable structure. In some examples, a DNN is configured in a feedforward network structure, where data flows from the input layer to the output layer without loops. In some examples, a DNN is configured in a fully connected network structure, where each node in one layer is connected to all nodes in the next layer. In some examples, a DNN is configured as a Recurrent Neural Network (RNN) structure, where data can flow in any direction.

[0025] An ANN having at least one convolutional layer for performing convolution operations can be called a Convolutional Neural Network (CNN). A CNN may include an input layer, an output layer, and hidden layers between the input and output layers. Hidden layers may include convolutional layers for performing convolutions (such as two-dimensional (2D) convolutions, as used in encoders). In one embodiment, the 2D convolution performed in the convolutional layer lies between the convolutional kernel (also called a filter or channel, e.g., a 5×5 matrix) and the input signal (e.g., a 2D matrix, such as a 2D block, a 256×256 matrix) fed into the convolutional layer. The dimension of the convolutional kernel (e.g., 5×5) is smaller than the dimension of the input signal (e.g., 256×256). During the convolution operation, a dot product operation is performed on a slice (e.g., a 5×5 region) of the same size as the convolutional kernel in the input signal (e.g., a 256×256 matrix) to generate an output signal for input to the next layer. A patch (e.g., a 5×5 region) in the input signal (e.g., a 256×256 matrix) that is the same size as the convolution kernel can be referred to as the receptive field of each node in the next layer.

[0026] During convolution, the dot product between the convolution kernel and the corresponding receptive field in the input signal is calculated. The convolution kernel includes weights as elements, each element of which is a weight applied to the corresponding sample in the receptive field. For example, a convolution kernel represented by a 5×5 matrix has 25 weights. In some examples, a bias is applied to the output signal of the convolutional layer, and the output signal is based on the sum of the dot product and the bias.

[0027] In some examples, the convolutional kernel can move along the input signal (e.g., a 2D matrix) by a size called stride; thus, the convolution operation generates a feature map or activation map (e.g., another 2D matrix), which in turn provides input to the next layer in the CNN. For example, if the input signal is a 2D block with 256×256 samples, then the stride is 2 samples (e.g., stride 2). In the case of stride 2, the convolutional kernel moves 2 samples along the X direction (e.g., horizontal direction) and / or the Y direction (e.g., vertical direction).

[0028] In some examples, multiple convolutional kernels can be applied to the input signal within the same convolutional layer to generate multiple feature maps, each representing a specific feature of the input signal. In some examples, the convolutional kernels can correspond to the feature maps. A convolutional layer with N kernels (or N channels) (each kernel having M×M samples and a stride S) can be specified as: Conv: M×M cN sS. For example, a convolutional layer with 192 kernels (or 192 channels) (each kernel having 5×5 samples and a stride of 2) can be specified as: Conv: 5×5 c192 s2. Hidden layers can include deconvolutional layers that perform deconvolution (such as 2D deconvolution) (e.g., used in a decoder). Deconvolution is the inverse process of convolution. A deconvolutional layer with 192 deconvolution kernels (or 192 channels) (each deconvolution kernel has 5×5 samples and a stride of 2) can be specified as: DeConv: 5×5 c192 s2.

[0029] In CNNs, a relatively large number of nodes can share the same filters (e.g., the same weights) and the same biases (if biases are used), thus reducing memory footprint because a single bias and a single weight vector can be used across all receptive fields sharing the same filters. For example, for an input signal with 100×100 samples, a convolutional layer with kernels of 5×5 samples has 25 learnable parameters (e.g., weights). If biases are used, one channel uses 26 learnable parameters (e.g., 25 weights and one bias). If the convolutional layer has N kernels, the total number of learnable parameters is 26×N. This is relatively small compared to fully connected feedforward neural network layers. For example, for a fully connected feedforward layer, 100×100 (i.e., 10000) weights are used to generate the resulting signal to be used as input to each node in the next layer. If the next layer has L nodes, the total number of learnable parameters is 10000×L.

[0030] CNNs can also include one or more other layers, such as pooling layers, fully connected layers that connect each node in one layer to each node in another layer, normalization layers, etc. The layers in a CNN can be arranged in any suitable order and with any suitable architecture (e.g., feedforward architecture, recurrent architecture). In one example, convolutional layers are followed by other layers, such as pooling layers, fully connected layers, normalization layers, etc.

[0031] Pooling layers can be used to reduce the dimensionality of data by combining the outputs of multiple nodes in one layer into a single node in the next layer. The pooling operation for a pooling layer with a feature map as input is described below. This description can be appropriately applied to other input signals. The feature map can be divided into sub-regions (e.g., rectangular sub-regions), and features in each sub-region can be independently downsampled (or pooled) to a single value, for example, by averaging in average pooling or maximizing in max pooling.

[0032] Pooling layers can perform pooling operations such as local pooling, global pooling, max pooling, and average pooling. Pooling is a form of non-linear downsampling. Local pooling combines a small number of nodes in the feature map (e.g., a local cluster of 2×2 nodes). Global pooling combines all nodes (e.g., all nodes in the feature map).

[0033] Pooling layers can reduce the size of the representation, thereby reducing the number of parameters, memory usage, and computational cost in a CNN. In one example, pooling layers are inserted between consecutive convolutional layers in a CNN. In another example, the pooling layer is followed by an activation function, such as a Rectified Linear Unit (ReLU) layer. In yet another example, pooling layers are omitted between consecutive convolutional layers in a CNN.

[0034] Normalization layers can be ReLU, leaked ReLU, Generalized Divisive Normalization (GDN), Inverse Generalized Divisive Normalization (IGDN), etc. ReLU can apply a non-saturating activation function, removing negative values ​​from the input signal (e.g., feature maps) by setting negative values ​​to zero. For negative values, leaked ReLU can have a smaller slope (e.g., 0.01) instead of a flat slope (e.g., 0). Therefore, if the value x is greater than 0, the output from leaked ReLU is x. Otherwise, the output from leaked ReLU is the value x multiplied by a smaller slope (e.g., 0.01). In one example, the slope is determined before training and therefore will not be learned during training.

[0035] The NIC framework can correspond to a compression model used for image compression. The NIC framework takes an input image x and outputs a reconstructed image corresponding to the input image x. The NIC framework may include a neural network encoder (e.g., an encoder based on a neural network such as a DNN) and a neural network decoder (e.g., a decoder based on a neural network such as a DNN). The input image x is provided as input to the neural network encoder to compute a compressed representation (e.g., a compact representation). This compression representation can be compact, for example, for storage and transmission purposes. Compression Representation It is provided as input to the neural network decoder to generate the reconstructed image. In various embodiments, the input image x and the reconstructed image Located in a spatial domain and represented in a compressed manner It resides in a domain different from this spatial domain. In some examples, this is for compressed representations. Perform quantization and entropy encoding.

[0036] In some examples, the NIC framework can use a Variational Autoencoder (VAE) structure. In a VAE structure, the entire input image x can be fed into a neural network encoder. The entire input image x is processed by a set of neural network layers (part of the neural network encoder), which act as a black box to compute a compressed representation. Compressed representation This is the output of the neural network encoder. The neural network decoder can compress the entire representation. As input. Compressed representation. Another set of neural network layers (belonging to the neural network decoder) can be used as another black box to compute the reconstructed image. The rate-distortion (RD) loss L( ) to optimize in order to achieve reconstructed image Distortion loss ) and compact representation with equilibrium hyperparameter λ The balance between bit consumption R can be optimized, for example, according to Equation 1: L ( ) = ) + R ( ) Equation 1 Neural networks (e.g., ANNs) can learn to perform tasks from examples without task-specific programming. An ANN can be configured with connected nodes or artificial neurons. Connections between nodes can transmit signals from a first node to a second node (e.g., a receiving node), and the signals can be modified by weights, indicated by the weight coefficients used for the connection. A receiving node can process signals from the nodes that transmitted signals to it (i.e., the input signals to the receiving node) and then generate an output signal by applying a function to the input signals. This function can be a linear function. In one example, the output signal is a weighted sum of the input signals. In another example, the output signal can be further modified by biases, represented by a bias term, so the output signal is the sum of the biases and the weighted sum of the input signals. The function can include nonlinear operations, such as a nonlinear operation on the weighted sum of the input signals or the biases and the sum of the weighted sums. The output signal can be sent to a node (a downstream node connected to the receiving node). An ANN can be represented or configured by parameters (e.g., the weights and / or biases of the connections). Weights and / or biases can be obtained by training an ANN (e.g., offline training, online training, etc.), where the weights and / or biases can be iteratively adjusted. A trained ANN configured with defined weights and / or defined biases can be used to perform tasks.

[0037] Figure 1 Examples of NIC frameworks (100) (e.g., NIC systems) are shown. NIC frameworks (100) may be based on neural networks, such as DNNs and / or CNNs. NIC frameworks (100) may be used to compress (e.g., encode) multiple images and decompress (e.g., decode or reconstruct) multiple compressed images (e.g., encoded images).

[0038] Specifically, in Figure 1 In the example, the compression model in the NIC framework (100) comprises two levels, referred to as the master level and superlevel of the compression model. The master and superlevels of the compression model can be implemented using neural networks. Figure 1 In the diagram, the main-level neural network used for the compression model is shown as the first sub-neural network (NN) (151), and the super-level neural network of the compression model is shown as the second sub-NN (152).

[0039] The first sub-NN (151) can be analogous to an autoencoder and can be trained to generate a compressed image of the input image x. And compressed images (i.e., encoded images). Decompress to obtain the reconstructed image The first sub-NN (151) may include multiple components (or modules), such as the master encoder neural network (or master encoder network) (111), quantizer (112), entropy encoder (113), entropy decoder (114) and master decoder neural network (or master decoder network) (115).

[0040] refer to Figure 1 The master encoder network (111) can generate latent values ​​or latent value representations y based on the input image x (e.g., an image to be compressed or encoded). In one example, the master encoder network (111) is implemented using a CNN. The relationship between the latent value representation y and the input image x can be described by Equation 2.

[0041] Equation 2 Among them, parameters The parameters represent, for example, the weights and biases used in the convolution kernels in the main encoder network (111) (if biases are used in the main encoder network (111)).

[0042] The latent value representation y can be quantized using a quantizer (112) to generate quantized latent values. It can quantify potential values. Compression can be performed, for example, by using lossless compression via an entropy encoder (113) to generate a compressed image (e.g., an encoded image). (131), the compressed image It is a compressed representation of the input image x. The entropy encoder (113) can use entropy coding techniques such as Huffman coding and arithmetic coding. In one example, the entropy encoder (113) uses arithmetic coding and is an arithmetic encoder. In one example, the encoded image (131) is transmitted as an encoded bitstream.

[0043] The encoded image (131) can be decompressed (e.g., entropy decoding) by an entropy decoder (114) to generate an output. The entropy decoder (114) can use an entropy coding technique corresponding to the entropy coding technique used in the entropy encoder (113), such as Huffman coding, arithmetic coding, etc. In one example, the entropy decoder (114) uses arithmetic decoding, and it is an arithmetic decoder. In one example, lossless compression is used in the entropy encoder (113), lossless decompression is used in the entropy decoder (114), and noise such as that generated due to the transmission of the encoded image (131) can be ignored, and the output from the entropy decoder (114) is the quantized latent value. .

[0044] The master decoder network (115) can process the quantized latent values Decode the image to generate a reconstructed image. In one example, a CNN is used to implement the main decoder network (115). Image reconstruction. (i.e., the output of the master decoder network (115)) and the quantized latent value The relationship between (i.e., the inputs of the main decoder network (115)) can be described by Equation 3.

[0045] Equation 3 Among them, parameters This represents parameters such as the weights and biases used in the convolutional kernels in the main decoder network (115) (if biases are used in the main decoder network (115)). Therefore, the first sub-NN (151) can compress (e.g., encode) the input image x to obtain an encoded image (131), and can decompress (e.g., decode) the encoded image (131) to obtain a reconstructed image. Because the quantizer (112) introduces quantization loss, the reconstructed image... It may be different from the input image x.

[0046] In some examples, the second sub-NN (152) can be used in the quantization of the latent values ​​for entropy encoding. The entropy model is learned from the prior probability model. Therefore, the entropy model can be a conditional entropy model, such as a Gaussian Mixture Model (GMM) or a Gaussian Scale Model (GSM) that depends on the input image x.

[0047] In some examples, the second sub-NN (152) may include a context model NN (116), an entropy parameter NN (117), a super encoder network (121), a quantizer (122), an entropy encoder (123), an entropy decoder (124), and a super decoder network (125). The entropy model used in the context model NN (116) may be a latent value (e.g., a quantized latent value). An autoregressive model on a super-encoder network (121), a quantizer (122), an entropy encoder (123), an entropy decoder (124), and a super-decoder network (125) form a super-prior model, which can be implemented using a neural network in the super-level (e.g., a super-prior NN). The super-prior model can represent information useful for correcting context-based predictions. Data from the context model NN (116) and the super-prior model can be combined using the entropy parameter NN (117). The entropy parameter NN (117) can generate parameters, such as the mean and scale parameters of an entropy model like a conditional Gaussian entropy model (e.g., a GMM).

[0048] refer to Figure 1 On the encoder side, the potential value quantized from the quantizer (112) is... The latent values ​​from the entropy decoder (114) are fed into the context model NN (116). On the decoder side, the quantized latent values ​​are... The information is fed into the context model NN (116). The context model NN (116) can be implemented using a neural network such as a CNN. The context model NN (116) can be based on context. Generate output The context These are latent values ​​that can be quantized for the contextual model NN(116). Context This can include: the previously quantized latent value on the encoder side, or the previously entropy-decoded quantized latent value on the decoder side. The output of the context model NN (116) can be described using Equation 4. and input (e.g.) The relationship between ).

[0049] Equation 4 Among them, parameters The parameters represent the weights and biases used in the convolution kernel, for example, in the context model NN (116) (if biases are used in the context model NN (116)).

[0050] The output from the context model NN (116) and the output from the super decoder network (125) Feeding the entropy parameter NN(117) to generate the output The entropy parameter NN(117) can be implemented using a neural network such as a CNN. The output of the entropy parameter NN(117) can be described using Equation 5. and input (e.g.) and The relationship between ).

[0051] Equation 5 Among them, parameters This represents the parameters, for example, the weights and biases used in the convolution kernel of entropy parameter NN(117) (if biases are used in entropy parameter NN(117)). The output of entropy parameter NN(117) This can be used to determine (e.g., constrain) entropy models, so that conditional entropy models can depend on the input image x, for example, through the output from the superdecoder network (125). In one example, the output is... This includes parameters used in constrained entropy models (e.g., GMMs), such as the mean and scale parameters. (See reference.) Figure 1 The entropy encoder (113) and entropy decoder (114) can use an entropy model (e.g., a conditional entropy model) in entropy encoding and entropy decoding, respectively.

[0052] The second sub-NN (152) can be described below. The latent value y can be fed into the superencoder network (121) to produce the super-latest value z. In one example, a neural network such as a CNN is used to implement the superencoder network (121). The relationship between the super-latest value z and the latent value y can be described using Equation 6.

[0053] Equation 6 Among them, parameters The parameters represent the weights and biases used in the convolution kernels, for example, in the super encoder network (121) if biases are used in the super encoder network (121).

[0054] The over-latency value z is quantized using quantizer (122) to produce the quantized latent value. It can quantify potential values. Compression is performed, for example, by using lossless compression with an entropy encoder (123) to generate side information, such as coded bits (132) from a hyperneural network. The entropy encoder (123) can use entropy coding techniques such as Huffman coding or arithmetic coding. In one example, the entropy encoder (123) uses arithmetic coding and is an arithmetic encoder. In one example, side information such as coded bits (132) can be transmitted in the encoded bitstream, for example, along with the encoded image (131).

[0055] The side information, such as the encoded bits (132), can be decompressed (e.g., entropy decoding) by the entropy decoder (124) to generate the output. The entropy decoder (124) can use entropy coding techniques such as Huffman coding and arithmetic coding. In the example, the entropy decoder (124) uses arithmetic decoding and is an arithmetic decoder. In one example, lossless compression is used in the entropy encoder (123), lossless decompression is used in the entropy decoder (124), and noise such as that caused by the transmission of side information is negligible, and the output from the entropy decoder (124) can be a quantized latent value. The super decoder network (125) can process the quantized latent values. Decode to generate output The output can be described using Equation 7. and quantified potential value The relationship between them.

[0056] Equation 7 Among them, parameters The parameters represent the weights and biases used in the convolution kernels, for example, in the superdecoder network (125) if biases are used in the superdecoder network (125).

[0057] As described above, compressed or encoded bits (132) can be added as side information to the encoded bitstream, enabling the entropy decoder (114) to use a conditional entropy model. Therefore, the entropy model can be image-dependent and spatially adaptive, thus being more accurate than a fixed entropy model.

[0058] For example, the NIC framework (100) can be appropriately adjusted to omit... Figure 1 One or more components shown, modified Figure 1 One or more components shown, and / or including Figure 1 One or more components not shown in the diagram. In one example, a NIC framework using a fixed entropy model may include a first sub-NN (151) but not a second sub-NN (152). In one example, a NIC framework may include components in a NIC framework (100) other than the entropy encoder (123) and the entropy decoder (124).

[0059] In one embodiment, a neural network (such as a CNN) can be used to achieve this. Figure 1 One or more components in the NIC framework (100) shown. Each NN-based component in the NIC framework (e.g., the NIC framework (100)) (e.g., the master encoder network (111), the master decoder network (115), the context model NN (116), the entropy parameter NN (117), the super encoder network (121), or the super decoder network (125)) may include any suitable architecture (e.g., with any suitable combination of layers), any suitable type of parameters (e.g., weights, biases, combinations of weights and biases and / or the like), and any suitable number of parameters.

[0060] In one embodiment, the main encoder network (111), the main decoder network (115), the context model NN (116), the entropy parameter NN (117), the super encoder network (121), and the super decoder network (125) are implemented using the corresponding CNN.

[0061] Figure 2 An exemplary CNN for a master encoder network (111) according to an embodiment of this disclosure is shown. For example, the master encoder network (111) includes four sets of layers, each set comprising a 5×5 c192 s2 convolutional layer followed by a GDN layer. Modifications and / or omissions are possible. Figure 2One or more layers are shown. Additional layers can be added to the main encoder network (111).

[0062] Figure 3 An exemplary CNN for a master decoder network (115) according to an embodiment of this disclosure is shown. For example, the master decoder network (115) includes three sets of layers, each set comprising a 5×5 c192 s2 deconvolutional layer followed by an IGDN layer. Furthermore, the three sets of layers are followed by a 5×5 c3 s2 deconvolutional layer followed by an IGDN layer. Modifications and / or omissions are possible. Figure 3 One or more layers are shown. Additional layers can be added to the main decoder network (115).

[0063] Figure 4 An exemplary CNN for a super encoder network (121) according to an embodiment of this disclosure is shown. For example, the super encoder network (121) includes: a 3×3 c192 s1 convolutional layer followed by a leaky ReLU, a 5×5 c192 s2 convolutional layer followed by a leaky ReLU, and another 5×5 c192 s2 convolutional layer. Modifications and / or omissions are possible. Figure 4 One or more layers are shown. Additional layers can be added to the super encoder network (121).

[0064] Figure 5 An exemplary CNN of a superdecoder network (125) according to an embodiment of this disclosure is shown. For example, the superdecoder network (125) includes: a 5×5 c192 s2 deconvolution layer followed by a leaked ReLU, a 5×5 c288 s2 deconvolution layer followed by a leaked ReLU, and a 3×3 c384 s1 deconvolution layer. Modifications and / or omissions are possible. Figure 5 One or more layers are shown. Additional layers can be added to the superdecoder network (125).

[0065] Figure 6 An exemplary CNN for a context model NN (116) according to an embodiment of this disclosure is shown. For example, the context model NN (116) includes a masked convolution of 5×5 c384 s1 for context prediction, thus the context in Equation 4 Includes limited context (e.g., 5×5 convolutional kernel). Can be modified. Figure 6 The convolutional layers in the context model NN (116) can be further expanded. Additional layers can be added to the context model NN (116).

[0066] Figure 7An exemplary CNN for an entropy parameter NN (117) according to an embodiment of this disclosure is shown. For example, the entropy parameter NN (117) includes: a 1×1 convolutional layer c640 s1 followed by a leaky ReLU, a 1×1 convolutional layer c512 s1 followed by a leaky ReLU, and a 1×1 convolutional layer c384 s1. Modifications and / or omissions are possible. Figure 7 One or more layers are shown. Additional layers can be added to the entropy parameter NN(117).

[0067] refer to Figures 2 to 7 The NIC framework (100) can be implemented using a CNN. The NIC framework (100) can be appropriately adapted so that one or more components of the NIC framework (100) (e.g., (111), (115), (116), (117), (121), and / or (125)) can be implemented using any suitable type of neural network (e.g., a CNN or a non-CNN-based neural network). One or more other components of the NIC framework (100) can be implemented using neural networks.

[0068] A NIC framework (100) can be trained to include neural networks (e.g., CNNs) to learn the parameters used in the neural networks. For example, when using a CNN, the parameters... These parameters represent, for example, the weights and biases used in the convolutional kernels of the main encoder network (111) (if biases are used in the main encoder network (111)), the weights and biases used in the convolutional kernels of the main decoder network (115) (if biases are used in the main decoder network (115)), the weights and biases used in the convolutional kernels of the super encoder network (121) (if biases are used in the super encoder network (121)), the weights and biases used in the convolutional kernels of the super decoder network (125) (if biases are used in the super decoder network (125)), the weights and biases used in the convolutional kernels of the context model NN (116) (if biases are used in the context model NN (116)), and the weights and biases used in the convolutional kernels of the entropy parameter NN (117) (if biases are used in the entropy parameter NN (117)). These parameters can be learned during the training process (e.g., offline training process, online training process, etc.).

[0069] In one example, reference Figure 2The master encoder network (111) comprises four convolutional layers, each with a 5×5 kernel and 192 channels. Therefore, the number of weights used in the convolutional kernels of the master encoder network (111) is 19200 (i.e., 4×5×5×192). The parameters used in the master encoder network (111) include 19200 weights and optional biases. Additional parameters may be included when biases and / or additional neural networks are used in the master encoder network (111).

[0070] refer to Figure 1 The NIC framework (100) includes at least one component or module built on a neural network. The at least one component may include one or more of the following: a master encoder network (111), a master decoder network (115), a super encoder network (121), a super decoder network (125), a context model NN (116), and an entropy parameter NN (117). At least one component can be trained individually. In one example, the training process is used to learn the parameters of each component separately. At least one component can be jointly trained as a group. In one example, the training process is used to jointly learn the parameters of a subset of at least one component. In one example, the training process is used to learn all the parameters of at least one component, hence the term E2E optimization.

[0071] During the training of one or more components in the NIC framework (100), the weights (or weight coefficients) of one or more components can be initialized. In one example, the weights are initialized based on the corresponding pre-trained neural network model (e.g., a DNN model, a CNN model). In another example, the weights are initialized by setting them to random numbers.

[0072] For example, after initializing the weights, a training image set can be used to train one or more components. The training image set can include any suitable image of any appropriate size. In some examples, the training image set includes raw images, natural images, computer-generated images, and / or similar images from the spatial domain. In some examples, the training image set includes images from residual images or residual images having residual data in the spatial domain. The residual data can be computed using a residual calculator. In some examples, the raw images and / or residual images including residual data can be directly used to train neural networks in a NIC framework (such as NIC framework (100)). Therefore, raw images, residual images, images from raw images, and / or images from residual images can be used to train neural networks in a NIC framework.

[0073] For the sake of brevity, training images are used below as examples to describe the training process (e.g., offline training, online training, etc.). This description can be appropriately applied to training blocks. Training images t from the training image set can be transmitted via... Figure 1 The encoding process in the code generates a compressed representation (e.g., encoded information placed in the bitstream). The encoded information can be transmitted via... Figure 1 The decoding process described in the document is used to calculate and reconstruct the reconstructed image. .

[0074] For the NIC framework (100), two competing objectives are balanced, such as reconstruction quality and bit consumption. The quality loss function (e.g., distortion or distortion loss) is used. () can be used to indicate reconstruction quality, such as the reconstructed object (e.g., reconstructed image). The rate (or rate loss) R can be used to indicate the bit consumption of the compressed representation. In one example, the rate loss R also includes, for example, side information used to determine the context model.

[0075] For neural image compression, the differentiable approximation of quantization can be used for E2E optimization. In various examples, noise injection is used to simulate quantization during the training process of neural network-based image compression, so quantization is simulated by noise injection rather than performed by a quantizer (e.g., quantizer (112)). Therefore, training with noise injection can variably approximate the quantization error. A bit-per-pixel (BPP) estimator can be used to simulate an entropy encoder, so entropy encoding is simulated by the BPP estimator rather than performed by an entropy encoder (e.g., (113)) and an entropy decoder (e.g., (114)). Therefore, during training, the rate loss R in the loss function L as shown in Equation 1 can be estimated, for example, based on noise injection and the BPP estimator. Generally, a higher rate R can allow for a lower distortion D, while a lower rate R may lead to a higher distortion D. Therefore, the joint RD loss L can be optimized using the equalization hyperparameter λ in Equation 1, where L can be optimized as a sum of λD and R. The training process can be used to tune the parameters of one or more components (e.g., (111), (115)) in the NIC framework (100) to minimize or optimize the joint rate-distortion (RD) loss L. In some examples, the joint rate-distortion (RD) loss can be optimized using the equalization hyperparameter λ, for example: L( ) = λ ) + R + Equation 8 Here, E measures the distortion of the decoded image residual relative to the original image residual before encoding, and serves as the regularization loss for the residual encoding / decoding DNN and the encoding / decoding DNN. β is a hyperparameter that balances the importance of the regularization loss.

[0076] Various models can be used to determine the distortion loss D and the rate loss R, thereby determining the joint RD loss L in Equation 1. In one example, the distortion loss... It is expressed as Peak Signal-to-Noise Ratio (PSNR), which is based on a metric of mean square error, a Multi-Scale Structural Similarity (MS-SSIM) quality index, a weighted combination of PSNR and MS-SSIM, etc.

[0077] In one example, the goal of the training process is to train an encoding neural network (e.g., an encoding DNN) and a decoding neural network (e.g., a decoding DNN), where the encoding neural network is, for example, a video encoder that will be used on the encoder side, and the decoding neural network is, for example, a video decoder that will be used on the decoder side. In one example, referencing... Figure 1 The encoding neural network may include: a master encoder network (111), a super encoder network (121), a super decoder network (125), a context model NN (116), and an entropy parameter NN (117). The decoding neural network may include: a master decoder network (115), a super decoder network (125), a context model NN (116), and an entropy parameter NN (117). The video encoder and / or video decoder may include other components based on NN and / or not based on NN.

[0078] The NIC framework can be trained in an E2E manner (e.g., NIC framework (100)). In one example, the encoding neural network and the decoding neural network are jointly updated in an E2E manner based on backpropagation gradients during training, for example using a gradient descent algorithm. The gradient descent algorithm can iteratively optimize the parameters of the NIC framework to find local minima of the differentiable function of the NIC framework (e.g., local minima of rate distortion loss). For example, the gradient descent algorithm can repeat steps in the opposite direction of the gradient (or approximate gradient) of the differentiable function at the current point.

[0079] After training the parameters of the neural network in the NIC framework (100), one or more components in the NIC framework (100) can be used to encode and / or decode the image. In one embodiment, on the encoder side, the image encoder is configured to encode the input image x into an encoded image (131) for transmission in the bitstream. The image encoder may include multiple components in the NIC framework (100). In one embodiment, on the decoder side, a corresponding image decoder is configured to decode the encoded image (131) carried in the bitstream into a reconstructed image. The image decoder may include multiple components in the NIC framework (100).

[0080] It should be noted that the image encoder and image decoder can have corresponding structures according to the NIC framework.

[0081] Figure 8 An exemplary image encoder (800) according to an embodiment of this disclosure is shown. The image encoder (800) includes: a main encoder network (811), a quantizer (812), an entropy encoder (813), and a second sub-NN (852). The main encoder network (811) is configured to resemble a main encoder network (111), the quantizer (812) is configured to resemble a quantizer (112), the entropy encoder (813) is configured to resemble an entropy encoder (113), and the second sub-NN (852) is configured to resemble a second sub-NN (152). Reference has been made above. Figure 1 A description was provided, but for clarity, it will be omitted here.

[0082] Figure 9 An exemplary image decoder (900) according to an embodiment of this disclosure is shown. The image decoder (900) may correspond to an image encoder (800). The image decoder (900) may include: a master decoder network (915), an entropy decoder (914), a context model NN (916), an entropy parameter NN (917), an entropy decoder (924), and a super decoder network (925). The master decoder network (915) is configured to resemble the master decoder network (115), the entropy decoder (914) is configured to resemble the entropy decoder (114), the context model NN (916) is configured to resemble the context model NN (116), the entropy parameter NN (917) is configured to resemble the entropy parameter NN (117), the entropy decoder (924) is configured to resemble the entropy decoder (124), and the super decoder network (925) is configured to resemble the super decoder network (125). The above has been referenced Figure 1 A description was provided, but for clarity, it will be omitted here.

[0083] refer to Figures 8 to 9On the encoder side, the image encoder (800) can generate an encoded image (831) and encoded bits (832) for transmission in the bitstream. On the decoder side, the image decoder (900) can receive the encoded image (931) and encoded bits (932) and decode them. The encoded image (931) and encoded bits (932) can be parsed from the received bitstream.

[0084] Figures 10 to 11 Exemplary image encoders (1000) and corresponding image decoders (1100) according to embodiments of this disclosure are shown respectively. Reference Figure 10 The image encoder (1000) includes a master encoder network (1011), a quantizer (1012), and an entropy encoder (1013). The master encoder network (1011) is configured similarly to the master encoder network (111), the quantizer (1012) is configured similarly to the quantizer (112), and the entropy encoder (1013) is configured similarly to the entropy encoder (113). The above has already referenced... Figure 1 A description was provided, but for clarity, it will be omitted here.

[0085] refer to Figure 11 The image decoder (1100) includes a main decoder network (1115) and an entropy decoder (1114). The main decoder network (1115) is configured similarly to the main decoder network (115), and the entropy decoder (1114) is configured similarly to the entropy decoder (114). (The above has been referenced...) Figure 1 A description was provided, but for clarity, it will be omitted here.

[0086] refer to Figure 10 and 11 The image encoder (1000) can generate an encoded image (1031) that will be included in the bitstream. The image decoder (1100) can receive the bitstream and decode the encoded image (1131) carried in the bitstream.

[0087] According to some aspects of this disclosure, image compression can remove redundancy from an image, thus allowing multiple compressed images to be represented using fewer bits. Image compression facilitates image transmission and storage. These multiple compressed images can be referred to as multiple images in the compression domain. Image processing of compressed images can be referred to as image processing in the compression domain. In some examples, image compression can be performed using different compression rates, and in some examples, the compression domain can be referred to as a multi-rate compression domain.

[0088] Computer vision (CV) is a field of artificial intelligence (AI) that uses computers with neural networks to detect, understand, and process content in images. CV tasks can include (but are not limited to): image classification, object detection, super-resolution (generating high-resolution images from one or more low-resolution images), image denoising, etc. In some relevant examples, multiple CV tasks are performed on multiple uncompressed images, such as the original uncompressed image, a reconstructed image from a compressed image, etc. In some examples, a compressed image is decompressed to generate a reconstructed image, and CV tasks are performed on the reconstructed image. Reconstruction can be computationally intensive. Performing CV tasks in the compression domain without image reconstruction can reduce the computational complexity and latency of CV tasks.

[0089] This disclosure provides techniques for neural networks used in multi-rate computer vision tasks within the compression domain. In some examples, these techniques can be used in an end-to-end (E2E) optimization framework that includes a model of a Compression Domain CV Task Framework (CDCVTF). The E2E optimization framework includes an encoder and a decoder. The encoder generates an encoded bitstream of the input image, and the decoder decodes the encoded bitstream to generate a result based on the CV task. Both the encoder and decoder support multi-rate image compression. The end-to-end (E2E) optimization framework can be a pre-trained framework based on an artificial neural network (ANN).

[0090] In some examples, multiple CV tasks on multiple compressed images can be performed by two parts of a neural network. In some examples, the two parts of the neural network form an end-to-end (E2E) framework. The two parts of the neural network consist of a first part of an image encoding neural network and a second part of a CV task neural network. The image encoding neural network is also known as an image compression encoder. The CV task neural network is also known as a CV task decoder. The image compression encoder encodes an image into an encoded bitstream. The CV task decoder decodes the encoded bitstream to generate CV task results in the compressed domain. The CV task decoder performs multiple CV tasks based on multiple compressed images.

[0091] In some examples, the encoder of the NIC framework compresses the image to generate an encoded bitstream carrying the compressed image or compressed feature map. Furthermore, compressed images or compressed feature maps can be fed to the neural network for a computer vision (CV) task to generate CV task results.

[0092] Figure 12A system (1200) for performing CV tasks in a compressed domain is illustrated in some examples. The system (1200) includes an image compression encoder (1220), an image compression decoder (1250), and a CV task decoder (1270). The image compression decoder (1250) may correspond to the image compression encoder (1220). For example, the image compression encoder (1220) may be configured similarly to the image encoder (800), and the image compression decoder (1250) may be configured similarly to the image decoder (900). In another example, the image compression encoder (1220) may be configured similarly to the image encoder (1000), and the image compression decoder (1250) may be configured similarly to the image decoder (1100). In some examples, the image compression encoder (1220) is configured as the encoding part (with an encoder model) of the NIC framework, and the image compression decoder (1250) is configured as the decoding part (with a decoder model) of the NIC framework. The NIC framework can be trained end-to-end to determine the pre-trained parameters of the encoder and decoder models used for E2E optimization of the framework. The image compression encoder (1220) and image compression decoder (1250) are configured according to the encoder model and the decoder model with pre-trained parameters. The image compression encoder (1220) can receive an input image, compress the input image, and generate an encoded bitstream carrying the compressed image corresponding to the input image. The image compression decoder (1250) can receive the encoded bitstream, decompress the compressed image, and generate a reconstructed image.

[0093] The CV task decoder (1270) is configured to decode the encoded bitstream carrying the compressed image and generate the CV task result corresponding to the input image. The CV task decoder (1270) can be a single-task decoder or a multi-task decoder. CV tasks can include, but are not limited to: super-resolution, object detection, image denoising, image classification, etc.

[0094] The CV task decoder (1270) includes a neural network (e.g., a CV task decoder model) trained on training data. For example, the training data may include: training images, compressed training images (e.g., via an image compression encoder (1220)), and guided CV task results of the training images. For example, the CV task decoder (1270) may take the compressed training images as input and generate the trained CV task results. The CV task decoder (1270) is then trained (e.g., using an adjustable neural network architecture and adjustable parameters) to minimize the loss between the guided CV task results and the trained CV task results. This training process determines the structure and pre-training parameters of the CV task decoder (1270).

[0095] It should be noted that the CV task decoder (1270) can have any suitable neural network architecture to generate CV task results. In some embodiments, the CV task decoder (1270) decodes the encoded bitstream to directly generate CV task results without image reconstruction. In some embodiments, the CV task decoder (1270) first decodes the encoded bitstream and generates a decompressed image (also known as a reconstructed image), and then applies a CV task model to the decompressed image to generate CV task results.

[0096] In some examples, the image compression encoder (1220), image compression decoder (1250), and CV task decoder (1270) are located in different electronic devices. For example, the image compression encoder (1220) is located in a first device (1210), the image compression decoder (1250) is located in a second device (1240), and the CV task decoder (1270) is located in a third device (1260). In some examples, the image compression decoder (1250) and the CV task decoder (1270) may be located in the same device. In some examples, the image compression encoder (1220) and the image compression decoder (1250) may be located in the same device. In some examples, the image compression encoder (1220) and the CV task decoder (1270) may be located in the same device. In some examples, the image compression encoder (1220), the image compression decoder (1250), and the CV task decoder (1270) may be located in the same device.

[0097] Figure 13 A system (1300) for performing CV tasks in a compressed domain is illustrated in some examples. The system (1300) includes an image compression encoder (1320) and a CV task decoder (1370). In some examples, the image compression encoder (1320) is configured as the encoding part (with an encoder model) of the NIC framework, and the CV task decoder (1370) is configured as the decoding part (with a decoder model) of the NIC framework. The NIC framework can be trained end-to-end based on training data to determine the pre-training parameters for the encoder and decoder models used to optimize the framework for E2E. For example, the training data may include training images (uncompressed) and guided CV task results of the training images. For example, the NIC framework may take the training images as input and generate trained CV task results. The NIC framework is then trained (using an adjustable neural network architecture and adjustable parameters) to minimize the loss between the guided CV task results and the trained CV task results. The image compression encoder (1320) and the CV task decoder (1370) are configured according to the pre-training parameters.

[0098] Then, the image compression encoder (1320) can receive the input image and generate an encoded bitstream carrying the compressed image or compressed feature map. The CV task decoder (1370) can receive the encoded bitstream, decompress the compressed image or compressed feature map, and generate the CV task result.

[0099] The CV task decoder (1370) can be a single-task decoder or a multi-task decoder. CV tasks can include, but are not limited to: super-resolution, object detection, image denoising, and image classification.

[0100] It should be noted that the CV task decoder (1370) can be any suitable neural network architecture. In some embodiments, the CV task decoder (1370) decodes the encoded bitstream to directly generate CV task results without image reconstruction. In some embodiments, the CV task decoder (1370) first decodes the encoded bitstream and generates a decompressed image (also known as a reconstructed image), and then applies a CV task model to the decompressed image to generate CV task results.

[0101] In some examples, the image compression encoder (1320) and the CV task decoder (1370) are located in different electronic devices. For example, the image compression encoder (1320) is located in a first device (1310), while the CV task decoder (1370) is located in a second device (1360). In some examples, the image compression encoder (1320) and the CV task decoder (1370) may be located in the same device.

[0102] According to some aspects of this disclosure, the compression domains in systems (1200) and (1300) can be appropriately adjusted for multi-rate compression. In some examples, the hyperparameter λ is used to adjust the compression ratio. In some examples, the compression ratio is defined as the number of bits per pixel and can be calculated according to Equation 9.

[0103] Equation 9 Where R is the bit consumption in the compressed image, also used in Equation 1, W is the width of the input image, and H is the height of the input image. According to Equations 1 and 9, the compression ratio increases as the hyperparameter λ increases.

[0104] This disclosure provides aspects of a multi-rate neural network model within the Compressed Domain Computer-Version Task Framework (CDCVTF). In some examples, the hyperparameter λ is the input to the CDCVTF, causing the CDCVTF to be trained to resolve the compression ratio, and then the value of the hyperparameter λ can be used to tune the compression ratio in the CDCVTF model. It should be noted that the use of the hyperparameter λ in the following description to illustrate techniques for multi-rate in CDCVTF can be appropriately adapted to use other suitable parameters to tune the compression ratio.

[0105] Figure 14 Some examples of systems (1400) for performing CV tasks in the multi-rate compression domain are shown. The system (1400) includes a multi-rate image compression encoder (1411), a multi-rate image compression decoder (1441), and a multi-rate CV task decoder (1461). In some examples, the multi-rate image compression decoder (1441) may correspond to the multi-rate image compression encoder (1411). In some examples, the multi-rate image compression encoder (1411) is configured as the encoding part (with an encoder model) of a multi-rate NIC framework, and the multi-rate image compression decoder (1441) is configured as the decoding part (with a decoder model) of a multi-rate NIC framework. The multi-rate NIC framework can be trained end-to-end (e.g., E2E multi-rate NIC training) to determine the pre-trained parameters of the encoder and decoder models of the E2E optimized framework for multi-rate image compression.

[0106] exist Figure 14 In the example, the multi-rate image compression encoder (1411) includes a transform module (1430) and an image compression encoder (1420). The transform module (1430) includes a neural network that transforms the hyperparameter λ into a tensor (1431). The image compression encoder (1420) can compress the input image based on the tensor (1431) to generate an encoded bitstream carrying a compressed image corresponding to the input image. In some examples, the neural network in the transform module (1430) includes an adjustable structure or adjustable parameters that can be adjusted during E2E multi-rate NIC training to determine pre-training parameters.

[0107] exist Figure 14 In the example, the multi-rate image compressor decoder (1441) includes a transform module (1445) and an image compressor decoder (1450). The transform module (1445) includes a neural network that transforms the hyperparameter λ into a tensor (1446). The image compressor decoder (1450) can decompress the compressed image based on the tensor (1446) to generate a reconstructed image. In some examples, the neural network in the transform module (1445) includes an adjustable structure or adjustable parameters that can be tuned during E2E multi-rate NIC training to determine pre-training parameters.

[0108] exist Figure 14 In the example, the multi-rate CV task decoder (1461) includes a transform module (1480) and a CV task decoder (1470). The transform module (1480) includes a neural network that transforms the hyperparameter λ into a tensor (1481). The CV task decoder (1470) can decode the compressed image based on the tensor (1481) to generate CV task results. In some examples, the neural network in the transform module (1480) includes an adjustable structure or adjustable parameters that can be tuned during training (referred to as CV task training) to determine pre-trained parameters.

[0109] The multi-rate CV task decoder (1461) can be a single-task decoder or a multi-task decoder. CV tasks can include, but are not limited to: super-resolution, object detection, image denoising, and image classification.

[0110] The multi-rate CV task decoder (1461) can be trained on CV task training based on training data. For example, the training data may include: training images, compressed training images at the corresponding rate (e.g., based on a multi-rate image compression decoder (1441)), and guided CV task results of the training images. For example, the multi-rate CV task decoder (1461) can take the compressed training images at the corresponding rate as input and generate the trained CV task results. The multi-rate CV task decoder (1461) is then trained (using an adjustable neural network structure and adjustable parameters) to minimize the loss between the guided CV task results and the trained CV task results.

[0111] In some embodiments, the multi-rate CV task decoder (1461) decodes the encoded bitstream to directly generate CV task results without image reconstruction. In some embodiments, the multi-rate CV task decoder (1461) first decodes the encoded bitstream and generates a decompressed image, and then applies a CV task model to the decompressed image to generate CV task results.

[0112] In some examples, the multi-rate image compression encoder (1411), the multi-rate image compression decoder (1441), and the multi-rate CV task decoder (1461) are located in different electronic devices. For example, the multi-rate image compression encoder (1411) is located in a first device, the multi-rate image compression decoder (1441) is located in a second device, and the multi-rate CV task decoder (1461) is located in a third device. In some examples, the multi-rate image compression decoder (1441) and the multi-rate CV task decoder (1461) may be located in the same device. In some examples, the multi-rate image compression encoder (1411) and the multi-rate image compression decoder (1441) may be located in the same device. In some examples, the multi-rate image compression encoder (1411), the multi-rate image compression decoder (1441), and the multi-rate CV task decoder (1461) may be located in the same device. In some examples, the multi-rate image compression encoder (1411), the multi-rate image compression decoder (1441), and the multi-rate CV task decoder (1461) may be located in the same device.

[0113] In some examples, transformation modules (1430), (1445), and (1480) can have the same neural network structure and the same pre-trained parameters. In some examples, transformation modules (1430), (1445), and (1480) can have the same neural network structure but different pre-trained parameters. In some examples, transformation modules (1430), (1445), and (1480) can have different neural network structures.

[0114] It should be noted that the tensor generated based on the value of the hyperparameter λ can be provided to any suitable layer in the neural network. For example, the tensor (1481) generated based on the value of the hyperparameter λ can be provided to one or more layers in the neural network of the CV task decoder (1470).

[0115] Figure 15 Some examples of systems (1500) for performing CV tasks in a multi-rate compression domain are shown. The system (1500) includes a multi-rate image compression encoder (1511) and a multi-rate CV task decoder (1561). In some examples, the multi-rate image compression encoder (1511) is configured as the encoding part (with an encoder model) of a multi-rate NIC framework, and the multi-rate CV task decoder (1561) is configured as the decoding part (with a decoder model) of the multi-rate NIC framework. The multi-rate NIC framework can be trained end-to-end (e.g., E2E multi-rate NIC training) to determine the pre-trained parameters of the encoder and decoder models of the E2E optimized framework for multi-rate CV tasks.

[0116] exist Figure 15 In the example, the multi-rate image compression encoder (1511) includes a transform module (1530) and an image compression encoder (1520). The transform module (1530) includes a neural network that transforms the hyperparameter λ into a tensor (1531). The image compression encoder (1520) can compress the input image based on the tensor (1531) to generate an encoded bitstream carrying a compressed image or compressed feature map corresponding to the input image. In some examples, the neural network in the transform module (1530) includes an adjustable structure or adjustable parameters that can be adjusted during E2E multi-rate NIC training to determine pre-training parameters.

[0117] exist Figure 15 In the example, the multi-rate CV task decoder (1561) includes a transform module (1580) and a CV task decoder (1570). The transform module (1580) includes a neural network that transforms the hyperparameter λ into a tensor (1581). The CV task decoder (1570) can decode the compressed image or compressed feature map based on the tensor (1581) to generate CV task results. In some examples, the neural network in the transform module (1580) includes an adjustable structure or adjustable parameters that can be tuned during E2E multi-rate NIC training to determine pre-training parameters.

[0118] The multi-rate CV task decoder (1561) can be a single-task decoder or a multi-task decoder. CV tasks can include, but are not limited to: super-resolution, object detection, image denoising, and image classification.

[0119] In E2E multi-rate NIC training based on training data, a multi-rate image compression encoder (1511) can be used to train a multi-rate CV task decoder (1561). For example, training data may include: training images, values ​​of hyperparameter λ, and guided CV task results for the training images. For instance, the multi-rate image compression encoder (1511) can take training images with values ​​of hyperparameter λ as input to generate compressed feature maps, and the compressed feature maps and the values ​​of hyperparameter λ can be input to the multi-rate CV task decoder (1561) to generate trained CV task results corresponding to the training images and the values ​​of hyperparameter λ. The multi-rate CV task decoder (1561) and the multi-rate image compression encoder (1511) are trained (using an adjustable neural network structure and adjustable parameters) to minimize the loss between the guided CV task results and the trained CV task results.

[0120] In some embodiments, the multi-rate CV task decoder (1561) decodes the encoded bitstream to directly generate CV task results without image reconstruction. In some embodiments, the multi-rate CV task decoder (1561) first decodes the encoded bitstream and generates a decompressed image, and then applies a CV task model to the decompressed image to generate CV task results.

[0121] In some examples, the multi-rate image compression encoder (1511) and the multi-rate CV task decoder (1561) are located in different electronic devices. In some examples, the multi-rate image compression encoder (1511) and the multi-rate CV task decoder (1561) are located in the same electronic device.

[0122] In some examples, the transformation module (1530) and the transformation module (1580) can have the same neural network structure and the same pre-training parameters. In some examples, the transformation module (1530) and the transformation module (1580) can have the same neural network structure but different pre-training parameters. In some examples, the transformation module (1530) and the transformation module (1580) can have different neural network structures.

[0123] It should be noted that the tensor generated by the transformation module based on the value of the hyperparameter λ can be provided to any suitable layer in the neural network. For example, the tensor (1581) generated based on the value of the hyperparameter λ can be provided to one or more layers in the neural network of the CV task decoder (1570).

[0124] In some examples, parameter values ​​controlling the compression rate are signaled in the encoded bitstream carrying compressed images or compressed feature maps. For example, the value of the hyperparameter λ is signaled in the encoded bitstream from the encoder side (e.g., multi-rate image compression encoder (1411), multi-rate image compression encoder (1511), etc.). Then, a CV task decoder such as a multi-rate CV task decoder (1461), multi-rate CV task decoder (1561), etc., can receive the hyperparameter λ information.

[0125] It should be noted that the network architecture of neural network structures such as image compression encoders, CV task decoders, and transform modules can have any suitable structure. In one embodiment, the transform module includes a set of convolutional layers. In another embodiment, the transform module includes convolutional layers with activation functions.

[0126] In some embodiments, the hyperparameter λ is selected from a set of predefined values ​​known to both the encoder and decoder sides. For the λ value selected on the encoder side, the index of the value in that set is signaled in the encoded bitstream. Based on this index, the decoder can determine the value of the hyperparameter λ used in encoding, and can use the same value of the hyperparameter λ as input to the decoder network.

[0127] In one example, the eight values ​​for the hyperparameter λ are predefined and placed in a set. Both the encoder and decoder sides have information about this set, such as the eight values ​​and their positions within the set. Instead of transmitting the selected value for hyperparameter λ on the encoder side, the encoder transmits an index from the encoder to the decoder to indicate the selected value from the set. For example, the index could be 0 to 7. Based on this index, the decoder can determine the selected value for hyperparameter λ on the encoder side.

[0128] According to one aspect of this disclosure, compared to a conventional CDCVTF, a multi-rate CDCVTF includes a multi-rate function, and parameters such as hyperparameter λ are inputs to the multi-rate CDCVTF. The values ​​of parameters such as hyperparameter λ can be changed to adjust the image compression rate.

[0129] Figure 16 A flowchart outlining a process (1600) according to an embodiment of the present disclosure is shown. The process (1600) is a coding process. The process (1600) can be executed in an electronic device. In some embodiments, the process (1600) is implemented as software instructions, so that when the processing circuit executes the software instructions, the processing circuit executes the process (1600). The process begins at (S1601) and proceeds to (S1610).

[0130] In (S1610), the values ​​of the parameters used to adjust the compression ratio are input to a multi-rate image compression encoder, such as a multi-rate image compression encoder (1411), a multi-rate image compression encoder (1511), etc. The multi-rate image compression encoder includes one or more neural networks that are used to encode multiple images using the individual values ​​of the parameters.

[0131] In (S1620), the multi-rate image compression encoder encodes the input image into a compressed image to be carried in the encoded bitstream, based on the value of this parameter. The value of this parameter adjusts the compression rate used to encode the input image into a compressed image.

[0132] In (S1630), the index is encoded into the encoded bitstream, and the index points to a value in the set of parameter values ​​(e.g., predefined values).

[0133] Then, process (1600) proceeds to (S1699) and ends.

[0134] The process (1600) can be appropriately adapted to various scenarios, and the steps in the process (1600) can be adjusted accordingly. One or more steps in the process (1600) can be adjusted, omitted, repeated, and / or combined. The process (1600) can be implemented using any suitable order. Additional steps can be added.

[0135] Figure 17 A flowchart outlining a process (1700) according to an embodiment of this disclosure is shown. Process (1700) is a decoding process. Process (1700) can be executed in an electronic device. In some embodiments, process (1700) is implemented as software instructions, so that when the processing circuit executes the software instructions, the processing circuit executes process (1700). The process begins at (S1701) and proceeds to (S1710).

[0136] In (S1710), an index is decoded from the encoded bitstream carrying the compressed image, where the index points to a value in the set of parameter values. The change in the value of the parameter is used to adjust the compression ratio of the compressed image, which is generated according to the parameter by a neural network-based encoder (e.g., a multi-rate image compression encoder (1411), a multi-rate image compression encoder (1511), etc.).

[0137] In (S1720), the value of the parameter is input into a multi-rate compression domain computer vision task decoder (e.g., a multi-rate CV task decoder (1461), a multi-rate CV task decoder (1561), etc.). The multi-rate compression domain computer vision task decoder includes one or more neural networks that perform computer vision tasks from various compressed images based on the corresponding values ​​of the parameter used to generate various compressed images.

[0138] In (S1730), the multi-rate compression domain computer vision task decoder generates the computer vision task result based on the compressed image in the encoded bitstream and the value of the parameter.

[0139] In some examples, a first neural network (e.g., transform module (1480), transform module (1580)) in a multi-rate compression domain computer vision task decoder converts the values ​​of the parameters into tensors. The tensors are then fed into one or more layers in a second neural network (e.g., CV task decoder (1470), CV task decoder (1570)) in the multi-rate compression domain computer vision task decoder. The second neural network generates the computer vision task results based on the compressed image and the tensors.

[0140] In some examples, the first neural network includes one or more convolutional layers. In some examples, the first neural network includes convolutional layers with activation functions.

[0141] In some examples, the second neural network is configured to generate computer vision task results without generating a reconstructed image from the compressed image.

[0142] In some examples, the second neural network is configured to generate a reconstructed image from the compressed image, and to generate computer vision task results from the reconstructed image.

[0143] In some examples, the neural network-based encoder is based on an encoder model within the Neural Image Compression (NIC) framework; the multi-rate compression domain computer vision task decoder is based on a decoder model within the NIC framework; the NIC framework is trained end-to-end, for example, in reference... Figure 15 As described.

[0144] In some examples, the decoder model of the multi-rate compression domain computer vision task decoder is trained separately from the encoder model of the neural network-based encoder, for example, reference Figure 14 As described.

[0145] In some examples, this parameter is a hyperparameter used to weight the distortion when calculating the rate distortion loss, such as the hyperparameter λ in Equation 1.

[0146] It should be noted that computer vision tasks can be any suitable computer vision task, such as image classification, image denoising, object detection, super-resolution, etc.

[0147] Then, process (1700) proceeds to (S1799) and ends.

[0148] The process (1700) can be appropriately adapted to various scenarios, and the steps in the process (1700) can be adjusted accordingly. One or more steps in the process (1700) can be adjusted, omitted, repeated, and / or combined. The process (1700) can be implemented using any suitable order. Additional steps can be added.

[0149] The above-described technology can be implemented as computer software that uses computer-readable instructions and is physically stored in one or more computer-readable media. For example, Figure 18 A computer system (1800) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0150] Computer software can be coded using any suitable machine code or computer language, which can be assembled, compiled, linked, or similarly to create code containing instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), or executed by the CPU or GPU via interpretation, microcode execution, or the like.

[0151] These instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, and Internet of Things (IoT) devices.

[0152] Figure 18 The components shown for the computer system (1800) are exemplary in nature and are not intended to suggest any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement relating to any one or a combination of the components shown in the exemplary embodiments of the computer system (1800).

[0153] The computer system (1800) may include certain human-computer interface input devices. Such human-computer interface input devices may respond to input from one or more human-computer interface users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). Human-computer interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0154] The input human-machine interface device may include one or more of the following (only one of each device is shown): keyboard (1801), mouse (1802), touchpad (1803), touch screen (1810), data glove (not shown), joystick (1805), microphone (1806), scanner (1807), camera (1808).

[0155] The computer system (1800) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include: tactile output devices (such as tactile feedback via a touch screen (1810), a data glove (not shown), or a joystick (1805), but may also include tactile feedback devices that are not used as input devices), audio output devices (such as speakers (1809), headphones (not shown)), and visual output devices (such as screens (1810) including CRT screens, LCD screens, plasma screens, OLED screens, each screen may or may not have touch input functionality, each screen may or may not have tactile feedback functionality, some of which may output two-dimensional visual output or more than three-dimensional output through means such as stereoscopic output, virtual reality glasses (not shown), holographic displays and smoke canisters (not shown), and printers (not shown).

[0156] The computer system (1800) may also include human-accessible storage devices and their associated media, such as optical media, including: CD / DVD ROM / RW with CD / DVD (1820) or similar media (1821), thumb drives (1822), removable hard disk drives or solid-state drives (1823), conventional magnetic media (such as magnetic tape and floppy disks (not shown)), devices based on dedicated ROM / ASIC / PLD (such as security dongles (not shown)), etc.

[0157] Those skilled in the art will also understand that the term "computer-readable medium" used in connection with the subject matter currently disclosed does not include transmission media, carrier waves, or other transient signals.

[0158] The computer system (1800) may also include an interface (1854) to one or more communication networks (1855). Networks may be, for example, wireless networks, wired networks, optical networks. Networks may also be local area networks, wide area networks, metropolitan area networks, vehicle and industrial networks, real-time networks, latency-tolerant networks, and so on. Examples of networks include: local area networks (e.g., Ethernet, wireless LAN), cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), wired or wireless wide area digital TV networks (including cable TV, satellite TV, and terrestrial broadcast TV), vehicle and industrial networks (including Controller Area Network Bus (CANBus)), and so on. Some networks typically require an external network interface adapter to connect to certain general-purpose data ports or peripheral buses (1849) (e.g., the computer system's USB port (1800)); others are typically integrated into the core of the computer system (1800) via a system bus (e.g., an Ethernet interface connected to a PC computer system, or a cellular network interface connected to a smartphone computer system). Using any of these networks, the computer system (1800) can communicate with other entities. This communication can be unidirectional, receive-only (e.g., broadcast television), send-only (e.g., to a CANbus device), or bidirectional (e.g., other computer systems using local or wide area digital networks). As described above, certain protocols and protocol stacks can be used on each of these networks and network interfaces.

[0159] The aforementioned human-computer interface device, human-accessible storage device, and network interface can be connected to the kernel (1840) of the computer system (1800).

[0160] The kernel (1840) may include one or more devices such as a central processing unit (CPU) (1841), a graphics processing unit (GPU) (1842), a dedicated programmable processing unit (FPGA) (1843), a hardware accelerator (1844) for certain tasks, a graphics adapter (1850), and so on. These devices, along with read-only memory (ROM) (1845), random access memory (1846), and internal mass storage (1847) (such as internal non-user-accessible hard disk drives, SSDs, etc.), can be connected via a system bus (1848). In some computer systems, the system bus (1848) may be accessed as one or more physical connectors to allow for expansion by adding CPUs, GPUs, etc. Peripheral devices may be connected directly to the kernel's system bus (1848) or via a peripheral bus (1849). In one example, a screen (1810) may be connected to a graphics adapter (1850). Peripheral bus architectures include PCI, USB, etc.

[0161] The CPU (1841), GPU (1842), FPGA (1843), and accelerator (1844) can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM (1845) or RAM (1846). Transient data can also be stored in RAM (1846), while permanent data can be stored, for example, in internal mass storage (1847). Fast storage and retrieval of any memory device can be achieved by using a cache memory; the cache memory can be closely associated with one or more CPUs (1841), GPUs (1842), mass storage (1847), ROM (1845), RAM (1846), etc.

[0162] The computer-readable medium may contain computer code for performing operations of various computer implementations. The medium and computer code may be specifically designed and constructed for the purposes of this disclosure, or the medium and computer code may be of a type known and available to those skilled in the art of computer software.

[0163] By way of example and not limitation, a computer system having an architecture (1800) (especially a kernel (1840)) can provide functionality as a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, and certain kernel memories (1840) having a non-transitory nature, such as internal kernel mass storage (1847) or ROM (1845). Software implementing various embodiments of this disclosure can be stored in such devices and executed by the kernel (1840). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can cause the kernel (1840) and its specific processor (including a CPU, GPU, FPGA, etc.) to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (1846) and modifying such data structures according to the processes defined by the software. As a supplement or alternative, a computer system may provide functionality through hard-wired logic or otherwise embedded circuitry (e.g., an accelerator (1844)) that may replace or operate in conjunction with software to perform a particular process or a particular portion of a particular process described herein. References to software may include logic, and vice versa, where appropriate. References to computer-readable media may include, where appropriate, circuitry storing software for execution (e.g., integrated circuits (ICs)), circuitry containing logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0164] Although several exemplary embodiments have been described in this disclosure, modifications, substitutions, and various alternative equivalents fall within the scope of this disclosure. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, while not expressly shown or described herein, embody the principles of this disclosure and are therefore within its spirit and scope.

Claims

1. An image processing method, characterized in that, include: Decode an index from an encoded bitstream carrying a compressed image, the index pointing to a value in a set of parameter values, the changes in which are used to adjust the compression ratio of the compressed image, the compressed image being generated by a neural network-based encoder based on the parameters; The values ​​of the parameters are input into the multi-rate compressed domain computer vision task decoder in the compressed domain computer vision task framework CDCVTF. The multi-rate compressed domain computer vision task decoder includes one or more neural networks, which are used to generate corresponding values ​​for each of the compressed images according to the parameters and perform computer vision tasks from each of the compressed images. as well as The multi-rate compression domain computer vision task decoder generates a computer vision task result based on the compressed image in the encoded bitstream, wherein the compressed image in the encoded bitstream is compressed according to the value of the parameter at a compression ratio corresponding to one of multiple different compression ratios. The generation of the computer vision task result includes: The first neural network in the multi-rate compressed domain computer vision task decoder converts the values ​​of the parameters into tensors. The tensor is input into one or more layers of a second neural network in the multi-rate compressed domain computer vision task decoder; and The second neural network generates the computer vision task result based on the compressed image and the tensor.

2. The method according to claim 1, characterized in that, The first neural network includes one or more convolutional layers.

3. The method according to claim 1, characterized in that, The first neural network includes convolutional layers with activation functions.

4. The method according to claim 1, characterized in that, The second neural network is configured to generate the computer vision task result without generating a reconstructed image based on the compressed image.

5. The method according to claim 1, characterized in that, The second neural network is configured to generate a reconstructed image based on the compressed image, and to generate the computer vision task result based on the reconstructed image.

6. The method according to claim 1, characterized in that, The neural network-based encoder is based on the encoder model in the neural image compression NIC framework, and the multi-rate compression domain computer vision task decoder is based on the decoder model in the NIC framework, which is trained end-to-end.

7. The method according to claim 1, characterized in that, The decoder model of the multi-rate compressed domain computer vision task decoder and the encoder model of the neural network-based encoder are trained separately.

8. The method according to claim 1, characterized in that, The parameters are hyperparameters used to weight the distortion when calculating rate distortion loss.

9. The method according to claim 1, characterized in that, The computer vision task includes at least one of image classification, image denoising, object detection, and super-resolution.

10. An image processing apparatus, characterized in that, include: Memory, used to store program instructions; as well as A processor for invoking program instructions stored in the memory to implement the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, Includes instructions that, when run on a computer, cause the computer to perform the method according to any one of claims 1 to 9.

Images