Image compression systems, image processing methods, encoding / decoding methods, and electronic devices

Abstract

This application provides an image compression system, an image processing method, an encoding / decoding method, and an electronic device. The image compression system includes a first selection module, an entropy encoding module, an entropy decoding module, a quantization module, N encoding networks, and one decoding network. The N encoding networks have different encoding losses. The first selection module is used to select a target encoding network from the N encoding networks based on the number of times the image to be encoded has been encoded. The target encoding network is used to perform feature transformation on the image to be encoded to obtain a first feature map. The quantization module is used to quantize the first feature map to obtain a second feature map. The entropy encoding module is used to entropy encode the second feature map to obtain a bitstream. The entropy decoding module is used to entropy decode the bitstream to obtain a third feature map. The decoding network is used to perform feature transformation based on the third feature map to obtain a reconstructed image. This effectively reduces the loss ratio of the image after multiple encoding and decoding operations.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image compression system, an image processing method, an encoding / decoding method, and an electronic device. Background Technology

[0002] With the development of neural networks in the field of image compression and their performance surpassing traditional compression methods, deep learning-based artificial intelligence (AI) image compression algorithms are gradually being widely used. Among them, AI image compression algorithms include various types, such as AI image compression algorithms based on the variational autoencoder (VAE) framework.

[0003] Many scenarios (such as software editing) require multiple encoding and decoding of images, which can cause a significant drop in rate-distortion performance and make it impossible to guarantee the quality of the image after multiple encoding and decoding, thus affecting the application of end-to-end image compression schemes. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides an image compression system, an image processing method, an encoding / decoding method, and an electronic device. The encoding / decoding method and image processing method are implemented based on the image compression system and can reduce the rate-distortion ratio of an image after multiple encoding / decoding operations (wherein, the rate-distortion loss of the last encoding / decoding operation minus the rate-distortion loss of the first encoding / decoding operation yields the loss difference; the ratio of the loss difference to the rate-distortion loss of the first encoding / decoding operation is the loss amplitude ratio).

[0005] It should be noted that this application can be applied to AI compression scenarios. The image compression system of this application can refer to an AI compression system, and the encoding / decoding method and image processing method implemented based on the image compression system can refer to an AI compression algorithm. The image compression system of this application can be used for multiple encoding / decoding operations.

[0006] For example, one application scenario of this application may be: a scenario in which images / videos are forwarded and displayed between multiple electronic devices; wherein, in the scenario in which images / videos are forwarded and displayed between multiple electronic devices, multiple encoding and decoding processes are performed.

[0007] For example, one application scenario of this application may be: a scenario of multiple image / video editing; wherein, in the scenario of multiple image / video editing, multiple encoding and decoding are performed.

[0008] In a first aspect, embodiments of this application provide an image compression system, which may include: a first selection module, an entropy encoding module, an entropy decoding module, a quantization module, N encoding networks, and one decoding network, wherein the N encoding networks have different encoding losses, and N is an integer greater than 1, wherein:

[0009] The first selection module is used to select a target coding network from N coding networks based on the number of times the image to be encoded has been encoded, wherein the image to be encoded is an uncoded image or a reconstructed image;

[0010] The target encoding network is used to perform feature transformation on the image to be encoded to obtain the first feature map;

[0011] The quantization module is used to quantize the first feature map to obtain the second feature map;

[0012] The entropy coding module is used to entropy code the second feature map to obtain the bitstream;

[0013] The entropy decoding module is used to perform entropy decoding on the bitstream to obtain the third feature map;

[0014] A decoding network is used to perform feature transformation based on the third feature map to obtain the reconstructed image.

[0015] It should be understood that the N encoding networks are all trained encoding networks, and the decoding network is a trained decoding network. The N encoding networks and the 1 decoding network are jointly trained. The N encoding networks have different loss weight coefficients, which are used to train encoding networks with different encoding losses. Therefore, after training, the N encoding networks have different encoding losses. The specific training process will be explained later. After jointly training the N encoding and decoding networks, for each of the N encoding networks, encoding the image based on that encoding network and decoding the encoded result based on the decoding network results in a rate-distortion loss of less than a preset rate-distortion loss (the preset rate-distortion loss can be set according to requirements). Therefore, after training, the decoding network can effectively accommodate the N encoding networks with different encoding losses. Thus, compared to existing technologies that use a single encoding network for decoding, the decoding network in this application has a higher reconstruction capability. Therefore, in multiple encoding and decoding processes, encoding based on one encoding network and decoding based on that decoding network each time can effectively reduce the magnitude of image loss after multiple encoding and decoding processes.

[0016] It should be noted that the number of entropy coding modules can be Y (where Y is a positive integer). For example, Y can be equal to 1, in which case N coding networks correspond to one entropy coding module. For example, Y can be equal to N, in which case N coding networks correspond to N entropy coding networks. For example, Y can be greater than 1 and less than N, in which case one or more of the N coding networks correspond to one entropy coding module.

[0017] It should be noted that the number of entropy decoding modules can be Y (where Y is a positive integer); that is, the number of entropy decoding modules is the same as the number of entropy encoding modules, and there is a one-to-one correspondence between Y entropy decoding modules and Y entropy encoding modules. This application uses Y equal to 1 as an example for illustration.

[0018] For example, the first feature map may refer to Figure 2a Feature map 1; the second feature map can refer to Figure 2a Feature map 2 in the middle, the third feature map can refer to Figure 2a Feature diagram 3.

[0019] For example, the image to be encoded can refer to Figure 2a Image 1 in the image; the reconstructed image can refer to... Figure 2a Image 2 in the image.

[0020] It should be understood that the image compression system may also include other modules, such as an entropy estimation module, etc., and this application does not limit this. For example, the entropy estimation module can be used for entropy estimation and output entropy estimation information. The entropy estimation information can be used in the entropy encoding process of the entropy encoding module and in the entropy decoding process of the entropy decoding module.

[0021] For example, the encoding network can be a convolutional neural network.

[0022] For example, the decoding network can be a convolutional neural network.

[0023] For example, when an image has been encoded once and decoded once, its encoding count is 1.

[0024] For example, there can be multiple decoding networks, with one decoding network corresponding to multiple encoding networks among N encoding networks. In this way, after selecting the target encoding network, one decoding network corresponding to the target encoding network can be selected from multiple decoding networks to perform feature transformation on the third feature map to obtain the reconstructed image.

[0025] For example, the second feature map and the third feature map may be the same or different, and this application does not impose any restrictions on this.

[0026] According to the first aspect, the N coding networks have the same network structure but different network parameters; or, the N coding networks have different network structures.

[0027] For example, each of the N encoding networks can be implemented using any existing neural network, and this application does not impose any restrictions on this. For instance, when the network structures of the N encoding networks are identical, each encoding network can be composed of four convolutional layers and three activation layers interleaved and cascaded. In this case, having identical network structures for the N encoding networks can save on the cost of designing the encoding networks.

[0028] For example, each of the N coding networks can correspond to a set of network parameters; thus, N sets of network parameters can be obtained. In the case where the N coding networks have the same network structure but different network parameters, the image compression system including N coding networks can also be understood as including only one coding network with the same structure as any one of the N coding networks, and this coding network corresponds to N sets of network parameters. In this case, selecting a target coding network from the N coding networks can also be understood as selecting target network parameters from the N sets of network parameters; configuring the network parameters of this coding network as the target network parameters, the target coding network can then be obtained.

[0029] For example, the network structure of an encoding network can refer to the network layers included in the encoding network and the connections between the network layers. The encoding network can include various types of network layers (e.g., convolutional, activation, etc.), and each type of network layer can be one or more.

[0030] For example, the network parameters of an encoding network can refer to the weight matrix and bias matrix corresponding to the multiple network layers included in the encoding network; wherein, the weight matrix is ​​composed of weight vectors of multiple network layers, and the bias matrix is ​​composed of bias vectors of multiple network layers. Each network layer can consist of one or more neurons, where a value in the weight vector of the network layer represents the weight value of a neuron in the network layer, and a value in the bias vector of the network layer represents the bias value of a neuron in the network layer.

[0031] According to the first aspect, or any implementation of the first aspect above, N coding networks share the first coding network; each of the N coding networks consists of the first coding network and the first gain vector, and the first gain vector of each of the N coding networks is different;

[0032] The target encoding network is specifically used to call the first encoding network to perform feature transformation on the image to be encoded to obtain the fourth feature map; and to scale the fourth feature map based on the first gain vector of the target encoding network to obtain the first feature map.

[0033] This saves on the cost of designing the encoding network. Furthermore, compared to having N independent encoding networks, it saves on the storage memory required for the encoding networks.

[0034] For example, the first encoding network can be implemented using any existing neural network, and this application does not limit it. For instance, the first encoding network can be composed of four convolutional layers and three activation layers interleaved and cascaded.

[0035] For example, the first gain vector has the same number of dimensions as the output channels of the first coding network.

[0036] For example, the fourth feature map can be as follows: Figure 4b Feature map 4 in the image, the first feature map can be as follows: Figure 4b Feature diagram 1.

[0037] According to the first aspect, or any implementation of the first aspect above, the image compression system further includes a second selection module and N second gain vectors, wherein the N second gain vectors correspond one-to-one with the N first gain vectors;

[0038] The second selection module is used to select a target gain vector from N second gain vectors based on the number of encoding iterations; and to scale the third feature map based on the target gain vector to obtain a fifth feature map; the decoding network is specifically used to perform feature transformation on the fifth feature map to obtain the reconstructed image. In this way, the feature map scaled based on the first gain vector during the encoding process can be recovered.

[0039] For example, the second gain vector has the same dimension as the first gain vector. In one possible approach, when the feature map is magnified based on the first gain vector, the feature map is shrunk based on the second gain vector; conversely, when the feature map is shrunk based on the first gain vector, the feature map is magnified based on the second gain vector.

[0040] In one possible approach, each element in the second gain vector is the reciprocal of the corresponding element in the first gain vector. It should be understood that the elements in the second gain vector may not be the reciprocals of the corresponding elements in the first gain vector; this application does not impose any restrictions on the numerical relationship between the elements in the second gain vector and the corresponding elements in the first gain vector.

[0041] In this way, by scaling the feature map using the first gain vector and the second gain vector, the difference between the quantized values ​​obtained by the quantization module in two adjacent encoding and decoding processes can be reduced; thus, the loss amplitude ratio of the image after two adjacent encoding and decoding processes can be reduced; thereby, the loss amplitude ratio of the image after multiple encoding and decoding processes can be reduced.

[0042] For example, the third feature map can be as follows: Figure 4b Feature map 3, the fifth feature map can be as follows Figure 4b Feature diagram 5.

[0043] According to the first aspect, or any implementation of the first aspect above, when N equals 2, the N encoding networks include a second encoding network and a third encoding network. The second encoding network is a non-reversible neural network, the third encoding network is a reversible neural network mapped according to a first preset direction, and the decoding network is a reversible neural network mapped according to a second preset direction.

[0044] For example, a reversible neural network can be a network that can be used for both encoding and decoding. A non-reversible neural network can be a network that can only be used for encoding or decoding; wherein, the second encoding network is a non-reversible neural network that can only be used for encoding.

[0045] For example, the first preset direction can refer to the direction that takes the image to be encoded as input and the feature map as output; the second prediction method can refer to the direction that takes the feature map as input and the reconstructed image as output. Alternatively, the reversible neural network can include R (R is an integer greater than 1) network layers, the first preset direction can refer to the direction that takes the first network layer as input and the Rth network layer as output; the second preset direction can refer to the direction that takes the Rth network layer as input and the first network layer as output.

[0046] Subsequently, a second encoding network can be used for the initial encoding and decoding, and a third encoding network can be used for subsequent multiple encodings.

[0047] Because the encoding and decoding processes in reversible neural networks share network parameters, reversible neural networks face more constraints, resulting in lower expressive power compared to non-reversible neural networks. Therefore, under the same image quality, the bitrate of the bitstream generated by encoding based on a reversible neural network is greater than that generated by encoding based on a non-reversible neural network. In other words, the rate-distortion loss of encoding and decoding based on a reversible neural network is greater than the rate-distortion loss of encoding and decoding based on both a non-reversible neural network and a decoding network. Therefore, using a second encoding network for the initial encoding and decoding can reduce the rate-distortion loss of the initial encoding and decoding. The rate-distortion loss of the initial encoding and decoding determines the upper limit of the rate-distortion performance of the image in subsequent encoding and decoding (where a larger rate-distortion loss results in worse rate-distortion performance), thereby improving the rate-distortion performance of the image after multiple subsequent encoding and decoding processes.

[0048] Furthermore, reversible neural networks are reversible, while non-reversible neural networks are irreversible. Compared to using non-reversible neural networks for multiple encoding and decoding operations, using reversible neural networks for multiple encoding and decoding operations results in a slower rate-distortion performance degradation of the image. Therefore, using a third encoding network for subsequent multiple encoding and decoding operations can slow down the rate of degradation of the image's rate-distortion performance after multiple encoding and decoding operations, thereby effectively reducing the loss magnitude ratio of the image after multiple encoding and decoding operations.

[0049] For example, rate-distortion loss refers to the loss of an image after the encoding and decoding processes, including bit rate loss and image distortion loss. Encoding loss refers to the loss of the image after feature transformation by the encoding network. The encoding loss of the encoding network affects the rate-distortion loss of the image; rate-distortion loss can be used to measure this encoding loss.

[0050] For example, the second encoding network can be implemented using any existing neural network, and this application does not limit it. For instance, the second encoding network can be composed of four convolutional layers and three activation layers interleaved and cascaded.

[0051] For example, a reversible neural network may include four blocks, an average layer, and a repeat layer. Each block may include three layers: a pixel shuffling layer (for transferring spatial information to channels), an invertible convolutional layer (for convolution), and coupling layers (for rearranging elements). For example, the average layer is used for channel shuffling; the repeat layer is used for channel copying. The pixel shuffling layer of Block 1 can refer to the first layer of the reversible neural network; the average and repeat layers can refer to the Rth layers of the reversible neural network.

[0052] It should be noted that when the reversible neural network is mapped according to the first preset direction, that is, when the PixelShuffling layer of Block1 is the input layer and the Average layer is the output layer, the reversible neural network is the third encoding network; when the reversible neural network is mapped according to the second preset direction, that is, when the PixelShuffling layer of Block1 is the output layer and the Repeat layer is the input layer, the reversible neural network is the decoding network.

[0053] According to the first aspect, or any implementation of the first aspect above, the decoding network and N encoding networks are jointly trained. The joint training includes M batches of training on the N encoding networks and 1 decoding network. Each batch of training is based on the loss weight coefficient corresponding to the i-th encoding network, training both the i-th encoding network and the decoding network. Each of the N encoding networks has a different loss weight coefficient, and these different loss weight coefficients are used to train encoding networks with different encoding losses. M is a positive integer greater than N, and i is an integer between 1 and N. Thus, by training the N encoding networks based on different loss weight coefficients, N encoding networks with different encoding losses can be obtained.

[0054] Secondly, embodiments of this application provide an encoding method, which includes: first, acquiring an image to be encoded, wherein the image to be encoded is an unencoded image or a reconstructed image; and acquiring the number of times the image to be encoded has been encoded; then, based on the number of times the image to be encoded has been encoded, selecting a target encoding network from N encoding networks, where N is an integer greater than 1; subsequently, inputting the image to be encoded into the target encoding network to obtain a first feature map output by the target encoding network; then, quantizing the first feature map to obtain a second feature map; and finally, performing entropy encoding on the second feature map to obtain a bitstream. The bitstream can then be decoded by a decoding end based on a decoding network. The decoding network and the N encoding networks are jointly trained, and the N encoding networks have different loss weight coefficients. These different loss weight coefficients are used to train encoding networks with different encoding losses. After training, the decoding network can be well compatible with the N encoding networks with different encoding losses. Compared with existing technologies that use a single encoding network for decoding, the decoding network of this application has a higher reconstruction capability. In this way, when encoding an image, the image is encoded based on one of the N encoding networks, and then decoded based on a decoding network with a higher reconstruction capability than the decoding network in the prior art. This can reduce the loss ratio of the image after this encoding and decoding compared to the previous encoding and decoding. Furthermore, it can effectively reduce the loss ratio of the image after multiple encoding and decoding.

[0055] For example, when an image has been encoded once and decoded once, its encoding count is 1.

[0056] According to the second aspect, selecting a target coding network from N coding networks based on the number of times the image to be encoded has been encoded includes: obtaining preset network selection conditions; and selecting a target coding network from N coding networks based on the number of times the image to be encoded has been encoded and the preset network selection conditions.

[0057] For example, the preset network selection conditions can be set as needed, specifically based on the network structure of N coding networks, and this application does not impose any restrictions on this.

[0058] For example, one coding network can be selected from N coding networks as the target coding network.

[0059] According to the second aspect, or any implementation of the second aspect above, the preset network selection conditions include: when the number of encoding times is 0, selecting the encoding network with the smallest difference between the compressed bit rate and the target compressed bit rate; wherein, the compressed bit rate is related to the encoding loss.

[0060] For example, for any coding network, the compression rate of the coding network is negatively correlated with the coding loss of the coding network. That is, the greater the coding loss of the coding network, the smaller the compression rate of the coding network; the smaller the coding loss of the coding network, the greater the compression rate of the coding network.

[0061] In other words, when encoding the image to be encoded for the first time, this application uses an encoding network whose compression bitrate is closest to the target compression bitrate. Subsequent decoding uses a decoding network with higher reconstruction capabilities than existing decoding networks. This ensures that, at the same bitrate (in the scenario of a single encoding network in existing technologies, the compression bitrate of the single encoding network is the target compression bitrate; that is, the compression bitrate corresponding to the initial encoding in this application is similar to or the same as the compression bitrate corresponding to the initial encoding in existing technologies), image distortion is low (at this time, the rate-distortion loss of the image is also low). The rate-distortion loss during the initial encoding and decoding determines the upper limit of the rate-distortion performance of the image in subsequent encoding and decoding, which can further improve the rate-distortion performance of subsequent images in multiple encoding and decoding processes.

[0062] According to the second aspect, or any implementation of the second aspect above, the preset network selection conditions include: when the number of encoding iterations is not zero, selecting a coding network with a coding loss less than the historical coding loss, where the historical coding loss refers to the coding loss of the coding network selected in the previous encoding of the image to be encoded. In other words, in subsequent compression processes, a coding network with a smaller coding loss than the previous one is selected each time, thus effectively reducing the magnitude of image loss after multiple encoding and decoding iterations.

[0063] According to the second aspect, or any implementation of the second aspect above, N coding networks share a first coding network; each of the N coding networks consists of a first coding network and a first gain vector, and the first gain vector of each of the N coding networks is different; inputting the image to be encoded into a target coding network to obtain a first feature map output by the target coding network includes: inputting the image to be encoded into the first coding network to obtain a fourth feature map; scaling the fourth feature map based on the first gain vector of the target coding network to obtain the first feature map.

[0064] According to the second aspect, or any implementation of the second aspect above, when N equals 2, the N encoding networks include a second encoding network and a third encoding network. The second encoding network is a non-reversible neural network, and the third encoding network is a reversible neural network mapped according to a first preset direction. The preset network selection conditions include: when the number of times the encoding has been performed is 0, the second encoding network is selected; when the number of times the encoding has been performed is not 0, the third encoding network is selected.

[0065] Because the encoding and decoding processes in reversible neural networks share network parameters, reversible neural networks face more constraints, resulting in lower expressive power compared to non-reversible neural networks. Therefore, under the same image quality, the bitrate of a bitstream encoded using a reversible neural network is higher than that of a bitstream encoded using a non-reversible neural network. In other words, the rate-distortion loss during encoding and decoding using a reversible neural network is greater than the rate-distortion loss during encoding and decoding using both a non-reversible neural network and a decoding network. Therefore, using a second encoding network for the initial encoding and decoding can reduce the rate-distortion loss during the initial encoding and decoding. The rate-distortion loss during the initial encoding and decoding determines the upper limit of the image's rate-distortion performance in subsequent encoding and decoding, thereby improving the image's rate-distortion performance after multiple encoding and decoding iterations.

[0066] Furthermore, reversible neural networks are reversible, while non-reversible neural networks are irreversible. Compared to using non-reversible neural networks for multiple encoding and decoding operations, using reversible neural networks for multiple encoding and decoding operations results in a slower rate-distortion performance degradation of the image. Therefore, using a third encoding network for subsequent multiple encoding and decoding operations can slow down the rate of degradation of the image's rate-distortion performance after multiple encoding and decoding operations, thereby effectively reducing the loss magnitude ratio of the image after multiple encoding and decoding operations.

[0067] According to the second aspect, or any implementation of the second aspect above, the method further includes: encapsulating the encoded number of times into the bitstream.

[0068] For example, in the bitstream, existing syntax elements can be used to represent the number of times the code has been encoded, or new syntax elements can be used to represent the number of times the code has been encoded; this application does not limit this.

[0069] The second aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the second aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0070] Thirdly, embodiments of this application provide a decoding method, which includes: first, acquiring a bitstream, the bitstream being obtained by an encoder encoding an image to be encoded based on a target encoding network among N encoding networks, where N is an integer greater than 1; next, performing entropy decoding on the bitstream to obtain a third feature map; then, inputting the third feature map into a decoding network to obtain a reconstructed image output by the decoding network. The decoding network and the N encoding networks are jointly trained, with different loss weight coefficients for each of the N encoding networks. These different loss weight coefficients are used to train encoding networks with different encoding losses. After training, the decoding network can effectively accommodate the N encoding networks with different encoding losses. Compared to existing decoding networks that correspond to a single encoding network, the decoding network of this application has a higher reconstruction capability. Therefore, decoding based on this decoding network can effectively reduce the loss magnitude ratio of the image after multiple encoding and decoding operations.

[0071] According to the third aspect, after obtaining the reconstructed image output by the decoding network, the method further includes updating the number of times the entropy decoding has been performed. This allows for the selection of a matching encoding network in the next encoding process.

[0072] For example, updating the number of encoded values ​​obtained from entropy decoding can mean adding 1 to the number of encoded values ​​obtained from entropy decoding.

[0073] According to the third aspect, or any implementation of the third aspect above, the method further includes: selecting a target gain vector from N second gain vectors based on the number of times it has been encoded obtained by entropy decoding; scaling the third feature map based on the target gain vector to obtain a fifth feature map; and inputting the third feature map into a decoding network to obtain a reconstructed image output by the decoding network, including: inputting the fifth feature map into the decoding network to obtain a reconstructed image output by the decoding network. In this way, a second gain vector corresponding to the first gain vector on the encoding side can be selected to recover the scaled feature map after encoding.

[0074] In this way, by scaling the feature map using the first gain vector and the second gain vector, the difference between the quantized values ​​obtained by the quantization module in two adjacent encoding and decoding processes can be reduced; thus, the loss amplitude ratio of the image after two adjacent encoding and decoding processes can be reduced; thereby, the loss amplitude ratio of the image after multiple encoding and decoding processes can be reduced.

[0075] Fourthly, embodiments of this application provide a preset network training method. The preset network includes N encoding networks and one decoding network, where N is an integer greater than 1. The method includes: first, acquiring training images; then, performing M batch training operations on the preset network based on the training images, where M is an integer greater than N. Each batch training operation on the preset network includes: determining a rate-distortion loss based on the training images, the reconstructed images, and the loss weight coefficients corresponding to the i-th encoding network; wherein the reconstructed image is obtained by encoding the training images using the i-th encoding network and then decoding the encoding result using the decoding network; and training the i-th encoding network and the decoding network based on the rate-distortion loss. Each of the N encoding networks has a different loss weight coefficient, and different loss weight coefficients can be used to train encoding networks with different encoding losses; therefore, after completing M batch training operations on the preset network, the encoding losses of the N encoding networks are also different.

[0076] It should be noted that the N encoding networks share the same decoding network. During batch training of the preset network, in the forward computation process (i.e., the process of obtaining the reconstructed image based on the training image), the network parameters of the decoding network are those obtained after training in the previous H-1 batches (where H is an integer between 1 and M; the current training is in the H-th batch). Similarly, the network parameters of the decoding network adjusted during backpropagation are also those obtained after training in the previous H-1 batches. After training the preset network in M ​​batches, for each of the N encoding networks, after encoding the image based on that encoding network and decoding the encoded result based on the decoding network, the rate-distortion loss of the image is less than the preset rate-distortion loss. Thus, the trained decoding network can be well-compatible with various encoding networks with different encoding losses, and its reconstruction capability is higher than that of the decoding network corresponding to a single encoding network in existing technologies. Furthermore, subsequent encoding and decoding based on an image compression system containing this preset network can effectively reduce the magnitude of image loss after multiple encoding and decoding operations.

[0077] It should be noted that during each batch training of the preset networks, one encoding network can be selected from N encoding networks for training. Then, based on a batch of training images, the selected encoding network and decoding network are trained. For ease of explanation, the encoding network selected from the N encoding networks for training can be called the i-th encoding network. i is an integer between 1 and N (inclusive).

[0078] According to the fourth aspect, the rate-distortion loss is determined based on the training image, the reconstructed image, and the loss weight coefficients corresponding to the i-th encoding network. This includes: obtaining the bitrate loss corresponding to encoding the training image using the i-th encoding network; determining the image distortion loss based on the training image and the reconstructed image; and performing a weighted calculation of the bitrate loss and the image distortion loss based on the loss weight coefficients corresponding to the i-th encoding network to obtain the rate-distortion loss. This allows for accurate calculation of the rate-distortion loss of the image after encoding and decoding, thereby improving the training efficiency of the preset network.

[0079] For example, the loss weight coefficients corresponding to each coding network may include a first weight coefficient and a second weight coefficient, wherein the first weight coefficient corresponds to the bit rate loss and the second weight coefficient corresponds to the image distortion loss.

[0080] In one possible approach, the first weight coefficients and the second weight coefficients of the N encoding networks are different.

[0081] In one possible approach, the first weight coefficients of the N encoding networks are the same, but the second weight coefficients are different.

[0082] In one possible approach, the first weight coefficients of the N coding networks are different, while the second weight coefficients are the same. For example, the first weight coefficients of the N coding networks are all Lagrange coefficients β with different values, and the second weight coefficients are all 1.

[0083] Fifthly, embodiments of this application provide an electronic device, including: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, cause the electronic device to perform the encoding method in the second aspect or any possible implementation of the second aspect.

[0084] The fifth aspect and any implementation thereof correspond to the second aspect and any implementation thereof, respectively. The technical effects of the fifth aspect and any implementation thereof can be found in the technical effects of the second aspect and any implementation thereof, as described above, and will not be repeated here.

[0085] In a sixth aspect, embodiments of this application provide an electronic device, including: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, when executed by the processor, causing the electronic device to perform the decoding method in the third aspect or any possible implementation of the third aspect.

[0086] The sixth aspect and any implementation thereof correspond to the third aspect and any implementation thereof, respectively. The technical effects of the sixth aspect and any implementation thereof are similar to those of the third aspect and any implementation thereof, and will not be repeated here.

[0087] In a seventh aspect, embodiments of this application provide an electronic device, including: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, when executed by the processor, causing the electronic device to execute a preset network training method in the fourth aspect or any possible implementation of the fourth aspect.

[0088] The seventh aspect and any implementation thereof correspond to the fourth aspect and any implementation thereof, respectively. The technical effects of the seventh aspect and any implementation thereof can be found in the technical effects of the fourth aspect and any implementation thereof, as described above, and will not be repeated here.

[0089] Eighthly, embodiments of this application provide an image processing apparatus, including one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, it causes the electronic device to perform the encoding method in the second aspect or any possible implementation of the second aspect.

[0090] The eighth aspect and any implementation thereof correspond to the second aspect and any implementation thereof, respectively. The technical effects corresponding to the eighth aspect and any implementation thereof are similar to those corresponding to the second aspect and any implementation thereof, and will not be repeated here.

[0091] Ninthly, embodiments of this application provide an image processing apparatus, including one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, it causes the electronic device to perform the decoding method in the third aspect or any possible implementation of the third aspect.

[0092] The ninth aspect and any implementation thereof correspond to the third aspect and any implementation thereof, respectively. The technical effects corresponding to the ninth aspect and any implementation thereof are similar to those corresponding to the third aspect and any implementation thereof, and will not be repeated here.

[0093] In a tenth aspect, embodiments of this application provide an image processing apparatus, including one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, it causes the electronic device to execute a preset network training method in the fourth aspect or any possible implementation of the fourth aspect.

[0094] The tenth aspect and any implementation thereof correspond to the fourth aspect and any implementation thereof, respectively. The technical effects corresponding to the tenth aspect and any implementation thereof are similar to those corresponding to the fourth aspect and any implementation thereof, and will not be repeated here.

[0095] Eleventhly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to execute the encoding method in the second aspect or any possible implementation thereof.

[0096] The eleventh aspect and any implementation thereof correspond to the second aspect and any implementation thereof, respectively. The technical effects corresponding to the eleventh aspect and any implementation thereof can be found in the technical effects corresponding to the second aspect and any implementation thereof, as described above, and will not be repeated here.

[0097] In a twelfth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to perform the decoding method in the third aspect or any possible implementation thereof.

[0098] The twelfth aspect and any implementation thereof correspond to the third aspect and any implementation thereof, respectively. The technical effects corresponding to the twelfth aspect and any implementation thereof are similar to those corresponding to the third aspect and any implementation thereof, and will not be repeated here.

[0099] In a thirteenth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to execute a preset network training method in the fourth aspect or any possible implementation thereof.

[0100] The thirteenth aspect and any implementation thereof correspond to the fourth aspect and any implementation thereof, respectively. The technical effects corresponding to the thirteenth aspect and any implementation thereof can be found in the above-mentioned fourth aspect and any implementation thereof, and will not be repeated here.

[0101] In a fourteenth aspect, embodiments of this application provide a computer program product, which includes a software program that, when executed by a computer or processor, causes the computer or processor to perform the encoding method in the second aspect or any possible implementation thereof.

[0102] The fourteenth aspect and any implementation thereof correspond to the second aspect and any implementation thereof, respectively. The technical effects corresponding to the fourteenth aspect and any implementation thereof are similar to those corresponding to the second aspect and any implementation thereof, and will not be repeated here.

[0103] In a fifteenth aspect, embodiments of this application provide a computer program product, which includes a software program that, when executed by a computer or processor, causes the computer or processor to perform the decoding method in the third aspect or any possible implementation thereof.

[0104] The fifteenth aspect and any implementation thereof correspond to the third aspect and any implementation thereof, respectively. The technical effects corresponding to the fifteenth aspect and any implementation thereof are similar to those corresponding to the third aspect and any implementation thereof, and will not be repeated here.

[0105] In a sixteenth aspect, embodiments of this application provide a computer program product, which includes a software program that, when executed by a computer or processor, causes the computer or processor to perform a preset network training method in the fourth aspect or any possible implementation thereof.

[0106] The sixteenth aspect and any implementation thereof correspond to the fourth aspect and any implementation thereof, respectively. The technical effects of the sixteenth aspect and any implementation thereof can be found in the technical effects of the fourth aspect and any implementation thereof, as described above, and will not be repeated here.

[0107] In a seventeenth aspect, embodiments of this application provide an apparatus for storing a bitstream, the apparatus comprising: a receiver and at least one storage medium, the receiver being used to receive the bitstream; at least one storage medium being used to store the bitstream; the bitstream being generated according to the second aspect and any implementation thereof.

[0108] The seventeenth aspect and any implementation thereof correspond to the second aspect and any implementation thereof, respectively. The technical effects corresponding to the seventeenth aspect and any implementation thereof are described above in the section on the second aspect and any implementation thereof, and will not be repeated here.

[0109] In an eighteenth aspect, embodiments of this application provide an apparatus for transmitting a bitstream, the apparatus comprising: a transmitter and at least one storage medium, the at least one storage medium being used to store the bitstream, the bitstream being generated according to the second aspect and any implementation thereof; the transmitter being used to obtain the bitstream from the storage medium and transmit the bitstream to an end-side device via the transmission medium.

[0110] The eighteenth aspect and any implementation thereof correspond to the second aspect and any implementation thereof, respectively. The technical effects corresponding to the eighteenth aspect and any implementation thereof are similar to those corresponding to the second aspect and any implementation thereof, and will not be repeated here.

[0111] Nineteenthly, embodiments of this application provide a system for distributing bitstreams, the system comprising: at least one storage medium for storing at least one bitstream, the at least one bitstream being generated according to the second aspect and any implementation thereof; and a streaming media device for obtaining a target bitstream from the at least one storage medium and sending the target bitstream to an end-side device, wherein the streaming media device includes a content server or a content distribution server.

[0112] The nineteenth aspect and any implementation thereof correspond to the second aspect and any implementation thereof, respectively. The technical effects corresponding to the nineteenth aspect and any implementation thereof are similar to those corresponding to the second aspect and any implementation thereof, and will not be repeated here.

[0113] In a twentieth aspect, embodiments of this application provide an image processing method, the method comprising: first, obtaining an image to be encoded and the number of times the image to be encoded has been encoded, wherein the image to be encoded is an unencoded image or a reconstructed image; and selecting a target encoding network from N encoding networks based on the number of times the image to be encoded has been encoded, wherein the encoding losses of the N encoding networks are different, and N is an integer greater than 1; then, inputting the image to be encoded into the target encoding network to obtain a first feature map output by the target encoding network; subsequently, quantizing the first feature map to obtain a second feature map; and entropy encoding the second feature map to obtain a bitstream; then, entropy decoding the bitstream to obtain a third feature map; and subsequently, inputting the third feature map into a decoding network to obtain a reconstructed image output by the decoding network.

[0114] According to aspect 20, selecting a target coding network from N coding networks based on the number of times the image to be encoded has been encoded includes: obtaining preset network selection conditions; and selecting a target coding network from N coding networks based on the number of times the image to be encoded has been encoded and the preset network selection conditions.

[0115] According to aspect 20, or any implementation of aspect 20 above, the preset network selection conditions include: when the number of encoding times is 0, selecting the encoding network with the smallest difference between the compressed bit rate and the target compressed bit rate; wherein, the compressed bit rate is related to the encoding loss.

[0116] According to aspect 20, or any implementation of aspect 20 above, the preset network selection conditions include: when the number of times it has been encoded is not 0, selecting a coding network whose coding loss is less than the historical coding loss, where the historical coding loss refers to the coding loss of the coding network selected in the previous encoding of the image to be encoded.

[0117] According to aspect 20, or any implementation thereof, N coding networks share a first coding network; each of the N coding networks consists of the first coding network and a first gain vector, and the first gain vector of each of the N coding networks is different; inputting the image to be encoded into the target coding network to obtain the first feature map output by the target coding network includes: inputting the image to be encoded into the first coding network to obtain a fourth feature map; scaling the fourth feature map based on the first gain vector of the target coding network to obtain the first feature map.

[0118] According to aspect 20, or any implementation of aspect 20 above, when N equals 2, the N encoding networks include a second encoding network and a third encoding network. The second encoding network is a non-reversible neural network, the third encoding network is a reversible neural network mapped according to a first preset direction, and the decoding network is a reversible neural network mapped according to a second preset direction. The preset network selection conditions include: when the number of times the encoding has been performed is 0, the second encoding network is selected; when the number of times the encoding has been performed is not 0, the third encoding network is selected.

[0119] According to aspect 20, or any implementation thereof, the method further includes: encapsulating the encoded number of times into the bitstream.

[0120] According to aspect 20, or any implementation thereof, after obtaining the reconstructed image output by the decoding network, the method further includes: updating the number of times the entropy decoding has been encoded.

[0121] According to aspect 20, or any implementation thereof, the method further includes: selecting a target gain vector from N second gain vectors based on the number of times it has been encoded obtained by entropy decoding, wherein the N second gain vectors correspond one-to-one with the N first gain vectors; scaling the third feature map based on the target gain vector to obtain a fifth feature map; and inputting the third feature map into the decoding network to obtain a reconstructed image output by the decoding network, including: inputting the fifth feature map into the decoding network to obtain a reconstructed image output by the decoding network.

[0122] According to aspect 20, or any implementation thereof, the decoding network and N encoding networks are jointly trained. The joint training includes training the N encoding networks and the decoding network in M ​​batches. Each batch of training is based on the loss weight coefficient corresponding to the i-th encoding network and the decoding network. The loss weight coefficients corresponding to each of the N encoding networks are different. Different loss weight coefficients are used to train encoding networks with different encoding losses. M is a positive integer greater than N, and i is an integer between 1 and N.

[0123] The twentieth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, or to the second aspect and any implementation thereof, or to the third aspect and any implementation thereof. The technical effects corresponding to the twentieth aspect and any implementation thereof can be found in the technical effects corresponding to the first aspect and any implementation thereof, or in the technical effects corresponding to the second aspect and any implementation thereof, or in the technical effects corresponding to the third aspect and any implementation thereof; further details are omitted here. Attached Figure Description

[0124] Figure 1a A schematic diagram of the main framework of artificial intelligence is shown as an example;

[0125] Figure 1b This is a schematic diagram illustrating an application scenario;

[0126] Figure 1c This is a schematic diagram illustrating an application scenario;

[0127] Figure 2a This is a schematic diagram of an image compression system framework, as exemplarily shown.

[0128] Figure 2b This is a schematic diagram illustrating a preset network training process as an example.

[0129] Figure 3a This is a schematic diagram of the network structure of an exemplary coding network;

[0130] Figure 3b This is a schematic diagram of an image compression system framework, as exemplarily shown.

[0131] Figure 4a This is a schematic diagram of the network structure of an exemplary coding network;

[0132] Figure 4b This is a schematic diagram of an image compression system framework, as exemplarily shown.

[0133] Figure 5a This is a schematic diagram of the network structure of an exemplary coding network;

[0134] Figure 5b This is a schematic diagram of an image compression system framework, as exemplarily shown.

[0135] Figure 6 This is a schematic diagram illustrating the encoding process as an example.

[0136] Figure 7This is a schematic diagram illustrating the decoding process as an example.

[0137] Figure 8 This is a schematic diagram illustrating the encoding process as an example.

[0138] Figure 9 This is a schematic diagram illustrating the decoding process as an example.

[0139] Figure 10 This is a schematic diagram illustrating the encoding process as an example.

[0140] Figure 11 This is a schematic diagram illustrating the decoding process as an example.

[0141] Figure 12 This is a schematic diagram of the structure of an exemplary device. Detailed Implementation

[0142] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0143] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0144] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0145] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0146] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0147] Figure 1aThis is an exemplary schematic diagram of the main framework of artificial intelligence, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence domain needs.

[0148] The above-mentioned artificial intelligence framework will be elaborated from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0149] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0150] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of artificial intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0151] (1) Infrastructure:

[0152] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0153] (2) Data

[0154] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0155] (3) Data processing

[0156] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0157] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0158] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0159] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0160] (4) General ability

[0161] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, texture mapping generation, etc.

[0162] (5) Smart Products and Industry Applications

[0163] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, and intelligent terminals.

[0164] The preset networks involved in this application can be used to implement machine learning, deep learning, search, reasoning, decision-making, etc. The preset networks mentioned in this application can include various types of neural networks, such as deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), residual networks, neural networks using transformer models, or other neural networks, etc., and this application does not impose any limitations on them.

[0165] For example, a neural network may include multiple network layers, wherein the operation of each network layer in the neural network can be expressed by a mathematical expression. To describe it: From a physical perspective, the work of each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are... The operation 4 is completed using +b, and the operation 5 is implemented using a. The term "space" is used here because the objects being classified are not individual things, but a class of things; space refers to the set of all individuals within this class of things. Here, W is the weight vector, where each value represents the weight of a neuron in that network layer. b is the bias vector, where each value represents the bias of a neuron in that network layer. These vectors W and b determine the spatial transformation from the input space to the output space, meaning that the weights W of each layer control how the space is transformed.

[0166] The goal of training a neural network is to ultimately obtain the weight matrices of all layers in the trained network (the weight matrix formed by the vectors W of many network layers and the bias matrix formed by the vectors b of many network layers; the weight matrix and bias matrix can be called network parameters). Therefore, the training process of a neural network is essentially learning how to control the transformation space, more specifically, learning the weight matrix and bias matrix.

[0167] For example, this application can be applied to AI compression scenarios. The image compression system of this application can refer to an AI compression system, and the encoding / decoding method and image processing method implemented based on the image compression system can refer to an AI compression algorithm. The image compression system of this application can be used for multiple encoding / decoding operations.

[0168] Figure 1b This is a schematic diagram illustrating an application scenario.

[0169] Figure 1b This illustrates a scenario where video is forwarded and displayed between multiple electronic devices. For example, a specific application scenario could be: User 1 records a video using electronic device 1 and shares it with User 2; User 2 plays the video using electronic device 2 and then forwards it to User 3; User 3 plays the video using electronic device 3. During the forwarding and display of the video between multiple electronic devices, multiple encoding and decoding processes are performed, as detailed below:

[0170] Reference Figure 1b For example, the video acquisition module of electronic device 1 acquires video data 1, and then outputs the acquired video data 1 to the video encoding module; where video data 1 is the raw video data without encoding or decoding. Next, the video encoding module encodes the video data 1 to obtain bitstream 1, and outputs bitstream 1 to the sending module. Subsequently, the sending module sends bitstream 1 to electronic device 2.

[0171] Continue to refer to Figure 1bFor example, after receiving stream 1, the receiving module of electronic device 2 can output stream 1 to the video decoding module. Then, the video decoding module decodes stream 1 to obtain video data 2 and outputs video data 2 to the display module for display. After displaying video data 2, the display module can output video data 2 to the video encoding module; subsequently, the video encoding module can encode video data 2 to obtain stream 2 and output stream 2 to the sending module. Then, the sending module sends stream 2 to electronic device 3.

[0172] Continue to refer to Figure 1b For example, after receiving the bitstream 2, the receiving module of electronic device 3 can output the bitstream 2 to the video decoding module. Then, the video decoding module decodes the bitstream 2 to obtain video data 3 and outputs the video data 3 to the display module for display.

[0173] It should be understood that, Figure 1b The electronic device described is merely one example of this application; the electronic device described in this application may include more than [examples of other electronic devices]. Figure 1b The application does not limit the number of modules shown, whether more or fewer.

[0174] It should be understood that electronic device 3 can also encode video data 3 into a bitstream and forward it to electronic device 4, which will then decode and play it, and so on. This application does not limit the number of times video data can be forwarded, that is, it does not limit the number of times video data can be encoded and decoded.

[0175] Figure 1c This is a schematic diagram illustrating an application scenario. Figure 1c The image shows a scenario where a video has been edited multiple times; in such a scenario, multiple encoding and decoding processes are performed.

[0176] Reference Figure 1c For example, video editing software can be used to edit video data 1 (raw video data without encoding or decoding) to obtain video data 2; then, the video editing software can encode video data 2 and store it; wherein, video encoding of video data 2 can obtain bitstream 1 and store bitstream 1.

[0177] Continue to refer to Figure 1c For example, when video data 2 needs to be edited, video stream 1 can be decoded to obtain video data 3; then, video editing software can be used to edit video data 3 to obtain video data 4. Subsequently, the video editing software can encode video data 4 and store it; wherein, video encoding of video data 4 can obtain and store video stream 2.

[0178] Continue to refer to Figure 1cFor example, when video data 4 needs to be edited, video stream 2 can be decoded to obtain video data 5; then, video editing software can be used to edit video data 5 to obtain video data 6. Subsequently, the video editing software can encode video data 6 and store it; wherein, video encoding of video data 6 can obtain and store video stream 3.

[0179] It should be understood that video editing can continue on the video data obtained from decoded bitstream 3, and then the edited video data can be encoded and stored, and so on; this application does not limit the number of times the video data can be edited, that is, it does not limit the number of times the video data can be encoded and decoded.

[0180] It should be understood. Figure 1b and Figure 1c The electronic devices included can be of various types, including but not limited to: personal computers, computer workstations, smartphones, tablets, servers, smart cameras, smart cars or other types of cellular phones, media consumption devices, wearable devices, set-top boxes, game consoles, etc.

[0181] Figure 2a This is a schematic diagram of the framework of an image compression system as an example.

[0182] Reference Figure 2a For example, an image compression system includes: a first selection module, N encoding networks, a quantization module, an entropy encoding module, an entropy estimation module, an entropy decoding module, and a decoding network.

[0183] It should be noted that the number of entropy coding modules can be Y (where Y is a positive integer). For example, Y can be equal to 1, in which case N coding networks correspond to one entropy coding module. For example, Y can be equal to N, in which case N coding networks correspond to N entropy coding networks. For example, Y can be greater than 1 and less than N, in which case one or more of the N coding networks correspond to one entropy coding module.

[0184] It should be noted that the number of entropy decoding modules can be Y (where Y is a positive integer); that is, the number of entropy decoding modules is the same as the number of entropy encoding modules, and there is a one-to-one correspondence between Y entropy decoding modules and Y entropy encoding modules. This application uses Y equal to 1 as an example for illustration.

[0185] It should be understood that, Figure 2a This is merely one example of the image compression system described in this application; the image compression system described in this application may include systems that compress images more than... Figure 2a The application does not limit the number of modules / networks shown.

[0186] Reference Figure 2aFor example, the N encoding networks can be: encoding network G1, encoding network G2, ..., encoding network G2, ..., encoding network GN. Here, i is an integer between 1 and N (inclusive).

[0187] For example, an encoding network (including encoding networks G1 to GN) can be used to perform feature transformation on an image, transforming the image to another space; its input is an image, and its output is a feature map. For example, the encoding network can be a convolutional neural network.

[0188] For example, the image input to the encoding network can be any one of RAW (raw) image, RGB (Red Green Blue) image, and YUV ("Y" represents luminance, "U" and "V" represent chrominance, "Chroma") image, and this application does not limit it.

[0189] For example, a quantization module can be used to perform quantization processing.

[0190] For example, an entropy estimation module can be used for entropy estimation and output entropy estimation information.

[0191] For example, an entropy coding module can be used to perform entropy coding based on entropy estimation information; its input is a feature map, and its output is a bitstream.

[0192] For example, entropy coding refers to coding that does not lose any information during the coding process according to the entropy principle. Entropy coding includes various types, such as Shannon coding, Huffman coding, and arithmetic coding, etc. This application does not limit the types of coding.

[0193] For example, an entropy decoding module can be used to perform entropy decoding based on entropy estimation information; its input is a bitstream and its output is a feature map.

[0194] For example, a decoding network can be used to perform feature transformation on the feature map obtained by entropy decoding (the reverse of the feature transformation process of an encoding network); its input is the feature map, and its output is the reconstructed image. For example, the decoding network can be a convolutional neural network.

[0195] For example, N encoding networks and one decoding network can be jointly trained in advance to ensure that the decoding network is well compatible with the N encoding networks (i.e., for each of the N encoding networks, after encoding the image based on that encoding network and decoding the encoded result based on the decoding network, the rate-distortion loss of the image is less than a preset rate-distortion loss. The preset rate-distortion loss can be set as needed, and this application does not impose any restrictions on it).

[0196] The following Figure 2a Based on the image compression system framework shown, the training process of the preset network (which includes N encoding networks and 1 decoding network, and the image compression system includes the preset network) is explained.

[0197] Figure 2b This is a schematic diagram illustrating a preset network training process as an example. Figure 2b The description is based on Figure 2a The training process of a pre-defined network based on this.

[0198] S201, Obtain training images.

[0199] In one possible approach, images acquired by the image acquisition module or downloaded from an image server can be used as training images.

[0200] In one possible approach, video data can be captured or downloaded from a video server; then, the image sequence included in the captured or downloaded video data can be used as training images.

[0201] For example, the training images may include multiple images, and this application does not limit the number of training images obtained or the method of obtaining them.

[0202] S202, train the preset network in M ​​batches based on the training images.

[0203] For example, M is an integer greater than N. Specifically, after training the preset network in M ​​batches, for each of the N encoding networks, after encoding the image based on that encoding network and decoding the encoding result based on the decoding network, the rate-distortion loss of the image is less than the preset rate-distortion loss.

[0204] For example, the training images can be divided into M batches, where a batch of training images may include one or more images. Then, a batch of training images is used to train the preset network.

[0205] For example, during each batch training process of the preset network, a batch of training images can be input to the first selection module, which selects one encoding network for training from N encoding networks; then, based on this batch of training images, the selected encoding network and decoding network are trained.

[0206] For example, the first selection module can select one encoding network for training from N encoding networks according to a preset training rule. The preset training rule can be set as needed, for example, selecting in turn; or randomly; etc., and this application does not limit this.

[0207] For ease of explanation, the encoding network selected from N encoding networks for training can be called the i-th encoding network (where i is an integer between 1 and N, inclusive) (e.g., ...). Figure 2a The following describes the process of batch training of a preset network by training the i-th encoding network (i.e., encoding network Gi) and the decoding network. Refer to S2021-S2022 for details.

[0208] S2021, the determination rate-distortion loss is based on the training image, the reconstructed image, and the loss weight coefficients corresponding to the i-th encoding network; wherein, the reconstructed image is obtained by encoding the training image using the i-th encoding network and then decoding the encoding result using the decoding network.

[0209] The following explanation uses a single training image from a batch of training images as an example.

[0210] Refer again Figure 2a For example, the first selection module can select the training image (i.e. Figure 2aImage 1) is input to the encoding network Gi, which performs feature transformation on the training image and outputs feature map 1 to the quantization module and entropy estimation module. Next, the quantization module quantizes feature map 1 to obtain feature map 2; and the entropy estimation module performs entropy estimation, outputting the entropy estimation information of the feature points contained in feature map 1 to the entropy encoding module and entropy decoding module. (It should be noted that feature map 1 and feature map 2 have the same size. The feature value of the feature point at position (x,y) in feature map 2 is the quantized result of the feature value of the feature point at position (x,y) in feature map 1; therefore, the entropy estimation information of the feature points contained in feature map 1 is also the entropy estimation information of the feature points contained in feature map 2.) For example, the quantization module can input feature map 2 to the entropy encoding module, which performs entropy encoding on the feature points contained in feature map 2 based on the entropy estimation information of the feature points contained in feature map 1, thus obtaining a bitstream; then, the bitstream is output to the entropy decoding module. Next, the entropy decoding module can perform entropy decoding on the bitstream based on the entropy estimation information of all feature points contained in feature map 1, and output feature map 3 to the decoding network (the decoding network is a decoding network trained in the previous H-1 batches, where H is an integer between 1 and M (the value of H includes 1 and M); the current training is the Hth batch). After that, the decoding network can perform feature transformation on feature map 3 to obtain image 2 (that is, the reconstructed image).

[0211] It should be understood that feature map 2 output by the quantization module can also be used to replace feature map 1 as input to the entropy estimation module. The entropy estimation module determines the entropy estimation information of the feature points contained in feature map 2 and outputs the entropy estimation information of the feature points contained in feature map 2 to the entropy encoding module and the entropy decoding module. This application does not limit this.

[0212] For example, a corresponding loss weight coefficient can be pre-set for each of the N encoding networks; wherein the loss weight coefficients for each of the N encoding networks are different; and different loss weight coefficients can be used to train encoding networks with different encoding losses. Then, after obtaining the reconstructed image output by the decoding network, the rate-distortion loss can be determined based on the training image, the reconstructed image, and the loss weight coefficients corresponding to the i-th encoding network.

[0213] In one possible approach, the bitrate loss corresponding to encoding the training image based on the i-th encoding network can be obtained; and the image distortion loss can be determined based on the training image and the reconstructed image; then, the bitrate loss and the image distortion loss can be weighted and calculated based on the loss weight coefficients corresponding to the i-th encoding network to obtain the rate-distortion loss.

[0214] For example, the entropy estimation module can determine the bit rate loss based on the entropy estimation information after obtaining the entropy estimation information, which can be represented by Loss1.

[0215] For example, the formula for calculating Loss1 can be shown in formula (1) below:

[0216]

[0217] Among them, S i1 H1 represents the entropy estimation information for the i1th feature point in feature map 2 (or feature map 1), where H1 is the total number of feature points in feature map 2 (or feature map 1). H1 is a positive integer, and i1 is an integer between 1 and H1 (inclusive).

[0218] For example, the entropy estimation module can be a probability estimation network, which can use deep learning-based convolutional neural networks, etc., and this application is not limited to this. For example, the probability estimation network can include various types, such as a Gaussian single model (GSM) or a Gaussian mixture model (GMM), where the corresponding entropy estimation information (also referred to as probability distribution information) includes the mean μ and variance σ. It can also be a Laplace distribution model, where the corresponding entropy estimation information (also referred to as probability distribution information) includes the location parameter μ and the scale parameter b. This application takes a Gaussian model as an example; for feature map 2 or feature map 1, its probability distribution function is shown in Equation 2.

[0219]

[0220] in, For feature map 2 (or feature map 1).

[0221] For example, based on the training image and the reconstructed image, the image distortion degree of the reconstructed image relative to the training image can be calculated to obtain the image distortion loss, which can be represented by Loss2.

[0222] In one possible approach, the formula for calculating Loss2 can be shown in formula (3) below:

[0223]

[0224] Where H2 is the number of pixels contained in the reconstructed image or training image; Y 2i It refers to the pixel at the i2th position in the reconstructed image (that is, the pixel value of the pixel at the i2th position in the reconstructed image); H2 refers to the pixel at position i2 in the training image (that is, the pixel value of the pixel at position i2 in the training image); H2 is a positive integer, and i2 is an integer between 1 and H2 (inclusive).

[0225] In one possible approach, Loss2 could also be the peak signal-to-noise ratio (PSNR) between the training and reconstructed images.

[0226] It should be understood that this application does not limit the calculation method of Loss2.

[0227] For example, the loss weight coefficients corresponding to each coding network may include a first weight coefficient and a second weight coefficient, wherein the first weight coefficient corresponds to the bit rate loss and the second weight coefficient corresponds to the image distortion loss.

[0228] In one possible approach, the first weight coefficients and the second weight coefficients of the N encoding networks are different.

[0229] In one possible approach, the first weight coefficients of the N encoding networks are the same, but the second weight coefficients are different.

[0230] In one possible approach, the first weight coefficients of the N coding networks are different, while the second weight coefficients are the same. For example, the first weight coefficients of the N coding networks are all Lagrange coefficients β with different values, and the second weight coefficient is 1.

[0231] For example, the loss can be calculated based on the bit rate loss Loss1 and the image distortion loss Loss2, referring to the following formula (4):

[0232] Li=β*Loss1+Loss2(4)

[0233] Where Li refers to the loss corresponding to the i-th batch of training.

[0234] S2022, based on rate-distortion loss, train the i-th encoder and decoder networks.

[0235] For example, after determining the rate-distortion loss, backpropagation can be performed based on the rate-distortion loss to adjust the network parameters of the i-th encoding network and the decoding network trained in the previous H-1 batches, thereby enabling the training of the i-th encoding network and the decoding network trained in the previous H-1 batches.

[0236] It should be understood that the loss weight coefficients corresponding to the N encoding networks are different, and the rate-distortion loss determined during the training process of each encoding and decoding network is also different; the rate-distortion loss is related to the encoding loss of the encoding network (for example, the greater the rate-distortion loss during training, the greater the encoding loss of the trained encoding network); therefore, after training the N encoding networks (that is, after training the M batches), the encoding loss of these N encoding networks will also be different.

[0237] For example, the larger the β corresponding to the encoding network, the larger the loss Li, and the larger the encoding loss of the encoding network.

[0238] It should be noted that the N encoding networks share the same decoding network. During the training of the preset network in a batch, the network parameters of the decoding network in the forward computation process (that is, the process of obtaining the reconstructed image based on the training image mentioned above) are the network parameters obtained after training in the previous H-1 batches; and the network parameters of the decoding network adjusted in the backpropagation process are also the network parameters obtained after training in the previous H-1 batches.

[0239] Thus, after training the preset network in M ​​batches, for each of the N encoding networks, encoding the image based on that encoding network and decoding the encoded result based on the decoding network results in a rate-distortion loss that is less than the preset rate-distortion loss. In other words, the trained decoding network can effectively accommodate multiple encoding networks with different encoding losses; therefore, its reconstruction capability is higher than that of existing technologies using decoding networks with a single encoding network. Consequently, subsequent image compression systems incorporating this preset network can effectively reduce the magnitude of image loss after multiple encoding and decoding operations.

[0240] In one possible approach, N encoding networks have the same network structure, but their network parameters differ after training. The following explanation uses the structure of a single encoding network as an example.

[0241] Figure 3a This is a schematic diagram illustrating the network structure of an exemplary coding network. It should be understood that... Figure 3a This is merely an example of this application. The encoding network of this application can be implemented using any existing neural network, and this application does not impose any restrictions on it.

[0242] For example, the encoding network can be composed of four convolutional layers and three activation layers interleaved and cascaded. (See reference...) Figure 3a For example, the activation layer can be a Generalized Divisive Normalization (GDN) layer; Figure 3a In this context, "Conv" means convolution, and "192×5×5 / 2↓" indicates that the kernel size of each convolutional layer is 5x5, the number of channels in the output feature map is 192, and the width and height of each convolutional layer are downsampled by a factor of 2 (i.e., the convolution stride is 2).

[0243] It should be understood that, Figure 3aThe parameters of the convolutional layer are merely an example in this application. This application does not limit the kernel size, number of channels, downsampling factor, downsampling times, etc. of the convolutional layers in the coding network. Furthermore, this application does not limit the number of convolutional layers, activation layers, etc. in the coding network.

[0244] It should be noted that, in Figure 3a In the scenario of the embodiment, the different network parameters of the N encoded networks after training can mean that some or all of the network parameters of the N encoded networks after training are different. For example, the weights and biases of the convolutional kernels of the convolutional layers are different, and / or the parameters of the activation layers are different, etc.

[0245] Figure 3b This is a schematic diagram illustrating an exemplary image compression system framework. Figure 3b Is Figure 3a and Figure 2a Based on this, the framework of the image compression system is shown.

[0246] For example, Figure 3b There are N coding networks (coding network G1, coding network G2, ..., coding network Gi, ..., coding network GN) with the same network structure but different network parameters.

[0247] In this way, the structures of N coding networks are all the same, which can save the cost of designing coding networks.

[0248] Figure 4a This is a schematic diagram illustrating the network structure of an exemplary coding network. Figure 4a In this model, N encoding networks share a first encoding network. Each of the N encoding networks consists of this first encoding network and a first gain vector (GainVector). The first gain vectors of each of the N encoding networks are different. For example, the dimension of each first gain vector is the same as the number of output channels of the first encoding network.

[0249] Reference Figure 4a For example, the N first gain vectors are: Gain Vector_A1, Gain Vector_A2, ..., Gain Vector_Ai, ..., Gain Vector_AN. Among them, Gain Vector_A1, Gain Vector_A2, ..., Gain Vector_Ai, ..., Gain Vector_AN are different, but have the same number of dimensions.

[0250] Continue to refer to Figure 4aFor example, the first encoding network and Gain Vector_A1 constitute encoding network G1; the first encoding network and Gain Vector_A2 constitute encoding network G2; ...; the first encoding network and Gain Vector_Ai constitute encoding network Gi; ...; the first encoding network and Gain Vector_AN constitute encoding network GN.

[0251] For example, Figure 4a The network structure of the first coding network in the embodiment can be referred to Figure 3a Of course, other neural networks can also be used, and this application does not limit them.

[0252] It should be understood that, in Figure 4a In the scenario of the embodiment, the different network parameters of the N encoded networks after training can mean that some network parameters are different and some network parameters are the same among the N encoded networks after training.

[0253] For example, when the encoding network is Figure 4a In the structure shown, the preset network may further include N second gain vectors, each corresponding one-to-one with the first gain vector. In one possible approach, when the feature map is magnified based on the first gain vector, the feature map is shrunk based on the second gain vector; conversely, when the feature map is shrunk based on the first gain vector, the feature map is magnified based on the second gain vector.

[0254] In one possible approach, each element in the second gain vector is the reciprocal of the corresponding element in the first gain vector. It should be understood that the element in the second gain vector may not be the reciprocal of the corresponding element in the first gain vector; this application does not impose any restrictions on the numerical relationship between the elements in the second gain vector and the corresponding elements in the first gain vector.

[0255] Figure 4b This is a schematic diagram illustrating an exemplary image compression system framework. Figure 4b Is Figure 4a and Figure 2a Based on this, an image compression system framework is shown. Among them, in Figure 4b In addition, the image compression system may also include a second selection module.

[0256] exist Figure 4bIn the given diagram, there are N second gain vectors: GainVector_B1, GainVector_B2, ..., Gain Vector_Bi, ..., GainVector_BN. These N second gain vectors are different, but have the same number of dimensions.

[0257] Reference Figure 4b For example, during a batch training process of a preset network, the process of determining the reconstructed image can be as follows: The first selection module can select the training image (i.e., Figure 4bImage 1) is input to the first encoding network, which transforms the training image and outputs feature map 4. Then, feature map 4 is scaled channel-by-channel based on the Gain Vector_Ai selected by the first selection module to obtain feature map 1, which is then output to the quantization module and the entropy estimation module. The first encoding network includes multiple output channels, corresponding to multiple channels of feature map 4. The dimension of GainVector_Ai is the same as the number of output channels of the first encoding network. Furthermore, for the k-th channel (k is an integer between 1 and P, where P is the number of output channels of the first encoding network), the element of the k-th dimension of GainVector_Ai is multiplied by the feature map 4 of the k-th channel to obtain feature map 1 of the k-th channel. In this way, P channels of feature map 1 can be obtained. Next, the quantization module quantizes feature map 1 (including P channels) to obtain feature map 2 (also including P channels); and the entropy estimation module performs entropy estimation, outputting the entropy estimation information of the feature points contained in feature map 1 to the entropy encoding module and the entropy decoding module. For example, the quantization module can output feature map 2 to the entropy encoding module, which entropy-encodes the feature points in feature map 2 based on the entropy estimation information of the feature points contained in feature map 1, thus obtaining a bitstream; then, the bitstream is output to the entropy decoding module. Next, the entropy decoding module can entropy-decode the bitstream based on the entropy estimation information of all feature points contained in feature map 1, obtaining feature map 3, and outputting feature map 3 to the second selection module. Then, the second selection module can select one second gain vector (GainVector_Bi) for training from N second gain vectors. For example, the second selection module can select one second gain vector (such as GainVector_Bi) for training from N second gain vectors according to a preset training rule. Subsequently, the second selection module can scale feature map 3 based on GainVector_Bi (similar to the process of scaling feature map 4 based on GainVector_Ai, which will not be repeated here) to obtain feature map 5, and output feature map 5 to the decoding network (the decoding network is a decoding network trained in the previous H-1 batches). Then, the decoding network can transform feature map 5 to obtain the reconstructed image (i.e., Figure 4b Image 2 in the image.

[0258] In this way, all N coding networks have the same structure, which saves on the cost of designing coding networks. In addition, compared to the case where N coding networks have different network parameters but the same structure, it can save on the storage memory of the coding networks.

[0259] In one possible approach, the network structures of the N coding networks are different.

[0260] In one possible approach, N=2. In this case, the N encoding networks include a second encoding network and a third encoding network. The second encoding network is a non-reversible encoding network, and the third encoding network is a reversible neural network mapped according to a first preset direction.

[0261] Figure 5a This is a schematic diagram of the structure of an exemplary coding network.

[0262] Reference Figure 5a (1), Figure 5a (1) is the second coding network; the network structure of the second coding network can be described by referring to Figure 3a The description in the document will not be repeated here. It should be understood that other neural networks can also be used as the second encoding network, and this application does not impose any restrictions on this.

[0263] Reference Figure 5a (2), Figure 5a (2) A reversible neural network. The reversible neural network may include four blocks (e.g., Block1, Block2, Block3, and Block4), an average layer, and a repeat layer. For example, each block may include three layers: a pixel shuffling layer (used to transfer spatial information to channels), an invertible convolutional layer (used for convolution), and coupling layers (used for rearranging elements). For example, the average layer is used for channel averaging; the repeat layer is used for channel replication. The pixel shuffling layer of Block1 can be referred to as the first network layer of the reversible neural network; the average layer and the repeat layer can be referred to as the Rth network layers of the reversible neural network.

[0264] It should be noted that, Figure 5a (2) The reversible neural network is both a third encoding network and a decoding network; when the reversible neural network maps according to the first preset direction (the first prediction direction can refer to the direction with the image to be encoded as input and the feature map as output, such as...), Figure 5a (2) When the direction of the right arrow in the reversible neural network is (i.e., the first network layer of the reversible neural network (i.e., the PixelShuffling layer of Block 1) is used as the input layer and the Rth network layer of the reversible neural network (i.e., the Average layer) is used as the output layer, the reversible neural network is the third encoding network. When the reversible neural network maps according to the second preset direction (the second prediction method refers to the direction with the feature map as input and the reconstructed image as output, such as...), the reversible neural network is the third encoding network. Figure 5a(2) When the left arrow of the reversible neural network is in the direction of the reversible neural network, that is, when the first network layer of the reversible neural network (i.e., the PixelShuffling layer of Block1) is the output layer and the Rth network layer of the reversible neural network (i.e., the Repeat layer) is the input layer, the reversible neural network is a decoding network.

[0265] It should be understood that, Figure 5a The reversible neural network described in this application is merely an example; the reversible neural network may include more than [examples of other neural networks]. Figure 5a Showing more or fewer blocks; each block can include more than Figure 5a The number of more or fewer layers shown, or each block may include more or fewer layers. Figure 5a The different network layers shown are not limited in this application.

[0266] Figure 5b This is a schematic diagram illustrating an exemplary image compression system framework. Figure 5b Is Figure 5a and Figure 2a Based on this, the framework of the image compression system is shown.

[0267] Reference Figure 5b For example, when performing batch training on the second encoder and decoder networks, the batch training process can be as follows: the training images (i.e., Figure 5b Image 1) is input to the second encoding network, which transforms the training image and outputs feature map 1 to the quantization module and entropy estimation module. The quantization module then quantizes feature map 1 to obtain feature map 2; and the entropy estimation module performs entropy estimation, outputting the entropy estimation information of the feature points contained in feature map 1 to the entropy encoding module and entropy decoding module. For example, the quantization module can output feature map 2 to the entropy encoding module, which entropy-encodes the feature points in feature map 2 based on the entropy estimation information of the feature points contained in feature map 1, obtaining a bitstream; then the bitstream is output to the entropy decoding module. Next, the entropy decoding module entropy-decodes the bitstream based on the entropy estimation information of all feature points contained in feature map 1, outputting feature map 3 to the reversible neural network (i.e., the decoding network) mapped according to the second prediction direction. Specifically, feature map 3 is output to the Repeat layer of the reversible neural network, and after processing through the Repeat layer, Block 4, Block 3, Block 2, and Block 1, the PixelShuffling layer of Block 1 outputs the reconstructed image (i.e., the image obtained from the encoding module). Figure 5b Image 2 in the image.

[0268] Reference Figure 5bFor example, when performing batch training on the third encoder and decoder networks, the batch training process can be as follows: the training images (i.e., Figure 5b Image 1) is input to a reversible neural network (i.e., the third encoding network) mapped according to the first prediction direction. Specifically, the training image is input to the PixelShuffling layer of Block 1 of the reversible neural network, and then processed sequentially through Block 1, Block 2, Block 3, Block 4, and the Average layer. The Average layer outputs feature map 1 to the quantization module and the entropy estimation module. Next, the quantization module quantizes feature map 1 to obtain feature map 2; and the entropy estimation module performs entropy estimation, outputting the entropy estimation information of the feature points contained in feature map 1 to the entropy encoding module and the entropy decoding module. For example, the quantization module can output feature map 2 to the entropy encoding module, which entropy-encodes the feature points contained in feature map 2 based on the entropy estimation information of the feature points contained in feature map 1, thus obtaining a bitstream; then, the bitstream is input to the entropy decoding module. Next, the entropy decoding module can perform entropy decoding on the bitstream based on the entropy estimation information of all feature points contained in feature map 1, and output feature map 3 to the reversible neural network (i.e., the decoding network) mapped according to the second prediction direction. That is, feature map 3 is output to the Repeat layer of the reversible neural network, and after being processed by the Repeat layer, Block 4, Block 3, Block 2 and Block 1 in sequence, the PixelShuffling layer of Block 1 outputs the reconstructed image (i.e., Figure 5b Image 2 in the image.

[0269] The following describes the encoding and decoding process based on an image compression system.

[0270] Figure 6 This is a schematic diagram illustrating an exemplary encoding process. Figure 6 Therefore Figure 2a The example uses an image compression system framework as its basis for the coding process.

[0271] S601, Obtain the image to be encoded; wherein, the image to be encoded is an unencoded image or a reconstructed image.

[0272] For example, the image to be encoded can be any of RAW, RGB, and YUV images, and this application does not limit it.

[0273] For example, the image to be encoded can be an unencoded image (i.e., the original image) or an image that has been encoded once or multiple times (i.e., the reconstructed image), and this application does not impose any restrictions on this. Wherein, after an image is encoded once and decoded once, the number of times the image has been encoded is 1.

[0274] S602, obtain the number of times the image to be encoded has been encoded.

[0275] In one possible approach, the image name of the image to be encoded may include a field that identifies the number of times the image has been encoded; thus, the number of times the image has been encoded can be obtained from the image name of the image to be encoded.

[0276] In one possible approach, the number of times the image to be encoded has been encoded can be stored in a file associated with the image to be encoded, and then the file associated with the image to be encoded can be retrieved to obtain the number of times the image to be encoded has been encoded.

[0277] S603, select the target coding network from N coding networks based on the number of times the image to be encoded has been encoded.

[0278] For example, the preset network includes N coding networks. Each encoding can select one coding network from the N coding networks as the target coding network; then the image to be encoded is encoded based on the target coding network.

[0279] For example, since the encoding losses of these N encoding networks are different, in order to meet the requirements of image distortion and bit rate, the target encoding network can be selected from the N encoding networks according to the number of times the image to be encoded has been encoded.

[0280] For example, preset network selection criteria can be set in advance. This allows a target coding network to be selected from N coding networks based on the number of times the image to be encoded has been encoded and the preset network selection criteria. Specifically, for... Figure 3a , Figure 4a and Figure 5a For different scenarios, corresponding preset network selection conditions can be set separately; details will be explained later.

[0281] For example, during the encoding process, the image to be encoded and the number of times the image to be encoded has been encoded can be input into the first selection module, and the first selection module can obtain the preset network selection conditions; then, the first selection module can select the target encoding network from N encoding networks based on the number of times the image to be encoded has been encoded and the preset network selection conditions.

[0282] It should be noted that the first selection module selects one encoding network from N encoding networks based on different conditions during the training process of the preset network and during the actual application encoding process. During the training process of the preset network, the first selection module selects one encoding network from N encoding networks according to preset training rules, while during the actual application encoding process, it selects one encoding network from N encoding networks according to preset network selection conditions.

[0283] For example, each of the N encoding networks obtained from the above training corresponds to a set of network parameters; thus, N sets of network parameters can be obtained. For example, in the case where the N encoding networks have the same network structure but different network parameters, the image compression system including N encoding networks can also be understood as including only one encoding network with the same structure as any one of the N encoding networks, and this encoding network corresponds to N sets of network parameters. In this case, S603 can also be understood as: selecting target network parameters from the N sets of network parameters based on the number of times the image to be encoded has been encoded; configuring the network parameters of this encoding network as the target network parameters, thus obtaining the target encoding network.

[0284] S604, input the image to be encoded into the target encoding network to obtain the first feature map output by the target encoding network.

[0285] For example, after selecting a target coding network, the first selection module can input the image to be encoded into the target coding network to obtain the first feature map output by the target coding network. (Refer to...) Figure 2a The image to be encoded can refer to image 1, the target encoding network can refer to encoding network Gi, and the first feature map is... Figure 2a Feature diagram 1.

[0286] S605, quantize the first feature map to obtain the second feature map.

[0287] S606, entropy encoding is performed on the second feature map to obtain the bitstream.

[0288] Next, the encoding network Gi can output the first feature map to the quantization module and the entropy estimation module. Then, the quantization module can quantize the first feature map to obtain the second feature map (i.e., Figure 2a The first feature map (2) is used for entropy estimation, and the entropy estimation module outputs the entropy estimation information of the feature points contained in the first feature map to the entropy coding module. For example, the quantization module can output the second feature map to the entropy coding module, and the entropy coding module entropy codes the feature points contained in the second feature map according to the entropy estimation information of the feature points contained in the first feature map to obtain the bitstream.

[0289] Figure 7 This is a schematic diagram illustrating the decoding process as an example. Figure 7 Therefore Figure 2a The decoding process based on the image compression system framework in the embodiment is, and Figure 6 The encoding process corresponds to this.

[0290] S701, acquire bitstream.

[0291] S702 performs entropy decoding on the bitstream to obtain the third feature map.

[0292] For example, after the entropy decoding module obtains the bitstream, it can instruct the entropy estimation module to perform entropy estimation based on the feature values ​​of the decoded feature points to determine the entropy estimation information of the feature points to be decoded; then, it obtains the entropy estimation information of the feature points to be decoded from the entropy estimation module. Next, the entropy decoding module can perform entropy decoding on the feature points to be decoded based on the entropy estimation information, obtaining the feature values ​​of the feature points to be decoded. In this way, after obtaining the feature values ​​of all feature points through entropy decoding, the third feature map can be obtained, which is... Figure 2a Feature diagram 3.

[0293] It should be understood that the third feature obtained by entropy decoding and the second feature map obtained by entropy encoding may be the same or different, and this application does not impose any restrictions on this.

[0294] S703, the third feature map is input into the decoding network to obtain the reconstructed image output by the decoding network.

[0295] For example, after obtaining the third feature map, the third feature map can be input into the decoding network, which will then perform feature transformation and output the reconstructed image.

[0296] Since the decoding network of this application is compatible with a variety of coding networks with different coding losses, its reconstruction capability is higher than that of the decoding network corresponding to the single coding network in the prior art; therefore, decoding based on this decoding network can effectively reduce the loss ratio of the image after multiple encoding and decoding.

[0297] The following are based on Figure 3b , Figure 4b and Figure 5b The image compression system framework described in this application describes the encoding and decoding process.

[0298] Figure 8 This is a schematic diagram illustrating an exemplary encoding process. Figure 8 by Figure 3b Based on the image compression system framework, the encoding process of the image to be encoded is described. Furthermore, in Figure 8 In this embodiment, N = 2.

[0299] S801, acquire the image to be encoded.

[0300] S802, obtain the number of times the image to be encoded has been encoded.

[0301] For example, S801 to S802 can be referred to the description of S601 to S602 above, and will not be repeated here.

[0302] S803, obtain the preset network selection conditions.

[0303] For example, the encoding network in the preset network is as follows: Figure 3a In the scenario shown, the preset network selection conditions can include: when the number of encoding iterations is 0, selecting the encoding network with the smallest difference between the compression bitrate and the target compression bitrate; when the number of encoding iterations is not 0, selecting the encoding network with a coding loss less than the historical coding loss, where the historical coding loss refers to the coding loss of the encoding network selected in the previous encoding of the image to be encoded.

[0304] For example, when N=2, the encoding network includes encoding network G1 and encoding network G2. Assuming that the encoding loss of encoding network G2 is less than that of encoding network G1, and the difference between the compression bitrate and the target compression bitrate of encoding network G1 is minimized, the preset network selection condition can be: when the number of times encoded is 0, select encoding network G1; when the number of times encoded is not 0, select encoding network G2.

[0305] It should be understood that when N is greater than 2, the preset network selection condition "when the number of encodings is not 0, select the encoding network with encoding loss less than the historical encoding loss" can be further subdivided, and this application does not impose any restrictions on this.

[0306] For example, when N=3, the encoding network includes encoding networks G1, G2, and G3. Assume that the encoding loss of encoding network G2 is less than that of encoding network G1, and the encoding loss of encoding network G3 is less than that of encoding network G2; the difference between the compression bitrate and the target compression bitrate of encoding network G1 is minimized. Then, the preset network selection criteria can be: when the number of encoded iterations is 0, select encoding network G1; when the number of encoded iterations is 1, select encoding network G2; when the number of encoded iterations is greater than 1, select encoding network G3.

[0307] For example, when N=10, the encoding network includes encoding networks G1 to G10. Assuming that the encoding loss of encoding networks G1 to G10 decreases sequentially, and the difference between the compression rate of encoding network G1 and the target compression rate is the smallest, the preset network selection conditions can be: when the number of encoded times is 0, select encoding network G1; when the number of encoded times is between 1 and 4, select encoding network G2; when the number of encoded times is between 5 and 8, select encoding network G3; and so on, when the number of encoded times is greater than 33, select encoding network G10.

[0308] In this way, the first selection module can obtain the image to be encoded, the number of times the image has been encoded, and the preset network selection conditions; then, it can select the target encoding network according to the preset network selection conditions; see S804 to S805 below:

[0309] S804: When the number of encoding iterations is 0, select the encoding network with the smallest difference between the compressed bit rate and the target compressed bit rate from the N encoding networks, and use it as the target encoding network.

[0310] For example, the compression bitrate is related to the coding loss. Specifically, for any coding network, the compression bitrate of that network is negatively correlated with its coding loss; that is, the greater the coding loss, the smaller the compression bitrate; conversely, the smaller the coding loss, the greater the compression bitrate. In one possible approach, for one of N coding networks, the compression bitrate can be estimated as follows: the image to be encoded is sequentially input into the coding network, the quantization module, and the entropy estimation module, and the entropy estimation module determines the compression bitrate of the coding network.

[0311] For example, when the first selection module determines that the number of times the image to be encoded has been encoded is 0, it can select the encoding network with the smallest difference between the compressed bitrate and the target compressed bitrate from N encoding networks as the target encoding network.

[0312] S805, when the number of times the encoding has been performed is not 0, select the encoding network with a coding loss less than the historical coding loss from the N encoding networks as the target encoding network. The historical coding loss refers to the coding loss of the encoding network selected in the previous encoding of the image to be encoded.

[0313] For example, in this application, the encoding loss of the N encoding networks is proportional to the rate-distortion loss obtained during the training of these N encoding networks. Therefore, the N encoding networks can be sorted based on the rate-distortion loss of each encoding network during the last batch of training. Thus, when the first selection module determines that the number of times the image to be encoded has been encoded is not zero, it can select the encoding network whose encoding loss is less than the encoding loss of the encoding network selected in the previous encoding of the image to be encoded, according to the order of the N encoding networks, as the target encoding network.

[0314] S806, input the image to be encoded into the target encoding network to obtain the first feature map output by the target encoding network.

[0315] For example, S806 can be described with reference to S604 above, and will not be repeated here.

[0316] S807, quantize the first feature map to obtain the second feature map.

[0317] S808 performs entropy encoding on the second feature map to obtain the bitstream.

[0318] For example, the quantization module quantizes the first feature map to obtain the second feature map, which can then be output to the entropy coding module. Next, the entropy coding module performs entropy coding on the second feature map to obtain its encoded data. This encoded data can then be encapsulated into a Network Abstraction Layer Unit (NALU). The NALU can include a NALU Header and a NALU Body. The NALU Header contains a detailed description of the corresponding data body (i.e., the NALU Body), such as NALU type information (including but not limited to: Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Supplemental Enhancement Information (SEI), I-frames, and P-frames). The entropy coding module can encapsulate the encoded data of the second feature map into the NALU Body; for example, it can encapsulate the encoded data of the second feature map into an I-frame / P-frame type NALU Body to obtain the bitstream.

[0319] S809 encapsulates the encoded number of times into the bitstream.

[0320] For example, the encoded count can also be encapsulated into the bitstream so that the encoded count can be updated during subsequent decoding.

[0321] For example, the entropy encoding module can encapsulate the encoded count into the NALU body. For instance, the encoded count can be encapsulated into the NALU body of an SEI type NALU.

[0322] For example, existing syntax elements can be used to represent the number of times encoded, or new syntax elements can be used to represent the number of times encoded; this application does not limit this.

[0323] Figure 9 This is a schematic diagram illustrating the decoding process as an example. Figure 9 by Figure 3b Based on the image compression system framework in [the document], it describes [the system] with [other components]. Figure 8 The encoding process corresponds to the decoding process.

[0324] S901, acquire bitstream.

[0325] S902 performs entropy decoding on the bitstream to obtain the third feature map and the number of times it has been encoded.

[0326] For example, after receiving the bitstream, the entropy decoding module can decapsulate (or parse) the bitstream, and then obtain the encoded data from the NALU. Figure 8 In this embodiment, the encoded data of the third feature map and the number of times it has been encoded are shown. Next, the entropy decoding module can perform entropy decoding on the encoded data based on the entropy estimation information determined by the entropy estimation module to obtain the third feature map.

[0327] It should be noted that the steps of decapsulating and entropy decoding of the bitstream by the entropy decoding module are both steps in the entropy decoding process.

[0328] S903, the third feature map is input into the decoding network to obtain the reconstructed image output by the decoding network.

[0329] S904, update the number of times the code has been encoded.

[0330] For example, the encoded count can be incremented by 1.

[0331] One possible approach is to replace the value in the field in the image name of the reconstructed image that identifies the number of times the image has been encoded with the updated number of times it has been encoded.

[0332] One possible approach is to replace the encoded counts in the file associated with the reconstructed image with the updated encoded counts.

[0333] For example, an image compression system may also include an update module ( Figure 3b (Not shown), the update module increments the encoded count by 1.

[0334] In other words, when encoding the image to be encoded for the first time, this application uses an encoding network whose compression bitrate is closest to the target compression bitrate. Subsequent decoding uses a decoding network with higher reconstruction capabilities than existing decoding networks. This ensures that, at the same bitrate (in the scenario of a single encoding network in existing technologies, the compression bitrate of the single encoding network is the target compression bitrate; that is, the compression bitrate corresponding to the initial encoding in this application is similar to or the same as the compression bitrate corresponding to the initial encoding in existing technologies), image distortion is low (at this time, the rate-distortion loss of the image is also low). The rate-distortion loss during the initial encoding and decoding determines the upper limit of the rate-distortion performance of the image in subsequent encoding and decoding, which can further improve the rate-distortion performance of subsequent images in multiple encoding and decoding processes.

[0335] Furthermore, in subsequent compression processes, selecting a coding network with a smaller encoding loss than the previous one each time can effectively reduce the magnitude of image loss after multiple encoding and decoding processes.

[0336] Tests have shown that the encoding and decoding method of this application (i.e., using...) Figure 3bThe encoding and decoding method based on the image compression system framework in [the document] outperforms existing encoding and decoding methods by 1.41% in BD-rate (BD-rate is one of the main parameters for evaluating the performance of video coding algorithms). The loss amplitude ratio (where the loss difference is obtained by subtracting the rate distortion loss of the first encoding and decoding from the rate distortion loss of the tenth encoding and decoding; the ratio of the loss difference to the rate distortion loss of the first encoding and decoding is the loss amplitude ratio) is reduced by an average of 8.62% compared with existing technologies.

[0337] For example, when the image compression system framework is as follows Figure 5b As shown, N can be equal to 2, and the N encoding networks can include a second encoding network and a third encoding network; the second encoding network is a non-reversible neural network, and the third encoding network is a reversible neural network mapped according to a first preset direction. In this case, the preset network selection conditions can include: selecting the second encoding network when the number of encoded iterations is 0; and selecting the third encoding network when the number of encoded iterations is not 0.

[0338] For example, in an image compression system framework such as Figure 5b In the scenario shown, during the encoding process, S804 can be replaced with: when the number of encodings is 0, select the second encoding network as the target encoding network; S805 can be replaced with: when the number of encodings is not 0, select the third encoding network as the target encoding network.

[0339] For example, in an image compression system framework such as Figure 5b In the scenario shown, when the second coding network is used as the target coding network, based on Figure 5b The encoding process can be as follows Figure 8 As shown in the examples, they will not be repeated here.

[0340] For example, in an image compression system framework such as Figure 5b In the scenario shown, when the third coding network is used as the target coding network, based on Figure 5b The encoding method can also be like Figure 8 As shown in the embodiment. In particular, the specific implementation process of inputting the image to be encoded into the target encoding network in S805 to obtain the first feature map output by the target encoding network can be as follows: the image to be encoded is input into the PixelShuffling layer of Block1 of the reversible neural network, and then processed by Block1, Block2, Block3, Block4 and Average layers in sequence, and the first feature map is output by the Average layer.

[0341] For example, in an image compression system framework such as Figure 5bIn the scenario shown, whether the second coding network or the third coding network is used as the target coding network, based on Figure 5b The decoding method can also be like Figure 9 As shown in the embodiment. In S903, the specific process of inputting the third feature map into the decoding network to obtain the reconstructed image output by the decoding network can be as follows: The third feature map can be input into the Repeat layer of the reversible neural network, and after being processed by the Repeat layer, Block4, Block3, Block2 and Block1 in sequence, the reconstructed image is output by the PixelShuffling layer of Block1.

[0342] For example, an image compression system framework such as Figure 5b The encoding / decoding method for the scenario shown, compared to existing technologies that only use reversible neural networks for encoding and decoding, suffers from more constraints due to the shared network parameters between encoding and decoding in reversible neural networks. This results in lower expressive power for reversible neural networks compared to non-reversible neural networks. Therefore, under the same image quality, the bitrate of the bitstream generated by encoding based on a reversible neural network is greater than that generated by encoding based on a non-reversible neural network. In other words, the rate-distortion loss of encoding and decoding based on a reversible neural network is greater than the rate-distortion loss of encoding and decoding based on both non-reversible and decoding networks. Therefore, using a second encoding network for the initial encoding and decoding can reduce the rate-distortion loss of the initial encoding and decoding. The rate-distortion loss of the initial encoding and decoding determines the upper limit of the rate-distortion performance of the image in subsequent encoding and decoding, thereby improving the rate-distortion performance of the image after multiple encoding and decoding iterations.

[0343] Furthermore, reversible neural networks are reversible, while non-reversible neural networks are irreversible. Compared to using non-reversible neural networks for multiple encoding and decoding operations, using reversible neural networks for multiple encoding and decoding operations results in a slower rate-distortion performance degradation of the image. Therefore, using a third encoding network for subsequent multiple encoding and decoding operations can slow down the rate of degradation of the image's rate-distortion performance after multiple encoding and decoding operations, thereby effectively reducing the loss magnitude ratio of the image after multiple encoding and decoding operations.

[0344] In the framework of image compression systems, such as Figure 5b In the scenario shown, tests demonstrate that the encoding / decoding method of this application (i.e., based on...) Figure 5b The image compression system framework for encoding and decoding methods (with a BD-rate that is 0.56% better than existing encoding and decoding methods) reduces the loss margin of ten encoding and decoding operations to 10 times that of existing methods.

[0345] Figure 10 This is a schematic diagram illustrating the encoding process as an example. Figure 10 by Figure 4bBased on the structure of the image compression system framework, the encoding process of the image to be encoded is described.

[0346] S1001, Obtain the image to be encoded.

[0347] S1002, obtain the number of times the image to be encoded has been encoded.

[0348] For example, S1001 to S1002 can be referred to the description of S601 to S602 above, and will not be repeated here.

[0349] S1003, Obtain preset network selection conditions.

[0350] For example, in Figure 10 In one embodiment, each coding network consists of a first coding network and a first gain vector; wherein the first gain vector is used to scale the feature map output by the first coding network.

[0351] For example, in Figure 10 In the embodiments, preset network selection conditions can be set according to actual needs. This application does not restrict how to set preset network selection conditions.

[0352] For example, the preset network selection criteria may include: when the number of times the network has been encoded is even, selecting the encoding network with an even number of serial numbers (wherein, the N encoding networks can be sorted in ascending or descending order according to the encoding loss); when the number of times the network has been encoded is odd, selecting the encoding network with an odd number of serial numbers.

[0353] For example, preset network selection criteria may include: when the number of times the network has been encoded is 0, selecting the encoding network with sequence number 1; when the number of times the network has been encoded is not 0, selecting the encoding network with the same sequence number as the number of times the network has been encoded.

[0354] For example, preset network selection criteria may include: when the number of encoding iterations is 0, selecting the coding network with the worst rate-distortion performance; when the number of encoding iterations is not 0, selecting other coding networks besides the coding network with the worst rate-distortion performance.

[0355] S1004: Select a target coding network from N coding networks based on the number of times the image to be encoded has been encoded and the preset network selection conditions.

[0356] S1005, the image to be encoded is input into the first encoding network shared by N encoding networks to obtain the fourth feature map.

[0357] S1006, the fourth feature map is scaled based on the first gain vector included in the target encoding network to obtain the first feature map.

[0358] Reference Figure 4bFor example, after selecting the target encoding network, the image to be encoded can be input into a first encoding network shared by N encoding networks. The first encoding network transforms the image to be encoded and outputs a fourth feature map (i.e., Figure 4b Feature map 4). Next, based on the first gain vector included in the target encoding network (such as... Figure 4b The GainVector_Ai in the first feature map is scaled channel by channel to obtain the first feature map (i.e., the first feature map). Figure 4b The feature map in Figure 1); the specific scaling process can be referred to the above description, and will not be repeated here.

[0359] S1007, quantize the first feature map to obtain the second feature map.

[0360] S1008, entropy encoding is performed on the second feature map to obtain the bitstream.

[0361] S1009 encapsulates the encoded number of times into the bitstream.

[0362] For example, S1007 to S1009 can be referred to the description of S807 to S809 above, and will not be repeated here.

[0363] Figure 11 This is a schematic diagram illustrating the decoding process as an example. Figure 11 by Figure 4b Based on the image compression system framework in [the document], it describes [the system] with [other components]. Figure 10 The encoding process corresponds to the decoding process.

[0364] S1101, Get the bitstream.

[0365] S1102, perform entropy decoding on the bitstream to obtain the third feature map and the number of times it has been encoded.

[0366] For example, S1101 to S1102 can be referred to the description of S901 to S902 above, and will not be repeated here.

[0367] S1103, Obtain preset network selection conditions.

[0368] S1104: Based on preset network selection conditions and the number of times encoded, select the target gain vector from N second gain vectors.

[0369] S1105, the third feature map is scaled based on the target gain vector to obtain the fifth feature map.

[0370] For example, the second selection module can obtain the third feature map, the number of times it has been encoded, and the preset network selection conditions; then, based on the preset network selection conditions and the number of times it has been encoded, it selects a target gain vector from N second gain vectors; next, it scales the third feature map based on the target gain vector to obtain the fifth feature map.

[0371] It should be noted that the second selection module selects the target gain vector from the N second gain vectors based on different conditions during the training process of the preset network and during the actual application encoding process. During the training process of the preset network, the second selection module selects the target gain vector from the N second gain vectors according to the preset training rules, while during the actual application encoding process, it selects the target gain vector from the N second gain vectors according to the preset network selection conditions.

[0372] For example, the second gain vector corresponds one-to-one with the first gain vector, and the first gain vector also corresponds one-to-one with the encoding network; therefore, based on the preset network selection conditions and the number of times the target encoding network has been encoded, after selecting the target encoding network, the target gain vector can also be selected from the N second gain vectors based on the target encoding network.

[0373] For example, refer to Figure 4b GainVector_Bi corresponds to GainVector_Ai, and the encoding network corresponding to GainVector_Ai is encoding network Gi. Assuming that the encoding side selects encoding network G5 as the target encoding network from N encoding networks based on the number of encoding iterations and preset network selection conditions, then Gain Vector_B5 can be selected from N second gain vectors as the target gain vector.

[0374] For example, S1105 is similar to S1005 above, and will not be described again here.

[0375] S1106, the fifth feature map is input into the decoding network to obtain the reconstructed image output by the decoding network.

[0376] S1107, Updated number of times the code has been encoded.

[0377] By scaling the feature map using the first and second gain vectors, the difference between the quantized values ​​obtained by the quantization module in two adjacent encoding and decoding processes can be reduced; thus, the loss magnitude ratio of the image after two adjacent encoding and decoding processes can be reduced; thereby, the loss magnitude ratio of the image after multiple encoding and decoding processes can be reduced.

[0378] Tests have shown that the encoding / decoding method of this application (i.e., based on...) Figure 4bThe image compression system framework for encoding and decoding methods shows that, compared with existing technologies, in the low bit rate (bit rate less than the preset bit rate threshold) region, the loss of ten encoding and decoding operations is reduced by an average of 6%.

[0379] In one example, Figure 12 A schematic block diagram of an apparatus 1200 according to an embodiment of this application is shown. The apparatus 1200 may include a processor 1201 and a transceiver / transceiver pin 1202, and optionally, a memory 1203.

[0380] The various components of device 1200 are coupled together via bus 1204, which includes a data bus, a power bus, a control bus, and a status signal bus. However, for clarity, all buses are referred to as bus 1204 in the figure.

[0381] Optionally, the memory 1203 can be used to store instructions from the foregoing method embodiments. The processor 1201 can be used to execute the instructions in the memory 1203, control the receive pin to receive signals, and control the transmit pin to transmit signals.

[0382] Device 1200 may be an electronic device or a chip of an electronic device in the above method embodiments.

[0383] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0384] This embodiment also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the methods described in the above embodiments.

[0385] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the methods described in the above embodiments.

[0386] This embodiment also provides an image processing apparatus, which may include one or more interface circuits and one or more processors; the interface circuit is used to receive signals from the memory of an electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, it causes the electronic device to perform the steps of the above embodiment. The interface circuit may refer to... Figure 12 The transceiver 1202 in the middle.

[0387] This embodiment also provides an apparatus for storing a bitstream, the apparatus comprising: a receiver and at least one storage medium, the receiver being used to receive the bitstream; the at least one storage medium being used to store the bitstream; the bitstream being generated according to the encoding method of the above embodiment.

[0388] This embodiment also provides an apparatus for transmitting a bitstream, the apparatus comprising: a transmitter and at least one storage medium, the at least one storage medium being used to store the bitstream, the bitstream being generated according to the encoding method of the above embodiment; the transmitter being used to obtain the bitstream from the storage medium and transmit the bitstream to an end-side device through the transmission medium.

[0389] This embodiment also provides a system for distributing bitstreams. The system includes: at least one storage medium for storing at least one bitstream, wherein the at least one bitstream is generated according to the encoding method of the above embodiment; and a streaming media device for obtaining a target bitstream from the at least one storage medium and sending the target bitstream to an end-side device, wherein the streaming media device includes a content server or a content distribution server.

[0390] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the methods in the above-described method embodiments.

[0391] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0392] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0393] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0394] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0395] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0396] Any content in the various embodiments of this application, as well as any content in the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0397] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0398] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0399] The steps of the methods or algorithms described in conjunction with the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0400] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0401] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image compression system, characterized in that, The image compression system includes a first selection module, an entropy encoding module, an entropy decoding module, a quantization module, N encoding networks, and one decoding network. The N encoding networks have different encoding losses, where N is an integer greater than 1. The first selection module is used to select a target coding network from the N coding networks based on the number of times the image to be encoded has been encoded, wherein the image to be encoded is an uncoded image or a reconstructed image; The target coding network is used to perform feature transformation on the image to be encoded to obtain a first feature map; The quantization module is used to quantize the first feature map to obtain a second feature map; The entropy coding module is used to entropy code the second feature map to obtain a bitstream; The entropy decoding module is used to perform entropy decoding on the bitstream to obtain a third feature map; The decoding network is used to perform feature transformation based on the third feature map to obtain the reconstructed image.

2. The system according to claim 1, characterized in that, The N coding networks have the same network structure but different network parameters; or, The N coding networks have different network structures.

3. The system according to claim 1 or 2, characterized in that, The N coding networks share a first coding network; each of the N coding networks consists of the first coding network and a first gain vector, and the first gain vector of each of the N coding networks is different. The target coding network is specifically used to call the first coding network to perform feature transformation on the image to be encoded to obtain a fourth feature map; and to scale the fourth feature map based on the first gain vector of the target coding network to obtain the first feature map.

4. The system according to claim 3, characterized in that, The image compression system further includes a second selection module and N second gain vectors, wherein the N second gain vectors correspond one-to-one with the N first gain vectors; The second selection module is configured to select a target gain vector from the N second gain vectors based on the number of times it has been encoded; and to scale the third feature map based on the target gain vector to obtain a fifth feature map. The decoding network is specifically used to perform feature transformation on the fifth feature map to obtain the reconstructed image.

5. The system according to claim 1 or 2, characterized in that, When N equals 2, the N encoding networks include a second encoding network and a third encoding network. The second encoding network is a non-reversible neural network, the third encoding network is a reversible neural network mapped according to a first preset direction, and the decoding network is the reversible neural network mapped according to a second preset direction.

6. The system according to any one of claims 1 to 5, characterized in that, The decoding network and the N encoding networks are jointly trained. The joint training includes M batch training sessions on the N encoding networks and the decoding network. Each batch training session is based on the loss weight coefficient corresponding to the i-th encoding network, and the i-th encoding network and the decoding network are trained. Each of the N encoding networks has a different loss weight coefficient, and different loss weight coefficients are used to train encoding networks with different encoding losses. M is a positive integer greater than N, and i is an integer between 1 and N.

7. An image processing method, characterized in that, The method includes: Obtain the image to be encoded and the number of times the image to be encoded has been encoded, wherein the image to be encoded is an unencoded image or a reconstructed image; Based on the number of times the image to be encoded has been encoded, a target encoding network is selected from N encoding networks, wherein the encoding losses of the N encoding networks are different, and N is an integer greater than 1; The image to be encoded is input into the target coding network to obtain the first feature map output by the target coding network; The first feature map is quantized to obtain the second feature map; The second feature map is entropy encoded to obtain a bitstream; Entropy decoding is performed on the bitstream to obtain a third feature map; The third feature map is input into the decoding network to obtain the reconstructed image output by the decoding network.

8. The method according to claim 7, characterized in that, The step of selecting a target coding network from N coding networks based on the number of times the image to be encoded has been encoded includes: Obtain the preset network selection criteria; Based on the number of times the image to be encoded has been encoded and the preset network selection conditions, the target encoding network is selected from the N encoding networks.

9. The method according to claim 8, characterized in that, The preset network selection criteria include: when the number of encoding iterations is 0, selecting the encoding network with the smallest difference between the compressed bitrate and the target compressed bitrate; wherein, the compressed bitrate is related to the encoding loss.

10. The method according to claim 8, characterized in that, The preset network selection conditions include: when the number of encoding times is not 0, selecting an encoding network with a encoding loss less than the historical encoding loss, wherein the historical encoding loss refers to the encoding loss of the encoding network selected in the previous encoding of the image to be encoded.

11. The method according to any one of claims 7 to 10, characterized in that, The N coding networks share a first coding network; each of the N coding networks consists of the first coding network and a first gain vector, and the first gain vector of each of the N coding networks is different. The step of inputting the image to be encoded into the target coding network to obtain the first feature map output by the target coding network includes: The image to be encoded is input into the first encoding network to obtain the fourth feature map; The fourth feature map is scaled based on the first gain vector of the target coding network to obtain the first feature map.

12. The method according to claim 8, characterized in that, When N equals 2, the N encoding networks include a second encoding network and a third encoding network. The second encoding network is a non-reversible neural network, the third encoding network is a reversible neural network mapped according to a first preset direction, and the decoding network is the reversible neural network mapped according to the second preset direction. The preset network selection criteria include: When the number of times the encoding has been performed is 0, the second encoding network is selected; When the number of times the code has been encoded is not zero, the third encoding network is selected.

13. The method according to any one of claims 7 to 12, characterized in that, The method further includes: The encoded number of times is encapsulated into the bitstream.

14. The method according to claim 13, characterized in that, After obtaining the reconstructed image output by the decoding network, the method further includes: Update the number of encoded values ​​obtained from the entropy decoding.

15. The method according to claim 11, characterized in that, The method further includes: Based on the number of times the encoding has been obtained from the entropy decoding, a target gain vector is selected from N second gain vectors, wherein the N second gain vectors correspond one-to-one with the N first gain vectors; The third feature map is scaled based on the target gain vector to obtain the fifth feature map; The step of inputting the third feature map into the decoding network to obtain the reconstructed image output by the decoding network includes: The fifth feature map is input into the decoding network to obtain the reconstructed image output by the decoding network.

16. The method according to any one of claims 7 to 15, characterized in that, The decoding network and the N encoding networks are jointly trained. The joint training includes M batch training sessions on the N encoding networks and the decoding network. Each batch training session is based on the loss weight coefficient corresponding to the i-th encoding network, and the i-th encoding network and the decoding network are trained. Each of the N encoding networks has a different loss weight coefficient, and different loss weight coefficients are used to train encoding networks with different encoding losses. M is a positive integer greater than N, and i is an integer between 1 and N.

17. An encoding method, characterized in that, The method includes: Obtain the image to be encoded, which is an unencoded image or a reconstructed image; Obtain the number of times the image to be encoded has been encoded; Based on the number of times the image to be encoded has been encoded, a target encoding network is selected from N encoding networks, where N is an integer greater than 1; The image to be encoded is input into the target coding network to obtain the first feature map output by the target coding network; The first feature map is quantized to obtain the second feature map; The second feature map is entropy encoded to obtain a bitstream; the bitstream is then decoded by the decoding end based on the decoding network, which is jointly trained with the N encoding networks. The N encoding networks have different loss weight coefficients, and different loss weight coefficients are used to train encoding networks with different encoding losses.

18. The method according to claim 17, characterized in that, The step of selecting a target coding network from N coding networks based on the number of times the image to be encoded has been encoded includes: Obtain the preset network selection criteria; Based on the number of times the image to be encoded has been encoded and the preset network selection conditions, the target encoding network is selected from the N encoding networks.

19. The method according to claim 18, characterized in that, The preset network selection criteria include: when the number of encoding iterations is 0, selecting the encoding network with the smallest difference between the compressed bitrate and the target compressed bitrate; wherein, the compressed bitrate is related to the encoding loss.

20. The method according to claim 18, characterized in that, The preset network selection conditions include: when the number of encoding times is not 0, selecting an encoding network with a encoding loss less than the historical encoding loss, wherein the historical encoding loss refers to the encoding loss of the encoding network selected in the previous encoding of the image to be encoded.

21. The method according to any one of claims 17 to 20, characterized in that, The N coding networks share a first coding network; each of the N coding networks consists of the first coding network and a first gain vector, and the first gain vector of each of the N coding networks is different. The step of inputting the image to be encoded into the target coding network to obtain the first feature map output by the target coding network includes: The image to be encoded is input into the first encoding network to obtain the fourth feature map; The fourth feature map is scaled based on the first gain vector of the target coding network to obtain the first feature map.

22. The method according to claim 18, characterized in that, When N equals 2, the N encoding networks include a second encoding network and a third encoding network, wherein the second encoding network is a non-reversible neural network and the third encoding network is a reversible neural network mapped according to a first preset direction; The preset network selection criteria include: When the number of times the encoding has been performed is 0, the second encoding network is selected; When the number of times the code has been encoded is not zero, the third encoding network is selected.

23. The method according to any one of claims 17 to 22, characterized in that, The method further includes: The encoded number of times is encapsulated into the bitstream.

24. A decoding method, characterized in that, The method includes: The bitstream is obtained by the encoder encoding the image to be encoded based on the target encoding network among N encoding networks, where N is an integer greater than 1; Entropy decoding is performed on the bitstream to obtain a third feature map; The third feature map is input into the decoding network to obtain the reconstructed image output by the decoding network; The decoding network and the N encoding networks are jointly trained. The N encoding networks have different loss weight coefficients, and different loss weight coefficients are used to train encoding networks with different encoding losses.

25. The method according to claim 24, characterized in that, After obtaining the reconstructed image output by the decoding network, the method further includes: Update the number of encoded values ​​obtained from the entropy decoding.

26. The method according to claim 24 or 25, characterized in that, The method further includes: Based on the number of times the code has been encoded obtained from the entropy decoding, a target gain vector is selected from N second gain vectors; The third feature map is scaled based on the target gain vector to obtain the fifth feature map; The step of inputting the third feature map into the decoding network to obtain the reconstructed image output by the decoding network includes: The fifth feature map is input into the decoding network to obtain the reconstructed image output by the decoding network.

27. A pre-set network training method, characterized in that, The preset network includes N encoding networks and 1 decoding network, where N is an integer greater than 1. The method includes: Acquire training images; The preset network is trained in M ​​batches based on the training images, where M is a positive integer greater than N; wherein, training the preset network in one batch includes: The determination rate-distortion loss is calculated based on the training image, the reconstructed image, and the loss weight coefficients corresponding to the i-th encoding network; wherein, the reconstructed image is obtained by encoding the training image using the i-th encoding network and then decoding the encoding result using the decoding network, and i is an integer between 1 and N; Based on the rate-distortion loss, the i-th encoding network and the decoding network are trained; The loss weight coefficients for each of the N coding networks are different.

28. The method according to claim 27, characterized in that, The determination rate-distortion loss, based on the training image, the reconstructed image, and the loss weight coefficients corresponding to the i-th encoding network, includes: Obtain the bitrate loss corresponding to encoding the training image based on the i-th encoding network; Based on the training image and the reconstructed image, determine the image distortion loss; Based on the loss weight coefficients corresponding to the i-th coding network, the bit rate loss and the image distortion loss are weighted and calculated to obtain the rate distortion loss.

29. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is coupled to the processor; The memory stores program instructions that, when executed by the processor, cause the electronic device to perform the method as described in any one of claims 7 to 28.

30. An image processing apparatus, characterized in that, The device includes one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from the memory of an electronic device and send the signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 7 to 28.

31. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed on a computer or processor, causes the computer or processor to perform the method as described in any one of claims 7 to 28.

32. A computer program product, characterized in that, The computer program product includes a software program that, when executed by a computer or processor, causes the steps of the method as described in any one of claims 7 to 28 to be performed.

33. An apparatus for storing a bitstream, characterized in that, The device includes: a receiver and at least one storage medium. The receiver is used to receive the bit stream; The at least one storage medium is used to store the bit stream; The bitstream is generated according to the encoding method described in any one of claims 17 to 23.

34. An apparatus for transmitting a code stream, characterized in that, The device includes: a transmitter and at least one storage medium. The at least one storage medium is used to store a bitstream, which is generated according to the encoding method according to any one of claims 17 to 23; The transmitter is used to obtain the bitstream from the storage medium and send the bitstream to the end-side device through the transmission medium.

35. A system for distributing bitstreams, characterized in that, The system includes: At least one storage medium for storing at least one bitstream, said at least one bitstream being generated according to the encoding method of any one of claims 17 to 23. A streaming media device is configured to acquire a target bitstream from the at least one storage medium and send the target bitstream to an end-side device, wherein the streaming media device includes a content server or a content distribution server.

Images