Image encoding and decoding methods, systems, devices and storage media

Abstract

This invention discloses an image encoding and decoding method, system, device, and storage medium. On the one hand, it improves the encoder and decoder in the image encoding and decoding model; on the other hand, when performing autoregressive hyperparameter context estimation, it uses row-by-row Rowcnn instead of traditional point-by-point Pixelcnn, which greatly reduces the fragmentation of computation I / O. Combining the above two aspects, it can reduce the computational complexity by one dimension, thereby accelerating the encoding and decoding speed, and can also improve image compression performance and ensure the quality of reconstructed images.

Description

Technical Field

[0001] This invention relates to the field of image encoding and decoding technology, and in particular to an image encoding and decoding method, system, device and storage medium. Background Technology

[0002] Currently, both academia and industry are using methods that pile up large models for deep learning encoding and decoding schemes, which makes it impossible to apply the encoding and decoding speed of these models in real-world environments.

[0003] In order for the encoding and decoding schemes to be practically implemented and for the encoding and decoding performance to not decline linearly as the model size decreases, it is urgent to make corresponding improvements to the existing encoding and decoding schemes. Summary of the Invention

[0004] The purpose of this invention is to provide an image encoding and decoding method, system, device, and storage medium that can improve image compression performance, accelerate encoding and decoding speed, and ensure image quality.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] An image encoding / decoding method, comprising:

[0007] An image encoding and decoding model is constructed, including: an encoding network, a super-prior probability model, and a decoding network; wherein, the first layer of the encoding network is a space_to_depth unit, and the last layer of the decoding network is a depth_to_space unit;

[0008] The image encoding / decoding model is trained by the following steps: First, an original image is input. Primary image features are extracted through an encoding network. The super-prior probability model contains two branches, both of which input the primary image features. In the first branch, the primary image features are quantized to obtain intermediate image features F, which are then input into a Maskcnn model. The Maskcnn model, combined with auxiliary information from the second branch, generates a first probability parameter A. The first probability parameter A is used to estimate the bitrate of the intermediate image features F, yielding a first bitrate estimation result. In the second branch, the primary image features are input to a super-prior encoder to obtain super-prior features. These super-prior features are then quantized to obtain quantized features Z. A second probability parameter B is generated through a hybrid Laplacian model. The second probability parameter B is used to estimate the bitrate of the quantized features Z, yielding the first bitrate estimation result. The two bitrate estimation results are used, and the quantized feature Z is also input into the super-prior decoder to obtain auxiliary information; the intermediate image feature F is also input into the decoding network to obtain the reconstructed image; the rate-distortion optimization loss function is constructed using the distortion of the reconstructed image and the original image and the two bitrate estimation results obtained from the super-prior probability model for training; the training phase includes two stages, both of which use the above training steps. In the first stage, the Maskcnn model uses the Pixelcnn model. After the training converges, the second stage begins. In the second stage, the Maskcnn model uses the Rowcnn model. The Rowcnn model is initialized using the parameters of the Pixelcnn model obtained in the first stage, and the last row of non-zero parameter values ​​in the weight parameters of the multi-scale mask convolutional layer is discarded;

[0009] The trained image encoding and decoding model is used to encode and decode the input image to obtain the reconstructed image. The Maskcnn in the trained image encoding and decoding model uses the Rowcnn trained in the second stage, and the bit rate estimation operation in the two branches is transformed into an arithmetic encoding and decoding operation.

[0010] An image encoding and decoding system, comprising:

[0011] The model building unit is used to build the image encoding and decoding model, including: an encoding network, a super-prior probability model, and a decoding network; wherein, the first layer of the encoding network is a space_to_depth unit, and the last layer of the decoding network is a depth_to_space unit;

[0012] The model training unit is used to train the image encoding / decoding model. The training steps include: inputting the original image, extracting primary image features through an encoding network, wherein the super-prior probability model contains two branches, and the input of both branches is the primary image features; in the first branch, the primary image features are quantized to obtain intermediate image features F, which are then input into the Maskcnn model. The Maskcnn model, combined with auxiliary information from the second branch, generates a first probability parameter A, and uses the first probability parameter A to estimate the bitrate of the intermediate image features F, obtaining a first bitrate estimation result; in the second branch, the primary image features are input to the super-prior encoder to obtain super-prior features, which are then quantized to obtain quantized features Z, which are then generated by a hybrid Laplacian model to generate a second probability parameter B, and the second probability parameter B is used to estimate the bitrate of the quantized features Z. The second bitrate estimation result is obtained, and the quantized feature Z is also input to the super-prior decoder to obtain auxiliary information; the intermediate image feature F is also input to the decoding network to obtain the reconstructed image; the rate-distortion optimization loss function is constructed using the distortion between the reconstructed image and the original image and the two bitrate estimation results obtained from the super-prior probability model for training; the training phase includes two stages, both of which use the above training steps. In the first stage, the Maskcnn model uses the Pixelcnn model. After the training converges, the second stage begins. In the second stage, the Maskcnn model uses the Rowcnn model. The Rowcnn model is initialized using the parameters of the Pixelcnn model obtained in the first stage, and the last row of non-zero parameter values ​​in the weight parameters of the multi-scale mask convolutional layer is discarded;

[0013] The image encoding / decoding unit is used to encode and decode the input image using the trained image encoding / decoding model to obtain the reconstructed image. The Maskcnn in the trained image encoding / decoding model uses the Rowcnn trained in the second stage, and the bit rate estimation operation in the two branches is transformed into an arithmetic encoding / decoding operation.

[0014] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0016] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0017] As can be seen from the technical solution provided by the present invention, on the one hand, the encoding network and decoding network in the image encoding and decoding model are improved; on the other hand, when performing autoregressive hyperparameter context estimation, the row-by-row Rowcnn model is used instead of the traditional point-by-point Pixelcnn model, which greatly reduces the fragmentation of computation I / O. Combining the above two aspects, the computational complexity can be reduced by one dimension, thereby speeding up the encoding and decoding speed, and can also improve the image compression performance and ensure the quality of the reconstructed image. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of an image encoding / decoding method provided in an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of an image encoding / decoding model provided in an embodiment of the present invention;

[0021] Figure 3 A schematic diagram illustrating the principle of the space_to_depth unit and the depth_to_space unit provided in the embodiments of the present invention;

[0022] Figure 4 A schematic diagram of a hyper-prior codec provided in an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of the Maskcnn model provided in an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram of the PixelCNN model provided in an embodiment of the present invention;

[0025] Figure 7 A schematic diagram of the Rowcnn model provided in an embodiment of the present invention;

[0026] Figure 8 A schematic diagram of an image encoding and decoding system provided in an embodiment of the present invention.

[0027] Figure 9 A schematic diagram of a processing device provided in an embodiment of the present invention; Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0029] First, the following explanations are provided for the terms that may be used in this article:

[0030] The terms “including,” “comprising,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0031] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0032] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0033] The following provides a detailed description of an image encoding / decoding method, system, device, and storage medium provided by the present invention. Contents not described in detail in the embodiments of the present invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of the present invention, they shall be performed according to conventional conditions in the art or conditions recommended by the manufacturer.

[0034] Example 1

[0035] This invention provides an image encoding and decoding method, such as... Figure 1 As shown, it mainly includes:

[0036] Step 1: Construct an image encoding / decoding model.

[0037] like Figure 2 As shown, the image encoding and decoding model mainly includes: an encoding network, a super-prior probability model, and a decoding network; wherein, the first layer of the encoding network is a space_to_depth unit, and the last layer of the decoding network is a depth_to_space unit.

[0038] like Figure 2 As shown, the encoding network includes multiple convolutional layers connected in sequence, with one space-to-depth unit for converting the image size spatial dimension to the depth dimension. The decoding network includes multiple deconvolutional layers connected in sequence and one depth-to-space unit for converting the depth dimension in the output features of the last deconvolutional layer to the image size spatial dimension. Figure 1 Examples are provided where the encoding network contains three convolutional layers and the decoding network contains three deconvolutional layers.

[0039] Figure 2 In this context, Conv represents a convolutional layer, and Deconv represents a deconvolutional layer; Conv A×A (Deconv A×A) indicates that the size of the convolutional kernel (deconvolutional kernel) is A×A. For example, Conv 4×4 indicates that the size of the convolutional kernel is 4×4, and Deconv 4×4 indicates that the size of the deconvolutional kernel is 4×4; ch=N indicates that the number of output channels is N; S represents the stride, for example, S2 indicates that the stride is 2; GDN is the most commonly used normalization and non-linear activation function in image compression.

[0040] like Figure 3 As shown, the space_to_depth unit and the depth_to_space unit are inverse operations, and both are existing technologies. Figure 3 In the example shown, data from each 2×2 segment of the image size spatial dimension is sequentially arranged into the depth dimension according to the sliding window size (2×2). In this embodiment of the invention, the first layer of the encoding network and the last layer of the decoding network, the two most time-consuming convolutional layers (deconvolutional layers), are replaced with space_to_depth units and depth_to_space units. This greatly reduces the time required for downsampling operations in existing schemes to be implemented through convolutional operations with a stride greater than 1 (i.e., reducing the time for convolutional computation), while also preserving the encoding and decoding performance to the greatest extent.

[0041] Step 2: Train the image encoding / decoding model.

[0042] In this embodiment of the invention, the training steps include: inputting an original image, extracting primary image features through an encoding network, wherein the super-prior probability model contains two branches, and the input of both branches is the primary image features; in the first branch, the primary image features are quantized to obtain intermediate image features F, which are then input into a MaskCNN model (autoregressive masked convolutional network model), and the MaskCNN model, combined with auxiliary information from the second branch, generates a first probability parameter A, which is used to estimate the bitrate of the intermediate image features F to obtain a first bitrate estimation result; in the second branch, the primary image features are input to a super-prior encoder to obtain super-prior features, which are then quantized to obtain quantized features Z, which are then generated by a hybrid Laplacian model to generate a second probability parameter B, which is used to estimate the bitrate of the quantized features Z. The second bitrate estimation result is obtained, and the quantized feature Z is input to the super-prior decoder to obtain auxiliary information. The intermediate image feature F is also input to the decoding network to obtain the reconstructed image. The rate-distortion optimization loss function is constructed using the distortion between the reconstructed image and the original image and the two bitrate estimation results obtained from the super-prior probability model for training. The training phase includes two stages, both of which use the above training steps. In the first stage, the Maskcnn model uses the Pixelcnn model. After the training converges, the second stage begins. In the second stage, the Maskcnn model uses the Rowcnn model. The Rowcnn model is initialized using the parameters of the Pixelcnn model obtained in the first stage, and the last row of non-zero parameter values ​​in the weight parameters of the multi-scale mask convolutional layer is discarded.

[0043] In this embodiment of the invention, training convergence is generally defined as the convergence of the loss function, or reaching a specified number of iterations (e.g., 1 million). Discarding the last row mainly refers to setting the last row of weights activated by 1 in the weight parameters of the multi-scale masked convolutional layer to 0, which means discarding the lateral dependencies in the autoregressive convolution, allowing an entire row of data to be computed in parallel.

[0044] Step 3: Use the trained image encoding and decoding model to encode and decode the input image to obtain the reconstructed image; wherein, the Maskcnn in the trained image encoding and decoding model uses the Rowcnn trained in the second stage, and the bit rate estimation operation in the two branches is transformed into an arithmetic encoding and decoding operation.

[0045] The main process of this step is as follows:

[0046] 1) Encoding section: Input image, primary image features are extracted through the encoding network in the trained image encoding / decoding model; then, input into the trained super-prior probability model, the primary image features are quantized in the first branch to obtain intermediate image features F, and input into the Rowcnn model trained in the second stage, the Rowcnn model trained in the second stage combines the auxiliary information from the second branch to generate the first probability parameter A, and the intermediate image features F are arithmetically encoded using the first probability parameter A to obtain the first bitstream; in the second branch, the primary image features are input into the super-prior encoder to obtain super-prior features and quantize them to obtain quantized features Z, and the quantized features Z are arithmetically encoded using the second probability parameter B provided by the trained hybrid Laplace model to obtain the second bitstream.

[0047] 2) Decoding part: The second bitstream is arithmetically decoded using the second probability parameter B to obtain quantization features Z, and then auxiliary information is obtained through the super prior decoder; the trained Rowcnn model combines the auxiliary information to generate the first probability parameter A, and the first bitstream is arithmetically decoded using the first probability parameter A to obtain intermediate image features F, and then the decoder network in the trained image encoding and decoding model is used to generate the decoded and reconstructed image.

[0048] This step belongs to the actual encoding and decoding stage. Entering this stage means that the image encoding and decoding model has been trained. The second probability parameter B is fixed information, which is directly stored in the encoding and decoding end and can be directly called when needed.

[0049] To facilitate understanding, a detailed introduction to the prior probability model will be provided below.

[0050] like Figure 2 As shown, the hyperprior probabilistic model mainly includes: a first quantization unit (Quant F), a hyperprior encoder (hyper analysis), a second quantization unit (Quant Z), a hybrid Laplace model (LMM), a hyperprior decoder (hyper synthesis), a Maskcnn model, a first rate estimation module, a second rate estimation module, a first arithmetic codec, and a second arithmetic codec. The first branch includes: the first quantization unit, the Maskcnn model, the first rate estimation module, and the first arithmetic codec; the second branch includes: the hyperprior encoder, the second quantization unit, the hybrid Laplace model, the hyperprior decoder, the second rate estimation module, and the second arithmetic codec.

[0051] like Figure 4 As shown, the hyper-prior encoder includes multiple sequentially connected convolutional layers; the hyper-prior decoder includes multiple sequentially connected deconvolutional layers.

[0052] Two bitrate estimation modules are used during the training phase, therefore, not in Figure 2 The diagram shows that two arithmetic codecs are used after training. The main principle is that arithmetic codecs cannot be used for training because the bitstream obtained during training is not important, but the codewords required to store these bitstreams are more important. Therefore, during training, each branch of the arithmetic encoder is replaced by a bitrate estimation module. In the first branch, the first bitrate estimation module takes the intermediate image feature F and the first probability parameter A as input and outputs the first bitrate estimation result. In the second branch, the second bitrate estimation module takes the quantization feature Z and the second probability parameter B as input and outputs the second bitrate estimation result. Furthermore, during training, the decoding network takes the intermediate image feature F output by the first quantization unit as input. After training, the two bitrate estimation modules are restored to their corresponding arithmetic codecs.

[0053] Figure 2 In this context, bit_coder represents an arithmetic codec, which includes an arithmetic encoder (BitEncoder) and an arithmetic decoder (BitDecoder). The bit stream (binary code stream) obtained by the arithmetic codec at the encoding end is transmitted to the decoding end, where the corresponding arithmetic decoder at the decoding end decodes the intermediate image features F without loss, and then generates the reconstructed image through the decoding network.

[0054] The arithmetic encoding and decoding described above all belong to entropy encoding and decoding, and specifically, lossless entropy encoding and decoding, meaning that the object encoded by arithmetic and the result of arithmetic decoding are completely identical. Those skilled in the art will understand that arithmetic encoding uses probability parameters to encode a string of target data into a stored binary code stream. Arithmetic decoding also requires corresponding probability parameters, and these parameters must be identical to those used during encoding to losslessly decode the target data. For example... Figure 2 As shown, in the two branches of the super-prior probability model, the target data input to the two arithmetic encoders are the intermediate image feature F and the quantized feature Z, respectively. Both features are obtained through corresponding quantization units (Quant F, Quant Z), and corresponding probability parameters also need to be input. The first probability parameter A and the second probability parameter B mentioned above have the same meaning; the main difference lies in their parameter values.

[0055] The following describes the processing flow for each branch during the training phase:

[0056] In the first branch: a first quantization unit is used to quantize the primary features of the image to obtain intermediate image features F; a Maskcnn model is used to generate a first probability parameter A by combining auxiliary information from the second branch; and a first bitrate estimation module is used to estimate the bitrate of the intermediate image features F using the first probability parameter A to obtain a first bitrate estimation result.

[0057] In the second branch: a super-prior encoder is used to perform super-prior encoding on the primary features of the image to obtain super-prior features; a second quantization unit is used to quantize the super-prior features to obtain quantized features Z; a hybrid Laplacian model is used during training to perform probability estimation on the super-prior features to obtain a second probability parameter B, and after training, a fixed second probability parameter B is used. Specifically, the hybrid Laplacian model has several trainable parameters, which are fixed after training. Therefore, the second probability parameter B is also fixed (i.e., the same second probability parameter B is output for any input quantized feature Z), and it is only used for arithmetic encoding and decoding of quantized feature Z; a second bitrate estimation module is used to estimate the bitrate of quantized feature Z using the second probability parameter B to obtain a second bitrate estimation result; and a super-prior decoder is used to generate auxiliary information (pre_MuSigm) using quantized feature Z.

[0058] After training, the two rate estimation modules are converted into two arithmetic codecs. The two arithmetic codecs have the same structure, differing mainly in their input probability parameters and feature data. The arithmetic encoders in both codecs use the input probability parameters to perform arithmetic encoding on the input feature data, obtaining a bitstream. During decoding, since the second probability parameter is fixed, arithmetic decoding can be performed directly to obtain the quantized feature Z. This quantized feature Z is then passed through a super-prior decoder to obtain auxiliary information, assisting in the arithmetic decoding of the first bitstream to obtain intermediate image features F. Finally, the decoding network generates the reconstructed image. In summary, the super-prior probability model primarily quantizes the primary image features extracted by the encoding network into intermediate image features F and performs arithmetic encoding to obtain a binary bitstream for storage, reducing storage space. Then, when needed, the bitstream is transmitted to the decoding end for decoding.

[0059] In this embodiment of the invention, the Maskcnn model can be implemented using existing models. The intermediate image feature F can be understood as multiple parts. The first probability parameter A used to perform bitrate estimation (or arithmetic encoding / decoding) on ​​the intermediate image feature F also contains multiple parts. Combined with the auxiliary information obtained by the super-prior decoder, and with the intermediate image feature F that has already been encoded / decoded (starting from 0), the probability parameters of each part of the intermediate image feature F are generated. Thus, each part of the intermediate image feature F can be encoded into a bitstream. During arithmetic decoding, each part is decoded step by step to finally obtain the intermediate image feature F.

[0060] In this embodiment of the invention, the Maskcnn model has two forms: the Pixelcnn model is used as the Maskcnn model in the first stage of the training process, and the Rowcnn model is used as the Maskcnn model in the second stage and after training is completed. Figure 5 The diagram illustrates the Maskcnn model, whose input includes auxiliary information (pre_MuSigm) from the super-prior decoder and intermediate image features F (feat). The intermediate image features F are processed through multi-scale mask convolutional layers (e.g., ...). Figure 5 The three multi-scale mask convolutional layers shown (357Maskconv) concatenate the outputs of each scale's mask convolutional layer, then concatenate them with auxiliary information, and then input them into multiple convolutional layers (e.g., Figure 5 The three convolutional layers shown are processed sequentially, with LeakyReLU activation layers between adjacent convolutional layers. The last convolutional layer outputs the first probability parameter A, namely two probability parameters, mu(μ) and sigam(σ).

[0061] In this embodiment of the invention, the Pixel CNN model generates data point-by-point (i.e., location points in the spatial dimension), meaning that only a single point's data can be generated at a time. After generating data for all points in a row, the data for each point in the next row is generated sequentially. The Row CNN model generates data row-by-row. The Row CNN model and the Pixel CNN model are essentially the same, both employing... Figure 5 The structures shown, including both RowCNN and PixelCNN, can be simply understood as setting the weight parameters of the multi-scale mask convolutional layers to 0. The main difference is that in the PixelCNN model, all weight parameters in the last row of non-zero weight parameters in the multi-scale mask convolutional layers are set to 0, forming the weights of the multi-scale mask convolutional layers in the RowCNN model. Therefore, compared to the PixelCNN model, the RowCNN model eliminates the dependency on the left-hand data in the spatial representation, thus reducing the time complexity dimension compared to the PixelCNN model. Specifically:

[0062] PixelCNN uses a recursive generation method, belonging to the ordered sequence prediction generation approach. Simply put, when generating an ordered sequence, it uses existing data as a reference to generate new data; and subsequent data generation uses previously generated data as a reference to generate the next data, and so on. PixelCNN generates data pixel-by-pixel in two-dimensional space; the first row of data must be generated before the second row can be generated, resulting in an I / O complexity of O(N^2). RowCNN uses parameters pre-trained with PixelCNN, then fine-tunes the training by masking the bottom row of parameters, making the MaskCNN convolutions horizontally independent. Therefore, an entire row of data can be input into the network for convolution, reducing the I / O complexity to O(N) and the performance loss to a negligible level. Figures 6-7 The principles of PixelCNN and RowCNN are demonstrated respectively.

[0063] Example 2

[0064] The present invention also provides an image encoding and decoding system, which is mainly implemented based on the method provided in the foregoing embodiments, such as... Figure 8 As shown, the system mainly includes:

[0065] The model building unit is used to build the image encoding and decoding model, including: an encoding network, a super-prior probability model, and a decoding network; wherein, the first layer of the encoding network is a space_to_depth unit, and the last layer of the decoding network is a depth_to_space unit;

[0066] The model training unit is used to train the image encoding / decoding model. The training steps include: inputting the original image, extracting primary image features through an encoding network, wherein the super-prior probability model contains two branches, and the input of both branches is the primary image features; in the first branch, the primary image features are quantized to obtain intermediate image features F, which are then input into the Maskcnn model. The Maskcnn model, combined with auxiliary information from the second branch, generates a first probability parameter A, and uses the first probability parameter A to estimate the bitrate of the intermediate image features F, obtaining a first bitrate estimation result; in the second branch, the primary image features are input to the super-prior encoder to obtain super-prior features, which are then quantized to obtain quantized features Z, which are then generated by a hybrid Laplacian model to generate a second probability parameter B, and the second probability parameter B is used to estimate the bitrate of the quantized features Z. The second bitrate estimation result is obtained, and the quantized feature Z is also input to the super-prior decoder to obtain auxiliary information; the intermediate image feature F is also input to the decoding network to obtain the reconstructed image; the rate-distortion optimization loss function is constructed using the distortion between the reconstructed image and the original image and the two bitrate estimation results obtained from the super-prior probability model for training; the training phase includes two stages, both of which use the above training steps. In the first stage, the Maskcnn model uses the Pixelcnn model. After the training converges, the second stage begins. In the second stage, the Maskcnn model uses the Rowcnn model. The Rowcnn model is initialized using the parameters of the Pixelcnn model obtained in the first stage, and the last row of non-zero parameter values ​​in the weight parameters of the multi-scale mask convolutional layer is discarded;

[0067] The image encoding / decoding unit is used to encode and decode the input image using the trained image encoding / decoding model to obtain the reconstructed image. The Maskcnn in the trained image encoding / decoding model uses the Rowcnn trained in the second stage, and the bit rate estimation operation in the two branches is transformed into an arithmetic encoding / decoding operation.

[0068] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0069] Example 3

[0070] The present invention also provides a processing device, such as Figure 9As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0071] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0072] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0073] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0074] The output device can be a display terminal;

[0075] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0076] Example 4

[0077] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0078] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0079] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method of image coding, characterized by, include: An image encoding and decoding model is constructed, including: an encoding network, a super-prior probability model, and a decoding network; wherein, the first layer of the encoding network is a space_to_depth unit, and the last layer of the decoding network is a depth_to_space unit; The image encoding / decoding model is trained by the following steps: First, an original image is input. Primary image features are extracted through an encoding network. The super-prior probability model contains two branches, both of which input the primary image features. In the first branch, the primary image features are quantized to obtain intermediate image features F, which are then input into a Maskcnn model. The Maskcnn model, combined with auxiliary information from the second branch, generates a first probability parameter A. The first probability parameter A is used to estimate the bitrate of the intermediate image features F, yielding a first bitrate estimation result. In the second branch, the primary image features are input to a super-prior encoder to obtain super-prior features. These super-prior features are then quantized to obtain quantized features Z. A second probability parameter B is generated through a hybrid Laplacian model. The second probability parameter B is used to estimate the bitrate of the quantized features Z, yielding the first bitrate estimation result. The two bitrate estimation results are used, and the quantized feature Z is also input into the super-prior decoder to obtain auxiliary information; the intermediate image feature F is also input into the decoding network to obtain the reconstructed image; the rate-distortion optimization loss function is constructed using the distortion of the reconstructed image and the original image and the two bitrate estimation results obtained from the super-prior probability model for training; the training phase includes two stages, both of which use the above training steps. In the first stage, the Maskcnn model uses the Pixelcnn model. After the training converges, the second stage begins. In the second stage, the Maskcnn model uses the Rowcnn model. The Rowcnn model is initialized using the parameters of the Pixelcnn model obtained in the first stage, and the last row of non-zero parameter values ​​in the weight parameters of the multi-scale mask convolutional layer is discarded; The trained image encoding and decoding model is used to encode and decode the input image to obtain the reconstructed image. The Maskcnn in the trained image encoding and decoding model uses the Rowcnn trained in the second stage, and the bit rate estimation operation in the two branches is transformed into an arithmetic encoding and decoding operation.

2. The method of claim 1, wherein, The encoding network includes a space_to_depth unit and multiple convolutional layers connected in sequence. The space_to_depth unit is used to transform the spatial dimension of the image size to the depth dimension.

3. The method of claim 1, wherein, The super-prior probability model includes: a first quantization unit, a super-prior encoder, a second quantization unit, a hybrid Laplace model, a super-prior decoder, a Maskcnn model, a first rate estimation module, a second rate estimation module, a first arithmetic codec, and a second arithmetic codec; wherein, the first branch includes: the first quantization unit, the Maskcnn model, the first rate estimation module, and the first arithmetic codec; the second branch includes: a super-prior encoder, a second quantization unit, a hybrid Laplace model, a super-prior decoder, the second rate estimation module, and the second arithmetic codec; the two rate estimation modules are used in the training phase, and the two arithmetic codecs are used in the actual encoding and decoding phase; during training: In the first branch: a first quantization unit is used to quantize the primary features of the image to obtain intermediate image features F; a Maskcnn model is used to generate a first probability parameter A by combining auxiliary information from the second branch; and a first bitrate estimation module is used to estimate the bitrate of the intermediate image features F using the first probability parameter A to obtain a first bitrate estimation result. In the second branch: a super-prior encoder is used to perform super-prior encoding processing on the primary features of the image to obtain super-prior features; a second quantization unit is used to quantize the super-prior features to obtain quantized features Z; a hybrid Laplacian model is used during training to perform probability estimation on the super-prior features to obtain a second probability parameter B; a second bitrate estimation module is used to estimate the bitrate of quantized features Z using the second probability parameter B to obtain a second bitrate estimation result; and a super-prior decoder is used to generate auxiliary information using quantized features Z.

4. The image encoding and decoding method according to claim 3, characterized in that, The advanced prior encoder includes multiple convolutional layers connected in sequence; the advanced prior decoder includes multiple deconvolutional layers connected in sequence.

5. The method of Claim 1, wherein, The decoding network includes: Multiple deconvolutional layers are connected in sequence, and a depth_to_space unit is used to transform the depth dimension in the output features of the last deconvolutional layer to the image size space dimension.

6. The method of claim 1, wherein, The pixelcnn generates data pixel by pixel, generating a row of data pixel by pixel and then generating the next row of data pixel by pixel. Rowcnn generates data row by row.

7. The method of claim 1 or 3, wherein, The steps for encoding and decoding the input image using the trained image encoding / decoding model to obtain the reconstructed image include: Encoding section: The input image is processed by the encoding network in the trained image encoding / decoding model to extract primary image features. These features are then input into the trained super-prior probability model. In the first branch, the primary image features are quantized to obtain intermediate image features F, which are then input into the Rowcnn model trained in the second stage. The Rowcnn model, combined with auxiliary information from the second branch, generates a first probability parameter A. The intermediate image features F are arithmetically encoded using the first probability parameter A to obtain a first bitstream. In the second branch, the primary image features are input to the super-prior encoder to obtain super-prior features, which are then quantized to obtain quantized features Z. The quantized features Z are arithmetically encoded using the second probability parameter B provided by the trained hybrid Laplace model to obtain a second bitstream. Decoding section: The second bitstream is arithmetically decoded using the second probability parameter B to obtain quantization features Z, and then auxiliary information is obtained through a super prior decoder; the trained Rowcnn model combines the auxiliary information to generate a first probability parameter A, and the first bitstream is arithmetically decoded using the first probability parameter A to obtain intermediate image features F, and then the decoder network in the trained image encoding and decoding model is used to generate a decoded and reconstructed image.

8. An image encoding and decoding system, characterized in that, Based on the method described in any one of claims 1 to 7, the system comprises: The model building unit is used to build the image encoding and decoding model, including: an encoding network, a super-prior probability model, and a decoding network; wherein, the first layer of the encoding network is a space_to_depth unit, and the last layer of the decoding network is a depth_to_space unit; The model training unit is used to train the image encoding / decoding model. The training steps include: inputting the original image, extracting primary image features through an encoding network, wherein the super-prior probability model contains two branches, and the input of both branches is the primary image features; in the first branch, the primary image features are quantized to obtain intermediate image features F, which are then input into the Maskcnn model. The Maskcnn model, combined with auxiliary information from the second branch, generates a first probability parameter A, and uses the first probability parameter A to estimate the bitrate of the intermediate image features F, obtaining a first bitrate estimation result; in the second branch, the primary image features are input to the super-prior encoder to obtain super-prior features, which are then quantized to obtain quantized features Z, which are then generated by a hybrid Laplacian model to generate a second probability parameter B, and the second probability parameter B is used to estimate the bitrate of the quantized features Z. The second bitrate estimation result is obtained, and the quantized feature Z is also input to the super-prior decoder to obtain auxiliary information; the intermediate image feature F is also input to the decoding network to obtain the reconstructed image; the rate-distortion optimization loss function is constructed using the distortion between the reconstructed image and the original image and the two bitrate estimation results obtained from the super-prior probability model for training; the training phase includes two stages, both of which use the above training steps. In the first stage, the Maskcnn model uses the Pixelcnn model. After the training converges, the second stage begins. In the second stage, the Maskcnn model uses the Rowcnn model. The Rowcnn model is initialized using the parameters of the Pixelcnn model obtained in the first stage, and the last row of non-zero parameter values ​​in the weight parameters of the multi-scale mask convolutional layer is discarded; The image encoding / decoding unit is used to encode and decode the input image using the trained image encoding / decoding model to obtain the reconstructed image. The Maskcnn in the trained image encoding / decoding model uses the Rowcnn trained in the second stage, and the bit rate estimation operation in the two branches is transformed into an arithmetic encoding / decoding operation.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images