Image encoding and decoding model training method and system based on deviation control

Abstract

The present invention provides a method and system for training an image coding and decoding model based on deviation control, which belongs to the field of image coding and decoding. The method comprises: determining a total loss function based on a deviation control loss function and a rate-distortion loss function, training the image coding and decoding model based on the total loss function, and obtaining a trained image coding and decoding model; the calculation formula of the deviation control loss function is:; by performing deviation control on a feature map and a Gaussian mean, the difference between the feature map and the estimated Gaussian mean value is reduced, so that the elements to be encoded are smaller, the feature map is closer to the peak value of the predicted Gaussian distribution, the variance of the Gaussian distribution is reduced, and the speed and encoding performance of entropy coding and decoding are improved.

Description

Technical Field

[0001] The present invention relates to the field of image coding and decoding technology, and in particular to an image coding and decoding model training method and system based on deviation control. Background Art

[0002] A typical codec framework for end-to-end image coding is as follows: Figure 1 As shown, during encoding and decoding, the specific steps are: at the encoding end, the image Input encoding transformation, output feature map On the one hand, the feature map The edge information is extracted by the edge information extraction module and output , the encoding end will Input probability estimation module, output each feature element The probability distribution of The probability modeling uses a Gaussian model with a mean and scale Two elements. Among them, the side information It is also a kind of feature information, which contains fewer feature elements than the feature map. On the other hand, the feature map Input the quantization module to obtain the quantized feature map The entropy coding module is based on each feature element The probability distribution of the corresponding quantized elements is derived The probability distribution of Perform entropy coding to obtain a compressed code stream. and scale Generate quantized feature map The probability distribution of is entropy encoded. The probability is estimated using the parameter decomposition entropy model, and entropy coding is also performed and written into the compressed bitstream. At the decoding end, the side information is first decoded , and then according to the side information Output y The probability distribution of The entropy decoding module is based on each feature element The probability distribution of each feature element Perform entropy decoding to obtain characteristic elements The value of the decoded feature map Input decoding transformation output reconstruction image .

[0003] When training the model, we generally use rate distortion loss to optimize all modules end-to-end through gradient backpropagation. During training, we use the probability distribution of the probability estimation network output to estimate the entropy coded feature map. and Required bit rate To solve the problem that the gradient of the quantization module cannot be back-transferred during training, a method of adding uniform noise or direct gradient transmission is used instead. The loss function of training is shown in formula (1):

[0004] (1)

[0005] in is the distortion loss between the original image and the reconstructed image, is the Lagrange multiplier that trades off bit rate and distortion.

[0006] When training an end-to-end image encoding and decoding model, some existing methods use other loss functions in addition to rate-distortion loss to improve the performance of the end-to-end image encoding and decoding model.

[0007] (1) Spatial correlation loss function: The feature map obtained by encoding the image , set a The window of size is used to calculate the correlation graph of the feature map, as shown in formula (2), where Represents the index of the center point of each window. At the same time, define a mask ,like Figure 2 As shown. The final spatial correlation loss function As shown in formula (3). When training the model, the spatial correlation loss function and the rate distortion function are used for joint training, as shown in formula (4), where is the weight parameter of spatial correlation loss.

[0008] (2)

[0009] (3)

[0010] (4)

[0011] The spatial correlation loss function is designed to remove the feature map to achieve better coding performance.

[0012] (2) Channel feature decoupling loss function: The feature map obtained by encoding the image , whose channel dimension size is , define the channel feature decoupling loss function as shown in formula (5), where represents the index of the channel, Representing characteristic elements No. u The components of the channels, Indicates that all characteristic elements are u When training the model, the channel feature decoupling loss function and the rate distortion function are used for joint training, as shown in formula (6), where The weight parameter for channel feature decoupling loss.

[0013] (5)

[0014] (6)

[0015] The design goal of the channel feature decoupling loss function is also to remove the feature map The redundancy in the feature map is not taken into account As the goal of entropy coding, it can be constrained by the loss function to achieve better entropy coding efficiency and rate-distortion performance. Summary of the invention

[0016] The technical problem to be solved by the present invention is how to improve the speed and encoding performance of entropy coding and entropy decoding.

[0017] The present invention solves the above technical problems through the following technical solutions: a method for training an image coding and decoding model based on deviation control, determining a total loss function based on a deviation control loss function and a rate-distortion loss function, training the image coding and decoding model based on the total loss function, and obtaining a trained image coding and decoding model;

[0018] The calculation formula of the deviation control loss function is:

[0019]

[0020] in, is the bias control loss function, The feature map i elements, For the i The Gaussian mean of the elements, The feature map The total number of elements in .

[0021] Preferably, the calculation formula of the rate-distortion loss function is:

[0022]

[0023] in, is the rate-distortion loss function, The feature map The code rate, Side Information The code rate, For images With the reconstruction The distortion loss between is the Lagrange multiplier that trades off bit rate and distortion.

[0024] Preferably, the calculation formula of the total loss function is:

[0025]

[0026] in, is the total loss function, The weight coefficient of the bias control loss function.

[0027] Preferably, the process of training the image encoding and decoding model based on the total loss function includes:

[0028] Step 1: Image Input to the encoding transformation module output feature map , feature map Input to the side information extraction module Output side information , the side information Input to the probability estimation module and output feature map The probability distribution of each characteristic element ;

[0029] Step 2: Feature map With Gaussian mean Input into the quantization module together and output the element to be encoded ,in Express Rounding to the nearest integer;

[0030] Step 3: Entropy coding module AE according to feature map The Gaussian distribution variance of each feature element Treating coded elements Each characteristic element in is entropy encoded to obtain a compressed code stream;

[0031] Step 4: Side Information Perform entropy coding, write compressed code stream, and entropy decoding module AD according to feature map The Gaussian distribution variance of each feature element Treating coded elements Each characteristic element in is entropy decoded to obtain the reconstructed value , will rebuild the value Input decoding transformation module, output reconstructed image ;

[0032] Step 5: Image and reconstruction graph Substitute the calculation formula of the total loss function. When the total loss function is minimized, stop training and obtain the trained image encoding and decoding model.

[0033] Preferably, the weight coefficient of the deviation control loss function is The value is 1.0.

[0034] The present invention also provides an image encoding and decoding model training system based on deviation control, comprising:

[0035] A loss function establishment module, used for determining a total loss function based on a deviation control loss function and a rate distortion loss function;

[0036] A joint training module is used to train the image encoding and decoding model based on the total loss function to obtain a trained image encoding and decoding model;

[0037] The calculation formula of the deviation control loss function is:

[0038]

[0039] in, is the bias control loss function, The feature map i elements, For the i The Gaussian mean of the elements, The feature map The total number of elements in .

[0040] Preferably, the calculation formula of the rate-distortion loss function is:

[0041]

[0042] in, is the rate-distortion loss function, The feature map The code rate, Side Information The code rate, For images With the reconstruction The distortion loss between is the Lagrange multiplier that trades off bit rate and distortion.

[0043] Preferably, the calculation formula of the total loss function is:

[0044]

[0045] in, is the total loss function, The weight coefficient of the bias control loss function.

[0046] Preferably, the training process of the joint training module includes:

[0047] Step 1: Image Input to the encoding transformation module output feature map , feature map Input to the side information extraction module Output side information , the side information Input to the probability estimation module and output feature map The probability distribution of each characteristic element ;

[0048] Step 2: Feature map With Gaussian mean Input into the quantization module together and output the element to be encoded ,in Express Rounding to the nearest integer;

[0049] Step 3: Entropy coding module AE according to feature map The Gaussian distribution variance of each feature element Treating coded elements Each characteristic element in is entropy encoded to obtain a compressed code stream;

[0050] Step 4: Side Information Perform entropy coding, write compressed code stream, and entropy decoding module AD according to feature map The Gaussian distribution variance of each feature element Treating coded elements Each characteristic element in is entropy decoded to obtain the reconstructed value , will rebuild the value Input decoding transformation module, output reconstructed image ;

[0051] Step 5: Image and reconstruction graph Substitute the calculation formula of the total loss function. When the total loss function is minimized, stop training and obtain the trained image encoding and decoding model.

[0052] Preferably, the weight coefficient of the deviation control loss function is The value is 1.0.

[0053] The advantages provided by the present invention are: With Gaussian mean Perform deviation control and reduce feature map With the estimated Gaussian mean The difference in values ​​makes the element to be encoded The overall size becomes smaller, and at the same time, the feature map Closer to predicting the peak of a Gaussian distribution , thereby reducing the variance of the Gaussian distribution , elements to be encoded The distribution range (characterized by the variance) determines the length of the cumulative distribution function (CDF) during entropy coding. The Gaussian distribution variance The smaller the value is, the smaller the distribution range of the elements to be encoded is, the shorter the CDF length is, the shorter the length of the CDF table search is, the faster the search speed is, and the faster the entropy coding time is. The present invention uses the deviation control loss function and the rate-distortion loss function to jointly train the image encoding and decoding model, which can train a model with better entropy coding performance and lower complexity, thereby improving the speed and coding performance of entropy coding and entropy decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of a typical coding framework for end-to-end image coding in the prior art;

[0055] Figure 2 A mask map defined for a spatial correlation loss function in the prior art;

[0056] Figure 3 A flowchart of an image encoding and decoding model training method based on deviation control provided by an embodiment of the present invention;

[0057] Figure 4 A flowchart of model training in an image encoding and decoding model training method based on deviation control provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in combination with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] like Figure 3 As shown, this embodiment provides an image encoding and decoding model training method based on deviation control. The deviation control is performed by With Gaussian mean This is achieved by constraining the training loss. The mean absolute error is used for constraint, and the deviation control loss function is shown in Equation (7).

[0060] (7)

[0061] in, is the bias control loss function, The feature map i elements, For the i The Gaussian mean of the elements, The feature map The total number of elements in .

[0062] The calculation formula of the rate-distortion loss function is:

[0063] (8)

[0064] in, is the rate-distortion loss function, The feature map The code rate, Side Information The code rate, For images With the reconstruction The distortion loss between is the Lagrange multiplier that trades off bit rate and distortion.

[0065] Based on the deviation control loss function and the rate distortion loss function, the total loss function is determined, and the image encoding and decoding model is trained based on the total loss function. The calculation formula of the total loss function is:

[0066] (9)

[0067] in, is the total loss function, The weight coefficient of the bias control loss function.

[0068] like Figure 4 As shown, the training process includes:

[0069] Step 1: Image Input to the encoding transformation module output feature map , feature map Input to the side information extraction module Output side information , the side information Input to the probability estimation module and output feature map The probability distribution of each characteristic element ;

[0070] Step 2: Feature map With Gaussian mean Input into the quantization module together and output the element to be encoded ; Represents the feature map With Gaussian mean The deviation value is rounded to the nearest integer.

[0071] Step 3: Entropy coding module AE according to feature map The Gaussian distribution variance of each feature element Treating coded elements Each characteristic element in is entropy encoded to obtain a compressed code stream;

[0072] Step 4: Side Information Perform entropy coding, write compressed code stream, and entropy decoding module AD according to feature map The Gaussian distribution variance of each feature element Treating coded elements Each characteristic element in is entropy decoded to obtain the reconstructed value , will rebuild the value Input decoding transformation module, output reconstructed image ;

[0073] Step 5: Image and reconstruction graph Substitute the calculation formula of the total loss function. When the total loss function is minimized, stop training and obtain the trained image encoding and decoding model.

[0074] When training the end-to-end image encoding and decoding model, the present invention sets , use the loss function as shown in formula (10) to train the model.

[0075] (10)

[0076] During model training, the feature map obtained by encoding the image , the mean of its Gaussian distribution will be estimated through the probability estimation network and Gaussian distribution variance The two parameters are used to further obtain the estimated bit rate loss term. In actual encoding and decoding, a probability estimation network is also needed to obtain the Gaussian distribution mean of the feature map. and Gaussian distribution variance , to treat the encoding element Perform entropy coding.

[0077] Elements to be encoded With Gaussian distribution variance The distribution of is crucial to the entropy coding process. First, the elements to be encoded The distribution range of (characterized by the variance) determines the length of the cumulative distribution function (CDF) during entropy coding. The smaller the element to be coded, the shorter the CDF length and the faster the entropy coding time. Secondly, the variance of the Gaussian distribution The smaller the value of , the smaller the distribution range of the elements to be encoded, the shorter the CDF length, the shorter the length of the CDF table search, the faster the search speed, and the faster the entropy coding time.

[0078] In order to obtain a feature map that is easier to entropy code, the present invention proposes to With Gaussian mean Perform deviation control and reduce feature map With the estimated Gaussian mean This operation will make the element to be encoded becomes smaller overall. At the same time, this operation makes the feature map Closer to predicting the peak of a Gaussian distribution , thereby reducing the variance of the Gaussian distribution .

[0079] The present invention also provides an image encoding and decoding model training system based on deviation control, comprising:

[0080] A loss function establishment module, used for determining a total loss function based on a deviation control loss function and a rate distortion loss function;

[0081] The calculation formula of the deviation control loss function is:

[0082] (7)

[0083] in, is the bias control loss function, The feature map i elements, For the i The Gaussian mean of the elements, The feature map The total number of elements in .

[0084] The calculation formula of the rate-distortion loss function is:

[0085] (8)

[0086] in, is the rate-distortion loss function, The feature map The code rate, Side Information The code rate, For images With the reconstruction The distortion loss between is the Lagrange multiplier that trades off bit rate and distortion.

[0087] The calculation formula of the total loss function is:

[0088] (9)

[0089] in, is the total loss function, The weight coefficient of the bias control loss function.

[0090] A joint training module is used to train the image encoding and decoding model based on the total loss function to obtain a trained image encoding and decoding model;

[0091] The training process includes:

[0092] Step 1: Image Input to the encoding transformation module output feature map , feature map Input to the side information extraction module Output side information , the side information Input to the probability estimation module and output feature map The probability distribution of each characteristic element ;

[0093] Step 2: Feature map With Gaussian mean Input into the quantization module together and output the element to be encoded ,in Express Rounding to the nearest integer;

[0094] Step 3: Entropy coding module AE according to feature map The Gaussian distribution variance of each feature element Treating coded elements Each characteristic element in is entropy encoded to obtain a compressed code stream;

[0095] Step 4: Side Information Perform entropy coding, write compressed code stream, and entropy decoding module AD according to feature map The Gaussian distribution variance of each feature element Treating coded elements Each characteristic element in is entropy decoded to obtain the reconstructed value , will rebuild the value Input decoding transformation module, output reconstructed image ;

[0096] Step 5: Image and reconstruction graph Substitute the calculation formula of the total loss function. When the total loss function is minimized, stop training and obtain the trained image encoding and decoding model.

[0097] When training the end-to-end image encoding and decoding model, the present invention sets , use the loss function as shown in formula (10) to train the model.

[0098] (10)

[0099] The image coding model training method based on bias control proposed in the present invention can train a model with better entropy coding performance and lower complexity. Table 1 shows the BD-Rate performance comparison of different models trained with and without the bias control loss function. Table 2 shows the entropy coding time statistics of the models trained with and without the bias control loss function. The model trained with the bias control loss function has better coding performance and reduced entropy coding time.

[0100] Table 1 Comparison of model performance using the image encoding and decoding model training method based on bias control

[0101]

[0102] Table 2 Comparison of model entropy coding time using the image encoding and decoding model training method based on bias control

[0103]

[0104] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image encoding and decoding model training method based on deviation control, characterized in that: Methods include: Based on the deviation control loss function and the rate distortion loss function, a total loss function is determined, and the image encoding and decoding model is trained based on the total loss function to obtain a trained image encoding and decoding model; Bias Control Loss Function The calculation formula is: in, The feature map i elements, For the i The Gaussian mean of the elements, The feature map The total number of elements in , Total loss function The calculation formula is: in, is the rate-distortion loss function, The weight coefficient of the bias control loss function.

2. The image encoding and decoding model training method based on deviation control according to claim 1 is characterized in that: The rate-distortion loss function The calculation formula is: in, The feature map The code rate, Side Information The code rate, For images With the reconstruction The distortion loss between is the Lagrange multiplier that trades off bit rate and distortion.

3. The image encoding and decoding model training method based on deviation control according to claim 1 is characterized in that: The process of training the image encoding and decoding model based on the total loss function includes: Step 1: Image Input to the encoding transformation module output feature map , feature map Input to the side information extraction module Output side information , the side information Input to the probability estimation module and output feature map The probability distribution of each characteristic element ; Step 2: Feature map With Gaussian mean Input into the quantization module together and output the element to be encoded ,in Express Rounding to the nearest integer; Step 3: Entropy coding module AE according to feature map The Gaussian distribution variance of each feature element Treating coded elements Each characteristic element in is entropy encoded to obtain a compressed code stream; Step 4: Side Information Perform entropy coding and write compressed code stream. The entropy decoding module AD is based on the feature map. The Gaussian distribution variance of each feature element Treating coded elements Each characteristic element in is entropy decoded to obtain the reconstructed value , will rebuild the value Input decoding transformation module, output reconstructed image ; Step 5: Image and reconstruction graph Substitute the calculation formula of the total loss function. When the total loss function is minimized, stop training and obtain the trained image encoding and decoding model.

4. The image encoding and decoding model training method based on deviation control according to claim 1 is characterized in that: The bias controls the weight coefficient of the loss function The value is 1.

0.

5. Image encoding and decoding model training system based on deviation control, characterized by: The system includes: A loss function establishment module, used for determining a total loss function based on a deviation control loss function and a rate distortion loss function; A joint training module is used to train the image encoding and decoding model based on the total loss function to obtain a trained image encoding and decoding model; Bias Control Loss Function The calculation formula is: in, The feature map i elements, For the i The Gaussian mean of the elements, The feature map The total number of elements in , Total loss function The calculation formula is: in, is the rate-distortion loss function, The weight coefficient of the bias control loss function.

6. The image encoding and decoding model training system based on deviation control according to claim 5, characterized in that: The rate-distortion loss function The calculation formula is: in, The feature map The code rate, Side Information The code rate, For images With the reconstruction The distortion loss between is the Lagrange multiplier that trades off bit rate and distortion.

7. The image encoding and decoding model training system based on deviation control according to claim 5, characterized in that: The training process of the joint training module includes: Step 1: Image Input to the encoding transformation module output feature map , feature map Input to the side information extraction module Output side information , the side information Input to the probability estimation module and output feature map The probability distribution of each characteristic element ; Step 2: Feature map With Gaussian mean Input into the quantization module together and output the element to be encoded ,in Express Rounding to the nearest integer; Step 3: Entropy coding module AE according to feature map The Gaussian distribution variance of each feature element Treating coded elements Each characteristic element in is entropy encoded to obtain a compressed code stream; Step 4: Side Information Perform entropy coding and write compressed code stream. The entropy decoding module AD is based on the feature map. The Gaussian distribution variance of each feature element Treating coded elements Each characteristic element in is entropy decoded to obtain the reconstructed value , will rebuild the value Input decoding transformation module, output reconstructed image ; Step 5: Image and reconstruction graph Substitute the calculation formula of the total loss function. When the total loss function is minimized, stop training and obtain the trained image encoding and decoding model.

8. The image encoding and decoding model training system based on deviation control according to claim 5, characterized in that: The bias controls the weight coefficient of the loss function The value is 1.0.

Images