Quantization parameter processing method and device, image decoding method and device, and storage medium

Abstract

The invention discloses a quantization parameter processing method, an image decoding method, an image encoding and decoding device and a computer storage medium. The quantization parameter processing method comprises the following steps: acquiring priori information of encoding and decoding; global information of a preset dimension of the prior information is extracted, and the preset dimension is a spatial dimension, a channel dimension or a global dimension; performing feature extraction based on the global information to generate quantization parameters; the quantization parameter is used for a residual information entropy model and / or a motion information entropy model. According to the method and the device, the quantization parameter which is adaptively generated aiming at the priori information of coding and decoding is provided and is used for feature quantization in the entropy model, so that the discrimination and compression performance of the features can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of image encoding and decoding technology, and in particular to a quantization parameter processing method, an image decoding method, an image encoding and decoding device, and a computer storage medium. Background Technology

[0002] Traditional image encoding and decoding techniques are designed for human visual characteristics. However, with the superior performance of deep neural networks in various machine vision tasks, such as image classification, object detection, and semantic segmentation, a large number of AI applications based on machine vision have emerged. To ensure that the performance of machine vision tasks is not compromised by the image encoding process, an analysis-then-encode approach is adopted to meet the demands of machine vision. This involves directly extracting features from the lossless image using a neural network at the image acquisition end, then encoding and transmitting the extracted features. The decoding end directly uses the decoded features as input to subsequent network structures to complete different machine vision tasks. Therefore, to save transmission bandwidth resources, it is necessary to research image encoding methods specifically for machine vision.

[0003] However, traditional video encoding and decoding technologies typically use the Round method to directly convert floating-point features into integer features during quantization, without considering the characteristics of the features to be quantized, which is not conducive to further reducing the feature bit rate. Summary of the Invention

[0004] This application provides a method for quantization parameter processing, an image decoding method, an image encoding / decoding device, and a computer storage medium.

[0005] One technical solution adopted in this application is to provide a method for processing quantization parameters, the method comprising:

[0006] Obtain prior information about encoding and decoding;

[0007] Extract global information of the prior information in a preset dimension, wherein the preset dimension is a spatial dimension, a channel dimension, or a global dimension;

[0008] Feature extraction is performed based on the global information to generate quantization parameters;

[0009] The quantization parameters are used in the residual information entropy model and / or the motion information entropy model.

[0010] Another technical solution adopted in this application is to provide an image decoding method, the image decoding method comprising:

[0011] The quantization parameters are obtained using the quantization parameter processing methods described above.

[0012] The first decoded feature output by entropy decoding is dequantized using the quantization parameters to obtain the second decoded feature;

[0013] The image is reconstructed by decoding based on the second decoding feature.

[0014] Another technical solution adopted in this application is to provide another image decoding method, which includes:

[0015] Obtain the first decoding feature;

[0016] Extract the prior decoding information, prediction information, temporal information transmitted by the decoded frame, and / or the fusion features of the first decoding features;

[0017] Based on the fused features, feature extraction is performed to generate compensation values;

[0018] The first decoding feature is compensated using the compensation value, and the compensated first decoding feature is dequantized using the quantization parameter to obtain the second decoding feature;

[0019] Decode the image based on the second decoding feature to obtain the reconstructed image;

[0020] Alternatively, the compensation value can be used to compensate the floating-point decoding feature after the first decoding feature has been dequantized by the quantization parameter;

[0021] The reconstructed image is obtained by decoding based on the compensated second decoding features.

[0022] Another technical solution adopted in this application is to provide an image encoding and decoding device, which includes a memory and a processor coupled to the memory;

[0023] The memory is used to store program data, and the processor is used to execute the program data to implement the quantization parameter processing method or the image decoding method described above.

[0024] Another technical solution adopted in this application is to provide a computer storage medium for storing program data, which, when executed by a computer, is used to implement the quantization parameter processing method or the image decoding method described above.

[0025] The beneficial effects of this application are: the image encoding / decoding device acquires prior information for encoding and decoding; extracts global information of a preset dimension from the prior information, wherein the preset dimension is a spatial dimension, a channel dimension, or a global dimension; performs feature extraction based on the global information to generate quantization parameters; the quantization parameters are used in a residual information entropy model and / or a motion information entropy model. This application proposes a quantization parameter adaptively generated for prior information of encoding and decoding, used for feature quantization in an entropy model, which can effectively improve the discriminative power and compression performance of features. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of an embodiment of the EEM overall framework provided in this application;

[0028] Figure 2 This is a schematic diagram of the framework of an embodiment of the entropy model provided in this application;

[0029] Figure 3 This is a flowchart of the calculation process for the quantization parameters in the existing scheme;

[0030] Figure 4 This is a flowchart illustrating an embodiment of the quantization parameter processing method provided in this application;

[0031] Figure 5 yes Figure 4 The flowchart of the adaptive quantization scheme is shown below;

[0032] Figure 6 This is a flowchart illustrating an embodiment of the fusion module for generating quantization parameters provided in this application;

[0033] Figure 7 This is a flowchart illustrating an embodiment of the global quantization parameter information extraction module provided in this application;

[0034] Figure 8 This is a flowchart illustrating an embodiment of the channel-level quantization parameter information extraction module provided in this application;

[0035] Figure 9 This is a flowchart illustrating an embodiment of the spatial-level quantization parameter information extraction module provided in this application;

[0036] Figure 10This is a flowchart illustrating the quantization parameter processing method of Implementation Example 1 provided in this application;

[0037] Figure 11 This is a flowchart illustrating the quantization parameter processing method of Implementation Example 2 provided in this application;

[0038] Figure 12 This is a schematic diagram of the entropy model framework based on quantization error compensation provided in this application;

[0039] Figure 13 This is a flowchart illustrating another embodiment of the quantization parameter processing method provided in this application;

[0040] Figure 14 This is a flowchart illustrating the post-quantization compensation method for quantization error compensation provided in this application.

[0041] Figure 15 This is a schematic diagram of the entropy model framework based on pre-quantization error compensation provided in this application;

[0042] Figure 16 This is a flowchart illustrating another embodiment of the quantization parameter processing method provided in this application;

[0043] Figure 17 This is a flowchart illustrating the calculation of compensation values ​​in the pre-quantization compensation method for quantization error compensation provided in this application.

[0044] Figure 18 This is a schematic diagram of the compensation-based quantization process in the pre-quantization compensation method for quantization error compensation provided in this application;

[0045] Figure 19 This is a flowchart illustrating an embodiment of the image decoding method provided in this application;

[0046] Figure 20 This is a flowchart illustrating another embodiment of the image decoding method provided in this application;

[0047] Figure 21 This is a schematic diagram of an embodiment of the image encoding and decoding apparatus provided in this application;

[0048] Figure 22 This is a schematic diagram of the structure of an embodiment of the computer storage medium provided in this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0050] In recent years, deep neural networks have made some progress in video processing, such as video detection, video super-resolution, video denoising, and video enhancement. Due to their powerful non-linear expressive capabilities and the advantages of joint training, deep neural networks have demonstrated great potential in the image / video field.

[0051] Currently, deep learning is also beginning to develop in the field of video compression. Its applications are mainly divided into two categories. The first is as a deep learning tool applied to traditional video encoders. To date, many works have proven that combining traditional coding modules with deep learning is very effective. These modules include, but are not limited to, motion compensation and frame interpolation networks, intra-frame predictive coding modules, bit rate control modules, and post-processing modules. The second is an end-to-end deep video compression framework with deep neural networks as the core of video coding. This framework makes full use of the powerful nonlinear expression capabilities of neural networks and the advantages of end-to-end joint optimization, which can further improve compression efficiency and accuracy.

[0052] Currently, the AVS4 standard is being explored and evolved in three directions. Among them, the exploration platform (software) for end-to-end video encoding and decoding is EEM (Embedded Event Manager). Please refer to [link / reference] for details. Figure 1 , Figure 1 This is a schematic diagram of an embodiment of the EEM overall framework provided in this application.

[0053] like Figure 1 As shown, the EEM framework includes, but is not limited to, the following components:

[0054] Motion information compression and reconstruction:

[0055] (1) Motion estimation: The current frame and the reference frame obtain inter-frame motion information through the motion estimation network.

[0056] (2) Motion Information Encoder: Reduces the dimensionality of motion information (or motion information residual) and extracts compact information to obtain the motion information to be encoded.

[0057] (3) Motion information entropy model: Obtain the probability of each character appearing in the quantized motion information to be encoded, perform arithmetic encoding, and output the motion information bitstream.

[0058] (4) Motion information Decoder: Upgrades the low-dimensional motion features of the decoded data and reconstructs the motion information.

[0059] Motion compensation and temporal prediction: Motion compensation operations such as warping are performed on reference frame feature 1 using motion information. Relevant information is extracted from the reference frame to obtain the predicted frame for the current frame. Reference frame feature 1 is either a previously reconstructed video frame image or a feature obtained through feature extraction processing.

[0060] Context (residual information) compression and reconstruction:

[0061] (1) Context Encoder: Performs dimensionality reduction and compression on the context information between the current frame and the predicted frame. The context information includes, but is not limited to, the difference information or concatenation information between the two frames. The purpose is to remove temporal and intra-spatial correlations between frames.

[0062] (2) Context entropy model: Obtain the probability of each character appearing in the quantized context information to be encoded, perform arithmetic encoding, and output the context information bitstream.

[0063] (3) Context Decoder: The low-dimensional context features obtained from decoding the above bitstream are upscaled and reconstructed to obtain context information. In this process, the context information is combined with the predicted frame to obtain preliminary reconstruction information. For example, if the context at the Encoder end is a difference, the Decoder end will add the predicted frame information to obtain the preliminary reconstruction information of the current frame.

[0064] Frame reconstruction: Further processing of the preliminary reconstruction information yields the reconstructed image and reconstructed features of the current frame. Both are stored in the cache information and can be used as a reference frame for the next frame.

[0065] in, Figure 1 Please refer to the entropy model structures of the residual information entropy model and / or motion information entropy model shown below. Figure 2 , Figure 2 This is a schematic diagram of the framework of an embodiment of the entropy model provided in this application.

[0066] like Figure 2 As shown, dashed lines represent optional information flow directions and optional components. For example, the residual information entropy model has no quantization parameter and inverse quantization branch, while the motion information entropy model has a quantization parameter branch and an inverse quantization branch.

[0067] Let the feature to be encoded (motion information or context information, etc.) be y, the scaled mean be μ, the quantization parameter be q (residual information does not have q), and the quantized integer feature to be encoded be y_int. Then the information quantization scheme of EEM can be expressed as:

[0068] For the motion information entropy model, we have:

[0069] y int =round[(y-μ) / q]

[0070] For the residual information entropy model, we have:

[0071] y int =round[(y-μ)]

[0072] The EEM information inverse quantization scheme can be expressed as:

[0073] For the motion information entropy model, we have:

[0074] yhat=yint*q+μ

[0075] For the residual information entropy model, we have:

[0076] yhat=yint+μ

[0077] In this scheme, the quantization parameter q has the same dimension as y, and each pixel value in y has a scaling factor. Let the dimensions of y be [h, w, c], corresponding to the width, height, and channels, respectively. For details on the structure of generating the quantization parameter q in the existing scheme's fusion module, please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is a flowchart of the calculation process for the quantization parameters in the existing scheme.

[0078] like Figure 3 As shown, the prior information includes super-prior decoding information, prediction information, and temporal information transmitted from the decoded frames (such as motion features, residual features, etc.). In existing schemes, the prior information is processed by feature extraction 1, and the output has the same dimension as the feature to be quantized, y, i.e., [h, w, c]. Then, the scaling parameter q is obtained by limiting the range of clipping.

[0079] Based on the above-mentioned technical foundation, in order to solve the problems of the existing technology, this application proposes a video end-to-end compression method based on adaptive quantization, which realizes global adaptive quantization and local adaptive quantization. According to prior information, this application uses different quantization step sizes for different regions of the features to be encoded, thereby reducing the codeword length and improving the encoding compression ratio.

[0080] Please refer to details. Figure 1 The overall framework of end-to-end video encoding and decoding Figure 4 , Figure 4 This is a flowchart illustrating an embodiment of the quantization parameter processing method provided in this application. The quantization parameter processing method of this application mainly targets... Figure 1 Improvements are made to the residual information entropy model and / or motion information entropy model shown.

[0081] Figure 1 The quantization process shown includes scaling and rounding, while the quantization parameter processing scheme provided in this application only improves the scaling process. Unlike the spatial quantization of existing schemes, each feature point of the information to be quantized, including the information to be encoded and the information to be decoded, has a different scaling factor.

[0082] The quantization parameter processing scheme provided in this application, based on the existing technology, designs a fusion module to generate global quantization parameters and local quantization parameters to scale the information to be quantized. This process can be regarded as global attention and channel-level attention, which pays more attention to important features (small quantization coefficients) and improves the discriminativeness and compression efficiency of features.

[0083] Global quantization parameters: In the input features to be encoded, all location points share a scaling factor.

[0084] Channel-level quantization parameters: Each feature channel shares a scaling factor.

[0085] Spatial-level quantization parameters: In the spatial dimension, different channels at each location point share the scaling factor.

[0086] It should be noted that the scaling in the dequantization process is the opposite of the scaling in the quantization process, and only includes the scaling process. The two scaling methods mentioned above can be used in combination with existing spatial scaling methods or alone.

[0087] like Figure 4 As shown, the specific steps are as follows:

[0088] Step S11: Obtain prior information for encoding and decoding.

[0089] In the embodiments of this application, please refer to Figure 5 , Figure 5 yes Figure 4 The diagram shows the quantization flowchart of the adaptive quantization scheme. Figure 5 As shown, the image encoding / decoding device acquires prior information, specifically including... Figure 5 The information includes prior decoding information, prediction information, and temporal information (such as motion features, residual features, etc.) transmitted from the decoded frames.

[0090] For example, the process of generating the aforementioned priori decoding information is as follows:

[0091] The image encoding and decoding device performs a priori encoding on the features to be encoded corresponding to the bitstream to be decoded, then obtains the binary bitstream through a quantization and entropy encoding model, and finally inputs the binary bitstream into the entropy decoding model for decoding to obtain the a priori decoding information.

[0092] Step S12: Extract global information of the preset dimension of the prior information, wherein the preset dimension is a spatial dimension, a channel dimension, or a global dimension.

[0093] In this embodiment of the application, the image encoding and decoding device utilizes Figure 5 The fusion module extracts features from prior information, generates global or local quantization parameters q, and scales the information to be quantized.

[0094] It should be noted that, Figure 5 The fusion module can be used to generate quantization parameters and probability parameters. This application embodiment only shows the part of the fusion module that generates quantization parameters.

[0095] The information to be quantified is: length and width are h and w, channel is c, and dimension is [h, w, c].

[0096] Step S13: Extract features based on global information and generate quantization parameters.

[0097] In the embodiments of this application, please refer to Figure 6 , Figure 6 This is a flowchart illustrating an embodiment of the fusion module for generating quantization parameters provided in this application.

[0098] like Figure 6 As shown, the fusion module includes a global / local information extraction module and a range limiting module (optional). The global / local information extraction module is used to acquire different global information and generate a global / local scaling parameter. The range limiting module is used to limit the numerical range of q (the range is a user-defined empirical value, not strictly limited), and methods include, but are not limited to, combinations of activation functions such as sigmoid and tanh with scaling factors, or truncation functions such as clip.

[0099] The difference between global quantization parameters and local quantization parameters lies in the dimension, meaning the information extraction modules differ, which can be categorized into three cases:

[0100] In the first case, the global quantization parameter has a dimension of [1], and only one quantization parameter q is output. The structure diagram of the information extraction module is as follows: Figure 7 As shown, Figure 7 This is a flowchart illustrating an embodiment of the global quantization parameter information extraction module provided in this application. The global information extraction module is used to acquire global information and generate a global scaling parameter (dimension [1]).

[0101] like Figure 7 As shown, the information extraction module includes at least one global information extraction module, including but not limited to global pooling, averaging, etc.; the output global scaling parameter is 1, and the dimension is [1]. The information extraction module also includes a feature extraction module, including but not limited to convolutional networks, convolution + attention combination and transformer neural networks.

[0102] In the second case, the quantization parameters are at the channel level, with dimensions [1, 1, c], and each channel shares one quantization parameter q. The information extraction module structure diagram is shown below. Figure 8 As shown, Figure 8 This is a flowchart illustrating an embodiment of the channel-level quantization parameter information extraction module provided in this application.

[0103] Among them, the channel-level global information extraction module is used to obtain the global information of each channel and generate the scaling parameter q (dimension [1, 1, c]) for each channel.

[0104] like Figure 8 As shown, the information extraction module includes:

[0105] Feature extraction 1. Feature transformation and dimensionality transformation, including but not limited to neural networks such as convolutional networks.

[0106] Information extraction module. It includes at least one channel-level global information extraction (each channel is converted into a scaling factor), including but not limited to spatial-level global pooling, averaging operations, etc.; outputs the global information of each channel, with dimensions [1, 1, c].

[0107] Feature extraction 2. Jointly learn information from multiple channels to generate quantization parameters adapted to each channel, including but not limited to neural networks such as convolutional networks, convolutional + attention combinations, and transformers.

[0108] The third case involves spatial-level quantization parameters, with dimensions [h, w, 1], where each space shares a single quantization parameter q. The information extraction module structure diagram is shown below. Figure 9 As shown, Figure 9 This is a flowchart illustrating an embodiment of the spatial-level quantization parameter information extraction module provided in this application.

[0109] Among them, the spatial-level global information extraction module is used to obtain global information for each space and generate scaling parameters q for each space (with dimensions [h, w, 1]).

[0110] like Figure 9 As shown, the information extraction module includes:

[0111] Feature extraction 1. Feature transformation and dimensionality transformation, including but not limited to neural networks such as convolutional networks.

[0112] Information extraction module. It includes at least one spatial-level global information extraction (all feature maps are transformed into one feature map), including but not limited to spatial-level global pooling (pooling all channels in each space), averaging operations, and fully connected operations; it outputs the global information of each spatial location point, with dimensions [h, w, 1].

[0113] Feature extraction 2. Jointly learn information from multiple channels to generate quantization parameters adapted to each channel, including but not limited to convolutional networks, convolutional + global pooling combinations, and transformers.

[0114] It should be noted that the quantization parameters in this application are applicable to the encoding or decoding process of the residual information entropy model, and also applicable to the encoding or decoding process of the motion information entropy model.

[0115] In the encoding process, the image codec uses quantization parameters obtained from quantization parameter processing methods to quantize the features to be encoded. Specifically, the image codec uses the generated quantization parameters to scale the input floating-point information y to be encoded. The scaled information is then integerized (F), using methods including, but not limited to, rounding.

[0116] yint = F[y / q]

[0117] For the decoding process, the dequantization scheme is the reverse of the scaling process in the quantization scheme, i.e., it is a scaling process. Therefore, the dequantized floating-point type of the decoded features is:

[0118] yfloat=yint*q

[0119] This application provides Implementation Example 1, which takes global adaptive quantization as an example. This method is used in a motion entropy model.

[0120] The process of generating the quantization parameter q in Implementation Example 1 is as follows:

[0121] Input: Prior information, including super-prior decoding information, prediction information, and temporal information (such as motion features, residual features, etc.) passed from the decoded frames.

[0122] Fusion module: Outputs a global quantization parameter q; see structure below. Figure 10 , Figure 10 This is a flowchart illustrating the quantization parameter processing method of Implementation Example 1 provided in this application.

[0123] The clip function controls the input range to [1, 1000], sets values ​​less than 1 to 1, sets values ​​greater than 1000 to 1000, and leaves other values ​​unchanged.

[0124] Scaling and integerization, the integerization function F is for rounding:

[0125] yint = Round[y / q]

[0126] Inverse quantization:

[0127] yfloat=yint*q

[0128] This application provides an implementation example 2, taking channel-level local adaptive quantization as an example. This method is used in the motion entropy model, where the dimension of the information to be quantized is [h, w, c], and the three variables are the length, width and channel, respectively.

[0129] The process of generating the quantization parameter q in Implementation Example 2 is as follows:

[0130] Input: Prior information, including super-prior decoding information, prediction information, and temporal information (such as motion features, residual features, etc.) passed from the decoded frames.

[0131] Fusion module: Outputs a channel-level quantization parameter q; see structure below. Figure 11 , Figure 11 This is a flowchart illustrating the quantization parameter processing method provided in Implementation Example 2 of this application.

[0132] The clip function controls the input range to [1, 2000]. Values ​​less than 1 are set to 1, and values ​​greater than 2000 are set to 2000. Other values ​​remain unchanged.

[0133] Scaling and integerization, the integerization function F is for rounding:

[0134] y int =Round[y / q]

[0135] Inverse quantization:

[0136] yfloat=yint*q

[0137] In this application, an image encoding / decoding device acquires prior information for encoding and decoding; extracts global information of a preset dimension from the prior information, wherein the preset dimension is a spatial dimension, a channel dimension, or a global dimension; performs feature extraction based on the global information to generate quantization parameters; the quantization parameters are used in a residual information entropy model and / or a motion information entropy model. This application proposes a quantization parameter adaptively generated for prior information of encoding and decoding, used for feature quantization in an entropy model, which can effectively improve the discriminative power and compression performance of features.

[0138] This application also proposes an end-to-end video compression method based on quantization error compensation to achieve quantization error compensation. This application includes compensation methods at the encoding or decoding end, which learn to generate compensation values ​​within a specific range based on existing prior information, and append these values ​​to the features to be encoded or decoded to compensate for information loss caused by integerization processes such as rounding.

[0139] The quantization error compensation scheme in this application includes quantization error compensation after quantization; please refer to [link / reference] for details. Figure 12 , Figure 12This is a schematic diagram of the entropy model framework based on quantization error compensation provided in this application.

[0140] like Figure 12 As shown, the entropy model mainly generates a specific range of compensation values ​​(simulating the range of numerical changes in rounding) based on prior information, temporal information transmitted by the decoded frame, and / or prediction information. These values ​​are then added to the integer features of the decoded data to compensate for the information loss caused by the rounding process during encoding, and then an inverse quantization operation is performed.

[0141] It should be noted that this scheme is a post-quantization compensation. Compensation is only performed on the decoded integer data at the decoding end.

[0142] based on Figure 12 In addition to the entropy model framework shown, this application proposes another method for processing quantization parameters. Please refer to [link / reference needed] for details. Figure 13 , Figure 13 This is a flowchart illustrating another embodiment of the quantization parameter processing method provided in this application.

[0143] like Figure 13 As shown, the specific steps are as follows:

[0144] Step S21: Extract the prior decoding information, prediction information, temporal information transmitted by the decoded frame, and / or the first fusion feature of the first decoding feature.

[0145] In the embodiments of this application, please refer to Figure 12 Continue reading Figure 14 , Figure 14 This is a flowchart illustrating the post-quantization compensation method for quantization error compensation provided in this application.

[0146] like Figure 14 As shown, the error compensation module structure in this embodiment includes two parts: calculating the compensation value and compensation. In the compensation value calculation part, the image encoding and decoding device inputs the prior decoding information, prediction information, temporal information transmitted from the decoded frame, and / or decoding information into the feature fusion module to extract fused features. The feature fusion method is to perform feature extraction and transformation on the above inputs, and the method includes, but is not limited to, convolutional networks, attention networks, and transformers.

[0147] It should be noted that the temporal information transmitted by the decoded frame is an intermediate variable generated during the decoding process of the decoded frame, which can be passed to the current frame as buffer information.

[0148] It should be noted that the first and second decoding features involved in this application can be integer decoding features or floating-point decoding features, and no specific limitation is made on the format and form of the features here.

[0149] Step S22: Extract features based on the first fusion feature to generate the first compensation value.

[0150] In this embodiment, the image encoding / decoding device can further limit the range of the feature fusion result to make the generated compensation value as consistent as possible with the quantization loss range. For example, for rounding to integer (quantization loss range is (-0.5, 0.5)), the compensation value is obtained.

[0151] Furthermore, such as Figure 12 As shown, the quantization error compensation module in the post-quantization compensation scheme has two embedding locations: before dequantization and after dequantization. The compensation value ranges differ between these two locations. For example, for rounding to integers, the following applies:

[0152] 1) When the quantization error compensation module is placed before dequantization. The range of value change after rounding (quantization loss range): (-0.5, 0.5).

[0153] 2) When the quantization error compensation module is placed after dequantization (dequantization of decoded information is multiplied by the quantization parameter q), the quantization loss range is: (-0.5, 0.5] * q.

[0154] If there is no quantization parameter, that is, no inverse quantization and no scaling during quantization, then the two embedding positions are equivalent.

[0155] Therefore, depending on the different embedding positions of the quantization error compensation module in the entropy model, the image encoding and decoding device needs to select one of steps S23 and S24 to execute.

[0156] Step S23: Use the first compensation value to compensate for the quantization error of the first decoded feature.

[0157] In this embodiment, if the quantization error compensation module is placed before dequantization, the image encoding and decoding device uses the compensation value to compensate for the quantization error of the integer decoding feature. The compensated integer decoding feature is then dequantized to obtain a floating-point decoding feature for image decoding.

[0158] At this time, the output range of the range limiting module can be [-0.5, 0.5]. It should be noted that the embodiments of this application do not limit whether the range boundaries of -0.5 and 0.5 are included.

[0159] Step S24: Use the first compensation value to compensate for the quantization error of the second decoding feature.

[0160] In this embodiment, if the quantization error compensation module is placed after dequantization, the image encoding and decoding device first dequantizes the integer decoding feature to obtain the floating-point decoding feature, and then uses the compensation value to compensate for the quantization error of the floating-point decoding feature to obtain the compensated floating-point decoding feature, which is used for image decoding.

[0161] At this point, the output range of the range limiting module can be: [-0.5*q, 0.5*q]. It should be noted that multiplying by q here is because the decoded features have already undergone dequantization, meaning that the dequantization process also needs compensation.

[0162] The compensation involved in the embodiments of this application is to pass the compensation value to the decoding information. The methods include, but are not limited to, splicing + convolutional network, addition, convolutional network + addition + multiplication spatial feature transformation methods, and combinations thereof.

[0163] This application provides implementation example 3, taking post-quantization compensation as an example. This compensation method is used in the motion entropy model, and the error compensation module is placed after entropy decoding.

[0164] The inputs to the error compensation module are as follows:

[0165] a) Input to the feature fusion module: prior decoding information, fusion results (cached motion features, residual features, and prediction information).

[0166] b) Decoding information: Decoding information output by entropy decoding.

[0167] The error compensation module calculates the compensation value as follows:

[0168] a) Feature fusion: Select residual network.

[0169] b) Range restriction: Select the tanh function to restrict the input to the range of (-1, 1), and then multiply by the scaling factor q = 0.5 to restrict the compensation value range to the range of (-0.5, 0.5).

[0170] The error compensation module compensates in the following way: Select the addition operation, that is, add the compensation value and the decoding information to obtain the compensated decoding information.

[0171] The quantization error compensation scheme in this application also includes quantization error compensation before quantization; please refer to [link / reference] for details. Figure 15 , Figure 15 This is a schematic diagram of the entropy model framework based on quantization error compensation before quantization provided in this application.

[0172] like Figure 15As shown, the entropy model mainly generates a specific range of compensation values ​​(simulating the range of numerical changes in rounding) based on prior information and / or temporal information and / or prediction information transmitted by the decoded frame. These values ​​are then added to the integer features of the decoded data to compensate for the information loss caused by the rounding process during encoding, and then an inverse quantization operation is performed.

[0173] It should be noted that this scheme is a pre-quantization compensation. On the encoding side, the information to be encoded is first appended with a compensation value, and then rounded to the nearest integer; on the decoding side, the decoded integer data needs to be subtracted from the compensation value to reduce quantization loss.

[0174] contrast Figure 12 The entropy model framework shown is based on quantization error compensation after quantization. The module for calculating the compensation value is the same as that in the entropy model framework for quantization error compensation before quantization in this application. The difference is that: at the encoding end, before the information to be encoded is integerized (such as rounded), the compensation value is added first to simulate the integerization process.

[0175] based on Figure 15 In addition to the entropy model framework shown, this application proposes another method for processing quantization parameters. Please refer to [link / reference needed] for details. Figure 16 , Figure 16 This is a flowchart illustrating another embodiment of the quantization parameter processing method provided in this application.

[0176] like Figure 16 As shown, the specific steps are as follows:

[0177] Step S31: Extract the prior decoding information, prediction information, temporal information transmitted by the decoded frame, and / or the second fusion feature of part of the first decoding feature.

[0178] In the embodiments of this application, please refer to Figure 15 Continue reading Figure 17 and Figure 18 , Figure 17 This is a flowchart illustrating the calculation of compensation values ​​in the pre-quantization compensation method for quantization error compensation provided in this application. Figure 18 This is a schematic diagram of the compensation-based quantization process in the pre-quantization compensation method for quantization error compensation provided in this application.

[0179] It should be noted that the second fusion feature extraction requires only a portion of the first decoded features, not all of them.

[0180] in, Figure 17 The process of calculating the compensation value shown is the same as the process of steps S21 and S22 above, and will not be repeated here.

[0181] At the encoding end, Figure 18 The compensation-based quantization process shown is as follows:

[0182] Let the information to be encoded be y, the quantization parameter be q, the integerization function be F (usually rounded), and the compensation value be A, then Figure 18 The scaling, compensation, and integerization process can be formalized as follows:

[0183]

[0184] in, It signifies compensation.

[0185] Image encoding and decoding devices pass compensation values ​​to encoded information, and the methods include, but are not limited to, splicing + convolutional networks, addition, convolutional networks + addition + multiplication spatial feature transformation methods, and combinations thereof.

[0186] It should be noted that, Figure 18 The compensation-based quantization process is as follows: first, the quantization parameters are used for scaling, and the scaled features are compensated. Finally, the quantization parameters are used to integerize the compensated features.

[0187] Step S32: Extract features based on the second fusion features to generate a second compensation value.

[0188] In this embodiment of the application, the anti-compensation method is as follows: Figure 18 The compensation process is the reverse process, and the compensation module also has two embedding positions, which correspond to the following steps S33 and S34 respectively.

[0189] Step S33: Use the second compensation value to perform inverse compensation on the first decoded feature.

[0190] In this embodiment of the application, if the anti-compensation module is placed after entropy decoding and before dequantization, the floating-point characteristic of the anti-compensation output at this position is:

[0191]

[0192] Then, the floating-point feature is dequantized.

[0193] Step S34: Use the second compensation value to perform inverse compensation on the second decoded feature obtained after inverse quantization of the first decoded feature.

[0194] In this embodiment, if the anti-compensation module is placed after dequantization, the decoded integer information has already undergone dequantization, and the compensation value can be dequantized or not. Therefore, the floating-point characteristics of the anti-compensation output have two types:

[0195]

[0196] or

[0197]

[0198] in, This indicates counter-compensation.

[0199] The anti-compensation methods involved in the embodiments of this application include, but are not limited to:

[0200] 1) If compensation includes addition, then countercompensation must at least include subtraction.

[0201] 2) If compensation includes splicing, then anti-compensation includes at least subtraction or splicing.

[0202] It should be noted that in methods 1) and 2) above, the additional modules may also include, but are not limited to, convolutional networks, transformers, etc.

[0203] This application provides implementation example 4, taking pre-quantization compensation as an example. This compensation method is used in the motion entropy model, and the anti-compensation module is placed after dequantization.

[0204] The process of calculating the compensation value is as follows:

[0205] a) Input:

[0206] i. Input to the feature fusion module: prior decoding information, fusion results (cached motion features, residual features, and prediction information).

[0207] ii. Decoding information: Decoding information output by entropy decoding.

[0208] b) Feature fusion: residual network.

[0209] c) Range limitation: Select the tanh function to limit the input to the range of (-1, 1), and then multiply by the scaling factor q0.5 to limit the compensation value range to the range of (-0.5, 0.5).

[0210] The compensation-based quantization process is as follows: compensation uses an addition operation, and integerization uses rounding. The resulting integer information is:

[0211] yint=Round[y / q+A], A∈(-0.5, 0.5)

[0212] If the inverse compensation module is placed after inverse quantization, the output of the inverse compensation will be:

[0213] yfloat=((yint*q)-(A*q))

[0214] Based on the above quantization parameter processing method, this application also provides a related image decoding method, for details please refer to [link / reference needed]. Figure 19 , Figure 19 This is a flowchart illustrating an embodiment of the image decoding method provided in this application.

[0215] like Figure 19 As shown, the specific steps are as follows:

[0216] Step S41: Obtain quantization parameters.

[0217] In this embodiment of the application, the quantization parameter can be the quantization parameter obtained by the above-described quantization parameter processing method, and the process will not be described in detail here.

[0218] Step S42: Use the quantization parameters to dequantize the first decoded feature output by entropy decoding to obtain the second decoded feature.

[0219] Step S43: Decode according to the second decoding feature to obtain the reconstructed image.

[0220] Please continue reading. Figure 20 , Figure 20 This is a flowchart illustrating another embodiment of the image decoding method provided in this application.

[0221] like Figure 20 As shown, the specific steps are as follows:

[0222] Step S51: Obtain the first decoding feature.

[0223] Step S52: Extract the super-prior decoding information, prediction information, temporal information transmitted by the decoded frame, and / or the fusion features of the first decoding features.

[0224] Step S53: Extract features based on the fused features and generate compensation values.

[0225] Step S54: Compensate the first decoded feature using the compensation value, and dequantize the compensated first decoded feature using the quantization parameter to obtain the second decoded feature.

[0226] Step S55: Decode according to the second decoding feature to obtain the reconstructed image.

[0227] Step S56: Use the compensation value to compensate the second decoded feature after the first decoded feature has been dequantized by the quantization parameter.

[0228] Step S57: Decode the image based on the compensated second decoding features to obtain the reconstructed image.

[0229] In this embodiment, the image decoding method employs a quantization error compensation scheme, including pre-quantization compensation and post-quantization compensation, to compensate for the decoding features and make up for information loss caused by integerization processes such as rounding. The quantization parameters involved in this scheme can be quantization parameters determined according to conventional techniques, or quantization parameters obtained by the aforementioned quantization parameter processing method; no specific limitation is made here.

[0230] This application proposes a global and local adaptive quantization method that quantizes the information to be encoded to different degrees. Important information is assigned small quantization parameters, while smooth regions are assigned large quantization parameters, thereby improving the discriminative power and compression performance of features.

[0231] This application proposes a global adaptive quantization method in which all positions of the input information to be encoded share a single quantization parameter, achieving simple and efficient adaptive quantization based on prior information.

[0232] This application proposes a channel-level local adaptive quantization. Each channel shares a quantization parameter, and based on prior information, each channel is given different levels of attention and quantization.

[0233] This application proposes a spatial-level local adaptive quantization method. Multiple channels share a single quantization parameter, but each spatial location uses a different quantization parameter, assigning different quantization levels to different spatial locations based on prior information.

[0234] This application proposes an error compensation scheme that learns to generate compensation values ​​within a specific range based on existing prior information, thereby compensating for information loss caused by integerization processes such as rounding.

[0235] This application proposes an error compensation scheme that can compensate at the encoding or decoding end to make up for the information loss caused by the integerization process.

[0236] The modules proposed in this application can be combined to form multiple solutions.

[0237] The above embodiments are merely one common example of this application and do not constitute any limitation on the technical scope of this application. Therefore, any minor modifications, equivalent changes, or alterations made to the above content based on the substance of the solution of this application shall still fall within the scope of the technical solution of this application.

[0238] Please continue reading Figure 21 , Figure 21 This is a schematic diagram of an embodiment of the image encoding / decoding apparatus provided in this application. The image encoding / decoding apparatus 600 of this application embodiment includes a processor 61, a memory 62, an input / output device 63, and a bus 64.

[0239] The processor 61, memory 62, and input / output device 63 are connected to the bus 64. The memory 62 stores program data, and the processor 61 is used to execute the program data to implement the quantization parameter processing method and / or image decoding method described in the above embodiments.

[0240] In this embodiment, processor 61 can also be referred to as a CPU (Central Processing Unit). Processor 61 may be an integrated circuit chip with signal processing capabilities. Processor 61 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 61 can be any conventional processor.

[0241] This application also provides a computer storage medium; please refer to the following: Figure 22 , Figure 22 This is a schematic diagram of a computer storage medium according to an embodiment of the present application. The computer storage medium 700 stores program data 71, which, when executed by a processor, is used to implement the quantization parameter processing method and / or image decoding method of the above embodiments.

[0242] When the embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0243] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for processing quantization parameters, characterized in that, The quantization parameter processing method includes: Obtain prior information about encoding and decoding; Extract global information of the prior information in a preset dimension, wherein the preset dimension is a spatial dimension, a channel dimension, or a global dimension; Feature extraction is performed based on the global information to generate quantization parameters; The quantization parameters are used in the residual information entropy model and / or the motion information entropy model.

2. The quantization parameter processing method according to claim 1, characterized in that, The global dimension includes one dimension, and its value is a preset value; The channel dimension includes three dimensions, and the value of one dimension is the channel dimension value of the information to be quantized, while the values ​​of the remaining dimensions are preset values. The spatial dimension includes three dimensions, with the values ​​of two of the dimensions being the length and width of the information to be quantified, respectively, and the values ​​of the remaining dimensions being preset values.

3. The quantization parameter processing method according to claim 1, characterized in that, The prior information includes one or more of the following: advanced prior decoding information, prediction information, temporal information transmitted from decoded frames, and information on the current frame that has been decoded.

4. The quantization parameter processing method according to claim 1, characterized in that, After performing feature extraction based on the global information and generating quantization parameters, the quantization parameter processing method further includes: Obtain the first decoded feature from the entropy decoding output; The first decoded feature is dequantized using the quantization parameters to obtain the second decoded feature.

5. The quantization parameter processing method according to claim 4, characterized in that, The quantization parameter processing method further includes: Extract the prior decoding information, prediction information, temporal information transmitted by the decoded frame, and / or the first fusion feature of the first decoding feature; Based on the first fused feature, feature extraction is performed to generate a first compensation value; After obtaining the first decoding feature of the entropy decoding output, the quantization parameter processing method further includes: The quantization error of the first decoding feature is compensated using the first compensation value; Alternatively, the first compensation value can be used to compensate for the quantization error of the second decoding feature.

6. The quantization parameter processing method according to claim 5, characterized in that, The step of compensating for the quantization error of the second decoding feature using the first compensation value includes: The quantization error of the second decoding feature is compensated using the first compensation value and the quantization parameter.

7. The quantization parameter processing method according to claim 4, characterized in that... The quantization parameter processing method further includes: Extract the prior decoding information, prediction information, temporal information transmitted by the decoded frame, and / or a portion of the second fusion feature of the first decoding feature; Based on the second fusion feature, feature extraction is performed to generate a second compensation value; After obtaining the first decoding feature of the entropy decoding output, the quantization parameter processing method further includes: The first decoded feature is inversely compensated using the second compensation value; Alternatively, the second compensation value can be used to perform inverse compensation on the second decoded feature obtained after inverse quantization of the first decoded feature.

8. The quantization parameter processing method according to claim 7, characterized in that, The step of inversely compensating the second decoded feature obtained by inverse quantization of the first decoded feature using the second compensation value includes: The second compensation value is dequantized using the quantization parameters; The second decoding feature obtained by dequantizing the first decoding feature is inversely compensated using the second compensation value obtained by dequantizing the first decoding feature.

9. The quantization parameter processing method according to claim 7 or 8, characterized in that, The quantization parameter processing method further includes: The first encoded feature is scaled using the quantization parameters; The second compensation value is used to compensate and integerize the scaled first coding feature to obtain the second coding feature.

10. An image decoding method, characterized in that, The image decoding method includes: The quantization parameters are obtained by the quantization parameter processing method according to any one of claims 1 to 9; The first decoded feature output by entropy decoding is dequantized using the quantization parameters to obtain the second decoded feature; The image is reconstructed by decoding based on the second decoding feature.

11. An image decoding method, characterized in that, The image decoding method includes: Obtain the first decoding feature; Extract the prior decoding information, prediction information, temporal information transmitted by the decoded frame, and / or the fusion features of the first decoding features; Based on the fused features, feature extraction is performed to generate compensation values; The first decoding feature is compensated using the compensation value, and the compensated first decoding feature is dequantized using the quantization parameter to obtain the second decoding feature; Decode the image based on the second decoding feature to obtain the reconstructed image; Alternatively, the compensation value can be used to compensate the second decoded feature after the first decoded feature has been dequantized using the quantization parameter; The reconstructed image is obtained by decoding based on the compensated second decoding features.

12. An image encoding and decoding apparatus, characterized in that, The image encoding / decoding device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the quantization parameter processing method as described in any one of claims 1 to 9, or the image decoding method as described in any one of claims 10 to 11.

13. A computer storage medium, characterized in that, The computer storage medium is used to store program data, which, when executed by the computer, is used to implement the quantization parameter processing method as described in any one of claims 1 to 9, or the image decoding method as described in any one of claims 10 to 11.

Images