Image information prediction method and device

Abstract

The embodiment of the invention provides an image information prediction method and device. The method comprises the following steps: acquiring an initial input feature and a hyper-prior feature corresponding to a target image; according to a selected early-stage image processing mode, generating an early-stage result feature corresponding to the target image based on the initial input feature and the hyper-prior feature; based on the early-stage image processing mode, determining a probability prediction model for later-stage image processing of the early-stage result features; wherein the model structure of the probability prediction model is adapted to the feature distribution of the early-stage result features; predicting probability distribution features of the early-stage result features through a probability prediction model based on the hyper-prior features; the probability distribution features are used for performing later image processing on the early result features. Therefore, the problem that the probability prediction model is not matched with the generation mode and the distribution characteristic of the early-stage result characteristics in the later-stage image processing process is solved, and the later-stage image processing efficiency is improved.

Description

Technical Field

[0001] Some embodiments of the present application relate to the field of image processing technology, and more specifically, to an image information prediction method and device. Background Art

[0002] With the development of technology, images / videos (composed of multiple consecutive frames of images) have been applied to a variety of scenarios. Among them, there is such a situation for image processing: the image is processed by different algorithms of the same image processing function in the early stage to obtain the early result features, and the same image processing algorithm is used in the later stage to continue the later image processing of the early result features in combination with the probability distribution features estimated for the early result features. For example, in the image encoding process, in order to losslessly compress the redundant information in the image, an entropy coding method is proposed in the relevant technology, which mainly provides a probability prediction model to estimate the probability distribution of the image features to be entropy encoded, and then entropy coding is performed based on the probability distribution.

[0003] However, in the process of implementing this application, the applicant found that due to the use of different image processing algorithms in the preliminary processing, the feature distributions of different preliminary result features obtained were different. Therefore, the unified probability prediction model used in the later stage could not adapt well to the various preliminary result features, resulting in poor accuracy of the estimated probability distribution, which in turn caused low efficiency of the later image processing. Summary of the invention

[0004] The exemplary embodiments of the present application provide an image information prediction method and device for solving the problem of poor compatibility between a unified probability prediction model and preliminary result features obtained by using different preliminary processing algorithms.

[0005] Some embodiments of the present application provide the following technical solutions:

[0006] In a first aspect, some embodiments of the present application provide an image information prediction method, including:

[0007] Obtaining initial input features and hyper-prior features corresponding to the target image;

[0008] According to the selected preliminary image processing mode, based on the initial input features and the super prior features, a preliminary result feature corresponding to the target image is generated;

[0009] Based on the previous image processing mode, determining a probability prediction model for the later image processing of the previous result features; wherein the model structure of the probability prediction model is adapted to the feature distribution of the previous result features;

[0010] Based on the super priori features, the probability distribution features of the previous result features are predicted through the probability prediction model; the probability distribution features are used to perform the post-image processing on the previous result features.

[0011] In a second aspect, some embodiments of the present application provide an image information prediction device, including:

[0012] a memory configured to store a computer program;

[0013] The processor is configured to, when calling a computer program, enable the image information prediction to implement the image information prediction method described in the first aspect.

[0014] In a third aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computing device, the computing device implements the image information prediction method described in the first aspect.

[0015] In a fourth aspect, some embodiments of the present application provide a computer program product, which, when executed on a computer, enables the computer to implement the image information prediction method.

[0016] It can be seen from the above technical scheme that the image information prediction method provided in the embodiment of the present application first obtains the initial input features and super prior features corresponding to the target image when processing the target image, and generates the preliminary result features corresponding to the target image to be subjected to post-processing based on the initial input features and the super prior features according to the selected preliminary image processing mode; the preliminary result features generated by different preliminary image processing modes have different feature distributions, so according to the preliminary image processing mode, a probability prediction model for post-image processing of the preliminary result features, whose model structure is adapted to the feature distribution of the preliminary result features, can be designed and determined; then, based on the super prior features, through the probability prediction model, The probability distribution characteristics of the previous result characteristics are predicted, and the previous result characteristics are subjected to post-image processing using the probability distribution characteristics to obtain the later result characteristics of the target image. In this way, different probability prediction models are designed for different previous image processing modes, which solves the problem of poor adaptability between the probability prediction model and the previous result characteristics in the later image processing process, so that the predicted probability distribution characteristics can match the characteristic distribution of the previous result characteristics obtained by the corresponding previous image processing mode, thereby improving the degree of fit between the probability prediction model and the characteristic distribution of the previous result characteristics, thereby improving the accuracy of the estimated probability distribution characteristics, and further improving the later image processing efficiency of the previous result characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the implementation methods of some embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0018] Figure 1 A schematic diagram showing the structure of an image coding framework in the second coding mode in some embodiments of the present application is shown;

[0019] Figure 2 A schematic diagram showing the structure of an image coding framework in a first coding mode in some embodiments of the present application is shown;

[0020] Figure 3 A schematic diagram showing the structure of an image decoding framework in a second image decoding mode in some embodiments of the present application is shown;

[0021] Figure 4 A schematic diagram showing the structure of an image decoding framework in a first image decoding mode in some embodiments of the present application is shown;

[0022] Figure 5 One of the flowcharts of the image encoding method in some embodiments of the present application is shown;

[0023] Figure 6 One of the flow charts of steps of a method for determining a super-a priori scale decoder in an image encoding method in some embodiments of the present application is shown;

[0024] Figure 7 One of the structural schematic diagrams of the super-a priori scale decoder in the first coding mode in some embodiments of the present application is shown;

[0025] Figure 8 Shows the use of Figure 7 The schematic diagram of the process of the super-prior scale decoder performing probability distribution prediction on the decoded super-prior features;

[0026] Fig. 9 The second structural schematic diagram of the super a priori scale decoder in the first coding mode in some embodiments of the present application is shown;

[0027] Fig.10 Shows the use of Fig. 9 The schematic diagram of the process of the super-prior scale decoder performing probability distribution prediction on the decoded super-prior features;

[0028] Fig.11 A second flowchart of the method for determining a super-a priori scale decoder in the image encoding method in some embodiments of the present application is shown;

[0029] Fig.12 A schematic diagram showing the structure of a super-a priori scale decoder in the second coding mode in some embodiments of the present application is shown;

[0030] Fig.13 Shows the use of Fig.12 The schematic diagram of the process of the super-prior scale decoder performing probability distribution prediction on the decoded super-prior features;

[0031] Fig.14 A flowchart showing the steps of the image decoding method in some embodiments of the present application is shown;

[0032] Fig.15 A flowchart showing the steps of an image information prediction method in some embodiments of the present application is shown. DETAILED DESCRIPTION

[0033] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0034] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0035] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0036] References to "some implementations", "some embodiments", etc. in the specification indicate that the described implementations or embodiments may include specific features, structures, or characteristics, but not every embodiment may include the specific features, structures, or characteristics. In addition, such phrases do not necessarily refer to the same implementation. In addition, when a specific feature, structure, or characteristic is described in connection with an embodiment, it is considered to be within the knowledge of those skilled in the art to implement such feature, structure, or characteristic in connection with other implementations (whether or not explicitly described herein).

[0037] The image information prediction method provided in the present application can be applied to the image processing scenario of performing coherent preliminary image processing and post-processing on the target image, and the preliminary image processing has at least two different processing modes to generate preliminary result features with different feature distribution characteristics, and the post-processing image processing relies on the predicted probability distribution of the preliminary result features.

[0038] Exemplarily, the image information prediction method can be applied to the process of entropy encoding / decoding residual features in the image encoding / decoding scenario. In the image encoding / decoding scenario, the "early image processing mode" corresponds to the "image encoding / decoding mode", the "first processing mode" and the "second processing mode" correspond to the "first encoding / decoding mode" and the "second encoding / decoding mode" respectively, the "early result features" correspond to the "residual features", the "late image processing" corresponds to the "entropy encoding / decoding", the "results obtained from the late image processing" correspond to the "encoded data" or "reconstructed target image", and the "probability prediction model" corresponds to the "super-prior scale decoder". Let's first combine Figures 1 to 14 The image encoding and decoding process of entropy coding is explained in detail.

[0039] The image encoding and decoding scenario of the embodiment of the present application relates to the field of image encoding and decoding technology, and in particular to intelligent image encoding and decoding technology based on deep learning. The international standards organization JPEG is formulating the JPEG AI standard, in which the intelligent image coding framework includes main parts such as analysis transform, intra prediction, hyper encoder, entropy encoding / decoding, hyper decoder, hyper scale decoder and synthesis transform. There are two main image encoding / decoding modes composed of the above-mentioned intelligent image coding framework, one is the base mode (base point), which is referred to as the first encoding / decoding mode in the embodiment of the present application, and the other is the high complexity mode (highpoint), which is referred to as the second encoding / decoding mode in the embodiment of the present application. Regardless of the image coding mode, the target image to be encoded will be subjected to intra-frame prediction processing inside the image to weaken the spatial redundancy caused by the similarity between adjacent pixels inside the image, and obtain the residual features corresponding to the target image.

[0040] Different image coding modes use different intra-frame prediction models, and the feature distribution of the residual features obtained will also be different. For example, the intra-frame prediction model used in the first coding mode is a simple convolution structure model, which performs convolution processing of intra-frame prediction on the overall spatial resolution corresponding to the target image, that is, the residual features are obtained after the super-prior features corresponding to the target image are directly convoluted, and the residual features are evenly distributed in the overall spatial range. The intra-frame prediction model used in the second coding mode is based on a multi-stage context modeling (MCM) network, and its internal operation process can be: first decode the encoded super-prior information, and down-shuffle, that is, reduce the spatial resolution by 4 times (2 times the length and width), and increase the number of channels by 4 times. At the same time, the initial input features obtained after analyzing the changes in the target image are sliced ​​and down-shuffled. In the prediction process, the first slice is first predicted, and then the second slice is predicted in combination with the reconstructed input features of the first slice. In the prediction of each slice, the first group of input features of the first slice are predicted using the super prior features of the first group. After the predicted input features (referred to as predicted input features) are generated, they are subtracted from the initial input features of the first group of the first slice to obtain the predicted residual features. The predicted residual features are entropy encoded and decoded and added back to the predicted input features to obtain the reconstructed input features of the first group of the first slice. Then, the reconstructed input features and the second group of super prior features of the first slice are used to predict the second group of features of the first slice, and the predicted input features of other groups are obtained by analogy. After the reconstruction of the four groups of features of the first slice is completed, they are enhanced by a channel enhancement network (Channel Net), and then the second slice is predicted. The prediction process of the second slice feature is basically the same as the prediction of the first slice feature, except that the enhanced reconstruction features of the group corresponding to the first slice are added as the input features of the prediction process. The residual features of each group of slices obtained above are combined and then reorganized by upsampling to obtain the overall residual features. It can be seen from the intra-frame prediction process that the second coding mode performs intra-frame prediction processing on the local spatial resolution obtained by downsampling the overall spatial resolution corresponding to the target image, and combines and upsamples the residual features obtained in each local area to obtain the residual features corresponding to the overall spatial resolution. The residual features obtained are spatially uniformly distributed in the grouped sub-image mode, but are unevenly distributed in the overall spatial range obtained by upsampling and reorganization.

[0041] In addition, no matter which image coding mode is used, after obtaining the residual feature, the residual feature can be entropy coded. In the process of entropy coding, the variance of the residual feature must first be predicted and modeled using a probability prediction model (i.e., the super prior scale decoder in the above image coding framework), that is, a probability model for entropy coding is established. For example, when a Gaussian distribution is used, the mean is 0, and the variance is the output of the super prior scale decoder. Afterwards, the residual feature is entropy coded using the obtained probability distribution characteristics of entropy coding.

[0042] On the basis of the above, if the probability distribution of the residual features is predicted based on the same super-prior features through the same probability prediction model (i.e. the above-mentioned super-prior scale decoder), the same probability distribution features will be obtained, which have poor adaptability to the residual features obtained under different image coding modes, resulting in low efficiency in the subsequent entropy encoding of the residual features based on the probability distribution features.

[0043] In response to the above problems, an embodiment of the present application proposes to design a model structure of an adaptive probability prediction model (i.e., a super-prior scale decoder in the above-mentioned image coding framework) based on the residual coding mode in the image coding mode or the generation method of the residual features, so as to improve the fit between the predicted probability distribution features and the residual features by adopting a super-prior scale decoder that is more matched with the generation method of the residual features, thereby improving the efficiency of entropy encoding the residual features using the probability distribution features.

[0044] The following first describes an image encoding and decoding framework for executing the image encoding method and image decoding method provided in the embodiments of the present application.

[0045] Reference Figure 1 As shown, the image coding framework under the second coding mode for executing the image coding method provided by the embodiment of the present application at least includes: an analysis transformation module 101, a super prior network 102, a quantization and entropy encoder 103, an entropy decoder and inverse quantization module 104, a super prior decoder 105, an intra-frame prediction model in the second coding mode (based on a multi-stage context modeling network) 106, a residual calculation module 107, an upsampling reorganization 108, a quantization module 109, a probability prediction model (super prior scale decoder) 110 and an entropy encoder 111.

[0046] In the image coding process, the image coding framework first analyzes and transforms the target image through the analysis and transformation module 101 to obtain the initial input feature (Input feature) corresponding to the target image. The initial input feature can be input into the hyper-prior network 102 to extract the hyper-prior information (hyperprior) of the initial input feature, and the extracted feature is entropy-encoded by the quantization and entropy encoder 103 to obtain the encoded hyper-prior information, and then the entropy decoder and dequantization module 104 are used to decode and dequantize to obtain the reconstructed feature. The reconstructed feature is then decoded / feature-transformed by the hyper-prior decoder 105 to obtain the decoded hyper-prior feature. The decoded hyper-prior feature can be input into the intra-frame prediction model (MCM network) 106 in the second coding mode together with the initial input feature, so as to be processed according to the internal grouping loop processing process of the MCM network described above to obtain the grouped predicted input feature. The predicted input feature can be calculated by subtracting the residual from the initial input feature of the corresponding group by the residual calculation module 107 to obtain the residual feature of the corresponding group. The residual features of these slice groups can be combined and processed by upsampling and reorganization 108 to obtain the residual features corresponding to the target image. The residual features are processed by the quantization module 109 to obtain the quantized residual features. At the same time, the reconstructed features output by the entropy decoder and the dequantization module 104 can be input into the probability prediction model (super prior scale decoder) 110 for probability prediction to obtain the probability distribution features. Then, the quantized residual features are entropy encoded by the entropy encoder 111 using the probability distribution features to obtain the encoded data corresponding to the target image.

[0047] Reference Figure 2 As shown, the image coding framework under the first coding mode for executing the image coding method provided by the embodiment of the present application at least includes: an analysis transformation module 201, a super prior network 202, a quantization and entropy encoder 203, an entropy encoder and dequantization module 204, a super prior decoder 205, an intra-frame prediction model (a model of a convolutional structure) 206 in the first coding mode, a residual calculation module 207, a quantization module 208, a probability prediction model (super prior scale decoder) 209 and an entropy encoder 210.

[0048] In the image coding process, the image coding framework first analyzes and transforms the target image through the analysis and transformation module 201 to obtain the initial input features corresponding to the target image. The initial input features can be input into the hyper-prior network 202 to extract the hyper-prior information of the initial input features, and the extracted features are quantized and entropy encoded by the quantization and entropy encoder 203 to obtain the encoded hyper-prior information, and then the entropy decoder and dequantization module 204 are used to decode and dequantize to obtain the reconstructed features. The reconstructed features are then decoded / feature transformed by the hyper-prior decoder 205 to obtain the decoded hyper-prior features. The decoded hyper-prior features can be convolved by the intra-frame prediction model (convolution structure model) 206 in the first coding mode to obtain the predicted input features corresponding to the target image. The predicted input features can be calculated by subtracting the residual from the initial input features by the residual calculation module 207 to obtain the residual features corresponding to the target image. The residual features are processed by the quantization module 208 to obtain the quantized residual features. At the same time, the reconstructed features output by the entropy decoder and dequantization module 204 can be input into the probability prediction model (super prior scale decoder) 209 for probability prediction to obtain probability distribution features. Then, the residual features are entropy encoded using the probability distribution features through the entropy encoder 210 to obtain the encoded data corresponding to the target image.

[0049] It should be noted that Figure 1 and Figure 2 The image coding framework shown mainly displays functional modules related to the image coding method in the embodiment of the present application, which may also include functional modules not shown, such as a spatial filling module and a cropping module.

[0050] Reference Figure 3 As shown, the image decoding framework under the second decoding mode for executing the image decoding method provided by the embodiment of the present application includes: an entropy decoder and dequantization module 301, a super priori decoder 302, a probability prediction model (super priori scale decoder) 303, an entropy decoder and dequantization module 304, an intra-frame prediction model in the second encoding mode (based on a multi-stage context modeling network) 305, an input feature reconstruction module 306 and an upsampling reorganization module 307.

[0051] In the image decoding process, the image decoding framework first uses the entropy decoder and dequantization module 301 to decode and dequantize the encoded super-prior information obtained after quantization and entropy coding to obtain the reconstructed features. The reconstructed features are decoded / feature transformed by the super-prior decoder 302 to obtain the decoded super-prior features. At the same time, the reconstructed features output by the entropy decoder and dequantization module 301 can be input into the probability prediction model (super-prior scale decoder) 303 for probability prediction processing to obtain the prediction result of the probability distribution of the residual features in the encoded data, that is, the probability distribution features. Then, the probability distribution features and the encoded data of the target image are input into the entropy decoder and dequantization module 304 for decoding and dequantization processing to obtain the residual features corresponding to the target image. Then, the decoded super-prior features output by the super-prior decoder 302 and the residual features are jointly input into the intra-frame prediction model (based on the multi-stage context modeling network) 305 in the second coding mode for slice grouping prediction of the input features to obtain the predicted input features of each group. Afterwards, the residual features of each group are combined with the predicted input features by the input feature reconstruction module 306 to obtain the reconstructed input features of each group. Finally, the reconstructed input features of each group are processed by the upsampling and reorganization module 307 to obtain the reconstructed target image.

[0052] Reference Figure 4 As shown, the image decoding framework in the first decoding mode for executing the image decoding method provided by the embodiment of the present application includes: an entropy decoder and dequantization module 401, a super priori decoder 402, a probability prediction model (super priori scale decoder) 403, an entropy decoder and dequantization module 404, an intra-frame prediction model (convolutional structure model) 405 in the first encoding mode, and an input feature reconstruction module 406.

[0053] In the image decoding process, the image decoding framework first uses the entropy decoder and dequantization module 401 to decode and dequantize the encoded super-prior information obtained after quantization and entropy coding to obtain reconstruction features. The reconstruction features are decoded / feature transformed by the super-prior decoder 402 to obtain decoded super-prior features. At the same time, the reconstruction features output by the entropy decoder and dequantization module 301 can be input into the probability prediction model (super-prior scale decoder) 403 for probability prediction processing to obtain the prediction result of the probability distribution of the residual features in the encoded data, that is, the probability distribution features. Then, the probability distribution features and the encoded data of the target image are input into the entropy decoder and dequantization module 404 for decoding and dequantization processing to obtain the residual features corresponding to the target image. Then, the decoded super-prior features output by the super-prior decoder 302 are input into the intra-frame prediction model (convolution structure model) 405 in the first coding mode to predict the input features and obtain the predicted input features corresponding to the target image. After that, the residual features and predicted input features obtained above are combined by the input feature reconstruction module 406 to obtain the reconstructed target image.

[0054] It should be noted that Figure 3 and Figure 4 The image decoding framework shown mainly displays functional modules related to the image decoding method in the embodiment of the present application, which may also include functional modules not shown, such as a spatial filling module and a cropping module.

[0055] Above Figure 1 to Figure 4 The probability prediction model (super-prior scale decoder) in the embodiment of the present application is designed according to the generation method of residual features in the second encoding mode (upsampling and reconstruction) and the generation method of residual features in the first encoding mode (convolution), which is adapted to the feature distribution of residual features generated in the corresponding image encoding mode.

[0056] The present application embodiment provides an image encoding method, referring to Figure 5 As shown, the image encoding method includes the following steps S51 to S55:

[0057] S51, obtaining initial input features and super prior features corresponding to the target image.

[0058] The target image is the image to be encoded and decoded. The initial input feature is the image feature obtained after the target image is analyzed and transformed, and has not been subjected to subsequent encoding processing. The super prior feature is the image feature obtained by entropy decoding the super prior information. The super prior information is some auxiliary information obtained by extracting the super prior information from the initial input feature corresponding to the target image, such as a finer resolution image of the target image, or the prior knowledge of the symmetry of the left and right faces of a human face, etc.

[0059] According to the foregoing description, the probability distribution prediction and entropy coding in the embodiments of the present application are for processing the output results (i.e., residual features) of the intra-frame prediction in the image coding process. Therefore, the target image must first undergo intra-frame prediction processing. The target image can be processed according to the requirements of the intra-frame prediction process in different image coding modes for the input features, and the initial input features corresponding to the target image can be obtained, and then the initial input features are processed to obtain the decoded super-prior features. As for whether the initial input features and super-prior features are processed in slices and groups, it depends on the image coding mode adopted.

[0060] In some embodiments, if the image encoding mode adopted is the first encoding mode (i.e., the basepoint mode), then generating the initial input features in S51 can be implemented as follows: first, the target image is analyzed and transformed to obtain the transformed input features; then, according to the spatial size of the output features of the intra-frame prediction model of the convolution structure in the first encoding mode, the transformed input features are spatially padded or cropped to obtain the initial input features, so as to ensure that when they are used for residual feature calculation, they are consistent with the spatial size of the predicted input features output by the intra-frame prediction model of the convolution structure.

[0061] Correspondingly, the generation of super-prior features in S51 can be implemented as follows: first, the encoded super-prior information is decoded to obtain the decoded super-prior features; then, according to the spatial size requirements of the input features of the intra-frame prediction model of the convolution structure in the first coding mode, the decoded super-prior features are spatially padded or cropped to obtain decoded super-prior features that are consistent with the spatial size of the input features of the intra-frame prediction model of the convolution structure.

[0062] In some embodiments, if the image encoding mode adopted is the second encoding mode (i.e., high complexity mode (highpoint)), then generating the initial input features in S51 can be implemented as follows: firstly, analyzing and transforming the target image to obtain the transformed input features; if the transformed input features do not meet the feature size requirement of an integer multiple of 2, the transformed input features can be spatially padded; finally, the padded and transformed input features are downsampled and reorganized to obtain the grouped initial input features.

[0063] Correspondingly, the generation of super-prior features in S51 can be implemented as follows: first, the encoded super-prior information is decoded to obtain the decoded super-prior features; if the feature size of the decoded super-prior features is inconsistent with the feature size of the transformed input features padded as mentioned above, the decoded super-prior features can be spatially padded or cropped to obtain decoded super-prior features with consistent feature sizes; finally, the decoded super-prior features after spatial size processing are downsampled and reorganized to obtain grouped super-prior features.

[0064] The spatial size filling parameter (such as 2) of the spatial padding in the above process can be determined according to the downsampling parameter preset in the downsampling reorganization module used for slicing grouping by the MCM network in the second coding mode, so as to ensure the consistency of the spatial size of the features flowing between the spatial padding, downsampling reorganization and MCM network. For example, if the downsampling is preset as a multiple of 2 in the downsampling reorganization, the spatial size filling parameter takes a value of 2; if the downsampling is set as a multiple of 4 in the downsampling reorganization, the spatial size filling parameter takes a value of 4. In addition, the slicing groupings are all positive integers greater than 1, and they can be determined according to the relevant preset parameter values ​​in the specific downsampling reorganization module. Exemplarily, the slicing parameter can take a value of 2, and the grouping parameter can take a value of 4.

[0065] S52: Generate residual features corresponding to the target image based on the initial input features and the super prior features according to the selected image coding mode.

[0066] According to the aforementioned Figure 1 and Figure 2 As shown in the figure, the selected image coding mode is different, the intra-frame prediction model used will be different, and the way of calculating the residual features using the initial input features and super prior features will also be different.

[0067] In some embodiments, if the selected image coding mode is the first coding mode, the intra-frame prediction model used is a convolutional structure model. In this way, the process of generating residual features can be implemented as follows: based on the super-prior features, the predicted input features corresponding to the target image are generated through the intra-frame prediction model of the convolutional structure; based on the initial input features and the predicted input features, the residual features are generated. That is, through the intra-frame prediction model of the convolutional structure, the super-prior features are directly convolved to obtain the predicted input features generated by predicting the target image. Then, the initial input features and the predicted input features are subtracted to obtain the residual features corresponding to the target image.

[0068] In other embodiments, if the selected image coding mode is the second coding mode, the intra-frame prediction model it adopts is based on a multi-stage context modeling network. In this way, the process of generating residual features can be implemented as follows: based on super-prior features and initial input features, a predicted input feature corresponding to the target image is generated through an intra-frame prediction model based on multi-stage context modeling; based on the initial input features and the predicted input features, a residual feature is generated. The internal processing process of the MCM network using super-prior features and initial input features for prediction can be found in the above description of the MCM network, which can output the predicted input features of the tile grouping, and can subtract it from the initial input features of the same tile grouping to obtain the residual features of the tile grouping. Combined with the above Figure 1According to the description of , the residual features of the sliced ​​groups can be combined, upsampled, reorganized, cropped, etc. to obtain the residual features corresponding to the target image. Combined with the description of the internal processing process of the above MCM network, the residual features obtained here can also be the reconstructed residual features obtained through the image reconstruction process.

[0069] S53. Based on the image coding mode, determine a probability prediction model for entropy coding of residual features, wherein the model structure of the probability prediction model is adapted to the feature distribution of the residual features.

[0070] According to the above analysis of the characteristic distribution characteristics of the residual features obtained in the first coding mode and the second coding mode, the distribution characteristics of the output residual features are different due to the different intra-frame prediction models in different image coding modes. Therefore, in order to make the predicted probability distribution characteristics more suitable for the residual features, the super-a priori scale decoder is designed according to the generation method of the residual features in the embodiment of the present application, that is, the model structure of the super-a priori scale decoder can be designed according to the model structure of the intra-frame prediction model that generates the residual features. For example, the generation method of the residual features in the first coding mode is direct generation by convolution, then the super-a priori scale decoder can be designed to directly output the probability distribution features by convolution to ensure that the output probability distribution features are also uniformly distributed within the overall spatial range. For another example, the generation method of the residual features in the second coding mode is to first downsample and reorganize for various processing and then upsample and reorganize, then the super-a priori scale decoder can also be designed to first downsample and then upsample and reorganize to output the probability distribution features to ensure that the probability distribution features are uniformly distributed in the sub-image mode and non-uniformly distributed within the overall spatial range. In this way, a corresponding super-prior scale decoder can be designed for each image coding mode, rather than using a unified super-prior scale decoder for all image coding modes. This makes the super-prior scale decoder more consistent with the generation method of residual features, thereby better modeling the probability distribution of residual features and improving the entropy coding efficiency of subsequent residual features.

[0071] Then, after the image coding mode is determined, the super a priori scale decoder used in the current image coding process can be determined according to the corresponding relationship between the image coding mode and the super a priori scale decoder.

[0072] In some embodiments, considering that the last processing layer for generating residual features in the intra-frame prediction model has a greater influence on the feature distribution of the residual features, when designing the super-a priori scale decoder, the structure of its last processing layer can be designed to be the same as the structure of the last processing layer of the intra-frame prediction model, without limiting the number of model processing layers and the structure of each processing layer of the super-a priori scale decoder before the last processing layer, so as to ensure that the final output method of the super-a priori scale decoder is the same as the generation method of the residual features.

[0073] S54. Based on the super prior features, the probability distribution characteristics of the residual features are predicted through a probability prediction model.

[0074] The super-prior features are input into the probability prediction model (i.e., the super-prior scale decoder) for processing, and the probability distribution features for probability estimation of the residual features are output.

[0075] In some embodiments, S54 can be implemented as follows: based on the super-prior feature, through a probability prediction model, predicting the variance feature corresponding to the residual feature; based on the variance feature and a preset probability distribution type, determining the probability distribution feature of the residual feature.

[0076] Among them, the preset probability distribution type is a pre-set probability distribution model, and its model form is related to the mean and variance, for example, it can be a Gaussian distribution model.

[0077] According to the above description, the super-prior scale decoder is used in the intelligent image coding framework to decode the variance of the residual feature to establish a probability distribution model for entropy coding. For example, when a Gaussian distribution is used, the mean is 0, and the variance is the output of the super-prior scale decoder. Then, after selecting an adaptive super-prior scale decoder according to the image coding mode, it can be used to process the super-prior feature and output the predicted variance feature. Combined with the mean in the preset probability distribution type, the probability distribution feature corresponding to the residual feature is estimated. The distribution characteristics of this probability distribution feature are more consistent with the distribution characteristics of the residual feature.

[0078] S55. Based on the probability distribution characteristics, entropy encoding is performed on the residual features to generate encoded data of the target image.

[0079] Based on the probability distribution characteristics as basic data, the residual characteristics are encoded according to the entropy coding method in the selected image coding mode to obtain the encoded data of the target image.

[0080] It is understandable that other processing, such as quantization, cropping, etc., can be performed on the residual features before / after entropy coding. At this time, the encoded data is the result of entropy coding and other processing.

[0081] The image coding method provided by the embodiment of the present application, when encoding a target image, first obtains the initial input features and super prior features corresponding to the target image, and generates the residual features corresponding to the target image to be entropy coded based on the initial input features and the super prior features according to the selected image coding mode; the feature distribution of the residual features depends on the selected image coding mode, so the probability prediction model of the feature distribution of the residual features can be designed and determined according to the image coding mode; then, based on the super prior features, the probability distribution features of the residual features are predicted through the probability prediction model; finally, the residual features are entropy coded using the probability distribution features to generate the coded data of the target image; in this way, different probability prediction models are designed for different image coding modes, the problem of poor adaptability between the probability prediction model in the entropy coding process and the residual features to be entropy coded is solved, so that the predicted probability distribution features can match the feature distribution of the residual features obtained by the corresponding image coding mode, the degree of fit between the probability prediction model and the feature distribution of the residual features is improved, thereby improving the accuracy of the estimated probability distribution features, and further improving the entropy coding efficiency of the residual features.

[0082] The embodiment of the present application also provides a flow chart of a method for determining a super-a priori scale decoder in an image coding method, which designs a model structure of a super-a priori scale decoder according to the structure of an intra-frame prediction model in a first coding mode, so as to improve the fit between the super-a priori scale decoder and the corresponding residual feature generation method in the first coding mode. Figure 6 As shown, the method for determining the super-a priori scale decoder in the image coding method includes the following steps S61-S62:

[0083] S61. If the image coding mode is the first coding mode, the last processing layer of the intra-frame prediction model is the first convolutional layer.

[0084] If the selected image coding mode is the first coding mode, then its intra-frame prediction model is a model with a simple convolution structure, and it can be determined that the last processing layer for generating residual features in the model with the convolution structure is a convolution layer, that is, the first convolution layer.

[0085] S62. Determine that the last processing layer of the probability prediction model is the second convolutional layer, and the convolution kernel size of the second convolutional layer is consistent with the convolution kernel size of the first convolutional layer.

[0086] According to the above description, the last processing layer of the super a priori scale decoder has the same structure as the last processing layer of the intra prediction model. Therefore, the last processing layer of the super a priori scale decoder can also be set as a convolution layer, that is, the second convolution layer. And the convolution kernel size of the second convolution layer is the same as the convolution kernel size of the first convolution layer. For example, if the convolution kernel size of the first convolution layer is 3*3, then the convolution kernel size of the second convolution layer is also set to 3*3. As for the convolution kernel parameters of the second convolution layer and the parameters of other processing layers in the super a priori scale decoder, they can be obtained through the model training process.

[0087] In some embodiments, in order to keep the model structure of the super-a priori scale decoder in the current JPEG AI standard as consistent as possible, the last processing layer mentioned above, that is, a second convolutional layer, may be added on the basis of the model structure of the super-a priori scale decoder in the JPEG AI standard.

[0088] Figure 7 FIG. 2 shows a schematic diagram of the structure of a super-a priori scale decoder in the first coding mode. Figure 7 As shown, the super-a priori scale decoder, which is closer to the model structure of the super-a priori scale decoder in the JPEG AI standard, includes: a first convolution combination of a convolution layer 701 including a 1*1 convolution kernel and an activation layer 702 (such as a nonlinear activation function ReLU), a second convolution combination of a convolution layer 703 including a 3*3 convolution kernel and an activation layer 704 (such as a nonlinear activation function ReLU), a third convolution combination of a convolution layer 705 including a 1*1 convolution kernel and an activation layer 706 (such as a nonlinear activation function ReLU), a first upsampling reorganization layer 707, a second convolution layer 708, and a cropping layer 709. The upsampling multiple of the first upsampling reorganization layer 707 is determined based on the spatial size difference between the initial input feature and the residual feature. For example, if the spatial size difference between the initial input feature and the residual feature is 4 times, the upsampling multiple is determined to be 4 times.

[0089] Taking the spatial size difference between the initial input feature [C,h,w] and the residual feature [C,4h,4w] as 4 times as an example, using Figure 7 The super-prior scale decoder shown estimates the probability distribution of the decoded super-prior features [C, h, w]. Figure 8As shown, the super-prior feature [C, h, w] with C channels and h rows and w columns is input into the super-prior scale decoder, and is processed by the convolution layer 801 with a 1*1 convolution kernel, and a convolution result with unchanged size is output; the convolution result is processed by the activation layer 802 and the convolution layer 803 with a 3*3 convolution kernel in turn, and a convolution result with unchanged size is output again; the convolution result is then processed by the activation layer 804 and the convolution layer 805 with a 1*1 convolution kernel and an output channel number that is 16 times the number of input channels, and the output channel number becomes a convolution result [16C, h, w] that is 16 times the number of input channels; the convolution result [16C, h ,w] is then processed by the activation layer 806 and the first upsampling reorganization layer 807 (such as PixelShuffle (4)) with an upsampling multiple of 4, and the 16-fold number of channels is sequentially combined into a spatial resolution 4*4 upsampling to obtain the upsampling result [C,4h,4w]; the upsampling result [C,4h,4w] is processed by the second convolution layer 808 to output a convolution result [C,4h,4w] with unchanged size; finally, the spatial resolution of the convolution result [C,4h,4w] is cut by the cropping layer 809 to obtain a probability distribution feature consistent with the spatial resolution of the residual feature.

[0090] In the above process, the number of output channels of the convolution layer 805 is set to facilitate the subsequent first upsampling and reorganization layer 807 to perform 4 times upsampling processing to obtain probability distribution features consistent with the spatial resolution of the residual features.

[0091] In some embodiments, in order to make the super a priori scale decoder corresponding to the first coding mode and the super a priori scale decoder corresponding to the second coding mode designed in the embodiments of the present application have better structural consistency, the super a priori scale decoder corresponding to the second coding mode can be designed according to the method of generating residual features in the MCM network in the second coding mode, and then a second convolution layer is added to the end of the model structure of the super a priori scale decoder corresponding to the second coding mode to form a super a priori scale decoder corresponding to the first coding mode. The design concept of the super a priori scale decoder corresponding to the second coding mode can be seen in the description of the subsequent embodiments.

[0092] Fig. 9 FIG. 4 shows a schematic diagram of the structure of another super-a priori scale decoder in the first coding mode. Fig. 9As shown, the super-a priori scale decoder corresponding to the first coding mode having better structural consistency with the super-a priori scale decoder corresponding to the second coding mode includes: a fourth convolution combination of a convolution layer 901 including a 3*3 convolution kernel and an activation layer 902 (such as a non-linear activation function ReLU), a fifth convolution combination of a convolution layer 903 including a 1*1 convolution kernel and an activation layer 904 (such as a non-linear activation function ReLU), a second upsampling reorganization layer 905, a sixth convolution combination of a convolution layer 906 including a 3*3 convolution kernel and an activation layer 907 (such as a non-linear activation function ReLU), a seventh convolution combination of a convolution layer 908 including a 1*1 convolution kernel and an activation layer 909 (such as a non-linear activation function ReLU), a third upsampling reorganization layer 910, a second convolution layer 911 and a cropping layer 912. The upsampling multiples of the second upsampling reorganization layer 905 and the third upsampling reorganization layer 910 are determined based on the spatial size difference between the initial input feature and the residual feature. For example, if the spatial size difference between the initial input feature and the residual feature is 4 times, and the above model structure includes two upsampling recombination layers, it can be determined that the upsampling multiples of the second upsampling recombination layer 905 and the third upsampling recombination layer 910 are both 2 times; for another example, if the spatial size difference between the initial input feature and the residual feature is 8 times, and the above model structure includes two upsampling recombination layers, it can be determined that the upsampling multiples of the second upsampling recombination layer 905 and the third upsampling recombination layer 910 are both 4 times, or the upsampling multiple of the second upsampling recombination layer 905 is 2 times, and the upsampling multiples of the third upsampling recombination layer 910 are both 6 times, etc.

[0093] Taking the spatial size difference between the initial input feature [C,h,w] and the residual feature [C,4h,4w] as 4 times as an example, using Fig. 9 The super-prior scale decoder shown estimates the probability distribution of the decoded super-prior features [C, h, w]. Fig.10As shown, the super-prior feature [C, h, w] with C channels and h rows and w columns is input into the super-prior scale decoder, and is processed by the convolution layer 1001 with a 3*3 convolution kernel, and a convolution result with unchanged size is output; the convolution result is successively processed by the activation layer 1002 and the convolution layer 1003 with a 1*1 convolution kernel and an output channel number that is 4 times the number of input channels, and the output channel number becomes a convolution result [4C, h, w] that is 4 times the number of input channels; the convolution result [4C, h, w] is then processed by the activation layer 1004 and the second upsampling reorganization layer 1005 (such as PixelShuffle (2)) with an upsampling factor of 2, and the 4-fold number of channels is sequentially combined into a spatial resolution 2*2 upsampling to obtain an upsampling result [C, 2h, 2w]; the upsampling result [C, 2h, 2w] is processed by the convolution layer 1006 with a 3*3 convolution kernel, and the convolution result [C, 2h, 2w] with unchanged size is output; the convolution result [C, 2h, 2w] is The convolution result [C, 2h, 2w] is processed by the activation layer 1007 and the convolution layer 1008 with a 1*1 convolution kernel and an output channel number that is 4 times the number of input channels, and the output channel number becomes the convolution result [4C, 2h, 2w] with a convolution number that is 4 times the number of input channels. The convolution result [4C, 2h, 2w] is then processed by the activation layer 1009 and the third upsampling reorganization layer 1010 (such as PixelShuffle (2)) with an upsampling multiple of 2, and the 4 The channels are combined in sequence to upsample the spatial resolution by 2*2 to obtain the upsampled result [C, 4h, 4w]; the upsampled result [C, 4h, 4w] is processed by the second convolution layer 1011 with a 3*3 convolution kernel to output the convolution result [C, 4h, 4w] with unchanged size; finally, the spatial resolution of the convolution result [C, 4h, 4w] is cut by the cropping layer 1012 to obtain the probability distribution feature consistent with the spatial resolution of the residual feature.

[0094] In the above process, the number of output channels of the convolutional layer 1003 and the convolutional layer 1008 is set to facilitate the subsequent 2x upsampling processing of the second upsampling and recombination layer 1005 and the third upsampling and recombination layer 1010 to obtain probability distribution characteristics consistent with the spatial resolution of the residual features.

[0095] The embodiment of the present application also provides a flow chart of a method for determining a super-a priori scale decoder in another image coding method, which designs the model structure of the super-a priori scale decoder according to the structure of the intra-frame prediction model in the second coding mode to improve the fit between the super-a priori scale decoder and the corresponding residual feature generation method in the second coding mode. Fig.11 As shown, the method for determining the super-a priori scale decoder in the image coding method includes the following steps S111-S112:

[0096] S111. If the image coding mode is the second coding mode, the last processing layer of the intra-frame prediction model is the fourth upsampling reorganization layer.

[0097] If the selected image coding mode is the second coding mode, then its intra-frame prediction model is the MCM network, and it can be determined that the last processing layer for generating residual features in the MCM network is the upsampling and reorganization layer, that is, the fourth upsampling and reorganization layer.

[0098] S112. Determine that the last processing layer of the probability prediction model is the fifth upsampling and reorganization layer, and the upsampling multiple of the fifth upsampling and reorganization layer is consistent with the upsampling multiple of the fourth upsampling and reorganization layer.

[0099] According to the above description, the last processing layer of the super a priori scale decoder has the same structure as the last processing layer of the intra-frame prediction model. Therefore, the last processing layer of the super a priori scale decoder can also be set to an upsampling reorganization layer, that is, the fifth upsampling reorganization layer. In addition, in order to further make the designed super a priori scale decoder more consistent with the characteristic distribution characteristics of the residual features, the upsampling multiple of the fifth upsampling reorganization layer can be set to the upsampling multiple of the fourth upsampling reorganization layer. For example, if the upsampling multiple of the fourth upsampling reorganization layer is 2, then the upsampling multiple of the fifth upsampling reorganization layer is also set to 2. As for the other various parameters in the super a priori scale decoder, they can be obtained through the model training process.

[0100] In some embodiments, Fig.12 FIG. 2 shows a schematic diagram of the structure of a super-a priori scale decoder in the second coding mode. Fig.12As shown, the super-a priori scale decoder corresponding to the second coding mode includes: an eighth convolution combination of a convolution layer 1201 including a 3*3 convolution kernel and an activation layer 1202 (such as a nonlinear activation function ReLU), a ninth convolution combination of a convolution layer 1203 including a 1*1 convolution kernel and an activation layer 1204 (such as a nonlinear activation function ReLU), a sixth upsampling reorganization layer 1205, a tenth convolution combination of a convolution layer 1206 including a 3*3 convolution kernel and an activation layer 1207 (such as a nonlinear activation function ReLU), a third convolution layer 1208 with a 1*1 convolution kernel, a fifth upsampling reorganization layer 1209 and a cropping layer 1210. The upsampling multiple of the sixth upsampling reorganization layer 1205 is determined based on the spatial size difference between the initial input feature and the residual feature, and the upsampling multiple of the fifth upsampling reorganization layer 1209. For example, if the spatial size difference between the initial input feature and the residual feature is 4 times, the MCM network determines that the upsampling multiple of the fifth upsampling recombinant layer 1209 is 2, then the upsampling multiple of the sixth upsampling recombinant layer 1205 can be determined to be 2 times; for another example, if the spatial size difference between the initial input feature and the residual feature is 6 times, the MCM network determines that the upsampling multiple of the fifth upsampling recombinant layer 1209 is 2, then the upsampling multiple of the sixth upsampling recombinant layer 1205 can be determined to be 4 times, and so on.

[0101] Taking the spatial size difference between the initial input feature [C,h,w] and the residual feature [C,4h,4w] as 4 times as an example, using Fig.12 The super-prior scale decoder shown estimates the probability distribution of the decoded super-prior features [C, h, w]. Fig.13As shown, the super-prior feature [C, h, w] with C channels, h rows and w columns is input into the super-prior scale decoder, and is processed by the convolution layer 1301 with a 3*3 convolution kernel, and a convolution result with unchanged size is output; the convolution result is sequentially processed by the activation layer 1302 and the convolution layer 1303 with a 1*1 convolution kernel and an output channel number that is 4 times the number of input channels, and the output channel number becomes a convolution result [4C, h, w] with a convolution number that is 4 times the number of input channels; the convolution result [4C, h, w] is then sequentially processed by the activation layer 1304 and the sixth upsampling reorganization layer 1305 (such as PixelShuffle (2)) with an upsampling multiple of 2, and the 4-fold number of channels is sequentially combined into a spatial resolution 2*2 upsampling, to obtain an upsampling result [C, 2h, 2w]; the upsampling result [C, 2h, 2w] is processed by the convolution layer 1304 with a 3*3 convolution kernel. The convolution layer 1306 processes and outputs a convolution result [C, 2h, 2w] with unchanged size. The convolution result [C, 2h, 2w] is processed in turn by the activation layer 1307 and the third convolution layer 1308 with a 1*1 convolution kernel and an output channel number that is 4 times the input channel number, and the output channel number becomes a convolution result [4C, 2h, 2w] with a convolution result number that is 4 times the input channel number. The convolution result [4C, 2h, 2w] is processed by the fifth upsampling reorganization layer 1309 (such as PixelShuffle (2)) with an upsampling multiple of 2, and the 4-fold number of channels is sequentially combined into a spatial resolution 2*2 upsampling to obtain an upsampling result [C, 4h, 4w]. Finally, the cropping layer 1310 is used to perform spatial resolution truncation on the upsampling result [C, 4h, 4w] to obtain a probability distribution feature that is consistent with the spatial resolution of the residual feature.

[0102] In the above process, the number of output channels of the convolutional layer 1303 and the convolutional layer 1308 is set to facilitate the subsequent 2x upsampling processing of the sixth upsampling and recombination layer 1305 and the fifth upsampling and recombination layer 1309 to obtain probability distribution characteristics consistent with the spatial resolution of the residual features.

[0103] It should be noted that Figure 7 to Figure 13 In the super-a priori size decoder shown, it is only necessary to ensure that the structure of the last processing layer before the cropping layer is the last processing layer for generating residual features in the intra-frame prediction model in the corresponding image coding mode. The number and structure of other processing layers before this processing layer may not be limited, as long as the settings of other processing layers are combined with the last processing layer to output probability distribution features consistent with the spatial resolution of the residual features.

[0104] The image coding method provided in each embodiment of the present application can be implemented by preparing a data set required for training a related neural network model, and training and testing the intelligent image coding framework including different super-a priori scale decoders in the above embodiments.

[0105] For example, the algorithm model of the intelligent image coding framework of this application is deployed on the PyTorch platform, and the model training and testing are completed on the NVIDIA GeForce RTX v100 GPU, so that the network can be trained end-to-end. The above-produced dataset is used as the input of the entire network, and the batch size is set to 8; the training strategy is the same as the JPEGAI standard; and the test is performed on the test set of the JPEG AI standard. Compared with the results of the JPEG AI standard, with 4-point and 5-point BDRate as the measurement standards respectively, the image coding method of the present application has improvements in multiple indicators such as the average value (AVG), multi-scale structural similarity (Multi Scale Structural Similarity Index, ms-ssim Torch), visual information fidelity (Visual Information Fidelity, vif), normalized Laplacian pyramid quality evaluation (Normalized Laplacian Pyramid, nlpd), evaluation of information content-weighted structural similarity (Evaluation of Information Content-Weighted SSIM, iw-ssim), and video quality multi-method fusion evaluation (Video Multi-method Assessment Fusion, vmaf), as shown in Tables 1 to 4 below.

[0106] Table 1 Comparison of results on multiple indicators using 4-point BDRate as the measurement standard

[0107]

[0108] Table 2 Comparison of results on multiple indicators using 5-point BDRate as the measurement standard

[0109]

[0110]

[0111] Table 3 Comparison of results on multiple indicators using 4-point BDRate as the measurement standard

[0112]

[0113] Table 4 Comparison of results on multiple indicators using 5-point BDRate as the measurement standard

[0114]

[0115] In some embodiments, the present application also provides an image decoding method. The explanation of the same or similar terms or steps in the image decoding method as those in the image encoding method provided in the above embodiments can be found in the description of the above embodiments, and will not be repeated here. Fig.14 As shown, the image decoding method includes the following steps S141 to S146:

[0116] S141, obtaining the encoding data and super-prior features of the target image.

[0117] The encoding data of the target image and the entropy-encoded super-prior information are obtained from the image encoding end. The super-prior information is entropy-decoded to obtain the decoded super-prior features.

[0118] S142. Based on the selected image decoding mode, determine a probability prediction model for entropy decoding of residual features in the encoded data, wherein the model structure of the probability prediction model is adapted to the feature distribution of the residual features in the encoded data.

[0119] In some embodiments, the structure of the last processing layer of the probability prediction model is the same as the structure of the last processing layer of the intra-frame prediction model for generating residual features in the image coding mode.

[0120] In some embodiments, S142 can be implemented as follows: if the image decoding mode is the first decoding mode corresponding to the first encoding mode, the last processing layer of the intra-frame prediction model in the first decoding mode is the fourth convolution layer; determine that the last processing layer of the probability prediction model is the fifth convolution layer, and the convolution kernel size of the fifth convolution layer is consistent with the convolution kernel size of the fourth convolution layer.

[0121] In one example, the model structure of the super-a priori scale decoder corresponding to the first decoding mode is Figure 7 The model structure shown is consistent. That is, the super prior scale decoder includes a first convolution combination, a second convolution combination, a third convolution combination, a first upsampling reorganization layer and a second convolution layer (corresponding to the fifth convolution layer); wherein the upsampling multiple of the first upsampling reorganization layer is determined based on the spatial size difference between the initial input feature and the residual feature; the convolution combination includes a convolution layer and an activation layer.

[0122] In another example, the model structure of the super-a priori scale decoder corresponding to the first decoding mode is Fig. 9 The model structure shown is consistent. That is, the super prior scale decoder includes: a fourth convolution combination, a fifth convolution combination, a second upsampling reorganization layer, a sixth convolution combination, a seventh convolution combination, a third upsampling reorganization layer and a second convolution layer (corresponding to the fifth convolution layer); wherein the upsampling multiples of the second upsampling reorganization layer and the third upsampling reorganization layer are determined based on the spatial size difference between the initial input feature and the residual feature.

[0123] In other embodiments, S142 can be implemented as follows: if the image decoding mode is the second decoding mode corresponding to the second encoding mode, the last processing layer of the intra-frame prediction model in the second decoding mode is the seventh upsampling reorganization layer; it is determined that the last processing layer of the probability prediction model is the eighth upsampling reorganization layer, and the upsampling multiple of the eighth upsampling reorganization layer is consistent with the upsampling multiple of the seventh upsampling reorganization layer.

[0124] In one example, the model structure of the super-a priori scale decoder corresponding to the second decoding mode is Fig.12 The model structure shown is consistent. That is, the super prior scale decoder includes: the eighth convolution combination, the ninth convolution combination, the sixth upsampling reorganization layer, the tenth convolution combination, the third convolution layer and the fifth upsampling reorganization layer (corresponding to the eighth upsampling reorganization layer) to generate the probability distribution characteristics of the residual features; wherein the upsampling multiple of the sixth upsampling reorganization layer is determined based on the spatial size difference between the initial input features and the residual features, and the upsampling multiple of the fifth upsampling reorganization layer.

[0125] S143. Based on the super prior features, the probability distribution characteristics of the residual features are predicted through a probability prediction model.

[0126] In some embodiments, S143 may be implemented as follows: based on the super-prior feature, using a probability prediction model, predicting the variance feature corresponding to the residual feature; based on the variance feature and a preset probability distribution type, determining the probability distribution feature of the residual feature.

[0127] S144. Based on the probability distribution characteristics, entropy decoding is performed on the encoded data to generate residual features.

[0128] Based on the probability distribution characteristics as basic data, the encoded data is decoded according to the entropy coding method in the image decoding mode corresponding to the selected image coding mode to obtain the residual characteristics of the target image.

[0129] It is understandable that, in order to be consistent with the processing process at the encoding end, other processing such as inverse quantization, space filling, cropping, etc. can also be performed on the encoded data before / after entropy decoding.

[0130] S145 . According to the image decoding mode, intra-frame prediction is performed based on the super-prior features to generate prediction input features corresponding to the target image.

[0131] S146: Generate a reconstructed target image based on the residual features and the predicted input features.

[0132] The residual features and the predicted input features are added and image reconstruction is performed to obtain a target image reconstructed from the encoded data.

[0133] The image decoding method provided by the embodiment of the present application adopts a probability prediction model consistent with the above-mentioned encoding process to predict the probability distribution characteristics corresponding to the residual features to be decoded during the decoding of the target image, thereby solving the problem of poor adaptability between the probability prediction model in the entropy decoding process and the residual features to be entropy decoded, thereby improving the accuracy of the estimated probability distribution characteristics, and further improving the entropy decoding efficiency of the residual features.

[0134] In some embodiments, the present application also provides an image information prediction method. The explanation of the same or corresponding terms or steps in the image information prediction method and the image encoding method provided in the above embodiments can be found in the description of the above embodiments, and will not be repeated here. Fig.15 As shown, the image information prediction method includes the following S151 to S154:

[0135] S151, obtaining initial input features and super prior features corresponding to the target image.

[0136] S152. Generate preliminary result features corresponding to the target image based on the initial input features and the super prior features according to the selected preliminary image processing mode.

[0137] In some embodiments, the preliminary image processing mode includes at least a first processing mode and a second processing mode. The first processing mode includes performing preliminary image processing on the overall spatial resolution corresponding to the target image to obtain an image processing process of preliminary result features. The second processing mode includes performing preliminary image processing on the local spatial resolution obtained by downsampling the overall spatial resolution, and combining and upsampling the local processing results to obtain an image processing process of preliminary result features corresponding to the overall spatial resolution.

[0138] S153. Based on the previous image processing mode, determine a probability prediction model for the later image processing of the previous result features, wherein the model structure of the probability prediction model is adapted to the feature distribution of the previous result features.

[0139] In some embodiments, the structure of the last processing layer of the probability prediction model is the same as the structure of the last processing layer for generating the previous result features in the previous image processing mode.

[0140] In some embodiments, S153 can be implemented as follows: if the previous image processing mode is the first processing mode, the last processing layer for generating the previous result features in the previous image processing mode is the first convolution layer; the last processing layer of the probability prediction model is determined to be the second convolution layer, and the convolution kernel size of the second convolution layer is consistent with the convolution kernel size of the first convolution layer.

[0141] In one example, the model structure of the probability prediction model corresponding to the first processing mode is Figure 7 The model structure shown is consistent. That is, the probability prediction model includes the first convolution combination, the second convolution combination, the third convolution combination, the first upsampling reorganization layer and the second convolution layer (corresponding to the fifth convolution layer); wherein the upsampling multiple of the first upsampling reorganization layer is determined based on the spatial size difference between the initial input features and the previous result features; the convolution combination includes a convolution layer and an activation layer.

[0142] In another example, the model structure of the probability prediction model corresponding to the first processing mode is Fig. 9 The model structure shown is consistent. That is, the probability prediction model includes: the fourth convolution combination, the fifth convolution combination, the second upsampling reorganization layer, the sixth convolution combination, the seventh convolution combination, the third upsampling reorganization layer and the second convolution layer (corresponding to the fifth convolution layer); wherein the upsampling multiples of the second upsampling reorganization layer and the third upsampling reorganization layer are determined based on the spatial size difference between the initial input features and the previous result features.

[0143] In other embodiments, S153 can be implemented as follows: if the previous image processing mode is the second processing mode, the last processing layer for generating the previous result features in the previous image processing mode is the fourth upsampling and reconstruction layer; the last processing layer for determining the probability prediction model is the fifth upsampling and reconstruction layer, and the upsampling multiple of the fifth upsampling and reconstruction layer is consistent with the upsampling multiple of the fourth upsampling and reconstruction layer.

[0144] In one example, the model structure of the probability prediction model corresponding to the second processing mode is Fig.12 The model structure shown is consistent. That is, the probability prediction model includes: the eighth convolution combination, the ninth convolution combination, the sixth upsampling reorganization layer, the tenth convolution combination, the third convolution layer and the fifth upsampling reorganization layer (corresponding to the eighth upsampling reorganization layer) to generate the probability distribution characteristics of the residual features; wherein the upsampling multiple of the sixth upsampling reorganization layer is determined based on the spatial size difference between the initial input features and the previous result features, and the upsampling multiple of the fifth upsampling reorganization layer.

[0145] S154. Based on the super prior features, the probability distribution features of the previous result features are predicted through a probability prediction model; the probability distribution features are used for post-image processing of the previous result features.

[0146] In some embodiments, if the previous image processing mode is the first processing mode, S154 can be implemented as: inputting the super-prior features into the probability prediction model, and sequentially processing through the first convolution combination, the second convolution combination, the third convolution combination, the first upsampling recombination layer and the second convolution layer in the probability prediction model to generate the probability distribution features of the previous result features.

[0147] In other embodiments, if the previous image processing mode is the first processing mode, S154 can be implemented as: inputting the super-prior features into the probability prediction model, and sequentially processing through the fourth convolution combination, the fifth convolution combination, the second upsampling recombination layer, the sixth convolution combination, the seventh convolution combination, the third upsampling recombination layer and the second convolution layer in the probability prediction model to generate the probability distribution features of the previous result features.

[0148] In some other embodiments, if the previous image processing mode is the second processing mode, S154 can be implemented as: inputting the super priori features into the probability prediction model, and sequentially processing through the eighth convolution combination, the ninth convolution combination, the sixth upsampling recombination layer, the tenth convolution combination, the third convolution layer and the fifth upsampling recombination layer in the probability prediction model to generate the probability distribution features of the previous result features.

[0149] In some embodiments, S154 can be implemented as follows: based on the super-prior feature, through the probability prediction model, predict the variance feature corresponding to the previous result feature; based on the variance feature and the preset probability distribution type, determine the probability distribution feature of the previous result feature.

[0150] The image information prediction method provided in the embodiment of the present application first obtains the initial input features and super prior features corresponding to the target image when processing the target image, and generates the preliminary result features corresponding to the target image to be subjected to the post-processing image based on the initial input features and the super prior features according to the selected preliminary image processing mode; the feature distributions of the preliminary result features generated by different preliminary image processing modes are different, so a probability prediction model for the post-processing of the preliminary result features, whose model structure is adapted to the feature distribution of the preliminary result features, can be designed and determined according to the preliminary image processing mode; and then, based on the super prior features, the preliminary result features are predicted through the probability prediction model. The probability distribution characteristics of the image are used to perform post-image processing on the early result characteristics by using the probability distribution characteristics to obtain the late result characteristics of the target image; in this way, different probability prediction models are designed for different early image processing modes, which solves the problem of poor adaptability between the probability prediction model and the early result characteristics in the late image processing process, so that the predicted probability distribution characteristics can match the characteristic distribution of the early result characteristics obtained by the corresponding early image processing mode, improves the degree of fit between the probability prediction model and the characteristic distribution of the early result characteristics, thereby improving the accuracy of the estimated probability distribution characteristics, and then improving the efficiency of the late image processing of the early result characteristics.

[0151] In some embodiments, some embodiments of the present application provide an image information prediction device, the image information prediction device comprising:

[0152] a memory configured to store a computer program;

[0153] The processor is configured to, when calling the computer program, enable the image information prediction device to implement the image information prediction method of any of the above embodiments.

[0154] In some embodiments, some embodiments of the present application provide an image encoding device, the image encoding device comprising:

[0155] a memory configured to store a computer program;

[0156] The processor is configured to, when calling a computer program, cause the image encoding device to implement the image encoding method of any of the above embodiments.

[0157] In some embodiments, some embodiments of the present application provide an image decoding device, the image decoding device comprising:

[0158] a memory configured to store a computer program;

[0159] The processor is configured to, when calling a computer program, enable the image decoding apparatus to implement the image decoding method of any of the above embodiments.

[0160] In some embodiments, some embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the image information prediction method of any of the above-mentioned embodiments, or the image encoding method of any of the above-mentioned embodiments, or the image decoding method of any of the above-mentioned embodiments.

[0161] In some embodiments, some embodiments of the present application provide a computer program product, which, when executed on a computer, enables the computer to implement the image information prediction method of any of the above embodiments, or the image encoding method of any of the above embodiments, or the image decoding method of any of the above embodiments.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0163] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A method for predicting image information, characterized in that: include: Obtaining initial input features and hyper-prior features corresponding to the target image; According to the selected preliminary image processing mode, based on the initial input features and the super prior features, a preliminary result feature corresponding to the target image is generated; Based on the previous image processing mode, determining a probability prediction model for the later image processing of the previous result features; wherein the model structure of the probability prediction model is adapted to the feature distribution of the previous result features; Based on the super priori features, predict the probability distribution features of the previous result features through the probability prediction model; The probability distribution features are used to perform the post-processing of the image on the pre-processing result features.

2. The method according to claim 1, characterized in that The first processing mode in the preliminary image processing mode includes an image processing process of performing preliminary image processing on the overall spatial resolution corresponding to the target image to obtain the preliminary result features; The second processing mode in the preliminary image processing mode includes performing preliminary image processing on the local spatial resolution obtained by sampling at the overall spatial resolution, and combining and upsampling the local processing results to obtain an image processing process of the preliminary result features corresponding to the overall spatial resolution.

3. The method according to claim 1 or 2, characterized in that: The structure of the last processing layer of the probability prediction model is the same as the structure of the last processing layer for generating the previous result features in the previous image processing mode.

4. The method according to claim 3, characterized in that The determining of a probability prediction model for the post-processing of the image based on the pre-processing image processing mode for the pre-processing result features comprises: If the early image processing mode is the first processing mode, the last processing layer for generating the early result features in the early image processing mode is the first convolution layer; Determine that the last processing layer of the probability prediction model is the second convolution layer, and the convolution kernel size of the second convolution layer is consistent with the convolution kernel size of the first convolution layer.

5. The method according to claim 4, characterized in that The method of predicting the probability distribution characteristics of the previous result characteristics based on the super prior characteristics through the probability prediction model includes: The super priori feature is input into the probability prediction model, and is processed in sequence by the first convolution combination, the second convolution combination, the third convolution combination, the first upsampling reorganization layer and the second convolution layer in the probability prediction model to generate the probability distribution feature of the previous result feature; wherein the upsampling multiple of the first upsampling reorganization layer is determined based on the spatial size difference between the initial input feature and the previous result feature; the convolution combination includes a convolution layer and an activation layer.

6. The method according to claim 4, characterized in that The method of predicting the probability distribution characteristics of the previous result characteristics based on the super prior characteristics through the probability prediction model includes: The super priori feature is input into the probability prediction model, and is processed in sequence by the fourth convolution combination, the fifth convolution combination, the second upsampling reorganization layer, the sixth convolution combination, the seventh convolution combination, the third upsampling reorganization layer and the second convolution layer in the probability prediction model to generate the probability distribution feature of the previous result feature; wherein the upsampling multiples of the second upsampling reorganization layer and the third upsampling reorganization layer are determined based on the spatial size difference between the initial input feature and the previous result feature.

7. The method according to claim 3, characterized in that The determining of a probability prediction model for the post-processing of the image based on the pre-processing image processing mode for the pre-processing result features comprises: If the previous image processing mode is the second processing mode, the last processing layer for generating the previous result features in the previous image processing mode is the fourth upsampling recombinant layer; It is determined that the last processing layer of the probability prediction model is a fifth upsampling and reorganization layer, and the upsampling multiple of the fifth upsampling and reorganization layer is consistent with the upsampling multiple of the fourth upsampling and reorganization layer.

8. The method according to claim 7, characterized in that The method of predicting the probability distribution characteristics of the previous result characteristics based on the super prior characteristics through the probability prediction model includes: The super priori feature is input into the probability prediction model, and is processed in sequence by the eighth convolution combination, the ninth convolution combination, the sixth upsampling reorganization layer, the tenth convolution combination, the third convolution layer and the fifth upsampling reorganization layer in the probability prediction model to generate the probability distribution feature of the previous result feature; wherein the upsampling multiple of the sixth upsampling reorganization layer is determined based on the spatial size difference between the initial input feature and the previous result feature, and the upsampling multiple of the fifth upsampling reorganization layer.

9. The method according to claim 1, characterized in that: The method of predicting the probability distribution characteristics of the previous result characteristics based on the super prior characteristics through the probability prediction model includes: Based on the super priori feature, predict the variance feature corresponding to the previous result feature through the probability prediction model; Based on the variance feature and a preset probability distribution type, the probability distribution feature of the previous result feature is determined.

10. An image information prediction device, characterized in that: include: a memory configured to store a computer program; The processor is configured to, when calling a computer program, enable the image information prediction device to implement the image information prediction method according to any one of claims 1 to 9.

Images