Image decoding method, image encoding method and device

Abstract

The embodiment of the invention provides an image decoding method and device, and an image coding method and device. The image coding method comprises the following steps: respectively generating M groups of initial input features and hyper-priori features based on a target image and hyper-priori information; for any current group in the groups: generating a position offset and a confidence mask between the reconstructed group and the current group by using an offset generation network based on the hyper-priori features of the current group and the reconstructed input features of the reconstructed group; aligning the reconstructed input features of the reconstructed group to the current group based on the reconstructed input features to generate aligned reconstructed input features; carrying out prediction according to the super-priori features of the current group, and generating prediction input features of the current group; generating a prediction residual error of the current group based on the prediction input feature and the initial input feature of the current group; and generating coded data of the target image based on the prediction residual of each group. In this way, the problem of spatial dislocation prediction between the current group and the adjacent group during intra-frame prediction coding is solved.

Description

Technical Field

[0001] Some embodiments of the present application relate to the field of image coding and decoding technology, and more specifically, to an image decoding method, an image coding method and a device. Background Art

[0002] With the widespread popularity of smart electronic products, the transmission of images / videos (composed of multiple consecutive frames of images) has long become an indispensable part of people's lives. The uploading and downloading of images / videos in social media, live streaming, virtual reality (VR) games, etc. all rely on image / video transmission and processing technologies, and image coding technology is the basis of many of the above-mentioned image / video transmission and processing technologies.

[0003] There is a strong correlation between adjacent pixels in an image. For example, the brightness and chrominance values ​​of two adjacent pixels are often close, which means that there is a lot of redundant information in the spatial domain of the image. In order to compress the redundant information of the image in the spatial domain, an image coding method based on intra-frame prediction is proposed in the related art. The image coding method based on intra-frame prediction is mainly divided into two parts: spatial group division and mode judgment. The main principle is to use different prediction modes and use the boundary pixels of the adjacent reconstructed group as reference pixels to predict the pixels of the current group. However, in the process of implementing the present application, the applicant found that in the process of predicting the pixels of the current group using the boundary pixels of the adjacent reconstructed group, there is a problem of spatial misalignment prediction between the current group and the adjacent group, resulting in low efficiency of intra-frame prediction. Summary of the invention

[0004] The exemplary embodiments of the present application provide an image decoding method, an image encoding method and an apparatus for solving the problem of spatial misaligned prediction between a current group and an adjacent group in an intra-frame prediction process.

[0005] Some embodiments of the present application provide the following technical solutions:

[0006] In a first aspect, some embodiments of the present application provide an image encoding method, including:

[0007] Based on the target image and the super prior information, at least M groups of initial input features and M groups of super prior features are generated respectively;

[0008] For any current group in each of the groups, execute:

[0009] Based on the super-prior features of the current group and the reconstructed input features of at least one reconstructed group, using an offset generation network, generating a position offset and a confidence mask between the reconstructed group and the current group;

[0010] Based on the position offset and the confidence mask, aligning the reconstructed input features of the reconstructed group to the current group to generate aligned reconstructed input features;

[0011] generating predicted input features of the current group based on the super-prior features of the current group and the aligned reconstructed input features;

[0012] generating a prediction residual of the current group based on the prediction input features and the initial input features of the current group;

[0013] Perform feature reconstruction based on the prediction residual and the prediction input feature to generate the reconstructed input feature of the current group; the reconstructed input feature of the current group is used for prediction processing of the next group;

[0014] Based on the prediction residuals of each of the groups, coded data of the target image is generated.

[0015] In a second aspect, some embodiments of the present application provide an image decoding method, including:

[0016] Acquire coded data of a target image, and generate at least M groups of reconstructed prediction residuals of the target image based on the coded data;

[0017] Generate M groups of super-prior features based on the super-prior information corresponding to the target image;

[0018] The following processing is performed for any current group in each of the groups:

[0019] Based on the super-prior features of the current group and the reconstructed input features of at least one reconstructed group, using an offset generation network, generating a position offset and a confidence mask between the reconstructed group and the current group;

[0020] Based on the position offset and the confidence mask, aligning the reconstructed input features of the reconstructed group to the current group to generate aligned reconstructed input features;

[0021] generating predicted input features of the current group based on the super-prior features of the current group and the aligned reconstructed input features;

[0022] generating a reconstruction input feature of the current group based on the prediction input feature and the reconstruction prediction residual of the current group;

[0023] A reconstructed target image is generated based on each of the groups of reconstructed input features.

[0024] In a third aspect, some embodiments of the present application provide an image encoding device, including:

[0025] a memory configured to store a computer program;

[0026] The processor is configured to, when calling a computer program, enable the image encoding device to implement the image encoding method described in the first aspect.

[0027] In a fourth aspect, some embodiments of the present application provide an image decoding device, including:

[0028] a memory configured to store a computer program;

[0029] The processor is configured to, when calling a computer program, enable the image decoding device to implement the image decoding method described in the second aspect.

[0030] In a fifth aspect, some embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the image encoding method described in the first aspect or the image decoding method described in the second aspect.

[0031] In a sixth aspect, some embodiments of the present application provide a computer program product, which, when executed on a computer, enables the computer to implement the image encoding method or the image decoding method described in the second aspect.

[0032] It can be seen from the above technical scheme that when encoding the target image, the image encoding method provided by the embodiment of the present application performs intra-frame prediction, so the target image and its corresponding super-prior information are spatially grouped to obtain M groups of initial input features and M groups of super-prior features; then for any group: because there is a spatial misalignment between the reconstructed input features of the reconstructed group and the super-prior features of the current group to be predicted, the offset generation network is used according to the super-prior features of the current group and the reconstructed input features of at least one reconstructed group to calculate the position offset and confidence mask between the reconstructed group and the current group, and the position offset and confidence mask are respectively Characterize the spatial position offset and pixel value difference information between the reconstructed group and the current group; then, align the reconstructed input features of the reconstructed group to the current group according to the position offset and the confidence mask obtained above, obtain the aligned reconstructed input features, and predict the input features of the current group according to the super-prior features of the current group and the aligned reconstructed input features, and obtain the predicted input features of the current group; then calculate the residual of the subtraction between the initial input features of the current group and the predicted input features, and obtain the predicted residual of the current group; finally, generate the encoding data of the target image by the predicted residual of each group. In this way, when implementing intra-frame prediction, the offset generation network is first used to calculate the spatial misalignment offset of the current group and the adjacent reconstructed group, and then the current group is predicted after adaptive shift processing, which solves the problem of spatial misalignment prediction between the current group and the adjacent group in intra-frame prediction coding, reduces the prediction residual of intra-frame prediction, thereby improving the prediction efficiency and compression efficiency of intra-frame prediction, and enhancing the image coding performance.

[0033] It can be seen from the above technical solutions that the image decoding method provided by the embodiment of the present application, in the process of decoding the target image, adopts the calculation of the position offset and confidence mask consistent with the above-mentioned encoding process when predicting the current group, as well as the alignment processing of the reconstructed input features of the reconstructed group, thereby solving the problem of spatial misaligned prediction between the current group and the adjacent group in intra-frame prediction decoding, and enhancing the image decoding performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the implementation methods of some embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0035] Figure 1 A schematic diagram showing the structure of an image coding framework in some embodiments of the present application is shown;

[0036] Figure 2A schematic diagram showing a process of performing image segmentation and grouping using a downsampling and reassembly module in some embodiments of the present application is shown;

[0037] Figure 3 A schematic diagram showing a process of performing image grouping using a downsampling and reassembly module in some embodiments of the present application is shown;

[0038] Figure 4 A schematic diagram showing the structure of an image decoding framework in some embodiments of the present application is shown;

[0039] Figure 5 One of the flowcharts of the image encoding method in some embodiments of the present application is shown;

[0040] Figure 6 A schematic diagram of the structure of a multi-stage context modeling network based on feature alignment for predicting the second group of the first slice in some embodiments of the present application is shown;

[0041] Figure 7 A schematic diagram of the structure of an offset generation network composed of a three-layer convolutional structure in some embodiments of the present application is shown;

[0042] Figure 8 A schematic diagram of the structure of an offset generation network composed of a U-shaped convolutional neural network in some embodiments of the present application is shown;

[0043] Fig. 9 A schematic diagram showing the structure of a fusion feature network in some embodiments of the present application is shown;

[0044] Fig.10 The second flowchart of the image encoding method in some embodiments of the present application is shown;

[0045] Fig.11 A schematic diagram of the structure of a multi-stage context modeling network based on feature alignment for predicting the first group of slices in some embodiments of the present application is shown;

[0046] Fig.12 A schematic diagram of the structure of a multi-stage context modeling network based on feature alignment for predicting the third group of the first slice in some embodiments of the present application is shown;

[0047] Fig.13 A schematic diagram of the structure of a multi-stage context modeling network based on feature alignment for predicting the fourth group of the first slice in some embodiments of the present application is shown;

[0048] Fig.14 A schematic diagram showing the structure of a channel enhancement network in some embodiments of the present application is shown;

[0049] Fig.15A schematic diagram of the structure of a multi-stage context modeling network based on feature alignment for predicting the first group of the second slice in some embodiments of the present application is shown;

[0050] Fig.16 A schematic diagram of the structure of a multi-stage context modeling network based on feature alignment for predicting the second group of the second slice in some embodiments of the present application is shown;

[0051] Fig.17 A schematic diagram of the structure of a multi-stage context modeling network based on feature alignment for predicting the third group of the second slice in some embodiments of the present application is shown;

[0052] Fig.18 A schematic diagram of the structure of a multi-stage context modeling network based on feature alignment for predicting the fourth group of the second slice in some embodiments of the present application is shown;

[0053] Fig.19 One of the flowcharts of the steps of the image decoding method in some embodiments of the present application is shown;

[0054] Fig. 20 The second flowchart of the steps of the image decoding method in other embodiments of the present application is shown. DETAILED DESCRIPTION

[0055] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0056] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0057] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0058] References to "some implementations", "some embodiments", etc. in the specification indicate that the described implementations or embodiments may include specific features, structures, or characteristics, but not every embodiment may include the specific features, structures, or characteristics. In addition, such phrases do not necessarily refer to the same implementation. In addition, when a specific feature, structure, or characteristic is described in connection with an embodiment, it is considered to be within the knowledge of those skilled in the art to implement such feature, structure, or characteristic in connection with other implementations (whether or not explicitly described herein).

[0059] The embodiments of the present application relate to the field of image coding and decoding technology, and in particular to intelligent image coding and decoding technology based on deep learning. The international standards organization JPEG is formulating the JPEG AI standard, in which the intelligent image coding framework includes main parts such as analysis transform, intra prediction, hyper encoder, entropy encoding / decoding, hyper decoder and synthesis transform. The intra prediction part here currently adopts a prediction method based on multi-stage context modeling (MCM).

[0060] The prediction method based on multi-stage context modeling first decodes the encoded super-prior information and performs down-shuffle, that is, the spatial resolution is reduced by 4 times (2 times in length and width) and the number of channels is increased by 4 times. At the same time, the input features obtained after the target image is analyzed and changed are sliced ​​and down-shuffled. In the prediction process, the first slice is predicted first, and then the second slice is predicted in combination with the reconstructed input features of the first slice. In the prediction of each slice, the first group of input features of the first slice are predicted using the super-prior information of the first group. After the predicted input features (referred to as predicted input features) are generated, they are subtracted from the input features of the first group of the first slice to obtain the predicted residual features, and the predicted residual features are entropy encoded and decoded and added back to the predicted input features to obtain the reconstructed input features of the first group of the first slice. Then, the reconstructed input features and the second group of super-prior information of the first slice are used to predict the second group of features of the first slice, and the predicted input features of other groups are obtained by analogy. After the reconstruction of the four groups of features of the first slice is completed, they are enhanced by a channel enhancement network (Channel Net), and then the second slice is predicted. The prediction process of the second piece of features is basically the same as the prediction of the first piece of features, except that the enhanced reconstruction features of the group where the corresponding position of the first piece is located (referred to as enhanced reconstruction features) are added as features input to the prediction process.

[0061] In the above process, the process of using the reconstructed input features of the first group to predict the features of the second group, and the subsequent process of using the reconstructed input features of the first group and the reconstructed input features of the second group together to predict the features of the third group, etc., all have the problem of misaligned prediction. That is, the first, second, third and fourth groups of features correspond to different spatial positions of the input features in the original spatial domain, and it is difficult to accurately generate the prediction information of the current group by directly using the information of different groups for prediction.

[0062] In response to the above problems, an embodiment of the present application proposes a misaligned prediction method based on deformable convolution for intra-frame prediction, so as to align the misaligned reconstructed input features through deformable convolution, and then predict the features after alignment, thereby solving the problem of spatial misaligned prediction between the current group and the adjacent group during intra-frame prediction, improving the efficiency of intra-frame prediction, and further improving the efficiency of intelligent image coding based on deep learning.

[0063] The following first describes an image encoding and decoding framework for executing the image encoding method and image decoding method provided in the embodiments of the present application.

[0064] Reference Figure 1 As shown, the image coding framework for executing the image coding method provided by the embodiment of the present application includes: an analysis transformation module 101, a spatial domain filling module 102, a downsampling reconstruction module 103, a downsampling reconstruction model 104, a super priori decoder 105, a multi-stage context modeling network 106, a residual calculation module 107, a quantization and entropy encoder 108, an inverse quantization and entropy decoder 109, an input feature reconstruction module 110 and an upsampling reconstruction and cropping module 111.

[0065] In the image coding process, the image coding framework first analyzes and transforms the target image through the analysis and transformation module 101 to obtain the input feature (Input feature) corresponding to the target image. The spatial size of the input feature is padded to an integer multiple of 2 through the spatial padding module 102 (if the spatial size of the input feature of the original input is an integer multiple of 2, the spatial padding operation is omitted). Here, the spatial size padding parameter (such as 2) of the spatial padding module 102 can be determined according to the preset downsampling parameter in the subsequent downsampling and reorganization module 103 / 104 to ensure the consistency of the spatial size of the features flowing between the spatial padding module 102, the downsampling and reorganization module 103 / 104 and the multi-stage context modeling network 106. For example, if the downsampling is preset to be a multiple of 2 in the downsampling and reorganization module 103 / 104, the spatial size padding parameter value is 2; if the downsampling is set to be a multiple of 4 in the downsampling and reorganization module 103 / 104, the spatial size padding parameter value is 4. Then, a down-sampling reorganization (down-shuffle) module 103 with slicing and grouping functions is used to expand the padded input features into N slices, and each slice is further reorganized into M parts to obtain M groups of initial input features (y). At the same time, a hyper-prior network is used to extract hyper-prior information (hyperprior) from the input features, and entropy coding is performed on it to obtain encoded hyper-prior information. The hyper-prior decoder 105 is then used to decode the entropy-coded hyper-prior information to obtain decoded hyper-prior features (also called reconstructed hyper-prior features), and the spatial size of the decoded hyper-prior features is padded to an integer multiple of 2 through the spatial domain filling module 102 (if the spatial size of the original input hyper-prior features is an integer multiple of 2, the spatial domain filling operation is omitted). Then, a down-sampling reorganization module 104 with a grouping function is used to reorganize the padded hyper-prior features into M parts (no slicing here) to obtain M groups of hyper-prior features. The initial input features (y) of each group and the super prior features of each group The input features are sequentially input into the multi-stage context modeling (MCM) network 106 for cyclic group prediction. After being processed by the multi-stage context modeling network 106, the predicted input features of each group can be obtained. The residual calculation module 107 is used to calculate the predicted input features of each group The residual is calculated by the initial input feature (y) to obtain the prediction residual (r). The prediction residual (r) is then quantized and processed by the entropy encoder 108 to obtain the encoded data of the target image with the same image size as the input feature. After that, the encoded data is dequantized and processed by the entropy decoder 109 to obtain the reconstructed prediction residual The reconstruction prediction residual and predict input features After being processed by the input feature reconstruction module 110 and the upsampling reorganization and cropping module 111, the reconstructed input features of each group can be obtained.

[0066] Wherein, the above N and M are both positive integers greater than 1, and the two values ​​can be determined according to relevant preset parameter values ​​in the specific downsampling and reorganization modules 103 and 104. For example, N can be 2, and M can be 4.

[0067] When N is 2 and M is 4, the schematic diagram of the process of using the downsampling and reorganization module 103 to slice and group the input features after the space domain filling can be seen in Figure 2 .like Figure 2 As shown in the figure, the original image a[2C,2h5,2w5] has 2C channels, each channel has 2h5 rows and 2w5 columns of pixels. After the slicing processing unit, the original image can be decomposed into the first slice a[0:C-1,2h5,2w5] and the second slice a[C:2C-1,2h5,2w5]. After the grouping processing unit, the original image can be transformed into an image feature of 8C channels with h5 rows and w5 columns of pixels for each channel. The schematic diagram of the process of using the above-mentioned downsampling and reassembly module 104 to perform image grouping (without fragmentation) can be seen in Figure 3 .like Figure 3 As shown in the figure, the original image a[C, 2h5, 2w5] has C channels, each channel has 2h5 rows and 2w5 columns of pixels. Without image segmentation, after the segmentation processing unit, the original image is still a[C, 2h5, 2w5]. After the grouping processing unit, the original image can be changed to an image feature map of 8C channels with h5 rows and w5 columns of pixels for each channel. The channel parameter C, the row pixel parameter h5, and the column pixel parameter w5 in this process can be set according to the actual image. For example, the value of C can be 128 in the JPEG AI standard.

[0068] The up-sampling and reorganization (up-shuffle) module is the reverse process of the down-sampling and reorganization (down-shuffle) module, which will not be described in detail here.

[0069] Reference Figure 4 As shown, the image decoding framework for executing the image decoding method provided by the embodiment of the present application includes: an inverse quantization and entropy decoder 401, a spatial domain filling module 402, a downsampling reconstruction module 403, a downsampling reconstruction module 404, a super priori decoder 405, a multi-stage context modeling network 406, an input feature reconstruction module 407 and an upsampling reconstruction and cropping module 408.

[0070] In the image decoding process, the image decoding framework first performs inverse quantization and entropy decoding on the coded data corresponding to the target image through the inverse quantization and entropy decoding encoder 401 to obtain the compression result of the reconstructed prediction residual, and then decompresses it in turn, uses the spatial domain filling module 402 to fill the spatial size, and uses the downsampling and reorganization module 403 with slicing and grouping functions to perform downsampling and reorganization to obtain N slices * M groups of reconstructed prediction residuals of the target image. At the same time, the super-prior information after entropy coding is decoded by the super-prior decoder 405 to obtain the decoded super-prior features, and the spatial size is filled by the spatial domain filling module 402 in turn, and the down-sampling and reorganization module 404 with grouping function is used to perform down-sampling and reorganization to obtain M groups of super-prior features. Then, the hyper-prior feature The input is sequentially input into the multi-stage context modeling network 406 for prediction, and the prediction input features of each group are obtained. Afterwards, the reconstructed prediction residuals of each group are input into the feature reconstruction module 407. and predict input features Combine to obtain the reconstruction input features of each group Finally, the upsampling, reorganization and cropping module 408 is used to reconstruct the input features of each group Perform corresponding processing and image reconstruction to obtain a reconstructed target image.

[0071] The present application embodiment provides an image encoding method, referring to Figure 5 As shown, the image encoding method includes the following steps S51 to S53:

[0072] S51. Based on the target image and the super-prior information corresponding to the target image, generate at least M groups of initial input features and M groups of super-prior features respectively.

[0073] The target image is the image to be encoded. The super prior information is some auxiliary information obtained by extracting super prior information from the input features corresponding to the target image. For example, it can be an image with a finer resolution of the target image, or it can be prior knowledge of the symmetry of the left and right faces of a human face. The initial input features are features obtained from the target image that have not been encoded by prediction and reconstruction and are to be input into the MCM network. The super prior features are features of the super prior information obtained from the super prior information and are to be input into the MCM network.

[0074] According to the requirements of the MCM network for input features, the target image and its corresponding super-prior information can be processed separately to obtain at least M groups of initial input features corresponding to the target image and M groups of super-prior features corresponding to the super-prior information. The processing method for grouping the target image and its corresponding super-prior information is not limited here, as long as the above-mentioned M groups of initial input features and super-prior information can be obtained.

[0075] In some embodiments, generating M groups of initial input features in S51 can be implemented as follows: firstly analyze and transform the target image to obtain the transformed input features; if the transformed input features do not meet the feature size requirement of an integer multiple of 2, the transformed input features can be spatially padded; finally, the padded and transformed input features are downsampled and grouped to obtain M groups of initial input features.

[0076] In some embodiments, generating M groups of super-prior features in S51 can be implemented as follows: first, the encoded super-prior information is decoded to obtain decoded super-prior features; if the decoded super-prior information is inconsistent with the feature size of the transformed input features padded as described above, the decoded super-prior information can be spatially padded or cropped to obtain decoded super-prior features with consistent feature sizes; finally, the decoded super-prior features after spatial size processing are downsampled and grouped to obtain M groups of super-prior features.

[0077] S52: Execute the following steps S521 to S524 for any current group in each group. The cyclic processing process of S521 to S524 corresponds to the internal processing process of the MCM network.

[0078] S521. Based on the super-prior features of the current group and the reconstructed input features of at least one reconstructed group, a position offset and a confidence mask between the reconstructed group and the current group are generated using an offset generation network.

[0079] Among them, the reconstructed group refers to the image group that has completed the reconstruction processing of the input features. The reconstructed input features are the input features reconstructed using the predicted input features and the predicted residuals. The offset generation network is a network model used to calculate the offset between two adjacent image groups. It can adopt any network structure and can obtain the effect of calculating the offset between the two groups through model training. The position offset offset is used to characterize the offset between two adjacent image groups in spatial position. The confidence mask mask is used to characterize the difference distribution of pixel values ​​between two adjacent image groups. In the offset calculation, the confidence mask can represent the credibility of the corresponding pixel value.

[0080] The MCM network can use the super-prior features of the current group to predict the input features of the current group. When the current group is not the first group, there is at least one reconstructed input feature of the reconstructed group. Then, in order to increase the accuracy of feature prediction, the super-prior features and at least one reconstructed input feature can be used simultaneously for feature prediction. However, the reconstructed group and the current group only have a spatial adjacent relationship, not a spatially completely corresponding relationship, and there is a problem of spatial misalignment prediction. Therefore, the offset generation network can be used first to process the super-prior features of the current group and the reconstructed input features of at least one reconstructed group to capture the position offset and confidence mask between the current group and the reconstructed group.

[0081] In some embodiments, S521 can be implemented as: fusing the super-prior features of the current group and the reconstructed input features of the reconstructed group to generate fused features; based on the fused features, using the offset generation network, generating a position offset and confidence mask between the reconstructed group and the current group.

[0082] For example, if the value of M is 4 and the current group is group 2, Figure 6 Figure 2 shows a schematic diagram of the structure of a multi-stage context modeling network based on feature alignment. Figure 6 First, the channel padding module 601 is used to fill the super-prior features of the second group Channel filling is performed to increase the number of channels from 2C to 5C / 2, and a super-prior feature of channel filling (p1(5C / 2)) is generated. At the same time, the reconstruction input feature of group 1 is merged into the feature fusion module 602. The fused feature (b1(3C)) is fused with the channel-filled hyper-prior feature (p1(5C / 2)) to obtain the fused feature (b1(3C)). After that, the fused feature (b1(3C)) is input into the offset generation network 603 (Offset Generation Network) for processing, and the number of output channels is 3C / 2, where the first C channel is the position offset (offset), and the last C / 2 channel is output as a confidence mask (mask) through the nonlinear activation function 604 (Sigmoid).

[0083] It should be noted that when the current group is the first group, since it has no corresponding reconstructed input features of the reconstructed group, only the first group's super-prior features are used for feature prediction. There is no problem of spatial misalignment prediction in this process, so the above position offset and confidence mask calculations do not need to be performed for the first group.

[0084] In some embodiments, the offset generation network can be implemented as a neural network model with a relatively simple network structure consisting of a three-layer convolution structure. Figure 7 Figure 2 shows a schematic diagram of the structure of the offset generation network consisting of a three-layer convolutional structure. Figure 7The convolution kernels of the three convolution layers are all 3*3. The number of channels of the fusion feature (b1) input to the first convolution layer 701 is 3C, and the number of output channels is 2C, and then it is output through the activation layer 702 (such as the non-linear activation function ReLU); the number of input channels of the second convolution layer 703 is 2C, and the number of output channels is C, and then it is output through the activation layer 704 (such as the non-linear activation function ReLU); the number of input channels of the third convolution layer 705 is C, and the number of output channels is 3C / 2, among which the first C channels are the position offset, and the latter C / 2 channels are output through the activation layer 706 (such as the non-linear activation function Sigmoid) as the confidence mask.

[0085] In some other embodiments, the offset generation network can be implemented as a U-shaped convolutional neural network model with a relatively classic network structure. The U-shaped convolutional neural network model is a convolutional neural network model in which the encoder and decoder in the model structure present a "U"-shaped architecture, which can also be referred to as a UNet network. Figure 8 Schematic diagram of the structure of the offset generation network composed of a U-shaped convolutional neural network. Figure 8 , a simple UNet network can be used to increase the receptive field in the offset generation process. The convolution kernels of each convolution layer in the UNet network are 3*3. The number of channels of the fusion feature b1 input to the first convolution layer 801 is 3C, and the number of output channels is C, and then it is output through the activation layer 802 (such as the nonlinear activation function ReLU); the number of input channels of the second convolution layer 803 is C, the number of output channels is C, and the convolution interval is 2 (stride = 2). After the feature processing, the spatial resolution is reduced to 1 / 2 of the original resolution, and then it is output through the activation layer 804 (such as the nonlinear activation function ReLU); the number of input channels of the third convolution layer 805 is C, the number of output channels is C, and then it is output through the activation layer 806 (such as the nonlinear activation function ReLU); the fourth is a deconvolution layer 807 (TransposedConv), and the number of convolution input channels is C , the number of output channels is C, the convolution interval is 2 (stride=2), and the spatial resolution after feature processing is increased to twice the input resolution (i.e., the original resolution of the entire module), and then it is output through the activation layer 808 (such as the nonlinear activation function ReLU); the input of the fifth convolution layer 809 is the sum of the output of the first convolution layer 801 and the output of the deconvolution layer 807, the number of input channels is C, the number of output channels is C, and then it is output through the activation layer 810 (such as the nonlinear activation function ReLU); the number of input channels of the sixth convolution layer 811 is C, and the number of output channels is 3C / 2, where the first C channels are the position offset, and the last C / 2 channels are output through the activation layer 810 (such as the nonlinear activation function Sigmoid) as the confidence mask.

[0086] Regardless of the network structure of the offset generation network used, after model training, the output position offset and confidence mask shape are the same.

[0087] S522: Based on the position offset and the confidence mask, align the reconstructed input features of the reconstructed group to the current group to generate aligned reconstructed input features.

[0088] The position offset and confidence mask obtained above can be used to align the reconstructed input features of the reconstructed group in terms of spatial position and pixel value differences, so as to minimize the spatial misalignment problem between the reconstructed group and the current group and obtain aligned reconstructed input features.

[0089] In some embodiments, the alignment process may be achieved by introducing more samples with complex deformations, using various data enhancements and tricks, and manually designing some manual features and algorithms.

[0090] In some other embodiments, the alignment process may be implemented by a deformable convolution layer, that is, based on the position offset and the confidence mask, the deformable convolution layer is used to perform alignment processing to generate aligned reconstructed input features. Figure 6 , the position offset (offset(C)) and confidence mask (mask(C / 2)) obtained above, as well as the reconstructed input features of the first group The deformable convolution (DConv) layer 605 is inputted together. The number of input channels of the deformable convolution layer 605 is C / 2, and the number of output channels is C / 2. Then, the output is outputted through the activation layer 606 (such as the nonlinear activation function ReLU). Then, the output is outputted through a convolution layer 607. The number of input channels of the convolution layer 607 is C / 2, and the number of output channels is C / 2, which is the aligned reconstructed input feature.

[0091] S523: Generate predicted input features of the current group based on the super-prior features of the current group and the aligned reconstructed input features.

[0092] The predicted input features are features obtained by predicting a group of input features.

[0093] Combining the super-prior features of the current group and the aligned reconstructed input features of the reconstructed group, the predicted input features of the current group are output after processing using the prediction fusion network. The network structure of the prediction fusion network here is not limited, and it can be trained to optimize the model parameters to achieve the function of feature prediction.

[0094] Continue to see Figure 6 , using the feature fusion module 608 to align the reconstructed input features The channel filling super prior feature (p1(5C / 2)) is fused, and the fusion result (a1(3C)) is input into the prediction fusion network 609. After being processed by the prediction fusion network 609, the prediction input feature of the second group is output

[0095] Fig. 9 A schematic diagram of the structure of a fusion feature network is shown. Fig. 9 The fusion feature network includes a convolution layer 901 with a convolution kernel of 1*1, an activation layer 902, a convolution layer 903 with a convolution kernel of 3*3, an activation layer 904, and a convolution layer 905 with a convolution kernel of 3*3. The above fusion result (a1(3C)) is processed by the convolution layer 901, the activation layer 902, the convolution layer 903, the activation layer 904, and the convolution layer 905 in sequence to output the corresponding prediction input feature.

[0096] S524: Generate a prediction residual of the current group based on the predicted input features and the initial input features of the current group.

[0097] Among them, the prediction residual is the difference distribution between the predicted input features and the true input features.

[0098] By subtracting the predicted input features of the current group from the initial input features, the prediction residual (r1) of the current group can be obtained.

[0099] S525 . Perform feature reconstruction based on the prediction residual and the prediction input features to generate the reconstructed input features of the current group.

[0100] In order to speed up image coding, the target image is divided into M groups for group prediction in the above process. The feature prediction of the subsequent group depends on the reconstructed input features of the reconstructed group. Therefore, after obtaining the prediction residual (r1) of the current group, the prediction residual can be compared with the prediction input features. Combine and complete feature reconstruction to obtain the reconstruction input features of the current group The reconstructed input features of the current group are used for prediction processing of the next group.

[0101] In some embodiments, in order to further improve the accuracy of reconstructing input features, thereby improving the accuracy of subsequent feature prediction, the above prediction residual can be further coded and decoded according to the image coding and decoding process, that is, the prediction residual (r1) of the current group is quantized, entropy encoded and entropy decoded to generate a reconstructed prediction residual (referred to as reconstructed prediction residual)

[0102] Based on these embodiments, S525 can be implemented as: performing feature reconstruction based on the reconstructed prediction residual and the predicted input features to generate the reconstructed input features of the current group. Figure 6 , combined with the reconstruction prediction residuals of group 1 and predict input features By performing image reconstruction, the reconstruction input features of the current group can be obtained.

[0103] S53. Generate coded data of the target image based on the prediction residuals of each group.

[0104] The prediction residuals obtained in the above process are grouped, so the prediction residuals of each group of the target image can be integrated into the entire image range to obtain the prediction residuals of the target image, and then quantization, entropy coding, compression and other coding processes are performed on it to obtain the coded data of the target image.

[0105] In some embodiments, the prediction residuals of each group may be encoded first and then integrated into the complete image range. In this way, S53 may be implemented as follows: for any group of prediction residuals (r i ) is quantized and entropy encoded to generate coding prediction residuals of corresponding groups, and then the coding prediction residuals of each group are integrated and compressed to generate coding data of the target image.

[0106] The image encoding method provided by the embodiment of the present application performs intra-frame prediction, so when encoding the target image, the target image and its corresponding super-prior information are spatially grouped to obtain M groups of initial input features and M groups of super-prior features; then for any group: because there is a spatial misalignment between the reconstructed input features of the reconstructed group and the super-prior features of the current group to be predicted, the offset generation network is used to calculate the position offset and confidence mask between the reconstructed group and the current group according to the super-prior features of the current group and the reconstructed input features of at least one reconstructed group, and the position offset and confidence mask respectively characterize the spatial position offset and pixel value difference between the reconstructed group and the current group; then, according to the position offset and confidence mask obtained above, the reconstructed input features of the reconstructed group are aligned to the current group to obtain aligned reconstructed input features, and the input features of the current group are predicted according to the super-prior features and the aligned reconstructed input features of the current group to obtain the predicted input features of the current group; then the residual of the subtraction between the initial input features of the current group and the predicted input features is calculated to obtain the predicted residual of the current group; finally, the predicted residuals of each group are used to generate the encoding data of the target image. In this way, when intra-frame prediction is implemented, the offset generation network is first used to calculate the spatial offset of the current group and the adjacent reconstructed group, and then the current group is predicted after adaptive shift processing. This solves the problem of spatial misaligned prediction between the current group and the adjacent group during intra-frame prediction coding, reduces the prediction residual of intra-frame prediction, thereby improving the prediction efficiency and compression efficiency of intra-frame prediction, and enhancing image coding performance.

[0107] The embodiment of the present application also provides another image encoding method, which divides the target image into N slices*M groups for processing to further improve the encoding efficiency. Fig.10 As shown, the image encoding method includes the following steps S1001 to S1018:

[0108] S101, analyzing and transforming the target image to generate transformed input features.

[0109] S102 , downsample and group the transformed input features to generate N pieces*M groups of initial input features.

[0110] The transformed input features can be processed through the spatial domain filling module and the downsampling reorganization module, and the transformed input features are first divided into N pieces, and each piece is further divided into M groups to obtain the initial input features of each piece and each group.

[0111] S103 . Execute the above-mentioned processes of S521 to S525 for any group in the first slice.

[0112] Assuming that N is 2 and M is 4, then for the feature prediction of the first group of the first slice, since there is no corresponding reconstructed group, only the hyper-prior features are used for prediction. Fig.11 , using the channel filling module 1101 to calculate the super prior features of the first group of the first slice Channel filling is performed to increase the number of channels from 2C to 3C, and the super prior features of channel filling (p0(3C)) are generated. Then, the prediction input features are generated through the prediction fusion network 1102 Then, the initial input features (y0) of the first group of the first slice are used together with the predicted input features Subtract them to obtain the corresponding prediction residual (r0). The prediction residual is quantized, entropy encoded and decoded to obtain the reconstructed prediction residual Afterwards, the prediction residuals are reconstructed and predict input features Add together to obtain the reconstructed input features of the first group of the first slice The reconstructed input features are used for the prediction process of the first slice and the second group and the prediction process of the second slice.

[0113] For feature prediction of the first slice and the second group, see Figure 6 and the description of S521 to S525 above.

[0114] For feature prediction of slice 1, group 3, see Fig.12 , first use the channel filling module 1201 to the super prior features of the third group Channel filling is performed to increase the number of channels from 2C to 5C / 2, and a super prior feature of channel filling (p2(5C / 2)) is generated. At the same time, the reconstruction input feature of group 1 is merged into the feature fusion module 1202. and the reconstructed input features of group 2 The fused features are obtained. The feature fusion module 1203 is then used to further fuse the fused features with the channel-filled hyper-prior features (p2(5C / 2)) to obtain the fused features (b2(7C / 2)). The fused features (b2(7C / 2)) are then input into the offset generation network 1204 for processing, and the number of output channels is 3C, of ​​which the first 2C channels are the position offset (offset(2C)), and the latter C channels are output as the confidence mask (mask(C)) through the activation layer 1205 (such as the nonlinear activation function Sigmoid). The position offset (offset(2C)) and confidence mask (mask(C)) obtained above, as well as the reconstructed input features of the first group can then be used as the basis for the reconstruction of the features. and the reconstructed input features of group 2 The fused features are input into the deformable convolution layer 1206, and are processed by the deformable convolution layer 1206, the activation layer 1207 and the convolution layer 1208 in sequence, and the aligned reconstructed input features are output. Afterwards, the aligned reconstructed input features are merged using feature fusion module 1209. The channel-filled hyper-priori feature (p2(5C / 2)) is fused, and the fusion result (a2(3C)) is processed by the prediction fusion network 1210 to output the prediction input feature of the third group. Using the initial input feature (y2) of the third group of the first slice and the predicted input feature Subtract the prediction residual (r2) of the corresponding group. The prediction residual is quantized, entropy encoded and decoded to obtain the reconstructed prediction residual Afterwards, the prediction residuals are reconstructed and predict input features Add together to obtain the reconstructed input features of the first slice and the third group The reconstructed input features are used for the prediction process of the 4th group of the 1st slice and the prediction process of the 2nd slice.

[0115] For feature prediction of slice 1, group 4, see Fig.13 , first use the channel filling module 1301 to the super prior features of the fourth group Channel filling is performed to increase the number of channels from 2C to 5C / 2, and a super-prior feature of channel filling (p3(5C / 2)) is generated. At the same time, the reconstructed input feature of group 1 is merged into the feature fusion module 1302. Reconstruction input features of group 2 and the reconstructed input features of group 2 The fused features are obtained. The feature fusion module 1303 is then used to further fuse the fused features with the channel-filled hyper-prior features (p3(5C / 2)) to obtain the fused features (b3(4C)). The fused features (b3(4C)) are then input into the offset generation network 1304 for processing, and the number of output channels is 9C / 2, of which the first 3C channels are the position offset (offset(3C)), and the last 3C / 2 channels are output as confidence masks (mask(3C / 2)) through the activation layer 1305 (such as the nonlinear activation function Sigmoid). The position offset and confidence mask obtained above, as well as the reconstructed input features of the first group can then be used as the basis for the reconstruction of the feature map. Reconstruction input features of group 2 and the reconstructed input features of group 2 The fused features are input into the deformable convolution layer 1306, and are processed by the deformable convolution layer 1306, the activation layer 1307 and the convolution layer 1308 in sequence to output the aligned reconstructed input features. Afterwards, the aligned reconstructed input features are merged using feature fusion module 1309. It is fused with the channel filling hyper-prior feature (p3(5C / 2)), and the fusion result (a3(3C)) is processed by the prediction fusion network to output the prediction input feature of the fourth group. Using the initial input feature (y3) of the 4th group of the 1st slice and the predicted input feature Subtract the prediction residual (r3) of the corresponding group. The prediction residual is quantized, entropy encoded and decoded to obtain the reconstructed prediction residual Afterwards, the prediction residuals are reconstructed and predict input features Add together to obtain the reconstructed input features of the 4th group of the 1st slice The reconstructed input features are used for the prediction process of the second slice.

[0116] S104 . For the first group of any current slice that is not the first slice, perform the following processing of S1041 to S1045 .

[0117] S1041 , based on the first group of super-prior features of the current slice and the first group of enhanced reconstruction features in the reconstructed slice, a prediction is performed using a prediction fusion network to generate a first group of prediction input features of the current slice.

[0118] The reconstructed slice is an image slice that has completed the reconstruction of the input features. The enhanced reconstruction features are the features after the features of each reconstructed input feature of the reconstructed slice are enhanced.

[0119] S1042: Generate a prediction residual of the first group of the current slice based on the prediction input features of the first group of the current slice and the initial input features of the first group of the current slice, and perform feature reconstruction based on the prediction residual and the prediction input features to generate a reconstructed input feature of the first group of the current slice.

[0120] For the feature prediction of the first group in the subsequent slice of the first slice, since there is no corresponding reconstructed group in the same slice, the hyper-prior feature can be used for prediction. At the same time, more image features can increase the knowledge that the network can learn, thereby improving the prediction efficiency, so the reconstruction input features of the same group in the reconstructed slice can be superimposed at the same time. Because different slices have certain feature differences, the reconstruction input features of the reconstructed slice can be enhanced through the channel enhancement network (Channel Net) to generate the enhanced reconstruction features of the corresponding group, so that the enhanced reconstruction features of the reconstructed slice are as close as possible to the features of the current slice.

[0121] Assume that N is 2 and M is 4, then we can use Fig.14 The channel enhancement network shown in the figure reconstructs the input features of each group in the first slice. The enhanced features of the corresponding group are generated by sequentially performing the processing of the upsampling and reorganization module 1401, the convolution layer 1402, the activation layer 1403, the convolution layer 1404, the activation layer 1405, the convolution layer 1406 and the downsampling and reorganization module 1407.

[0122] On the basis of the above, see Fig.15 , S1041-S1042 can be specifically implemented as follows: using the channel filling module 1501 to calculate the super-prior features of the first group of the second slice Channel filling is performed to increase the number of channels from 2C to 5C / 2, and the super-prior feature (p4(5C / 2)) of channel filling is generated. Then, the super-prior feature (p4(5C / 2)) and the enhanced reconstruction feature of the first group of the first slice are combined by the feature fusion module 1502. Then, the prediction fusion network 1503 is used to predict the fused feature (a4(3C)) to obtain the predicted input feature of the first group of the second slice. The subsequent prediction residuals and reconstruction prediction residuals and reconstruct the input features The process of obtaining can refer to the relevant descriptions of each example in the above S103.

[0123] S105. For any current group in any current slice other than the first slice, perform the following processing of S1051 to S1055.

[0124] S1051. Based on the super-prior features of the current group, the enhanced reconstruction features of the current group in at least one reconstructed slice corresponding to the current slice, and the reconstruction input features of at least one reconstructed group in the current slice, an offset generation network is used to generate a position offset and a confidence mask between the reconstructed group and the current group.

[0125] For subsequent groups in subsequent slices, the reconstructed input features of the reconstructed group can also be used for prediction, or the enhanced reconstruction features of the reconstructed group can be used to improve the information content and prediction efficiency. In this process, there is a spatial misalignment between the super-prior features of the current group and the reconstructed group in the current slice, so the position offset and confidence mask can be calculated.

[0126] In some embodiments, S1051 can be implemented as: splicing the super-prior features of the current group and the enhanced reconstruction features of the current group in at least one reconstructed slice corresponding to the current slice to generate a spliced ​​feature; fusing the spliced ​​feature and the reconstructed input features of the reconstructed group to generate a fused feature; based on the fused feature, using an offset generation network to generate a position offset and confidence mask between the reconstructed group and the current group.

[0127] S1052: Based on the position offset and the confidence mask, align the reconstructed input features of the reconstructed group to the current group to generate aligned reconstructed input features.

[0128] S1053 . Generate predicted input features of the current group based on the super-prior features of the current group, the enhanced reconstruction features of the current group in the reconstructed slices, and the aligned reconstruction input features.

[0129] Similarly, during prediction, the super-prior features of the current group can be combined with the enhanced reconstruction features of the current group in the reconstructed slice and the aligned reconstruction input features to increase the information content of the features used for feature prediction, thereby improving prediction accuracy and prediction efficiency.

[0130] S1054: Generate a prediction residual of the current group based on the predicted input features and the initial input features of the current group.

[0131] S1055: Perform feature reconstruction based on the prediction residual and the prediction input features to generate the reconstructed input features of the current group.

[0132] For explanations of the steps S1051 to S1055, please refer to the relevant descriptions of S521 to S525. Here, taking N as 2 and M as 4 as an example, the prediction process of the 2nd, 3rd and 4th groups of the 2nd slice is described.

[0133] See also Fig.16 The prediction process for the second group of the second slice is as follows: first, the super-prior features of the second group of the second slice are fused by the feature fusion module 1601. Enhanced reconstruction features with the first slice and the second group The splicing is then performed, and the splicing result is combined with the reconstructed input feature of the first group of the second slice through the feature fusion module 1602. The fused features (b5(3C)) are then processed by an offset generation network 1603, and the number of output channels is 3C / 2, where the first C channel is the position offset (offset(C)), and the second C / 2 channel is output through an activation layer 1604 (such as a nonlinear activation function Sigmoid) as a mask (mask(C / 2)). The position offset and confidence mask obtained above, as well as the reconstructed input features of the first group of the second slice, can be used as the The deformable convolution layer 1605 is inputted together, and the deformable convolution layer 1605, the activation layer 1606 and the convolution layer 1607 are processed in sequence to output the aligned reconstructed input features of the second piece and the first group. Afterwards, the aligned reconstructed input features are merged using feature fusion module 1608 And the super prior features of the second group of the second film Enhanced reconstruction features with the first slice and the second group The fusion features of the second slice and the second group are fused, and the fusion result (a5(3C)) is processed by the prediction fusion network 1609 to output the prediction input features of the second slice and the second group The subsequent prediction residuals and reconstruction prediction residuals and reconstruct the input features The process of obtaining may refer to the relevant description of each example in the above S103. The reconstructed input feature is used for prediction processing of the third group of the second slice.

[0134] See also Fig.17 The prediction process for the third group of the second slice is as follows: first, the super-prior features of the third group of the second slice are fused into Enhanced reconstruction features with the first slice and the third group The reconstruction input feature of the first group of the second slice is fused by the fusion feature module 1702 Reconstructed input features of group 2 of slice 2 The two fusion results are further fused through the fusion feature module 1703 to generate a fusion feature (b6(7C / 2)). Then, the fusion feature (b6(7C / 2)) is processed by an offset generation network 1704, and the number of output channels is 3C, of ​​which the first 2C channels are the position offset (offset(2C)), and the latter C channels are output through the activation layer 1705 (such as the nonlinear activation function Sigmoid) as the mask (mask(C)). The position offset and confidence mask obtained above, as well as the reconstructed input features of the first group of the second slice can be used as the output of the mask (mask(C)). Reconstructed input features of group 2 of slice 2 The fused features are input into the deformable convolution layer 1706, and are processed by the deformable convolution layer 1706, the activation layer 1707 and the convolution layer 1708 in sequence to output the aligned reconstructed input features of the second piece and the second group. Afterwards, the aligned reconstructed input features are merged using feature fusion module 1709. And the super prior features Enhanced reconstruction features with the first slice and the third group The fusion features of the second slice and the third group are fused, and the fusion result (a6(3C)) is processed by the prediction fusion network 1710 to output the prediction input feature of the second slice and the third group The subsequent prediction residuals and reconstruction prediction residuals and reconstruct the input features The process of obtaining may refer to the relevant description of each example in the above S103. The reconstructed input feature is used for prediction processing of the fourth group of the second slice.

[0135] See also Fig.18 The prediction process for the 4th group of the 2nd piece is as follows: first, the super prior features of the 4th group of the 2nd piece are fused by the feature fusion module 1801 Enhanced reconstruction features with the 1st slice, group 4 The reconstruction input feature of the first group of the second slice is fused by the fusion feature module 1802 Reconstructed input features of the second slice and the second group and the reconstructed input features of the 2nd slice and the 3rd group The two fusion results are further fused through the fusion feature module 1803 to generate a fusion feature (b7(4C)). Then, the fusion feature (b7(4C)) is processed by an offset generation network 1804, and the number of output channels is 9C / 2, of which the first 3C channels are the position offset (offset(3C)), and the latter 3C / 2 channels are output through the activation layer 1805 (such as the nonlinear activation function Sigmoid) as the mask (mask(3C / 2)). The position offset and confidence mask obtained above, as well as the reconstructed input features of the first group of the second slice can be used as the output of the mask (mask(3C / 2)). Reconstructed input features of the second slice and the second group and the reconstructed input features of the 2nd slice and the 3rd group The fused features are input into the deformable convolution layer 1806, and are processed by the deformable convolution layer 1806, the activation layer 1807 and the convolution layer 1808 in sequence to output the aligned reconstructed input features of the second piece and the third group. Afterwards, the aligned reconstructed input features are merged using feature fusion module 1809. And the super prior features of the 2nd piece and the 4th group Enhanced reconstruction features with the 1st slice, group 4 The fusion features of the second slice and the fourth group are fused, and the fusion result (a7(3C)) is processed by the prediction fusion network 1810 to output the prediction input feature of the second slice and the fourth group The subsequent prediction residuals and reconstruction prediction residuals and reconstruct the input features The process of obtaining can refer to the relevant descriptions of each example in the above S103.

[0136] S106. Generate coded data of the target image based on the prediction residuals of each group.

[0137] In some embodiments, S106 may be implemented as follows: quantizing and entropy coding the prediction residuals of any group in any slice to generate the coded prediction residuals of the corresponding group in the corresponding slice; and generating the coded data of the target image based on the coded prediction residuals of each group in each slice. In addition to quantizing, entropy coding, compressing, and other coding processes for the prediction residuals of each group in the same slice, the coded prediction residuals of each slice may also be integrated to generate the coded data of the target image.

[0138] It should be noted that the slice grouping processing in the above-mentioned image coding method can be selected and combined according to the actual situation. For example, only the features of the slice grouping are used to calculate the offset of the spatial dislocation, and the calculated offset is spliced ​​for each slice to obtain the offset features of the grouping without slices (i.e., position offset and confidence mask), and then the prediction residual and reconstruction input feature processing can be performed without slices. For another example, only the features of the grouping can be used to calculate the offset of the spatial dislocation, and the calculated offset can be sliced ​​to obtain the offset features of each slice and group, and then the prediction residual and reconstruction input feature processing can be performed in slices and groups.

[0139] The image coding method provided in each embodiment of the present application can be implemented by producing a data set required for training a related neural network model, and training and testing an intelligent image coding framework of a model of multi-stage context modeling including an offset generation network and a deformable convolutional layer in the above embodiments.

[0140] For example, the algorithm model of the intelligent image coding framework of this application is deployed on the PyTorch platform, and the model training and testing are completed on the NVIDIA GeForce RTX v100 GPU, so that the network can be trained end-to-end. The above-produced dataset is used as the input of the entire network, and the batch size is set to 8; the training strategy is the same as the JPEGAI standard; and the test is performed on the test set of the JPEG AI standard. Compared with the results of the JPEG AI standard, with 4-point and 5-point BDRate as the measurement standards respectively, the image coding method of the present application has improvements in multiple indicators such as the average value (AVG), multi-scale structural similarity (Multi Scale Structural Similarity Index, ms-ssim Torch), visual information fidelity (Visual Information Fidelity, vif), normalized Laplacian pyramid quality evaluation (Normalized Laplacian Pyramid, nlpd), evaluation of information content-weighted structural similarity (Evaluation of Information Content-Weighted SSIM, iw-ssim), and video quality multi-method fusion evaluation (Video Multi-method Assessment Fusion, vmaf), as shown in Tables 1 and 2 below.

[0141] Table 1 Comparison of results on multiple indicators using 4-point BDRate as the measurement standard

[0142]

[0143] Table 2 Comparison of results on multiple indicators using 4-point BDRate as the measurement standard

[0144]

[0145] In some embodiments, the present application also provides an image decoding method, referring to Fig.19 As shown, the image decoding method includes the following steps S191 to S194:

[0146] S191. Obtain encoding data of a target image, and generate at least M groups of reconstructed prediction residuals of the target image based on the encoding data.

[0147] In some embodiments, S191 can be implemented as: entropy decoding, inverse quantization and decompression of the encoded data of the target image in sequence to generate a reconstructed prediction residual of the target image; downsampling and grouping the reconstructed prediction residual of the target image to generate M groups of reconstructed prediction residuals.

[0148] S192. Generate M groups of super-prior features based on the super-prior information corresponding to the target image.

[0149] The super prior information here can be the entropy coded super prior information obtained by the image encoding end based on the input features of the target image. The acquisition process can be referred to above. Figure 1 Corresponding instructions.

[0150] In some embodiments, S192 can be implemented as: decoding the super-prior information to generate decoded super-prior features; downsampling and grouping the decoded super-prior features to generate M groups of super-prior features.

[0151] S193. For any current group in each group, the following processing of S1931 to S1934 is performed. Since the feature prediction process in the image decoding method is consistent with the feature prediction process in the image encoding method, the explanation of S1931 to S1934 can refer to the relevant description of each embodiment in the above-mentioned image encoding method. To avoid redundancy, it will not be repeated here. It can also be understood that the loop processing process of S1931 to S1934 is the internal processing process of the MCM model in the image decoding end.

[0152] S1931. Based on the super-prior features of the current group and the reconstructed input features of at least one reconstructed group, a position offset and a confidence mask between the reconstructed group and the current group are generated using an offset generation network.

[0153] In some embodiments, S1931 can be implemented as: fusing the super-prior features of the current group and the reconstructed input features of the reconstructed group to generate fused features; based on the fused features, using the offset generation network, generating the position offset and confidence mask between the reconstructed group and the current group.

[0154] It should be noted that when the current group is the first group, since it has no corresponding reconstructed input features of the reconstructed group, only the first group's super-prior features are used for feature prediction. There is no problem of spatial misalignment prediction in this process, so the above position offset and confidence mask calculations do not need to be performed for the first group.

[0155] In some embodiments, the process of using an offset generation network to process fused features to obtain position offsets and confidence masks can be implemented as follows: using a neural network model composed of a three-layer convolutional structure or a U-shaped convolutional neural network model to perform offset processing on the fused features to generate position offsets and confidence masks.

[0156] S1932. Based on the position offset and the confidence mask, align the reconstructed input features of the reconstructed group to the current group to generate aligned reconstructed input features.

[0157] In some embodiments, S1934 may be implemented as follows: using a deformable convolutional layer, based on a position offset and a confidence mask, aligning the reconstructed input features of the reconstructed group to generate aligned reconstructed input features.

[0158] S1933: Generate predicted input features of the current group based on the super-prior features of the current group and the aligned reconstructed input features.

[0159] S1934: Generate a reconstruction input feature of the current group based on the prediction input feature and the reconstruction prediction residual of the current group.

[0160] During decoding, the predicted input features and the corresponding reconstructed prediction residuals are added to obtain the reconstructed input features of the current group.

[0161] S194. Generate a reconstructed target image based on the reconstructed input features of each group.

[0162] The reconstructed input features of each group are converted from the feature domain back to the pixel domain and spliced ​​to obtain the reconstructed target image.

[0163] The image decoding method provided by the embodiment of the present application adopts the calculation of position offset and confidence mask consistent with the above-mentioned encoding process and alignment processing of reconstructed input features of the reconstructed group when predicting the current group during the process of decoding the target image, thereby solving the problem of spatial misaligned prediction between the current group and the adjacent group in intra-frame prediction decoding and enhancing the image decoding performance.

[0164] In some other embodiments, the present application provides another image decoding method, referring to Fig. 20 As shown, the image decoding method includes the following steps S201 to S206:

[0165] S201 , performing entropy decoding, inverse quantization and decompression on the coded data of the target image in sequence to generate a reconstructed prediction residual of the target image.

[0166] S202 , downsample and group the reconstructed prediction residuals of the target image to generate N slices*M groups of reconstructed prediction residuals.

[0167] The reconstructed prediction residuals of the target image can be processed by the spatial domain filling module and the downsampling and reorganization module, and the target image is first divided into N slices, and each slice is further divided into M groups to obtain the reconstructed prediction residuals of each slice and each group.

[0168] S203 . For any group in the first slice, execute the above-mentioned processing of S1931 to S1934 to generate the reconstructed input features of the corresponding group in the first slice.

[0169] S204 . For the first group of any current slice that is not the first slice, perform the following processing of S2041 to S2042 .

[0170] S2041 , based on the first group of super-prior features of the current slice and the first group of enhanced reconstruction features in the reconstructed slice, prediction is performed using a prediction fusion network to generate a first group of prediction input features of the current slice.

[0171] The explanation of S2041 can be found in the relevant description of S1041 in the above-mentioned image encoding method. To avoid redundancy, it will not be repeated here.

[0172] S2042 . Generate a first group of reconstructed input features of the current slice based on the first group of predicted input features of the current slice and the first group of reconstructed prediction residuals of the current slice.

[0173] S205. For any current group in any current slice other than the first slice, perform the following processing of S2051 to S2055.

[0174] S2051. Based on the super-prior features of the current group, the enhanced reconstruction features of the current group in at least one reconstructed slice corresponding to the current slice, and the reconstruction input features of at least one reconstructed group in the current slice, an offset generation network is used to generate a position offset and a confidence mask between the reconstructed group and the current group.

[0175] In some embodiments, S2051 can be implemented as: splicing the super-prior features of the current group and the enhanced reconstruction features of the current group in at least one reconstructed slice corresponding to the current slice to generate a spliced ​​feature; fusing the spliced ​​feature and the reconstructed input features of the reconstructed group to generate a fused feature; based on the fused feature, using an offset generation network to generate a position offset and confidence mask between the reconstructed group and the current group.

[0176] The explanation of S2051 can refer to the relevant description of S1051 in the above-mentioned image encoding method. To avoid redundancy, it will not be repeated here.

[0177] S2052: Based on the position offset and the confidence mask, align the reconstructed input features of the reconstructed group to the current group to generate aligned reconstructed input features.

[0178] The explanation of S2052 can be found in the relevant description of S1052 in the above-mentioned image encoding method. To avoid redundancy, it will not be repeated here.

[0179] S2053: Generate predicted input features of the current group based on the super-prior features of the current group, the enhanced reconstruction features of the current group in the reconstructed slices, and the aligned reconstruction input features.

[0180] The explanation of S2053 can be found in the relevant description of S1053 in the above-mentioned image encoding method. To avoid redundancy, it will not be repeated here.

[0181] S2054: Generate a reconstruction input feature of the current group based on the prediction input feature and the reconstruction prediction residual of the current group.

[0182] S206 : Generate a reconstructed target image based on the reconstructed input features of each group.

[0183] It should also be noted that the slice grouping processing in the above-mentioned image decoding method can be selected and combined according to the actual situation. For example, only the features of the slice grouping are used to calculate the offset of the spatial dislocation, and the calculated offset is spliced ​​for each slice to obtain the offset features of the grouping without slices (i.e., position offset and confidence mask), and then the prediction and reconstruction of the input features can be performed without slices. For another example, only the features of the grouping can be used to calculate the offset of the spatial dislocation, and the calculated offset is sliced ​​to obtain the offset features of each slice and group, and then the prediction and reconstruction of the input features can be performed in slices and groups.

[0184] In some embodiments, some embodiments of the present application provide an image encoding device, the image encoding device comprising:

[0185] a memory configured to store a computer program;

[0186] The processor is configured to, when calling a computer program, cause the image encoding device to implement the image encoding method of any of the above embodiments.

[0187] In some embodiments, some embodiments of the present application provide an image decoding device, the image decoding device comprising:

[0188] a memory configured to store a computer program;

[0189] The processor is configured to, when calling a computer program, enable the image decoding apparatus to implement the image decoding method of any of the above embodiments.

[0190] In some embodiments, some embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the image encoding method described in any of the above embodiments or the image decoding method described in any of the above embodiments.

[0191] In some embodiments, some embodiments of the present application provide a computer program product, which, when executed on a computer, enables the computer to implement the image encoding method described in any of the above embodiments or the image decoding method described in any of the above embodiments.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0193] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. An image coding method, characterized in that: include: Based on the target image and the super prior information, at least M groups of initial input features and M groups of super prior features are generated respectively; For any current group in each of the groups, execute: Based on the super-prior features of the current group and the reconstructed input features of at least one reconstructed group, using an offset generation network, generating a position offset and a confidence mask between the reconstructed group and the current group; Based on the position offset and the confidence mask, aligning the reconstructed input features of the reconstructed group to the current group to generate aligned reconstructed input features; generating predicted input features of the current group based on the super-prior features of the current group and the aligned reconstructed input features; generating a prediction residual of the current group based on the predicted input features and the initial input features of the current group; Perform feature reconstruction based on the prediction residual and the prediction input feature to generate the reconstructed input feature of the current group; The reconstructed input features of the current group are used for prediction of the next group; Based on the prediction residuals of each of the groups, coded data of the target image is generated.

2. The method according to claim 1, characterized in that Generating at least M groups of initial input features based on the target image, including: Analyzing and transforming the target image to generate transformed input features; The transformed input features are downsampled and grouped to generate N pieces*M groups of the initial input features.

3. The method according to claim 2, characterized in that For the current group in the current slice that is not the first slice, generating a position offset and a confidence mask between the reconstructed group and the current group using an offset generation network based on the super-prior features of the current group and the reconstructed input features of at least one reconstructed group, including: The position offset and the confidence mask between the reconstructed group and the current group are generated using the offset generation network based on the super-prior features of the current group, the enhanced reconstruction features of the current group in at least one reconstructed slice corresponding to the current slice, and the reconstruction input features of at least one reconstructed group in the current slice.

4. The method according to claim 2 or 3, characterized in that: The generating the predicted input features of the current group based on the super-prior features of the current group and the aligned reconstructed input features comprises: The predicted input features of the current group are generated based on the super-prior features of the current group, the enhanced reconstruction features of the current group in the reconstructed slices, and the aligned reconstructed input features.

5. The method according to claim 2 or 3, characterized in that: The step of generating the coded data of the target image based on the prediction residuals of each of the groups comprises: quantizing and entropy coding the prediction residuals of any of the groups in any of the slices to generate coded prediction residuals of the corresponding group in the corresponding slice; Based on the encoded prediction residuals of each of the groups in each of the slices, encoded data of the target image is generated.

6. The method according to claim 3, characterized in that: The generating the position offset and the confidence mask between the reconstructed group and the current group using the offset generation network based on the super-prior features of the current group, the enhanced reconstruction features of the current group in at least one reconstructed slice corresponding to the current slice, and the reconstruction input features of at least one reconstructed group in the current slice, comprises: Splicing the super-prior features of the current group and the enhanced reconstruction features of the current group in at least one reconstructed slice corresponding to the current slice to generate a spliced ​​feature; Fusing the splicing feature with the reconstructed input feature of the reconstructed group to generate a fused feature; Based on the fused features, the offset generation network is used to generate the position offset and the confidence mask between the reconstructed group and the current group.

7. The method according to claim 6, characterized in that The generating the position offset and the confidence mask between the reconstructed group and the current group based on the fusion feature and using the offset generation network comprises: The fused features are offset processed by using a neural network model consisting of a three-layer convolutional structure or a U-shaped convolutional neural network model to generate the position offset and the confidence mask.

8. The method according to claim 1, characterized in that The step of aligning the reconstructed input features of the reconstructed group to the current group based on the position offset and the confidence mask to generate aligned reconstructed input features includes: Using a deformable convolutional layer, based on the position offset and the confidence mask, the reconstructed input features of the reconstructed group are aligned to generate reconstructed input features aligned with the above.

9. The method according to claim 1, characterized in that: After generating the prediction residual of the current group based on the prediction input features and the initial input features of the current group, the method further includes: quantizing, entropy coding and entropy decoding the prediction residual to generate a reconstructed prediction residual; The performing feature reconstruction based on the prediction residual and the prediction input feature to generate the reconstructed input feature of the current group includes: Feature reconstruction is performed based on the reconstructed prediction residual and the predicted input features to generate the reconstructed input features of the current group.

10. An image decoding method, characterized in that: include: Acquire coded data of a target image, and generate at least M groups of reconstructed prediction residuals of the target image based on the coded data; Generate M groups of super-prior features based on the super-prior information corresponding to the target image; The following processing is performed for any current group in each of the groups: Based on the super-prior features of the current group and the reconstructed input features of at least one reconstructed group, using an offset generation network, generating a position offset and a confidence mask between the reconstructed group and the current group; Based on the position offset and the confidence mask, aligning the reconstructed input features of the reconstructed group to the current group to generate aligned reconstructed input features; generating predicted input features of the current group based on the super-prior features of the current group and the aligned reconstructed input features; generating a reconstruction input feature of the current group based on the prediction input feature and the reconstruction prediction residual of the current group; A reconstructed target image is generated based on each of the groups of reconstructed input features.

11. The method according to claim 10, characterized in that The generating at least M groups of reconstructed prediction residuals of the target image based on the coded data comprises: Performing entropy decoding, inverse quantization and decompression on the coded data of the target image in sequence to generate a reconstructed prediction residual of the target image; The reconstructed prediction residuals of the target image are downsampled and grouped to generate N slices*M groups of the reconstructed prediction residuals.

12. The method according to claim 11, characterized in that For the current group in the current slice that is not the first slice, generating a position offset and a confidence mask between the reconstructed group and the current group using an offset generation network based on the super-prior features of the current group and the reconstructed input features of at least one reconstructed group, including: The position offset and the confidence mask between the reconstructed group and the current group are generated using the offset generation network based on the super-prior features of the current group, the enhanced reconstruction features of the current group in at least one reconstructed slice corresponding to the current slice, and the reconstruction input features of at least one reconstructed group in the current slice.

13. The method according to claim 12, characterized in that The generating the position offset and the confidence mask between the reconstructed group and the current group using the offset generation network based on the super-prior features of the current group, the enhanced reconstruction features of the current group in at least one reconstructed slice corresponding to the current slice, and the reconstruction input features of at least one reconstructed group in the current slice, comprises: Splicing the super-prior features of the current group and the enhanced reconstruction features of the current group in at least one reconstructed slice corresponding to the current slice to generate a spliced ​​feature; Fusing the splicing feature with the reconstructed input feature of the reconstructed group to generate a fused feature; Based on the fused features, the offset generation network is used to generate the position offset and the confidence mask between the reconstructed group and the current group.

14. An image encoding device, characterized in that: include: a memory configured to store a computer program; The processor is configured to, when calling a computer program, enable the image encoding device to implement the image encoding method according to any one of claims 1 to 9.

15. An image decoding device, characterized in that: include: a memory configured to store a computer program; The processor is configured to, when calling a computer program, enable the image decoding device to implement the image decoding method according to any one of claims 10 to 13.

Images