Image compression method, image encoding method, image decoding method, and apparatus

Abstract

The application discloses an image compression method, an image coding method, an image decoding method and devices. The image compression method comprises the following steps: performing convolution processing on an image feature to be processed based on a multi-scale receptive field, so as to obtain a plurality of context features of the image feature to be processed; the image feature to be processed is determined based on an image feature of an image to be compressed; the plurality of context features are fused to obtain a sample feature of the image feature to be processed; and a compression result of the image to be compressed is obtained based on the sample feature of the image feature to be processed. The application can effectively utilize neighborhood information in a plurality of ranges, and can effectively eliminate coding / decoding redundancy.

Description

Technical Field

[0001] This application relates to the field of image encoding and decoding technology, and in particular to an end-to-end image compression method, image encoding method, image decoding method and apparatus. Background Technology

[0002] Image encoding and decoding methods may include a context processing step, which determines sample features of the image features to be processed based on contextual information of the image features to be processed. However, related technologies only utilize convolution with a receptive field of one size to extract contextual information of the samples to be processed, that is, the context model only utilizes neighborhood information within a certain range, without effectively eliminating encoding / decoding redundancy. Summary of the Invention

[0003] This application provides an image compression method, an image encoding method, an image decoding method, and an apparatus that can effectively utilize neighborhood information within a variety of ranges and effectively eliminate encoding / decoding redundancy.

[0004] To achieve the above objectives, this application provides an end-to-end image compression method, the method comprising:

[0005] Based on multi-scale receptive fields, convolution processing is performed on the features of the image to be processed to obtain multiple contextual features of the features of the image to be processed; the features of the image to be processed are determined based on the image features of the image to be compressed.

[0006] The various contextual features are fused to obtain sample features of the image to be processed;

[0007] Based on the sample features of the image to be processed, the compression result of the image to be compressed is obtained.

[0008] To achieve the above objectives, this application also provides an end-to-end image coding method, the method comprising:

[0009] Obtain the features of the image to be processed based on the image features of the image to be compressed;

[0010] The image compression method described above is used to process the features of the image to be processed, and the compression result of the image to be compressed is obtained.

[0011] The encoded bitstream of the image to be compressed is obtained based on the compression result.

[0012] To achieve the above objectives, this application also provides an end-to-end image decoding method, the method comprising:

[0013] The bitstream of the image to be compressed is decoded to obtain the image features of the image to be compressed;

[0014] The above image compression method is used to process the image features within the image features of the image to be compressed, and the compression result of the image to be compressed is obtained.

[0015] The decoded image of the bitstream is obtained based on the compression result.

[0016] To achieve the above objectives, this application also provides an encoder that includes a processor; the processor is configured to execute instructions to implement the steps of the above method.

[0017] To achieve the above objectives, this application also provides a decoder, which includes a processor; the processor is used to execute instructions to implement the steps of the above method.

[0018] To achieve the above objectives, this application also provides a computer-readable storage medium for storing instruction / program data that can be executed to implement the above methods.

[0019] In the image compression method of this application, based on multi-scale receptive fields, convolution processing is performed on the features of the image to be processed to obtain multiple contextual features of the features of the image to be processed, so that the multiple contextual features can be fused in the subsequent process to obtain sample features of the features of the image to be processed. Thus, the image compression method of this embodiment proposes a multi-scale receptive field approach. By combining receptive fields of different scales, the information of neighborhood points in different ranges can be effectively utilized, which can effectively eliminate encoding / decoding redundancy and thus improve image processing efficiency. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a schematic diagram of the structure of one embodiment of the image encoding / decoding network of this application;

[0022] Figure 2 This is a flowchart illustrating one embodiment of the image compression method of this application;

[0023] Figure 3 This is a schematic diagram of an embodiment of the image compression method of this application;

[0024] Figure 4 This is a schematic diagram of another embodiment of the image compression method of this application;

[0025] Figure 5 This is a schematic diagram of yet another embodiment of the image compression method of this application;

[0026] Figure 6This is a flowchart illustrating another embodiment of the image compression method of this application;

[0027] Figure 7 This is a flowchart illustrating one embodiment of the image encoding method of this application;

[0028] Figure 8 This is a flowchart illustrating one embodiment of the image decoding method of this application;

[0029] Figure 9 This is a schematic diagram of another embodiment of the image compression method of this application;

[0030] Figure 10 This is a schematic diagram of one embodiment of the encoder of this application;

[0031] Figure 11 This is a schematic diagram of the structure of one embodiment of the decoder of this application;

[0032] Figure 12 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application. In addition, unless otherwise specified (e.g., "or additionally" or "or in alternatives"), the term "or" as used herein refers to a non-exclusive "or" (i.e., "and / or"). Furthermore, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0034] Image compression methods, such as image encoding and decoding, can be implemented using models. For example, an end-to-end image codec composed entirely of neural networks can be used to perform image compression.

[0035] Among them, such as Figure 1 As shown, the encoding / decoding model can include a main encoding network and an entropy model network.

[0036] The main coding network may include transform, inverse transform model, quantization, dequantization, entropy coding and / or entropy decoding.

[0037] In transform / inverse transform networks, nonlocal attention modules can be used, where the element value of each feature is equal to the weighted sum of its original value and the weights of the entire feature location. The transform process continuously reduces dimensionality to decrease the amount of data; that is, its role is to represent the main features of the original image using a more compact representation, while reducing the image's dimensionality and data volume. Inverse transform networks, on the other hand, continuously increase dimensionality to restore the original data volume.

[0038] Quantization is one of the steps in lossy encoding, which converts data to integer form to improve compression ratio. Dequantization (optional) is the opposite operation, but it can also be omitted because the powerful nonlinear capabilities of neural networks can encompass the effect of dequantization.

[0039] Entropy coding is a lossless process that primarily calculates the probability of the sign bit in each feature using a pre-built probability model and encodes it into a binary representation, which is then written into the bitstream. Entropy decoding is the reverse process of entropy coding.

[0040] The entropy model network may include auxiliary transform, auxiliary inverse transform, quantization, inverse quantization, entropy encoding, entropy decoding, probabilistic model construction, prediction module, and / or context module. Except for the probabilistic model construction, prediction module, and / or context module, the functions of the remaining modules are similar to their corresponding modules in the main encoding network.

[0041] The construction of a probabilistic model mainly involves learning the model parameters through a neural network to calculate the probability of the feature to be encoded by the main encoding network.

[0042] In the auxiliary transform / inverse transform, a grouped convolution method is used. The features are evenly divided according to the channels, then grouped into a fixed number of groups, each group is convolved separately, and finally the groups are merged.

[0043] The prediction module can be used for difference encoding, so that the output of the inverse transform is fed into the prediction module, and then the difference between the output and the transformed quantized features is calculated, and this difference is fed into the encoding module. Optionally, the prediction module can also perform uniform channel division.

[0044] Furthermore, since each sample to be processed in the features depends on the previous processed sample, the context model can be used to learn their correlations, reducing redundancy.

[0045] While some image compression methods utilize contextual features, they often rely solely on convolutions with a single receptive field size (typically 5x5, 7x7, or 11x11) to extract these features. Larger kernel sizes refer to a wider neighborhood, but this weakens the correlation between distant information and the current point, increasing the likelihood of prediction errors. Conversely, smaller kernels refer to a smaller neighborhood but may miss highly relevant neighboring points. Consequently, the context model utilizes only a limited range of neighborhood information, failing to effectively eliminate encoding / decoding redundancy and resulting in low image processing efficiency.

[0046] Based on this, this application proposes a deep learning-based image compression method. This method utilizes convolution of receptive fields of various sizes to extract contextual features of the image, thereby effectively utilizing neighborhood information within a variety of ranges in the context model. This effectively eliminates encoding / decoding redundancy and improves image processing efficiency.

[0047] Specifically, such as Figure 2 As shown, the image compression method proposed in this application may include the following steps. It should be noted that the step numbers are for simplification only and are not intended to limit the execution order of the steps. The execution order of each step in this embodiment can be arbitrarily changed without departing from the technical concept of this application.

[0048] S101: Based on the multi-scale receptive field, convolution processing is performed on the features of the image to be processed to obtain multiple contextual features of the features of the image to be processed.

[0049] Optionally, in the image compression method, convolution processing can be performed on the features of the image to be processed based on multi-scale receptive fields to obtain multiple contextual features of the features of the image to be processed. These multiple contextual features can then be fused to obtain sample features of the features of the image to be processed. Thus, the image compression method of this embodiment proposes a multi-scale receptive field approach. By combining receptive fields of different scales, it effectively utilizes neighborhood point information within different ranges, effectively eliminating encoding / decoding redundancy, thereby improving image processing efficiency.

[0050] The image features to be processed are determined based on the image features of the image to be compressed. Optionally, the image features to be processed can be a single pixel or a pixel block within the image features of the image to be compressed.

[0051] In the first feasible method, such as Figure 3As shown, in step S101, convolution kernels corresponding to receptive fields of different scales can be used to perform convolution processing on the image features to be processed, obtaining multiple contextual features of the image features to be processed. The size of the convolution kernels corresponding to different receptive fields varies. For example, convolution kernels of 5x5, 7x7, and 11x11 sizes can be pre-trained. In step S101, a 5x5 convolution kernel can be used to perform convolution processing on the image features to be processed, obtaining one contextual feature; a 7x7 convolution kernel can be used to perform convolution processing on the image features to be processed, obtaining another contextual feature; and an 11x11 convolution kernel can be used to perform convolution processing on the image features to be processed, obtaining yet another contextual feature. By combining receptive fields of different scales and effectively utilizing neighborhood point information within different ranges, encoding / decoding redundancy can be effectively eliminated, thereby improving image processing efficiency.

[0052] In this feasible approach, the step of performing convolution processing on the image features to be processed using a convolution kernel may include: determining the context information of the image features to be processed in the corresponding receptive field using a mask of the same size as the convolution kernel; and performing calculations on the context information of the image features to be processed in the corresponding receptive field using the convolution kernel to obtain the context features corresponding to the convolution kernel.

[0053] The process of using a mask of the same size as the convolution kernel to determine the contextual information of the image features within the corresponding receptive field can be understood as extracting information from the region centered on the image features of the image to be compressed using the mask. Thus, the receptive field corresponding to the convolution kernel is related to the values ​​of each point in the mask corresponding to the kernel. Specifically, the receptive field corresponding to the convolution kernel can refer to the region within the area centered on the image features of the image to be processed that corresponds to the non-zero region in the mask corresponding to the convolution kernel (hereinafter referred to as the effective region for ease of description). The scale of the receptive field corresponding to the mask can be equal to the scale of the smallest bounding rectangle of the effective region of the mask.

[0054] Considering that during image decoding using image compression methods, the image features of the image to be compressed include samples to be decoded, and the data of these samples is invalid, to avoid the invalid data of the samples to be decoded affecting the extraction of context features, in step S101 when calculating context features, the convolution kernel can be used to operate only on the decoded samples in the neighborhood of the image feature to be processed (i.e., the current sample to be decoded), so as to predict the context features of the image feature to be processed relatively accurately, thereby avoiding the influence of invalid data of the samples to be decoded on the prediction of the image feature to be processed. Specifically, in order to achieve the operation of the convolution kernel only on the decoded samples in the neighborhood of the image feature to be processed, the parameters in the mask corresponding to the convolution kernel can be set based on the prediction direction, so that the context information of the image feature to be processed in the corresponding receptive field obtained by the mask corresponding to the convolution kernel only includes the decoded samples.

[0055] For example, such as Figure 4 As shown, a mask is placed on the image feature Y of the image to be compressed. If the center of the mask is the image feature to be processed, then the mask region centered on the image feature to be processed in the image feature of the image to be compressed can be determined. This mask region includes the current sample to be decoded (i.e., the image feature to be processed). Figure 4 (filled grid in the middle), decoded sample ( Figure 4 (the diagonal-filled grid in the middle) and the remaining samples to be decoded ( Figure 4 (Small grid filling cells in the image). To facilitate the use of convolution kernels to operate only on decoded samples in the neighborhood of the image features being processed when predicting them, the values ​​of the mask corresponding to the decoded samples can be set to non-zero values ​​(e.g., 1), and the values ​​of the mask corresponding to the samples to be decoded can be set to 0. Specifically, as shown in... Figure 4 As shown, assuming the prediction direction is from left to right and from top to bottom, the values ​​of points in the mask located directly to the left and above the center point can be set to non-zero values, while the values ​​of the remaining points in the mask can be set to 0. In other embodiments, assuming the prediction direction is from right to left and from bottom to top, the values ​​of points in the mask located directly to the right and below the center point can be set to non-zero values, while the values ​​of the remaining points in the mask can be set to 0. It is understood that the image features of the image to be compressed can be image features decoded from the bitstream.

[0056] Accordingly, in the image encoding process of the image compression method, a mask of each size is placed on the image features of the image to be compressed. If the center of the mask is the image feature to be processed, then the mask region centered on the image feature to be processed in the image features of the image to be compressed can be determined. This mask region includes the current sample to be encoded (i.e., the image feature to be processed), the encoded samples, and the remaining samples to be encoded. The context information of the image feature to be processed extracted based on the mask can only include the encoded samples in the mask region. The image features of the image to be compressed can be features obtained by processing the image, or it can be the image itself.

[0057] In addition, the step of using the convolution kernel to operate on the context information of the image features to be processed within the corresponding receptive field to obtain the context features corresponding to the convolution kernel can be as follows: multiply each point value in the context information of the image features to be processed by the parameters of the corresponding points in the convolution kernel to obtain the product of each point in the context information of the image features to be processed; and obtain the context features corresponding to the convolution kernel based on the product of all points in the context information of the image features to be processed.

[0058] In a second feasible approach, the image features to be processed are convolved using at least one convolution kernel and at least two masks to obtain a variety of contextual features.

[0059] In this implementation, at least one convolutional kernel corresponds to at least two masks. For a convolutional kernel corresponding to at least two masks (hereinafter referred to as the first convolutional kernel for ease of description), the size of the at least two masks corresponding to the first convolutional kernel is equal to the size of the first convolutional kernel. The only difference is that the effective regions of the at least two masks corresponding to the first convolutional kernel are set differently. The effective region of a mask is the region within the mask that is not zero. Thus, different neighborhood information of the features of the image to be processed can be extracted through different masks, resulting in multiple receptive fields during convolution processing. Combining different receptive fields effectively utilizes neighborhood point information within different ranges, effectively eliminating encoding / decoding redundancy and improving image compression efficiency. Furthermore, this implementation reduces the number of training convolutional kernels. Compared to the first feasible approach, this does not significantly increase model complexity and effectively learns the correlation between different neighborhood ranges and the points to be decoded, reducing redundancy.

[0060] The scale of the receptive field corresponding to the mask can be equal to the scale of the minimum bounding rectangle of the mask's effective region. Preferably, at least two masks corresponding to at least one first convolutional kernel have different receptive field scales. This allows for the combination of receptive fields of different scales, effectively utilizing neighborhood point information within different ranges and effectively eliminating encoding / decoding redundancy. For example, there might be a first convolutional kernel with a scale of 11×11, corresponding to three masks with receptive field scales of 5×5, 7×7, and 11×11, respectively.

[0061] Furthermore, as described in the first achievable manner, in order to facilitate the use of the convolution kernel to operate only on the processed image features in the neighborhood of the image features to be processed when predicting the image features to be processed, the value of the mask corresponding to the region of the processed image features can be set to a non-zero value (e.g., 1), and the value of the mask corresponding to the region of the image features to be processed and the unprocessed image features can be set to 0.

[0062] In this feasible approach, the step of performing convolution processing on the image features to be processed using at least one convolution kernel and at least two masks to obtain multiple contextual features may include: determining the contextual information of the image features to be processed in the corresponding receptive field using each mask corresponding to each convolution kernel; and performing operations on the contextual information corresponding to each mask using the convolution kernel to obtain the contextual features corresponding to the convolution kernel and each mask.

[0063] In a specific example, such as Figure 5 As shown, three 11x11 masks are preset, with receptive field scales of 5x5, 7x7, and 11x11 respectively, and 11x11 convolutional kernels are trained. In step S101, the 11x11 convolutional kernel is used to process the context information of the 5x5 receptive field mask to obtain the context feature Y3 corresponding to the 11x11 convolutional kernel and the 5x5 receptive field mask; the 11x11 convolutional kernel is used to process the context information of the 7x7 receptive field mask to obtain the context feature Y2 corresponding to the 11x11 convolutional kernel and the 7x7 receptive field mask; the 11x11 convolutional kernel is used to process the context information of the 11x11 receptive field mask to obtain the context feature Y1 corresponding to the 11x11 convolutional kernel and the 11x11 receptive field mask.

[0064] Furthermore, when using the convolution kernel to perform operations on the context information corresponding to each mask to obtain the context features corresponding to the convolution kernel and each mask, the product of each point in the context information can be calculated and pre-stored. When determining the context features corresponding to each mask, the pre-stored product of the corresponding points can be directly used for calculation. In this way, for the same points in the context information corresponding to at least two masks of the first convolution kernel, there is no need to repeatedly calculate their product, thereby reducing the impact of setting masks of multiple receptive field sizes on model complexity and computational complexity. When only one convolution kernel is set, the computational cost of this method is almost the same as that of a single scale.

[0065] In addition, considering that the data of the sample to be decoded (i.e. the sample to be processed) in the mask area are all invalid data during the image decoding process using image compression method, in order to avoid the invalid data from affecting the context features during the convolution processing in step S102, the parameter corresponding to the sample to be processed in the convolution kernel can be set to 0.

[0066] Optionally, in the above implementation, the width and height of the convolution kernel can be the same or different. The number of different types of convolution kernels is also unlimited.

[0067] S102: Fuse multiple contextual features to obtain sample features of the image to be processed.

[0068] After obtaining multiple contextual features, these features can be fused to obtain sample features of the image to be processed.

[0069] The fusion method is unrestricted; for example, multiple contextual features can be fused using any of the following methods.

[0070] In one feasible method, multiple contextual features can be weighted to obtain sample features of the image to be processed. The weighting coefficients for each contextual feature are unrestricted; they can be preset or trained. The sum of the weighting coefficients of all contextual features can equal 1.

[0071] In another feasible method, multiple contextual features can be concatenated to obtain concatenated features, which are then convolved to obtain sample features of the image to be processed. In other words, multiple contextual features can be fused using convolutional fusion to obtain sample features of the image to be processed.

[0072] S103: Based on the sample features of the image to be processed, obtain the compression result of the image to be compressed.

[0073] After obtaining the sample features of the image to be processed, the compression result of the image to be compressed can be obtained based on the sample features of the image to be processed.

[0074] Optionally, if the image feature to be processed is not the last feature among the image features of the image to be compressed, after obtaining the sample features of the image feature to be processed, the image feature to be processed has become a processed image feature. The next feature of the image feature to be processed can be used as the image feature to be processed, and the process returns to step S101 to determine the sample features of the feature using steps S101 and S102. This process of "using the next feature of the image feature to be processed as the image feature to be processed", S101 and S102 is repeated until the image feature to be processed is the last feature among the image features of the image to be compressed, so as to determine the sample features of all features in the image to be compressed, thereby obtaining the context processing result of the image to be compressed.

[0075] Optionally, the context processing result of the image to be compressed can be output as the compression result of the image to be compressed. Alternatively, the context processing result of the image to be compressed can be processed to obtain the compression result of the image to be compressed.

[0076] In this embodiment, the image compression method can perform convolution processing on the features of the image to be processed based on multi-scale receptive fields to obtain multiple contextual features of the features of the image to be processed. These multiple contextual features can then be fused to obtain sample features of the features of the image to be processed. Thus, the image compression method of this embodiment proposes a multi-scale receptive field approach. By combining receptive fields of different scales, it can effectively utilize neighborhood point information within different ranges, effectively eliminate encoding / decoding redundancy, and thereby improve image processing efficiency.

[0077] In end-to-end image codec solutions, the image features of the image to be compressed include not only spatial dimensions but also channel dimensions. In the context models of related technologies, the same convolutional kernel is used to process all channels, resulting in low image processing efficiency.

[0078] Based on this, such as Figure 6 As shown, this application proposes another image compression method to simultaneously and fully learn channel information and utilize spatial information to process the features of the image to be processed. This image compression method may include the following steps. It should be noted that the step numbers are for simplification only and are not intended to limit the execution order of the steps. The execution order of each step in this embodiment can be arbitrarily changed without departing from the technical concept of this application.

[0079] S201: Divide the image features of the image to be compressed into multiple sub-features according to channels.

[0080] In one alternative implementation, the image features of the image to be compressed can be divided into multiple sub-features according to channels based on the distribution of inter-channel information in the image features of the image to be compressed. Then, the multiple sub-features can be convolved separately, and the convolution results of the multiple sub-features can be concatenated to obtain the target features of the image features of the image to be compressed.

[0081] Optionally, the inter-channel information distribution refers to the distribution of image information among different channels in the feature; that is, the inter-channel information distribution can refer to the distribution of image information content among different channels in the feature. The image information content of at least one channel in the image feature of the image to be compressed can refer to the proportion of the original image recovered based on that at least one channel. For example, if the feature has 4 channels, and the first channel of the feature can recover 40% of the image, the second channel can recover 30%, the third channel can recover 20%, and the fourth channel can recover 10%, then the image information content in the four channels of the feature is 40%, 30%, 20%, and 10% respectively, and the inter-channel information distribution in the feature is (40%, 30%, 20%, 10%).

[0082] In this optional implementation, based on the distribution of inter-channel information in the image features of the image to be compressed, the image features are divided into multiple sub-features according to channels to balance the information content of different sub-features. This allows the convolutional module corresponding to each sub-feature to learn the feature, thereby facilitating the extraction of information from the feature by each convolutional module. This ensures that the convolutional parameters in the grouped convolutional model can fully learn the feature, improving the feature representation effect and thus enhancing the image processing performance. For example, assuming the image features are divided into four sub-features according to step S201, the information content of different sub-features can be balanced, ensuring that the difference between the image information content of each of the four sub-features and 25% is within a difference threshold. The difference threshold can be set according to actual conditions and is not limited here; for example, it can be 1%, 5%, or 7%.

[0083] In a specific example, an engineer can manually determine the distribution of information between channels in the image features of the image to be compressed based on domain knowledge, and the engineer can then divide the image features of the image to be compressed by channel based on the determined distribution of information between channels.

[0084] In another specific example, deep learning methods can be used to process the image features of the image to be compressed, determining the number of channels for each of the multiple sub-features; based on the number of channels for each sub-feature, the image features of the image to be compressed are divided into multiple sub-features. It can be understood that the deep learning method also determines the number of channels for each sub-feature based on the distribution of information between channels in the image features of the image to be compressed.

[0085] In another specific example, a base value for the number of channels for each of the multiple sub-features can be preset; and a deep learning method can be used to process the image features of the image to be compressed to determine the offset value for the number of channels for each of the multiple sub-features; the base value for the number of channels and the offset value for the number of channels for each sub-feature are added together to obtain the number of channels for each sub-feature; and the image features of the image to be compressed are divided into the multiple sub-features according to the number of channels for each of the multiple sub-features.

[0086] In another alternative implementation, the number of channels for each of the multiple sub-features can be preset. In step S201, the image features of the image to be compressed can be divided into multiple sub-features according to the preset number of channels for each of the multiple sub-features.

[0087] The number of channels for different sub-features can be equal or unequal, and no restriction is imposed here.

[0088] In addition, the number of feature groups can be set according to the actual situation and is not limited here. For example, it can be 5, 8 or 10, etc.

[0089] S202: Based on the multi-scale receptive field, the image features to be processed in each sub-feature are convolved to obtain multiple contextual features of the image features to be processed in each sub-feature.

[0090] For details, please refer to step S101, which will not be elaborated here.

[0091] Optionally, the coefficients of convolution kernels of the same size for different sub-features can be the same or different.

[0092] In one feasible approach, for each sub-feature, a corresponding convolution kernel can be pre-trained. Thus, in step S202, the convolution kernel corresponding to each sub-feature can be used to perform convolution processing on various contextual information of each sub-feature to obtain multiple contextual features of the image features to be processed in each sub-feature. For specific operation steps, please refer to S101.

[0093] In another possible approach, at least one total convolutional kernel can be trained for the image features to be processed. Then, convolutional kernels corresponding to the positions of each sub-feature can be extracted from each total convolutional kernel to obtain at least one convolutional kernel corresponding to each sub-feature. Subsequently, the convolutional kernels corresponding to each sub-feature are used to perform convolution processing on various contextual information of each sub-feature to obtain multiple contextual features of the image features to be processed in each sub-feature. For specific operation steps, please refer to S101.

[0094] S203: Fuse multiple contextual features of each sub-feature to obtain sample features of the image features to be processed in each sub-feature.

[0095] For details, please refer to step S102, which will not be elaborated here.

[0096] S204: Based on the sample features of the image features to be processed in each sub-feature, the compression result of the image to be compressed is obtained.

[0097] If the image feature to be processed is not the last feature in the image features of the image to be compressed, after obtaining the sample features of the image feature to be processed in the sub-features, the next feature of the image feature to be processed can be used as the image feature to be processed, and the process returns to step S202, so as to determine the sample features of the feature using steps S202 and S203; this process of "using the next feature of the image feature to be processed as the image feature to be processed", S202 and S203 is repeated until the image feature to be processed is the last feature in the sub-features, so as to determine the sample features of all features in each sub-feature, thereby obtaining the context processing result of the sub-features; then the context processing results of the sub-features can be concatenated to obtain the context processing result of the image to be compressed.

[0098] After obtaining the context processing result of the image to be compressed, the context processing result of the image to be compressed can be used as the compression result of the image to be compressed, or the context processing result of the image to be compressed can be processed to obtain the compression result of the image to be compressed.

[0099] Please continue reading. Figure 7 , Figure 7 This is a flowchart illustrating one embodiment of the image encoding method provided in this application.

[0100] like Figure 7 As shown, the image encoding method of this embodiment includes the following steps:

[0101] S301: Obtain the features of the image to be processed based on the image features of the image to be compressed.

[0102] S302: The image features to be processed are processed using the above image compression method to obtain the compression result of the image to be compressed.

[0103] S303: Obtain the encoded bitstream of the image to be compressed based on the compression result.

[0104] Please continue reading. Figure 8 , Figure 8 This is a flowchart illustrating one embodiment of the image decoding method provided in this application.

[0105] like Figure 8 As shown, the image decoding method of this embodiment includes the following steps:

[0106] S401: Decode the bitstream of the image to be compressed to obtain the image features of the image to be compressed.

[0107] S402: Using the image compression method described above, the image features to be processed within the image features of the image to be compressed are processed to obtain the compression result of the image to be compressed.

[0108] In the image decoding process, step S402 may include: performing convolution processing on at least a portion of samples in the image to be compressed, including features to be processed, based on multi-scale receptive fields, to obtain multiple intermediate features of the image to be compressed; the data of the image features to be processed in the intermediate features corresponding to each receptive field is the context feature corresponding to each receptive field, and the data of the processed image features in the intermediate features is equivalent to the data in the image features of the image to be compressed; fusing the multiple intermediate features to obtain the updated features of the image to be compressed, and the data of the image features to be processed in the updated features of the image to be compressed is the sample feature of the image features to be processed; if the image features to be processed are not the image features to be compressed... The last feature in the image features of the image is used as the updated feature of the image to be compressed, and the next feature of the image to be processed is used as the image feature to be processed; the process returns to the step of performing convolution processing on at least a portion of the samples in the image to be compressed, including the image feature to be processed, based on a multi-scale receptive field, to obtain multiple intermediate features of the image to be compressed, until the image feature to be processed is the last feature in the image features of the image to be compressed, thus obtaining the context processing result of the image to be compressed; the context processing result of the image to be compressed can be used as the compression result of the image to be compressed, or the context processing result of the image to be compressed can be processed to obtain the compression result of the image to be compressed.

[0109] Furthermore, considering that the data of the processed image features in the intermediate features of the image to be compressed remains unchanged, during the convolution process of at least a portion of the samples in the image to be compressed, including the features to be processed, based on the multi-scale receptive field, it is not necessary to calculate the sample features of the processed image features. That is, only the sample features of the features to be processed and the sample features of the unprocessed image features need to be calculated. The calculation method of the sample features of the unprocessed image features can refer to the calculation method of the sample features of the features to be processed, and will not be elaborated here.

[0110] Furthermore, when determining the compression result of the image to be compressed using the image compression method of the second embodiment, based on the content of steps S102 and S203, it is known that the updated features of each sub-feature can be obtained based on step S402. In this step, the updated features of multiple sub-features can be concatenated to obtain the updated features of the image to be compressed; then, the updated features of the image to be compressed are used as the image features of the image to be compressed, and the next feature of the image features to be processed is used as the image features to be processed, and the process returns to step S201, "dividing the image features of the image to be compressed into multiple sub-features according to channels," and steps S201, S202, S203, and S204 are executed sequentially. This process is repeated to sequentially obtain the sample characteristics of all features in the image features of the image features to be compressed, thereby obtaining the context processing result of the image to be compressed. Of course, in other embodiments, the sample features of all features in each sub-feature can also be determined first to obtain the context processing result of each sub-feature; then, the context processing results of multiple sub-features can be concatenated to obtain the context processing result of the feature to be compressed.

[0111] Additionally, as described in step S102, the multiple intermediate features can be fused using methods such as weighted fusion or convolutional fusion. For example, the multiple intermediate features can be directly weighted to obtain the updated features of the image to be compressed. Alternatively, the multiple intermediate features can be sequentially concatenated and convolved to obtain the updated features of the image to be compressed.

[0112] S403: Obtain the decoded image of the bitstream based on the compression result.

[0113] To better illustrate the image decoding method of this application, the following specific embodiments of image decoding are provided as examples:

[0114] Example 1

[0115] a. such as Figure 3 As shown, three receptive fields of different sizes are set, and convolution is performed using 5x5, 7x7 and 11x11 convolution kernels respectively. The mask labels are similar and related to the convolution kernel size, identifying decoded samples (diagonal filled parts), image features to be processed (horizontal filled parts) and other samples to be decoded (small grid filled areas).

[0116] b. Input a 512x512x128 feature Y, where the (x,y) position in this feature is the image feature to be processed, x=y=128. Copy the feature Y into three parts, and perform mask convolutions of 5x5, 7x7 and 11x11 in sequence.

[0117] c. Perform convolutional fusion (by channel dimension) on the output features Y1, Y2, and Y3 at the three scales to obtain the updated feature Y' of 512x512x128.

[0118] d. such as Figure 4 and Figure 1 As shown, the updated feature Y' is used as the image feature to be processed, and the next sample to be decoded of the image feature to be processed is used as the image feature to be processed. Then, the process returns to step a to determine the context processing result of each sample in the image feature to be processed in turn, and then the decoded image is obtained based on the context processing result of the image.

[0119] Example 2

[0120] 1. For example Figure 5 As shown, three receptive fields are set, using only 11x11 convolution kernels, while the mask design is related to the size of 5x5, 7x7, and 11x11, marking the decoded samples (diagonal filled parts), the image features to be processed (horizontal filled parts), and the remaining samples to be decoded (small grid filled areas).

[0121] 2. Input a 512x512x128 feature Y, where the (x,y) position in this feature is the image feature to be processed, x=y=128. Copy the feature Y into three parts, and perform mask convolutions of 5x5, 7x7 and 11x11 in sequence.

[0122] 3. Perform convolutional fusion (by channel dimension) on the output features Y1, Y2, and Y3 at the three scales to finally obtain the 512x512x128 feature Y'.

[0123] 4. For example Figure 4 and Figure 1 As shown, the updated feature Y' is used as the image feature to be processed, and the next sample to be decoded of the image feature to be processed is used as the image feature to be processed. Then, the process returns to step 1 to determine the context processing result of each sample in the image feature to be processed in turn, and then the decoded image is obtained based on the context processing result of the image.

[0124] Example 3

[0125] Features with N channels are grouped according to their channel dimension, with each group containing independent sub-features. These independent features are then subjected to multi-scale convolutions. The resulting independent features are then merged again. Efficient channel grouping implies different grouping strategies. Since efficient channels are generally clustered at the beginning, while inefficient channels are generally clustered at the end, efficient channel regions can be subdivided into smaller groups, while inefficient channel regions can be coarsely grouped. For example... Figure 9 As shown, the steps can be as follows:

[0126] 1. Input a 512x512x128 feature Y, and group it from front to back according to the number of channels 16, 16, 32, 64 to obtain sub-features Y1, Y2, Y3, Y4.

[0127] 2. Set three receptive fields of different sizes, and use 3x3, 5x5 and 11x11 convolution kernels for convolution respectively. The mask labels are similar and related to the size of the convolution kernel. They are labeled as decoded samples (diagonal filled parts), image features to be processed (horizontal filled parts) and other samples to be decoded (small grid filled areas).

[0128] 3. Taking sub-feature Y1 as an example, where the (x,y) position in this sub-feature is the image feature to be processed, x=y=128. Copy the sub-feature Y three times, and perform mask convolutions of 3x3, 5x5 and 11x11 in sequence.

[0129] 4. Perform convolutional fusion (by channel dimension) on the output features Y1_1, Y1_2, and Y1_3 at the three scales to finally obtain the 512x512x128 sub-feature Y1'.

[0130] 5. Similarly, sub-features Y2, Y3, and Y4 are processed through steps 2, 3, and 4 to obtain sub-features Y2', Y3', and Y4'. These sub-features are then merged to finally output Y'.

[0131] Please see Figure 10 , Figure 10 This is a schematic diagram of one embodiment of the encoder of this application. The encoder 10 includes a processor 12, which executes instructions to implement the prediction method and image encoding method described above. For detailed implementation processes, please refer to the description of the above embodiments, which will not be repeated here.

[0132] Processor 12 can also be referred to as a CPU (Central Processing Unit). Processor 12 may be an integrated circuit chip with signal processing capabilities. Processor 12 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor, or processor 12 can be any conventional processor.

[0133] The encoder 10 may further include a memory 11 for storing instructions and data required for the processor 12 to run.

[0134] The processor 12 is used to execute instructions to implement the methods provided by any embodiment of the prediction method and image encoding method of this application and any non-conflicting combination thereof.

[0135] Please see Figure 11 , Figure 11 This is a schematic diagram of one embodiment of the decoder of this application. The decoder 20 includes a processor 22, which executes instructions to implement the prediction method and image encoding method described above. For detailed implementation processes, please refer to the description of the above embodiments; they will not be repeated here.

[0136] Processor 22 can also be referred to as CPU (Central Processing Unit). Processor 22 may be an integrated circuit chip with signal processing capabilities. Processor 22 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor can be a microprocessor, or processor 22 can be any conventional processor.

[0137] The decoder 20 may further include a memory 21 for storing instructions and data required for the processor 22 to run.

[0138] The processor 22 is used to execute instructions to implement the methods provided by any embodiment of the prediction method and image coding method of this application and any non-conflicting combination thereof.

[0139] Please see Figure 12 , Figure 12This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of this application. The computer-readable storage medium 30 in this embodiment stores instruction / program data 31. When executed, this instruction / program data 31 implements the methods provided by any embodiment of the image compression method and image encoding method of this application, as well as any non-conflicting combination thereof. The instruction / program data 31 can be formed into a program file and stored in the storage medium 30 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) or processor can execute all or part of the steps of the methods in various embodiments of this application. The aforementioned storage medium 30 includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or terminal devices such as computers, servers, mobile phones, and tablets.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0141] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0142] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0143] The above are merely embodiments of this application and do not limit the scope of this patent application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. An end-to-end image compression method, characterized in that, The method includes: Based on multi-scale receptive fields, convolution processing is performed on the features of the image to be processed to obtain multiple contextual features of the features of the image to be processed; the features of the image to be processed are determined based on the image features of the image to be compressed. The various contextual features are fused to obtain sample features of the image to be processed; Based on the sample features of the image to be processed, the compression result of the image to be compressed is obtained; The method of performing convolution processing on the image features to be processed based on multi-scale receptive fields to obtain multiple contextual features of the image features to be processed includes: using at least one convolution kernel and at least two masks to perform convolution processing on the image features to be processed to obtain multiple contextual features, wherein the effective regions of the at least two masks corresponding to each convolution kernel are different, and the effective region of the mask is the region in the mask that is not zero.

2. The image compression method according to claim 1, characterized in that, The multi-scale receptive field is used to perform convolution processing on the features of the image to be processed, resulting in various contextual features of the features, including: The image features to be processed are convolved using convolution kernels corresponding to the receptive fields at each scale to obtain various contextual features; wherein: Different receptive fields correspond to different convolution kernel sizes.

3. The image compression method according to claim 1, characterized in that, The compression direction of the image is from the first direction to the second direction, and from the third direction to the fourth direction; Points in the mask that have a non-zero value are located in the third upward direction and the positive first direction from the center point of the mask.

4. The image compression method according to claim 1, characterized in that, The process of fusing the multiple contextual features includes: The various contextual features are convolved and fused to obtain sample features of the image features to be processed.

5. The image compression method according to claim 1, characterized in that, The multi-scale receptive field is used to perform convolution processing on the features of the image to be processed, resulting in various contextual features of the features, including: The image features of the image to be compressed are divided into multiple sub-features according to channels; based on the multi-scale receptive field, the image features to be processed in each sub-feature are convolved to obtain multiple contextual features of the image features to be processed in each sub-feature; The step of fusing the multiple contextual features to obtain sample features of the image features to be processed includes: fusing multiple contextual features of the image features to be processed for each sub-feature to obtain sample features of the image features to be processed for each sub-feature; The step of obtaining the compression result of the image to be compressed based on the sample features of the image to be processed includes: obtaining the compression result of the image to be compressed based on the sample features of the image to be processed for each sub-feature.

6. The image compression method according to claim 5, characterized in that, The step of dividing the image features of the image to be compressed into multiple sub-features according to channels includes: dividing the image features of the image to be compressed into multiple sub-features according to channels based on the distribution of inter-channel information in the image features of the image to be compressed.

7. An end-to-end image coding method, characterized in that, The method includes: Obtain the features of the image to be processed based on the image features of the image to be compressed; The image compression method according to any one of claims 1-6 is used to process the features of the image to be processed to obtain the compression result of the image to be compressed. The encoded bitstream of the image to be compressed is obtained based on the compression result.

8. An end-to-end image decoding method, characterized in that, The method includes: The bitstream of the image to be compressed is decoded to obtain the image features of the image to be compressed; The image compression method according to claim 1 is used to process the image features to be processed within the image features of the image to be compressed, thereby obtaining the compression result of the image to be compressed; The decoded image of the bitstream is obtained based on the compression result.

9. The image decoding method according to claim 8, characterized in that, The image compression method according to claim 1 processes the image features to be processed within the image features of the image to be compressed to obtain the compression result of the image to be compressed, including: Based on multi-scale receptive fields, convolution processing is performed on the image features to be processed to obtain multiple intermediate features of the image to be compressed; the data of the image features to be processed in the intermediate features corresponding to each receptive field is the context feature corresponding to each receptive field, and the data of the processed image features in the intermediate features is equivalent to the data in the image features of the image to be compressed. The multiple intermediate features are fused to obtain the updated features of the image to be compressed; If the image feature to be processed is not the last feature among the image features of the image to be compressed, the updated feature of the image to be compressed is used as the image feature of the image to be compressed, and the next feature of the image feature to be processed is used as the image feature to be processed. Return to the step of performing convolution processing on the image features to be processed based on multi-scale receptive fields to obtain multiple intermediate features of the image to be compressed, until the image features to be processed are the last feature in the image features of the image to be compressed, and obtain the compression result.

10. An encoder, characterized in that, The encoder includes a processor; the processor is configured to execute instructions to implement the steps of the method as described in claim 7.

11. A decoder, characterized in that, The decoder includes a processor; the processor is configured to execute instructions to implement the steps of the method as described in claim 8 or 9.

12. A computer-readable storage medium storing instruction / program data thereon, characterized in that, When the instruction / program data is executed, it implements the steps of the method described in any one of claims 1-6, 7, or 8-9.

Images