Coding method and apparatus, and decoding method and apparatus

Abstract

The present application relates to the field of image coding and decoding. Disclosed is an image coding or decoding method, which can be applied to next-generation image or video coding and decoding standards such as JPEG-AI. The method comprises: acquiring a first feature of an image, wherein the shape of the first feature is \[C, h, w]; performing reordering on the first feature, so as to obtain a second feature, wherein the shape of the second feature is \[4C, h / 2, w / 2], the second feature comprises C sub-features, and the shape of each sub-feature is \[4, h / 2, w / 2]; performing recombining on the second feature, so as to obtain K third features, wherein the shape of a kth third feature is \[C, h / 2, w / 2], and the kth third feature comprises one channel of each sub-feature in the second feature, K being a positive integer greater than 2; and coding or decoding the image on the basis of the K third features. The present application can fully utilize adjacent pixel information in the same channel of a hidden variable and pixel information at the same position in neighboring channels, such that the hidden variable predicted by a context model is more accurate, thereby reducing the number of bits required for coding the hidden variable, and reducing the code rate.

Description

A method and device for encoding and decoding Technical Field

[0001] The embodiments of the present application relate to the field of image or video encoding and decoding, and in particular to an encoding and decoding method and device. Background Art

[0002] The AI ​​(Artificial Intelligence) image compression algorithm is based on deep learning and has better compression effects than traditional image compression technologies such as JPEG (Joint Photographic Experts Group) and BPG (Better Portable Graphics).

[0003] When using AI compression (also known as encoding) and decompression (also known as decoding), the input data is complex data such as images and videos. The encoder and super encoder output latent variables and super latent variables, and perform entropy encoding on the latent variables and super latent variables. The decoding process first decodes the super latent variables, then uses the super decoder to predict the latent variables, and restores the latent variables based on the results of entropy decoding and the context model. Finally, the decoder is used to restore the original data.

[0004] In AI encoding and decoding, a context model is needed to predict latent variables. The prediction accuracy of the context model will directly affect the overall effect of encoding and decoding.

[0005] Summary of the Invention

[0006] This application provides an encoding and decoding method and apparatus. This method can fully utilize the pixel information adjacent to the same channel of the latent variable and the pixel information at the same position in the surrounding channels, making the context model prediction more accurate, thereby reducing the number of bits required for entropy coding the latent variable and achieving better encoding performance.

[0007] In a first aspect, an embodiment of the present application provides an encoding or decoding method, the method comprising: obtaining a first feature of an image (in the entire encoding and decoding process of the image, the first feature can be referred to as a priori feature), the shape of the first feature being [C, h, w], wherein C represents the number of channels of the first feature, h represents the length of the first feature, w represents the width of the first feature, C, h and w are all positive integer multiples of 2, each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, and each of the pixel blocks includes 4 pixels in the upper left, lower right, upper right and lower left; reordering the first feature to obtain a second feature, the shape of the second feature being [4C, h / 2, w / 2], and the second feature including C sub-features , each of the sub-features has a shape of [4, h / 2, w / 2], the cth sub-feature of the second feature corresponds to the cth channel of the first feature, c is a positive integer and satisfies 1≤c≤C, and the four channels in the cth sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all the pixel blocks of the cth channel of the first feature; recombining the second features to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes one channel of each sub-feature in the second feature, K is a positive integer greater than 2, k is a positive integer and satisfies 1≤k≤K; encoding or decoding the image based on the K third features.

[0008] In the present application, by reordering and recombining the prior features (i.e., the first features), a more reasonable prior feature for segmentation is obtained, and then the context model segmentation is input to predict the latent variable, so that when predicting the segment after the latent variable, the pixel information adjacent to the same channel and the pixel information at the same position of the surrounding channel in the segment of the latent variable that has been predicted before can be fully utilized, thereby improving the prediction accuracy of the latent variable, thereby reducing the number of bits required for encoding the latent variable and improving the coding efficiency. After the bits required in the code stream are reduced, the communication bandwidth required for transmitting the code stream and the storage resources required for storing the code stream can be saved, and the corresponding technical effects can be reflected in the encoding process, transmission process, and decoding process. The prior features are processed in the same way as the encoding side on the decoding side to ensure correct decoding.

[0009] In a possible implementation of the first aspect, as shown in Figures 4a to 4d, the upper left, lower right, upper right, and lower left pixels of each 2×2 pixel block in the cth channel of the first feature are represented by 0, 1, 2, 3 or 4, 5, 6, and 7, respectively, where c is a positive integer and satisfies 1≤c≤C. That is, the upper left pixel of each pixel block in the same channel is represented by 0 or 4, the lower right pixel of each pixel block in the same channel is represented by 1 or 5, the upper right pixel of each pixel block in the same channel is represented by 2 or 5, and the lower left pixel of each pixel block in the same channel is represented by 4 or 7.

[0010] It is worth noting that Figures 4a to 4d only give several specific examples of the solutions provided in this application. This application does not limit the value of C to 2 or 4, and the values ​​of h and w are not limited to 4, as long as C, h and w are all positive integers and integer multiples of 2; this application does not limit the channel arrangement of each third feature in the K third features, as long as the positions of N pixels at the same position in the N channels of the kth third feature are different in the N channels of the first feature, where N is a positive integer and 2≤N≤C; this application does not limit the value of K to 3 or 4, as long as K is a positive integer greater than 2.

[0011] In the method provided in this application, the number of channels of each of the K third features may be the same or different, but the total number of channels of all the K third features is required to be equal to 4 times of C.

[0012] In a possible implementation of the first aspect, the third feature obtained after recombining satisfies the following conditions: the positions of N pixels at the same position in N channels of the kth third feature are different from each other in the N channels of the first feature, wherein N is a positive integer and 2≤N≤C, wherein the position of the pixel can be represented by spatial coordinates [x,y], x represents the horizontal coordinate offset of the pixel relative to the upper left corner of the image, and y represents the vertical coordinate offset of the pixel relative to the upper left corner of the image; the position of the pixel can also be indicated by a symbol representing the pixel. The positions of N pixels at the same position in N channels of the kth third feature are different from each other in the N channels of the first feature, which means that the pixels included in the N channels exclude the following situations: (1) all are upper left pixels in the 2×2 pixel block of the first feature; (2) all are lower right pixels in the 2×2 pixel block of the first feature; (3) all are upper right pixels in the 2×2 pixel block of the first feature; (4) all are lower left pixels in the 2×2 pixel block of the first feature. By limiting the positions of the N pixels at the same position in the N channels of the kth third feature after the recombination to be different from each other in the N channels of the first feature, it can be ensured that when the third feature is used to predict the latent variable, the pixel information at the same position in the surrounding channels can be fully utilized (because the pixel points at the same position in the surrounding channels have been obtained in the previous segmented prediction), which can improve the prediction accuracy of the latent variable, thereby reducing the number of bits required to encode the latent variable and reducing the bit rate. It should be noted that in this application, it is only necessary to limit at least one third feature among all K third features to meet the above conditions, and it is not required that all third features meet the above conditions.

[0013] In a possible implementation of the first aspect, C / 2 channels in the kth third feature are composed of the i-th channel of each subfeature in the first portion of subfeatures in the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each subfeature in the second portion of subfeatures in the second feature, where i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j, the first portion of subfeatures and the second portion of subfeatures each include C / 2 subfeatures in the second feature, and the first portion of subfeatures and the second portion of subfeatures together constitute the second feature. When K=4 and C=2, possible combinations of this implementation are listed in Table 1. The first, second, third, and fourth channels of any subfeature in the second feature are composed, in sequence, of the top left, bottom right, top right, and bottom left pixels in all 2×2 pixel blocks in the same channel of the first feature. Therefore, by limiting i≠j, it can be ensured that the C / 2 channels in the kth third feature and the remaining C / 2 channels are composed of pixel points at different positions in the pixel block in the first feature, thereby ensuring that the division method can make full use of the pixel information adjacent to the same channel and the pixel information at the same position of the surrounding channels when the context model predicts the latent variable, so that the latent variable predicted by the context model is more accurate, reducing the number of bits required to transmit the residual of the latent variable and improving the coding efficiency. It should be noted that in this application, it is only necessary to limit at least one third feature among all K third features to meet the above conditions, and it is not required that all K third features meet the above conditions.

[0014] In a possible embodiment of the above implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features. If C=2, it is the example given in Figure 4a, that is, the first third feature is composed of the first channel of the first subfeature of the second feature and the third channel of the second subfeature of the second feature; the second third feature is composed of the second channel of the first subfeature of the second feature and the fourth channel of the second subfeature of the second feature; the third third feature is composed of the third channel of the first subfeature of the second feature and the first channel of the second subfeature of the second feature; the fourth third feature is composed of the fourth channel of the first subfeature of the second feature and the second channel of the second subfeature of the second feature.

[0015] In a possible embodiment of the above implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of sub-features. If C=2, it is the example given in Figure 4b, that is, the first third feature is composed of the first channel of the first subfeature of the second feature and the fourth channel of the second subfeature of the second feature; the second third feature is composed of the second channel of the first subfeature of the second feature and the third channel of the second subfeature of the second feature; the third third feature is composed of the third channel of the first subfeature of the second feature and the second channel of the second subfeature of the second feature; the fourth third feature is composed of the fourth channel of the first subfeature of the second feature and the first channel of the second subfeature of the second feature.

[0016] In a possible embodiment of the above implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of sub-features.

[0017] In a possible embodiment of the above implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features.

[0018] In a possible embodiment of the above implementation, K=3. In this case, the shapes of the first and second third features are [C, h / 2, w / 2], and the shape of the third feature is [2C, h / 2, w / 2]; the C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; the C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; the C channels in the third third feature are composed of the third and fourth channels of each sub-feature in the first part of sub-features, and the remaining C channels in the third third feature are composed of the first and second channels of each sub-feature in the second part of sub-features.

[0019] In a possible implementation of the first aspect, K=4, and the third features are combined as follows: the first third feature is composed of the 4cth channel of the first sub-feature and the 4c+2th channel of the second sub-feature; the second third feature is composed of the 4c+1th channel of the first sub-feature and the 4c+3th channel of the second sub-feature; the third third feature is composed of the 4c+2th channel of the first sub-feature and the 4cth channel of the second sub-feature; and the fourth third feature is composed of the 4c+3th channel of the first sub-feature and the 4c+1th channel of the second sub-feature. Where c is an integer, and the numbering index starts at 0.

[0020] In a possible implementation manner of the first aspect, K=4, and a shape of each of the K third features is [C, h / 2, w / 2].

[0021] In a possible implementation of the first aspect, K=3, the shapes of two of the K third features are [C, h / 2, w / 2], and the shape of one of the K third features is [2C, h / 2, w / 2]. This application does not limit the third feature with a channel number of 2C to the number of the K third features. It can be the first third feature, the second third feature, or the third third feature.

[0022] In a possible implementation of the first aspect, encoding or decoding the image based on the K third features includes: for k=1, 2, ..., K, using a context model to predict the kth part of the latent variable of the image based on the kth third feature respectively; encoding or decoding the image based on the K parts of the latent variable. Specifically, on the encoding side, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, and then the residual of the latent variable is obtained by taking the difference between the original latent variable and the predicted latent variable, and then the residual of the latent variable is entropy encoded to obtain a bitstream; and on the decoding side, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, the bitstream is entropy decoded to obtain the residual of the latent variable, and then the predicted latent variable and the residual of the decoded latent variable are summed to obtain the latent variable, and then the latent variable is decoded by the decoder to obtain a reconstructed image.

[0023] In a possible implementation of the first aspect, the first feature represents a luminance component and / or a chrominance component of an image. When the luminance component and chrominance component of an image are represented by different first features, the combinations of the third features corresponding to the luminance component and chrominance component of the image may be the same or different.

[0024] When the method of the first aspect is implemented by an encoder, that is, when the method of the first aspect is an encoding method, the method further includes encoding a first syntax element, where the first syntax element is used to indicate the combination method adopted by the third feature.

[0025] When the method of the first aspect is implemented by a decoder, that is, when the method of the first aspect is a decoding method, the method further includes decoding a first syntax element, where the first syntax element is used to indicate the combination method adopted by the third feature.

[0026] In a possible implementation manner, the first syntax element is encoded in a picture header, a model header, or a tools header of a code stream.

[0027] In a possible implementation manner of the first aspect, the encoding side and the decoding side agree that the combination manner adopted by the third feature is a specific combination manner in Table 1, and then the first syntax element does not need to be transmitted.

[0028] In a second aspect, the present application provides an image encoding device, characterized in that the device includes: an acquisition module for acquiring a first feature of an image, the shape of the first feature is [C, h, w], wherein C represents the number of channels of the first feature, h represents the length of the first feature, w represents the width of the first feature, C, h and w are all positive integer multiples of 2, each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, and each of the pixel blocks includes 4 pixels in the upper left, lower right, upper right and lower left; a processing module for reordering the first feature to obtain a second feature, the shape of the second feature is [4C, h / 2, w / 2], the second feature includes C sub-features, and the shape of each sub-feature is [4, h / 2, w / 2], the cth sub-feature of the second feature corresponds to the cth channel of the first feature, c is a positive integer and satisfies 1≤c≤C, and the four channels in the cth sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all the pixel blocks of the cth channel in the first feature; the processing module is further used to recombine the second features to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes one channel for each sub-feature in the second feature, K is a positive integer greater than 2, k is a positive integer and satisfies 1≤k≤K; the encoding module is used to encode the image based on the K third features.

[0029] In a possible implementation of the second aspect, the positions of N pixels at the same position in the N channels in the kth third feature are different from each other in the N channels in the first feature, where N is a positive integer and 2≤N≤C.

[0030] In a possible implementation of the second aspect, C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of sub-features in the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of sub-features in the second feature, where i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j, the first part of sub-features and the second part of sub-features each include C / 2 sub-features in the second feature, and the first part of sub-features and the second part of sub-features together constitute the second feature.

[0031] In one embodiment of this implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features. When C=2, the first third feature is composed of the first channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; the second third feature is composed of the second channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; the third third feature is composed of the third channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature; the fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature.

[0032] In another embodiment of this implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of sub-features.

[0033] In another embodiment of this implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of sub-features.

[0034] In another embodiment of this implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features.

[0035] In another embodiment of this implementation, K=3, in which case the shapes of the first and second third features are [C, h / 2, w / 2], and the shape of the third feature is [2C, h / 2, w / 2]; the C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; the C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; the C channels in the third third feature are composed of the third and fourth channels of each sub-feature in the first part of sub-features, and the remaining C channels in the third third feature are composed of the first and second channels of each sub-feature in the second part of sub-features.

[0036] In a possible implementation manner of the second aspect, K=4, and a shape of each of the K third features is [C, h / 2, w / 2].

[0037] In a possible implementation of the second aspect, K=3, shapes of two of the K third features are [C, h / 2, w / 2], and shape of one of the K third features is [2C, h / 2, w / 2].

[0038] In a possible implementation manner of the second aspect, encoding the image based on the K third features includes:

[0039] For k=1, 2, ..., K, based on the kth third feature, a context model is used to predict the kth part of the latent variable of the image; and the image is encoded based on the K parts of the latent variable. Specifically, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, and the residual of the latent variable is obtained by subtracting the original latent variable from the predicted latent variable. The residual of the latent variable is then entropy coded to obtain a bitstream.

[0040] In a possible implementation manner of the second aspect, the first feature represents a luminance component and / or a chrominance component of the image.

[0041] In a possible implementation of the second aspect, the method further includes encoding a first syntax element, where the first syntax element is used to indicate a combination method used by the third feature. The first syntax element is encoded in a picture header, a model header, or a tools header of a codestream.

[0042] The technical effects of the second aspect and any implementation method of the second aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0043] In a third aspect, the present application provides an image decoding device, characterized in that the device includes: an acquisition module for acquiring a first feature of an image, the shape of the first feature is [C, h, w], wherein C represents the number of channels of the first feature, h represents the length of the first feature, w represents the width of the first feature, C, h and w are all positive integer multiples of 2, each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, and each of the pixel blocks includes 4 pixels in the upper left, lower right, upper right and lower left; a processing module for reordering the first feature to obtain a second feature, the shape of the second feature is [4C, h / 2, w / 2], the second feature includes C sub-features, and the shape of each sub-feature is [ 4,h / 2,w / 2], the cth sub-feature of the second feature corresponds to the cth channel of the first feature, c is a positive integer and satisfies 1≤c≤C, and the four channels in the cth sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all the pixel blocks of the cth channel in the first feature; the processing module is further used to recombine the second features to obtain K third features, wherein the shape of the kth third feature is [C,h / 2,w / 2], the kth third feature includes a channel for each sub-feature in the second feature, K is a positive integer greater than 2, k is a positive integer and satisfies 1≤k≤K; a decoding module is used to decode based on the K third features to obtain a reconstructed image.

[0044] In a possible implementation of the third aspect, the positions of N pixels at the same position in the N channels in the kth third feature are different from each other in the N channels in the first feature, where N is a positive integer and 2≤N≤C.

[0045] In a possible implementation of the third aspect, C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of sub-features in the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of sub-features in the second feature, where i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j, the first part of sub-features and the second part of sub-features each include C / 2 sub-features in the second feature, and the first part of sub-features and the second part of sub-features together constitute the second feature.

[0046] In one embodiment of this implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features. When C=2, the first third feature is composed of the first channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; the second third feature is composed of the second channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; the third third feature is composed of the third channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature; the fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature.

[0047] In another embodiment of this implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of sub-features.

[0048] In another embodiment of this implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of sub-features.

[0049] In another embodiment of this implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features.

[0050] In another embodiment of this implementation, K=3, in which case the shapes of the first and second third features are [C, h / 2, w / 2], and the shape of the third feature is [2C, h / 2, w / 2]; the C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; the C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; the C channels in the third third feature are composed of the third and fourth channels of each sub-feature in the first part of sub-features, and the remaining C channels in the third third feature are composed of the first and second channels of each sub-feature in the second part of sub-features.

[0051] In a possible implementation manner of the third aspect, K=4, and a shape of each of the K third features is [C, h / 2, w / 2].

[0052] In a possible implementation of the third aspect, K=3, shapes of two of the K third features are [C, h / 2, w / 2], and shape of one of the K third features is [2C, h / 2, w / 2].

[0053] In a possible implementation manner of the third aspect, performing decoding based on the K third features includes:

[0054] For k=1, 2, ..., K, based on the kth third feature, a context model is used to predict the kth part of the latent variable of the image; encoding is performed based on the K parts of the latent variable to obtain a reconstructed image. Specifically, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, the bitstream can be entropy decoded to obtain a residual of the latent variable, the predicted latent variable and the residual of the decoded latent variable can be summed to obtain a latent variable, and the latent variable can be decoded by a decoder to obtain the reconstructed image.

[0055] In a possible implementation manner of the third aspect, the first feature represents a luminance component and / or a chrominance component of the image.

[0056] In a possible implementation of the third aspect, the method further includes decoding a first syntax element, where the first syntax element is used to indicate a combination method used for the third feature. The first syntax element is encoded in a picture header, a model header, or a tools header of a codestream.

[0057] The technical effects of the third aspect and any implementation method of the third aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0058] In a fourth aspect, the present application provides an encoding or decoding device, characterized in that it includes: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the encoding or decoding device executes the method described in the first aspect and any one of the implementation methods of the first aspect.

[0059] The technical effects of the fourth aspect and any implementation method of the fourth aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0060] In a fifth aspect, the present application provides a coding and decoding system, characterized in that the coding and decoding system includes the coding device as described in the second aspect and any one of the implementations of the second aspect, and / or the decoding device as described in the third aspect and any one of the implementations of the third aspect.

[0061] The technical effects of the fifth aspect and any implementation method of the fifth aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0062] In a sixth aspect, the present application provides a computer-readable storage medium, characterized in that the medium stores computer-readable instructions, and when the computer-readable instructions are executed by a computer, the method described in the above-mentioned first aspect and any one of the implementation methods of the first aspect is implemented.

[0063] The technical effects of the sixth aspect and any implementation method of the sixth aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0064] In a seventh aspect, the present application provides a computer program product, characterized in that it includes computer-readable instructions, which, when the computer-readable instructions are run on a computer, enable the computer to execute the method described in the above-mentioned first aspect and any one of the implementation methods of the first aspect.

[0065] The technical effects of the seventh aspect and any implementation method of the seventh aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0066] In an eighth aspect, the present application provides a computer-readable storage medium, characterized in that the computer-readable medium stores a code stream encoded according to the encoding method described in the first aspect and any one of the implementations of the first aspect.

[0067] The technical effects of the eighth aspect and any implementation method of the eighth aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0068] In a ninth aspect, the present application provides a device for storing a code stream, characterized in that it includes at least one storage medium and a communication interface; the communication interface is used to receive or send a code stream; the at least one storage medium is used to store the code stream; the code stream is encoded by an encoder according to the encoding method described in the first aspect and any one of the implementations of the first aspect.

[0069] The technical effects of the ninth aspect and any implementation method of the ninth aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0070] In a tenth aspect, the present application provides a method for storing a code stream, characterized in that it includes: receiving a code stream through a communication interface; storing the code stream in one or more storage media, wherein the code stream is encoded by an encoder according to the encoding method described in the first aspect and any implementation of the first aspect.

[0071] The technical effects of the tenth aspect and any implementation method of the tenth aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0072] In an eleventh aspect, the present application provides a system for distributing a code stream, characterized in that it includes at least one storage medium and a video streaming device; the at least one storage medium is used to store the code stream, and the code stream is encoded by the encoder according to the encoding method in the first aspect and any one of the implementations of the first aspect; the video streaming device is used to respond to a request from a decoder so that the code stream in the at least one storage medium is sent to the decoder.

[0073] The technical effects of the eleventh aspect and any implementation method of the eleventh aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0074] In a twelfth aspect, the present application provides a method for distributing a code stream, characterized in that it includes: receiving a first request; selecting a code stream from at least one storage medium in response to the first request; sending the code stream to a destination device; the at least one storage medium is used to store the code stream, and the code stream is encoded by an encoder according to the encoding method in the first aspect and any one of the implementations of the first aspect.

[0075] The technical effects of the twelfth aspect and any implementation method of the twelfth aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0076] In a thirteenth aspect, the present application provides a system for processing a code stream, characterized in that it includes an image source device, an encoder device, one or more storage media and a destination device; the image source device is used to provide image data; the encoder device is used to obtain the image data of the image source device through an interface, and encode the image data to obtain one or more code streams, wherein the code stream is encoded by the encoder according to the encoding method in the first aspect and any one of the implementations of the first aspect; the encoder device is used to store the one or more code streams in one or more storage media; or the encoder device is used to encapsulate the one or more code streams to obtain a transmission code stream; the encoder device is used to transmit the transmission code stream to the destination device via a communication link or a communication network; the destination device is used to decapsulate the transmission code stream to obtain the one or more code streams; and the destination device is used to decode the one or more code streams to obtain decoded data.

[0077] The technical effects of the thirteenth aspect and any implementation method of the thirteenth aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] FIG1a is a schematic diagram of an application scenario of an embodiment of the present application;

[0079] FIG1b is a schematic diagram of an application scenario of an embodiment of the present application;

[0080] FIG1c is a schematic diagram of an application scenario of an embodiment of the present application;

[0081] FIG2a is a schematic diagram of an AI encoding unit and an AI decoding unit according to an embodiment of the present application;

[0082] FIG2 b is a chip hardware structure provided in an embodiment of the present application;

[0083] FIG3 is a flow chart of an encoding or decoding method provided in an embodiment of the present application;

[0084] FIG4a is a schematic diagram of an encoding or decoding method provided in an embodiment of the present application;

[0085] FIG4 b is a schematic diagram of an encoding or decoding method provided in an embodiment of the present application;

[0086] FIG4c is a schematic diagram of an encoding or decoding method provided in an embodiment of the present application;

[0087] FIG4 d is a schematic diagram of an encoding or decoding method provided in an embodiment of the present application;

[0088] FIG4e is a schematic diagram of an encoding or decoding method provided in an embodiment of the present application;

[0089] FIG4f is a schematic diagram of an encoding or decoding method provided in an embodiment of the present application;

[0090] FIG5 is a schematic structural diagram of an image encoding device provided in an embodiment of the present application;

[0091] FIG6 is a schematic structural diagram of an image decoding device provided in an embodiment of the present application;

[0092] FIG7 is a schematic diagram of a structure of a device provided in an embodiment of the present application;

[0093] FIG8 is a schematic diagram of a structure of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0094] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0095] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0096] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0097] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0098] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0099] To facilitate understanding, some basic concepts involved in this application are introduced below:

[0100] □Tensor: A high-dimensional matrix. A 3D tensor refers to a tensor divided into three dimensions (number of channels, height, width). The shape (shape) or size (size) of a tensor is usually expressed as [C,H,W], where C represents the number of channels, H represents the height, and W represents the width.

[0101] Hidden variables: A tensor formed by the encoder transform, which can be restored to the original data or data similar to the original data by the decoder transform. It is usually a 3D tensor, with each dimension representing the length, width, and number of channels.

[0102] □Entropy coding: The process of converting data into a string of 0 / 1.

[0103] □ Entropy decoding: The process of restoring a 0 / 1 string to data, which is the inverse process of encoding.

[0104] □Encoder: A model that transforms data (images, videos, etc.) into latent variables.

[0105] □Decoder: A model that restores latent variables to original data.

[0106] □Super encoder: A model that further encodes latent variables into super latent variables.

[0107] □Superdecoder: A model that decodes super latent variables and restores them to latent variables, including super decoding prediction model and super scale decoding model.

[0108] □Superscale decoding model: A model that restores super latent variables to scale variables. The scale variables are used as conditional inputs for entropy decoding and output residuals. The residuals are combined with the super decoder output and the context model output to obtain latent variables.

[0109] □Context model: Divide the latent variable recovery into several parts, each part outputs a part of the latent variable, and the result of each part is determined by the super decoder output, the super scale decoding model output, and the partial results of the latent variables predicted before.

[0110] Pixel shuffle (also known as down-shuffle): A simple operator that rearranges the positions of three-dimensional tensor elements such as latent variables, halving the length and width and increasing the number of channels to four times the original number.

[0111] Since the embodiments of the present application involve the application of a large number of neural networks, for ease of understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0112] (1) Neural Network

[0113] A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0114] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of Xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0115] (2) Deep Neural Networks

[0116] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. Based on the location of the different layers, the neural network within a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. Each layer is fully connected, meaning that any neuron in layer i is connected to any neuron in layer i+1.

[0117] Although DNN looks complicated, the work of each layer is actually not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0118] In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as

[0119] It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0120] (3) Convolutional Neural Networks

[0121] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. It consists of a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each of which can be composed of a rectangular arrangement of neurons. Neurons in the same feature plane share weights, which are referred to as convolution kernels. Shared weights can be understood as extracting image information in a position-independent manner. Convolution kernels can be initialized as matrices of random size, and during CNN training, they can learn to acquire reasonable weights. Furthermore, shared weights have the direct benefit of reducing the number of connections between layers of the CNN, thereby reducing the risk of overfitting.

[0122] (4) Concat

[0123] The concatenation operation refers to concatenating multiple tensors in a certain dimension. All tensors must have the same shape (except for the concatenation dimension). This patent refers to concatenation on channels.

[0124] (5) Bit rate: In image compression tasks, the average coding length required to encode a unit pixel.

[0125] (6) Rate-distortion performance: An indicator used to measure the performance of a compression algorithm, which comprehensively considers both the bit rate and the distortion of the decoded image.

[0126] The following describes the application scenarios to which the method provided in the embodiments of the present application is applicable.

[0127] The method provided in the embodiments of the present application can be applied to business scenarios such as public cloud, private cloud, video surveillance, live broadcast technology, storage, transmission, etc. The method provided in the embodiments of the present application can be implemented as an image file type or storage standard, or an image data transmission protocol, or a compression software or storage device.

[0128] Figure 1a illustrates an example application scenario provided by an embodiment of the present application. This application scenario can also be understood as a codec application scenario. This application scenario includes an AI encoding unit, a storage / transmission unit, and an AI decoding unit. The method provided by an embodiment of the present application can be applied in the AI ​​encoding unit / AI decoding unit.

[0129] The AI ​​encoding unit is used to encode input data (such as images, videos, etc.) according to the encoding method provided in the embodiment of the present application.

[0130] The storage / transmission unit is used to store (terminal, video surveillance, public cloud, private cloud, etc.) or transmit (public cloud, private cloud, live broadcast technology, etc.) the encoded input data.

[0131] The AI ​​decoding unit is used to decode the encoded input data according to the decoding method provided in the embodiment of the present application to obtain reconstructed data (which can also be understood as data after reconstructing the encoding features of the input data).

[0132] The application scenarios mentioned above have multiple situations, which are described in detail below:

[0133] The first one is the storage application scenario.

[0134] This storage application scenario includes: terminal photo album, video surveillance, etc. The schematic diagram of this scenario is shown in Figure 1b, which includes: AI encoding unit, file saving unit, file loading unit and AI decoding unit.

[0135] The AI ​​encoding unit is used to encode input data (such as images, videos, etc.) according to the encoding method provided in the embodiment of the present application to obtain a binary file.

[0136] The file saving unit is used to save the binary file output by the AI ​​encoding unit.

[0137] The loader file unit is used to load binary files.

[0138] The AI ​​decoding unit is used to decode the loaded binary file according to the decoding method provided in the embodiment of the present application to obtain reconstructed data (which can also be understood as data after reconstructing the encoding features of the input data).

[0139] The second type is the terminal-cloud interaction scenario.

[0140] The terminal-cloud interaction scenarios include: live broadcast scenarios, public cloud, etc. The schematic diagram of this scenario is shown in Figure 1c, which includes: encoding unit, decoding unit, AI encoding unit, AI decoding unit, encoding unit, and decoding unit.

[0141] Among them, the terminal side includes an encoding unit and a decoding unit, and the cloud side includes a decoding unit, an encoding unit, an AI encoding unit, and an AI decoding unit.

[0142] The terminal obtains input data and encodes the input data through the encoding unit to obtain encoding features. The terminal sends the encoding features to the cloud. After the cloud receives the encoding features, it decodes the encoding features through the decoding unit to obtain reconstructed data. The reconstructed data is then encoded by the AI ​​encoding unit according to the encoding method provided in the embodiment of the present application to obtain a code stream / file. The cloud decodes the code stream / file through the AI ​​decoding unit according to the decoding method provided in the embodiment of the present application to obtain reconstructed data. The cloud encodes the reconstructed data through the encoding unit to obtain encoding features. And sends the encoding features to the terminal. After the terminal receives the encoding features, it decodes the encoding features through the decoding unit to obtain reconstructed data.

[0143] It is understandable that the encoding methods used by the encoding unit and the AI ​​encoding unit in Figure 1c may be inconsistent (or it can be understood that the operation of the decoding unit and the AI ​​encoding unit in the cloud can be called a transcoding operation, and the operation of the AI ​​decoding unit and the encoding unit in the cloud can be called a transcoding operation). For example, the cloud usually has more computing power resources than the terminal, so an algorithm with high computing power requirements and higher compression efficiency (AI encoding) can be used to re-compress the input data (transcoding) to obtain a higher compression multiple and save cloud storage space. The process of transcoding: decode first, then encode in another way.

[0144] It is understandable that the above scenarios are just examples, and there may be other scenarios in actual applications, which are not specifically limited here.

[0145] The following describes an AI encoding unit and an AI decoding unit provided by an embodiment of the present application in conjunction with Figure 2a. This architecture can be used in the AI ​​encoding unit / AI decoding unit shown in Figures 1a to 1c to complete the encoding / decoding process.

[0146] Figure 2a is a schematic diagram of the AI ​​encoding unit and AI decoding unit of an embodiment of the present application. The AI ​​compression model infrastructure is a two-layer encoder-decoder architecture. Among them, the input data is complex data such as images and videos. The encoder and super encoder output latent variables and super latent variables, and entropy encoding is performed on the latent variables and super latent variables. The decoding process first decodes the super latent variables, then uses the super decoder (including super decoding prediction model, superscale decoding model) to predict the latent variables, and restores the latent variables based on the results of entropy decoding and the context model, and finally uses the decoder to restore the original data.

[0147] Assume the input data is x, the latent variable is y, and the super latent variable is z. The entire encoding process is expressed as follows:

[0148] y=Enc(x)

[0149] z=HEnc(x),bs z =EntropyEncoding(z)

[0150] Among them, Enc, HEnc represent encoder, super encoder, super decoding prediction model HDec and super scale decoding model HSDec constitute super decoder, EntropyEncoding is entropy coding operation, Context is context model, σ is scale variable, It is the output of the super decoding prediction model, which can be called a priori features and will be used as the input of the context model to predict latent variables The output bit stream is bs z ,bs r .

[0151] The decoding process is based on bs z ,bs r Decoding is expressed as follows:

[0152] z=EntropyDecoding(bs z )

[0153] EntropyDecoding is the entropy decoding operation, and Dec is the decoder. The context input here is different from the context on the encoding side, but the operations are equivalent. The principles are explained in detail below.

[0154] Note that in the image compression model, are all 3D tensors with the same shape.

[0155] Among them, the core of the context model is the high-dimensional latent variable The prediction is divided into multiple steps, each time only predicting and make full use of the predicted and the corresponding dimension y (or ), predict the current part of the dimension Since the prediction is made step by step and the predicted information is fully utilized, the context model predicts y more accurately, so the entropy coding residual Fewer coding bits are required, resulting in better coding performance.

[0156] The core of the context model is to The forecast is divided into several parts (correspond The same division is used). For each part The context model formula is as follows:

[0157] in, is is The Context process of the encoding process can be specifically written in the following form:

[0158] where, y<i is [y1,…,y i-1 .

[0159] This application focuses on how to reasonably divide in the field of image compression implementation solutions.

[0160] The following introduces a chip hardware structure provided by an embodiment of this application.

[0161] Figure 2b is a chip hardware structure provided by an embodiment of the present invention. The chip includes a neural network processor 40. The chip can be arranged in the AI encoding unit / AI decoding unit shown in Figures 1a to 1c to complete encoding / decoding processing.

[0162] The neural network processor 40 can be a neural-network processing unit (NPU), a tensor processing unit (TPU), a graphics processing unit (GPU), or any other processor suitable for large-scale exclusive OR operation processing. Taking the NPU as an example: The neural network processor 40 is mounted as a coprocessor on the main central processing unit (CPU) (host CPU), and tasks are assigned by the main CPU. The core part of the NPU is the arithmetic circuit 403, and the controller 404 controls the arithmetic circuit 403 to extract data from the memory (weight memory or input memory) and perform operations.

[0163] In some implementations, the arithmetic circuit 403 internally includes multiple processing units (process engine, PE). In some implementations, the arithmetic circuit 403 is a two-dimensional systolic array. The arithmetic circuit 403 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 403 is a general matrix processor.

[0164] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 403 fetches the corresponding data of matrix B from the weight memory 402 and caches it on each PE in the arithmetic circuit. The arithmetic circuit fetches the data of matrix A from the input memory 401 and performs matrix operations with matrix B, and the partial results or final results of the obtained matrix are saved in the accumulator 408.

[0165] The vector calculation unit 407 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 407 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0166] In some implementations, the vector calculation unit 407 stores the processed output vector to the unified memory 406. For example, the vector calculation unit 407 can apply a nonlinear function to the output of the operation circuit 403, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 407 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 403, for example, for use in a subsequent layer in a neural network.

[0167] The unified memory 406 is used to store input data and output data.

[0168] The weight data is directly transferred from the external memory to the input memory 401 and / or the unified memory 406 through the direct memory access controller (DMAC) 405, the weight data in the external memory is stored in the weight memory 402, and the data in the unified memory 406 is stored in the external memory.

[0169] The bus interface unit (BIU) 410 is used to implement interaction between the main CPU, DMAC and instruction fetch memory 409 through the bus.

[0170] An instruction fetch buffer 409 connected to the controller 404 is used to store instructions used by the controller 404 .

[0171] The controller 404 is used to call the instructions cached in the instruction fetch memory 409 to control the working process of the computing accelerator.

[0172] Generally, the unified memory 406, the input memory 401, the weight memory 402 and the instruction fetch memory 409 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0173] The method provided by the embodiment of the present application is described below. The method can be performed by an encoding device or a decoding device, or by a component of the encoding device or the decoding device (such as a processor, a chip, or a chip system, etc.). The encoding device or the decoding device can be a cloud device or a terminal device. Of course, the method can also be performed by a system consisting of a cloud device and a terminal device. Optionally, the method can be processed by a processor such as a CPU, NPU, or GPU in the encoding device or the decoding device, and this application does not limit it.

[0174] FIG3 is a schematic diagram of an encoding or decoding method 1000 provided in an embodiment of the present application, which includes the following steps:

[0175] 1001, obtaining a first feature of an image, where the shape of the first feature is [C, h, w].

[0176] 1003. Down-shuffle the first feature to obtain a second feature. The second feature has a shape of [4C, h / 2, w / 2]. The second feature includes C sub-features, and each sub-feature has a shape of [4, h / 2, w / 2].

[0177] 1005, recombining (cross-cat) the second features to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2];

[0178] 1007. Encode or decode the image based on the K third features.

[0179] In some possible implementations, step 1007 is an optional step.

[0180] In step 1001, a first feature of the image is obtained. The shape of the first feature is [C, h, w], where C represents the number of channels of the first feature, h represents the length of the first feature, and w represents the width of the first feature. C, h, and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, that is, pixel blocks with (h / 2) rows and (w / 2) columns. Each of the pixel blocks includes 4 pixels in the upper left, lower right, upper right, and lower left.

[0181] In this application, a pixel block may also be referred to as a feature block, and a pixel point may also be referred to as a feature point.

[0182] In one possible implementation, as shown in Figures 4a to 4f, the upper left, lower right, upper right, and lower left pixels of each 2×2 pixel block in the cth channel of the first feature are represented by 0, 1, 2, 3 or 4, 5, 6, and 7, respectively, where c is a positive integer and satisfies 1≤c≤C. That is, the upper left pixel of each pixel block in the same channel is represented by 0 or 4, the lower right pixel of each pixel block in the same channel is represented by 1 or 5, the upper right pixel of each pixel block in the same channel is represented by 2 or 5, and the lower left pixel of each pixel block in the same channel is represented by 4 or 7.

[0183] In one possible implementation, on the encoding side, the encoder encodes the image to obtain latent variables, the super-encoder encodes the latent variables into super-latent variables, and the super-decoder restores the super-latent variables into prior features (i.e., the first feature mentioned above) and scale variables.

[0184] In one possible implementation, on the decoding side, the code stream including the encoded super-latent variable data is entropy decoded to obtain the super-latent variables, and then the super-decoder restores the super-latent variables into prior features (i.e., the first features mentioned above) and scale variables.

[0185] Step 1003: Reorder the first feature to obtain a second feature, where the second feature includes C sub-features, each of which has a shape of [4, h / 2, w / 2]. In one possible implementation, the cth sub-feature of the second feature corresponds to the cth channel of the first feature, where c is a positive integer and satisfies 1≤c≤C, and the four channels in the cth sub-feature are respectively composed of the top left, bottom right, top right, and bottom left pixels in all pixel blocks of the cth channel of the first feature.

[0186] In the examples given in Figures 4a to 4c, and Figure 4e, the number of channels of the first feature is C = 2, the height h = 4, and the width w = 4. The second feature is obtained by reordering the first feature. The second feature includes 2 sub-features, and the shape of each sub-feature is [4, 2, 2]. The first sub-feature of the second feature is composed of the 1st to 4th channels of the second feature. The 4 channels of the first sub-feature are respectively the upper left pixel (0), the lower right pixel (1), the upper right pixel (2), and the lower left pixel in all pixel blocks of the 1st channel of the first feature. (3), all channels of the first sub-feature of the second feature together include all pixels of the first channel of the first feature; the second sub-feature of the second feature is composed of the 5th to 8th channels of the second feature, and the 4 channels of the second sub-feature are respectively composed of the upper left pixel (4), lower right pixel (5), upper right pixel (6) and lower left pixel (7) in all pixel blocks of the second channel of the first feature, and all channels of the second sub-feature of the second feature together include all pixels of the second channel of the first feature.

[0187] In the examples given in Figures 4d and 4f, the number of channels of the first feature is C = 4, the height h = 4, and the width w = 4. The second feature is obtained by reordering the first feature. The second feature includes 4 sub-features, and the shape of each sub-feature is [4, 2, 2]. The first sub-feature of the second feature is composed of the 1st to 4th channels of the second feature. The 4 channels of the 1st sub-feature are respectively composed of the upper left pixel (0), lower right pixel (1), upper right pixel (2), and lower left pixel (3) in all pixel blocks of the 1st channel of the first feature. All channels of the 1st sub-feature of the second feature include all pixels of the 1st channel of the first feature. The second sub-feature of the second feature is composed of the 5th to 8th channels of the second feature. The 4 channels of the 2nd sub-feature are respectively composed of the upper left pixel (4), lower right pixel (5), upper right pixel (6), and lower left pixel (7) in all pixel blocks of the 2nd channel of the first feature. The second sub-feature of the second feature is composed of all the channels of the second channel of the first feature; the third sub-feature of the second feature is composed of the 9th to 12th channels of the second feature, and the four channels of the third sub-feature are respectively composed of the upper left pixel (0), the lower right pixel (1), the upper right pixel (2) and the lower left pixel (3) in all the pixel blocks of the third channel of the first feature, and all the channels of the third sub-feature of the second feature are respectively composed of all the pixels of the third channel of the first feature; the fourth sub-feature of the second feature is composed of the 13th to 16th channels of the second feature, and the four channels of the fourth sub-feature are respectively composed of the upper left pixel (4), the lower right pixel (5), the upper right pixel (6) and the lower left pixel (7) in all the pixel blocks of the fourth channel of the first feature, and all the channels of the fourth sub-feature of the second feature are respectively composed of the upper left pixel (4), the lower right pixel (5), the upper right pixel (6) and the lower left pixel (7) in all the pixel blocks of the fourth channel of the first feature, and all the channels of the fourth sub-feature of the second feature are respectively composed of all the pixels of the fourth channel of the first feature.

[0188] It is worth noting that Figures 4a to 4f only give several specific examples of the solutions provided in this application. This application does not limit the value of C to 2 or 4, and the values ​​of h and w are not limited to 4, as long as C, h and w are all positive integers and integer multiples of 2; this application does not limit the channel arrangement of each third feature in the K third features, as long as the positions of N pixels at the same position in the N channels of the kth third feature are different in the N channels of the first feature, where N is a positive integer and 2≤N≤C; this application does not limit the value of K to 3 or 4, as long as K is a positive integer greater than 2.

[0189] Step 1005: Recombining the second features to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes a channel for each sub-feature in the second feature, K is a positive integer greater than 2, and k is a positive integer that satisfies 1≤k≤K;

[0190] In the examples given in Figures 4a, 4b, 4d, and 4f, the shape of each of the K third features is the same, which is [C, h / 2, w / 2]. Specifically, in the examples given in Figures 4a, 4b, and 4e, the shape of each third feature is [2, 2, 2], and in the examples given in Figures 4d and 4f, the shape of each third feature is [4, 2, 2]. However, the present application does not limit the shape of each of the K third features to the same. For example, in the example given in Figure 4c, the shapes of the first and second third features are the same, which are [2, 2, 2], and the shape of the last third feature is different from the shapes of the first and second third features, which is [4, 2, 2]. In the method provided in the present application, the number of channels of each of the K third features may be the same or different, but it is required that the sum of the number of channels of all the third features of the K third features is equal to 4 times C.

[0191] In a possible implementation of step 1005, the third feature obtained after recombining satisfies the following conditions: the positions of N pixels at the same position in the N channels of the kth third feature are different from each other in the N channels of the first feature, where N is a positive integer and 2≤N≤C, where the position of the pixel can be represented by the spatial coordinates [x,y], x represents the horizontal coordinate offset of the pixel relative to the upper left corner of the image, and y represents the vertical coordinate offset of the pixel relative to the upper left corner of the image; the position of the pixel can also be indicated by the symbol representing the pixel, for example, in the examples given in Figures 4a to 4d, the upper left, lower right, upper right and lower left pixels of each 2×2 pixel block in the cth channel of the first feature are represented by 0, 1, 2, 3 or 4, 5, 6, 7, respectively, where c is a positive integer and satisfies 1≤c≤C. That is, the upper left pixel of each pixel block in the same channel is represented by 0 or 4, the lower right pixel of each pixel block in the same channel is represented by 1 or 5, the upper right pixel of each pixel block in the same channel is represented by 2 or 5, and the lower left pixel of each pixel block in the same channel is represented by 4 or 7. The positions of the N pixels at the same position in the N channels of the kth third feature are different from each other in the N channels of the first feature, which means that the pixels included in the N channels exclude the following situations: (1) all are the upper left pixel points in the 2×2 pixel blocks in the first feature; (2) all are the lower right pixel points in the 2×2 pixel blocks in the first feature; (3) all are the upper right pixel points in the 2×2 pixel blocks in the first feature; (4) all are the lower left pixel points in the 2×2 pixel blocks in the first feature. It is worth noting that this application only limits at least one third feature among all K third features to satisfy the above conditions, and does not require all K third features to satisfy the above conditions. For example, in the examples given in Figures 4a to 4d, all K third features satisfy the above conditions, but in the example given in Figure 4e, only the last two third features satisfy the above conditions, and in the example given in Figure 4f, only the first and third third features satisfy the above conditions.

[0192] When K=4, the kth third feature is composed of the 8c+dth in the second feature. 1k +1,8c+d 2k +1,c=0,…,C / 2-1 channels.

[0193] d 1k ,d 2k There are many combinations of (as shown in Table 1 below), and the only requirement is that the combination is not (0,4)→(1,5)→(2,6)→(3,7).

[0194] Table 1 Possible combinations of the third feature when K=4

[0195] In the example given in Figure 4a, the first combination in Table 1 is used, namely (0,6)→(1,7)→(2,4)→(3,5);

[0196] In the example given in FIG4 b , the fifth combination in Table 1 is adopted, namely (0,7)→(1,6)→(2,5)→(3,4).

[0197] It is worth noting that this application does not list all combinations. This application only limits the combination to not (0,4)→(1,5)→(2,6)→(3,7). In other words, it excludes the situation where each combination of the third feature is composed of pixels with the same position in the pixel block of the first feature.

[0198] Figures 4a and 4b show examples of C = 2. When C = 4, in the example shown in Figure 4d, d 1k ,d 2k The combination of still adopts the first combination in Table 1, that is, (0,6)→(1,7)→(2,4)→(3,5). At this time, the number of channels of each third feature is also 4. The first third feature is composed of the 1st, 7th, 9th, and 15th channels of the second feature. The second third feature is composed of the 2nd, 8th, 10th, and 16th channels of the second feature. The third third feature is composed of the 3rd, 5th, 11th, and 13th channels of the second feature. The fourth third feature is composed of the 4th, 6th, 12th, and 14th channels of the second feature.

[0199] It should be noted that in this application, the channel numbering starts from 1 and increases. In some implementations, the channel numbering can also start from 0 and increase. When the channel numbering starts from 0 and increases, all channel numbers mentioned in this application need to be subtracted by 1.

[0200] In one possible implementation of step 1005, C / 2 channels in the kth third feature are composed of the i-th channel of each subfeature in the first portion of subfeatures in the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each subfeature in the second portion of subfeatures in the second feature, where i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j. The first portion of subfeatures and the second portion of subfeatures each include C / 2 subfeatures in the second feature, and the first portion of subfeatures and the second portion of subfeatures together constitute the second feature. When K=4 and C=2, possible combinations of this implementation are listed in Table 1. The first, second, third, and fourth channels of any subfeature in the second feature are composed, in order, of the top left, bottom right, top right, and bottom left pixels in all 2×2 pixel blocks in the same channel of the first feature. Therefore, by limiting i≠j, it can be ensured that the C / 2 channels and the remaining C / 2 channels in the kth third feature are composed of pixel points at different positions in the pixel block in the first feature, thereby ensuring that the division method can make full use of the pixel information adjacent to the same channel and the pixel information at the same position of the surrounding channel when the context model predicts the latent variable, so that the latent variable predicted by the context model is more accurate, the number of bits required to transmit the residual of the latent variable is reduced, and the bit rate is reduced. It is worth noting that this application only limits at least one third feature among all K third features to meet the above conditions, and does not require all K third features to meet the above conditions. For example, in the examples given in Figures 4a to 4d, all K third features meet the above conditions, but in the example given in Figure 4e, only the last two third features meet the above conditions, and in the example given in Figure 4f, only the first third feature and the third third feature meet the above conditions.

[0201] In a possible embodiment of the above implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features. If C=2, it is the example given in Figure 4a, that is, the first third feature is composed of the first channel of the first subfeature of the second feature and the third channel of the second subfeature of the second feature; the second third feature is composed of the second channel of the first subfeature of the second feature and the fourth channel of the second subfeature of the second feature; the third third feature is composed of the third channel of the first subfeature of the second feature and the first channel of the second subfeature of the second feature; the fourth third feature is composed of the fourth channel of the first subfeature of the second feature and the second channel of the second subfeature of the second feature.

[0202] In a possible embodiment of the above implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of sub-features. If C=2, it is the example given in Figure 4b, that is, the first third feature is composed of the first channel of the first subfeature of the second feature and the fourth channel of the second subfeature of the second feature; the second third feature is composed of the second channel of the first subfeature of the second feature and the third channel of the second subfeature of the second feature; the third third feature is composed of the third channel of the first subfeature of the second feature and the second channel of the second subfeature of the second feature; the fourth third feature is composed of the fourth channel of the first subfeature of the second feature and the first channel of the second subfeature of the second feature.

[0203] In a possible embodiment of the above implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of sub-features.

[0204] In a possible embodiment of the above implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features.

[0205] In a possible embodiment of the above implementation, K=3. In this case, the shapes of the first and second third features are [C, h / 2, w / 2], and the shape of the third feature is [2C, h / 2, w / 2]; the C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; the C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; the C channels in the third third feature are composed of the third and fourth channels of each sub-feature in the first part of sub-features, and the remaining C channels in the third third feature are composed of the first and second channels of each sub-feature in the second part of sub-features.

[0206] In a possible embodiment of the above implementation, K=3. In this case, the shapes of the first and third third features are [C, h / 2, w / 2], and the shape of the second third feature is [2C, h / 2, w / 2]; the C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; the C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the second part of sub-features; the C channels in the second third feature are composed of the third and fourth channels of each sub-feature in the first part of sub-features, and the remaining C channels in the second third feature are composed of the first and second channels of each sub-feature in the second part of sub-features.

[0207] In a possible embodiment of the above implementation, K=3. In this case, the shapes of the second and third third features are [C, h / 2, w / 2], and the shape of the first third feature is [2C, h / 2, w / 2]; the C / 2 channels in the second third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; the C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the second part of sub-features; the C channels in the first third feature are composed of the third and fourth channels of each sub-feature in the first part of sub-features, and the remaining C channels in the first third feature are composed of the first and second channels of each sub-feature in the second part of sub-features.

[0208] In one possible implementation of step 1005, K=4, and the third features are combined as follows: the first third feature is composed of the 4cth channel of the first sub-feature and the 4c+2th channel of the second sub-feature; the second third feature is composed of the 4c+1th channel of the first sub-feature and the 4c+3th channel of the second sub-feature; the third third feature is composed of the 4c+2th channel of the first sub-feature and the 4cth channel of the second sub-feature; and the fourth third feature is composed of the 4c+3th channel of the first sub-feature and the 4c+1th channel of the second sub-feature. Where c is an integer, and the index starts at 0.

[0209] In a possible implementation of step 1005 , K=4, and the shape of each of the K third features is [C, h / 2, w / 2].

[0210] In a possible implementation of step 1005, K=3, the shapes of two of the K third features are [C, h / 2, w / 2], and the shape of one of the K third features is [2C, h / 2, w / 2]. This application does not limit the third feature with a channel number of 2C to the number of the K third features. It can be the first third feature, the second third feature, or the third third feature.

[0211] Step 1007: Encode or decode the image based on the K third features. In one possible implementation of step 1007, encoding or decoding the image based on the K third features includes: for k=1, 2, ..., K, using a context model to predict the kth part of the latent variable of the image based on the kth third feature; and encoding or decoding the image based on the K parts of the latent variable. Specifically, on the encoding side, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, and then the residual of the latent variable is obtained by subtracting the original latent variable from the predicted latent variable. The residual of the latent variable is then entropy encoded to obtain a bitstream. On the decoding side, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, the bitstream is entropy decoded to obtain the residual of the latent variable, and then the predicted latent variable and the residual of the decoded latent variable are summed to obtain the latent variable. The latent variable is then decoded by the decoder to obtain a reconstructed image.

[0212] In one possible implementation of the present application, the first feature represents the luminance component and / or chrominance component of the image. When the luminance component and chrominance component of the image are represented by different first features, the combination of the third features corresponding to the luminance component and chrominance component of the image can be the same or different.

[0213] When method 1000 is implemented by an encoder, that is, when method 1000 is an encoding method, the method further includes encoding a first syntax element, where the first syntax element is used to indicate the combination method adopted by the third feature.

[0214] When method 1000 is implemented by a decoder, that is, when method 1000 is a decoding method, the method further includes decoding a first syntax element, where the first syntax element is used to indicate a combination method adopted by the third feature.

[0215] In a possible implementation manner, the first syntax element is encoded in a picture header, a model header, or a tools header of a code stream.

[0216] In one possible implementation, the value of the first syntax element corresponds to the combination in Table 1 in the following manner:

[0217] In one possible implementation, the encoding side and the decoding side agree that the combinations of the third feature are several of those in Table 1, then the first syntax element can be transmitted with fewer bits. For example, if there are only four possible combinations of the third feature, then only two bits are needed to transmit the first syntax element.

[0218] In a possible implementation, the encoding side and the decoding side agree that the combination method used by the third feature is a specific combination method in Table 1, and then the first syntax element does not need to be transmitted.

[0219] The above describes the encoding and decoding method in the embodiment of the present application. The following describes the device in the embodiment of the present application. Please refer to Figure 5. An embodiment of the encoding device 1200 in the embodiment of the present application includes: an acquisition module 1201, a processing module 1202 and an encoding module 1203.

[0220] An acquisition module 1201 is configured to acquire a first feature of an image. The shape of the first feature is [C, h, w], where C represents the number of channels of the first feature, h represents the length of the first feature, and w represents the width of the first feature. C, h, and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, and each pixel block includes four pixels, namely, upper left, lower right, upper right, and lower left.

[0221] Among them, the introduction of the acquisition module 1201 can refer to the introduction of step 1001 in the above embodiment, and will not be repeated here.

[0222] Processing module 1202 is used to reorder the first feature to obtain a second feature, where the shape of the second feature is [4C, h / 2, w / 2], the second feature includes C sub-features, each of the sub-features has a shape of [4, h / 2, w / 2], the c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, c is a positive integer and satisfies 1≤c≤C, and the four channels in the c-th sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all the pixel blocks of the c-th channel of the first feature.

[0223] The processing module 1202 is also used to recombine the second features to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes a channel for each sub-feature in the second feature, K is a positive integer greater than 2, k is a positive integer and satisfies 1≤k≤K.

[0224] It is worth noting that the present application does not limit the value of C to 2 or 4, and the values ​​of h and w are not limited to 4, as long as C, h and w are all positive integers and integer multiples of 2; the present application does not limit the channel arrangement of each third feature in the K third features, as long as the positions of N pixels at the same position in the N channels of the kth third feature are different in the N channels of the first feature, where N is a positive integer and 2≤N≤C; the present application does not limit the value of K to 3 or 4, as long as K is a positive integer greater than 2.

[0225] In one possible implementation, the third feature obtained after recombining satisfies the following conditions: the positions of N pixels at the same position in the N channels of the kth third feature are different from each other in the N channels of the first feature, where N is a positive integer and 2≤N≤C, where the position of the pixel can be represented by the spatial coordinates [x,y], x represents the horizontal coordinate offset of the pixel relative to the upper left corner of the image, and y represents the vertical coordinate offset of the pixel relative to the upper left corner of the image; the position of the pixel can also be indicated by the symbol representing the pixel. For example, in the examples given in Figures 4a to 4d, the upper left, lower right, upper right and lower left pixels of each 2×2 pixel block in the cth channel of the first feature are represented by 0, 1, 2, 3 or 4, 5, 6, 7, respectively, where c is a positive integer and satisfies 1≤c≤C. That is, the upper left pixel of each pixel block in the same channel is represented by 0 or 4, the lower right pixel of each pixel block in the same channel is represented by 1 or 5, the upper right pixel of each pixel block in the same channel is represented by 2 or 5, and the lower left pixel of each pixel block in the same channel is represented by 4 or 7. The positions of the N pixels at the same position in the N channels of the kth third feature are different from each other in the N channels of the first feature, which means that the pixels included in the N channels exclude the following situations: (1) all are the upper left pixel points in the 2×2 pixel blocks in the first feature; (2) all are the lower right pixel points in the 2×2 pixel blocks in the first feature; (3) all are the upper right pixel points in the 2×2 pixel blocks in the first feature; (4) all are the lower left pixel points in the 2×2 pixel blocks in the first feature.

[0226] In one possible implementation, C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first portion of sub-features in the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second portion of sub-features in the second feature, where i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j, the first portion of sub-features and the second portion of sub-features each include C / 2 sub-features in the second feature, and the first portion of sub-features and the second portion of sub-features together constitute the second feature. The first, second, third, and fourth channels of any sub-feature in the second feature are composed, in sequence, of the top left, bottom right, top right, and bottom left pixels in all 2×2 pixel blocks in the same channel of the first feature. Therefore, by limiting i≠j, it can be ensured that the C / 2 channels in the kth third feature and the remaining C / 2 channels are composed of pixel points at different positions in the pixel block in the first feature, thereby ensuring that the division method can make full use of the pixel information adjacent to the same channel and the pixel information at the same position in the surrounding channels when the context model predicts the latent variable, so that the latent variable predicted by the context model is more accurate, reducing the number of bits required to transmit the residual of the latent variable and improving the coding efficiency.

[0227] In a possible embodiment of the above implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features. If C=2, it is the example given in Figure 4a, that is, the first third feature is composed of the first channel of the first subfeature of the second feature and the third channel of the second subfeature of the second feature; the second third feature is composed of the second channel of the first subfeature of the second feature and the fourth channel of the second subfeature of the second feature; the third third feature is composed of the third channel of the first subfeature of the second feature and the first channel of the second subfeature of the second feature; the fourth third feature is composed of the fourth channel of the first subfeature of the second feature and the second channel of the second subfeature of the second feature.

[0228] In one possible implementation, K = 4, and the third features are combined as follows: the first third feature is composed of the 4cth channel of the first sub-feature and the 4c+2th channel of the second sub-feature; the second third feature is composed of the 4c+1th channel of the first sub-feature and the 4c+3th channel of the second sub-feature; the third third feature is composed of the 4c+2th channel of the first sub-feature and the 4cth channel of the second sub-feature; and the fourth third feature is composed of the 4c+3th channel of the first sub-feature and the 4c+1th channel of the second sub-feature. Where c is an integer, and the index starts at 0.

[0229] In a possible implementation, K=4, and the shape of each of the K third features is [C, h / 2, w / 2].

[0230] In one possible implementation, K=3, the shapes of two of the K third features are [C, h / 2, w / 2], and the shape of one of the K third features is [2C, h / 2, w / 2]. This application does not limit the third feature with a channel number of 2C to the number of the K third features. It can be the first third feature, the second third feature, or the third third feature.

[0231] For a more detailed introduction to the processing module 1202 , please refer to the introduction to steps 1003 and 1005 in the above embodiment, which will not be repeated here.

[0232] The encoding module 1203 is configured to encode the image based on the K third features.

[0233] In one possible implementation, encoding the image based on the K third features includes: for k=1, 2, ..., K, using a context model to predict a kth portion of a latent variable of the image based on the kth third feature; and encoding the image based on the K portions of the latent variable. Specifically, the K portions of the predicted latent variable can be combined into a complete predicted latent variable, and a residual of the latent variable is obtained by subtracting the original latent variable from the predicted latent variable. The residual of the latent variable is then entropy coded to obtain a bitstream.

[0234] Among them, the introduction of the encoding module 1203 can refer to the introduction of step 1007 in the above embodiment, and will not be repeated here.

[0235] In one possible implementation, the first feature represents the luminance component and / or chrominance component of the image. When the luminance component and chrominance component of the image are represented by different first features, the combination of the third features corresponding to the luminance component and chrominance component of the image can be the same or different.

[0236] Optionally, the encoding module 1203 is further used to encode a first syntax element, where the first syntax element is used to indicate a combination method used by the third feature.

[0237] In a possible implementation manner, the first syntax element is encoded in a picture header, a model header, or a tools header of a code stream.

[0238] Please refer to FIG. 6 . An embodiment of a decoding device 1300 in an embodiment of the present application includes: an acquisition module 1301 , a processing module 1302 and a decoding module 1303 .

[0239] An acquisition module 1301 is configured to acquire a first feature of an image. The shape of the first feature is [C, h, w], where C represents the number of channels of the first feature, h represents the length of the first feature, and w represents the width of the first feature. C, h, and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, and each pixel block includes four pixels, namely, upper left, lower right, upper right, and lower left.

[0240] Among them, the introduction of the acquisition module 1301 can refer to the introduction of step 1001 in the above embodiment, and will not be repeated here.

[0241] A processing module 1302 is configured to reorder the first features to obtain a second feature, where the shape of the second feature is [4C, h / 2, w / 2]. The second feature includes C sub-features, each of which has a shape of [4, h / 2, w / 2]. The c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, where c is a positive integer and satisfies 1≤c≤C. The four channels in the c-th sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all the pixel blocks of the c-th channel of the first feature.

[0242] The processing module 1302 is further used to recombine the second features to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes a channel for each sub-feature in the second feature, K is a positive integer greater than 2, and k is a positive integer and satisfies 1≤k≤K.

[0243] The introduction of the processing module 1302 can refer to the introduction of steps 1003 and 1005 in the above embodiment, and will not be repeated here.

[0244] The decoding module 1303 is configured to perform decoding based on the K third features to obtain a reconstructed image.

[0245] In one possible implementation, decoding the image based on the K third features includes: for k=1, 2, ..., K, using a context model to predict the kth portion of a latent variable of the image based on the kth third feature; and encoding or decoding the image based on the K portions of the latent variable. Specifically, the K portions of the predicted latent variable can be combined into a complete predicted latent variable, the bitstream can be entropy decoded to obtain a residual of the latent variable, the predicted latent variable and the residual of the decoded latent variable can be summed to obtain a latent variable, and the latent variable can be decoded by a decoder to obtain a reconstructed image.

[0246] The introduction of the decoding module 1303 can refer to the introduction of step 1007 in the above embodiment, which will not be repeated here.

[0247] In one possible implementation, the first feature represents the luminance component and / or chrominance component of the image. When the luminance component and chrominance component of the image are represented by different first features, the combination of the third features corresponding to the luminance component and chrominance component of the image can be the same or different.

[0248] Optionally, the decoding module 1303 is further used to decode a first syntax element, where the first syntax element is used to indicate a combination method adopted by the third feature.

[0249] In a possible implementation manner, the first syntax element is encoded in a picture header, a model header, or a tools header of a code stream.

[0250] Next, a device provided in an embodiment of the present application is introduced. The device can be an encoding device or a decoding device. Please refer to Figure 7. Figure 7 is a structural diagram of a device provided in an embodiment of the present application. The device 1400 can be specifically manifested as a virtual reality VR device, a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., which is not limited here. Among them, the device 1400 can be deployed with the device described in the corresponding embodiment of Figure 5 or Figure 6. Specifically, the device 1400 includes: a receiver 1401, a transmitter 1402, a processor 1403 and a memory 1404 (wherein the number of processors 1403 in the device 1400 can be one or more, and Figure 7 takes one processor as an example), wherein the processor 1403 may include an application processor 14031 and a communication processor 14032. In some embodiments of the present application, the receiver 1401, the transmitter 1402, the processor 1403 and the memory 1404 can be connected via a bus or other means.

[0251] Memory 1404 may include read-only memory and random access memory, and provides instructions and data to processor 1403. A portion of memory 1404 may also include non-volatile random access memory (NVRAM). Memory 1404 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0252] Processor 1403 controls the operation of the device. In specific applications, the various components of the device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0253] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1403. Processor 1403 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1403. The above processor 1403 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1403 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1404, and processor 1403 reads the information in memory 1404 and, in conjunction with its hardware, completes the steps of the above method.

[0254] Receiver 1401 can be used to receive input digital or character information and generate signal input related to device settings and function control. Transmitter 1402 can be used to output digital or character information through the first interface. Transmitter 1402 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1402 can also include a display device such as a display screen.

[0255] The present application also provides an embodiment of a device, which can be an encoding device or a decoding device. Please refer to Figure 8, which is a structural diagram of a device provided in an embodiment of the present application. The device 1500 can be deployed with the device described in the corresponding embodiment of Figure 5 or Figure 6. Specifically, the device 1500 is implemented by one or more servers. The device 1500 may have relatively large differences due to different configurations or performance. It may include one or more central processing units (CPUs) 1522 (for example, one or more processors) and memory 1532, and one or more storage media 1530 (for example, one or more mass storage devices) for storing application programs 1542 or data 1544. Among them, the memory 1532 and the storage medium 1530 can be temporary storage or permanent storage. The program stored in the storage medium 1530 may include one or more modules (not shown in the figure), each module may include a series of instruction operations in the device. Furthermore, the central processing unit 1522 can be configured to communicate with the storage medium 1530 to execute a series of instruction operations in the storage medium 1530 on the device 1500.

[0256] The device 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input and output interfaces 1558, and / or one or more operating systems 1541, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0257] An embodiment of the present application also provides a computer program product (or computer program) storing one or more computers. When the computer program product is executed by the processor, the processor executes the method of the possible implementation method of the above-mentioned image processing device.

[0258] An embodiment of the present application also provides a chip system, which includes at least one processor for supporting a terminal device or server to implement the functions involved in the possible implementation of the above-mentioned image processing device. Optionally, the chip system also includes an interface circuit, which provides program instructions and / or data to the at least one processor. In one possible design, the chip system may also include a memory for storing the necessary program instructions and data for the terminal device. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0259] An embodiment of the present application further provides a readable storage medium, which stores a code stream encoded using the method or apparatus in the embodiments shown in FIG. 1 a to FIG. 6 .

[0260] An embodiment of the present application further provides a device for storing a code stream, characterized in that it includes at least one storage medium and a communication interface; the communication interface is used to receive or send the code stream; the at least one storage medium is used to store the code stream; the code stream is encoded by an encoder using the method or apparatus in the embodiments shown in Figures 1a to 6 above.

[0261] An embodiment of the present application further provides a method for storing a code stream, the method comprising: receiving a code stream through a communication interface; and storing the code stream in one or more storage media, wherein the code stream is encoded by an encoder using the method or apparatus in the embodiments shown in FIG. 1 a to FIG. 6 .

[0262] An embodiment of the present application further provides a system for distributing a bitstream, the system comprising at least one storage medium and a video streaming device; the at least one storage medium is used to store the bitstream, where the bitstream is encoded by an encoder using the method or apparatus in the embodiments shown in FIG. 1 a to FIG. 6 ; and the video streaming device is used to respond to a request from a decoder so that the target bitstream in the at least one storage medium is sent to the decoder.

[0263] An embodiment of the present application further provides a method for distributing a bitstream, the method comprising: receiving a first request; selecting a target bitstream from at least one storage medium in response to the first request; and sending the target bitstream to a destination device; the at least one storage medium being used to store the bitstream, where the bitstream is encoded by an encoder using the method or apparatus in the embodiments shown in FIG. 1A to FIG. 6 .

[0264] An embodiment of the present application also provides a system for processing a code stream, the system including an image source device, an encoder device, one or more storage media, and a destination device; the image source device is used to provide image data; the encoder device is used to obtain image data from the image source device through an interface and encode the image data to obtain one or more code streams, where the code streams are encoded by the encoder using the method or apparatus in the embodiments shown in Figures 1a to 6 above; the encoder device is used to store the one or more code streams in one or more storage media; or the encoder device is used to encapsulate the one or more code streams to obtain a transmission code stream; the encoder device is used to transmit the transmission code stream to the destination device via a communication link or communication network; the destination device is used to decapsulate the transmission code stream to obtain one or more code streams; and the destination device is used to decode the one or more code streams to obtain decoded data.

[0265] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0266] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Any of the implementation methods provided in the present application can be combined with each other in any way under the premise of logical coherence. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0267] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0268] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0269] Through the description of the above embodiments, it is clear to those skilled in the art that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course it can also be implemented by means of dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits, or dedicated circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, device, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0270] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0271] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, device or data center to another website, computer, device or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a device integrated with one or more available media, a data center, etc. The available medium can be a magnetic medium, (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive (SSD)).

Claims

1. A method for encoding or decoding an image, characterized in that: include: Acquire a first feature of the image, where the shape of the first feature is [C, h, w], where C represents the number of channels of the first feature, h represents the length of the first feature, and w represents the width of the first feature. C, h, and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, and each of the pixel blocks includes 4 pixels in total, namely, upper left, lower right, upper right, and lower left. Reorder the first feature to obtain a second feature, wherein the shape of the second feature is [4C, h / 2, w / 2], the second feature includes C sub-features, each of the sub-features has a shape of [4, h / 2, w / 2], the c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, c is a positive integer and satisfies 1≤c≤C, and the four channels in the c-th sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all the pixel blocks of the c-th channel of the first feature; Recombining the second features to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes a channel of each sub-feature in the second feature, K is a positive integer greater than 2, k is a positive integer and satisfies 1≤k≤K; The image is encoded or decoded based on the K third features.

2. The method according to claim 1, characterized in that The positions of N pixels at the same position in the N channels in the kth third feature are different from each other in the N channels in the first feature, where N is a positive integer and 2≤N≤C.

3. The method according to claim 1 or 2, characterized in that: The C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of sub-features in the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of sub-features in the second feature, wherein i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j, the first part of sub-features and the second part of sub-features each include C / 2 sub-features in the second feature, and the first part of sub-features and the second part of sub-features together constitute the second feature.

4. The method according to any one of claims 1 to 3, characterized in that K=4, and the shape of each of the K third features is [C, h / 2, w / 2].

5. The method according to claim 1 or 2, characterized in that: K=3, shapes of two third features among the K third features are [C, h / 2, w / 2], and shape of one third feature among the K third features is [2C, h / 2, w / 2].

6. The method according to claim 3, characterized in that K=4, at this time, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features.

7. The method according to claim 6, characterized in that C=2, at this time, The first third feature is composed of the first channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; The second third feature is composed of the second channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; The third third feature is composed of the third channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature; The fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature.

8. The method according to claim 3, characterized in that K=4, at this time, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of sub-features; The C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of sub-features.

9. The method according to claim 3, characterized in that: K=4, at this time, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of sub-features; The C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of sub-features.

10. The method according to claim 3, characterized in that K=4, at this time, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features.

11. The method according to claim 3, characterized in that K=3, in this case, the shapes of the first third feature and the second third feature are [C, h / 2, w / 2], and the shape of the third third feature is [2C, h / 2, w / 2]; C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; The C channels in the third third feature are composed of the 3rd channel and the 4th channel of each sub-feature in the first part of sub-features, and the remaining C channels in the third third feature are composed of the 1st channel and the 2nd channel of each sub-feature in the second part of sub-features.

12. The method according to any one of claims 1 to 11, characterized in that The encoding or decoding of the image based on the K third features includes: For k=1, 2, ..., K, respectively, based on the k-th third feature, using the context model to predict the k-th part of the latent variable of the image; The image is encoded or decoded based on a total of K parts of the latent variables.

13. The method according to any one of claims 1 to 12, characterized in that The first feature represents a luminance component and / or a chrominance component of the image.

14. The method according to any one of claims 1 to 13, characterized in that When the method is a coding method, it also includes: A first syntax element is encoded, where the first syntax element is used to indicate a combination method adopted by the third feature.

15. The method according to any one of claims 1 to 13, characterized in that When the method is a decoding method, it also includes: A first syntax element is decoded, where the first syntax element is used to indicate a combination method adopted by the third feature.

16. An image encoding device, characterized in that: The device comprises: an acquisition module, configured to acquire a first feature of an image, wherein the shape of the first feature is [C, h, w], wherein C represents the number of channels of the first feature, h represents the length of the first feature, and w represents the width of the first feature, and C, h, and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, and each of the pixel blocks includes 4 pixels in total, namely, upper left, lower right, upper right, and lower left. a processing module, configured to reorder the first feature to obtain a second feature, wherein the shape of the second feature is [4C, h / 2, w / 2], the second feature includes C sub-features, each of the sub-features has a shape of [4, h / 2, w / 2], the c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, c is a positive integer and satisfies 1≤c≤C, and the four channels in the c-th sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all the pixel blocks of the c-th channel of the first feature; The processing module is further used to recombine the second features to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes a channel of each sub-feature in the second feature, K is a positive integer greater than 2, and k is a positive integer and satisfies 1≤k≤K; An encoding module is used to encode the image based on the K third features.

17. The encoding device according to claim 16, characterized in that The positions of N pixels at the same position in the N channels in the kth third feature are different from each other in the N channels in the first feature, where N is a positive integer and 2≤N≤C.

18. The encoding device according to claim 16 or 17, characterized in that: The C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of sub-features in the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of sub-features in the second feature, wherein i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j, the first part of sub-features and the second part of sub-features each include C / 2 sub-features in the second feature, and the first part of sub-features and the second part of sub-features together constitute the second feature.

19. The encoding device according to claim 18, characterized in that K=4, at this time, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features.

20. The encoding device according to claim 19, characterized in that C=2, at this time, The first third feature is composed of the first channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; The second third feature is composed of the second channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; The third third feature is composed of the third channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature; The fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature.

21. An image decoding device, characterized in that: The device comprises: an acquisition module, configured to acquire a first feature of an image, wherein the shape of the first feature is [C, h, w], wherein C represents the number of channels of the first feature, h represents the length of the first feature, and w represents the width of the first feature, and C, h, and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, and each of the pixel blocks includes 4 pixels in total, namely, upper left, lower right, upper right, and lower left. a processing module, configured to reorder the first feature to obtain a second feature, wherein the shape of the second feature is [4C, h / 2, w / 2], the second feature includes C sub-features, each of the sub-features has a shape of [4, h / 2, w / 2], the c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, c is a positive integer and satisfies 1≤c≤C, and the four channels in the c-th sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all the pixel blocks of the c-th channel of the first feature; The processing module is further used to recombine the second features to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes a channel of each sub-feature in the second feature, K is a positive integer greater than 2, and k is a positive integer and satisfies 1≤k≤K; A decoding module is used to perform decoding based on the K third features to obtain a reconstructed image.

22. The decoding device according to claim 21, characterized in that The positions of N pixels at the same position in the N channels in the kth third feature are different from each other in the N channels in the first feature, where N is a positive integer and 2≤N≤C.

23. The decoding device according to claim 21 or 22, characterized in that: The C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of sub-features in the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of sub-features in the second feature, wherein i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j, the first part of sub-features and the second part of sub-features each include C / 2 sub-features in the second feature, and the first part of sub-features and the second part of sub-features together constitute the second feature.

24. The decoding device according to claim 23, characterized in that: K=4, at this time, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features.

25. The decoding device according to claim 24, characterized in that: C=2, at this time, The first third feature is composed of the first channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; The second third feature is composed of the second channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; The third third feature is composed of the third channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature; The fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature.

26. An encoding or decoding device, characterized in that include: A processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the encoding or decoding device executes the method according to any one of claims 1-15.

27. A coding and decoding system, characterized in that: The encoding and decoding system comprises an encoding device as described in any one of claims 16-20, and / or a decoding device as described in any one of claims 21-25.

28. A computer-readable storage medium, characterized in that: The medium stores computer instructions, and when the computer instructions are executed by a computer, the method according to any one of claims 1 to 15 is implemented.

29. A computer program product, characterized in that The method comprises computer instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 15.

30. A computer-readable storage medium, characterized in that: The computer-readable medium stores a code stream encoded according to any one of the encoding methods of claims 1-15.

31. A device for storing a code stream, characterized in that: comprising at least one storage medium and a communication interface; The communication interface is used to receive or send code streams; The at least one storage medium is used to store the code stream; The code stream is obtained by encoding by an encoder according to any encoding method in claims 1-15.

32. A method for storing a code stream, characterized in that: include: Receiving a code stream via a communication interface; The code stream is stored in one or more storage media, and the code stream is an encoder Obtained by encoding according to any one of the encoding methods of claims 1-15.

33. A system for distributing code streams, characterized in that: including at least one storage medium and a video streaming device; The at least one storage medium is used to store a code stream, and the code stream is an encoder Obtained by encoding according to any one of the encoding methods of claims 1-15; The video stream device is used to respond to a request of a decoder so that the code stream in the at least one storage medium can be sent to the decoder.

34. A method for distributing code streams, characterized in that: include: receiving a first request; In response to the first request, selecting a code stream from at least one storage medium; Sending the code stream to a destination device; The at least one storage medium is used to store a code stream, and the code stream is an encoder Obtained by encoding according to any one of the encoding methods of claims 1-15.

35. A system for processing a code stream, characterized in that: It includes an image source device, an encoder device, one or more storage media and a destination device; The image source device is used to provide image data; The encoder device is used to obtain the image data of the image source device through an interface, and encode the image data to obtain one or more code streams, wherein the code streams are encoded by the encoder according to any one of the encoding methods of claims 1-15; The encoder device is used to store the one or more code streams in one or more storage media; or The encoder device is used to encapsulate the one or more code streams to obtain a transmission code stream; The encoder device is used to transmit the transmission code stream to the destination device through a communication link or a communication network; The destination device is used to decapsulate the transport code stream to obtain the one or more code streams; The destination device is used to decode the one or more code streams to obtain decoded data.

Images