Coding method and apparatus, and decoding method and apparatus

Abstract

The present application relates to the field of image coding and decoding. Disclosed is an image coding or decoding method, which can be applied to next-generation image or video coding and decoding standards such as JPEG-AI. The method comprises: acquiring a first feature of an image, wherein the shape of the first feature is \[C, h, w]; performing reordering on the first feature, so as to obtain a second feature, wherein the shape of the second feature is \[4C, h / 2, w / 2], the second feature comprises C sub-features, and the shape of each sub-feature is \[4, h / 2, w / 2]; performing recombining on the second feature, so as to obtain K third features, wherein the shape of a kth third feature is \[C, h / 2, w / 2], and the kth third feature comprises one channel of each sub-feature in the second feature, K being a positive integer greater than 2; and coding or decoding the image on the basis of the K third features. The present application can fully utilize adjacent pixel information in the same channel of a hidden variable and pixel information at the same position in neighboring channels, such that the hidden variable predicted by a context model is more accurate, thereby reducing the number of bits required for coding the hidden variable, and reducing the code rate.

Description

An encoding and decoding method and apparatus Technical Field

[0001] This application relates to the field of image or video encoding and decoding, and more particularly to an encoding and decoding method and apparatus. Background Technology

[0002] AI (Artificial Intelligence) image compression algorithms are based on deep learning and offer better compression results compared to traditional image compression technologies such as JPEG (Joint Photographic Experts Group) and BPG (Better Portable Graphics).

[0003] When using AI compression (also known as encoding) and decompression (also known as decoding), the input data is complex data such as images and videos. The encoder and super encoder output latent variables and super latent variables, and entropy encoding is performed on the latent variables and super latent variables. The decoding process first decodes the super latent variables, then uses the super decoder to predict the latent variables, and recovers the latent variables based on the entropy decoding results and the context model. Finally, the decoder is used to recover the original data.

[0004] In AI encoding and decoding, context models are needed to predict latent variables, and the accuracy of the context model's predictions directly affects the overall encoding and decoding performance.

[0005] Summary of the Invention

[0006] This application provides an encoding and decoding method and apparatus. This method can fully utilize pixel information adjacent to latent variables in the same channel and pixel information at the same position in surrounding channels, making context model prediction more accurate, thereby reducing the number of bits required for entropy encoding of latent variables and achieving better encoding performance.

[0007] In a first aspect, embodiments of this application provide an encoding or decoding method, the method comprising: acquiring a first feature of an image (in the entire encoding and decoding process of the image, the first feature can be referred to as a priori feature), the shape of the first feature being [C, h, w], where C represents the number of channels of the first feature, h represents the length of the first feature, w represents the width of the first feature, and C, h, and w are all positive integer multiples of 2, each channel of the first feature comprising (h / 2) × (w / 2) 2×2 pixel blocks, each pixel block comprising four pixels: upper left, lower right, upper right, and lower left; reordering the first feature to obtain a second feature, the shape of the second feature being [4C, h / 2, w / 2], the second feature comprising C sub-features. Each of the sub-features has a shape of [4, h / 2, w / 2]. The c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, where c is a positive integer and satisfies 1 ≤ c ≤ C. The four channels of the c-th sub-feature are respectively composed of the top-left, bottom-right, top-right, and bottom-left pixels of all the pixel blocks of the c-th channel of the first feature. The second feature is recombined to obtain K third features, where the k-th third feature has a shape of [C, h / 2, w / 2]. The k-th third feature includes one channel of each sub-feature of the second feature, where K is a positive integer greater than 2, and k is a positive integer and satisfies 1 ≤ k ≤ K. The image is encoded or decoded based on the K third features.

[0008] In this application, by reordering and recombining prior features (i.e., the first feature), more rationally segmented prior features are obtained. These are then input into the context model to segment and predict latent variables. This allows subsequent segment predictions of latent variables to fully utilize pixel information from adjacent pixels in the same channel and pixels at the same position in surrounding channels within the previously predicted segments, thereby improving the prediction accuracy of latent variables. This, in turn, reduces the number of bits required to encode latent variables, improving coding efficiency. Reducing the number of bits required in the bitstream also saves communication bandwidth for transmission and storage resources for storage, demonstrating corresponding technical benefits in the encoding, transmission, and decoding processes. Processing prior features on the decoding side in the same way as on the encoding side ensures correct decoding.

[0009] In one possible implementation of the first aspect, as shown in Figures 4a to 4d, the top-left, bottom-right, top-right, and bottom-left pixels of each 2×2 pixel block in the c-th channel of the first feature are represented by 0, 1, 2, 3 or 4, 5, 6, 7, respectively, where c is a positive integer and satisfies 1 ≤ c ≤ C. That is, the top-left pixel of each pixel block in the same channel is represented by 0 or 4, the bottom-right pixel of each pixel block in the same channel is represented by 1 or 5, the top-right pixel of each pixel block in the same channel is represented by 2 or 5, and the bottom-left pixel of each pixel block in the same channel is represented by 4 or 7.

[0010] It is worth noting that Figures 4a to 4d only provide several specific examples of the scheme provided in this application. This application does not limit the value of C to 2 or 4, nor does it limit the values ​​of h and w to 4, as long as C, h, and w are all positive integers and multiples of 2. This application also does not limit the channel arrangement of each of the K third features, as long as the positions of the N pixels at the same position in the N channels of the kth third feature are all different in the N channels of the first feature, where N is a positive integer and 2≤N≤C. This application also does not limit the value of K to 3 or 4, as long as K is a positive integer greater than 2.

[0011] In the method provided in this application, the number of channels of each of the K third features may be the same or different, but the total number of channels of all the K third features must be equal to 4 times C.

[0012] In one possible implementation of the first aspect, the third feature obtained after recombination satisfies the following condition: the N pixels at the same position in the N channels of the k-th third feature are all different in the N channels of the first feature, where N is a positive integer and 2≤N≤C, and the position of the pixel can be represented by spatial coordinates [x,y], where x represents the horizontal coordinate offset of the pixel relative to the upper left corner of the image, and y represents the vertical coordinate offset of the pixel relative to the upper left corner of the image; the position of the pixel can also be indicated by the pixel symbol. The N pixels at the same position in the N channels of the k-th third feature are all different in the N channels of the first feature, meaning that the pixels included in the N channels exclude the following cases: (1) all are upper left pixels in the 2×2 pixel block of the first feature; (2) all are lower right pixels in the 2×2 pixel block of the first feature; (3) all are upper right pixels in the 2×2 pixel block of the first feature; (4) all are lower left pixels in the 2×2 pixel block of the first feature. By restricting the N pixels at the same position in the N channels of the recombined k-th third feature to be distinct from the N channels of the first feature, it can be ensured that when using the third feature to predict latent variables, the pixel information at the same position in surrounding channels can be fully utilized (because the pixels at the same position in surrounding channels have already been obtained in the previous segmented prediction). This can improve the prediction accuracy of latent variables, thereby reducing the number of bits required to encode latent variables and lowering the code rate. It should be noted that in this application, it is only necessary to restrict that at least one of the K third features satisfies the above condition; it is not required that all third features satisfy the above condition.

[0013] In one possible implementation of the first aspect, C / 2 channels in the k-th third feature are composed of the i-th channel of each sub-feature in the first part of the second feature, and the remaining C / 2 channels in the k-th third feature are composed of the j-th channel of each sub-feature in the second part of the second feature, where i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j. The first part of the sub-feature and the second part of the sub-feature each include C / 2 sub-features in the second feature, and the first part of the sub-feature and the second part of the sub-feature together constitute the second feature. When K=4 and C=2, the possible combinations of this implementation are listed in Table 1. In this implementation, the 1st, 2nd, 3rd, and 4th channels of any sub-feature in the second feature are sequentially composed of the top-left, bottom-right, top-right, and bottom-left pixels of all 2×2 pixel blocks in the same channel of the first feature. Therefore, by limiting i≠j, it can be ensured that C / 2 channels in the k-th third feature and the remaining C / 2 channels are composed of pixels at different positions in the pixel block of the first feature. This ensures that the partitioning method can fully utilize the pixel information of adjacent pixels in the same channel and the pixel information of the same position in surrounding channels when the context model predicts latent variables, making the latent variables predicted by the context model more accurate, reducing the number of bits required to transmit the residual of the latent variables, and improving coding efficiency. It should be noted that in this application, it is only necessary to limit that at least one of the K third features satisfies the above condition; it is not required that all K third features satisfy the above condition.

[0014] In one possible embodiment of the above implementation, K = 4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of the sub-features. If C = 2, then the example given in Figure 4a is as follows: the first third feature is composed of the first channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; the second third feature is composed of the second channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; the third third feature is composed of the third channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature; and the fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature.

[0015] In one possible embodiment of the above implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of the sub-features. If C = 2, then it is the example given in Figure 4b, that is, the first third feature is composed of the first channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; the second third feature is composed of the second channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; the third third feature is composed of the third channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature; the fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature.

[0016] In one possible embodiment of the above implementation, K = 4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of the sub-features.

[0017] In one possible embodiment of the above implementation, K = 4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of the sub-features.

[0018] In one possible embodiment of the above implementation, K = 3. In this case, the shapes of the first and second third features are [C, h / 2, w / 2], and the shape of the third feature is [2C, h / 2, w / 2]. The C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features. The C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of the sub-features. The C channels in the third third feature are composed of the third and fourth channels of each sub-feature in the first part of the sub-features, and the remaining C channels in the third third feature are composed of the first and second channels of each sub-feature in the second part of the sub-features.

[0019] In one possible implementation of the first aspect, K = 4, and the combination of the third features is as follows: the first third feature consists of the 4c-th channel of the first sub-feature and the 4c+2-th channel of the second sub-feature; the second third feature consists of the 4c+1-th channel of the first sub-feature and the 4c+3-th channel of the second sub-feature; the third third feature consists of the 4c+2-th channel of the first sub-feature and the 4c-th channel of the second sub-feature; and the fourth third feature consists of the 4c+3-th channel of the first sub-feature and the 4c+1-th channel of the second sub-feature. Here, c is an integer, and the indexing starts from 0.

[0020] In one possible implementation of the first aspect, K = 4, and the shape of each of the K third features is [C, h / 2, w / 2].

[0021] In one possible implementation of the first aspect, K = 3, the shape of two of the K third features is [C, h / 2, w / 2], and the shape of one of the K third features is [2C, h / 2, w / 2]. This application does not limit the third feature with 2C channels to be which of the K third features; it can be the first third feature, the second third feature, or the third third feature.

[0022] In one possible implementation of the first aspect, encoding or decoding the image based on the K third features includes: for k = 1, 2, ..., K, predicting the k-th part of the latent variable of the image using a context model based on the k-th third feature; and encoding or decoding the image based on the K parts of the latent variable. Specifically, on the encoding side, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, and the difference between the original latent variable and the predicted latent variable can be calculated to obtain the residual of the latent variable. The residual of the latent variable is then entropy-encoded to obtain the bitstream. On the decoding side, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, and the bitstream can be entropy-decoded to obtain the residual of the latent variable. The residual of the predicted latent variable and the decoded latent variable can be summed to obtain the latent variable, and the latent variable is then decoded by the decoder to obtain the reconstructed image.

[0023] In one possible implementation of the first aspect, the aforementioned first feature represents the luminance component and / or chrominance component of the image. When the luminance component and chrominance component of the image are represented by different first features, the combination of the third features corresponding to the luminance component and chrominance component of the image can be the same or different.

[0024] When the method of the first aspect is implemented by an encoder, that is, when the method of the first aspect is an encoding method, the method further includes encoding a first syntax element, which is used to indicate the combination method adopted by the third feature.

[0025] When the method of the first aspect is implemented by a decoder, that is, when the method of the first aspect is a decoding method, the method further includes decoding a first syntax element, which is used to indicate the combination method adopted by the third feature.

[0026] In one possible implementation, the first syntax element is encoded in the bitstream's picture header, model header, or tools header.

[0027] In one possible implementation of the first aspect, if the encoding and decoding sides agree that the combination method used for the third feature is a specific combination method in Table 1, then the first syntax element does not need to be transmitted.

[0028] Secondly, this application provides an image encoding apparatus, characterized in that the apparatus comprises: an acquisition module, configured to acquire a first feature of an image, the shape of the first feature being [C, h, w], wherein C represents the number of channels of the first feature, h represents the length of the first feature, w represents the width of the first feature, and C, h, and w are all positive integer multiples of 2, each channel of the first feature comprising (h / 2) × (w / 2) 2×2 pixel blocks, each pixel block comprising four pixels: upper left, lower right, upper right, and lower left; and a processing module, configured to reorder the first feature to obtain a second feature, the shape of the second feature being [4C, h / 2, w / 2], the second feature comprising C sub-features, each sub-feature having a shape of... [4, h / 2, w / 2], where the c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, where c is a positive integer and satisfies 1 ≤ c ≤ C, and the four channels of the c-th sub-feature are respectively composed of the top-left, bottom-right, top-right, and bottom-left pixels of all the pixel blocks of the c-th channel in the first feature; the processing module is further configured to recombine the second feature to obtain K third features, wherein the shape of the k-th third feature is [C, h / 2, w / 2], and the k-th third feature includes one channel of each sub-feature of the second feature, where K is a positive integer greater than 2, and k is a positive integer and satisfies 1 ≤ k ≤ K; the encoding module is configured to encode the image based on the K third features.

[0029] In one possible implementation of the second aspect, the N pixels at the same position in the N channels of the kth third feature are all different in the N channels of the first feature, where N is a positive integer and 2≤N≤C.

[0030] In one possible implementation of the second aspect, the C / 2 channels of the kth third feature are composed of the i-th channel of each sub-feature in the first part of the second feature, and the remaining C / 2 channels of the kth third feature are composed of the j-th channel of each sub-feature in the second part of the second feature, where i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j. The first part of the sub-feature and the second part of the sub-feature each include C / 2 sub-features in the second feature, and the first part of the sub-feature and the second part of the sub-feature together constitute the second feature.

[0031] In one embodiment of this implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of the sub-features. When C=2, the first third feature is composed of the first channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; the second third feature is composed of the second channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; the third third feature is composed of the third channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature; the fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature.

[0032] In another embodiment of this implementation, K = 4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of the sub-features.

[0033] In another embodiment of this implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of the sub-features.

[0034] In another embodiment of this implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of the sub-features.

[0035] In another embodiment of this implementation, K = 3. In this case, the shapes of the first and second third features are [C, h / 2, w / 2], and the shape of the third feature is [2C, h / 2, w / 2]. The C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features. The C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of the sub-features. The C channels in the third third feature are composed of the third and fourth channels of each sub-feature in the first part of the sub-features, and the remaining C channels in the third third feature are composed of the first and second channels of each sub-feature in the second part of the sub-features.

[0036] In one possible implementation of the second aspect, K = 4, and the shape of each of the K third features is [C, h / 2, w / 2].

[0037] In one possible implementation of the second aspect, K = 3, the shapes of two of the K third features are [C, h / 2, w / 2], and the shape of one of the K third features is [2C, h / 2, w / 2].

[0038] In one possible implementation of the second aspect, encoding the image based on the K third features includes:

[0039] For k = 1, 2, ..., K, based on the kth third feature, a context model is used to predict the kth part of the latent variable of the image; the image is then encoded based on the K parts of the latent variable. Specifically, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, and the difference between the original latent variable and the predicted latent variable can be used to obtain the residual of the latent variable. Then, the residual of the latent variable is entropy encoded to obtain the bitstream.

[0040] In one possible implementation of the second aspect, the first feature represents the luminance component and / or chrominance component of the image.

[0041] In one possible implementation of the second aspect, a first syntax element is further encoded, which indicates the combination method used by the third feature. The first syntax element is encoded in the picture header, model header, or tools header of the bitstream.

[0042] The technical effects of the second aspect and any implementation thereof are described in the first aspect and its implementation thereof, and will not be repeated here.

[0043] Thirdly, this application provides an image decoding apparatus, characterized in that the apparatus comprises: an acquisition module, configured to acquire a first feature of an image, the shape of the first feature being [C, h, w], wherein C represents the number of channels of the first feature, h represents the length of the first feature, w represents the width of the first feature, and C, h, and w are all positive integer multiples of 2, each channel of the first feature comprising (h / 2) × (w / 2) 2×2 pixel blocks, each pixel block comprising four pixels: upper left, lower right, upper right, and lower left; and a processing module, configured to reorder the first feature to obtain a second feature, the shape of the second feature being [4C, h / 2, w / 2], the second feature comprising C sub-features, each sub-feature having a shape of […]. [4, h / 2, w / 2], where the c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, where c is a positive integer and satisfies 1 ≤ c ≤ C, and the four channels of the c-th sub-feature are respectively composed of the top-left, bottom-right, top-right, and bottom-left pixels of all the pixel blocks of the c-th channel in the first feature; the processing module is further configured to recombine the second feature to obtain K third features, wherein the shape of the k-th third feature is [C, h / 2, w / 2], and the k-th third feature includes one channel of each sub-feature in the second feature, where K is a positive integer greater than 2, and k is a positive integer and satisfies 1 ≤ k ≤ K; the decoding module is configured to decode based on the K third features to obtain a reconstructed image.

[0044] In one possible implementation of the third aspect, the N pixels at the same position in the N channels of the kth third feature are all different in the N channels of the first feature, where N is a positive integer and 2≤N≤C.

[0045] In one possible implementation of the third aspect, the C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of the second feature, where i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j. The first part of the sub-feature and the second part of the sub-feature each include C / 2 sub-features in the second feature, and the first part of the sub-feature and the second part of the sub-feature together constitute the second feature.

[0046] In one embodiment of this implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of the sub-features. When C=2, the first third feature is composed of the first channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; the second third feature is composed of the second channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; the third third feature is composed of the third channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature; the fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature.

[0047] In another embodiment of this implementation, K = 4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of the sub-features.

[0048] In another embodiment of this implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of the sub-features.

[0049] In another embodiment of this implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of the sub-features.

[0050] In another embodiment of this implementation, K = 3. In this case, the shapes of the first and second third features are [C, h / 2, w / 2], and the shape of the third feature is [2C, h / 2, w / 2]. The C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features. The C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of the sub-features. The C channels in the third third feature are composed of the third and fourth channels of each sub-feature in the first part of the sub-features, and the remaining C channels in the third third feature are composed of the first and second channels of each sub-feature in the second part of the sub-features.

[0051] In one possible implementation of the third aspect, K = 4, and the shape of each of the K third features is [C, h / 2, w / 2].

[0052] In one possible implementation of the third aspect, K = 3, the shapes of two of the K third features are [C, h / 2, w / 2], and the shape of one of the K third features is [2C, h / 2, w / 2].

[0053] In one possible implementation of the third aspect, decoding is performed based on the K third features, including:

[0054] For k = 1, 2, ..., K, based on the kth third feature, a context model is used to predict the kth part of the latent variable of the image; the reconstructed image is obtained by encoding the K parts of the latent variable. Specifically, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, and entropy decoding is performed on the bitstream to obtain the residual of the latent variable. Then, the residual of the predicted latent variable and the decoded latent variable are summed to obtain the latent variable, and the latent variable is decoded by the decoder to obtain the reconstructed image.

[0055] In one possible implementation of the third aspect, the first feature represents the luminance component and / or chrominance component of the image.

[0056] In one possible implementation of the third aspect, the method further includes decoding a first syntax element, which indicates the combination method used for the third feature. The first syntax element is encoded in the picture header, model header, or tools header of the bitstream.

[0057] The technical effects of the third aspect and any implementation thereof are described in the first aspect and its implementation thereof, and will not be repeated here.

[0058] Fourthly, this application provides an encoding or decoding device, characterized in that it includes: a processor coupled to a memory for storing programs or instructions, wherein when the program or instructions are executed by the processor, the encoding or decoding device performs the method as described in the first aspect and any implementation thereof.

[0059] The technical effects of the fourth aspect and any implementation thereof are described in the first aspect and its implementation thereof, and will not be repeated here.

[0060] Fifthly, this application provides an encoding and decoding system, characterized in that the encoding and decoding system includes an encoding device as described in the second aspect and any implementation thereof, and / or a decoding device as described in the third aspect and any implementation thereof.

[0061] The technical effects of the fifth aspect and any implementation thereof are described in the first aspect and its implementation thereof, and will not be repeated here.

[0062] In a sixth aspect, this application provides a computer-readable storage medium, characterized in that the medium stores computer-readable instructions, which, when executed by a computer, implement the method described in the first aspect and any implementation thereof.

[0063] The technical effects of the sixth aspect and any implementation thereof are described in the first aspect and its implementation thereof, and will not be repeated here.

[0064] In a seventh aspect, this application provides a computer program product, characterized in that it includes computer-readable instructions, which, when executed on a computer, cause the computer to perform the method described in the first aspect and any implementation thereof.

[0065] The technical effects of the seventh aspect and any implementation thereof are described in the first aspect and its implementation thereof, and will not be repeated here.

[0066] Eighthly, this application provides a computer-readable storage medium, characterized in that the computer-readable medium stores a bitstream encoded according to the encoding method described in the first aspect and any implementation thereof.

[0067] The technical effects of the eighth aspect and any implementation thereof are described in the first aspect and its implementation thereof, and will not be repeated here.

[0068] Ninthly, this application provides a device for storing a bitstream, characterized in that it includes at least one storage medium and a communication interface; the communication interface is used to receive or transmit the bitstream; the at least one storage medium is used to store the bitstream; the bitstream is encoded by an encoder according to the encoding method described in the first aspect and any implementation thereof.

[0069] The technical effects of the ninth aspect and any implementation thereof are described in the first aspect and its implementation thereof, and will not be repeated here.

[0070] In a tenth aspect, this application provides a method for storing a bitstream, characterized in that it includes: receiving a bitstream through a communication interface; storing the bitstream in one or more storage media, wherein the bitstream is encoded by an encoder according to the encoding method described in the first aspect and any implementation thereof.

[0071] The technical effects of the tenth aspect and any implementation thereof are described in the first aspect and its implementation thereof, and will not be repeated here.

[0072] Eleventhly, this application provides a system for distributing bitstreams, characterized in that it includes at least one storage medium and a video streaming device; the at least one storage medium is used to store the bitstream, which is encoded by an encoder according to an encoding method in the first aspect and any implementation thereof; the video streaming device is used to send the bitstream in the at least one storage medium to the decoder in response to a request from the decoder.

[0073] The technical effects of the eleventh aspect and any implementation of the eleventh aspect are described in the first aspect and its implementation, and will not be repeated here.

[0074] In a twelfth aspect, this application provides a method for distributing a bitstream, characterized in that it includes: receiving a first request; in response to the first request, selecting a bitstream from at least one storage medium; sending the bitstream to a destination device; wherein the at least one storage medium is used to store the bitstream, and the bitstream is encoded by an encoder according to an encoding method in the first aspect and any implementation thereof.

[0075] The technical effects of the twelfth aspect and any implementation thereof are described in the first aspect and its implementation thereof, and will not be repeated here.

[0076] In a thirteenth aspect, this application provides a system for processing bitstreams, characterized in that it includes an image source device, an encoder device, one or more storage media, and a destination device; the image source device is used to provide image data; the encoder device is used to acquire the image data from the image source device through an interface and encode the image data to obtain one or more bitstreams, wherein the bitstreams are encoded by the encoder according to the encoding method in the first aspect and any implementation thereof; the encoder device is used to store the one or more bitstreams in one or more storage media; or the encoder device is used to encapsulate the one or more bitstreams to obtain a transmission bitstream; the encoder device is used to transmit the transmission bitstreams to the destination device through a communication link or communication network; the destination device is used to decapsulate the transmission bitstreams to obtain the one or more bitstreams; and the destination device is used to decode the one or more bitstreams to obtain decoded data.

[0077] The technical effects of any implementation of the thirteenth aspect and the thirteenth aspect are described in the first aspect and its implementations above, and will not be repeated here. Attached Figure Description

[0078] Figure 1a is a schematic diagram of an application scenario of an embodiment of this application;

[0079] Figure 1b is a schematic diagram of an application scenario of an embodiment of this application;

[0080] Figure 1c is a schematic diagram of an application scenario of this application embodiment;

[0081] Figure 2a is a schematic diagram of the AI ​​encoding unit and AI decoding unit in an embodiment of this application;

[0082] Figure 2b shows a chip hardware structure provided in an embodiment of this application;

[0083] Figure 3 is a flowchart of an encoding or decoding method provided in an embodiment of this application;

[0084] Figure 4a is a schematic diagram of an encoding or decoding method provided in an embodiment of this application;

[0085] Figure 4b is a schematic diagram of an encoding or decoding method provided in an embodiment of this application;

[0086] Figure 4c is a schematic diagram of an encoding or decoding method provided in an embodiment of this application;

[0087] Figure 4d is a schematic diagram of an encoding or decoding method provided in an embodiment of this application;

[0088] Figure 4e is a schematic diagram of an encoding or decoding method provided in an embodiment of this application;

[0089] Figure 4f is a schematic diagram of an encoding or decoding method provided in an embodiment of this application;

[0090] Figure 5 is a schematic diagram of an image encoding device provided in an embodiment of this application;

[0091] Figure 6 is a schematic diagram of an image decoding device provided in an embodiment of this application;

[0092] Figure 7 is a schematic diagram of a device provided in an embodiment of this application;

[0093] Figure 8 is a schematic diagram of a device provided in an embodiment of this application. Detailed Implementation

[0094] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0095] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0096] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0097] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0098] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0099] To facilitate understanding, some basic concepts involved in this application will be introduced below:

[0100] □ Tensor: A high-dimensional matrix. A 3D tensor refers to a tensor with three dimensions: (number of channels, height, and width). The shape or size of a tensor is usually represented as [C, H, W], where C represents the number of channels, H represents the height, and W represents the width.

[0101] □ Latent variables: Tensors formed by the encoder transformation of data, which can be transformed by the decoder to recover the original data, or data similar to the original data. They are usually 3-dimensional tensors, with each dimension representing length, width, and number of channels, respectively.

[0102] Entropy coding: The process of converting data into a 0 / 1 string.

[0103] Entropy decoding: The process of restoring a 0 / 1 string back to data; it is the reverse process of encoding.

[0104] □ Encoder: A model that transforms data (images, videos, etc.) into latent variables.

[0105] □ Decoder: A model that restores latent variables to the original data.

[0106] □ Hyperencoder: A model that further encodes latent variables into hyperlatent variables.

[0107] □ Superdecoder: A model that decodes super latent variables to restore latent variables, including superdecoding prediction models and superscale decoding models.

[0108] □ Superscale decoding model: A model that restores super-latent variables to scale variables. The scale variables serve as the conditional input for entropy decoding, and the output is the residual. The residual, along with the superdecoder output and the context model output, yields the latent variables.

[0109] □ Context Model: The latent variable recovery is divided into several parts, and each part outputs a portion of the latent variables. The result of each part is determined by the output of the superdecoder, the output of the superscale decoding model, and the result of some previously predicted latent variables.

[0110] □ Pixel shuffle (also known as down-shuffle): A simple operator that rearranges the positions of three-dimensional tensor elements such as latent variables, halving the length and width and increasing the number of channels to four times the original.

[0111] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts of neural networks that may be involved in the embodiments of this application will be introduced below.

[0112] (1) Neural Network

[0113] A neural network can be composed of neural units, which can be operational units that take xs and an intercept of 1 as inputs, and whose output can be:

[0114] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of Xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0115] (2) Deep Neural Networks

[0116] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple hidden layers. Based on the position of the layers, the internal neural network of a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0117] Although DNNs seem complex, the operation of each layer is actually not complicated. Simply put, it involves the following linear relationship expression: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is also relatively large. The definitions of these parameters in DNNs are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0118] In summary, the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as...

[0119] It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0120] (3) Convolutional Neural Network

[0121] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of a series of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are called the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is that it reduces the connections between layers in the CNN, while also reducing the risk of overfitting.

[0122] (4) Concatenation

[0123] The splicing operation refers to splicing multiple tensors along a certain dimension, where all tensors must have the same shape (except for the splicing dimension). This patent refers to splicing along a channel.

[0124] (5) Bitrate: In image compression tasks, the average coding length required to encode a unit pixel.

[0125] (6) Rate-distortion performance: an indicator used to measure the performance of compression algorithms, which takes into account both bit rate and the distortion of the decoded image.

[0126] The following describes the application scenarios to which the methods provided in the embodiments of this application are applicable.

[0127] The method provided in this application can be applied to business scenarios such as public cloud, private cloud, video surveillance, live streaming technology, storage, and transmission. The method provided in this application can be implemented as an image file type or storage standard, an image data transmission protocol, or compression software or storage device.

[0128] Figure 1a illustrates an application scenario example provided by an embodiment of this application, which can also be understood as an encoding / decoding application scenario. This application scenario includes an AI encoding unit, a storage / transmission unit, and an AI decoding unit. The method provided by this embodiment can be applied to the AI ​​encoding unit / AI decoding unit.

[0129] The AI ​​encoding unit is used to encode input data (such as images, videos, etc.) according to the encoding method provided in the embodiments of this application.

[0130] The storage / transmission unit is used to store (terminals, video surveillance, public cloud, private cloud, etc.) or transmit (public cloud, private cloud, live streaming technology, etc.) the encoded input data.

[0131] The AI ​​decoding unit is used to decode the encoded input data according to the decoding method provided in the embodiments of this application to obtain reconstructed data (which can also be understood as data reconstructed from the encoded features of the input data).

[0132] The application scenarios illustrated above have multiple applications, which will be described in detail below:

[0133] The first type is storage application scenarios.

[0134] This storage application scenario includes: terminal photo album, video surveillance, etc. A schematic diagram of this scenario is shown in Figure 1b, which includes: an AI encoding unit, a file saving unit, a file loading unit, and an AI decoding unit.

[0135] The AI ​​encoding unit is used to encode input data (such as images, videos, etc.) according to the encoding method provided in the embodiments of this application to obtain binary files.

[0136] The file storage unit is used to store the binary files output by the AI ​​encoding unit.

[0137] The load file unit is used to load binary files.

[0138] The AI ​​decoding unit is used to decode the loaded binary file according to the decoding method provided in the embodiments of this application to obtain reconstructed data (which can also be understood as data reconstructed from the encoded features of the input data).

[0139] The second type is the terminal-cloud interaction scenario.

[0140] The terminal-cloud interaction scenarios include: live streaming scenarios, public cloud scenarios, etc. A schematic diagram of this scenario is shown in Figure 1c. This scenario includes: encoding unit, decoding unit, AI encoding unit, AI decoding unit, encoding unit, and decoding unit.

[0141] The terminal side includes an encoding unit and a decoding unit, while the cloud side includes a decoding unit, an encoding unit, an AI encoding unit, and an AI decoding unit.

[0142] The terminal acquires input data and encodes it using an encoding unit to obtain encoded features. The terminal sends the encoded features to the cloud. Upon receiving the encoded features, the cloud decodes them using a decoding unit to obtain reconstructed data. The reconstructed data is then encoded using an AI encoding unit according to the encoding method provided in this embodiment to obtain a bitstream / file. The cloud decodes the bitstream / file using an AI decoding unit according to the decoding method provided in this embodiment to obtain reconstructed data. The cloud encodes the reconstructed data using an encoding unit to obtain encoded features and sends these features to the terminal. Upon receiving the encoded features, the terminal decodes them using a decoding unit to obtain reconstructed data.

[0143] It is understandable that the encoding methods used by the encoding unit and the AI ​​encoding unit in Figure 1c may be different (or, the operations of the decoding unit and the AI ​​encoding unit in the cloud can be called transcoding operations, and the operations of the AI ​​decoding unit and the encoding unit in the cloud can be called transcoding operations). For example, the cloud usually has more abundant computing resources than the terminal, so it can use algorithms with higher computing power requirements and higher compression efficiency (AI encoding) to re-compress (transcode) the input data to achieve a higher compression ratio and save cloud storage space. The transcoding process involves first decoding, and then encoding in another way.

[0144] It is understandable that the above scenarios are just examples, and there may be other scenarios in actual applications, which are not limited here.

[0145] The following description, with reference to Figure 2a, introduces an AI encoding unit and an AI decoding unit provided in an embodiment of this application. This architecture can be used in the AI ​​encoding unit / AI decoding unit shown in Figures 1a to 1c to complete encoding / decoding processing.

[0146] Figure 2a is a schematic diagram of the AI ​​encoding unit and AI decoding unit in an embodiment of this application. The basic architecture of the AI ​​compression model is a two-layer encoder-decoder architecture. The input data is complex data such as images and videos. The encoder and super encoder output latent variables and super-latent variables, and entropy encoding is performed on the latent and super-latent variables. The decoding process first decodes the super-latent variables, then uses the super decoder (including a super-decoding prediction model and a super-scale decoding model) to predict the latent variables, and recovers the latent variables based on the entropy decoding results and the context model. Finally, the decoder is used to recover the original data.

[0147] Let the input data be x, the latent variable be y, and the super-latent variable be z. The entire encoding process can be represented by the following formula:

[0148] y = Enc(x)

[0149] z = HEnc(x), bs z =EntropyEncoding(z)

[0150] Where Enc and HEnc represent the encoder and superencoder, respectively; the superdecoder consists of the superdecoder (HDec) and the superscale decoding model (HSDec); EntropyEncoding is the entropy encoding operation; Context is the context model; and σ is the scale variable. This is the output of the super-decoding prediction model, which can be called prior features. It will be used as the input to the context model to predict latent variables. The output bitstream is bs z ,bs r .

[0151] Decoding process based on bs z ,bs r Decoding, expressed by the formula as follows:

[0152] z = EntropyDecoding(bs) z )

[0153] Here, EntropyDecoding is the entropy decoding operation, and Dec is the decoder. The Context input here differs from the Context on the encoding side, but their operations are equivalent. The principle will be explained in detail below.

[0154] Note that in image compression models, All are 3D tensors and have the same shape.

[0155] The core of the context model is the handling of high-dimensional latent variables. The prediction is performed in multiple steps, with only one prediction made each time. One dimension, and make full use of the already predicted and the corresponding dimension y (or Predicting a portion of the current dimensions. Because predictions are made progressively and by fully utilizing previously predicted information, the context model's predictions of y are more accurate, thus entropy-encoded residuals. It requires fewer bits of encoding and can achieve better encoding performance.

[0156] The core of the context model is to... The prediction is divided into several parts. (correspond (It's the same division method). For each part The context model formula is expressed as follows:

[0157] in, For For The Context process of the encoding process can be specifically written in the following form:

[0158] where y<i is [y1,…,y i-1 .

[0159] This application focuses on how to reasonably divide in the field of image compression implementation solutions.

[0160] The following introduces a chip hardware structure provided by an embodiment of this application.

[0161] Figure 2b is a chip hardware structure provided by an embodiment of the present invention. The chip includes a neural network processor 40. The chip can be disposed in the AI encoding unit / AI decoding unit shown in Figures 1a to 1c to complete encoding / decoding processing.

[0162] The neural network processor 40 can be a neural-network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU), etc., all processors suitable for large-scale exclusive OR operation processing. Taking the NPU as an example: The neural network processor 40 is mounted on the main central processing unit (CPU) (host CPU) as a coprocessor, and tasks are assigned by the main CPU. The core part of the NPU is the arithmetic circuit 403, and the controller 404 controls the arithmetic circuit 403 to extract data from the memory (weight memory or input memory) and perform operations.

[0163] In some implementations, the arithmetic circuit 403 internally includes multiple processing units (process engine, PE). In some implementations, the arithmetic circuit 403 is a two-dimensional systolic array. The arithmetic circuit 403 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 403 is a general matrix processor.

[0164] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 403 fetches the corresponding data of matrix B from the weight memory 402 and caches it on each PE in the arithmetic circuit. The arithmetic circuit fetches the data of matrix A from the input memory 401 and performs matrix operations with matrix B, and the partial results or final results of the obtained matrix are stored in the accumulator 408.

[0165] The vector computation unit 407 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector computation unit 407 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, local response normalization, etc.

[0166] In some implementations, vector computation unit 407 stores the processed output vector in unified memory 406. For example, vector computation unit 407 can apply a nonlinear function to the output of arithmetic circuit 403, such as a vector of accumulated values, to generate activation values. In some implementations, vector computation unit 407 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to arithmetic circuit 403, for example, for use in subsequent layers of a neural network.

[0167] The unified memory 406 is used to store input data and output data.

[0168] The weight data is directly transferred from the external memory to the input memory 401 and / or the unified memory 406 through the direct memory access controller (DMAC) 405, the weight data in the external memory is stored in the weight memory 402, and the data in the unified memory 406 is stored in the external memory.

[0169] The bus interface unit (BIU) 410 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 409 via a bus.

[0170] The instruction fetch buffer 409, which is connected to the controller 404, is used to store the instructions used by the controller 404.

[0171] The controller 404 is used to call the instructions cached in the instruction fetch memory 409 to control the operation of the computing accelerator.

[0172] Generally, the unified memory 406, input memory 401, weight memory 402, and instruction fetch memory 409 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0173] The method provided in the embodiments of this application is described below. This method can be executed by an encoding device or a decoding device, or by a component of the encoding or decoding device (e.g., a processor, chip, or chip system). The encoding or decoding device can be a cloud device or a terminal device. Alternatively, the method can be executed by a system composed of a cloud device and a terminal device. Optionally, the method can be processed by a processor such as a CPU, NPU, or GPU in the encoding or decoding device; this application does not impose any limitations.

[0174] Figure 3 is a schematic diagram of an encoding or decoding method 1000 provided in an embodiment of this application, including the following steps:

[0175] 1001. Obtain the first feature of the image, where the shape of the first feature is [C, h, w].

[0176] 1003, reorder (down-shuffle) the first feature to obtain the second feature, the second feature having a shape of [4C,h / 2,w / 2], the second feature comprising C sub-features, each of the sub-features having a shape of [4,h / 2,w / 2];

[0177] 1005, recombine (cross-cat) the second feature to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2];

[0178] 1007. Based on the K third features, the image is encoded or decoded.

[0179] In some possible implementations, step 1007 is an optional step.

[0180] In step 1001, a first feature of the image is obtained. The shape of the first feature is [C, h, w], where C represents the number of channels of the first feature, h represents the length of the first feature, and w represents the width of the first feature. C, h, and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2) × (w / 2) 2×2 pixel blocks, that is, pixel blocks with (h / 2) rows and (w / 2) columns. Each pixel block includes four pixels: top left, bottom right, top right, and bottom left.

[0181] In this application, a pixel block can also be called a feature block, and a pixel point can also be called a feature point.

[0182] In one possible implementation, as shown in Figures 4a to 4f, the top-left, bottom-right, top-right, and bottom-left pixels of each 2×2 pixel block in the c-th channel of the first feature are represented by 0, 1, 2, 3 or 4, 5, 6, 7, respectively, where c is a positive integer and satisfies 1 ≤ c ≤ C. That is, the top-left pixel of each pixel block in the same channel is represented by 0 or 4, the bottom-right pixel of each pixel block in the same channel is represented by 1 or 5, the top-right pixel of each pixel block in the same channel is represented by 2 or 5, and the bottom-left pixel of each pixel block in the same channel is represented by 4 or 7.

[0183] In one possible implementation, on the encoding side, the encoder encodes the image to obtain latent variables, the super encoder encodes the latent variables into super latent variables, and the super decoder restores the super latent variables into prior features (i.e., the first feature mentioned above) and scale variables.

[0184] In one possible implementation, on the decoding side, entropy decoding is performed on the bitstream including the encoded super latent variable data to obtain the super latent variables, and then the super decoder restores the super latent variables into prior features (i.e. the first feature mentioned above) and scale variables.

[0185] Step 1003: Reorder the first feature to obtain a second feature, the second feature comprising C sub-features, each sub-feature having a shape of [4, h / 2, w / 2]. In one possible implementation, the c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, where c is a positive integer and satisfies 1 ≤ c ≤ C. The four channels of the c-th sub-feature are respectively composed of the top-left, bottom-right, top-right, and bottom-left pixels of all pixel blocks of the c-th channel of the first feature.

[0186] In the examples given in Figures 4a to 4c and Figure 4e, the first feature has 2 channels C, 4 heights h, and 4 widths w. After reordering the first feature, the second feature is obtained. The second feature includes two sub-features, each with a shape of [4,2,2]. The first sub-feature of the second feature is composed of the first to fourth channels of the second feature. The four channels of the first sub-feature are respectively composed of the top-left pixel (0), bottom-right pixel (1), top-right pixel (2), and bottom-left pixel in all pixel blocks of the first channel of the first feature. (3) Composition: The first sub-feature of the second feature consists of all the channels of the first channel of the first feature. The second sub-feature of the second feature consists of the fifth to eighth channels of the second feature. The four channels of the second sub-feature are respectively composed of the top left pixel (4), bottom right pixel (5), top right pixel (6) and bottom left pixel (7) of all the pixel blocks of the second channel of the first feature. The second sub-feature of the second feature consists of all the pixels of the second channel of the first feature.

[0187] In the examples given in Figures 4d and 4f, the first feature has 4 channels (C=4), a height (h=4), and a width (w=4). After reordering the first feature, the second feature is obtained. The second feature includes 4 sub-features, each with a shape of [4,2,2]. The first sub-feature of the second feature is composed of the first to fourth channels of the second feature. The four channels of the first sub-feature are composed of the top-left pixel (0), bottom-right pixel (1), top-right pixel (2), and bottom-left pixel (3) of all pixel blocks in the first channel of the first feature. All channels of the first sub-feature of the second feature together include all pixels in the first channel of the first feature. The second sub-feature of the second feature is composed of the fifth to eighth channels of the second feature. The four channels of the second sub-feature are composed of the top-left pixel (4), bottom-right pixel (5), top-right pixel (6), and bottom-left pixel (7) of all pixel blocks in the second channel of the first feature. The second feature consists of all channels of the second sub-feature of the second feature, which together include all pixels of the second channel of the first feature. The third sub-feature of the second feature consists of the 9th to 12th channels of the second feature. The four channels of the third sub-feature are respectively composed of the top left pixel (0), bottom right pixel (1), top right pixel (2), and bottom left pixel (3) of all pixel blocks of the third channel of the first feature. The fourth sub-feature of the second feature consists of the 13th to 16th channels of the second feature. The four channels of the fourth sub-feature are respectively composed of the top left pixel (4), bottom right pixel (5), top right pixel (6), and bottom left pixel (7) of all pixel blocks of the fourth channel of the first feature. The fourth sub-feature of the second feature consists of all pixels of the fourth channel of the first feature.

[0188] It is worth noting that Figures 4a to 4f only provide several specific examples of the scheme provided in this application. This application does not limit the value of C to 2 or 4, nor does it limit the values ​​of h and w to 4, as long as C, h, and w are all positive integers and multiples of 2. This application also does not limit the channel arrangement of each of the K third features, as long as the positions of the N pixels at the same position in the N channels of the kth third feature are all different in the N channels of the first feature, where N is a positive integer and 2≤N≤C. This application also does not limit the value of K to 3 or 4, as long as K is a positive integer greater than 2.

[0189] Step 1005: Recombine the second feature to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], and the kth third feature includes one channel of each sub-feature in the second feature, where K is a positive integer greater than 2, and k is a positive integer that satisfies 1 ≤ k ≤ K;

[0190] In the examples given in Figures 4a, 4b, and 4d to 4f, each of the K third features has the same shape, [C, h / 2, w / 2]. Specifically, in the examples given in Figures 4a, 4b, and 4e, each third feature has a shape of [2, 2, 2], and in the examples given in Figures 4d and 4f, each third feature has a shape of [4, 2, 2]. However, this application does not limit the shape of each of the K third features to be the same. For example, in the example given in Figure 4c, the first and second third features have the same shape, [2, 2, 2], while the last third feature has a different shape, [4, 2, 2]. In the method provided in this application, the number of channels for each of the K third features can be the same or different, but it is required that the sum of the number of channels for all the K third features is equal to four times C.

[0191] In one possible implementation of step 1005, the third feature obtained after recombination satisfies the following condition: the N pixels at the same position in the N channels of the k-th third feature are all different in the N channels of the first feature, where N is a positive integer and 2≤N≤C. The position of the pixel can be represented by spatial coordinates [x,y], where x represents the horizontal coordinate offset of the pixel relative to the upper left corner of the image, and y represents the vertical coordinate offset of the pixel relative to the upper left corner of the image. The position of the pixel can also be indicated by the pixel symbol. For example, in the examples given in Figures 4a to 4d, the upper left, lower right, upper right, and lower left pixels of each 2×2 pixel block in the c-th channel of the first feature are represented by 0, 1, 2, 3 or 4, 5, 6, 7, respectively, where c is a positive integer and 1≤c≤C. That is, the top-left pixel of each pixel block in the same channel is represented by 0 or 4, the bottom-right pixel of each pixel block in the same channel is represented by 1 or 5, the top-right pixel of each pixel block in the same channel is represented by 2 or 5, and the bottom-left pixel of each pixel block in the same channel is represented by 4 or 7. The N pixels at the same position in the N channels of the kth third feature are not in the same position in the N channels of the first feature, which means that the pixels included in the N channels exclude the following cases: (1) they are all top-left pixels in the 2×2 pixel block of the first feature; (2) they are all bottom-right pixels in the 2×2 pixel block of the first feature; (3) they are all top-right pixels in the 2×2 pixel block of the first feature; (4) they are all bottom-left pixels in the 2×2 pixel block of the first feature. It is worth noting that this application only requires that at least one of the K third features satisfies the above condition, and does not require that all K third features satisfy the above condition. For example, in the examples given in Figures 4a to 4d, all K third features satisfy the above condition, but in the example given in Figure 4e, only the last two third features satisfy the above condition, and in the example given in Figure 4f, only the first and third third features satisfy the above condition.

[0192] When K=4, the kth third feature is derived from the 8th (c+d)th second feature. 1k +1,8c+d 2k It consists of +1, c=0,…,C / 2-1 channels.

[0193] d 1k ,d 2k There are multiple ways to combine them (as shown in Table 1 below), as long as the combination is not (0,4)→(1,5)→(2,6)→(3,7).

[0194] Table 1 shows the possible combinations of the third feature when K = 4.

[0195] In the example given in Figure 4a, the first combination method in Table 1 is used, namely (0,6)→(1,7)→(2,4)→(3,5);

[0196] In the example given in Figure 4b, the fifth combination method in Table 1 is used, namely (0,7)→(1,6)→(2,5)→(3,4).

[0197] It is worth noting that this application does not list all the combination methods. This application only limits the combination method to not being (0,4)→(1,5)→(2,6)→(3,7). In other words, it excludes the case where each combination method of the third feature is composed of pixels with the same position in the pixel block of the first feature.

[0198] Figures 4a and 4b show examples where C = 2. When C = 4, in the example shown in Figure 4d, d 1k ,d 2k The combination still uses the first combination method in Table 1, that is, (0,6)→(1,7)→(2,4)→(3,5). At this time, the number of channels of each third feature also becomes 4. The first third feature is composed of the 1st, 7th, 9th and 15th channels of the second feature, the second third feature is composed of the 2nd, 8th, 10th and 16th channels of the second feature, the third third feature is composed of the 3rd, 5th, 11th and 13th channels of the second feature, and the fourth third feature is composed of the 4th, 6th, 12th and 14th channels of the second feature.

[0199] It should be noted that in this application, the channel number starts from 1 and increments. In some implementations, the channel number may also start from 0 and increment. When the channel number starts from 0 and increments, all channel numbers mentioned in this application need to be subtracted by 1.

[0200] In one possible implementation of step 1005, C / 2 channels in the k-th third feature are composed of the i-th channel of each sub-feature in the first part of the second feature, and the remaining C / 2 channels in the k-th third feature are composed of the j-th channel of each sub-feature in the second part of the second feature, where i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j. The first part of the sub-feature and the second part of the sub-feature each include C / 2 sub-features in the second feature, and the first part of the sub-feature and the second part of the sub-feature together constitute the second feature. When K=4 and C=2, the possible combinations of this implementation are listed in Table 1. Specifically, the 1st, 2nd, 3rd, and 4th channels of any sub-feature in the second feature are sequentially composed of the top-left, bottom-right, top-right, and bottom-left pixels of all 2×2 pixel blocks in the same channel of the first feature. Therefore, by limiting i≠j, it can be ensured that C / 2 channels in the k-th third feature and the remaining C / 2 channels are composed of pixels at different positions in the pixel block of the first feature. This ensures that the partitioning method can fully utilize the pixel information of adjacent pixels in the same channel and the pixel information of the same position in the surrounding channels when the context model predicts the latent variables, making the latent variables predicted by the context model more accurate, reducing the number of bits required to transmit the residual of the latent variables, and lowering the bit rate. It is worth noting that this application only requires that at least one of the K third features satisfies the above condition, and does not require that all K third features satisfy the above condition. For example, in the examples given in Figures 4a to 4d, all K third features satisfy the above condition, but in the example given in Figure 4e, only the last two third features satisfy the above condition, and in the example given in Figure 4f, only the first and third third features satisfy the above condition.

[0201] In one possible embodiment of the above implementation, K = 4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of the sub-features. If C = 2, then the example given in Figure 4a is as follows: the first third feature is composed of the first channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; the second third feature is composed of the second channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; the third third feature is composed of the third channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature; and the fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature.

[0202] In one possible embodiment of the above implementation, K=4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of the sub-features. If C = 2, then it is the example given in Figure 4b, that is, the first third feature is composed of the first channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; the second third feature is composed of the second channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; the third third feature is composed of the third channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature; the fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature.

[0203] In one possible embodiment of the above implementation, K = 4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of the sub-features.

[0204] In one possible embodiment of the above implementation, K = 4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of the sub-features.

[0205] In one possible embodiment of the above implementation, K = 3. In this case, the shapes of the first and second third features are [C, h / 2, w / 2], and the shape of the third feature is [2C, h / 2, w / 2]. The C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features. The C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of the sub-features. The C channels in the third third feature are composed of the third and fourth channels of each sub-feature in the first part of the sub-features, and the remaining C channels in the third third feature are composed of the first and second channels of each sub-feature in the second part of the sub-features.

[0206] In one possible embodiment of the above implementation, K = 3. In this case, the shapes of the first and third third features are [C, h / 2, w / 2], and the shape of the second third feature is [2C, h / 2, w / 2]. The C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features. The C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the second part of the sub-features. The C channels in the second third feature are composed of the third and fourth channels of each sub-feature in the first part of the sub-features, and the remaining C channels in the second third feature are composed of the first and second channels of each sub-feature in the second part of the sub-features.

[0207] In one possible embodiment of the above implementation, K = 3. In this case, the shapes of the second and third third features are [C, h / 2, w / 2], and the shape of the first third feature is [2C, h / 2, w / 2]. The C / 2 channels in the second third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features. The C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the second part of the sub-features. The C channels in the first third feature are composed of the third and fourth channels of each sub-feature in the first part of the sub-features, and the remaining C channels in the first third feature are composed of the first and second channels of each sub-feature in the second part of the sub-features.

[0208] In one possible implementation of step 1005, K = 4, and the combination of the third features is as follows: the first third feature consists of the 4cth channel of the first sub-feature and the 4c+2th channel of the second sub-feature; the second third feature consists of the 4c+1th channel of the first sub-feature and the 4c+3th channel of the second sub-feature; the third third feature consists of the 4c+2th channel of the first sub-feature and the 4cth channel of the second sub-feature; and the fourth third feature consists of the 4c+3th channel of the first sub-feature and the 4c+1th channel of the second sub-feature. Here, c is an integer, and the indexing starts from 0.

[0209] In one possible implementation of step 1005, K = 4, and the shape of each of the K third features is [C, h / 2, w / 2].

[0210] In one possible implementation of step 1005, K = 3, the shape of two of the K third features is [C, h / 2, w / 2], and the shape of one of the K third features is [2C, h / 2, w / 2]. This application does not limit which third feature among the K third features the third feature with 2C channels is, it can be the first third feature, the second third feature, or the third third feature.

[0211] Step 1007: Encode or decode the image based on the K third features. In one possible implementation of step 1007, encoding or decoding the image based on the K third features includes: for k = 1, 2, ..., K, predicting the k-th part of the latent variable of the image using a context model based on the k-th third feature; and encoding or decoding the image based on the K parts of the latent variable. Specifically, on the encoding side, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, and the difference between the original latent variable and the predicted latent variable can be calculated to obtain the residual of the latent variable. The residual of the latent variable is then entropy-encoded to obtain the bitstream. On the decoding side, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, and the bitstream can be entropy-decoded to obtain the residual of the latent variable. The residual of the predicted latent variable and the decoded latent variable can be summed to obtain the latent variable, and the latent variable is then decoded by the decoder to obtain the reconstructed image.

[0212] In one possible implementation of this application, the aforementioned first feature represents the luminance component and / or chrominance component of an image. When the luminance component and chrominance component of an image are represented by different first features, the combination of the third features corresponding to each of the luminance component and chrominance component can be the same or different.

[0213] When method 1000 is implemented by an encoder, that is, when method 1000 is an encoding method, the method further includes encoding a first syntax element, the first syntax element being used to indicate the combination method adopted by the third feature.

[0214] When method 1000 is implemented by a decoder, that is, when method 1000 is a decoding method, the method further includes decoding a first syntax element, the first syntax element being used to indicate the combination method adopted by the third feature.

[0215] In one possible implementation, the first syntax element is encoded in the bitstream's picture header, model header, or tools header.

[0216] In one possible implementation, the values ​​of the first syntax element correspond to the following combinations in Table 1:

[0217] In one possible implementation, the encoding and decoding sides agree on the combination methods of the third feature as shown in Table 1. In this case, the first syntax element can be transmitted with fewer bits. For example, if there are only 4 possible combinations of the third feature, then only 2 bits are needed to transmit the first syntax element.

[0218] In one possible implementation, if the encoding and decoding sides agree that the combination of the third feature is one of the specific combinations in Table 1, then the first syntax element does not need to be transmitted.

[0219] The encoding and decoding methods in the embodiments of this application have been described above. The apparatus in the embodiments of this application is described below. Please refer to FIG5. An embodiment of the encoding apparatus 1200 in the embodiments of this application includes: an acquisition module 1201, a processing module 1202 and an encoding module 1203.

[0220] The acquisition module 1201 is used to acquire a first feature of an image. The shape of the first feature is [C, h, w], where C represents the number of channels of the first feature, h represents the length of the first feature, w represents the width of the first feature, and C, h and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2) × (w / 2) 2 × 2 pixel blocks, and each pixel block includes four pixels: top left, bottom right, top right and bottom left.

[0221] The description of the acquisition module 1201 can be found in the description of step 1001 in the above embodiments, and will not be repeated here.

[0222] Processing module 1202 is used to reorder the first feature to obtain a second feature. The shape of the second feature is [4C, h / 2, w / 2]. The second feature includes C sub-features, each of which has a shape of [4, h / 2, w / 2]. The c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, where c is a positive integer and satisfies 1 ≤ c ≤ C. The four channels of the c-th sub-feature are respectively composed of the top-left, bottom-right, top-right, and bottom-left pixels of all the pixel blocks of the c-th channel of the first feature.

[0223] The processing module 1202 is further configured to recombine the second feature to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes one channel of each sub-feature in the second feature, K is a positive integer greater than 2, and k is a positive integer that satisfies 1 ≤ k ≤ K.

[0224] It is worth noting that this application does not limit the value of C to 2 or 4, nor does it limit the values ​​of h and w to 4, as long as C, h, and w are all positive integers and multiples of 2; this application also does not limit the channel arrangement of each of the K third features, as long as the positions of the N pixels at the same position in the N channels of the kth third feature are all different in the N channels of the first feature, where N is a positive integer and 2≤N≤C; this application also does not limit the value of K to 3 or 4, as long as K is a positive integer greater than 2.

[0225] In one possible implementation, the third feature obtained after recombination satisfies the following condition: the N pixels at the same position in the N channels of the k-th third feature are all different in the N channels of the first feature, where N is a positive integer and 2≤N≤C. The position of the pixel can be represented by spatial coordinates [x,y], where x represents the horizontal coordinate offset of the pixel relative to the top left corner of the image, and y represents the vertical coordinate offset of the pixel relative to the top left corner of the image. The position of the pixel can also be indicated by the pixel symbol. For example, in the examples given in Figures 4a to 4d, the top left, bottom right, top right, and bottom left pixels of each 2×2 pixel block in the c-th channel of the first feature are represented by 0, 1, 2, 3 or 4, 5, 6, 7, respectively, where c is a positive integer and 1≤c≤C. That is, the top-left pixel of each pixel block in the same channel is represented by 0 or 4, the bottom-right pixel of each pixel block in the same channel is represented by 1 or 5, the top-right pixel of each pixel block in the same channel is represented by 2 or 5, and the bottom-left pixel of each pixel block in the same channel is represented by 4 or 7. The N pixels at the same position in the N channels of the kth third feature are not in the same position in the N channels of the first feature, which means that the pixels included in the N channels exclude the following cases: (1) they are all top-left pixels in the 2×2 pixel block of the first feature; (2) they are all bottom-right pixels in the 2×2 pixel block of the first feature; (3) they are all top-right pixels in the 2×2 pixel block of the first feature; (4) they are all bottom-left pixels in the 2×2 pixel block of the first feature.

[0226] In one possible implementation, C / 2 channels of the k-th third feature are composed of the i-th channel of each sub-feature in the first part of the second feature, and the remaining C / 2 channels of the k-th third feature are composed of the j-th channel of each sub-feature in the second part of the second feature, where i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j. The first part of the sub-feature and the second part of the sub-feature each include C / 2 sub-features of the second feature, and together they constitute the second feature. Furthermore, the 1st, 2nd, 3rd, and 4th channels of any sub-feature in the second feature are sequentially composed of the top-left, bottom-right, top-right, and bottom-left pixels of all 2×2 pixel blocks in the same channel of the first feature. Therefore, by limiting i≠j, we can ensure that C / 2 channels in the kth third feature and the remaining C / 2 channels are composed of pixels at different positions in the pixel block of the first feature. This ensures that the partitioning method can make full use of the pixel information of adjacent pixels in the same channel and the pixel information of the same position in the surrounding channels when the context model predicts the latent variables, making the latent variables predicted by the context model more accurate, reducing the number of bits required to transmit the residual of the latent variables, and improving coding efficiency.

[0227] In one possible embodiment of the above implementation, K = 4. In this case, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of the sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of the sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of the sub-features. If C = 2, then the example given in Figure 4a is as follows: the first third feature is composed of the first channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; the second third feature is composed of the second channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; the third third feature is composed of the third channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature; and the fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature.

[0228] In one possible implementation, K=4, and the third feature is combined as follows: the first third feature consists of the 4c-th channel of the first sub-feature and the 4c+2-th channel of the second sub-feature; the second third feature consists of the 4c+1-th channel of the first sub-feature and the 4c+3-th channel of the second sub-feature; the third third feature consists of the 4c+2-th channel of the first sub-feature and the 4c-th channel of the second sub-feature; and the fourth third feature consists of the 4c+3-th channel of the first sub-feature and the 4c+1-th channel of the second sub-feature. Here, c is an integer, and the indexing starts from 0.

[0229] In one possible implementation, K = 4, and the shape of each of the K third features is [C, h / 2, w / 2].

[0230] In one possible implementation, K = 3, and the shapes of two of the K third features are [C, h / 2, w / 2], and the shape of one of the K third features is [2C, h / 2, w / 2]. This application does not limit which third feature among the K third features the third feature with 2C channels is, it can be the first third feature, the second third feature, or the third third feature.

[0231] For a more detailed description of the processing module 1202, please refer to the description of steps 1003 and 1005 in the above embodiments, which will not be repeated here.

[0232] The encoding module 1203 is used to encode the image based on the K third features.

[0233] In one possible implementation, encoding the image based on the K third features includes: for k = 1, 2, ..., K, predicting the k-th part of the latent variable of the image using a context model based on the k-th third feature; and encoding the image based on the K parts of the latent variable. Specifically, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, and the residual of the latent variable can be obtained by subtracting the original latent variable from the predicted latent variable. Then, the residual of the latent variable is entropy encoded to obtain the bitstream.

[0234] The description of the encoding module 1203 can be found in step 1007 of the above embodiments, and will not be repeated here.

[0235] In one possible implementation, the first feature described above represents the luminance component and / or chrominance component of the image. When the luminance component and chrominance component of the image are represented by different first features, the combination of the third features corresponding to the luminance component and chrominance component of the image can be the same or different.

[0236] Optionally, the encoding module 1203 is further configured to encode a first syntax element, which indicates the combination method used by the third feature.

[0237] In one possible implementation, the first syntax element is encoded in the bitstream's picture header, model header, or tools header.

[0238] Please refer to Figure 6. One embodiment of the decoding device 1300 in this application includes: an acquisition module 1301, a processing module 1302, and a decoding module 1303.

[0239] The acquisition module 1301 is used to acquire a first feature of an image. The shape of the first feature is [C, h, w], where C represents the number of channels of the first feature, h represents the length of the first feature, w represents the width of the first feature, and C, h and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2) × (w / 2) 2×2 pixel blocks, and each pixel block includes four pixels: top left, bottom right, top right and bottom left.

[0240] The description of the acquisition module 1301 can be found in the description of step 1001 in the above embodiments, and will not be repeated here.

[0241] Processing module 1302 is used to reorder the first feature to obtain a second feature. The shape of the second feature is [4C, h / 2, w / 2]. The second feature includes C sub-features, each of which has a shape of [4, h / 2, w / 2]. The c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, where c is a positive integer and satisfies 1 ≤ c ≤ C. The four channels of the c-th sub-feature are respectively composed of the top-left, bottom-right, top-right, and bottom-left pixels of all the pixel blocks of the c-th channel of the first feature.

[0242] The processing module 1302 is further configured to recombine the second feature to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes one channel of each sub-feature in the second feature, K is a positive integer greater than 2, and k is a positive integer that satisfies 1≤k≤K.

[0243] The description of the processing module 1302 can be found in steps 1003 and 1005 of the above embodiments, and will not be repeated here.

[0244] The decoding module 1303 is used to decode based on the K third features to obtain the reconstructed image.

[0245] In one possible implementation, decoding the image based on the K third features includes: for k = 1, 2, ..., K, predicting the k-th part of the latent variable of the image using a context model based on the k-th third feature; and encoding or decoding the image based on the K parts of the latent variable. Specifically, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, entropy decoding can be performed on the bitstream to obtain the residual of the latent variable, and then the residual of the predicted latent variable and the decoded latent variable can be summed to obtain the latent variable. The latent variable is then decoded by the decoder to obtain the reconstructed image.

[0246] The description of the decoding module 1303 can be found in step 1007 of the above embodiment, and will not be repeated here.

[0247] In one possible implementation, the first feature described above represents the luminance component and / or chrominance component of the image. When the luminance component and chrominance component of the image are represented by different first features, the combination of the third features corresponding to the luminance component and chrominance component of the image can be the same or different.

[0248] Optionally, the decoding module 1303 is also used to decode a first syntax element, which is used to indicate the combination method adopted by the third feature.

[0249] In one possible implementation, the first syntax element is encoded in the bitstream's picture header, model header, or tools header.

[0250] The following describes a device provided in an embodiment of this application. This device can be an encoding device or a decoding device. Please refer to Figure 7, which is a structural schematic diagram of the device provided in an embodiment of this application. The device 1400 can specifically be a virtual reality (VR) device, a mobile phone, a tablet, a laptop, a smart wearable device, etc., and is not limited here. The device 1400 can be equipped with the devices described in the embodiments corresponding to Figures 5 or 6. Specifically, the device 1400 includes: a receiver 1401, a transmitter 1402, a processor 1403, and a memory 1404 (the number of processors 1403 in the device 1400 can be one or more; Figure 7 shows one processor as an example). The processor 1403 can include an application processor 14031 and a communication processor 14032. In some embodiments of this application, the receiver 1401, transmitter 1402, processor 1403, and memory 1404 can be connected via a bus or other means.

[0251] Memory 1404 may include read-only memory and random access memory, and provides instructions and data to processor 1403. A portion of memory 1404 may also include non-volatile random access memory (NVRAM). Memory 1404 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0252] Processor 1403 controls the operation of the device. In specific applications, the various components of the device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses in the diagram are referred to as the bus system.

[0253] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 1403. The processor 1403 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuits in the hardware of the processor 1403 or by instructions in software form. The processor 1403 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1403 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1404. Processor 1403 reads the information in memory 1404 and, in conjunction with its hardware, completes the steps of the above method.

[0254] Receiver 1401 can be used to receive input digital or character information, and to generate signal inputs related to device settings and function control. Transmitter 1402 can be used to output digital or character information through the first interface; transmitter 1402 can also be used to send instructions to the disk group through the first interface to modify data in the disk group; transmitter 1402 may also include a display device such as a display screen.

[0255] This application also provides a device, which can be an encoding device or a decoding device. Referring to Figure 8, which is a schematic diagram of a device provided in this application embodiment, the device 1500 can be deployed with the device described in the corresponding embodiment of Figure 5 or Figure 6. Specifically, the device 1500 is implemented by one or more servers. The device 1500 can vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 1522 (e.g., one or more processors) and a memory 1532, and one or more storage media 1530 (e.g., one or more mass storage devices) for storing application programs 1542 or data 1544. The memory 1532 and storage media 1530 can be temporary or persistent storage. The program stored in the storage media 1530 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the device. Furthermore, the CPU 1522 may be configured to communicate with the storage media 1530 and execute the series of instruction operations in the storage media 1530 on the device 1500.

[0256] Device 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input / output interfaces 1558, and / or one or more operating systems 1541, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0257] This application also provides a computer program product (or computer program) that stores one or more computers. When the computer program product is executed by the processor, the processor executes the method of the image processing device that may be implemented as described above.

[0258] This application also provides a chip system including at least one processor for supporting a terminal device or server in implementing the functions involved in the possible implementations of the image processing device described above. Optionally, the chip system further includes an interface circuit that provides program instructions and / or data to the at least one processor. In one possible design, the chip system may further include a memory for storing necessary program instructions and data for the terminal device. The chip system may be composed of chips or may include chips and other discrete devices.

[0259] This application also provides a readable storage medium storing a bitstream encoded using the methods or apparatus shown in the embodiments of Figures 1a to 6.

[0260] This application also provides a device for storing a bitstream, characterized in that it includes at least one storage medium and a communication interface; the communication interface is used to receive or send the bitstream; at least one storage medium is used to store the bitstream; the bitstream is encoded by an encoder using the method or device shown in the embodiments of Figures 1a to 6 above.

[0261] This application also provides a method for storing a bitstream, the method comprising: receiving the bitstream through a communication interface; storing the bitstream in one or more storage media, wherein the bitstream is encoded by an encoder using the method or apparatus shown in the embodiments of Figures 1a to 6 above.

[0262] This application also provides a system for distributing bitstreams, the system including at least one storage medium and a video streaming device; the at least one storage medium is used to store the bitstream, which is encoded by an encoder using the method or apparatus shown in the embodiments of Figures 1a to 6 above; the video streaming device is used to send the target bitstream in the at least one storage medium to the decoder in response to a request from the decoder.

[0263] This application embodiment also provides a method for distributing a bitstream, the method comprising: receiving a first request; in response to the first request, selecting a target bitstream from at least one storage medium; sending the target bitstream to a destination device; wherein at least one storage medium is used to store the bitstream, the bitstream being encoded by an encoder using the method or apparatus shown in the embodiments of Figures 1a to 6 above.

[0264] This application also provides a system for processing bitstreams. The system includes an image source device, an encoder device, one or more storage media, and a destination device. The image source device provides image data. The encoder device acquires the image data from the image source device through an interface and encodes the image data to obtain one or more bitstreams. The bitstreams are encoded by the encoder using the methods or apparatus shown in the embodiments of Figures 1a to 6. The encoder device stores one or more bitstreams in one or more storage media; or the encoder device encapsulates one or more bitstreams to obtain a transmission bitstream. The encoder device transmits the transmission bitstream to the destination device through a communication link or communication network. The destination device decapsulates the transmission bitstream to obtain one or more bitstreams. The destination device decodes one or more bitstreams to obtain decoded data.

[0265] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0266] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Any implementation provided in this application can be combined with each other in any way, provided that the logic is sound. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, apparatuses, or units, and may be electrical, mechanical, or other forms.

[0267] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0268] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0269] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, device, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0270] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0271] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, device, or data center to another website, computer, device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a device or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. An image encoding or decoding method, characterized in that, include: The first feature of the image is obtained, and the shape of the first feature is [C,h,w], where C represents the number of channels of the first feature, h represents the length of the first feature, w represents the width of the first feature, and C, h and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, and each pixel block includes 4 pixels in total, namely, the top left, bottom right, top right and bottom left. The first feature is reordered to obtain the second feature, which has a shape of [4C, h / 2, w / 2]. The second feature includes C sub-features, each of which has a shape of [4, h / 2, w / 2]. The c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, where c is a positive integer and satisfies 1 ≤ c ≤ C. The four channels in the c-th sub-feature are respectively composed of the top-left, bottom-right, top-right, and bottom-left pixels in all the pixel blocks of the c-th channel of the first feature. The second feature is recombine to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], and the kth third feature includes one channel of each sub-feature in the second feature, where K is a positive integer greater than 2, and k is a positive integer that satisfies 1 ≤ k ≤ K; The image is encoded or decoded based on the K third features.

2. The method according to claim 1, characterized in that, The N pixels at the same position in the N channels of the kth third feature are all different in the N channels of the first feature, where N is a positive integer and 2≤N≤C.

3. The method according to claim 1 or 2, characterized in that, The C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of the second feature, where i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j. The first part of the sub-feature and the second part of the sub-feature each include C / 2 sub-features in the second feature, and the first part of the sub-feature and the second part of the sub-feature together constitute the second feature.

4. The method according to any one of claims 1 to 3, characterized in that, K = 4, and the shape of each of the K third features is [C, h / 2, w / 2].

5. The method according to claim 1 or 2, characterized in that, K = 3, the shapes of two of the K third features are [C, h / 2, w / 2], and the shape of one of the K third features is [2C, h / 2, w / 2].

6. The method according to claim 3, characterized in that, When K = 4, at this time... The C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third feature are composed of the first channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of the sub-features.

7. The method according to claim 6, characterized in that, When C = 2, at this time... The first third feature is composed of the first channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; The second third feature is composed of the second channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; The third feature is composed of the third channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature; The fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature.

8. The method according to claim 3, characterized in that, When K = 4, at this time... The C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third feature are composed of the second channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of the sub-features.

9. The method according to claim 3, characterized in that, When K = 4, at this time... The C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third feature are composed of the second channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of the sub-features.

10. The method according to claim 3, characterized in that, When K = 4, at this time... The C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third feature are composed of the first channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of the sub-features.

11. The method according to claim 3, characterized in that, When K=3, the shapes of the first and second third features are [C,h / 2,w / 2], and the shape of the third feature is [2C,h / 2,w / 2]. The C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of the sub-features; The C channels in the third feature are composed of the third and fourth channels of each sub-feature in the first part of the sub-features, and the remaining C channels in the third feature are composed of the first and second channels of each sub-feature in the second part of the sub-features.

12. The method according to any one of claims 1-11, characterized in that, The process of encoding or decoding the image based on the K third features includes: For k = 1, 2, ..., K, based on the kth third feature, the context model is used to predict the kth part of the latent variables of the image; The image is encoded or decoded based on K parts of the latent variables.

13. The method according to any one of claims 1-12, characterized in that, The first feature represents the luminance component and / or chrominance component of the image.

14. The method according to any one of claims 1-13, characterized in that, When the method is an encoding method, it further includes: Encode a first syntax element, which indicates the combination method used by the third feature.

15. The method according to any one of claims 1-13, characterized in that, When the method is a decoding method, it also includes: Decode the first syntax element, which indicates the combination method used by the third feature.

16. An image encoding device, characterized in that, The device includes: The acquisition module is used to acquire the first feature of the image. The shape of the first feature is [C, h, w], where C represents the number of channels of the first feature, h represents the length of the first feature, w represents the width of the first feature, and C, h and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2) × (w / 2) 2×2 pixel blocks, and each pixel block includes four pixels: top left, bottom right, top right and bottom left. The processing module is used to reorder the first feature to obtain the second feature. The shape of the second feature is [4C, h / 2, w / 2]. The second feature includes C sub-features, each of which has a shape of [4, h / 2, w / 2]. The c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, where c is a positive integer and satisfies 1 ≤ c ≤ C. The four channels in the c-th sub-feature are respectively composed of the top-left, bottom-right, top-right, and bottom-left pixels in all the pixel blocks of the c-th channel of the first feature. The processing module is further configured to recombine the second feature to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes one channel of each sub-feature in the second feature, K is a positive integer greater than 2, and k is a positive integer that satisfies 1≤k≤K; An encoding module is used to encode the image based on the K third features.

17. The encoding device according to claim 16, characterized in that, The N pixels at the same position in the N channels of the kth third feature are all different in the N channels of the first feature, where N is a positive integer and 2≤N≤C.

18. The encoding device according to claim 16 or 17, characterized in that, The C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of the second feature, where i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j. The first part of the sub-feature and the second part of the sub-feature each include C / 2 sub-features in the second feature, and the first part of the sub-feature and the second part of the sub-feature together constitute the second feature.

19. The encoding device according to claim 18, characterized in that, When K = 4, at this time... The C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third feature are composed of the first channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of the sub-features.

20. The encoding device according to claim 19, characterized in that, When C = 2, at this time... The first third feature is composed of the first channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; The second third feature is composed of the second channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; The third feature is composed of the third channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature; The fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature.

21. An image decoding device, characterized in that, The device includes: The acquisition module is used to acquire the first feature of the image. The shape of the first feature is [C, h, w], where C represents the number of channels of the first feature, h represents the length of the first feature, w represents the width of the first feature, and C, h and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2) × (w / 2) 2×2 pixel blocks, and each pixel block includes four pixels: top left, bottom right, top right and bottom left. The processing module is used to reorder the first feature to obtain the second feature. The shape of the second feature is [4C, h / 2, w / 2]. The second feature includes C sub-features, each of which has a shape of [4, h / 2, w / 2]. The c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, where c is a positive integer and satisfies 1 ≤ c ≤ C. The four channels in the c-th sub-feature are respectively composed of the top-left, bottom-right, top-right, and bottom-left pixels in all the pixel blocks of the c-th channel of the first feature. The processing module is further configured to recombine the second feature to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes one channel of each sub-feature in the second feature, K is a positive integer greater than 2, and k is a positive integer that satisfies 1≤k≤K; The decoding module is used to decode based on the K third features to obtain the reconstructed image.

22. The decoding apparatus according to claim 21, characterized in that, The N pixels at the same position in the N channels of the kth third feature are all different in the N channels of the first feature, where N is a positive integer and 2≤N≤C.

23. The decoding apparatus according to claim 21 or 22, characterized in that, The C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of the second feature, where i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j. The first part of the sub-feature and the second part of the sub-feature each include C / 2 sub-features in the second feature, and the first part of the sub-feature and the second part of the sub-feature together constitute the second feature.

24. The decoding apparatus according to claim 23, characterized in that, When K = 4, at this time... The C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the third feature are composed of the third channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the third feature are composed of the first channel of each sub-feature in the second part of the sub-features; The C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of the sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of the sub-features.

25. The decoding apparatus according to claim 24, characterized in that, When C = 2, at this time... The first third feature is composed of the first channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; The second third feature is composed of the second channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; The third feature is composed of the third channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature; The fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature.

26. An encoding or decoding device, characterized in that, include: A processor coupled to a memory for storing programs or instructions that, when executed by the processor, cause the encoding or decoding device to perform the method as described in any one of claims 1-15.

27. A codec system, characterized in that, The encoding / decoding system includes an encoding device as described in any one of claims 16-20, and / or a decoding device as described in any one of claims 21-25.

28. A computer-readable storage medium, characterized in that, The medium stores computer instructions that, when executed by a computer, implement the method as described in any one of claims 1-15.

29. A computer program product, characterized in that, Includes computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-15.

30. A computer-readable storage medium, characterized in that, The computer-readable medium stores a bitstream encoded according to any one of the encoding methods in claims 1-15.

31. A device for storing bitstreams, characterized in that, Includes at least one storage medium and a communication interface; The communication interface is used to receive or send code streams; The at least one storage medium is used to store the bit stream; The bitstream is encoded by the encoder according to any one of the encoding methods in claims 1-15.

32. A method for storing a bitstream, characterized in that, include: Receive the bitstream through the communication interface; The bitstream is stored in one or more storage media, and the bitstream is an encoder. Encoded according to any one of the encoding methods in claims 1-15.

33. A system for distributing bitstreams, characterized in that, Includes at least one storage medium and a video streaming device; The at least one storage medium is used to store a bitstream, which is an encoder. Encoded according to any one of the encoding methods in claims 1-15; The video streaming device is configured to, in response to a request from the decoder, transmit the bitstream in the at least one storage medium to the decoder.

34. A method for distributing a bitstream, characterized in that, include: Receive the first request; In response to the first request, a bitstream is selected from at least one storage medium; Send the bitstream to the destination device; The at least one storage medium is used to store a bitstream, which is an encoder. Encoded according to any one of the encoding methods in claims 1-15.

35. A system for processing bitstreams, characterized in that, Includes an image source device, an encoder device, one or more storage media, and a destination device; The image source device is used to provide image data; The encoder device is used to acquire the image data of the image source device through an interface, and encode the image data to obtain one or more bitstreams, wherein the bitstreams are encoded by the encoder according to any one of the encoding methods in claims 1-15; The encoder device is used to store the one or more bit streams into one or more storage media; or The encoder device is used to encapsulate the one or more code streams to obtain a transmission code stream; The encoder device is used to transmit the transmitted code stream to the destination device via a communication link or communication network; The destination device is used to decapsulate the transmitted code stream to obtain the one or more code streams; The target device is used to decode the one or more bitstreams to obtain decoded data.

Images