Coding and decoding method and device

Abstract

The invention relates to the field of image coding and decoding, and discloses an image coding or decoding method, which can be applied to JPEG-AI and other next generation image or video coding and decoding standards, and comprises the following steps: obtaining a first feature of an image, the shape of the first feature being [C, h, w]; the first feature is reordered to obtain a second feature, the shape of the second feature is [4C, h / 2, w / 2], the second feature comprises C sub-features, and the shape of each sub-feature is [4, h / 2, w / 2]; the second features are recombined to obtain K third features, the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature comprises a channel of each sub-feature in the second features, and K is a positive integer greater than 2; and encoding or decoding the image based on the K third features. According to the method, the adjacent pixel information of the same channel of the hidden variable and the pixel information of the same position of the peripheral channel can be fully utilized, so that the hidden variable predicted by the context model is more accurate, the number of bits required for encoding the hidden variable is reduced, and the code rate is reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image or video encoding and decoding, and in particular, to an encoding and decoding method and device. Background Art

[0002] The AI ​​(Artificial Intelligence) image compression algorithm is based on deep learning and has better compression effect than traditional image compression technologies (such as JPEG (Joint Photographic Experts Group), BPG (Better Portable Graphics), etc.).

[0003] When using AI compression (also known as encoding) and decompression (also known as decoding), the input data is complex data such as images and videos. The encoder and super encoder output latent variables and super latent variables, and perform entropy encoding on the latent variables and super latent variables. The decoding process first decodes the super latent variables, then uses the super decoder to predict the latent variables, and restores the latent variables based on the results of entropy decoding and the context model. Finally, the decoder is used to restore the original data.

[0004] In AI encoding and decoding, a context model is needed to predict latent variables. The prediction accuracy of the context model will directly affect the overall effect of encoding and decoding. Summary of the invention

[0005] The present application provides a coding and decoding method and device. The method can make full use of the pixel information adjacent to the same channel of the latent variable and the pixel information at the same position of the surrounding channels, so that the context model prediction is more accurate, thereby reducing the number of bits required for entropy coding of the latent variable, and obtaining better coding performance.

[0006] In a first aspect, an embodiment of the present application provides an encoding or decoding method, the method comprising: obtaining a first feature of an image (in the entire encoding and decoding process of the image, the first feature can be called a priori feature), the shape of the first feature is [C, h, w], wherein C represents the number of channels of the first feature, h represents the length of the first feature, w represents the width of the first feature, C, h and w are all positive integer multiples of 2, each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, each of the pixel blocks includes 4 pixels in the upper left, lower right, upper right and lower left; reordering the first feature to obtain a second feature, the shape of the second feature is [4C, h / 2, w / 2], and the second feature includes C sub-features , each of the sub-features has a shape of [4, h / 2, w / 2], the cth sub-feature of the second feature corresponds to the cth channel of the first feature, c is a positive integer and satisfies 1≤c≤C, and the four channels in the cth sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all the pixel blocks of the cth channel in the first feature; the second features are recombined to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes a channel of each sub-feature in the second feature, K is a positive integer greater than 2, k is a positive integer and satisfies 1≤k≤K; based on the K third features, the image is encoded or decoded.

[0007] In the present application, by reordering and recombining the prior features (i.e., the first features), a more reasonable prior feature for segmentation is obtained, and then the context model is input to predict the latent variable in segmentation, so that the segmentation behind the latent variable can make full use of the pixel information adjacent to the same channel and the pixel information at the same position of the surrounding channel in the segmentation of the latent variable that has been predicted before, thereby improving the prediction accuracy of the latent variable, thereby reducing the number of bits required for encoding the latent variable and improving the coding efficiency. After the required bits in the code stream are reduced, the communication bandwidth required for transmitting the code stream and the storage resources required for storing the code stream can also be saved, and the corresponding technical effects can be reflected in the encoding process, transmission process and decoding process. Processing the prior features on the decoding side in the same way as the encoding side can ensure correct decoding.

[0008] In a possible implementation of the first aspect, Figures 4a to 4dAs shown, the upper left, lower right, upper right and lower left pixels of each 2×2 pixel block in the cth channel of the first feature are represented by 0, 1, 2, 3 or 4, 5, 6, 7, respectively, where c is a positive integer and satisfies 1≤c≤C. That is, the upper left pixel of each pixel block in the same channel is represented by 0 or 4, the lower right pixel of each pixel block in the same channel is represented by 1 or 5, the upper right pixel of each pixel block in the same channel is represented by 2 or 5, and the lower left pixel of each pixel block in the same channel is represented by 4 or 7.

[0009] It is worth noting that Figures 4a to 4d Only a few specific examples of the solutions provided in the present application are given. The present application does not limit the value of C to 2 or 4, and the values ​​of h and w are not limited to 4, as long as C, h and w are all positive integers and are integer multiples of 2; the present application does not limit the channel arrangement of each of the K third features, as long as the positions of N pixels at the same position in the N channels of the kth third feature are different in the N channels of the first feature, where N is a positive integer and 2≤N≤C; the present application does not limit the value of K to 3 or 4, as long as K is a positive integer greater than 2.

[0010] In the method provided in the present application, the number of channels of each of the K third features may be the same or different, but the total number of channels of all the third features of the K third features is required to be equal to 4 times of C.

[0011] In a possible implementation of the first aspect, the third feature obtained after recombining satisfies the following conditions: the positions of N pixels at the same position in N channels of the kth third feature in the N channels of the first feature are different from each other, wherein N is a positive integer and 2≤N≤C, wherein the position of the pixel can be represented by the spatial coordinates [x, y], x represents the horizontal coordinate offset of the pixel relative to the upper left corner of the image, and y represents the vertical coordinate offset of the pixel relative to the upper left corner of the image; the position of the pixel can also be indicated by a symbol representing the pixel. The positions of N pixels at the same position in N channels of the kth third feature in the N channels of the first feature are different from each other, which means that the pixels included in the N channels exclude the following situations: (1) all are upper left pixels in the 2×2 pixel block in the first feature; (2) all are lower right pixels in the 2×2 pixel block in the first feature; (3) all are upper right pixels in the 2×2 pixel block in the first feature; (4) all are lower left pixels in the 2×2 pixel block in the first feature. By limiting the positions of the N pixels at the same position in the N channels of the kth third feature after the recombination to be different from each other in the N channels of the first feature, it can be ensured that when the third feature is used to predict the latent variable, the pixel information at the same position in the surrounding channels can be fully utilized (because the pixels at the same position in the surrounding channels have been obtained in the previous segmented prediction), which can improve the prediction accuracy of the latent variable, thereby reducing the number of bits required to encode the latent variable and reducing the bit rate. It should be noted that in this application, it is only necessary to limit at least one of all K third features to meet the above conditions, and it is not required that all third features meet the above conditions.

[0012] In a possible implementation of the first aspect, C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of sub-features in the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of sub-features in the second feature, wherein i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j, the first part of sub-features and the second part of sub-features each include C / 2 sub-features in the second feature, and the first part of sub-features and the second part of sub-features together constitute the second feature. When K=4 and C=2, possible combinations of this implementation are listed in Table 1. Among them, the first channel, the second channel, the third channel and the fourth channel of any sub-feature in the second feature are composed of the upper left, lower right, upper right and lower left pixel points in all 2×2 pixel blocks in the same channel of the first feature. Therefore, by limiting i≠j, it can be ensured that the C / 2 channels in the kth third feature and the remaining C / 2 channels are composed of pixel points at different positions in the pixel block in the first feature, thereby ensuring that the division method can make full use of the pixel information adjacent to the same channel and the pixel information at the same position of the surrounding channels when the context model predicts the latent variable, so that the latent variable predicted by the context model is more accurate, reducing the number of bits required to transmit the residual of the latent variable, and improving the coding efficiency. It should be noted that in the present application, it is only necessary to limit at least one third feature among all K third features to meet the above conditions, and it is not required that all K third features meet the above conditions.

[0013] In a possible embodiment of the above implementation, K=4, in which case, C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the second channel of each sub-feature in the second part of sub-features. If C=2, then Figure 4aThe example given is, that is, the first third feature is composed of the first channel of the first subfeature of the second feature and the third channel of the second subfeature of the second feature; the second third feature is composed of the second channel of the first subfeature of the second feature and the fourth channel of the second subfeature of the second feature; the third third feature is composed of the third channel of the first subfeature of the second feature and the first channel of the second subfeature of the second feature; the fourth third feature is composed of the fourth channel of the first subfeature of the second feature and the second channel of the second subfeature of the second feature.

[0014] In a possible embodiment of the above implementation, K=4, in which case, C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the second channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the first channel of each sub-feature in the second part of sub-features. If C=2, then Figure 4b The example given is, that is, the first third feature is composed of the first channel of the first subfeature of the second feature and the fourth channel of the second subfeature of the second feature; the second third feature is composed of the second channel of the first subfeature of the second feature and the third channel of the second subfeature of the second feature; the third third feature is composed of the third channel of the first subfeature of the second feature and the second channel of the second subfeature of the second feature; the fourth third feature is composed of the fourth channel of the first subfeature of the second feature and the first channel of the second subfeature of the second feature.

[0015] In a possible embodiment of the above implementation, K=4, in which case C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the second channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the first channel of each sub-feature in the second part of sub-features.

[0016] In a possible embodiment of the above implementation, K=4, in which case C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the second channel of each sub-feature in the second part of sub-features.

[0017] In a possible embodiment of the above implementation, K=3, in which case the shapes of the first and second third features are [C, h / 2, w / 2], and the shape of the third feature is [2C, h / 2, w / 2]; C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; C channels in the third third feature are composed of the third and fourth channels of each sub-feature in the first part of sub-features, and the remaining C channels in the third third feature are composed of the first and second channels of each sub-feature in the second part of sub-features.

[0018] In a possible implementation of the first aspect, K=4, and the third features are combined as follows: the first third feature is composed of the 4cth channel in the first sub-feature and the 4c+2th channel in the second sub-feature; the second third feature is composed of the 4c+1th channel of the first sub-feature and the 4c+3th channel of the second sub-feature; the third third feature is composed of the 4c+2th channel of the first sub-feature and the 4cth channel of the second sub-feature; the fourth third feature is composed of the 4c+3th channel of the first sub-feature and the 4c+1th channel of the second sub-feature. Wherein c is an integer, and the numbering index starts from 0.

[0019] In a possible implementation manner of the first aspect, K=4, and a shape of each of the K third features is [C, h / 2, w / 2].

[0020] In a possible implementation manner of the first aspect, K=3, shapes of two of the K third features are [C, h / 2, w / 2], and shape of one of the K third features is [2C, h / 2, w / 2]. The present application does not limit the third feature with a channel number of 2C to be the number of the K third features, and it may be the first, second, or third third feature.

[0021] In a possible implementation of the first aspect, encoding or decoding the image based on the K third features includes: for k=1, 2, ..., K, respectively based on the kth third feature, using a context model to predict the kth part of the latent variable of the image; encoding or decoding the image based on the K parts of the latent variable. Specifically, on the encoding side, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, and then the residual of the latent variable is obtained by subtracting the original latent variable from the predicted latent variable, and then the residual of the latent variable is entropy encoded to obtain a bitstream; and on the decoding side, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, the bitstream is entropy decoded to obtain the residual of the latent variable, and then the residual of the predicted latent variable and the decoded latent variable are summed to obtain the latent variable, and then the latent variable is decoded by the decoder to obtain a reconstructed image.

[0022] In a possible implementation of the first aspect, the first feature represents a brightness component and / or a chromaticity component of an image. When the brightness component and the chromaticity component of an image are represented by different first features, the combination of the third features corresponding to the brightness component and the chromaticity component of the image may be the same or different.

[0023] When the method of the first aspect is implemented by an encoder, that is, when the method of the first aspect is an encoding method, the method further includes encoding a first syntax element, where the first syntax element is used to indicate a combination method adopted by the third feature.

[0024] When the method of the first aspect is implemented by a decoder, that is, when the method of the first aspect is a decoding method, the method further includes decoding a first syntax element, where the first syntax element is used to indicate a combination method adopted by the third feature.

[0025] In a possible implementation manner, the first syntax element is encoded in a picture header, or a model header, or a tools header of a code stream.

[0026] In a possible implementation manner of the first aspect, the encoding side and the decoding side agree that the combination method used by the third feature is a specific combination method in Table 1, and the first syntax element does not need to be transmitted.

[0027] In a second aspect, the present application provides an image encoding device, characterized in that the device includes: an acquisition module, used to acquire a first feature of an image, the shape of the first feature is [C, h, w], wherein C represents the number of channels of the first feature, h represents the length of the first feature, w represents the width of the first feature, C, h and w are all positive integer multiples of 2, each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, and each of the pixel blocks includes 4 pixels in the upper left, lower right, upper right and lower left; a processing module, used to reorder the first feature to obtain a second feature, the shape of the second feature is [4C, h / 2, w / 2], the second feature includes C sub-features, and the shape of each sub-feature is [4,h / 2,w / 2], the cth sub-feature of the second feature corresponds to the cth channel of the first feature, c is a positive integer and satisfies 1≤c≤C, and the four channels in the cth sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all the pixel blocks of the cth channel in the first feature; the processing module is further used to recombine the second features to obtain K third features, wherein the shape of the kth third feature is [C,h / 2,w / 2], the kth third feature includes a channel for each sub-feature in the second feature, K is a positive integer greater than 2, k is a positive integer and satisfies 1≤k≤K; the encoding module is used to encode the image based on the K third features.

[0028] In a possible implementation manner of the second aspect, positions of N pixels at the same position in the N channels in the kth third feature are different from each other in the N channels in the first feature, where N is a positive integer and 2≤N≤C.

[0029] In a possible implementation manner of the second aspect, C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of sub-features in the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of sub-features in the second feature, where i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j, the first part of sub-features and the second part of sub-features each include C / 2 sub-features in the second feature, and the first part of sub-features and the second part of sub-features together constitute the second feature.

[0030] In one embodiment of this implementation, K=4, in which case C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the second channel of each sub-feature in the second part of sub-features. When C=2, the first third feature is composed of the first channel of the first subfeature of the second feature and the third channel of the second subfeature of the second feature; the second third feature is composed of the second channel of the first subfeature of the second feature and the fourth channel of the second subfeature of the second feature; the third third feature is composed of the third channel of the first subfeature of the second feature and the first channel of the second subfeature of the second feature; the fourth third feature is composed of the fourth channel of the first subfeature of the second feature and the second channel of the second subfeature of the second feature.

[0031] In another embodiment of this implementation, K=4, in which case C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the second channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the first channel of each sub-feature in the second part of sub-features.

[0032] In another embodiment of this implementation, K=4, in which case C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the second channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the first channel of each sub-feature in the second part of sub-features.

[0033] In another embodiment of this implementation, K=4, in which case C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the second channel of each sub-feature in the second part of sub-features.

[0034] In another embodiment of the implementation, K=3, in which case the shapes of the first and second third features are [C, h / 2, w / 2], and the shape of the third feature is [2C, h / 2, w / 2]; the C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; the C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; the C channels in the third third feature are composed of the third and fourth channels of each sub-feature in the first part of sub-features, and the remaining C channels in the third third feature are composed of the first and second channels of each sub-feature in the second part of sub-features.

[0035] In a possible implementation manner of the second aspect, K=4, and a shape of each of the K third features is [C, h / 2, w / 2].

[0036] In a possible implementation manner of the second aspect, K=3, shapes of two third features among the K third features are [C, h / 2, w / 2], and shape of one third feature among the K third features is [2C, h / 2, w / 2].

[0037] In a possible implementation manner of the second aspect, encoding the image based on the K third features includes:

[0038] For k=1, 2, ..., K, based on the kth third feature, respectively, the context model is used to predict the kth part of the latent variable of the image; and the image is encoded based on the K parts of the latent variable. Specifically, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, and then the residual of the latent variable is obtained by subtracting the original latent variable from the predicted latent variable, and then the residual of the latent variable is entropy encoded to obtain a bitstream.

[0039] In a possible implementation manner of the second aspect, the first feature represents a brightness component and / or a chrominance component of the image.

[0040] In a possible implementation manner of the second aspect, the method further includes encoding a first syntax element, wherein the first syntax element is used to indicate a combination method adopted by the third feature. The first syntax element is encoded in a picture header, a model header, or a tools header of a code stream.

[0041] The technical effects of the second aspect and any implementation method of the second aspect refer to the description of the first aspect and the implementation method of the first aspect, and will not be repeated here.

[0042] In a third aspect, the present application provides an image decoding device, characterized in that the device includes: an acquisition module, used to acquire a first feature of an image, the shape of the first feature is [C, h, w], wherein C represents the number of channels of the first feature, h represents the length of the first feature, w represents the width of the first feature, C, h and w are all positive integer multiples of 2, each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, and each of the pixel blocks includes 4 pixels in the upper left, lower right, upper right and lower left; a processing module, used to reorder the first feature to obtain a second feature, the shape of the second feature is [4C, h / 2, w / 2], the second feature includes C sub-features, and the shape of each sub-feature is [ 4,h / 2,w / 2], the cth sub-feature of the second feature corresponds to the cth channel of the first feature, c is a positive integer and satisfies 1≤c≤C, and the four channels in the cth sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all the pixel blocks of the cth channel in the first feature; the processing module is further used to recombine the second features to obtain K third features, wherein the shape of the kth third feature is [C,h / 2,w / 2], the kth third feature includes a channel for each sub-feature in the second feature, K is a positive integer greater than 2, k is a positive integer and satisfies 1≤k≤K; a decoding module is used to decode based on the K third features to obtain a reconstructed image.

[0043] In a possible implementation manner of the third aspect, positions of N pixels at the same position in the N channels in the kth third feature are different from each other in the N channels in the first feature, where N is a positive integer and 2≤N≤C.

[0044] In a possible implementation manner of the third aspect, C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of sub-features in the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of sub-features in the second feature, where i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j, the first part of sub-features and the second part of sub-features each include C / 2 sub-features in the second feature, and the first part of sub-features and the second part of sub-features together constitute the second feature.

[0045] In one embodiment of this implementation, K=4, in which case C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the second channel of each sub-feature in the second part of sub-features. When C=2, the first third feature is composed of the first channel of the first subfeature of the second feature and the third channel of the second subfeature of the second feature; the second third feature is composed of the second channel of the first subfeature of the second feature and the fourth channel of the second subfeature of the second feature; the third third feature is composed of the third channel of the first subfeature of the second feature and the first channel of the second subfeature of the second feature; the fourth third feature is composed of the fourth channel of the first subfeature of the second feature and the second channel of the second subfeature of the second feature.

[0046] In another embodiment of this implementation, K=4, in which case C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the second channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the first channel of each sub-feature in the second part of sub-features.

[0047] In another embodiment of this implementation, K=4, in which case C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the second channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the first channel of each sub-feature in the second part of sub-features.

[0048] In another embodiment of this implementation, K=4, in which case C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the second channel of each sub-feature in the second part of sub-features.

[0049] In another embodiment of the implementation, K=3, in which case the shapes of the first and second third features are [C, h / 2, w / 2], and the shape of the third feature is [2C, h / 2, w / 2]; the C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; the C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; the C channels in the third third feature are composed of the third and fourth channels of each sub-feature in the first part of sub-features, and the remaining C channels in the third third feature are composed of the first and second channels of each sub-feature in the second part of sub-features.

[0050] In a possible implementation manner of the third aspect, K=4, and a shape of each of the K third features is [C, h / 2, w / 2].

[0051] In a possible implementation manner of the third aspect, K=3, shapes of two third features among the K third features are [C, h / 2, w / 2], and shape of one third feature among the K third features is [2C, h / 2, w / 2].

[0052] In a possible implementation manner of the third aspect, decoding is performed based on the K third features, including:

[0053] For k=1, 2, ..., K, based on the kth third feature, respectively, the context model is used to predict the kth part of the latent variable of the image; based on the K parts of the latent variable, encoding is performed to obtain a reconstructed image. Specifically, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, the bitstream is entropy decoded to obtain the residual of the latent variable, and then the predicted latent variable and the residual of the decoded latent variable are summed to obtain the latent variable, and then the latent variable is decoded by the decoder to obtain the reconstructed image.

[0054] In a possible implementation manner of the third aspect, the first feature represents a brightness component and / or a chrominance component of the image.

[0055] In a possible implementation manner of the third aspect, the method further includes decoding a first syntax element, where the first syntax element is used to indicate a combination method adopted by the third feature. The first syntax element is encoded in a picture header, a model header, or a tools header of a code stream.

[0056] The technical effects of the third aspect and any implementation method of the third aspect refer to the description of the first aspect and the implementation method of the first aspect, and will not be repeated here.

[0057] In a fourth aspect, the present application provides an encoding or decoding device, characterized in that it includes: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the encoding or decoding device executes the method described in the first aspect and any one of the implementation methods of the first aspect.

[0058] The technical effects of the fourth aspect and any implementation method of the fourth aspect refer to the description of the first aspect and the implementation method of the first aspect, and will not be repeated here.

[0059] In a fifth aspect, the present application provides a coding and decoding system, characterized in that the coding and decoding system includes an encoding device as described in the second aspect and any one of the implementations of the second aspect, and / or a decoding device as described in the third aspect and any one of the implementations of the third aspect.

[0060] The technical effects of the fifth aspect and any implementation method of the fifth aspect refer to the description of the first aspect and the implementation method of the first aspect, and will not be repeated here.

[0061] In a sixth aspect, the present application provides a computer-readable storage medium, characterized in that the medium stores computer-readable instructions, and when the computer-readable instructions are executed by a computer, the method described in the above-mentioned first aspect and any one of the implementation methods of the first aspect is implemented.

[0062] The technical effects of the sixth aspect and any implementation method of the sixth aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0063] In a seventh aspect, the present application provides a computer program product, characterized in that it includes computer-readable instructions, and when the computer-readable instructions are executed on a computer, the computer executes the method described in the first aspect and any one of the implementation methods of the first aspect.

[0064] The technical effects of the seventh aspect and any implementation method of the seventh aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0065] In an eighth aspect, the present application provides a computer-readable storage medium, characterized in that the computer-readable medium stores a code stream encoded according to the encoding method described in the first aspect and any one of the implementations of the first aspect.

[0066] The technical effects of the eighth aspect and any implementation method of the eighth aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0067] In a ninth aspect, the present application provides a device for storing a code stream, characterized in that it includes at least one storage medium and a communication interface; the communication interface is used to receive or send a code stream; the at least one storage medium is used to store the code stream; the code stream is encoded by an encoder according to the encoding method described in the first aspect and any one of the implementations of the first aspect.

[0068] The technical effects of the ninth aspect and any implementation method of the ninth aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0069] In the tenth aspect, the present application provides a method for storing a code stream, characterized in that it includes: receiving a code stream through a communication interface; storing the code stream in one or more storage media, wherein the code stream is encoded by an encoder according to the encoding method described in the first aspect and any one of the implementations of the first aspect.

[0070] The technical effects of the tenth aspect and any implementation method of the tenth aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0071] In an eleventh aspect, the present application provides a system for distributing code streams, characterized in that it includes at least one storage medium and a video stream device; the at least one storage medium is used to store the code stream, and the code stream is encoded by the encoder according to the encoding method in the first aspect and any one of the implementations of the first aspect; the video stream device is used to respond to the request of the decoder so that the code stream in the at least one storage medium can be sent to the decoder.

[0072] The technical effects of the eleventh aspect and any implementation method of the eleventh aspect refer to the description of the first aspect and the implementation method of the first aspect, and will not be repeated here.

[0073] In a twelfth aspect, the present application provides a method for distributing a code stream, characterized in that it includes: receiving a first request; in response to the first request, selecting a code stream from at least one storage medium; sending the code stream to a destination device; the at least one storage medium is used to store the code stream, and the code stream is encoded by the encoder according to the encoding method in the first aspect and any one of the implementations of the first aspect.

[0074] The technical effects of the twelfth aspect and any implementation method of the twelfth aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here.

[0075] In a thirteenth aspect, the present application provides a system for processing a code stream, characterized in that it includes an image source device, an encoder device, one or more storage media and a destination device; the image source device is used to provide image data; the encoder device is used to obtain the image data of the image source device through an interface, and encode the image data to obtain one or more code streams, wherein the code stream is encoded by the encoder according to the encoding method in the first aspect and any one of the implementations of the first aspect; the encoder device is used to store the one or more code streams in one or more storage media; or the encoder device is used to encapsulate the one or more code streams to obtain a transmission code stream; the encoder device is used to transmit the transmission code stream to the destination device through a communication link or a communication network; the destination device is used to decapsulate the transmission code stream to obtain the one or more code streams; the destination device is used to decode the one or more code streams to obtain decoded data.

[0076] The technical effects of the thirteenth aspect and any implementation method of the thirteenth aspect refer to the description of the first aspect and the implementation method of the first aspect above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1a A schematic diagram of an application scenario of an embodiment of the present application;

[0078] Figure 1b A schematic diagram of an application scenario of an embodiment of the present application;

[0079] Figure 1c A schematic diagram of an application scenario of an embodiment of the present application;

[0080] Figure 2a A schematic diagram of an AI encoding unit and an AI decoding unit according to an embodiment of the present application;

[0081] Figure 2b A chip hardware structure provided in an embodiment of the present application;

[0082] Figure 3 A flowchart of an encoding or decoding method provided in an embodiment of the present application;

[0083] Figure 4a A schematic diagram of an encoding or decoding method provided in an embodiment of the present application;

[0084] Figure 4b A schematic diagram of an encoding or decoding method provided in an embodiment of the present application;

[0085] Figure 4c A schematic diagram of an encoding or decoding method provided in an embodiment of the present application;

[0086] Figure 4d A schematic diagram of an encoding or decoding method provided in an embodiment of the present application;

[0087] Figure 4e A schematic diagram of an encoding or decoding method provided in an embodiment of the present application;

[0088] Figure 4f A schematic diagram of an encoding or decoding method provided in an embodiment of the present application;

[0089] Figure 5 A schematic diagram of the structure of an image encoding device provided in an embodiment of the present application;

[0090] Figure 6 A schematic diagram of the structure of an image decoding device provided in an embodiment of the present application;

[0091] Figure 7 A schematic diagram of the structure of a device provided in an embodiment of the present application;

[0092] Figure 8 A schematic diagram of the structure of the device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0093] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0094] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0095] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0096] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0097] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0098] For ease of understanding, some basic concepts involved in this application are introduced below:

[0099] Tensor: A high-dimensional matrix. A 3D tensor refers to a tensor divided into three dimensions (number of channels, height, width). The shape or size of a tensor is usually expressed as [C, H, W], where C represents the number of channels, H represents height, and W represents width.

[0100] Hidden variable: A tensor formed by the data being transformed by the encoder, which can be restored to the original data or data similar to the original data by the decoder. It is usually a 3-dimensional tensor, with each dimension representing the length, width, and number of channels.

[0101] Entropy coding: The process of converting data into a string of 0 / 1.

[0102] Entropy decoding: The process of restoring a 0 / 1 string to data, which is the reverse process of encoding.

[0103] Encoder: A model that transforms data (images, videos, etc.) into latent variables.

[0104] Decoder: A model that restores latent variables to original data.

[0105] Super Encoder: A model that further encodes latent variables into super latent variables.

[0106] Super decoder: A model that decodes super latent variables and restores them to latent variables, including super decoding prediction model and super scale decoding model.

[0107] Super-scale decoding model: A model that restores super-hidden variables to scale variables. The scale variables are used as conditional inputs for entropy decoding and output residuals. The residuals are combined with the super-decoder output and the context model output to obtain the hidden variables.

[0108] Context model: Divide the latent variable recovery into several parts, each part outputs a part of the latent variable, and the result of each part is determined by the super decoder output, the super scale decoding model output, and the partial results of the latent variables predicted before.

[0109] Pixel reorganization / reordering (pixel-shuffle, also known as down-shuffle): A simple operator that rearranges the positions of the elements of a three-dimensional tensor such as a hidden variable, halving the length and width and increasing the number of channels to four times the original.

[0110] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0111] (1) Neural Network

[0112] A neural network may be composed of neural units, and a neural unit may refer to an operation unit with xs and intercept 1 as input, and the output of the operation unit may be:

[0113] Where s=1, 2, ..., n, n is a natural number greater than 1, Ws is the weight of Xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0114] (2) Deep Neural Networks

[0115] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0116] Although DNN looks complicated, the work of each layer is not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0117] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0118] It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0119] (3) Convolutional Neural Network

[0120] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connection between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0121] (4) Concat

[0122] The concatenation operation refers to concatenating multiple tensors in a certain dimension, and all tensors must have the same shape (except for the concatenation dimension). This patent refers to concatenation on channels.

[0123] (5) Bit rate: In image compression tasks, the average coding length required to encode a unit pixel.

[0124] (6) Rate-distortion performance: An indicator used to measure the performance of a compression algorithm, which comprehensively considers both bit rate and decoded image distortion.

[0125] The following describes the application scenarios to which the method provided in the embodiments of the present application is applicable.

[0126] The method provided in the embodiment of the present application can be applied to business scenarios such as public cloud, private cloud, video surveillance, live broadcast technology, storage, transmission, etc. The method provided in the embodiment of the present application can be implemented as an image file type or storage standard, or an image data transmission protocol, or a compression software or storage device.

[0127] Figure 1a An example of an application scenario provided in an embodiment of the present application can also be understood as a coding and decoding application scenario. The application scenario includes: an AI coding unit, a storage / transmission unit, and an AI decoding unit. The method provided in an embodiment of the present application can be applied in an AI coding unit / AI decoding unit.

[0128] The AI ​​encoding unit is used to encode input data (such as images, videos, etc.) according to the encoding method provided in the embodiment of the present application.

[0129] The storage / transmission unit is used to store (terminal, video surveillance, public cloud, private cloud, etc.) or transmit (public cloud, private cloud, live broadcast technology, etc.) the encoded input data.

[0130] The AI ​​decoding unit is used to decode the encoded input data according to the decoding method provided in the embodiment of the present application to obtain reconstructed data (which can also be understood as data after reconstructing the encoding features of the input data).

[0131] There are many application scenarios in the above examples, which are described in detail below:

[0132] The first one is the storage application scenario.

[0133] The storage application scenarios include: terminal photo album, video surveillance, etc. The schematic diagram of this scenario is as follows Figure 1b As shown, the scene includes: an AI encoding unit, a file saving unit, a file loading unit, and an AI decoding unit.

[0134] The AI ​​encoding unit is used to encode input data (such as images, videos, etc.) according to the encoding method provided in the embodiment of the present application to obtain a binary file.

[0135] The file saving unit is used to save the binary file output by the AI ​​encoding unit.

[0136] The loader file unit is used to load binary files.

[0137] The AI ​​decoding unit is used to decode the loaded binary file according to the decoding method provided in the embodiment of the present application to obtain reconstructed data (which can also be understood as data after reconstructing the encoding features of the input data).

[0138] The second is the terminal-cloud interaction scenario.

[0139] The terminal-cloud interaction scenarios include: live broadcast scenarios, public clouds, etc. The schematic diagram of this scenario is as follows Figure 1c As shown, the scenario includes: an encoding unit, a decoding unit, an AI encoding unit, an AI decoding unit, an encoding unit, and a decoding unit.

[0140] Among them, the terminal side includes an encoding unit and a decoding unit, and the cloud side includes a decoding unit, an encoding unit, an AI encoding unit, and an AI decoding unit.

[0141] The terminal obtains input data, and encodes the input data through the encoding unit to obtain encoding features. The terminal sends the encoding features to the cloud. After the cloud receives the encoding features, it decodes the encoding features through the decoding unit to obtain reconstructed data. Then the AI ​​encoding unit encodes the reconstructed data according to the encoding method provided in the embodiment of the present application to obtain a code stream / file. The cloud decodes the code stream / file through the AI ​​decoding unit according to the decoding method provided in the embodiment of the present application to obtain reconstructed data. The cloud encodes the reconstructed data through the encoding unit to obtain the encoding features. And send the encoding features to the terminal. After the terminal receives the encoding features, it decodes the encoding features through the decoding unit to obtain the reconstructed data.

[0142] Understandably, Figure 1c The encoding methods used by the encoding unit and the AI ​​encoding unit may be inconsistent (or it can be understood that the operations of the decoding unit and the AI ​​encoding unit in the cloud can be called transcoding operations, and the operations of the AI ​​decoding unit and the encoding unit in the cloud can be called transcoding operations). For example, the cloud usually has more computing power resources than the terminal, so an algorithm with high computing power requirements and higher compression efficiency (AI encoding) can be used to re-compress the input data (transcoding) to obtain a higher compression multiple and save cloud storage space. The process of transcoding: decode first, then encode in another way.

[0143] It is understandable that the above-mentioned scenarios are just examples, and there may be other scenarios in actual applications, which are not specifically limited here.

[0144] Combine the following Figure 2a An AI encoding unit and an AI decoding unit provided in an embodiment of the present application are introduced. The architecture can be used for Figure 1a to Figure 1c The AI ​​encoding unit / AI decoding unit shown is used to complete the encoding / decoding process.

[0145] Figure 2a Schematic diagram of the AI ​​encoding unit and AI decoding unit of the embodiment of the present application, the AI ​​compression model infrastructure is a two-layer encoder-decoder architecture. Among them, the input data is complex data such as images and videos, and the encoder and super encoder output hidden variables and super hidden variables, and entropy encode the hidden variables and super hidden variables. The decoding process first decodes the super hidden variables, and then uses the super decoder (including super decoding prediction model, superscale decoding model) to predict the hidden variables, and restores the hidden variables according to the results of entropy decoding and the context model, and finally uses the decoder to restore the original data.

[0146] Assume input data is x, latent variable is y, and super latent variable is z. The whole encoding process is expressed by the formula as follows:

[0147] y=Enc(x)

[0148] z=HEnc(x),bs z =EntropyEncodimg(z)

[0149]

[0150]

[0151] Where Enc, HEnc represent the encoder, super encoder, super decoding prediction model HDec and super scale decoding model HSDec constitute super decoder, EntropyEncoding is the entropy coding operation, Context is the context model, σ is the scale variable, It is the output of the super decoding prediction model, which can be called a priori features and will be used as the input of the context model to predict latent variables The output bit stream is bs z ,bs r .

[0152] The decoding process is based on bs z ,bs r Decoding, expressed as the following formula:

[0153] z=EntropyDecoding(bs z )

[0154] σ=HSDec(z),

[0155]

[0156]

[0157] EntropyDecoding is the entropy decoding operation, and Dec is the decoder. The context input here is different from the context on the encoding side, but their operations are equivalent. The principle is explained in detail below.

[0158] Note that in the image compression model, are all 3D tensors with the same shape.

[0159] Among them, the core of the context model is the high-dimensional latent variable The prediction is divided into multiple steps, each time only predicting and take advantage of the already predicted and the corresponding dimension y (or ), predict the current part of the dimension Since the prediction is performed step by step and the already predicted information is fully utilized, the context model makes a more accurate prediction of y, so the entropy coding residual Fewer coding bits are required, resulting in better coding performance.

[0160] The core of the context model is to The forecast is divided into several parts (correspond The same division is used). For each part The context model formula is as follows:

[0161]

[0162] in, for for The context flow of the encoding process can be specifically written as follows:

[0163]

[0164] Among them, y ＜i is [y1,...,y i-1 ].

[0165] This application focuses on how to reasonably divide the field of image compression implementation scheme.

[0166] The following introduces a chip hardware structure provided by an embodiment of the present application.

[0167] Figure 2b A chip hardware structure provided in an embodiment of the present invention includes a neural network processor 40. The chip can be arranged as follows: Figure 1a to Figure 1c The AI ​​encoding unit / AI decoding unit shown is used to complete the encoding / decoding process.

[0168] The neural network processor 40 can be a neural network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU) and any other processor suitable for large-scale XOR processing. Take NPU as an example: the neural network processor 40 is mounted on the main central processing unit (CPU) (host CPU) as a coprocessor, and the main CPU assigns tasks. The core part of the NPU is the operation circuit 403, and the controller 404 controls the operation circuit 403 to extract data from the memory (weight memory or input memory) and perform operations.

[0169] In some implementations, the operation circuit 403 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 403 is a two-dimensional systolic array. The operation circuit 403 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 403 is a general-purpose matrix processor.

[0170] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 403 takes the corresponding data of the matrix B from the weight memory 402 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 401 and performs a matrix operation with the matrix B, and the partial result or the final result of the matrix is ​​stored in the accumulator 408.

[0171] The vector calculation unit 407 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 407 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0172] In some implementations, the vector calculation unit 407 stores the vector of processed outputs to the unified memory 406. For example, the vector calculation unit 407 can apply a nonlinear function to the output of the operation circuit 403, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 407 generates a normalized value, a merged value, or both. In some implementations, the vector of processed outputs can be used as an activation input to the operation circuit 403, such as for use in a subsequent layer in a neural network.

[0173] The unified memory 406 is used to store input data and output data.

[0174] The weight data is directly transferred from the external memory to the input memory 401 and / or the unified memory 406 through the direct memory access controller (DMAC) 405, the weight data in the external memory is stored in the weight memory 402, and the data in the unified memory 406 is stored in the external memory.

[0175] The bus interface unit (BIU) 410 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 409 through the bus.

[0176] An instruction fetch buffer 409 connected to the controller 404 is used to store instructions used by the controller 404 .

[0177] The controller 404 is used to call the instructions cached in the instruction fetch memory 409 to control the working process of the computing accelerator.

[0178] Generally, the unified memory 406, the input memory 401, the weight memory 402 and the instruction fetch memory 409 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM for short), a high bandwidth memory (HBM) or other readable and writable memory.

[0179] The method provided by the embodiment of the present application is described below. The method can be performed by a coding device or a decoding device, or by a component of the coding device or the decoding device (such as a processor, a chip, or a chip system, etc.). The coding device or the decoding device can be a cloud device or a terminal device. Of course, the method can also be performed by a system consisting of a cloud device and a terminal device. Optionally, the method can be processed by a processor such as a CPU, NPU, GPU in a coding device or a decoding device, and this application is not limited.

[0180] Figure 3 A schematic diagram of an encoding or decoding method 1000 provided in an embodiment of the present application includes the following steps:

[0181] 1001, obtaining a first feature of an image, wherein the shape of the first feature is [C, h, w];

[0182] 1003, reorder (down-shuffle) the first feature to obtain a second feature, the shape of the second feature is [4C, h / 2, w / 2], the second feature includes C sub-features, and the shape of each sub-feature is [4, h / 2, w / 2];

[0183] 1005, recombining (cross-cat) the second features to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2];

[0184] 1007. Encode or decode the image based on the K third features.

[0185] In some possible implementations, step 1007 is an optional step.

[0186] Among them, step 1001, obtains the first feature of the image, and the shape of the first feature is [C, h, w], wherein C represents the number of channels of the first feature, h represents the length of the first feature, and w represents the width of the first feature. C, h and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, that is, pixel blocks with (h / 2) rows and (w / 2) columns, and each of the pixel blocks includes 4 pixels in total, namely, upper left, lower right, upper right and lower left.

[0187] In this application, a pixel block may also be referred to as a feature block, and a pixel point may also be referred to as a feature point.

[0188] In one possible implementation, Figures 4a to 4f As shown, the upper left, lower right, upper right and lower left pixels of each 2×2 pixel block in the cth channel of the first feature are represented by 0, 1, 2, 3 or 4, 5, 6, 7, respectively, where c is a positive integer and satisfies 1≤c≤C. That is, the upper left pixel of each pixel block in the same channel is represented by 0 or 4, the lower right pixel of each pixel block in the same channel is represented by 1 or 5, the upper right pixel of each pixel block in the same channel is represented by 2 or 5, and the lower left pixel of each pixel block in the same channel is represented by 4 or 7.

[0189] In one possible implementation, on the encoding side, the encoder encodes the image to obtain latent variables, the super encoder encodes the latent variables into super latent variables, and the super decoder restores the super latent variables into prior features (i.e., the first feature mentioned above) and scale variables.

[0190] In one possible implementation, on the decoding side, the bitstream including the encoded super-latent variable data is entropy decoded to obtain super-latent variables, and then the super-decoder restores the super-latent variables into prior features (i.e., the first features mentioned above) and scale variables.

[0191] Step 1003: reorder the first feature to obtain a second feature, wherein the second feature includes C sub-features, and each sub-feature has a shape of [4, h / 2, w / 2]. In a possible implementation, the c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, where c is a positive integer and satisfies 1≤c≤C, and the four channels in the c-th sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all pixel blocks of the c-th channel of the first feature.

[0192] exist Figures 4a to 4c ,as well as Figure 4eIn the given example, the number of channels of the first feature is C=2, the height is h=4, and the width is w=4. The second feature is obtained by reordering the first feature. The second feature includes 2 sub-features, and the shape of each sub-feature is [4,2,2]. The first sub-feature of the second feature is composed of the first to fourth channels of the second feature. The four channels of the first sub-feature are respectively composed of the upper left pixel (0), the lower right pixel (1), the upper right pixel (2), and the lower left pixel (3) in all pixel blocks of the first channel of the first feature. All channels of the first sub-feature of the second feature include all pixels of the first channel of the first feature. The second sub-feature of the second feature is composed of the fifth to eighth channels of the second feature. The four channels of the second sub-feature are respectively composed of the upper left pixel (4), the lower right pixel (5), the upper right pixel (6), and the lower left pixel (7) in all pixel blocks of the second channel of the first feature. All channels of the second sub-feature of the second feature include all pixels of the second channel of the first feature.

[0193] exist Figure 4d and Figure 4fIn the given example, the number of channels of the first feature is C=4, the height is h=4, and the width is w=4. The second feature is obtained by reordering the first feature. The second feature includes 4 sub-features, and the shape of each sub-feature is [4,2,2]. The first sub-feature of the second feature is composed of the 1st to 4th channels of the second feature. The 4 channels of the first sub-feature are respectively composed of the upper left pixel (0), the lower right pixel (1), the upper right pixel (2), and the lower left pixel (3) in all pixel blocks of the 1st channel of the first feature. All channels of the first sub-feature of the second feature together include all pixels of the 1st channel of the first feature; the second sub-feature of the second feature is composed of the 5th to 8th channels of the second feature. The 4 channels of the second sub-feature are respectively composed of the upper left pixel (4), the lower right pixel (5), the upper right pixel (6), and the lower left pixel (7) in all pixel blocks of the 2nd channel of the first feature. All channels of the second sub-feature of the feature include all pixels of the second channel of the first feature; the third sub-feature of the second feature is composed of the 9th to 12th channels of the second feature, and the four channels of the third sub-feature are respectively composed of the upper left pixel (0), the lower right pixel (1), the upper right pixel (2) and the lower left pixel (3) in all pixel blocks of the third channel of the first feature, and all channels of the third sub-feature of the second feature include all pixels of the third channel of the first feature; the fourth sub-feature of the second feature is composed of the 13th to 16th channels of the second feature, and the four channels of the fourth sub-feature are respectively composed of the upper left pixel (4), the lower right pixel (5), the upper right pixel (6) and the lower left pixel (7) in all pixel blocks of the fourth channel of the first feature, and all channels of the fourth sub-feature of the second feature include all pixels of the fourth channel of the first feature.

[0194] It is worth noting that Figures 4a to 4f Only a few specific examples of the solutions provided in the present application are given. The present application does not limit the value of C to 2 or 4, and the values ​​of h and w are not limited to 4, as long as C, h and w are all positive integers and are integer multiples of 2; the present application does not limit the channel arrangement of each of the K third features, as long as the positions of N pixels at the same position in the N channels of the kth third feature are different in the N channels of the first feature, where N is a positive integer and 2≤N≤C; the present application does not limit the value of K to 3 or 4, as long as K is a positive integer greater than 2.

[0195] Step 1005: recombining the second features to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes a channel of each sub-feature in the second feature, K is a positive integer greater than 2, k is a positive integer and satisfies 1≤k≤K;

[0196] exist Figure 4a , Figure 4b , Figure 4d to Figure 4f In the example given, the shape of each of the K third features is the same, which is [C, h / 2, w / 2]. Specifically, in Figure 4a , Figure 4b and Figure 4e In the example given, the shape of each third feature is [2,2,2], in Figure 4d and Figure 4f In the example given, the shape of each third feature is [4,2,2]. However, the present application does not limit the shape of each third feature in the K third features to be the same. Figure 4c In the example given, the shapes of the first and second third features are the same, both [2,2,2], and the shape of the last third feature is different from the shapes of the first and second third features, which is [4,2,2]. In the method provided in the present application, the number of channels of each of the K third features can be the same or different, but the sum of the number of channels of all the third features of the K third features is required to be equal to 4 times C.

[0197] In a possible implementation of step 1005, the third feature obtained after recombining satisfies the following conditions: the positions of N pixels at the same position in the N channels in the kth third feature are different from each other in the N channels in the first feature, wherein N is a positive integer and 2≤N≤C, wherein the position of the pixel can be represented by the spatial coordinates [x, y], x represents the horizontal coordinate offset of the pixel relative to the upper left corner of the image, and y represents the vertical coordinate offset of the pixel relative to the upper left corner of the image; the position of the pixel can also be indicated by a symbol representing the pixel, for example, in Figures 4a to 4dIn the given example, the upper left, lower right, upper right and lower left pixels of each 2×2 pixel block in the cth channel of the first feature are represented by 0, 1, 2, 3 or 4, 5, 6, 7, respectively, where c is a positive integer and satisfies 1≤c≤C. That is, the upper left pixel of each pixel block in the same channel is represented by 0 or 4, the lower right pixel of each pixel block in the same channel is represented by 1 or 5, the upper right pixel of each pixel block in the same channel is represented by 2 or 5, and the lower left pixel of each pixel block in the same channel is represented by 4 or 7. The N pixels at the same position in the N channels of the kth third feature have different positions in the N channels of the first feature, which means that the pixels included in the N channels exclude the following situations: (1) all are the upper left pixel points in the 2×2 pixel block in the first feature; (2) all are the lower right pixel points in the 2×2 pixel block in the first feature; (3) all are the upper right pixel points in the 2×2 pixel block in the first feature; (4) all are the lower left pixel points in the 2×2 pixel block in the first feature. It is worth noting that the present application only stipulates that at least one of all K third features satisfies the above conditions, and does not require that all K third features satisfy the above conditions. For example, in Figures 4a to 4d In the example given, all K third features satisfy the above conditions, but Figure 4e In the example given, only the last two third features meet the above conditions, while Figure 4f In the example given, only the 1st third feature and the 3rd third feature meet the above conditions.

[0198] When K=4, the kth third feature is composed of the 8c+dth in the second feature 1k +1,8c+d 2k +1,c=0,...,C / 2-1 channels.

[0199] d 1k ,d 2k There are many combinations of (as shown in Table 1 below), and the only requirement is that the combination is not (0,4)→(1,5)→(2,6)→(3,7).

[0200] Table 1 Possible combinations of the third feature when K=4

[0201] Combination number <![CDATA[(d 10 ,d 20 )]]> <![CDATA[(d 11 ,d 21 )]]> <![CDATA[(d 12 ,d 22 )]]> <![CDATA[(d 13 ,d 23 )]]> 1 (0,6) (1,7) (2,4) (3,5) 2 (0,6) (1,7) (2,5) (3,4) 3 (0,6) (1,4) (2,5) (3,7) 4 (0,6) (1,4) (2,7) (3,5) 5 (0,7) (1,6) (2,5) (3,4) 6 (0,7) (1,6) (2,4) (3,5) 7 (0，7) (1，4) (2，5) (3,6) 8 (0,5) (1,7) (2,4) (3,6) 9 (0,5) (1,6) (2,7) (3,4) 10 (0,5) (1,6) (2,4) (3,7) 11 (0,5) (1,4) (2,7) (3,6) 12 (0,4) (1,5) (2,7) (3,6) 13 (0,6) (1,5) (2,4) (3,7) 14 (0,5) (1,6) (2,4) (3,7) ... ... ... ... ...

[0202] exist Figure 4a In the example given, the first combination in Table 1 is used, that is, (0,6)→(1,7)→(2,4)→(3,5);

[0203] exist Figure 4bIn the example given, the fifth combination in Table 1 is used, namely (0,7)→(1,6)→(2,5)→(3,4).

[0204] It is worth noting that this application does not list all the combinations. This application only limits the combinations to not (0,4)→(1,5)→(2,6)→(3,7). In other words, it excludes the situation where each combination of the third features is composed of pixels with the same position in the pixel block of the first feature.

[0205] Figure 4a and Figure 4b The example given is C = 2. When C = 4, Figure 4d In the example given, d 1k ,d 2k The combination of still adopts the first combination in Table 1, that is, (0,6)→(1,7)→(2,4)→(3,5). At this time, the number of channels of each third feature becomes 4. The first third feature is composed of the 1st, 7th, 9th, and 15th channels of the second feature. The second third feature is composed of the 2nd, 8th, 10th, and 16th channels of the second feature. The third third feature is composed of the 3rd, 5th, 11th, and 13th channels of the second feature. The fourth third feature is composed of the 4th, 6th, 12th, and 14th channels of the second feature.

[0206] It should be noted that in the present application, the channel numbering starts from 1 and increases. In some implementations, the channel numbering may also start from 0 and increase. When the channel numbering starts from 0 and increases, all the channel numbers mentioned in the present application need to be subtracted by 1.

[0207] In a possible implementation of step 1005, C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of sub-features in the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of sub-features in the second feature, wherein i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j, the first part of sub-features and the second part of sub-features each include C / 2 sub-features in the second feature, and the first part of sub-features and the second part of sub-features together constitute the second feature. When K=4 and C=2, possible combinations of this implementation are listed in Table 1. Among them, the first channel, the second channel, the third channel and the fourth channel of any sub-feature in the second feature are composed of the upper left, lower right, upper right and lower left pixel points in all 2×2 pixel blocks in the same channel of the first feature. Therefore, by limiting i≠j, it can be ensured that the C / 2 channels in the kth third feature and the remaining C / 2 channels are composed of pixels at different positions in the pixel block in the first feature, thereby ensuring that the division method can make full use of the pixel information adjacent to the same channel and the pixel information at the same position of the surrounding channels when the context model predicts the latent variable, so that the latent variable predicted by the context model is more accurate, the number of bits required to transmit the residual of the latent variable is reduced, and the bit rate is reduced. It is worth noting that the present application only limits at least one third feature among all K third features to meet the above conditions, and does not require all K third features to meet the above conditions. For example, in Figures 4a to 4d In the example given, all K third features satisfy the above conditions, but Figure 4e In the example given, only the last two third features meet the above conditions, while Figure 4f In the example given, only the 1st third feature and the 3rd third feature meet the above conditions.

[0208] In a possible embodiment of the above implementation, K=4, in which case, C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the second channel of each sub-feature in the second part of sub-features. If C=2, then Figure 4a The example given is, that is, the first third feature is composed of the first channel of the first subfeature of the second feature and the third channel of the second subfeature of the second feature; the second third feature is composed of the second channel of the first subfeature of the second feature and the fourth channel of the second subfeature of the second feature; the third third feature is composed of the third channel of the first subfeature of the second feature and the first channel of the second subfeature of the second feature; the fourth third feature is composed of the fourth channel of the first subfeature of the second feature and the second channel of the second subfeature of the second feature.

[0209] In a possible embodiment of the above implementation, K=4, in which case, C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the second channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the first channel of each sub-feature in the second part of sub-features. If C=2, then Figure 4b The example given is, that is, the first third feature is composed of the first channel of the first subfeature of the second feature and the fourth channel of the second subfeature of the second feature; the second third feature is composed of the second channel of the first subfeature of the second feature and the third channel of the second subfeature of the second feature; the third third feature is composed of the third channel of the first subfeature of the second feature and the second channel of the second subfeature of the second feature; the fourth third feature is composed of the fourth channel of the first subfeature of the second feature and the first channel of the second subfeature of the second feature.

[0210] In a possible embodiment of the above implementation, K=4, in which case C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the second channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the first channel of each sub-feature in the second part of sub-features.

[0211] In a possible embodiment of the above implementation, K=4, in which case C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the second channel of each sub-feature in the second part of sub-features.

[0212] In a possible embodiment of the above implementation, K=3, in which case the shapes of the first and second third features are [C, h / 2, w / 2], and the shape of the third feature is [2C, h / 2, w / 2]; C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; C channels in the third third feature are composed of the third and fourth channels of each sub-feature in the first part of sub-features, and the remaining C channels in the third third feature are composed of the first and second channels of each sub-feature in the second part of sub-features.

[0213] In a possible embodiment of the above implementation, K=3, in which case the shapes of the first and third third features are [C, h / 2, w / 2], and the shape of the second third feature is [2C, h / 2, w / 2]; the C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; the C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the second part of sub-features; the C channels in the second third feature are composed of the third and fourth channels of each sub-feature in the first part of sub-features, and the remaining C channels in the second third feature are composed of the first and second channels of each sub-feature in the second part of sub-features.

[0214] In a possible embodiment of the above implementation, K=3, in which case the shapes of the second and third third features are [C, h / 2, w / 2], and the shape of the first third feature is [2C, h / 2, w / 2]; the C / 2 channels in the second third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; the C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the second part of sub-features; the C channels in the first third feature are composed of the third and fourth channels of each sub-feature in the first part of sub-features, and the remaining C channels in the first third feature are composed of the first and second channels of each sub-feature in the second part of sub-features.

[0215] In a possible implementation of step 1005, K=4, and the third features are combined as follows: the first third feature is composed of the 4cth channel in the first sub-feature and the 4c+2th channel in the second sub-feature; the second third feature is composed of the 4c+1th channel in the first sub-feature and the 4c+3th channel in the second sub-feature; the third third feature is composed of the 4c+2th channel in the first sub-feature and the 4cth channel in the second sub-feature; the fourth third feature is composed of the 4c+3th channel in the first sub-feature and the 4c+1th channel in the second sub-feature. Where c is an integer, and the numbering index starts from 0.

[0216] In a possible implementation of step 1005 , K=4, and the shape of each of the K third features is [C, h / 2, w / 2].

[0217] In a possible implementation of step 1005, K=3, shapes of two of the K third features are [C, h / 2, w / 2], and shape of one of the K third features is [2C, h / 2, w / 2]. The present application does not limit the third feature with a channel number of 2C to be the number of the K third features, and it may be the first, second, or third third feature.

[0218] Step 1007, encoding or decoding the image based on the K third features. In a possible implementation of step 1007, encoding or decoding the image based on the K third features includes: for k=1, 2, ..., K, respectively based on the kth third feature, using the context model to predict the kth part of the latent variable of the image; encoding or decoding the image based on the K parts of the latent variable. Specifically, on the encoding side, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, and then the residual of the latent variable is obtained by subtracting the original latent variable from the predicted latent variable, and then the residual of the latent variable is entropy encoded to obtain a bitstream; on the decoding side, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, the bitstream is entropy decoded to obtain the residual of the latent variable, and then the residual of the predicted latent variable and the decoded latent variable are summed to obtain the latent variable, and then the latent variable is decoded by the decoder to obtain the reconstructed image.

[0219] In a possible implementation of the present application, the first feature represents the brightness component and / or chrominance component of the image. When the brightness component and chrominance component of the image are represented by different first features, the combination of the third features corresponding to the brightness component and chrominance component of the image can be the same or different.

[0220] When method 1000 is implemented by an encoder, that is, when method 1000 is an encoding method, the method further includes encoding a first syntax element, where the first syntax element is used to indicate a combination method adopted by the third feature.

[0221] When method 1000 is implemented by a decoder, that is, when method 1000 is a decoding method, the method further includes decoding a first syntax element, where the first syntax element is used to indicate a combination method adopted by the third feature.

[0222] In a possible implementation manner, the first syntax element is encoded in a picture header, or a model header, or a tools header of a code stream.

[0223] In a possible implementation, the value of the first syntax element and the combination in Table 1 have the following corresponding relationship:

[0224]

[0225] In one possible implementation, the encoding side and the decoding side agree that the combinations of the third feature are those in Table 1, then the first syntax element can be transmitted with fewer bits. For example, if there are only four possible combinations of the third feature, then only two bits are needed to transmit the first syntax element.

[0226] In a possible implementation, the encoding side and the decoding side agree that the combination method used by the third feature is a specific combination method in Table 1, and then the first syntax element does not need to be transmitted.

[0227] The encoding and decoding method in the embodiment of the present application is described above. The device in the embodiment of the present application is described below. Figure 5 An embodiment of the encoding device 1200 in the embodiment of the present application includes: an acquisition module 1201, a processing module 1202 and an encoding module 1203.

[0228] The acquisition module 1201 is used to acquire a first feature of an image, wherein the shape of the first feature is [C, h, w], wherein C represents the number of channels of the first feature, h represents the length of the first feature, and w represents the width of the first feature. C, h, and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, and each of the pixel blocks includes 4 pixels in total, namely, upper left, lower right, upper right, and lower left.

[0229] Among them, the introduction of the acquisition module 1201 can refer to the introduction of step 1001 in the above embodiment, and will not be repeated here.

[0230] The processing module 1202 is used to reorder the first feature to obtain a second feature, wherein the shape of the second feature is [4C, h / 2, w / 2], the second feature includes C sub-features, each of the sub-features has a shape of [4, h / 2, w / 2], the c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, c is a positive integer and satisfies 1≤c≤C, and the four channels in the c-th sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all the pixel blocks of the c-th channel of the first feature.

[0231] The processing module 1202 is also used to recombine the second features to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes a channel for each sub-feature in the second feature, K is a positive integer greater than 2, k is a positive integer and satisfies 1≤k≤K.

[0232] It is worth noting that the present application does not limit the value of C to 2 or 4, and the values ​​of h and w are not limited to 4, as long as C, h and w are all positive integers and are integer multiples of 2; the present application does not limit the channel arrangement of each third feature in the K third features, as long as the positions of N pixels at the same position in the N channels of the kth third feature are different in the N channels of the first feature, where N is a positive integer and 2≤N≤C; the present application does not limit the value of K to 3 or 4, as long as K is a positive integer greater than 2.

[0233] In a possible implementation, the third feature obtained after recombining satisfies the following conditions: the positions of N pixels at the same position in the N channels of the kth third feature are different from each other in the N channels of the first feature, wherein N is a positive integer and 2≤N≤C, wherein the position of the pixel can be represented by the spatial coordinates [x, y], x represents the horizontal coordinate offset of the pixel relative to the upper left corner of the image, and y represents the vertical coordinate offset of the pixel relative to the upper left corner of the image; the position of the pixel can also be indicated by the representation symbol of the pixel, for example, in Figures 4a to 4d In the given example, the upper left, lower right, upper right and lower left pixels of each 2×2 pixel block in the cth channel of the first feature are represented by 0, 1, 2, 3 or 4, 5, 6, 7, respectively, where c is a positive integer and satisfies 1≤c≤C. That is, the upper left pixel of each pixel block in the same channel is represented by 0 or 4, the lower right pixel of each pixel block in the same channel is represented by 1 or 5, the upper right pixel of each pixel block in the same channel is represented by 2 or 5, and the lower left pixel of each pixel block in the same channel is represented by 4 or 7. The positions of N pixels at the same position in N channels of the kth third feature are different from each other in the N channels of the first feature, which means that the pixels included in the N channels exclude the following situations: (1) they are all upper left pixels in the 2×2 pixel block in the first feature; (2) they are all lower right pixels in the 2×2 pixel block in the first feature; (3) they are all upper right pixels in the 2×2 pixel block in the first feature; (4) they are all lower left pixels in the 2×2 pixel block in the first feature.

[0234] In a possible implementation, C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of sub-features in the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of sub-features in the second feature, wherein i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j, the first part of sub-features and the second part of sub-features each include C / 2 sub-features in the second feature, and the first part of sub-features and the second part of sub-features together constitute the second feature. The first channel, the second channel, the third channel, and the fourth channel of any sub-feature in the second feature are composed of the upper left, lower right, upper right, and lower left pixel points in all 2×2 pixel blocks in the same channel of the first feature, respectively. Therefore, by limiting i≠j, it can be ensured that the C / 2 channels in the kth third feature and the remaining C / 2 channels are composed of pixel points at different positions in the pixel block in the first feature, thereby ensuring that the division method can make full use of the pixel information adjacent to the same channel and the pixel information at the same position of the surrounding channels when the context model predicts the latent variables, so that the latent variables predicted by the context model are more accurate, reducing the number of bits required to transmit the residual of the latent variables and improving the coding efficiency.

[0235] In a possible embodiment of the above implementation, K=4, in which case, C / 2 channels in the first third feature consist of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature consist of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature consist of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature consist of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature consist of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature consist of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature consist of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature consist of the second channel of each sub-feature in the second part of sub-features. If C=2, then Figure 4aThe example given is, that is, the first third feature is composed of the first channel of the first subfeature of the second feature and the third channel of the second subfeature of the second feature; the second third feature is composed of the second channel of the first subfeature of the second feature and the fourth channel of the second subfeature of the second feature; the third third feature is composed of the third channel of the first subfeature of the second feature and the first channel of the second subfeature of the second feature; the fourth third feature is composed of the fourth channel of the first subfeature of the second feature and the second channel of the second subfeature of the second feature.

[0236] In a possible implementation, K=4, and the third features are combined as follows: the first third feature is composed of the 4cth channel in the first sub-feature and the 4c+2th channel in the second sub-feature; the second third feature is composed of the 4c+1th channel in the first sub-feature and the 4c+3th channel in the second sub-feature; the third third feature is composed of the 4c+2th channel in the first sub-feature and the 4cth channel in the second sub-feature; the fourth third feature is composed of the 4c+3th channel in the first sub-feature and the 4c+1th channel in the second sub-feature. Where c is an integer, and the numbering index starts from 0.

[0237] In a possible implementation, K=4, and the shape of each of the K third features is [C, h / 2, w / 2].

[0238] In one possible implementation, K=3, shapes of two of the K third features are [C, h / 2, w / 2], and shape of one of the K third features is [2C, h / 2, w / 2]. The present application does not limit the third feature with a channel number of 2C to be the number of the K third features, and it may be the first, second, or third third feature.

[0239] Among them, a more detailed introduction of the processing module 1202 can refer to the introduction of steps 1003 and 1005 in the above embodiment, which will not be repeated here.

[0240] The encoding module 1203 is used to encode the image based on the K third features.

[0241] In a possible implementation, encoding the image based on the K third features includes: for k=1, 2, ..., K, respectively based on the kth third feature, using a context model to predict the kth part of the latent variable of the image; encoding the image based on a total of K parts of the latent variable. Specifically, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, and then the residual of the latent variable is obtained by subtracting the original latent variable from the predicted latent variable, and then the residual of the latent variable is entropy encoded to obtain a bitstream.

[0242] Among them, the introduction of the encoding module 1203 can refer to the introduction of step 1007 in the above embodiment, and will not be repeated here.

[0243] In a possible implementation, the first feature represents the brightness component and / or chrominance component of the image. When the brightness component and chrominance component of the image are represented by different first features, the combination of the third features corresponding to the brightness component and chrominance component of the image can be the same or different.

[0244] Optionally, the encoding module 1203 is further used to encode a first syntax element, where the first syntax element is used to indicate a combination method used by the third feature.

[0245] In a possible implementation manner, the first syntax element is encoded in a picture header, or a model header, or a tools header of a code stream.

[0246] See also Figure 6 An embodiment of a decoding device 1300 in an embodiment of the present application includes: an acquisition module 1301, a processing module 1302 and a decoding module 1303.

[0247] The acquisition module 1301 is used to acquire a first feature of an image, wherein the shape of the first feature is [C, h, w], wherein C represents the number of channels of the first feature, h represents the length of the first feature, and w represents the width of the first feature. C, h, and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, and each of the pixel blocks includes 4 pixels in total, namely, upper left, lower right, upper right, and lower left.

[0248] Among them, the introduction of the acquisition module 1301 can refer to the introduction of step 1001 in the above embodiment, and will not be repeated here.

[0249] The processing module 1302 is used to reorder the first feature to obtain a second feature, wherein the shape of the second feature is [4C, h / 2, w / 2], the second feature includes C sub-features, each of the sub-features has a shape of [4, h / 2, w / 2], the c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, c is a positive integer and satisfies 1≤c≤C, and the four channels in the c-th sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all the pixel blocks of the c-th channel of the first feature.

[0250] The processing module 1302 is also used to recombine the second features to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes a channel for each sub-feature in the second feature, K is a positive integer greater than 2, k is a positive integer and satisfies 1≤k≤K.

[0251] Among them, the introduction of the processing module 1302 can refer to the introduction of steps 1003 and 1005 in the above embodiment, and will not be repeated here.

[0252] The decoding module 1303 is used to perform decoding based on the K third features to obtain a reconstructed image.

[0253] In a possible implementation, the decoding of the image based on the K third features includes: for k=1, 2, ..., K, respectively based on the kth third feature, using a context model to predict the kth part of the latent variable of the image; encoding or decoding the image based on a total of K parts of the latent variable. Specifically, the K parts of the predicted latent variable can be combined into a complete predicted latent variable, the bitstream is entropy decoded to obtain the residual of the latent variable, and then the predicted latent variable and the residual of the decoded latent variable are summed to obtain the latent variable, and then the latent variable is decoded by the decoder to obtain the reconstructed image.

[0254] Among them, the introduction of the decoding module 1303 can refer to the introduction of step 1007 in the above embodiment, and will not be repeated here.

[0255] In a possible implementation, the first feature represents the brightness component and / or chrominance component of the image. When the brightness component and chrominance component of the image are represented by different first features, the combination of the third features corresponding to the brightness component and chrominance component of the image can be the same or different.

[0256] Optionally, the decoding module 1303 is further used to decode a first syntax element, where the first syntax element is used to indicate a combination method adopted by the third feature.

[0257] In a possible implementation manner, the first syntax element is encoded in a picture header, or a model header, or a tools header of a code stream.

[0258] Next, a device provided in an embodiment of the present application is introduced. The device may be an encoding device or a decoding device. Figure 7 , Figure 7 A schematic diagram of a structure of a device provided in an embodiment of the present application, the device 1400 may be a virtual reality VR device, a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., which is not limited here. Among them, the device 1400 may be deployed with Figure 5 or Figure 6 Specifically, the device 1400 includes: a receiver 1401, a transmitter 1402, a processor 1403 and a memory 1404 (wherein the number of the processor 1403 in the device 1400 may be one or more, Figure 7 In the example of FIG. 1403 , the processor 1403 may include an application processor 14031 and a communication processor 14032. In some embodiments of the present application, the receiver 1401, the transmitter 1402, the processor 1403 and the memory 1404 may be connected via a bus or other means.

[0259] The memory 1404 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1403. A portion of the memory 1404 may also include a non-volatile random access memory (NVRAM). The memory 1404 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0260] The processor 1403 controls the operation of the device. In a specific application, the various components of the device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0261] The method disclosed in the above embodiment of the present application can be applied to the processor 1403, or implemented by the processor 1403. The processor 1403 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1403. The above processor 1403 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1403 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1404, and the processor 1403 reads the information in the memory 1404 and completes the steps of the above method in combination with its hardware.

[0262] The receiver 1401 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the device. The transmitter 1402 can be used to output digital or character information through the first interface; the transmitter 1402 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1402 can also include a display device such as a display screen.

[0263] The present application also provides a device, which may be a coding device or a decoding device. Figure 8 , Figure 8 This is a schematic diagram of a structure of a device provided in an embodiment of the present application. The device 1500 may be deployed with Figure 5 or Figure 6Corresponding to the apparatus described in the embodiment, specifically, the device 1500 is implemented by one or more servers, and the device 1500 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1522 (for example, one or more processors) and memory 1532, and one or more storage media 1530 (for example, one or more mass storage devices) storing application programs 1542 or data 1544. Among them, the memory 1532 and the storage medium 1530 can be short-term storage or persistent storage. The program stored in the storage medium 1530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the device. Furthermore, the central processing unit 1522 can be configured to communicate with the storage medium 1530 to execute a series of instruction operations in the storage medium 1530 on the device 1500.

[0264] The device 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input and output interfaces 1558, and / or one or more operating systems 1541, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0265] An embodiment of the present application also provides a computer program product (or computer program) storing one or more computers. When the computer program product is executed by the processor, the processor executes the method of the possible implementation mode of the above-mentioned image processing device.

[0266] The embodiment of the present application also provides a chip system, which includes at least one processor for supporting a terminal device or a server to implement the functions involved in the possible implementation of the above-mentioned image processing device. Optionally, the chip system also includes an interface circuit, which provides program instructions and / or data for the at least one processor. In one possible design, the chip system may also include a memory, which is used to store the necessary program instructions and data for the terminal device. The chip system can be composed of chips, or it can include chips and other discrete devices.

[0267] The embodiment of the present application also provides a readable storage medium, which stores the above-mentioned Figures 1a to 6 The code stream obtained by encoding by the method or device in the illustrated embodiment.

[0268] The embodiment of the present application also provides a device for storing a code stream, characterized in that it includes at least one storage medium and a communication interface; the communication interface is used to receive or send a code stream; at least one storage medium is used to store a code stream; the code stream is a coder using the aforementioned Figures 1a to 6 The method or device in the illustrated embodiment is encoded.

[0269] The embodiment of the present application also provides a method for storing a code stream, the method comprising: receiving a code stream through a communication interface; storing the code stream in one or more storage media, the code stream being the code stream of the encoder using the aforementioned Figures 1a to 6 The method or device in the illustrated embodiment is encoded.

[0270] The embodiment of the present application also provides a system for distributing code streams, the system comprising at least one storage medium and a video stream device; at least one storage medium is used to store the code stream, the code stream is the code stream of the encoder using the aforementioned Figures 1a to 6 The method or apparatus in the illustrated embodiment is used to encode the target code stream obtained by encoding; the video stream device is used to respond to the request of the decoder so that the target code stream in at least one storage medium can be sent to the decoder.

[0271] The embodiment of the present application also provides a method for distributing a bitstream, the method comprising: receiving a first request; selecting a target bitstream from at least one storage medium in response to the first request; sending the target bitstream to a destination device; at least one storage medium is used to store the bitstream, the bitstream being the coder using the aforementioned Figures 1a to 6 The method or device in the illustrated embodiment is encoded.

[0272] The embodiment of the present application also provides a system for processing a code stream, the system comprising an image source device, an encoder device, one or more storage media and a destination device; the image source device is used to provide image data; the encoder device is used to obtain the image data of the image source device through an interface, and encode the image data to obtain one or more code streams, the code stream is the encoder using the aforementioned Figures 1a to 6 The method or apparatus in the illustrated embodiment is used for encoding; the encoder device is used to store one or more code streams in one or more storage media; or the encoder device is used to encapsulate one or more code streams to obtain a transmission code stream; the encoder device is used to transmit the transmission code stream to a destination device through a communication link or a communication network; the destination device is used to decapsulate the transmission code stream to obtain one or more code streams; the destination device is used to decode the one or more code streams to obtain decoded data.

[0273] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0274] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Any of the implementation methods provided in the present application can be combined with each other in any way under the premise of logical coherence. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0275] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0276] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0277] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, disk or optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, device, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0278] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0279] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, device or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, device or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a device or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for encoding or decoding an image, characterized in that: include: Acquire a first feature of the image, where the shape of the first feature is [C, h, w], where C represents the number of channels of the first feature, h represents the length of the first feature, and w represents the width of the first feature. C, h, and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, and each of the pixel blocks includes 4 pixels in total, namely, upper left, lower right, upper right, and lower left. Reorder the first feature to obtain a second feature, wherein the shape of the second feature is [4C, h / 2, w / 2], the second feature includes C sub-features, each of the sub-features has a shape of [4, h / 2, w / 2], the c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, c is a positive integer and satisfies 1≤c≤C, and the four channels in the c-th sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all the pixel blocks of the c-th channel of the first feature; Recombining the second features to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes a channel of each sub-feature in the second feature, K is a positive integer greater than 2, k is a positive integer and satisfies 1≤k≤K; The image is encoded or decoded based on the K third features.

2. The method according to claim 1, characterized in that The positions of N pixels at the same position in the N channels in the kth third feature are different from each other in the N channels in the first feature, where N is a positive integer and 2≤N≤C.

3. The method according to claim 1 or 2, characterized in that: The C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of sub-features in the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of sub-features in the second feature, wherein i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j, the first part of sub-features and the second part of sub-features each include C / 2 sub-features in the second feature, and the first part of sub-features and the second part of sub-features together constitute the second feature.

4. The method according to any one of claims 1 to 3, characterized in that K=4, and the shape of each of the K third features is [C, h / 2, w / 2].

5. The method according to claim 1 or 2, characterized in that: K=3, shapes of two third features among the K third features are [C, h / 2, w / 2], and shape of one third feature among the K third features is [2C, h / 2, w / 2].

6. The method according to claim 3, characterized in that K=4, at this time, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features.

7. The method according to claim 6, characterized in that C=2, at this time, The first third feature is composed of the first channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; The second third feature is composed of the second channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; The third third feature is composed of the third channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature; The fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature.

8. The method according to claim 3, characterized in that K=4, at this time, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of sub-features; The C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of sub-features.

9. The method according to claim 3, characterized in that: K=4, at this time, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the second channel of each sub-feature in the second part of sub-features; The C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the first channel of each sub-feature in the second part of sub-features.

10. The method according to claim 3, characterized in that: K=4, at this time, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features.

11. The method according to claim 3, characterized in that K=3, in this case, the shapes of the first third feature and the second third feature are [C, h / 2, w / 2], and the shape of the third third feature is [2C, h / 2, w / 2]; C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the third channel of each sub-feature in the second part of sub-features; The C channels in the third third feature are composed of the 3rd channel and the 4th channel of each sub-feature in the first part of sub-features, and the remaining C channels in the third third feature are composed of the 1st channel and the 2nd channel of each sub-feature in the second part of sub-features.

12. The method according to any one of claims 1 to 11, characterized in that The encoding or decoding of the image based on the K third features includes: For k=1, 2, ..., K, respectively, based on the k-th third feature, using the context model to predict the k-th part of the latent variable of the image; The image is encoded or decoded based on a total of K parts of the latent variables.

13. The method according to any one of claims 1 to 12, characterized in that The first feature represents a luminance component and / or a chrominance component of the image.

14. The method according to any one of claims 1 to 13, characterized in that When the method is a coding method, it also includes: A first syntax element is encoded, where the first syntax element is used to indicate a combination method adopted by the third feature.

15. The method according to any one of claims 1 to 13, characterized in that When the method is a decoding method, it also includes: A first syntax element is decoded, where the first syntax element is used to indicate a combination method adopted by the third feature.

16. An image encoding device, characterized in that: The device comprises: an acquisition module, configured to acquire a first feature of an image, wherein the shape of the first feature is [C, h, w], wherein C represents the number of channels of the first feature, h represents the length of the first feature, and w represents the width of the first feature, and C, h, and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, and each of the pixel blocks includes 4 pixels in total, namely, upper left, lower right, upper right, and lower left. a processing module, configured to reorder the first feature to obtain a second feature, wherein the shape of the second feature is [4C, h / 2, w / 2], the second feature includes C sub-features, each of the sub-features has a shape of [4, h / 2, w / 2], the c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, c is a positive integer and satisfies 1≤c≤C, and the four channels in the c-th sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all the pixel blocks of the c-th channel of the first feature; The processing module is further used to recombine the second features to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes a channel of each sub-feature in the second feature, K is a positive integer greater than 2, and k is a positive integer and satisfies 1≤k≤K; An encoding module is used to encode the image based on the K third features.

17. The encoding device according to claim 16, characterized in that The positions of N pixels at the same position in the N channels in the kth third feature are different from each other in the N channels in the first feature, where N is a positive integer and 2≤N≤C.

18. The encoding device according to claim 16 or 17, characterized in that: The C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of sub-features in the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of sub-features in the second feature, wherein i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j, the first part of sub-features and the second part of sub-features each include C / 2 sub-features in the second feature, and the first part of sub-features and the second part of sub-features together constitute the second feature.

19. The encoding device according to claim 18, characterized in that K=4, at this time, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features.

20. The encoding device according to claim 19, characterized in that C=2, at this time, The first third feature is composed of the first channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; The second third feature is composed of the second channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; The third third feature is composed of the third channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature; The fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature.

21. An image decoding device, characterized in that: The device comprises: an acquisition module, configured to acquire a first feature of an image, wherein the shape of the first feature is [C, h, w], wherein C represents the number of channels of the first feature, h represents the length of the first feature, and w represents the width of the first feature, and C, h, and w are all positive integer multiples of 2. Each channel of the first feature includes (h / 2)×(w / 2) 2×2 pixel blocks, and each of the pixel blocks includes 4 pixels in total, namely, upper left, lower right, upper right, and lower left. a processing module, configured to reorder the first feature to obtain a second feature, wherein the shape of the second feature is [4C, h / 2, w / 2], the second feature includes C sub-features, each of the sub-features has a shape of [4, h / 2, w / 2], the c-th sub-feature of the second feature corresponds to the c-th channel of the first feature, c is a positive integer and satisfies 1≤c≤C, and the four channels in the c-th sub-feature are respectively composed of the upper left, lower right, upper right, and lower left pixel points in all the pixel blocks of the c-th channel of the first feature; The processing module is further used to recombine the second features to obtain K third features, wherein the shape of the kth third feature is [C, h / 2, w / 2], the kth third feature includes a channel of each sub-feature in the second feature, K is a positive integer greater than 2, and k is a positive integer and satisfies 1≤k≤K; A decoding module is used to perform decoding based on the K third features to obtain a reconstructed image.

22. The decoding device according to claim 21, characterized in that The positions of N pixels at the same position in the N channels in the kth third feature are different from each other in the N channels in the first feature, where N is a positive integer and 2≤N≤C.

23. The decoding device according to claim 21 or 22, characterized in that: The C / 2 channels in the kth third feature are composed of the i-th channel of each sub-feature in the first part of sub-features in the second feature, and the remaining C / 2 channels in the kth third feature are composed of the j-th channel of each sub-feature in the second part of sub-features in the second feature, wherein i and j are positive integers, 1≤i≤4, 1≤j≤4, and i≠j, the first part of sub-features and the second part of sub-features each include C / 2 sub-features in the second feature, and the first part of sub-features and the second part of sub-features together constitute the second feature.

24. The decoding device according to claim 23, characterized in that: K=4, at this time, C / 2 channels in the first third feature are composed of the first channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the first third feature are composed of the third channel of each sub-feature in the second part of sub-features; C / 2 channels in the second third feature are composed of the second channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the second third feature are composed of the fourth channel of each sub-feature in the second part of sub-features; C / 2 channels in the third third feature are composed of the third channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the third third feature are composed of the first channel of each sub-feature in the second part of sub-features; C / 2 channels in the fourth third feature are composed of the fourth channel of each sub-feature in the first part of sub-features, and the remaining C / 2 channels in the fourth third feature are composed of the second channel of each sub-feature in the second part of sub-features.

25. The encoding device according to claim 24, characterized in that C=2, at this time, The first third feature is composed of the first channel of the first sub-feature of the second feature and the third channel of the second sub-feature of the second feature; The second third feature is composed of the second channel of the first sub-feature of the second feature and the fourth channel of the second sub-feature of the second feature; The third third feature is composed of the third channel of the first sub-feature of the second feature and the first channel of the second sub-feature of the second feature; The fourth third feature is composed of the fourth channel of the first sub-feature of the second feature and the second channel of the second sub-feature of the second feature.

26. An encoding or decoding device, characterized in that include: A processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the encoding or decoding device executes the method according to any one of claims 1-15.

27. A coding and decoding system, characterized in that: The encoding and decoding system comprises an encoding device as described in any one of claims 16-20, and / or a decoding device as described in any one of claims 21-25.

28. A computer-readable storage medium, characterized in that: The medium stores computer instructions, and when the computer instructions are executed by a computer, the method according to any one of claims 1 to 15 is implemented.

29. A computer program product, characterized in that The method comprises computer instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 15.

30. A computer-readable storage medium, characterized in that: The computer-readable medium stores a code stream encoded according to any one of the encoding methods of claims 1-15.

31. A device for storing a code stream, characterized in that: comprising at least one storage medium and a communication interface; The communication interface is used to receive or send code streams; The at least one storage medium is used to store the code stream; The code stream is obtained by encoding by an encoder according to any encoding method in claims 1-15.

32. A method for storing a code stream, characterized in that: include: Receiving a code stream via a communication interface; The code stream is stored in one or more storage media, and the code stream is an encoder Obtained by encoding according to any one of the encoding methods of claims 1-15.

33. A system for distributing code streams, characterized in that: including at least one storage medium and a video streaming device; The at least one storage medium is used to store a code stream, and the code stream is an encoder Obtained by encoding according to any one of the encoding methods of claims 1-15; The video stream device is used to respond to a request of a decoder so that the code stream in the at least one storage medium can be sent to the decoder.

34. A method for distributing code streams, characterized in that: include: receiving a first request; In response to the first request, selecting a code stream from at least one storage medium; Sending the code stream to a destination device; The at least one storage medium is used to store a code stream, and the code stream is an encoder Obtained by encoding according to any one of the encoding methods of claims 1-15.

35. A system for processing a code stream, characterized in that: It includes an image source device, an encoder device, one or more storage media and a destination device; The image source device is used to provide image data; The encoder device is used to obtain the image data of the image source device through an interface, and encode the image data to obtain one or more code streams, wherein the code streams are encoded by the encoder according to any one of the encoding methods of claims 1-15; The encoder device is used to store the one or more code streams in one or more storage media; or The encoder device is used to encapsulate the one or more code streams to obtain a transmission code stream; The encoder device is used to transmit the transmission code stream to the destination device through a communication link or a communication network; The destination device is used to decapsulate the transport code stream to obtain the one or more code streams; The destination device is used to decode the one or more code streams to obtain decoded data.

Images