Encoding and decoding method and electronic equipment

Abstract

The invention provides a coding and decoding method and electronic equipment. The method comprises the following steps: performing motion estimation on a current frame to obtain a first optical flow; obtaining first optical flow super-prior information based on the feature information of the first optical flow and the prior information of the first optical flow; encoding the first optical flow hyper-prior information to obtain a first code stream; probability estimation is carried out based on the second optical flow hyper-prior information and the prior information of the first optical flow, and optical flow probability distribution is obtained; entropy coding is carried out on the feature information of the first optical flow based on the optical flow probability distribution to obtain a second code stream; performing inter-frame prediction on the current frame to obtain prediction information; determining residual information based on the prediction information and the original image; and encoding the residual information to obtain a third code stream. Compared with the prior art that the determined first optical flow hyper-prior information is used for determining the optical flow probability distribution of all the pixel points in the current frame, the first optical flow hyper-prior information is used for determining the optical flow probability distribution of part of the pixel points in the current frame, and the code rate overhead can be reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of coding and decoding, and in particular to a coding and decoding method and an electronic device. Background Art

[0002] As videos evolve from high-definition to ultra-high-definition, people have higher and higher requirements for video quality. At the same time, high-definition videos have higher and higher requirements for bandwidth and storage. Accordingly, due to cost control considerations for bandwidth, transmission delay, and storage, the demand for video encoding has become more and more urgent.

[0003] Among them, the artificial intelligence (AI) video compression (or codec) algorithm is based on deep learning and has better compression effect than traditional video compression technologies (such as H265 and H266).

[0004] Since there is temporal continuity between the image frames of video data, the amount of change between adjacent frames is relatively small, that is, there is temporal redundancy between video frames; thus, inter-frame coding can be used to reduce the bitrate overhead. In the inter-frame coding process of AI video coding, the residual information is usually entropy encoded according to the probability distribution of the residual information, and the optical flow is entropy encoded according to the probability distribution of the optical flow, as well as the information used to determine the probability distribution of the residual information and the information used to determine the probability distribution of the optical flow; so that the decoding end can decode the residual information and the optical flow for subsequent video reconstruction. Summary of the invention

[0005] In view of this, the present application provides a coding and decoding method and an electronic device, wherein the coding and decoding method can reduce bit rate overhead.

[0006] In a first aspect, an embodiment of the present application provides a coding method, which includes: first, performing motion estimation on a current frame to obtain a first optical flow of the current frame; then, based on the feature information of the first optical flow of the current frame and the prior information of the first optical flow of the current frame, determining the first optical flow super-prior information of the current frame; then, encoding the first optical flow super-prior information of the current frame to obtain a first code stream; then, performing probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain an optical flow probability distribution of the current frame; wherein the second optical flow super-prior information of the current frame is obtained by decoding the first code stream; then, based on the optical flow probability distribution of the current frame, entropy encoding is performed on the feature information of the first optical flow of the current frame to obtain a second code stream; then, performing inter-frame prediction on the current frame to obtain prediction information of the current frame; then, based on the prediction information of the current frame and the original image of the current frame, determining the residual information of the current frame; then, encoding the residual information of the current frame to obtain a third code stream.

[0007] Exemplarily, the prior information of the first optical flow of the current frame may refer to the empirical information (or historical information) that can be obtained before entropy encoding the feature information of the first optical flow of the current frame, which can be used for subsequent inference and decision-making (such as the prior information of the first optical flow of the current frame can be used to determine the probability distribution of the feature information of the first optical flow of K1 pixels in the current frame (i.e., the optical flow probability distribution)). Exemplarily, the super prior information of the first optical flow of the current frame is used to determine the optical flow probability distribution of K2 pixels in the current frame. Wherein, the current frame may include K pixels, K is a positive integer, K1 and K2 are both positive integers less than K, and the union of K1 pixels and K2 pixels is the K pixels included in the current frame. It should be noted that, in one possible method, K1 pixel points and K2 pixel points have no intersection, that is, K1 pixel points and K2 pixel points can be complementary to each other, K=K1+K2; for example, assuming that the current frame includes 100 pixels, K1=K2=50, then the optical flow probability distribution of 50 pixels out of the 100 pixels can be determined by using the prior information of the first optical flow of the current frame alone; the optical flow probability distribution of another 50 pixels out of the 100 pixels can be determined by using the super prior information of the first optical flow of the current frame alone. In one possible approach, K1 pixels and K2 pixels intersect, and K1+K2 is greater than K; for example, assuming that the current frame includes 100 pixels, K1=50, K2=60, then using the prior information of the first optical flow of the current frame alone, the optical flow probability distribution of 40 pixels among the 100 pixels can be determined; using the first optical flow super prior information of the current frame alone, the optical flow probability distribution of another 50 pixels among the 100 pixels can be determined; and combining the first optical flow prior information of the current frame and the first optical flow super prior information of the current frame, the optical flow probability distribution of another 10 pixels among the 100 pixels can be determined.

[0008] Among them, the prior art determines the first optical flow super-prior information of the current frame based on the characteristic information of the first optical flow of the current frame, and the determined first optical flow super-prior information of the current frame is used to determine the probability distribution of the characteristic information of the first optical flow of all pixels in the current frame; while the present application determines the first optical flow super-prior information of the current frame based on the characteristic information of the first optical flow of the current frame and the prior information of the first optical flow of the current frame. Since the prior information of the first optical flow of the current frame can be used to determine the probability distribution of the characteristic information of the first optical flow of some pixels in the current frame, the first optical flow super-prior information of the current frame determined by the present application is used to determine the probability distribution information of the characteristic information of the first optical flow of another part of the pixels in the current frame; it can be seen that compared with the first optical flow super-prior information of the current frame determined by the prior art, the first optical flow super-prior information of the current frame determined by the present application has less data, and thus the present application can reduce the bit rate overhead.

[0009] Exemplarily, the prior information of the first optical flow of the current frame can be determined first, which can be used to determine the probability distribution of the feature information of the first optical flow of which pixels in the current frame; then, the feature information of the first optical flow of the current frame can be determined, which can be used to determine the probability distribution of the feature information of the first optical flow of other pixels in the current frame, and features can be extracted from this information to obtain the super prior information of the first optical flow of the current frame.

[0010] Exemplarily, inter-frame prediction may be performed based on the second optical flow of the current frame to obtain prediction information of the current frame; wherein the second optical flow may be obtained based on decoding of the second bitstream.

[0011] Exemplarily, feature information of the first optical flow of the current frame may be obtained by performing feature extraction on the first optical flow of the current frame.

[0012] Exemplarily, the residual information of the current frame may be the first residual of the current frame (the first residual may refer to the residual between the feature information of the original image of the current frame and the prediction information of the current frame, or the residual between the original image of the current frame and the prediction information of the current frame), or may be the feature information of the first residual of the current frame (which may be obtained by feature extraction of the first residual), and the present application does not impose any limitation on this. The present application takes the residual information of the current frame as the feature information of the first residual of the current frame as an example for explanation.

[0013] It should be noted that the motion estimation of the current frame may be performed based on the reference frame corresponding to the current frame to obtain the first optical flow of the current frame; wherein the first optical flow of the current frame may be understood as the first optical flow of the current frame relative to the corresponding reference frame.

[0014] It should be noted that the first optical flow super-prior information and the second optical flow super-prior information may be the same or different, and this application does not impose any limitation on this.

[0015] It should be noted that, in one possible manner, the third code stream, the second code stream and the first code stream can be three code streams. In one possible manner, the third code stream, the second code stream and the first code stream are three parts of the same code stream, that is, one code stream carries residual information, the first optical stream and the first optical stream super-prior information at the same time. In one possible manner, the third code stream and the second code stream are one code stream, and the first code stream is another code stream. In one possible manner, the third code stream and the first code stream are one code stream, and the second code stream is another code stream. In one possible manner, the first code stream and the second code stream are one code stream, and the third code stream is another code stream; this application does not limit this.

[0016] According to the first aspect, the prior information of the first optical flow of the current frame includes at least one of the time domain prior information of the first optical flow of the current frame or the space domain prior information of the first optical flow of the current frame.

[0017] That is to say, the prior information of the first optical flow of the current frame may only include the time domain prior information of the first optical flow of the current frame; or, the prior information of the first optical flow of the current frame may only include the spatial domain prior information of the first optical flow of the current frame; or, the prior information of the first optical flow of the current frame may include the spatial domain prior information of the first optical flow of the current frame and the time domain prior information of the first optical flow of the current frame.

[0018] Exemplarily, the time domain prior information of the first optical flow of the current frame may refer to information related to the first optical flow of the current frame in the time domain; the space domain prior information of the first optical flow of the current frame may refer to information related to the first optical flow of the current frame in the space domain. In this way, the prior information of the first optical flow of the current frame can be used to determine the probability distribution of feature information of the first optical flow of some pixels in the current frame.

[0019] According to the first aspect, or any implementation of the first aspect above, the time domain prior information of the first optical flow of the current frame includes feature information of the second optical flow of the m frames before the current frame, where m is a positive integer; the spatial domain prior information of the first optical flow of the current frame includes feature information of the reconstructed image of the reference frame corresponding to the current frame.

[0020] Exemplarily, the first m frames of the current frame may refer to image frames whose reconstructed images have a similarity with the original image of the current frame that is higher than a first threshold.

[0021] Exemplarily, when m=1, the time domain prior information of the first optical flow of the current frame may include feature information of the second optical flow of the reference frame corresponding to the current frame.

[0022] According to the first aspect, or any implementation manner of the first aspect above, encoding the residual information of the current frame to obtain a third bitstream includes: determining first residual super prior information of the current frame based on the residual information of the current frame and prior information of the residual information of the current frame; encoding the first residual super prior information of the current frame to obtain a fourth bitstream; performing probability estimation based on the second residual super prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame; wherein the second residual super prior information of the current frame is obtained based on decoding the fourth bitstream; and entropy encoding the residual information of the current frame based on the residual probability distribution of the current frame to obtain a third bitstream.

[0023] Exemplarily, the prior information of the residual information of the current frame may refer to the empirical information (or historical information) that can be obtained before entropy encoding the residual information of the current frame, which can be used for subsequent inference and decision-making (such as the prior information of the residual information of the current frame can be used to determine the probability distribution of the residual information of K3 pixels in the current frame (i.e., the residual probability distribution)). Exemplarily, the first residual information super prior information of the current frame is used to determine the residual probability distribution of K4 pixels in the current frame. Among them, the current frame may include K pixels, K is a positive integer, K3 and K4 are both positive integers less than K, and the union of K3 pixels and K4 pixels is the K pixels included in the current frame. It should be noted that, in one possible method, K3 pixels and K4 pixels have no intersection, that is, K3 pixels and K4 pixels can be complementary to each other, K=K3+K4; for example, assuming that the current frame includes 100 pixels, K3=K4=50, then the prior information of the residual information of the current frame alone can be used to determine the residual probability distribution of 50 pixels among the 100 pixels; the first residual information super prior information of the current frame alone can be used to determine the residual probability distribution of another 50 pixels among the 100 pixels. In one possible approach, K3 pixels and K4 pixels have an intersection, and K3+K4 is greater than K; for example, assuming that the current frame includes 100 pixels, K3=50, K4=60, then using the prior information of the residual information of the current frame alone, the residual probability distribution of 40 pixels among the 100 pixels can be determined; using the first residual information super prior information of the current frame alone, the residual probability distribution of another 50 pixels among the 100 pixels can be determined; and combining the prior information of the residual information of the current frame and the first residual super prior information of the current frame, the residual probability distribution of another 10 pixels among the 100 pixels can be determined.

[0024] Among them, the prior art determines the first residual super-prior information of the current frame based on the residual information of the current frame, and the determined first residual super-prior information of the current frame is used to determine the probability distribution of the residual information of all pixels in the current frame (i.e., the residual probability distribution); while the present application determines the first residual super-prior information of the current frame based on the residual information of the current frame and the prior information of the residual information of the current frame. Since the prior information of the residual information of the current frame can be used to determine the probability distribution of the residual information of some pixels in the current frame, the first residual super-prior information of the current frame determined by the present application is used to determine the probability distribution of the residual information of another part of the pixels in the current frame. It can be seen that compared with the first residual super-prior information of the current frame determined by the prior art, the first residual super-prior information of the current frame determined by the present application has less data, which can further reduce the bit rate overhead.

[0025] It should be noted that the fourth code stream may be independent of the code stream composed of at least one of the first code stream, the second code stream and the third code stream in the above-mentioned first aspect and any implementation of the first aspect, or may be a part of the code stream composed of at least one of the first code stream, the second code stream and the third code stream in the above-mentioned first aspect and any implementation of the first aspect, and the present application does not impose any restriction on this.

[0026] According to the first aspect, or any implementation of the first aspect above, the prior information of the residual information of the current frame includes at least one of the time domain prior information of the residual information of the current frame or the spatial domain prior information of the residual information of the current frame.

[0027] That is to say, the prior information of the residual information of the current frame may only include the time domain prior information of the residual information of the current frame; or, the prior information of the residual information of the current frame may only include the spatial domain prior information of the residual information of the current frame; or, the prior information of the residual information of the current frame may include the spatial domain prior information of the residual information of the current frame and the time domain prior information of the residual information of the current frame.

[0028] Exemplarily, the time domain prior information of the residual information of the current frame may refer to information related to the residual information of the current frame in the time domain; the spatial domain prior information of the residual information of the current frame may refer to information related to the residual information of the current frame in the spatial domain. In this way, the prior information of the residual information of the current frame can be used to determine the probability distribution of the residual information of some pixels in the current frame.

[0029] According to the first aspect, or any implementation of the first aspect above, the time domain prior information of the residual information of the current frame includes the residual information of the n frames before the current frame, where n is a positive integer; the spatial domain prior information of the residual information of the current frame includes the prediction information of the current frame.

[0030] Exemplarily, the first n frames of the current frame may refer to image frames whose reconstructed images have a similarity with the original image of the current frame that is higher than a second threshold.

[0031] Exemplarily, the time domain prior information of the residual information of the current frame includes residual information of n frames before the current frame, where n is a positive integer. When n is equal to 1, the time domain prior information of the residual information of the current frame may be residual information of a reference frame corresponding to the current frame.

[0032] According to the first aspect, or any implementation of the first aspect above, the method is applied to an AI video coding framework, the AI ​​video coding framework includes a motion estimation module, an entropy estimation module, an entropy coding module, a prediction module and a residual coding module, wherein:

[0033] A motion estimation module, used to perform motion estimation on the current frame to obtain a first optical flow of the current frame;

[0034] An entropy estimation module is used to determine the super prior information of the first optical flow of the current frame based on the feature information of the first optical flow of the current frame and the prior information of the first optical flow of the current frame; and output the super prior information of the first optical flow of the current frame to the entropy coding module;

[0035] An entropy coding module, used for encoding the first optical flow super-prior information of the current frame to obtain a first code stream;

[0036] The entropy estimation module is further used to perform probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain the optical flow probability distribution of the current frame; and output the optical flow probability distribution to the entropy coding module;

[0037] The entropy coding module is further used to entropy code the feature information of the first optical flow of the current frame based on the optical flow probability distribution of the current frame to obtain a second code stream;

[0038] The prediction module is used to perform inter-frame prediction on the current frame to obtain prediction information of the current frame; and output the prediction information of the current frame to the residual coding module;

[0039] The residual coding module is used to determine the residual information of the current frame based on the prediction information of the current frame and the original image of the current frame; and output the residual information of the current frame to the entropy coding module;

[0040] The entropy coding module is also used to encode the residual information of the current frame to obtain a third code stream.

[0041] Exemplarily, the entropy estimation module may correspond to Figure 2A The first entropy estimation module in the entropy coding module can correspond to Figure 2A The first entropy coding module in.

[0042] In a second aspect, an embodiment of the present application provides a decoding method, which includes: first, receiving a code stream; wherein the code stream includes a first code stream, a second code stream and a third code stream; then, decoding the first code stream to obtain the second optical flow super-prior information of the current frame; and performing probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain the optical flow probability distribution of the current frame; then, entropy decoding the second code stream based on the optical flow probability distribution of the current frame to obtain feature information of the second optical flow of the current frame; then, performing inter-frame prediction based on the second optical flow of the current frame to obtain prediction information of the current frame; wherein the second optical flow of the current frame is obtained by feature recovery based on the feature information of the second optical flow of the current frame; then, based on the third code stream, obtaining the decoded residual information of the current frame; then, reconstructing based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

[0043] According to the second aspect, the prior information of the first optical flow of the current frame includes at least one of the time domain prior information of the first optical flow of the current frame or the space domain prior information of the first optical flow of the current frame.

[0044] According to the second aspect, or any implementation of the second aspect above, the time domain prior information of the first optical flow of the current frame includes feature information of the second optical flow of the m frames before the current frame, where m is a positive integer; the spatial domain prior information of the first optical flow of the current frame includes feature information of the reconstructed image of the reference frame corresponding to the current frame.

[0045] According to the second aspect, or any implementation manner of the above second aspect, the bitstream also includes a fourth bitstream; based on the third bitstream, obtaining the residual information of the current frame includes: decoding the fourth bitstream to obtain second residual super-prior information of the current frame; performing probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame; entropy decoding the third bitstream based on the residual probability distribution of the current frame to obtain decoded residual information of the current frame.

[0046] According to the second aspect, or any implementation of the second aspect above, the prior information of the residual information of the current frame includes at least one of the time domain prior information of the decoded residual information of the current frame or the spatial domain prior information of the decoded residual information of the current frame.

[0047] According to the second aspect, or any implementation method of the second aspect above, the time domain prior information of the decoded residual information of the current frame includes the decoded residual information of the previous n frames of the current frame, where n is a positive integer; the spatial domain prior information of the decoded residual information of the current frame includes the prediction information of the current frame.

[0048] According to the second aspect, or any implementation of the second aspect above, the method is applied to an AI video decoding framework, the AI ​​video decoding framework includes a motion estimation module, an entropy estimation module, an entropy decoding module, a prediction module and a reconstruction module, wherein:

[0049] An entropy decoding module is used to decode the first code stream to obtain the second optical flow super-prior information of the current frame; and output the second optical flow super-prior information of the current frame to the entropy estimation module;

[0050] An entropy estimation module is used to perform probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain the optical flow probability distribution of the current frame; and output the optical flow probability distribution to the entropy decoding module;

[0051] The entropy decoding module is further used to perform entropy decoding on the second code stream based on the optical flow probability distribution of the current frame to obtain feature information of the second optical flow of the current frame;

[0052] A prediction module, used for performing inter-frame prediction based on the second optical flow of the current frame to obtain prediction information of the current frame; and outputting the prediction information of the current frame to the reconstruction module;

[0053] An entropy decoding module, used for obtaining decoded residual information of the current frame based on the third bit stream; and outputting the decoded residual information of the current frame to the reconstruction module;

[0054] The reconstruction module is used to reconstruct based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

[0055] Exemplarily, the entropy estimation module may correspond to Figure 2B The first entropy estimation module in the entropy decoding module may correspond to Figure 2B The first entropy decoding module in.

[0056] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0057] In a third aspect, an embodiment of the present application also provides a coding method, which includes: first, performing inter-frame prediction on a current frame to obtain prediction information of the current frame; then, determining the residual information of the current frame based on the prediction information of the current frame and the original image of the current frame; and determining the first residual super-prior information of the current frame based on the residual information of the current frame and the prior information of the residual information of the current frame; then, encoding the first residual super-prior information of the current frame to obtain a first code stream; then, performing probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame; wherein the second residual super-prior information of the current frame is obtained based on decoding of the first code stream; then, entropy encoding is performed on the residual information of the current frame based on the residual probability distribution of the current frame to obtain a second code stream.

[0058] Exemplarily, the prior information of the residual information of the current frame is used to determine the residual probability distribution of K3 pixels in the current frame, and the first residual super prior information of the current frame is used to determine the residual probability distribution of K4 pixels in the current frame.

[0059] Among them, the prior art determines the first residual super-prior information of the current frame based on the residual information of the current frame, and the determined first residual super-prior information of the current frame is used to determine the probability distribution of the residual information of all pixels in the current frame (i.e., the residual probability distribution); while the present application determines the first residual super-prior information of the current frame based on the residual information of the current frame and the prior information of the residual information of the current frame. Since the prior information of the residual information of the current frame can be used to determine the probability distribution of the residual information of some pixels in the current frame, the first residual super-prior information of the current frame determined by the present application is used to determine the probability distribution of the residual information of another part of the pixels in the current frame; therefore, compared with the first residual super-prior information of the current frame determined by the prior art, the first residual super-prior information of the current frame determined by the present application has less data, which can further reduce the bit rate overhead.

[0060] It should be noted that the first residual super a priori information and the second residual super a priori information may be the same or different, and this application does not impose any limitation on this.

[0061] It should be noted that, in one possible manner, the first code stream and the second code stream may be two code streams. In one possible manner, the first code stream and the second code stream are two parts of the same code stream, that is, one code stream carries both the residual information and the first residual super-prior information; this application does not impose any restrictions on this.

[0062] According to the third aspect, the prior information of the residual information of the current frame includes at least one of the time domain prior information of the residual information of the current frame or the spatial domain prior information of the residual information of the current frame.

[0063] According to the third aspect, or any implementation method of the third aspect above, the time domain prior information of the residual information of the current frame includes the residual information of the previous n frames of the current frame, where n is a positive integer; the spatial domain prior information of the residual information of the current frame includes the prediction information of the current frame.

[0064] According to the third aspect, or any implementation of the third aspect above, the method is applied to an AI video coding framework, the AI ​​video coding framework includes an entropy estimation module, an entropy coding module, an inter-frame prediction module and a residual coding module, wherein:

[0065] The inter-frame prediction module is used to perform inter-frame prediction on the current frame to obtain prediction information of the current frame; and output the prediction information of the current frame to the residual coding module;

[0066] The residual coding module is used to determine the residual information of the current frame based on the prediction information of the current frame and the original image of the current frame; and output the residual information of the current frame to the entropy estimation module;

[0067] An entropy estimation module is used to determine first residual super prior information of the current frame based on the residual information of the current frame and the prior information of the residual information of the current frame; and output the first residual super prior information of the current frame to the entropy coding module;

[0068] An entropy coding module, used for encoding the first residual super-prior information of the current frame to obtain a first bit stream;

[0069] The entropy estimation module is further used to perform probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain the residual probability distribution of the current frame; and output the residual probability distribution to the entropy coding module;

[0070] The entropy coding module is further used to perform entropy coding on the residual information of the current frame based on the residual probability distribution of the current frame to obtain a second code stream.

[0071] Exemplarily, the entropy estimation module may correspond to Figure 2A The second entropy estimation module in the entropy coding module can correspond to Figure 2A The second entropy coding module in.

[0072] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0073] In a fourth aspect, an embodiment of the present application also provides a decoding method, which includes: first, receiving a bit stream; wherein the bit stream includes a first bit stream and a second bit stream; then, decoding the first bit stream to obtain second residual super-prior information of the current frame; and performing probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame; then, entropy decoding the second bit stream based on the residual probability distribution of the current frame to obtain decoded residual information of the current frame; thereafter, performing inter-frame prediction on the current frame to obtain prediction information of the current frame; then, reconstructing based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

[0074] According to a fourth aspect, the prior information of the residual information of the current frame includes at least one of the time domain prior information of the decoded residual information of the current frame or the spatial domain prior information of the decoded residual information of the current frame.

[0075] According to the fourth aspect, or any implementation method of the fourth aspect above, the time domain prior information of the decoded residual information of the current frame includes the decoded residual information of the previous n frames of the current frame, where n is a positive integer; the spatial domain prior information of the decoded residual information of the current frame includes the prediction information of the current frame.

[0076] According to the fourth aspect, or any implementation of the fourth aspect above, the method is applied to an AI video decoding framework, the AI ​​video decoding framework includes an inter-frame prediction module, an entropy estimation module, an entropy decoding module, a prediction module and a reconstruction module, wherein:

[0077] An entropy decoding module is used to decode the first bit stream to obtain the second residual super-prior information of the current frame; and output the second residual super-prior information of the current frame to the entropy estimation module;

[0078] An entropy estimation module is used to perform probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain the residual probability distribution of the current frame; and output the residual probability distribution to the entropy coding module;

[0079] An entropy coding module is used to perform entropy decoding on the second bit stream based on the residual probability distribution of the current frame to obtain decoded residual information of the current frame; and output the decoded residual information of the current frame to the reconstruction module;

[0080] The inter-frame prediction module is used to perform inter-frame prediction on the current frame to obtain prediction information of the current frame; and output the prediction information of the current frame to the reconstruction module;

[0081] The reconstruction module is used to reconstruct based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

[0082] Exemplarily, the entropy estimation module may correspond to Figure 2B The second entropy estimation module in the entropy decoding module may correspond to Figure 2B The second entropy decoding module in.

[0083] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0084] In a fifth aspect, an embodiment of the present application provides a coding device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, and when the program instructions are executed by the processor, the coding device executes the coding method in the first aspect or any possible implementation of the first aspect; or, executes the coding method in the third aspect or any possible implementation of the third aspect.

[0085] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively; or correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or can refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, which will not be repeated here.

[0086] In a sixth aspect, an embodiment of the present application provides a decoding device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, and when the program instructions are executed by the processor, the decoding device executes the decoding method in the second aspect or any possible implementation of the second aspect; or, executes the decoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0087] The sixth aspect and any implementation of the sixth aspect correspond to the second aspect and any implementation of the second aspect, respectively; or correspond to the fourth aspect and any implementation of the fourth aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or can refer to the technical effects corresponding to the above-mentioned fourth aspect and any implementation of the fourth aspect, which will not be repeated here.

[0088] In the seventh aspect, an embodiment of the present application provides a decoding device, comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the first aspect or any implementation of the first aspect, or the method in the third aspect or any implementation of the third aspect is executed.

[0089] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively; or correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, or can refer to the above-mentioned third aspect and any implementation of the third aspect, which will not be repeated here.

[0090] In an eighth aspect, an embodiment of the present application provides a decoding device, comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the second aspect or any implementation of the second aspect, or the method in the fourth aspect or any implementation of the fourth aspect is executed.

[0091] The eighth aspect and any implementation of the eighth aspect correspond to the second aspect and any implementation of the second aspect, respectively; or correspond to the fourth aspect and any implementation of the fourth aspect, respectively. The technical effects corresponding to the eighth aspect and any implementation of the eighth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or can refer to the above-mentioned fourth aspect and any implementation of the fourth aspect, which will not be repeated here.

[0092] In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect; or, executes the encoding method in the third aspect or any possible implementation of the third aspect.

[0093] The ninth aspect and any implementation of the ninth aspect correspond to the first aspect and any implementation of the first aspect, respectively; or correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the ninth aspect and any implementation of the ninth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or can refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, which will not be repeated here.

[0094] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the encoding method in the second aspect or any possible implementation of the second aspect; or, executes the decoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0095] The tenth aspect and any implementation of the tenth aspect correspond to the second aspect and any implementation of the second aspect, respectively; or correspond to the fourth aspect and any implementation of the fourth aspect, respectively. The technical effects corresponding to the tenth aspect and any implementation of the tenth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or can refer to the above-mentioned fourth aspect and any implementation of the fourth aspect, which will not be repeated here.

[0096] In the eleventh aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect, or executes the encoding method in the third aspect or any possible implementation of the third aspect.

[0097] The eleventh aspect and any implementation of the eleventh aspect correspond to the first aspect and any implementation of the first aspect, respectively; or correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the eleventh aspect and any implementation of the eleventh aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, or can refer to the technical effects corresponding to the above-mentioned third aspect and any implementation of the third aspect, which will not be repeated here.

[0098] In the twelfth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the decoding method in the second aspect or any possible implementation of the second aspect, or executes the decoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0099] The twelfth aspect and any implementation of the twelfth aspect correspond to the second aspect and any implementation of the second aspect, respectively; or correspond to the fourth aspect and any implementation of the fourth aspect, respectively. The technical effects corresponding to the twelfth aspect and any implementation of the twelfth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or can refer to the technical effects corresponding to the above-mentioned fourth aspect and any implementation of the fourth aspect, which will not be repeated here.

[0100] In a thirteenth aspect, an embodiment of the present application provides a device for storing a code stream, the device comprising: a receiver and at least one storage medium, the receiver being used to receive the code stream; the at least one storage medium being used to store the code stream; the code stream is generated according to the first aspect and any one of the implementation methods of the first aspect, or, generated according to the third aspect and any one of the implementation methods of the third aspect.

[0101] The thirteenth aspect and any implementation of the thirteenth aspect respectively correspond to the first aspect and any implementation of the first aspect; or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the thirteenth aspect and any implementation of the thirteenth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, or can refer to the technical effects corresponding to the above-mentioned third aspect and any implementation of the third aspect, which will not be repeated here.

[0102] In a fourteenth aspect, an embodiment of the present application provides a device for transmitting a code stream, the device comprising: a transmitter and at least one storage medium, the at least one storage medium is used to store the code stream, the code stream is generated according to the first aspect and any one of the implementation methods of the first aspect, or, is generated according to the third aspect and any one of the implementation methods of the third aspect; the transmitter is used to obtain the code stream from the storage medium and send the code stream to the terminal side device through the transmission medium.

[0103] The fourteenth aspect and any implementation of the fourteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively; or correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the fourteenth aspect and any implementation of the fourteenth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, or can refer to the technical effects corresponding to the above-mentioned third aspect and any implementation of the third aspect, which will not be repeated here.

[0104] In a fifteenth aspect, an embodiment of the present application provides a system for distributing code streams, the system comprising: at least one storage medium for storing at least one code stream, the at least one code stream being generated according to the first aspect and any one of the implementation methods of the first aspect, or generated according to the third aspect and any one of the implementation methods of the third aspect, a streaming media device for obtaining a target code stream from the at least one storage medium and sending the target code stream to a terminal side device, wherein the streaming media device comprises a content server or a content distribution server.

[0105] The fifteenth aspect and any implementation of the fifteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively; or correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the fifteenth aspect and any implementation of the fifteenth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, or can refer to the technical effects corresponding to the above-mentioned third aspect and any implementation of the third aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0106] Figure 1A is a schematic diagram of an application framework shown as an example;

[0107] Figure 1B A schematic diagram of a storage framework is shown as an example;

[0108] Figure 1C is a schematic diagram of an exemplary transmission framework;

[0109] Figure 2A Schematic diagram of the structure of an AI video encoding framework shown as an example;

[0110] Figure 2B Schematic diagram of the structure of an AI video decoding framework shown as an example;

[0111] Figure 3A is a schematic structural diagram of a first entropy estimation module shown as an example;

[0112] Figure 3B is a schematic diagram of an exemplary encoding process;

[0113] Figure 4 is a schematic diagram of an exemplary decoding process;

[0114] Figure 5A is a schematic structural diagram of a second entropy estimation module shown as an example;

[0115] Figure 5B is a schematic diagram of an exemplary encoding process;

[0116] Figure 6is a schematic diagram of an exemplary decoding process;

[0117] Figure 7 is a schematic diagram of an exemplary encoding process;

[0118] Figure 8 is a schematic diagram of an exemplary decoding process;

[0119] Fig. 9 Schematic diagram of the structure of the device shown as an example. DETAILED DESCRIPTION

[0120] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0121] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0122] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0123] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0124] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0125] Figure 1A The following is a schematic diagram of an application framework shown as an example.

[0126] Reference Figure 1AFor example, a camera can capture video and obtain video data; then, an AI video encoding framework can be used to perform AI video encoding on the video data to obtain a bitstream. Afterwards, the bitstream can be stored locally or transmitted to a remote device.

[0127] Continue to refer to Figure 1A In one possible approach, after the bitstream is stored locally, when the video needs to be played or edited, the AI ​​video decoding framework can be used to perform AI video decoding on the bitstream to obtain reconstructed video data; then, the reconstructed video data can be played or edited.

[0128] Continue to refer to Figure 1A In one possible way, after the remote device receives the bit stream, when it needs to play or edit the video, it can use the AI ​​video decoding framework to perform AI video decoding on the bit stream to obtain reconstructed video data; then, the reconstructed video data can be played or edited.

[0129] Figure 1B A schematic diagram of a storage framework is shown as an example.

[0130] Reference Figure 1B , exemplarily, the AI ​​video encoding framework may include an AI encoding module (also referred to as an AI encoding unit) and an entropy encoding module (also referred to as an entropy encoding module), and the AI ​​video decoding framework may include an AI decoding module (also referred to as an AI decoding unit) and an entropy decoding module (also referred to as an entropy decoding module); it should be understood that, Figure 1B This is only an example of the present application. The AI ​​video encoding framework and the AI ​​video decoding framework may include Figure 1B More modules are shown, but this application does not limit this.

[0131] Exemplarily, the AI ​​encoding module and the AI ​​decoding module can be set in an embedded neural network processor (Neural network Processing Unit, NPU) or a graphics processor (Graphics Processing Unit, GPU). Exemplarily, the entropy encoding module and the entropy decoding module can be set in a central processing unit (Central Processing Unit, CPU). Exemplarily, the file saving module and the file loading module can be set in the CPU.

[0132] Reference Figure 1B, exemplarily, after the Camera captures the video data, it can be input into the AI ​​encoding module. Then, the AI ​​encoding module can perform the following processing for each frame of the video data: perform prediction (including intra-frame prediction and inter-frame prediction, this application takes inter-frame prediction as an example), determine the prediction information; and determine the residual information based on the prediction information and the original image, and determine the probability distribution of the residual information; then, input the residual information and the probability distribution of the residual information into the entropy encoding module. Afterwards, the entropy encoding module can entropy encode the residual information according to the probability distribution of the residual information to obtain a code stream; and input the code stream into the file saving module for saving to obtain a file.

[0133] Continue to refer to Figure 1B For example, when a video needs to be played or edited, the file loading module can load the file, and then input the bitstream in the file into the entropy decoding module. Then, the entropy decoding module can obtain the probability distribution of the residual information from the AI ​​decoding module, and according to the probability distribution of the residual information, entropy decode the residual information from the bitstream and output the residual information to the AI ​​decoding module. After that, the AI ​​decoding module can perform inter-frame prediction to obtain prediction information; then, reconstruct according to the prediction information and the residual information to obtain a reconstructed image; and then, reconstructed video data can be obtained.

[0134] Figure 1C FIG. 1 is a schematic diagram of an exemplary transmission framework.

[0135] Reference Figure 1C , exemplarily, the encoding process of the encoding end may be as follows:

[0136] Exemplarily, after the Camera captures the video data, it can be input into the AI ​​encoding module. Then, the AI ​​encoding module can perform the following processing for each frame of the video data: perform prediction (including intra-frame prediction and inter-frame prediction, this application takes inter-frame prediction as an example), determine the prediction information; and determine the residual information based on the prediction information and the original image, and determine the probability distribution of the residual information; then, input the residual information and the probability distribution of the residual information into the entropy encoding module. Afterwards, the entropy encoding module can entropy encode the residual information according to the probability distribution of the residual information to obtain a bitstream; and send the bitstream to a server in the cloud.

[0137] Afterwards, the server can send the code stream to the decoding end. The server can be a single server or a server cluster, which is not limited in this application.

[0138] Continue to refer to Figure 1C , for example, after the decoding end receives the code stream sent by the server, the decoding process can be as follows:

[0139] Exemplarily, the file loading module can load the file and then send the code stream in the file to the entropy decoding module. Then, the AI ​​decoding network can perform inter-frame prediction to obtain prediction information; after that, the entropy decoding module can obtain the probability distribution of residual information from the AI ​​decoding module, and according to the probability distribution of residual information, entropy decode the residual information from the code stream and output the residual information to the AI ​​decoding module. Subsequently, reconstruction is performed based on the prediction information and residual information to obtain a reconstructed image; and then, reconstructed video data can be obtained.

[0140] It should be understood that the encoding end can also directly send the code stream to the decoding end without forwarding through the server, and this application does not impose any restrictions on this.

[0141] Figure 2A FIG. 1 is a schematic diagram of the structure of an AI video encoding framework. Figure 2A In the embodiment, the inter-frame coding process is shown.

[0142] Reference Figure 2A , exemplarily, the AI ​​encoding module may include a motion estimation module, an optical flow encoding module, a first entropy estimation module, an optical flow decoding module, a prediction module, a feature extraction module, a residual encoding module, a second entropy estimation module, a residual decoding module and a reconstruction module; it should be understood that the AI ​​encoding module may also include Figure 2A The present application is not limited to more or fewer modules shown.

[0143] Reference Figure 2A , exemplarily, the entropy coding module may include a first entropy coding module and a second entropy coding module.

[0144] Reference Figure 2A , exemplarily, the AI ​​video encoding framework may further include an entropy decoding module, and the entropy decoding module may include a first entropy decoding module and a second entropy decoding module.

[0145] It should be noted that Figure 2A The feature extraction module in the dotted box is an optional module.

[0146] It should be noted that the various modules included in the AI ​​video coding framework can be implemented only by a neural network, or by a combination of an algorithm and a neural network; or, some modules included in the AI ​​video coding framework can be implemented only by an algorithm, and this application does not impose any restrictions on this.

[0147] Continue to refer to Figure 2A , taking inter-frame coding of an image frame in video data as an example for explanation; wherein the AI ​​video coding process can be as follows:

[0148] Exemplarily, the original image of the current frame and the reconstructed image of the reference frame corresponding to the current frame can be input into the motion estimation module. Then, the motion estimation module can perform motion estimation based on the original image of the current frame and the reconstructed image of the reference frame to determine the optical flow of the current frame (hereinafter referred to as the first optical flow); and output the first optical flow of the current frame to the optical flow encoding module. The optical flow encoding module (also referred to as the optical flow feature extraction module) can encode the first optical flow of the current frame (it should be noted that the encoding performed by the optical flow encoding module is essentially feature extraction), obtain the feature information of the first optical flow of the current frame (also referred to as the compressed feature of the first optical flow of the current frame) and output it to the first entropy estimation module.

[0149] Exemplarily, the first entropy estimation module can perform probability estimation on the feature information of the first optical flow of the current frame, and determine the probability distribution of the feature information of the first optical flow of the current frame (hereinafter referred to as the optical flow probability distribution of the current frame); then, the feature information of the first optical flow of the current frame and the optical flow probability distribution of the current frame are output to the first entropy coding module.

[0150] Exemplarily, the first entropy coding module may perform entropy coding on feature information of the first optical flow of the current frame according to the optical flow probability distribution of the current frame to obtain an optical flow code stream.

[0151] Exemplarily, on the one hand, the optical flow code stream can be sent / saved (for example, the optical flow code stream is sent to a server or a decoding end). On the other hand, the optical flow code stream can be output to the first entropy decoding module; in addition, the first entropy estimation module can also output the optical flow probability distribution of the current frame to the first entropy decoding module.

[0152] Exemplarily, the first entropy decoding module can entropy decode the optical flow code stream according to the optical flow probability distribution of the current frame to obtain the characteristic information of the first optical flow after decoding of the current frame (which can also be called the characteristic information of the second optical flow of the current frame); and output the characteristic information of the second optical flow of the current frame to the optical flow decoding module.

[0153] Exemplarily, the optical flow decoding module can decode the feature information of the second optical flow of the current frame (it should be noted that the decoding performed by the optical flow decoding module is essentially feature recovery), obtain the second optical flow of the current frame and output the second optical flow of the current frame to the prediction module.

[0154] In one possible manner, when the AI ​​encoding module includes a feature extraction module, the reconstructed image of the reference frame can be input into the feature extraction module, and the feature extraction module extracts the features of the reconstructed image of the reference frame to obtain the feature information of the reconstructed image of the reference frame; and the feature information of the reconstructed image of the reference frame is output to the prediction module. The prediction module can make a prediction based on the second optical flow of the current frame and the feature information of the reconstructed image of the reference frame to obtain the prediction information of the current frame; and the prediction information of the current frame is output to the residual encoding module. Correspondingly, the original image of the current frame can be input into the feature extraction module, and the feature extraction module extracts the features of the original image of the current frame to obtain the feature information of the original image of the current frame, and the feature information of the original image of the current frame is output to the residual encoding module. The residual encoding module outputs the residual information of the current frame (hereinafter referred to as the first residual information of the current frame) to the second entropy estimation module based on the feature information of the original image of the current frame and the prediction information of the current frame. Among them, the first residual information of the current frame can be the first residual of the current frame (wherein, the first residual can refer to the residual between the feature information of the original image of the current frame and the prediction information of the current frame, or it can be the feature information of the first residual of the current frame (wherein, the feature information of the first residual can also be called the compression feature of the first residual, which can be obtained by encoding the first residual of the residual encoding module (it should be noted that the encoding performed by the residual encoding module is essentially feature extraction)), and the present application does not limit this. The present application takes the first residual information of the current frame as the feature information of the first residual of the current frame as an example for explanation.

[0155] In one possible way, when the AI ​​encoding module does not include a feature extraction module, the reconstructed image of the reference frame can be directly input into the prediction module, and the prediction module predicts the current frame based on the second optical flow of the current frame and the reconstructed image of the reference frame to obtain the prediction information of the current frame; and outputs the prediction information of the current frame to the residual encoding module. Correspondingly, the original image of the current frame can be directly input into the residual encoding module; the residual encoding module can output the first residual information of the current frame to the second entropy estimation module based on the original image of the current frame and the prediction information of the current frame.

[0156] Exemplarily, the second entropy estimation module can perform probability estimation on the first residual information of the current frame to determine the probability distribution of the first residual information of the current frame (hereinafter referred to as the residual probability distribution of the current frame); then, the first residual information of the current frame and the residual probability distribution of the current frame are output to the second entropy coding module.

[0157] Exemplarily, the second entropy coding module may perform entropy coding on the first residual information of the current frame according to the residual probability distribution of the current frame to obtain a residual code stream.

[0158] Exemplarily, on the one hand, the residual code stream can be sent / saved (for example, the residual code stream is sent to a server or a decoding end). On the other hand, the residual code stream can be output to the second entropy decoding module; in addition, the second entropy estimation module can also output the residual probability distribution of the current frame to the second entropy decoding module.

[0159] Exemplarily, the second entropy decoding module can entropy decode the residual code stream according to the residual probability distribution of the current frame to obtain the first residual information after entropy decoding of the current frame. When the first residual information of the current frame is the characteristic information of the first residual of the current frame, the first residual information after entropy decoding of the current frame can be output to the residual decoding module.

[0160] Exemplarily, the residual decoding module can decode the first residual information after entropy decoding of the current frame (it should be noted that the decoding performed by the residual decoding module is essentially feature recovery), obtain the second residual information of the current frame and output the second residual information of the current frame to the reconstruction module. It should be noted that when the first residual information of the current frame is the first residual of the current frame, the AI ​​video encoding framework may not include a residual decoding module; at this time, the second residual information of the current frame is the decoded first residual information obtained by entropy decoding of the current frame. In addition, it can also be said that the second residual information of the current frame is obtained by decoding the encoding result of the first residual information of the current frame (i.e., the residual code stream); when the first residual information of the current frame is the feature information of the first residual of the current frame, the decoding in this step includes entropy decoding and feature recovery; when the first residual information of the current frame is the first residual of the current frame, the decoding in this step is entropy decoding.

[0161] Exemplarily, the prediction module can also output the prediction information of the current frame to the reconstruction module. When the AI ​​encoding module includes a feature extraction module, the reconstruction module can first reconstruct according to the second residual information of the current frame and the prediction information of the current frame to obtain the feature information of the reconstructed image of the current frame; then, the reconstruction module can perform feature transformation on the feature information of the reconstructed image of the current frame to obtain the reconstructed image of the current frame. When the AI ​​encoding module does not include a feature extraction module, the reconstruction module reconstructs according to the second residual information of the current frame and the prediction information of the current frame to obtain the reconstructed image of the current frame.

[0162] Exemplarily, the present application may also encode information for determining the optical flow probability distribution of the current frame, and send it to the decoding end; so that the decoding end determines the optical flow probability distribution to obtain the second optical flow of the current frame; and encode information for determining the residual probability distribution of the current frame, and send it to the decoding end; so that the decoding end determines the residual probability distribution to obtain the second residual information of the current frame.

[0163] Figure 2B The figure is a schematic diagram of the structure of an AI video decoding framework shown as an example.

[0164] Reference Figure 2B , exemplarily, the AI ​​decoding module may include: a first entropy estimation module, a second entropy estimation module, an optical flow decoding module, a prediction module, a feature extraction module, a residual decoding module and a reconstruction module; it should be understood that the AI ​​encoding module may also include Figure 2B The present application is not limited to more or fewer modules shown.

[0165] Reference Figure 2B , exemplarily, the entropy decoding module may include a first entropy decoding module and a second entropy encoding module.

[0166] It should be noted that Figure 2B The feature extraction module in the dotted box is an optional module.

[0167] It should be noted that the various modules included in the AI ​​video decoding framework can be implemented only by a neural network, or by a combination of an algorithm and a neural network; or, some modules included in the AI ​​video decoding framework can be implemented only by an algorithm, and this application does not impose any restrictions on this.

[0168] Continue to refer to Figure 2B , taking inter-frame decoding of an image frame in video data as an example for explanation; wherein the AI ​​video decoding process can be as follows:

[0169] Exemplarily, the decoding end may receive a code stream, which may include an optical flow code stream, a residual code stream, a code stream of information for determining an optical flow probability distribution, and a code stream of information for determining a residual probability distribution. The first entropy decoding module may perform entropy decoding on the code stream of information for determining an optical flow probability distribution, obtain information for determining an optical flow probability distribution of a current frame, and output the information for determining an optical flow probability distribution of a current frame to a first entropy estimation module. Then, the first entropy estimation module may perform probability estimation based on the information for determining an optical flow probability distribution of a current frame, determine an optical flow probability distribution of a current frame, and output it to the first entropy decoding module.

[0170] Exemplarily, the first entropy decoding module may perform entropy decoding on the optical flow code stream according to the optical flow probability distribution of the current frame, obtain feature information of the second optical flow of the current frame, and output the feature information to the optical flow decoding module.

[0171] Exemplarily, the optical flow decoding module may perform feature recovery on feature information of the second optical flow of the current frame, obtain the second optical flow of the current frame, and output the second optical flow of the current frame to the prediction module.

[0172] Exemplarily, the second entropy decoding module can perform entropy decoding on the code stream of information used to determine the residual probability distribution, obtain information used to determine the residual probability distribution of the current frame, and output the information used to determine the residual probability distribution of the current frame to the second entropy estimation module; then, the second entropy estimation module performs probability estimation based on the information for determining the residual probability distribution of the current frame, determines the residual probability distribution of the current frame and outputs it to the second entropy decoding module.

[0173] Exemplarily, the second entropy decoding module may perform entropy decoding on the residual code stream according to the residual probability distribution of the current frame, obtain the first residual information after entropy decoding of the current frame, and output it to the residual decoding module.

[0174] Exemplarily, the residual decoding module may perform feature recovery on the first residual information of the current frame after entropy decoding, obtain the second residual information of the current frame, and output the second residual information of the current frame to the reconstruction module.

[0175] In one possible manner, when the AI ​​decoding module includes a feature extraction module, the reconstructed image of the reference frame can be input into the feature extraction module, and the feature extraction module extracts the features of the reconstructed image of the reference frame to obtain feature information of the reconstructed image of the reference frame; and the feature information of the reconstructed image of the reference frame is output to the prediction module. The prediction module can make a prediction based on the second optical flow of the current frame and the feature information of the reconstructed image of the reference frame to obtain the prediction information of the current frame; and the prediction information of the current frame is output to the reconstruction module. Then, the reconstruction module can first reconstruct based on the second residual information of the current frame and the prediction information of the current frame to obtain the feature information of the reconstructed image of the current frame; thereafter, the reconstruction module can perform feature transformation (such as feature recovery) on the feature information of the reconstructed image of the current frame to obtain the reconstructed image of the current frame.

[0176] In one possible way, when the AI ​​decoding module does not include a feature extraction module, the reconstructed image of the reference frame can be directly input into the prediction module, and the prediction module predicts based on the second optical flow of the current frame and the reconstructed image of the reference frame to obtain the prediction information of the current frame; and outputs the prediction information of the current frame to the reconstruction module. Then, the reconstruction module reconstructs based on the second residual information of the current frame and the prediction information of the current frame to obtain the reconstructed image of the current frame.

[0177] It should be understood that the present invention is Figure 2A The AI ​​video encoding framework shown in FIG. 1 includes the names of various modules, and Figure 2B The names of the modules included in the AI ​​video decoding framework shown in the figure are not limited; the modules included in the AI ​​video encoding framework can also be called units; and the modules included in the AI ​​video decoding framework can also be called units.

[0178] It should be noted that in one possible manner, the residual code stream and the optical stream code stream are two code streams. In one possible manner, the residual code stream and the optical stream code stream may be two parts of the same code stream, that is, the same code stream carries the residual information and the first optical stream.

[0179] Figure 3A FIG. 4 is a schematic diagram of the structure of a first entropy estimation module shown as an example.

[0180] Reference Figure 3A Exemplarily, the first entropy estimation module includes a first super priori encoding network, a first super priori decoding network and a first fusion module. And the entropy encoding module in the AI ​​video encoding framework may also include a third entropy encoding module, and the entropy decoding modules in the AI ​​video encoding framework and the AI ​​video decoding framework may also include a third entropy decoding module.

[0181] The following is based on the above Figure 2A , Figure 2B and Figure 3A , the encoding and decoding process of video data is explained.

[0182] Figure 3B FIG. 1 is a schematic diagram of an exemplary encoding process. Figure 3B In the embodiment, inter-frame decoding of an image frame in video data is taken as an example for description.

[0183] S301, performing motion estimation on the current frame to obtain a first optical flow of the current frame.

[0184] Reference Figure 2A For example, the original image of the current frame and the image information of the reference frame corresponding to the current frame (the reconstructed image of the reference frame or the feature information of the reconstructed image of the reference frame) can be input into the motion estimation module, and the motion estimation module calls the motion estimation network to perform motion estimation to obtain the first optical flow of the current frame.

[0185] S302, extracting features from the first optical flow of the current frame to obtain feature information of the first optical flow of the current frame.

[0186] Next, the motion estimation module can output the first optical flow of the current frame to the optical flow encoding module, and the optical flow encoding module outputs the feature information of the first optical flow of the current frame to the first entropy estimation module; thereafter, entropy estimation can be performed on the feature information of the first optical flow of the current frame with reference to the following S303 to S305.

[0187] S303: Determine super-prior information of the first optical flow of the current frame based on the feature information of the first optical flow of the current frame and the prior information of the first optical flow of the current frame.

[0188] Reference Figure 3AExemplarily, the feature information of the first optical flow of the current frame and the prior information of the first optical flow of the current frame can be input into the first super-prior coding network, and the first super-prior coding network encodes the feature information of the first optical flow of the current frame and the prior information of the first optical flow of the current frame (it should be noted that the encoding performed by the first super-prior coding network is essentially feature extraction), and obtains the optical flow super-prior information of the current frame (hereinafter referred to as the first optical flow super-prior information of the current frame, where the first optical flow super-prior information can also be referred to as the first optical flow super-prior feature). Then, the first super-prior coding network can output the first optical flow super-prior information of the current frame to the third entropy coding module.

[0189] Exemplarily, at the encoding end, the prior information of the first optical flow of the current frame may refer to the empirical information (or historical information) that can be obtained before entropy encoding the feature information of the first optical flow of the current frame, which can be used for subsequent inference and decision-making (such as used to determine the probability distribution of the feature information of the first optical flow of K1 pixels in the current frame (i.e., the optical flow probability distribution)).

[0190] Exemplarily, the first super-prior coding network can determine the prior information of the first optical flow of the current frame, which can be used to determine the probability distribution of the feature information of the first optical flow of which pixels in the current frame; then, the feature information of the first optical flow of the current frame can be determined, which can be used to determine the probability distribution of the feature information of the first optical flow of other pixels in the current frame, and feature extraction is performed on this information to obtain the super-prior information of the first optical flow of the current frame. In other words, the super-prior information of the first optical flow of the current frame obtained by feature extraction is used to determine the probability distribution of the feature information of the first optical flow of K2 pixels in the current frame. In this way, the amount of data used to determine the probability distribution of the optical flow (i.e., the first optical flow super-prior information) can be reduced, thereby reducing the bit rate overhead.

[0191] The current frame may include K pixels, K is a positive integer, K1 and K2 are both positive integers less than K, and the union of K1 pixels and K2 pixels is the K pixels included in the current frame. It should be noted that in one possible method, K1 pixels and K2 pixels have no intersection, that is, K1 pixels and K2 pixels can be complementary sets, K = K1 + K2; for example, assuming that the current frame includes 100 pixels, K1 = K2 = 50, then the optical flow probability distribution of 50 pixels out of the 100 pixels can be determined by using the prior information of the first optical flow of the current frame alone; the optical flow probability distribution of another 50 pixels out of the 100 pixels can be determined by using the super prior information of the first optical flow of the current frame alone. In one possible approach, K1 pixels and K2 pixels intersect, and K1+K2 is greater than K; for example, assuming that the current frame includes 100 pixels, K1=50, K2=60, then using the prior information of the first optical flow of the current frame alone, the optical flow probability distribution of 40 pixels among the 100 pixels can be determined; using the first optical flow super prior information of the current frame alone, the optical flow probability distribution of another 50 pixels among the 100 pixels can be determined; and combining the first optical flow prior information of the current frame and the first optical flow super prior information of the current frame, the optical flow probability distribution of another 10 pixels among the 100 pixels can be determined.

[0192] Exemplarily, the prior information of the first optical flow of the current frame includes at least one of the time domain prior information of the first optical flow of the current frame or the spatial domain prior information of the first optical flow of the current frame. That is, the prior information of the first optical flow of the current frame may only include the time domain prior information of the first optical flow of the current frame; or, the prior information of the first optical flow of the current frame may only include the spatial domain prior information of the first optical flow of the current frame; or, the prior information of the first optical flow of the current frame may include the spatial domain prior information of the first optical flow of the current frame and the time domain prior information of the first optical flow of the current frame.

[0193] Exemplarily, the time domain prior information of the first optical flow of the current frame may refer to information related to the first optical flow of the current frame in the time domain; the spatial domain prior information of the first optical flow of the current frame may refer to information related to the first optical flow of the current frame in the spatial domain.

[0194] Exemplarily, the time domain prior information of the first optical flow of the current frame may include feature information of the second optical flow of the m frames before the current frame; wherein m is a positive integer. When m=1, the time domain prior information of the first optical flow of the current frame may include feature information of the second optical flow of the reference frame corresponding to the current frame.

[0195] Exemplarily, the spatial domain prior information of the first optical flow of the current frame may include feature information of a reconstructed image of a reference frame corresponding to the current frame.

[0196] S304, encode the first optical flow super-prior information of the current frame to obtain a first code stream.

[0197] Exemplarily, the third entropy coding module may entropy encode the first optical flow super-prior information of the current frame to obtain a first bitstream. On the one hand, the third entropy coding module may send the first bitstream to a decoding end (or server) or save it locally; on the other hand, the third entropy coding module may output the first bitstream to a third entropy decoding module.

[0198] S305, performing probability estimation based on the second optical flow super prior information of the current frame and the first optical flow prior information of the current frame to obtain the optical flow probability distribution of the current frame; wherein the second optical flow super prior information of the current frame is obtained based on decoding of the first bitstream.

[0199] Next, the third entropy decoding module can entropy decode the first bitstream to obtain the first optical flow super-prior information after entropy decoding of the current frame; and output the first optical flow super-prior information after entropy decoding of the current frame to the first super-prior decoding network. Afterwards, the first super-prior decoding network can decode the first optical flow super-prior information after entropy decoding of the current frame (it should be noted that the encoding performed by the first super-prior decoding network is essentially feature recovery) to obtain the second optical flow super-prior information of the current frame (wherein the second optical flow super-prior information can also be called the second optical flow super-prior feature); and output the second optical flow super-prior information of the current frame to the first fusion module. In other words, the "decoding" based on the first bitstream decoding in S305 includes two steps: entropy decoding and feature recovery.

[0200] Exemplarily, the prior information of the first optical flow of the current frame can also be input into the first fusion module; then, the first fusion module can perform probability estimation based on the super prior information of the second optical flow of the current frame and the prior information of the first optical flow of the current frame to obtain the optical flow probability distribution of the current frame; and the optical flow probability distribution of the current frame is input into the first entropy coding module. The optical flow probability distribution of the current frame is also the probability distribution of the feature information of the first optical flow of the current frame.

[0201] S306 , entropy encoding the feature information of the first optical flow of the current frame based on the optical flow probability distribution of the current frame to obtain a second code stream.

[0202] S307, performing inter-frame prediction on the current frame to obtain prediction information of the current frame.

[0203] Exemplarily, inter-frame prediction may be performed based on the second optical flow of the current frame to obtain prediction information of the current frame; wherein the second optical flow may be obtained based on decoding of the second bitstream.

[0204] It should be noted that the "decoding" in the decoding based on the second bit stream may include two steps: entropy decoding and feature recovery; Figure 2A The description in the embodiments will not be repeated here.

[0205] S308 , determining residual information of the current frame based on the prediction information of the current frame and the original image of the current frame.

[0206] S309, encoding the residual information of the current frame to obtain a third bit stream.

[0207] For example, S306 to S309 can refer to the above Figure 2A The description in the embodiment will not be repeated here. It should be noted that the residual information of the current frame in S308 and S309 refers to Figure 2A The first residual information of the current frame in the embodiment.

[0208] Exemplarily, S309 may include: determining a residual probability distribution of the current frame; and then, performing entropy coding on residual information of the current frame according to the residual probability distribution of the current frame to obtain a third bitstream.

[0209] Exemplarily, the entropy coding module in the AI ​​video coding framework may also include a fifth entropy coding module and a fifth entropy decoding module; the second entropy estimation module includes a third super-prior coding network, a third super-prior decoding network and a third fusion module. Specifically, the process of determining the residual probability distribution of the current frame can be as follows: the residual information of the current frame can be input into the third super-prior coding network to obtain the third residual super-prior information of the current frame (wherein the third residual super-prior information can also be referred to as the third residual super-prior feature); the third residual super-prior information of the current frame is input into the fifth entropy coding module. Then, the fifth entropy coding module entropy encodes the third residual super-prior information of the current frame to obtain a fourth bitstream; on the one hand, the fourth bitstream is sent to the decoding end (or server) or saved locally; on the other hand, the fourth bitstream is output to the fifth entropy decoding module. Afterwards, the fifth entropy decoding module can entropy decode the fourth bitstream to obtain the third residual super-prior information after entropy decoding of the current frame; and output the third residual super-prior information after entropy decoding of the current frame to the third super-prior coding network. Then, the third super prior coding network performs feature recovery on the third residual super prior information after entropy decoding of the current frame to obtain the fourth residual super prior information of the current frame (wherein the fourth residual super prior information may also be referred to as the fourth residual super prior feature); and outputs the fourth residual super prior information of the current frame to the third fusion module. And the prior information of the residual information of the current frame may also be input to the third fusion module, and the third fusion module performs probability estimation based on the prior information of the residual information of the current frame and the fourth residual super prior information of the current frame to determine the residual probability distribution of the current frame.

[0210] Figure 4 FIG. 4 is a schematic diagram showing an exemplary decoding process. Figure 4 The decoding process in the embodiment is the same as Figure 3B The encoding process in the embodiment corresponds.

[0211] S401, receiving code streams; wherein the code streams include a first code stream, a second code stream and a third code stream.

[0212] S402, decoding the first code stream to obtain the second optical flow super-prior information of the current frame.

[0213] Reference Figure 3A , exemplarily, the received first code stream can be input into the third entropy decoding module, and the third entropy decoding module can perform entropy decoding on the first code stream to obtain the first optical flow super-prior information after entropy decoding of the current frame; and output the first optical flow super-prior information after entropy decoding of the current frame to the first super-prior decoding network. Afterwards, the first super-prior decoding network can perform feature recovery on the first optical flow super-prior information after entropy decoding of the current frame to obtain the second optical flow super-prior information of the current frame; and output the second optical flow super-prior information of the current frame to the first fusion module.

[0214] S403, performing probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain an optical flow probability distribution of the current frame.

[0215] Exemplarily, at the decoding end, the prior information of the first optical flow of the current frame may refer to information that can be obtained before entropy decoding the second bitstream (or before obtaining feature information of the first optical flow after entropy decoding of the current frame).

[0216] Exemplarily, the prior information of the first optical flow of the current frame can also be input into the first fusion module; then, the first fusion module can perform probability estimation based on the super prior information of the second optical flow of the current frame and the prior information of the first optical flow of the current frame to obtain the optical flow probability distribution of the current frame; and input the optical flow probability distribution of the current frame into the first entropy coding module.

[0217] S404, entropy decoding is performed on the second bitstream based on the optical flow probability distribution of the current frame to obtain feature information of the second optical flow of the current frame.

[0218] S405, performing feature recovery on feature information of the second optical flow of the current frame to obtain the second optical flow of the current frame.

[0219] S406, performing inter-frame prediction based on the second optical flow of the current frame to obtain prediction information of the current frame; wherein the second optical flow of the current frame is obtained by performing feature recovery based on feature information of the second optical flow of the current frame.

[0220] S407, based on the third bit stream, obtain decoded residual information of the current frame.

[0221] S408, reconstructing based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

[0222] For example, S405 to S408 can refer to the above Figure 2B The description of the embodiment will not be repeated here. It should be noted that the decoded residual information in S407 and S408 may refer to Figure 2B The second residual information of the current frame in the embodiment.

[0223] Exemplarily, the code stream received by the decoding end may also include a fourth code stream; and the entropy decoding module in the AI ​​video decoding framework may also include a fifth entropy decoding module. Among them, based on the third code stream, the process of obtaining the decoded residual information of the current frame can be as follows: the fourth code stream can be input into the fifth entropy decoding module, and the fifth entropy decoding module can entropy decode the fourth code stream to obtain the third residual super-prior information after entropy decoding of the current frame; and output the third residual super-prior information after entropy decoding of the current frame to the third super-prior coding network. Then, the third super-prior coding network performs feature recovery on the third residual super-prior information after entropy decoding of the current frame to obtain the fourth residual super-prior information of the current frame; and output the fourth residual super-prior information of the current frame to the third fusion module. And the prior information of the residual information of the current frame can also be input into the third fusion module, and the third fusion module performs probability estimation based on the prior information of the residual information of the current frame and the fourth residual super-prior information of the current frame to determine the residual probability distribution of the current frame. Afterwards, the third bitstream may be entropy decoded according to the residual probability distribution of the current frame to obtain decoded residual information of the current frame.

[0224] It should be noted that Figure 3A Examples and Figure 4 In the embodiment, the first code stream may refer to the code stream of the first optical flow super-prior information, and the second code stream may refer to the code stream of the optical flow (corresponding to Figure 2A and Figure 2B The third code stream can be a residual code stream (corresponding to Figure 2A and Figure 2B The fourth bitstream may refer to the bitstream of the first residual super-prior information.

[0225] Figure 5A FIG. 4 is a schematic diagram of the structure of a second entropy estimation module shown as an example.

[0226] Reference Figure 5A, exemplarily, the second entropy estimation module also includes a second super priori encoding network, a second super priori decoding network and a second fusion module. And the entropy encoding module in the AI ​​video encoding framework may also include a fourth entropy encoding module, and the entropy decoding module in the AI ​​video encoding framework and the AI ​​video decoding framework may also include a fourth entropy decoding module.

[0227] The following is based on the above Figure 2A , Figure 2B and Figure 5A , the encoding and decoding process of video data is explained.

[0228] Figure 5B FIG. 1 is a schematic diagram of an exemplary encoding process. Figure 5B In the embodiment, inter-frame decoding of an image frame in video data is taken as an example for description.

[0229] S501, performing inter-frame prediction on the current frame to obtain prediction information of the current frame.

[0230] For example, the inter-frame prediction process in S501 can refer to the above Figure 2A The description in the embodiment will not be repeated here. It should be noted that S501 can be performed by the inter-frame prediction module in the AI ​​video coding framework, wherein the inter-frame prediction module in the AI ​​video coding framework may include the above Figure 2A The motion estimation module, optical flow encoding module, first entropy estimation module, optical flow decoding module and prediction module are included in the embodiment.

[0231] Exemplary, exemplary, and the entropy coding module in the AI ​​video coding framework may also include a sixth entropy coding module and a sixth entropy decoding module; the first entropy estimation network also includes a fourth super-prior coding network, a fourth super-prior decoding network and a fourth fusion module. Specifically, the process of determining the probability distribution of optical flow can be as follows: the feature information of the first optical flow of the current frame can be input into the fourth super-prior coding network to obtain the third optical flow super-prior information of the current frame (wherein the third optical flow super-prior information can also be called the third optical flow super-prior feature); the third optical flow super-prior information of the current frame is input into the sixth entropy coding module. Then, the sixth entropy coding module entropy encodes the third optical flow super-prior information of the current frame to obtain a third code stream; on the one hand, the third code stream is sent to the decoding end (or server) or saved locally; on the other hand, the third code stream is output to the sixth entropy decoding module. Afterwards, the sixth entropy decoding module can entropy decode the third code stream to obtain the third optical flow super-prior information after entropy decoding of the current frame; and output the third optical flow super-prior information after entropy decoding of the current frame to the fourth super-prior coding network. Then, the fourth super-prior coding network performs feature recovery on the third optical flow super-prior information after entropy decoding of the current frame to obtain the fourth optical flow super-prior information of the current frame (wherein the fourth optical flow super-prior information may also be referred to as the fourth optical flow super-prior feature); and outputs the fourth optical flow super-prior information of the current frame to the fourth fusion module. The prior information of the first optical flow of the current frame may also be input to the fourth fusion module, and the fourth fusion module performs probability estimation based on the prior information of the first optical flow of the previous frame and the fourth optical flow super-prior information of the current frame to determine the optical flow probability distribution of the current frame.

[0232] Afterwards, based on the optical flow probability distribution of the current frame, entropy coding may be performed on the feature information of the first optical flow of the current frame to obtain a fourth bitstream.

[0233] S502, determining residual information of the current frame based on the prediction information of the current frame and the original image of the current frame.

[0234] For example, S502 may refer to the above Figure 2A The description in the embodiments will not be repeated here.

[0235] S503 : Determine first residual super a priori information of the current frame based on the residual information of the current frame and the a priori information of the residual information of the current frame.

[0236] Reference Figure 5AExemplarily, the residual information of the current frame and the prior information of the residual information of the current frame can be input into the second super-prior coding network, and the second super-prior coding network extracts features from the residual information of the current frame and the prior information of the residual information of the current frame to obtain the first residual super-prior information of the current frame (wherein the first residual super-prior information can also be referred to as the first residual super-prior feature). Then, the second super-prior coding network can output the first residual super-prior information of the current frame to the fourth entropy coding module.

[0237] Exemplarily, at the encoding end, the prior information of the residual information of the current frame may refer to the empirical information (or historical information) that can be obtained before entropy encoding the residual information of the current frame, which can be used for subsequent inference and decision-making (such as used to determine the probability distribution of the residual information of K3 pixels in the current frame (i.e., the residual probability distribution)).

[0238] Exemplarily, the second super prior coding network can determine the prior information of the residual information of the current frame, which can be used to determine the probability distribution of the residual information of which pixels in the current frame; then, the residual information of the current frame can be determined, and the information that can be used to determine the probability distribution of the residual information of other pixels in the current frame can be used to extract features from this information to obtain the first residual super prior information of the current frame. In other words, the first residual super prior information of the current frame obtained by feature extraction is used to determine the probability distribution of the residual information of K4 pixels in the current frame. In this way, the amount of data used to determine the residual probability distribution (i.e., the first residual super prior information) can be reduced, thereby reducing the bit rate overhead.

[0239] Among them, the current frame may include K pixels, K is a positive integer, K3 and K4 are both positive integers less than K, and the union of K3 pixels and K4 pixels is the K pixels included in the current frame. It should be noted that in one possible way, K3 pixels and K4 pixels have no intersection, that is, K3 pixels and K4 pixels can be complementary sets, K = K3 + K4; for example, assuming that the current frame includes 100 pixels, K3 = K4 = 50, then the prior information of the residual information of the current frame is used alone to determine the residual probability distribution of 50 pixels among the 100 pixels, and the first residual information super prior information of the current frame is used alone to determine the residual probability distribution of another 50 pixels among the 100 pixels. In one possible approach, K3 pixels and K4 pixels have an intersection, and K3+K4 is greater than K; for example, assuming that the current frame includes 100 pixels, K3=50, K4=60, then using the prior information of the residual information of the current frame alone, the residual probability distribution of 40 pixels among the 100 pixels can be determined; using the first residual information super prior information of the current frame alone, the residual probability distribution of another 50 pixels among the 100 pixels can be determined; and combining the prior information of the residual information of the current frame and the first residual super prior information of the current frame, the residual probability distribution of another 10 pixels among the 100 pixels can be determined.

[0240] Exemplarily, the prior information of the residual information of the current frame includes at least one of the time domain prior information of the residual information of the current frame or the spatial domain prior information of the residual information of the current frame. In other words, the prior information of the residual information of the current frame may only include the time domain prior information of the residual information of the current frame; or, the prior information of the residual information of the current frame may only include the spatial domain prior information of the residual information of the current frame; or, the prior information of the residual information of the current frame may include the spatial domain prior information of the residual information of the current frame and the time domain prior information of the residual information of the current frame.

[0241] Exemplarily, the time domain prior information of the residual information of the current frame may refer to information related to the residual information of the current frame in the time domain; the spatial domain prior information of the residual information of the current frame may refer to information related to the residual information of the current frame in the spatial domain.

[0242] Exemplarily, the time domain prior information of the residual information of the current frame includes residual information of n frames before the current frame, where n is a positive integer. When n is equal to 1, the time domain prior information of the residual information of the current frame may be residual information of a reference frame corresponding to the current frame.

[0243] Exemplarily, the spatial domain prior information of the residual information of the current frame includes prediction information of the current frame.

[0244] S504, encode the first residual super-prior information of the current frame to obtain a first bit stream.

[0245] Exemplarily, the fourth entropy coding module may perform entropy coding on the first residual super-prior information of the current frame to obtain a first bitstream. On the one hand, the fourth entropy coding module may send the first bitstream to a decoding end (or server) or save it locally; on the other hand, the fourth entropy coding module may output the first bitstream to a fourth entropy decoding module.

[0246] S505, performing probability estimation based on the second residual super priori information of the current frame and the priori information of the residual information of the current frame to obtain the residual probability distribution of the current frame; wherein the second residual super priori information of the current frame is obtained based on decoding of the first bitstream.

[0247] Next, the fourth entropy decoding module can perform entropy decoding on the first bitstream to obtain the first residual super-prior information after entropy decoding of the current frame; and output the first residual super-prior information after entropy decoding of the current frame to the first super-prior decoding network. Afterwards, the first super-prior decoding network can perform feature recovery on the first residual super-prior information after entropy decoding of the current frame to obtain the second residual super-prior information of the current frame (wherein the second residual super-prior information can also be called the second residual super-prior feature); and output the second residual super-prior information of the current frame to the second fusion module. In other words, the "decoding" based on the first bitstream decoding in S505 includes two steps: entropy decoding and feature recovery.

[0248] Exemplarily, the prior information of the residual information of the current frame can also be input into the second fusion module; then, the second fusion module can perform probability estimation based on the second residual super prior information of the current frame and the prior information of the residual information of the current frame to obtain the residual probability distribution of the current frame; and the residual probability distribution of the current frame is input into the second entropy coding module. Among them, the residual probability distribution of the current frame is also the probability distribution of the residual information of the current frame.

[0249] S506: Perform entropy coding on the residual information of the current frame based on the residual probability distribution of the current frame to obtain a second bit stream.

[0250] For example, S506 can refer to the above Figure 2A The description of the embodiments will not be repeated here.

[0251] It should be noted that the residual information of the current frame in S502, S503, S505 and S506 is Figure 2A The first residual information of the current frame.

[0252] Figure 6 FIG. 4 is a schematic diagram showing an exemplary decoding process. Figure 6 The decoding process and Figure 5BThe encoding process corresponds to .

[0253] S601, receiving a code stream; wherein the code stream includes a first code stream and a second code stream.

[0254] S602, decoding the first bit stream to obtain second residual super-prior information of the current frame.

[0255] Exemplarily, the first bitstream can be input to the fourth entropy decoding module, and the fourth entropy decoding module can perform entropy decoding on the first bitstream to obtain the first residual super-prior information after entropy decoding of the current frame; and output the first residual super-prior information after entropy decoding of the current frame to the second super-prior decoding network. Afterwards, the second super-prior decoding network can perform feature recovery on the first residual super-prior information after entropy decoding of the current frame to obtain the second residual super-prior information of the current frame; and output the second residual super-prior information of the current frame to the second fusion module.

[0256] S603 , performing probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame.

[0257] Exemplarily, the prior information of the residual information of the current frame can also be input into the second fusion module; then, the second fusion module can perform probability estimation based on the second residual super prior information of the current frame and the prior information of the residual information of the current frame to obtain the residual probability distribution of the current frame; and the residual probability distribution of the current frame is input into the second entropy coding module. Among them, the residual probability distribution of the current frame is also the probability distribution of the residual information of the current frame.

[0258] S604: Perform entropy decoding on the second bitstream based on the residual probability distribution of the current frame to obtain decoded residual information of the current frame.

[0259] Afterwards, the second entropy coding module may perform entropy decoding on the second bitstream based on the residual probability distribution of the current frame to obtain decoded residual information of the current frame.

[0260] S605, performing inter-frame prediction on the current frame to obtain prediction information of the current frame.

[0261] Exemplarily, the bitstream received by the decoding end also includes: a third bitstream and a fourth bitstream. And the entropy decoding module in the AI ​​video decoding framework may also include a sixth entropy decoding module.

[0262] Exemplarily, the third code stream can be output to the sixth entropy decoding module, and the sixth entropy decoding module can perform entropy decoding on the third code stream to obtain the third optical flow super-prior information after entropy decoding of the current frame; and output the third optical flow super-prior information after entropy decoding of the current frame to the fourth super-prior coding network. Then, the fourth super-prior coding network performs feature recovery on the third optical flow super-prior information after entropy decoding of the current frame to obtain the fourth optical flow super-prior information of the current frame; and output the fourth optical flow super-prior information of the current frame to the fourth fusion module. And the prior information of the feature information of the first optical flow of the current frame can also be input into the fourth fusion module, and the fourth fusion module performs probability estimation based on the prior information of the feature information of the first optical flow of the current frame and the fourth optical flow super-prior information of the current frame to determine the optical flow probability distribution of the current frame; and the optical flow probability distribution of the current frame is input into the first entropy decoding module. Then, the first entropy decoding module entropy decodes the fourth bit stream to obtain the feature information of the first optical flow after entropy decoding of the current frame; and the feature information of the first optical flow after entropy decoding of the current frame is input into the optical flow decoding module, and the optical flow decoding module performs feature recovery on the feature information of the first optical flow after entropy decoding of the current frame to obtain the second optical flow of the current frame. Then, prediction is performed based on the second optical flow of the current frame and the image information of the reference frame to obtain the prediction information of the current frame; refer to the above Figure 2B The description in will not be repeated here.

[0263] Exemplarily, S605 may be performed by an inter-frame prediction module in an AI video decoding framework, wherein the inter-frame prediction module in the AI ​​video decoding framework may include Figure 2B The first entropy estimation module, optical flow decoding module and prediction module in.

[0264] S606, reconstructing based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

[0265] For example, S606 can refer to the above Figure 2B It should be noted that the decoded residual information in S603, S604 and S606 is Figure 2B The second residual information of the current frame in .

[0266] It should be noted that Figure 5A Examples and Figure 6 In the embodiment, the first bitstream may refer to the bitstream of the first residual super-prior information, and the second bitstream may refer to the bitstream of the residual (corresponding to Figure 2A and Figure 2B The third bitstream may be the bitstream of the first optical flow super-prior information, and the fourth bitstream may refer to the bitstream of the optical flow (corresponding to Figure 2A and Figure 2B The optical flow code stream in .

[0267] Figure 7 FIG. 1 is a schematic diagram of an exemplary encoding process.

[0268] S701, performing motion estimation on the current frame to obtain a first optical flow of the current frame.

[0269] S702, extracting features from the first optical flow of the current frame to obtain feature information of the first optical flow of the current frame.

[0270] S703: Determine super a priori information of the first optical flow of the current frame based on the feature information of the first optical flow of the current frame and the a priori information of the first optical flow of the current frame.

[0271] S704, encode the first optical flow super-prior information of the current frame to obtain a first code stream.

[0272] S705, performing probability estimation based on the second optical flow super prior information of the current frame and the first optical flow prior information of the current frame to obtain the optical flow probability distribution of the current frame; wherein the second optical flow super prior information of the current frame is obtained based on decoding of the first bitstream.

[0273] S706 , performing entropy coding on feature information of the first optical flow of the current frame based on the optical flow probability distribution to obtain a second code stream.

[0274] S707, performing inter-frame prediction on the current frame to obtain prediction information of the current frame.

[0275] S708, determining residual information of the current frame based on the prediction information of the current frame and the original image of the current frame.

[0276] S709 , determining first residual super a priori information of the current frame based on the residual information of the current frame and the a priori information of the residual information of the current frame.

[0277] S710, encode the first residual super-prior information of the current frame to obtain a fourth bit stream.

[0278] S711, perform probability estimation based on the second residual super priori information of the current frame and the priori information of the residual information of the current frame to obtain the residual probability distribution of the current frame; wherein the second residual super priori information of the current frame is obtained based on decoding of the fourth bitstream.

[0279] S712, performing entropy coding on the residual information of the current frame based on the residual probability distribution of the current frame to obtain a third bit stream.

[0280] For example, S701 to S712 may refer to the above Figure 3B Examples and Figure 5B The description in the embodiments will not be repeated here.

[0281] Figure 8 FIG. 4 is a schematic diagram showing an exemplary decoding process. Figure 8 The decoding process in the embodiment is Figure 7 The encoding process in the embodiment corresponds.

[0282] S801, receiving code streams; wherein the code streams include a first code stream, a second code stream and a third code stream.

[0283] S802, decoding the first code stream to obtain the second optical flow super-prior information of the current frame.

[0284] S803, performing probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain an optical flow probability distribution of the current frame.

[0285] S804, entropy decoding is performed on the second bitstream based on the optical flow probability distribution of the current frame to obtain feature information of the second optical flow of the current frame.

[0286] S805, performing feature recovery on feature information of the second optical flow of the current frame to obtain the second optical flow of the current frame.

[0287] S806, performing inter-frame prediction based on the second optical flow of the current frame to obtain prediction information of the current frame; wherein the second optical flow of the current frame is obtained by performing feature recovery based on feature information of the second optical flow of the current frame.

[0288] S807, decoding the fourth bit stream to obtain second residual super-prior information of the current frame.

[0289] S808 , performing probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame.

[0290] S809, entropy decoding is performed on the third bitstream based on the residual probability distribution of the current frame to obtain decoded residual information of the current frame.

[0291] S810, reconstructing based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

[0292] For example, S801 to S810 may refer to the above Figure 4 Examples and Figure 6 The description in the embodiments will not be repeated here.

[0293] It should be noted that Figure 7 Examples and Figure 8 In the embodiment, the first code stream may refer to the code stream of the first optical flow super-prior information, and the second code stream may refer to the code stream of the optical flow (corresponding to Figure 2A and Figure 2BThe third code stream can be a residual code stream (corresponding to Figure 2A and Figure 2B The fourth bitstream may refer to the bitstream of the first residual super-prior information.

[0294] The following test set is used Figure 7 The encoding method and Figure 8 The decoding method of the embodiment, the traditional encoding and decoding method of the prior art (such as H266) and the AI ​​video encoding and decoding method of the prior art are tested and compared with the present application. Figure 7 The encoding method and Figure 8 The decoding method of the embodiment and the compression performance of the AI ​​video encoding and decoding method of the prior art.

[0295] For example, the test set includes 5: two 4k test sets (one 4K (A1) test set has a complex motion type, and one 4K test set (A2) has a simple motion type), a 1080p test set, an 832*480 test set (target object motion) and a 720p test set (such as conference video); this application Figure 7 The encoding method and Figure 8 The peak signal-to-noise ratio (PSNR) of the decoding method of the embodiment relative to H266, and the PSNR of the AI ​​video encoding and decoding method of the prior art relative to H266 can be shown in the following Table 1:

[0296] Table 1

[0297]

[0298]

[0299] Referring to Table 1, Y-PSNR represents the PSNR of the Y component, U-PSNR represents the PSNR of the U component, and V-PSNR represents the PSNR of the V component.

[0300] It should be noted that the video data used for testing in the test set contains YUV420 images, in which Y component: U component: V component = 8:1:1; therefore, Table 1 only focuses on Y-PSNR. Figure 7 The encoding method and Figure 8 The decoding method of the embodiment can improve the compression effect by encoding and decoding, that is, at the same bit rate, the reconstructed image quality is higher, or, at the same quality, the bit rate is lower.

[0301] In addition, since the feature information, residual information, and super-prior coding network of the first optical flow have a small scale, the increased amount of calculation is very small; and the decoding complexity of the decoding end is not increased.

[0302] It should be noted that, in the above embodiment, the fourth code stream, the third code stream, the second code stream and the first code stream can be four code streams. In one possible way, the fourth code stream, the third code stream, the second code stream and the first code stream are four parts of the same code stream, that is, one code stream carries the residual information, the first optical stream, the first optical stream super-prior information and the first residual super-prior information at the same time. In one possible way, any two of the fourth code stream, the third code stream, the second code stream and the first code stream can form one code stream, so that the fourth code stream, the third code stream, the second code stream and the first code stream can form two code streams. In one possible way, any three of the fourth code stream, the third code stream, the second code stream and the first code stream can form one code stream, so that the fourth code stream, the third code stream, the second code stream and the first code stream can form two code streams.

[0303] In one example, Fig. 9 A schematic block diagram of a device 900 according to an embodiment of the present application is shown. The device 900 may include: a processor 901 and a transceiver / transceiver pin 902 , and optionally, a memory 903 .

[0304] The components of the device 900 are coupled together via a bus 904, wherein the bus 904 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, all buses are referred to as bus 904 in the figure.

[0305] Optionally, the memory 903 may be used to store instructions in the aforementioned method embodiment. The processor 901 may be used to execute instructions in the memory 903, and control the receiving pin to receive a signal, and control the sending pin to send a signal.

[0306] The apparatus 900 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0307] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0308] The embodiment of the present application also provides a chip, including one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the above-mentioned related method steps implement the steps of the method in the above-mentioned embodiment. The interface circuit is a transceiver / transceiver pin 902.

[0309] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0310] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0311] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the methods in the above-mentioned method embodiments.

[0312] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0313] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0314] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0315] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0316] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0317] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0318] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0319] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (ReadOnly Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disk (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0320] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0321] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A coding method, characterized in that: The method comprises: Performing motion estimation on the current frame to obtain a first optical flow of the current frame; Determine the super-prior information of the first optical flow of the current frame based on the feature information of the first optical flow of the current frame and the prior information of the first optical flow of the current frame; Encoding the first optical flow super-prior information of the current frame to obtain a first bitstream; Probability estimation is performed based on the second optical flow super prior information of the current frame and the first optical flow prior information of the current frame to obtain the optical flow probability distribution of the current frame; wherein the second optical flow super prior information of the current frame is obtained based on decoding of the first bitstream; Performing entropy coding on feature information of a first optical flow of the current frame based on an optical flow probability distribution of the current frame to obtain a second bitstream; Performing inter-frame prediction on the current frame to obtain prediction information of the current frame; Determining residual information of the current frame based on the prediction information of the current frame and the original image of the current frame; Encode the residual information of the current frame to obtain a third bit stream.

2. The method according to claim 1, characterized in that The prior information of the first optical flow of the current frame includes at least one of the time domain prior information of the first optical flow of the current frame or the space domain prior information of the first optical flow of the current frame.

3. The method according to claim 2, characterized in that The time domain prior information of the first optical flow of the current frame includes feature information of the second optical flow of m frames before the current frame, where m is a positive integer; The spatial domain prior information of the first optical flow of the current frame includes feature information of a reconstructed image of a reference frame corresponding to the current frame.

4. The method according to any one of claims 1 to 3, characterized in that: The encoding of the residual information of the current frame to obtain a third bitstream includes: Determine first residual super prior information of the current frame based on the residual information of the current frame and the prior information of the residual information of the current frame; Probability estimation is performed based on the second residual super prior information of the current frame and the prior information of the residual information of the current frame to obtain the residual probability distribution of the current frame; wherein the second residual super prior information of the current frame is obtained based on the decoding of the fourth bitstream; The residual information of the current frame is entropy encoded based on the residual probability distribution of the current frame to obtain the third code stream.

5. The method according to claim 4, characterized in that The prior information of the residual information of the current frame includes at least one of the time domain prior information of the residual information of the current frame or the spatial domain prior information of the residual information of the current frame.

6. The method according to claim 5, characterized in that The time domain prior information of the residual information of the current frame includes the residual information of n frames before the current frame, where n is a positive integer; The spatial domain prior information of the residual information of the current frame includes prediction information of the current frame.

7. The method according to any one of claims 1 to 6, characterized in that: The method is applied to an AI video coding framework, which includes a motion estimation module, an entropy estimation module, an entropy coding module, a prediction module and a residual coding module, wherein: The motion estimation module is used to perform motion estimation on the current frame to obtain a first optical flow of the current frame; The entropy estimation module is used to determine the super prior information of the first optical flow of the current frame based on the feature information of the first optical flow of the current frame and the prior information of the first optical flow of the current frame; and output the super prior information of the first optical flow of the current frame to the entropy coding module; The entropy coding module is used to encode the first optical flow super-prior information of the current frame to obtain the first bit stream; The entropy estimation module is further used to perform probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain the optical flow probability distribution of the current frame; and output the optical flow probability distribution to the entropy coding module; The entropy coding module is further used to perform entropy coding on feature information of the first optical flow of the current frame based on the optical flow probability distribution of the current frame to obtain the second code stream; The prediction module is used to perform inter-frame prediction on the current frame to obtain prediction information of the current frame; and output the prediction information of the current frame to the residual coding module; The residual coding module is used to determine the residual information of the current frame based on the prediction information of the current frame and the original image of the current frame; and output the residual information of the current frame to the entropy coding module; The entropy coding module is further used to encode the residual information of the current frame to obtain the third code stream.

8. A decoding method, characterized in that: The method comprises: Receive a code stream; wherein the code stream includes a first code stream, a second code stream and a third code stream; Decoding the first bitstream to obtain super-prior information of the second optical flow of the current frame; Performing probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain an optical flow probability distribution of the current frame; Performing entropy decoding on the second bitstream based on the optical flow probability distribution of the current frame to obtain feature information of the second optical flow of the current frame; Performing inter-frame prediction based on the second optical flow of the current frame to obtain prediction information of the current frame; wherein the second optical flow of the current frame is obtained by performing feature recovery based on feature information of the second optical flow of the current frame; Based on the third bitstream, obtaining decoded residual information of the current frame; Reconstruction is performed based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

9. The method according to claim 8, characterized in that The prior information of the first optical flow of the current frame includes at least one of the time domain prior information of the first optical flow of the current frame or the space domain prior information of the first optical flow of the current frame.

10. The method according to claim 9, characterized in that The time domain prior information of the first optical flow of the current frame includes feature information of the second optical flow of m frames before the current frame, where m is a positive integer; The spatial domain prior information of the first optical flow of the current frame includes feature information of a reconstructed image of a reference frame corresponding to the current frame.

11. The method according to any one of claims 8 to 10, characterized in that: The bitstream also includes a fourth bitstream; and obtaining the residual information of the current frame based on the third bitstream includes: Decoding the fourth bit stream to obtain second residual super-prior information of the current frame; Probability estimation is performed based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame; The third bitstream is entropy decoded based on the residual probability distribution of the current frame to obtain decoded residual information of the current frame.

12. The method according to claim 11, characterized in that The prior information of the residual information of the current frame includes at least one of time domain prior information of the decoded residual information of the current frame or space domain prior information of the decoded residual information of the current frame.

13. The method according to claim 12, characterized in that The time domain prior information of the decoded residual information of the current frame includes the decoded residual information of n frames before the current frame, where n is a positive integer; The spatial domain prior information of the decoded residual information of the current frame includes prediction information of the current frame.

14. The method according to any one of claims 8 to 13, characterized in that The method is applied to an AI video decoding framework, which includes a motion estimation module, an entropy estimation module, an entropy decoding module, a prediction module and a reconstruction module, wherein: The entropy decoding module is used to decode the first bit stream to obtain the second optical flow super-prior information of the current frame; and output the second optical flow super-prior information of the current frame to the entropy estimation module; The entropy estimation module is used to perform probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain the optical flow probability distribution of the current frame; and output the optical flow probability distribution to the entropy decoding module; The entropy decoding module is further used to perform entropy decoding on the second bitstream based on the optical flow probability distribution of the current frame to obtain feature information of the second optical flow of the current frame; The prediction module is used to perform inter-frame prediction based on the second optical flow of the current frame to obtain prediction information of the current frame; and output the prediction information of the current frame to the reconstruction module; The entropy decoding module is used to obtain the decoded residual information of the current frame based on the third bit stream; and output the decoded residual information of the current frame to the reconstruction module; The reconstruction module is used to reconstruct based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

15. A coding method, characterized in that: The method comprises: Performing inter-frame prediction on the current frame to obtain prediction information of the current frame; Determining residual information of the current frame based on the prediction information of the current frame and the original image of the current frame; Determine first residual super prior information of the current frame based on the residual information of the current frame and the prior information of the residual information of the current frame; Encoding the first residual super-prior information of the current frame to obtain a first bit stream; Probability estimation is performed based on the second residual super prior information of the current frame and the prior information of the residual information of the current frame to obtain the residual probability distribution of the current frame; wherein the second residual super prior information of the current frame is obtained based on decoding of the first bitstream; Perform entropy coding on the residual information of the current frame based on the residual probability distribution of the current frame to obtain a second code stream.

16. The method according to claim 11, characterized in that The prior information of the residual information of the current frame includes at least one of the time domain prior information of the residual information of the current frame or the spatial domain prior information of the residual information of the current frame.

17. The method according to claim 16, characterized in that The time domain prior information of the residual information of the current frame includes the residual information of n frames before the current frame, where n is a positive integer; The spatial domain prior information of the residual information of the current frame includes prediction information of the current frame.

18. The method according to any one of claims 15 to 17, characterized in that The method is applied to an AI video coding framework, which includes an entropy estimation module, an entropy coding module, an inter-frame prediction module and a residual coding module, wherein: The inter-frame prediction module is used to perform inter-frame prediction on the current frame to obtain prediction information of the current frame; and output the prediction information of the current frame to the residual coding module; The residual coding module is used to determine the residual information of the current frame based on the prediction information of the current frame and the original image of the current frame; and output the residual information of the current frame to the entropy estimation module; The entropy estimation module is used to determine the first residual super prior information of the current frame based on the residual information of the current frame and the prior information of the residual information of the current frame; and output the first residual super prior information of the current frame to the entropy coding module; The entropy coding module is used to encode the first residual super-prior information of the current frame to obtain a first bit stream; The entropy estimation module is further used to perform probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain the residual probability distribution of the current frame; and output the residual probability distribution to the entropy coding module; The entropy coding module is further used to perform entropy coding on the residual information of the current frame based on the residual probability distribution of the current frame to obtain the second code stream.

19. A decoding method, characterized in that: The method comprises: Receive a code stream; wherein the code stream includes a first code stream and a second code stream; Decoding the first bitstream to obtain second residual super-prior information of the current frame; Probability estimation is performed based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame; Performing entropy decoding on the second bitstream based on the residual probability distribution of the current frame to obtain decoded residual information of the current frame; Performing inter-frame prediction on the current frame to obtain prediction information of the current frame; Reconstruction is performed based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

20. The method according to claim 19, characterized in that The prior information of the residual information of the current frame includes at least one of time domain prior information of the decoded residual information of the current frame or space domain prior information of the decoded residual information of the current frame.

21. The method according to claim 20, characterized in that The time domain prior information of the decoded residual information of the current frame includes the decoded residual information of n frames before the current frame, where n is a positive integer; The spatial domain prior information of the decoded residual information of the current frame includes prediction information of the current frame.

22. The method according to any one of claims 19 to 21, characterized in that The method is applied to an AI video decoding framework, which includes an inter-frame prediction module, an entropy estimation module, an entropy decoding module, a prediction module and a reconstruction module, wherein: The entropy decoding module is used to decode the first bit stream to obtain the second residual super-prior information of the current frame; and output the second residual super-prior information of the current frame to the entropy estimation module; The entropy estimation module is used to perform probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain the residual probability distribution of the current frame; and output the residual probability distribution to the entropy coding module; The entropy coding module is used to perform entropy decoding on the second bit stream based on the residual probability distribution of the current frame to obtain the decoded residual information of the current frame; and output the decoded residual information of the current frame to the reconstruction module; The inter-frame prediction module is used to perform inter-frame prediction on the current frame to obtain prediction information of the current frame; and output the prediction information of the current frame to the reconstruction module; The reconstruction module is used to reconstruct based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

23. A coding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the encoding device executes the method according to any one of claims 1 to 7 or any one of claims 15 to 18.

24. A decoding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the decoding device performs the method according to any one of claims 8 to 14 or any one of claims 19 to 22.

25. A decoding device, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method as described in any one of claims 1 to 7 or any one of claims 15 to 18 are executed.

26. A decoding device, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method as described in any one of claims 8 to 14 or any one of claims 19 to 22 are executed.

27. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 22.

28. A computer program product, characterized in that The computer program product comprises computer instructions, which, when executed by a computer or a processor, cause the steps of the method according to any one of claims 1 to 22 to be performed.

Images