Coding method, decoding method and electronic device

Abstract

Provided in the present application are a coding method, a decoding method and an electronic device. The coding method comprises: performing motion estimation on a current frame to obtain a first optical flow; on the basis of feature information of the first optical flow and prior information of the first optical flow, obtaining first optical flow hyperprior information; coding the first optical flow hyperprior information to obtain a first data stream; performing probability estimation on the basis of second optical flow hyperprior information and the prior information of the first optical flow to obtain optical flow probability distribution; on the basis of the optical flow probability distribution, performing entropy coding on the feature information of the first optical flow to obtain a second data stream; performing inter-frame prediction on the current frame to obtain prediction information; on the basis of the prediction information and an original image, determining residual information; and coding the residual information to obtain a third data stream. Compared with the prior art that the first optical flow hyperprior information is used for determining the optical flow probability distribution of all pixels in a current frame, the first optical flow hyperprior information of the present application is used for determining the optical flow probability distribution for part of pixels in the current frame, thus reducing data rate overhead.

Description

Coding and decoding method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 15, 2023, with application number 202311035328.2 and application name “Encoding and Decoding Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of coding and decoding, and in particular to a coding and decoding method and an electronic device. Background Art

[0003] As video evolves from HD to UHD, people's expectations for video quality are growing. At the same time, HD video places increasing demands on bandwidth and storage. Consequently, the need for video encoding is becoming increasingly urgent due to cost control considerations for bandwidth, transmission latency, and storage.

[0004] Among them, the artificial intelligence (AI) video compression (or codec) algorithm is based on deep learning and has better compression effect than traditional video compression technologies (such as H265 and H266).

[0005] Since there is temporal continuity between the image frames of video data, the amount of change between adjacent frames is relatively small, which means that there is temporal redundancy between video frames; thus, inter-frame coding can be used to reduce bitrate overhead. In the inter-frame coding process of AI video coding, the residual information is usually entropy coded according to the probability distribution of the residual information, and the optical flow is entropy coded according to the probability distribution of the optical flow, as well as the information used to determine the probability distribution of the residual information and the information used to determine the probability distribution of the optical flow; so that the decoding end can decode the residual information and the optical flow for subsequent video reconstruction.

[0006] Summary of the Invention

[0007] In view of this, the present application provides a coding and decoding method and an electronic device, wherein the coding and decoding method can reduce bit rate overhead.

[0008] In a first aspect, an embodiment of the present application provides an encoding method, which includes: first, performing motion estimation on a current frame to obtain a first optical flow of the current frame; then, determining the first optical flow super-prior information of the current frame based on the feature information of the first optical flow of the current frame and the prior information of the first optical flow of the current frame; then, encoding the first optical flow super-prior information of the current frame to obtain a first code stream; then, performing probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain an optical flow probability distribution of the current frame; wherein the second optical flow super-prior information of the current frame is obtained based on decoding the first code stream; then, entropy encoding the feature information of the first optical flow of the current frame based on the optical flow probability distribution of the current frame to obtain a second code stream; then, performing inter-frame prediction on the current frame to obtain prediction information of the current frame; then, determining the residual information of the current frame based on the prediction information of the current frame and the original image of the current frame; then, encoding the residual information of the current frame to obtain a third code stream.

[0009] Exemplarily, the prior information of the first optical flow of the current frame may refer to the empirical information (or historical information) that can be obtained before entropy encoding the feature information of the first optical flow of the current frame, which can be used for subsequent inference and decision-making (such as the prior information of the first optical flow of the current frame can be used to determine the probability distribution of the feature information of the first optical flow of K1 pixels in the current frame (i.e., the optical flow probability distribution)). Exemplarily, the super prior information of the first optical flow of the current frame is used to determine the optical flow probability distribution of K2 pixels in the current frame. The current frame may include K pixels, K is a positive integer, K1 and K2 are both positive integers less than K, and the union of K1 pixels and K2 pixels is the K pixels included in the current frame. It should be noted that, in one possible method, K1 pixels and K2 pixels have no intersection, that is, K1 pixels and K2 pixels can be complementary to each other, K = K1 + K2; for example, assuming that the current frame includes 100 pixels, K1 = K2 = 50, then the prior information of the first optical flow of the current frame alone can be used to determine the optical flow probability distribution of 50 pixels among the 100 pixels; the super prior information of the first optical flow of the current frame alone can be used to determine the optical flow probability distribution of the other 50 pixels among the 100 pixels. In one possible approach, K1 pixels and K2 pixels intersect, and K1+K2 is greater than K; for example, assuming that the current frame includes 100 pixels, K1=50, K2=60, then using the prior information of the first optical flow of the current frame alone, the optical flow probability distribution of 40 pixels among the 100 pixels can be determined; using the first optical flow super prior information of the current frame alone, the optical flow probability distribution of another 50 pixels among the 100 pixels can be determined; and combining the first optical flow prior information of the current frame and the first optical flow super prior information of the current frame, the optical flow probability distribution of another 10 pixels among the 100 pixels can be determined.

[0010] Among them, the existing technology determines the first optical flow super-prior information of the current frame based on the characteristic information of the first optical flow of the current frame, and the determined first optical flow super-prior information of the current frame is used to determine the probability distribution of the characteristic information of the first optical flow of all pixels in the current frame; while the present application determines the first optical flow super-prior information of the current frame based on the characteristic information of the first optical flow of the current frame and the prior information of the first optical flow of the current frame. Since the prior information of the first optical flow of the current frame can be used to determine the probability distribution of the characteristic information of the first optical flow of some pixels in the current frame, the first optical flow super-prior information of the current frame determined by the present application is used to determine the probability distribution information of the characteristic information of the first optical flow of another part of the pixels in the current frame; it can be seen that compared with the first optical flow super-prior information of the current frame determined by the existing technology, the amount of data of the first optical flow super-prior information of the current frame determined by the present application is less, and thus the present application can reduce the bit rate overhead.

[0011] Exemplarily, the prior information of the first optical flow of the current frame can be determined first, which can be used to determine the probability distribution of the feature information of the first optical flow of which pixels in the current frame; then, the feature information of the first optical flow of the current frame can be determined, which can be used to determine the probability distribution of the feature information of the first optical flow of other pixels in the current frame, and feature extraction is performed on this information to obtain the super prior information of the first optical flow of the current frame.

[0012] Exemplarily, inter-frame prediction may be performed based on the second optical flow of the current frame to obtain prediction information of the current frame; wherein the second optical flow may be obtained based on decoding of the second code stream.

[0013] Exemplarily, feature information of the first optical flow of the current frame may be obtained by performing feature extraction on the first optical flow of the current frame.

[0014] Exemplarily, the residual information of the current frame may be the first residual of the current frame (the first residual may refer to the residual between the feature information of the original image of the current frame and the prediction information of the current frame, or the residual between the original image of the current frame and the prediction information of the current frame), or may be the feature information of the first residual of the current frame (which may be obtained by feature extraction of the first residual). This application does not impose any restrictions on this. This application is described by taking the residual information of the current frame as the feature information of the first residual of the current frame as an example.

[0015] It should be noted that the motion estimation of the current frame can be performed based on the reference frame corresponding to the current frame to obtain the first optical flow of the current frame; wherein the first optical flow of the current frame can be understood as the first optical flow of the current frame relative to the corresponding reference frame.

[0016] It should be noted that the first optical flow super-prior information and the second optical flow super-prior information may be the same or different, and this application does not impose any limitation on this.

[0017] It should be noted that, in one possible embodiment, the third code stream, the second code stream, and the first code stream can be three code streams. In one possible embodiment, the third code stream, the second code stream, and the first code stream are three parts of the same code stream, that is, one code stream simultaneously carries residual information, the first optical stream, and the first optical stream super-prior information. In one possible embodiment, the third code stream and the second code stream are one code stream, and the first code stream is another code stream. In one possible embodiment, the third code stream and the first code stream are one code stream, and the second code stream is another code stream. In one possible embodiment, the first code stream and the second code stream are one code stream, and the third code stream is another code stream; this application does not impose any restrictions on this.

[0018] According to the first aspect, the priori information of the first optical flow of the current frame includes at least one of the time domain priori information of the first optical flow of the current frame or the space domain priori information of the first optical flow of the current frame.

[0019] That is to say, the prior information of the first optical flow of the current frame may only include the time domain prior information of the first optical flow of the current frame; or, the prior information of the first optical flow of the current frame may only include the spatial domain prior information of the first optical flow of the current frame; or, the prior information of the first optical flow of the current frame may include the spatial domain prior information of the first optical flow of the current frame and the time domain prior information of the first optical flow of the current frame.

[0020] For example, the temporal prior information of the first optical flow of the current frame may refer to information related to the first optical flow of the current frame in the temporal domain; and the spatial prior information of the first optical flow of the current frame may refer to information related to the first optical flow of the current frame in the spatial domain. In this way, the prior information of the first optical flow of the current frame can be used to determine the probability distribution of feature information of the first optical flow of some pixels in the current frame.

[0021] According to the first aspect, or any implementation of the first aspect above, the time domain prior information of the first optical flow of the current frame includes feature information of the second optical flow of the m frames before the current frame, where m is a positive integer; the spatial domain prior information of the first optical flow of the current frame includes feature information of the reconstructed image of the reference frame corresponding to the current frame.

[0022] Exemplarily, the m frames before the current frame may refer to image frames whose reconstructed images have a similarity with the original image of the current frame that is higher than a first threshold.

[0023] Exemplarily, when m=1, the time-domain priori information of the first optical flow of the current frame may include feature information of the second optical flow of the reference frame corresponding to the current frame.

[0024] According to the first aspect, or any implementation of the first aspect above, encoding the residual information of the current frame to obtain a third bitstream includes: determining first residual super-prior information of the current frame based on the residual information of the current frame and prior information of the residual information of the current frame; encoding the first residual super-prior information of the current frame to obtain a fourth bitstream; performing probability estimation based on second residual super-prior information of the current frame and prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame; wherein the second residual super-prior information of the current frame is obtained based on decoding the fourth bitstream; and performing entropy encoding on the residual information of the current frame based on the residual probability distribution of the current frame to obtain the third bitstream.

[0025] Exemplarily, the prior information of the residual information of the current frame may refer to the empirical information (or historical information) that can be obtained before entropy encoding the residual information of the current frame, which can be used for subsequent inference and decision-making (such as the prior information of the residual information of the current frame can be used to determine the probability distribution of the residual information of K3 pixels in the current frame (i.e., the residual probability distribution)). Exemplarily, the first residual information super prior information of the current frame is used to determine the residual probability distribution of K4 pixels in the current frame. The current frame may include K pixels, K is a positive integer, K3 and K4 are both positive integers less than K, and the union of K3 pixels and K4 pixels is the K pixels included in the current frame. It should be noted that, in one possible method, K3 pixels and K4 pixels have no intersection, that is, K3 pixels and K4 pixels can be complementary to each other, K = K3 + K4; for example, assuming that the current frame includes 100 pixels, K3 = K4 = 50, then the prior information of the residual information of the current frame alone can be used to determine the residual probability distribution of 50 pixels among the 100 pixels; the first residual information super prior information of the current frame alone can be used to determine the residual probability distribution of the other 50 pixels among the 100 pixels. In one possible approach, K3 pixels and K4 pixels intersect, and K3+K4 is greater than K; for example, assuming that the current frame includes 100 pixels, K3=50, K4=60, then using the prior information of the residual information of the current frame alone, the residual probability distribution of 40 pixels among the 100 pixels can be determined; using the first residual information super prior information of the current frame alone, the residual probability distribution of another 50 pixels among the 100 pixels can be determined; and combining the prior information of the residual information of the current frame and the first residual super prior information of the current frame, the residual probability distribution of another 10 pixels among the 100 pixels can be determined.

[0026] Among them, the existing technology determines the first residual super-prior information of the current frame based on the residual information of the current frame, and the determined first residual super-prior information of the current frame is used to determine the probability distribution of the residual information of all pixel points in the current frame (i.e., the residual probability distribution); while the present application determines the first residual super-prior information of the current frame based on the residual information of the current frame and the prior information of the residual information of the current frame. Since the prior information of the residual information of the current frame can be used to determine the probability distribution of the residual information of some pixel points in the current frame, the first residual super-prior information of the current frame determined by the present application is used to determine the probability distribution of the residual information of another part of the pixel points in the current frame; it can be seen that compared with the first residual super-prior information of the current frame determined by the existing technology, the first residual super-prior information of the current frame determined by the present application has less data volume, which can further reduce the bit rate overhead.

[0027] It should be noted that the fourth code stream may be independent of the code stream consisting of at least one of the first code stream, the second code stream, and the third code stream in the first aspect and any implementation of the first aspect, or may be a part of the code stream consisting of at least one of the first code stream, the second code stream, and the third code stream in the first aspect and any implementation of the first aspect. This application does not impose any restrictions on this.

[0028] According to the first aspect, or any implementation of the first aspect above, the prior information of the residual information of the current frame includes at least one of the time domain prior information of the residual information of the current frame or the spatial domain prior information of the residual information of the current frame.

[0029] That is to say, the prior information of the residual information of the current frame may only include the time domain prior information of the residual information of the current frame; or, the prior information of the residual information of the current frame may only include the spatial domain prior information of the residual information of the current frame; or, the prior information of the residual information of the current frame may include the spatial domain prior information of the residual information of the current frame and the time domain prior information of the residual information of the current frame.

[0030] For example, the temporal prior information of the residual information of the current frame may refer to information related to the residual information of the current frame in the temporal domain; and the spatial prior information of the residual information of the current frame may refer to information related to the residual information of the current frame in the spatial domain. In this way, the prior information of the residual information of the current frame can be used to determine the probability distribution of the residual information of some pixels in the current frame.

[0031] According to the first aspect, or any implementation method of the first aspect above, the time domain prior information of the residual information of the current frame includes the residual information of the n frames before the current frame, where n is a positive integer; the spatial domain prior information of the residual information of the current frame includes the prediction information of the current frame.

[0032] Exemplarily, the first n frames of the current frame may refer to image frames whose reconstructed images have a similarity with the original image of the current frame that is higher than a second threshold.

[0033] Exemplarily, the time domain prior information of the residual information of the current frame includes residual information of n frames before the current frame, where n is a positive integer. When n is 1, the time domain prior information of the residual information of the current frame may be residual information of a reference frame corresponding to the current frame.

[0034] According to the first aspect, or any implementation of the first aspect above, the method is applied to an AI video coding framework, the AI ​​video coding framework includes a motion estimation module, an entropy estimation module, an entropy coding module, a prediction module and a residual coding module, wherein,

[0035] A motion estimation module is used to perform motion estimation on the current frame to obtain the first optical flow of the current frame;

[0036] An entropy estimation module is configured to determine super-prior information of the first optical flow of the current frame based on feature information of the first optical flow of the current frame and prior information of the first optical flow of the current frame; and output the super-prior information of the first optical flow of the current frame to the entropy coding module;

[0037] An entropy coding module is used to encode the first optical flow super-prior information of the current frame to obtain a first bit stream;

[0038] The entropy estimation module is further used to perform probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain the optical flow probability distribution of the current frame; and output the optical flow probability distribution to the entropy coding module;

[0039] The entropy coding module is further configured to perform entropy coding on feature information of the first optical flow of the current frame based on the optical flow probability distribution of the current frame to obtain a second code stream;

[0040] The prediction module is used to perform inter-frame prediction on the current frame to obtain prediction information of the current frame; the prediction information of the current frame is output to the residual coding module;

[0041] The residual coding module is used to determine the residual information of the current frame based on the prediction information of the current frame and the original image of the current frame; and output the residual information of the current frame to the entropy coding module;

[0042] The entropy coding module is also used to encode the residual information of the current frame to obtain a third code stream.

[0043] Exemplarily, the entropy estimation module may correspond to the first entropy estimation module in FIG. 2A , and the entropy coding module may correspond to the first entropy coding module in FIG. 2A .

[0044] In a second aspect, an embodiment of the present application provides a decoding method, comprising: first, receiving a code stream; wherein the code stream includes a first code stream, a second code stream, and a third code stream; then, decoding the first code stream to obtain a second optical flow super-prior information of the current frame; and performing probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain an optical flow probability distribution of the current frame; then, entropy decoding the second code stream based on the optical flow probability distribution of the current frame to obtain feature information of the second optical flow of the current frame; then, performing inter-frame prediction based on the second optical flow of the current frame to obtain prediction information of the current frame; wherein the second optical flow of the current frame is obtained by feature recovery based on the feature information of the second optical flow of the current frame; then, obtaining decoded residual information of the current frame based on the third code stream; then, reconstructing based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

[0045] According to the second aspect, the prior information of the first optical flow of the current frame includes at least one of the time domain prior information of the first optical flow of the current frame or the space domain prior information of the first optical flow of the current frame.

[0046] According to the second aspect, or any implementation method of the above second aspect, the time domain prior information of the first optical flow of the current frame includes the feature information of the second optical flow of the m frames before the current frame, where m is a positive integer; the spatial domain prior information of the first optical flow of the current frame includes the feature information of the reconstructed image of the reference frame corresponding to the current frame.

[0047] According to the second aspect, or any implementation of the second aspect above, the bitstream further includes a fourth bitstream; and obtaining residual information of the current frame based on the third bitstream includes: decoding the fourth bitstream to obtain second residual super-a priori information of the current frame; performing probability estimation based on the second residual super-a priori information of the current frame and prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame; and performing entropy decoding on the third bitstream based on the residual probability distribution of the current frame to obtain decoded residual information of the current frame.

[0048] According to the second aspect, or any implementation of the second aspect above, the prior information of the residual information of the current frame includes at least one of the time domain prior information of the decoded residual information of the current frame or the spatial domain prior information of the decoded residual information of the current frame.

[0049] According to the second aspect, or any implementation method of the above second aspect, the time domain prior information of the decoded residual information of the current frame includes the decoded residual information of the previous n frames of the current frame, where n is a positive integer; the spatial domain prior information of the decoded residual information of the current frame includes the prediction information of the current frame.

[0050] According to the second aspect, or any implementation of the second aspect above, the method is applied to an AI video decoding framework, the AI ​​video decoding framework includes a motion estimation module, an entropy estimation module, an entropy decoding module, a prediction module and a reconstruction module, wherein,

[0051] An entropy decoding module is used to decode the first code stream to obtain the second optical flow super-prior information of the current frame; and output the second optical flow super-prior information of the current frame to the entropy estimation module;

[0052] An entropy estimation module is used to perform probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain the optical flow probability distribution of the current frame; and output the optical flow probability distribution to the entropy decoding module;

[0053] The entropy decoding module is further configured to perform entropy decoding on the second bitstream based on the optical flow probability distribution of the current frame to obtain feature information of the second optical flow of the current frame;

[0054] A prediction module is configured to perform inter-frame prediction based on the second optical flow of the current frame to obtain prediction information of the current frame; and output the prediction information of the current frame to a reconstruction module;

[0055] An entropy decoding module is used to obtain decoded residual information of the current frame based on the third code stream; and output the decoded residual information of the current frame to the reconstruction module;

[0056] The reconstruction module is used to reconstruct the image based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

[0057] Exemplarily, the entropy estimation module may correspond to the first entropy estimation module in FIG. 2B , and the entropy decoding module may correspond to the first entropy decoding module in FIG. 2B .

[0058] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0059] In a third aspect, an embodiment of the present application also provides an encoding method, which includes: first, performing inter-frame prediction on the current frame to obtain prediction information of the current frame; then, determining the residual information of the current frame based on the prediction information of the current frame and the original image of the current frame; and determining the first residual super-prior information of the current frame based on the residual information of the current frame and the prior information of the residual information of the current frame; then, encoding the first residual super-prior information of the current frame to obtain a first code stream; then, performing probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame; wherein the second residual super-prior information of the current frame is obtained based on decoding the first code stream; then, entropy encoding is performed on the residual information of the current frame based on the residual probability distribution of the current frame to obtain a second code stream.

[0060] Exemplarily, the prior information of the residual information of the current frame is used to determine the residual probability distribution of K3 pixels in the current frame, and the first residual super prior information of the current frame is used to determine the residual probability distribution of K4 pixels in the current frame.

[0061] Among them, the existing technology determines the first residual super-prior information of the current frame based on the residual information of the current frame, and the determined first residual super-prior information of the current frame is used to determine the probability distribution of the residual information of all pixel points in the current frame (i.e., the residual probability distribution); while the present application determines the first residual super-prior information of the current frame based on the residual information of the current frame and the prior information of the residual information of the current frame. Since the prior information of the residual information of the current frame can be used to determine the probability distribution of the residual information of some pixel points in the current frame, the first residual super-prior information of the current frame determined by the present application is used to determine the probability distribution of the residual information of another part of the pixel points in the current frame; therefore, compared with the first residual super-prior information of the current frame determined by the existing technology, the first residual super-prior information of the current frame determined by the present application has less data volume, which can further reduce the bit rate overhead.

[0062] It should be noted that the first residual super-prior information and the second residual super-prior information may be the same or different, and this application does not impose any limitation on this.

[0063] It should be noted that, in one possible embodiment, the first code stream and the second code stream may be two code streams. In another possible embodiment, the first code stream and the second code stream are two parts of the same code stream, that is, the same code stream carries both the residual information and the first residual super-prior information; this application does not impose any restrictions on this.

[0064] According to the third aspect, the prior information of the residual information of the current frame includes at least one of the time domain prior information of the residual information of the current frame or the spatial domain prior information of the residual information of the current frame.

[0065] According to the third aspect, or any implementation method of the above third aspect, the time domain prior information of the residual information of the current frame includes the residual information of the previous n frames of the current frame, where n is a positive integer; the spatial domain prior information of the residual information of the current frame includes the prediction information of the current frame.

[0066] According to the third aspect, or any implementation of the third aspect above, the method is applied to an AI video coding framework, the AI ​​video coding framework includes an entropy estimation module, an entropy coding module, an inter-frame prediction module, and a residual coding module, wherein:

[0067] The inter-frame prediction module is used to perform inter-frame prediction on the current frame to obtain prediction information of the current frame; and output the prediction information of the current frame to the residual coding module;

[0068] The residual coding module is used to determine the residual information of the current frame based on the prediction information of the current frame and the original image of the current frame; and output the residual information of the current frame to the entropy estimation module;

[0069] An entropy estimation module is configured to determine first residual super prior information of the current frame based on the residual information of the current frame and the prior information of the residual information of the current frame; and output the first residual super prior information of the current frame to the entropy coding module;

[0070] An entropy coding module, configured to encode first residual super-prior information of a current frame to obtain a first bitstream;

[0071] The entropy estimation module is further configured to perform probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame; and output the residual probability distribution to the entropy coding module;

[0072] The entropy coding module is further configured to perform entropy coding on the residual information of the current frame based on the residual probability distribution of the current frame to obtain a second code stream.

[0073] Exemplarily, the entropy estimation module may correspond to the second entropy estimation module in FIG. 2A , and the entropy coding module may correspond to the second entropy coding module in FIG. 2A .

[0074] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0075] In a fourth aspect, an embodiment of the present application further provides a decoding method, which includes: first, receiving a code stream; wherein the code stream includes a first code stream and a second code stream; then, decoding the first code stream to obtain second residual super-prior information of the current frame; and performing probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame; then, entropy decoding the second code stream based on the residual probability distribution of the current frame to obtain decoded residual information of the current frame; thereafter, performing inter-frame prediction on the current frame to obtain prediction information of the current frame; and then, reconstructing based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

[0076] According to a fourth aspect, the prior information of the residual information of the current frame includes at least one of the time domain prior information of the decoded residual information of the current frame or the spatial domain prior information of the decoded residual information of the current frame.

[0077] According to the fourth aspect, or any implementation method of the above fourth aspect, the time domain prior information of the decoded residual information of the current frame includes the decoded residual information of the previous n frames of the current frame, where n is a positive integer; the spatial domain prior information of the decoded residual information of the current frame includes the prediction information of the current frame.

[0078] According to the fourth aspect, or any implementation of the fourth aspect above, the method is applied to an AI video decoding framework, the AI ​​video decoding framework includes an inter-frame prediction module, an entropy estimation module, an entropy decoding module, a prediction module and a reconstruction module, wherein,

[0079] An entropy decoding module is used to decode the first bit stream to obtain the second residual super-prior information of the current frame; and output the second residual super-prior information of the current frame to the entropy estimation module;

[0080] An entropy estimation module is used to perform probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame; and output the residual probability distribution to the entropy coding module;

[0081] An entropy coding module is used to perform entropy decoding on the second bit stream based on the residual probability distribution of the current frame to obtain decoded residual information of the current frame; and output the decoded residual information of the current frame to the reconstruction module;

[0082] The inter-frame prediction module is used to perform inter-frame prediction on the current frame to obtain prediction information of the current frame; and output the prediction information of the current frame to the reconstruction module;

[0083] The reconstruction module is used to reconstruct the image based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

[0084] Exemplarily, the entropy estimation module may correspond to the second entropy estimation module in FIG. 2B , and the entropy decoding module may correspond to the second entropy decoding module in FIG. 2B .

[0085] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0086] In a fifth aspect, an embodiment of the present application provides a coding device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the coding device to execute the coding method in the first aspect or any possible implementation of the first aspect; or, to execute the coding method in the third aspect or any possible implementation of the third aspect.

[0087] The fifth aspect and any implementation of the fifth aspect respectively correspond to the first aspect and any implementation of the first aspect; or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and will not be repeated here.

[0088] In the sixth aspect, an embodiment of the present application provides a decoding device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the decoding device to execute the decoding method in the second aspect or any possible implementation of the second aspect; or, to execute the decoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0089] The sixth aspect and any implementation thereof respectively correspond to the second aspect and any implementation thereof; or respectively correspond to the fourth aspect and any implementation thereof. The technical effects corresponding to the sixth aspect and any implementation thereof may refer to the technical effects corresponding to the second aspect and any implementation thereof, or to the technical effects corresponding to the fourth aspect and any implementation thereof, and are not further elaborated here.

[0090] In a seventh aspect, an embodiment of the present application provides a decoding device comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the first aspect or any implementation of the first aspect, or the method in the third aspect or any implementation of the third aspect is executed.

[0091] The seventh aspect and any implementation of the seventh aspect respectively correspond to the first aspect and any implementation of the first aspect; or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and will not be repeated here.

[0092] In an eighth aspect, an embodiment of the present application provides a decoding device comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the second aspect or any implementation of the second aspect, or the method in the fourth aspect or any implementation of the fourth aspect is executed.

[0093] The eighth aspect and any implementation thereof respectively correspond to the second aspect and any implementation thereof; or respectively correspond to the fourth aspect and any implementation thereof. The technical effects corresponding to the eighth aspect and any implementation thereof may refer to the technical effects corresponding to the second aspect and any implementation thereof, or to the technical effects corresponding to the fourth aspect and any implementation thereof, and are not further elaborated here.

[0094] In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, it enables the computer or the processor to execute the encoding method in the first aspect or any possible implementation of the first aspect; or, execute the encoding method in the third aspect or any possible implementation of the third aspect.

[0095] The ninth aspect and any implementation of the ninth aspect respectively correspond to the first aspect and any implementation of the first aspect; or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the ninth aspect and any implementation of the ninth aspect may refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and will not be repeated here.

[0096] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, it enables the computer or the processor to execute the encoding method in the second aspect or any possible implementation of the second aspect; or, execute the decoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0097] The tenth aspect and any implementation of the tenth aspect respectively correspond to the second aspect and any implementation of the second aspect; or respectively correspond to the fourth aspect and any implementation of the fourth aspect. The technical effects corresponding to the tenth aspect and any implementation of the tenth aspect may refer to the technical effects corresponding to the second aspect and any implementation of the second aspect, or refer to the technical effects corresponding to the fourth aspect and any implementation of the fourth aspect, and will not be repeated here.

[0098] In the eleventh aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect, or executes the encoding method in the third aspect or any possible implementation of the third aspect.

[0099] The eleventh aspect and any implementation thereof respectively correspond to the first aspect and any implementation thereof; or respectively correspond to the third aspect and any implementation thereof. The technical effects corresponding to the eleventh aspect and any implementation thereof may refer to the technical effects corresponding to the first aspect and any implementation thereof, or to the technical effects corresponding to the third aspect and any implementation thereof, and are not further elaborated here.

[0100] In the twelfth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the decoding method in the second aspect or any possible implementation of the second aspect, or executes the decoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0101] The twelfth aspect and any implementation of the twelfth aspect respectively correspond to the second aspect and any implementation of the second aspect; or respectively correspond to the fourth aspect and any implementation of the fourth aspect. The technical effects corresponding to the twelfth aspect and any implementation of the twelfth aspect may refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or refer to the technical effects corresponding to the above-mentioned fourth aspect and any implementation of the fourth aspect, and will not be repeated here.

[0102] In a thirteenth aspect, an embodiment of the present application provides an apparatus for storing a code stream, the apparatus comprising: a receiver and at least one storage medium, the receiver being configured to receive the code stream; the at least one storage medium being configured to store the code stream; the code stream being generated according to the first aspect and any one of the implementations of the first aspect, or according to the third aspect and any one of the implementations of the third aspect.

[0103] The thirteenth aspect and any implementation of the thirteenth aspect respectively correspond to the first aspect and any implementation of the first aspect; or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the thirteenth aspect and any implementation of the thirteenth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and will not be repeated here.

[0104] In a fourteenth aspect, an embodiment of the present application provides a device for transmitting a code stream, the device comprising: a transmitter and at least one storage medium, the at least one storage medium being used to store the code stream, the code stream being generated according to the first aspect and any one of the implementation methods of the first aspect, or generated according to the third aspect and any one of the implementation methods of the third aspect; the transmitter being used to obtain the code stream from the storage medium and send the code stream to the end-side device through the transmission medium.

[0105] The fourteenth aspect and any implementation of the fourteenth aspect respectively correspond to the first aspect and any implementation of the first aspect; or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the fourteenth aspect and any implementation of the fourteenth aspect may refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and will not be repeated here.

[0106] In a fifteenth aspect, an embodiment of the present application provides a system for distributing code streams, the system comprising: at least one storage medium for storing at least one code stream, the at least one code stream being generated according to the first aspect and any one of the implementations of the first aspect, or generated according to the third aspect and any one of the implementations of the third aspect; a streaming media device for obtaining a target code stream from the at least one storage medium and sending the target code stream to an end-side device, wherein the streaming media device comprises a content server or a content distribution server.

[0107] The fifteenth aspect and any implementation of the fifteenth aspect respectively correspond to the first aspect and any implementation of the first aspect; or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the fifteenth aspect and any implementation of the fifteenth aspect may refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] FIG1A is a schematic diagram illustrating an exemplary application framework;

[0109] FIG1B is a schematic diagram of an exemplary storage framework;

[0110] FIG1C is a schematic diagram illustrating an exemplary transmission framework;

[0111] FIG2A is a schematic diagram illustrating the structure of an AI video encoding framework;

[0112] FIG2B is a schematic diagram illustrating the structure of an AI video decoding framework;

[0113] FIG3A is a schematic structural diagram of an exemplary first entropy estimation module;

[0114] FIG3B is a schematic diagram illustrating an exemplary encoding process;

[0115] FIG4 is a schematic diagram illustrating an exemplary decoding process;

[0116] FIG5A is a schematic structural diagram of a second entropy estimation module; ...

[0117] FIG5B is a schematic diagram illustrating an exemplary encoding process;

[0118] FIG6 is a schematic diagram illustrating an exemplary decoding process;

[0119] FIG7 is a schematic diagram illustrating an exemplary encoding process;

[0120] FIG8 is a schematic diagram illustrating an exemplary decoding process;

[0121] FIG9 is a schematic structural diagram of an exemplary device. DETAILED DESCRIPTION

[0122] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0123] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0124] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0125] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0126] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0127] FIG1A is a schematic diagram illustrating an exemplary application framework.

[0128] Referring to Figure 1A , for example, a camera can capture video and obtain video data. Then, an AI video encoding framework can be used to perform AI video encoding on the video data to generate a bitstream. This bitstream can then be stored locally or transmitted to a remote device.

[0129] Continuing with reference to FIG1A , in one possible approach, after the code stream is stored locally, when a video needs to be played or edited, the AI ​​video decoding framework can be used to perform AI video decoding on the code stream to obtain reconstructed video data; subsequently, the reconstructed video data can be played or edited.

[0130] Continuing with reference to Figure 1A, in one possible approach, after the remote device receives the code stream, when it needs to play or edit a video, it can use the AI ​​video decoding framework to perform AI video decoding on the code stream to obtain reconstructed video data; then, the reconstructed video data can be played or edited.

[0131] FIG. 1B is a schematic diagram illustrating an exemplary storage framework.

[0132] 1B , exemplarily, the AI ​​video encoding framework may include an AI encoding module (also referred to as an AI encoding unit) and an entropy encoding module (also referred to as an entropy encoding unit), and the AI ​​video decoding framework may include an AI decoding module (also referred to as an AI decoding unit) and an entropy decoding module (also referred to as an entropy decoding unit); it should be understood that FIG1B is only an example of the present application, and the AI ​​video encoding framework and the AI ​​video decoding framework may include more modules than shown in FIG1B , and the present application does not limit this.

[0133] Exemplarily, the AI ​​encoding module and the AI ​​decoding module can be set in an embedded neural network processor (NPU) or a graphics processing unit (GPU). Exemplarily, the entropy encoding module and the entropy decoding module can be set in a central processing unit (CPU). Exemplarily, the file saving module and the file loading module can be set in a CPU.

[0134] Referring to Figure 1B, illustratively, after the camera captures the video data, it can be input into the AI ​​encoding module. Then, the AI ​​encoding module can perform the following processing for each frame image in the video data: perform prediction (including intra-frame prediction and inter-frame prediction, this application takes inter-frame prediction as an example to illustrate), determine the prediction information; and determine the residual information based on the prediction information and the original image, and determine the probability distribution of the residual information; then, input the residual information and the probability distribution of the residual information into the entropy coding module. Afterwards, the entropy coding module can entropy encode the residual information according to the probability distribution of the residual information to obtain a code stream; and input the code stream into the file saving module for saving to obtain a file.

[0135] Continuing with Figure 1B, for example, when a video needs to be played or edited, the file loading module can load the file and then input the code stream in the file into the entropy decoding module. Next, the entropy decoding module can obtain the probability distribution of the residual information from the AI ​​decoding module, and based on the probability distribution of the residual information, entropy decode the residual information from the code stream and output the residual information to the AI ​​decoding module. Afterwards, the AI ​​decoding module can perform inter-frame prediction to obtain prediction information; then, reconstruct based on the prediction information and residual information to obtain a reconstructed image; and then, reconstructed video data can be obtained.

[0136] FIG1C is a schematic diagram illustrating an exemplary transmission framework.

[0137] 1C , illustratively, the encoding process at the encoding end may be as follows:

[0138] For example, after the camera collects the video data, it can be input into the AI ​​encoding module. Then, the AI ​​encoding module can perform the following processing for each frame image in the video data: perform prediction (including intra-frame prediction and inter-frame prediction, this application takes inter-frame prediction as an example), determine the prediction information; and determine the residual information based on the prediction information and the original image, and determine the probability distribution of the residual information; then, the residual information and the probability distribution of the residual information are input into the entropy coding module. Afterwards, the entropy coding module can entropy encode the residual information according to the probability distribution of the residual information to obtain a code stream; and send the code stream to the server in the cloud.

[0139] Afterwards, the server can send the code stream to the decoding end. Among them, the server can be a single server or a server cluster, which is not limited in this application.

[0140] Continuing with FIG1C , illustratively, after the decoding end receives the code stream sent by the server, the decoding process may be as follows:

[0141] For example, the file loading module can load a file and then send the bitstream in the file to the entropy decoding module. The AI ​​decoding network can then perform inter-frame prediction to obtain prediction information. The entropy decoding module can then obtain the probability distribution of residual information from the AI ​​decoding module. Based on the probability distribution of residual information, it entropy decodes the residual information from the bitstream and outputs the residual information to the AI ​​decoding module. Subsequently, reconstruction is performed based on the prediction information and residual information to obtain a reconstructed image, thereby obtaining reconstructed video data.

[0142] It should be understood that the encoding end can also directly send the code stream to the decoding end without forwarding through the server, and this application does not impose any restrictions on this.

[0143] Figure 2A is a schematic diagram of the structure of an exemplary AI video coding framework. In the embodiment of Figure 2A, an inter-frame coding process is shown.

[0144] 2A , exemplarily, the AI ​​encoding module may include a motion estimation module, an optical flow encoding module, a first entropy estimation module, an optical flow decoding module, a prediction module, a feature extraction module, a residual encoding module, a second entropy estimation module, a residual decoding module and a reconstruction module; it should be understood that the AI ​​encoding module may also include more or fewer modules than shown in FIG. 2A , and this application does not impose any restrictions on this.

[0145] 2A , exemplarily, the entropy coding module may include a first entropy coding module and a second entropy coding module.

[0146] 2A , exemplarily, the AI ​​video encoding framework may further include an entropy decoding module, and the entropy decoding module may include a first entropy decoding module and a second entropy decoding module.

[0147] It should be noted that the feature extraction module in the dotted box in FIG2A is an optional module.

[0148] It should be noted that the various modules included in the AI ​​video coding framework can be implemented only by a neural network, or by a combination of an algorithm and a neural network; or, some modules included in the AI ​​video coding framework can be implemented only by an algorithm, and this application does not impose any restrictions on this.

[0149] 2A , an example of inter-frame coding of an image frame in video data is used for description. The AI ​​video coding process may be as follows:

[0150] Exemplarily, the original image of the current frame and the reconstructed image of the reference frame corresponding to the current frame can be input into the motion estimation module. Then, the motion estimation module can perform motion estimation based on the original image of the current frame and the reconstructed image of the reference frame to determine the optical flow of the current frame (hereinafter referred to as the first optical flow); and output the first optical flow of the current frame to the optical flow encoding module. The optical flow encoding module (also referred to as the optical flow feature extraction module) can encode the first optical flow of the current frame (it should be noted that the encoding performed by the optical flow encoding module is essentially feature extraction), obtain the feature information of the first optical flow of the current frame (also referred to as the compressed feature of the first optical flow of the current frame) and output it to the first entropy estimation module.

[0151] Exemplarily, the first entropy estimation module can perform probability estimation on the feature information of the first optical flow of the current frame, determine the probability distribution of the feature information of the first optical flow of the current frame (hereinafter referred to as the optical flow probability distribution of the current frame); then, output the feature information of the first optical flow of the current frame and the optical flow probability distribution of the current frame to the first entropy coding module.

[0152] Exemplarily, the first entropy coding module may perform entropy coding on feature information of the first optical flow of the current frame according to the optical flow probability distribution of the current frame to obtain an optical flow code stream.

[0153] For example, on one hand, the optical flow code stream can be sent / saved (for example, the optical flow code stream is sent to a server or a decoding end). On the other hand, the optical flow code stream can be output to the first entropy decoding module; in addition, the first entropy estimation module can also output the optical flow probability distribution of the current frame to the first entropy decoding module.

[0154] Exemplarily, the first entropy decoding module can perform entropy decoding on the optical flow code stream according to the optical flow probability distribution of the current frame to obtain the feature information of the first optical flow after decoding of the current frame (which can also be called the feature information of the second optical flow of the current frame); and output the feature information of the second optical flow of the current frame to the optical flow decoding module.

[0155] Exemplarily, the optical flow decoding module can decode the feature information of the second optical flow of the current frame (it should be noted that the decoding performed by the optical flow decoding module is essentially feature recovery), obtain the second optical flow of the current frame and output the second optical flow of the current frame to the prediction module.

[0156] In one possible way, when the AI ​​encoding module includes a feature extraction module, the reconstructed image of the reference frame can be input into the feature extraction module, and the feature extraction module extracts the features of the reconstructed image of the reference frame to obtain the feature information of the reconstructed image of the reference frame; and the feature information of the reconstructed image of the reference frame is output to the prediction module. The prediction module can make a prediction based on the second optical flow of the current frame and the feature information of the reconstructed image of the reference frame to obtain the prediction information of the current frame; and output the prediction information of the current frame to the residual coding module. Correspondingly, the original image of the current frame can be input into the feature extraction module, and the feature extraction module extracts the features of the original image of the current frame to obtain the feature information of the original image of the current frame, and output the feature information of the original image of the current frame to the residual coding module. The residual coding module outputs the residual information of the current frame (hereinafter referred to as the first residual information of the current frame) to the second entropy estimation module based on the feature information of the original image of the current frame and the prediction information of the current frame. Among them, the first residual information of the current frame can be the first residual of the current frame (wherein, the first residual can refer to the residual between the feature information of the original image of the current frame and the prediction information of the current frame, or it can be the feature information of the first residual of the current frame (wherein, the feature information of the first residual can also be called the compressed feature of the first residual, which can be obtained by encoding the first residual of the residual encoding module (it should be noted that the encoding performed by the residual encoding module is essentially feature extraction)), and this application does not impose any restrictions on this. This application takes the first residual information of the current frame as the feature information of the first residual of the current frame as an example for explanation.

[0157] In one possible approach, when the AI ​​encoding module does not include a feature extraction module, the reconstructed image of the reference frame can be directly input into the prediction module. The prediction module then performs a prediction based on the second optical flow of the current frame and the reconstructed image of the reference frame to obtain prediction information for the current frame. This prediction information is then output to the residual encoding module. Alternatively, the original image of the current frame can be directly input into the residual encoding module. The residual encoding module then outputs the first residual information of the current frame to the second entropy estimation module based on the original image of the current frame and the prediction information of the current frame.

[0158] Exemplarily, the second entropy estimation module can perform probability estimation on the first residual information of the current frame to determine the probability distribution of the first residual information of the current frame (hereinafter referred to as the residual probability distribution of the current frame); then, the first residual information of the current frame and the residual probability distribution of the current frame are output to the second entropy coding module.

[0159] Exemplarily, the second entropy coding module may perform entropy coding on the first residual information of the current frame according to the residual probability distribution of the current frame to obtain a residual code stream.

[0160] For example, on the one hand, the residual bitstream can be sent / saved (for example, the residual bitstream can be sent to a server or a decoding end). On the other hand, the residual bitstream can be output to the second entropy decoding module; in addition, the second entropy estimation module can also output the residual probability distribution of the current frame to the second entropy decoding module.

[0161] Exemplarily, the second entropy decoding module may entropy decode the residual code stream according to the residual probability distribution of the current frame to obtain the first residual information after entropy decoding of the current frame. When the first residual information of the current frame is characteristic information of the first residual of the current frame, the first residual information after entropy decoding of the current frame may be output to the residual decoding module.

[0162] Exemplarily, the residual decoding module can decode the first residual information after entropy decoding of the current frame (it should be noted that the decoding performed by the residual decoding module is essentially feature recovery), obtain the second residual information of the current frame and output the second residual information of the current frame to the reconstruction module. It should be noted that when the first residual information of the current frame is the first residual of the current frame, the AI ​​video encoding framework may not include a residual decoding module; at this time, the second residual information of the current frame is the decoded first residual information obtained by entropy decoding of the current frame. In addition, it can also be said that the second residual information of the current frame is obtained by decoding the encoding result of the first residual information of the current frame (i.e., the residual code stream); when the first residual information of the current frame is the feature information of the first residual of the current frame, the decoding in this step includes entropy decoding and feature recovery; when the first residual information of the current frame is the first residual of the current frame, the decoding in this step is entropy decoding.

[0163] Exemplarily, the prediction module can also output the prediction information of the current frame to the reconstruction module. When the AI ​​encoding module includes a feature extraction module, the reconstruction module can first reconstruct based on the second residual information of the current frame and the prediction information of the current frame to obtain the feature information of the reconstructed image of the current frame; then, the reconstruction module can perform feature transformation on the feature information of the reconstructed image of the current frame to obtain the reconstructed image of the current frame. When the AI ​​encoding module does not include a feature extraction module, the reconstruction module can reconstruct based on the second residual information of the current frame and the prediction information of the current frame to obtain the reconstructed image of the current frame.

[0164] Exemplarily, the present application may also encode information for determining the optical flow probability distribution of the current frame and send it to the decoding end; so that the decoding end determines the optical flow probability distribution to obtain the second optical flow of the current frame; and encode information for determining the residual probability distribution of the current frame and send it to the decoding end; so that the decoding end determines the residual probability distribution to obtain the second residual information of the current frame.

[0165] FIG2B is a schematic diagram showing the structure of an exemplary AI video decoding framework.

[0166] 2B , exemplarily, the AI ​​decoding module may include: a first entropy estimation module, a second entropy estimation module, an optical flow decoding module, a prediction module, a feature extraction module, a residual decoding module and a reconstruction module; it should be understood that the AI ​​encoding module may also include more or fewer modules than shown in FIG. 2B , and this application does not impose any restrictions on this.

[0167] 2B , exemplarily, the entropy decoding module may include a first entropy decoding module and a second entropy encoding module.

[0168] It should be noted that the feature extraction module in the dotted box in FIG2B is an optional module.

[0169] It should be noted that the various modules included in the AI ​​video decoding framework can be implemented only by a neural network, or by a combination of an algorithm and a neural network; or, some modules included in the AI ​​video decoding framework can be implemented only by an algorithm, and this application does not impose any restrictions on this.

[0170] 2B , the inter-frame decoding of an image frame in the video data is taken as an example for description. The AI ​​video decoding process may be as follows:

[0171] Exemplarily, the decoding end may receive a bitstream that may include an optical flow bitstream, a residual bitstream, a bitstream of information for determining an optical flow probability distribution, and a bitstream of information for determining a residual probability distribution. The first entropy decoding module may perform entropy decoding on the bitstream of information for determining an optical flow probability distribution to obtain information for determining an optical flow probability distribution of a current frame, and output the information for determining an optical flow probability distribution of a current frame to a first entropy estimation module. The first entropy estimation module may then perform probability estimation based on the information for determining an optical flow probability distribution of a current frame, determine an optical flow probability distribution of the current frame, and output the information to the first entropy decoding module.

[0172] Exemplarily, the first entropy decoding module may perform entropy decoding on the optical flow code stream according to the optical flow probability distribution of the current frame, obtain feature information of the second optical flow of the current frame, and output the obtained feature information to the optical flow decoding module.

[0173] Exemplarily, the optical flow decoding module may perform feature recovery on feature information of the second optical flow of the current frame, obtain the second optical flow of the current frame, and output the second optical flow of the current frame to the prediction module.

[0174] Exemplarily, the second entropy decoding module can perform entropy decoding on the code stream of information used to determine the residual probability distribution, obtain information used to determine the residual probability distribution of the current frame, and output the information used to determine the residual probability distribution of the current frame to the second entropy estimation module; then, the second entropy estimation module performs probability estimation based on the information for determining the residual probability distribution of the current frame, determines the residual probability distribution of the current frame and outputs it to the second entropy decoding module.

[0175] Exemplarily, the second entropy decoding module may perform entropy decoding on the residual code stream according to the residual probability distribution of the current frame, obtain the first residual information after entropy decoding of the current frame, and output it to the residual decoding module.

[0176] Exemplarily, the residual decoding module may perform feature recovery on the first residual information of the current frame after entropy decoding, obtain the second residual information of the current frame, and output the second residual information of the current frame to the reconstruction module.

[0177] In one possible approach, when the AI ​​decoding module includes a feature extraction module, the reconstructed image of the reference frame can be input into the feature extraction module, and the feature extraction module extracts the features of the reconstructed image of the reference frame to obtain feature information of the reconstructed image of the reference frame; and the feature information of the reconstructed image of the reference frame is output to the prediction module. The prediction module can make a prediction based on the second optical flow of the current frame and the feature information of the reconstructed image of the reference frame to obtain prediction information of the current frame; and output the prediction information of the current frame to the reconstruction module. Then, the reconstruction module can first reconstruct based on the second residual information of the current frame and the prediction information of the current frame to obtain feature information of the reconstructed image of the current frame; thereafter, the reconstruction module can perform feature transformation (such as feature recovery) on the feature information of the reconstructed image of the current frame to obtain the reconstructed image of the current frame.

[0178] In one possible approach, when the AI ​​decoding module does not include a feature extraction module, the reconstructed image of the reference frame can be directly input into the prediction module. The prediction module performs prediction based on the second optical flow of the current frame and the reconstructed image of the reference frame to obtain prediction information for the current frame. The prediction information for the current frame is then output to the reconstruction module. The reconstruction module then reconstructs the current frame based on the second residual information of the current frame and the prediction information of the current frame to obtain a reconstructed image of the current frame.

[0179] It should be understood that the present invention does not limit the names of the modules included in the AI ​​video encoding framework shown in Figure 2A, and the names of the modules included in the AI ​​video decoding framework shown in Figure 2B; and the modules included in the AI ​​video encoding framework can also be called units; and the modules included in the AI ​​video decoding framework can also be called units.

[0180] It should be noted that in one possible approach, the residual code stream and the optical stream code stream are two code streams. In one possible approach, the residual code stream and the optical stream code stream can be two parts of the same code stream, that is, the same code stream carries the residual information and the first optical stream.

[0181] FIG3A is a schematic structural diagram of an exemplary first entropy estimation module.

[0182] 3A , exemplarily, the first entropy estimation module includes a first super-prior encoding network, a first super-prior decoding network, and a first fusion module. Furthermore, the entropy encoding module in the AI ​​video encoding framework may further include a third entropy encoding module, and the entropy decoding modules in the AI ​​video encoding framework and the AI ​​video decoding framework may further include a third entropy decoding module.

[0183] The following describes the encoding and decoding process of video data based on FIG. 2A , FIG. 2B and FIG. 3A .

[0184] Fig. 3B is a schematic diagram of an exemplary encoding process. In the embodiment of Fig. 3B, inter-frame decoding of an image frame in video data is taken as an example for description.

[0185] S301 , performing motion estimation on the current frame to obtain a first optical flow of the current frame.

[0186] 2A , exemplarily, the original image of the current frame and the image information of the reference frame corresponding to the current frame (the reconstructed image of the reference frame or the feature information of the reconstructed image of the reference frame) can be input into the motion estimation module, and the motion estimation module calls the motion estimation network to perform motion estimation to obtain the first optical flow of the current frame.

[0187] S302: Perform feature extraction on the first optical flow of the current frame to obtain feature information of the first optical flow of the current frame.

[0188] Next, the motion estimation module can output the first optical flow of the current frame to the optical flow encoding module, and the optical flow encoding module outputs the feature information of the first optical flow of the current frame to the first entropy estimation module; thereafter, the feature information of the first optical flow of the current frame can be entropy estimated with reference to the following S303 to S305.

[0189] S303 : Determine super-prior information of the first optical flow of the current frame based on the feature information of the first optical flow of the current frame and the prior information of the first optical flow of the current frame.

[0190] 3A , illustratively, the feature information of the first optical flow of the current frame and the prior information of the first optical flow of the current frame can be input into the first super-prior coding network, and the first super-prior coding network encodes the feature information of the first optical flow of the current frame and the prior information of the first optical flow of the current frame (it should be noted that the encoding performed by the first super-prior coding network is essentially feature extraction), and obtains the optical flow super-prior information of the current frame (hereinafter referred to as the first optical flow super-prior information of the current frame, wherein the first optical flow super-prior information may also be referred to as the first optical flow super-prior feature). Then, the first super-prior coding network can output the first optical flow super-prior information of the current frame to the third entropy coding module.

[0191] Exemplarily, at the encoding end, the prior information of the first optical flow of the current frame may refer to the empirical information (or historical information) that can be obtained before entropy encoding the feature information of the first optical flow of the current frame, which can be used for subsequent inference and decision-making (such as for determining the probability distribution of the feature information of the first optical flow of K1 pixels in the current frame (i.e., the optical flow probability distribution)).

[0192] Exemplarily, the first super-prior coding network can determine the prior information of the first optical flow of the current frame, which can be used to determine the probability distribution of the feature information of the first optical flow of which pixels in the current frame; then, it can determine the information in the feature information of the first optical flow of the current frame that can be used to determine the probability distribution of the feature information of the first optical flow of other pixels in the current frame, perform feature extraction on this information, and obtain the super-prior information of the first optical flow of the current frame. In other words, the super-prior information of the first optical flow of the current frame obtained by feature extraction is used to determine the probability distribution of the feature information of the first optical flow of K2 pixels in the current frame. In this way, the amount of data used to determine the probability distribution of the optical flow (i.e., the super-prior information of the first optical flow) can be reduced, thereby reducing the bitrate overhead.

[0193] The current frame may include K pixels, K is a positive integer, K1 and K2 are both positive integers less than K, and the union of K1 pixels and K2 pixels is the K pixels included in the current frame. It should be noted that, in one possible manner, K1 pixels and K2 pixels have no intersection, that is, K1 pixels and K2 pixels can be complementary to each other, K = K1 + K2; for example, assuming that the current frame includes 100 pixels, K1 = K2 = 50, then the optical flow probability distribution of 50 pixels out of the 100 pixels can be determined by using the prior information of the first optical flow of the current frame alone; the optical flow probability distribution of the other 50 pixels out of the 100 pixels can be determined by using the super prior information of the first optical flow of the current frame alone. In one possible approach, K1 pixels and K2 pixels intersect, and K1+K2 is greater than K; for example, assuming that the current frame includes 100 pixels, K1=50, K2=60, then using the prior information of the first optical flow of the current frame alone, the optical flow probability distribution of 40 pixels among the 100 pixels can be determined; using the first optical flow super prior information of the current frame alone, the optical flow probability distribution of another 50 pixels among the 100 pixels can be determined; and combining the first optical flow prior information of the current frame and the first optical flow super prior information of the current frame, the optical flow probability distribution of another 10 pixels among the 100 pixels can be determined.

[0194] Exemplarily, the prior information of the first optical flow of the current frame includes at least one of the temporal prior information of the first optical flow of the current frame or the spatial prior information of the first optical flow of the current frame. That is, the prior information of the first optical flow of the current frame may only include the temporal prior information of the first optical flow of the current frame; or, the prior information of the first optical flow of the current frame may only include the spatial prior information of the first optical flow of the current frame; or, the prior information of the first optical flow of the current frame may include both the spatial prior information of the first optical flow of the current frame and the temporal prior information of the first optical flow of the current frame.

[0195] Exemplarily, the time domain prior information of the first optical flow of the current frame may refer to information related to the first optical flow of the current frame in the time domain; the spatial domain prior information of the first optical flow of the current frame may refer to information related to the first optical flow of the current frame in the spatial domain.

[0196] Exemplarily, the temporal prior information of the first optical flow of the current frame may include feature information of the second optical flow of the m frames preceding the current frame, where m is a positive integer. When m=1, the temporal prior information of the first optical flow of the current frame may include feature information of the second optical flow of the reference frame corresponding to the current frame.

[0197] Exemplarily, the spatial domain prior information of the first optical flow of the current frame may include feature information of a reconstructed image of a reference frame corresponding to the current frame.

[0198] S304 , encode the first optical flow super-prior information of the current frame to obtain a first code stream.

[0199] Exemplarily, the third entropy coding module may perform entropy coding on the first optical flow super-prior information of the current frame to obtain a first bitstream. On the one hand, the third entropy coding module may send the first bitstream to a decoding end (or server) or store it locally; on the other hand, the third entropy coding module may output the first bitstream to a third entropy decoding module.

[0200] S305 , performing probability estimation based on the second optical flow super prior information of the current frame and the first optical flow prior information of the current frame to obtain an optical flow probability distribution of the current frame; wherein the second optical flow super prior information of the current frame is obtained based on decoding of the first bitstream.

[0201] Next, the third entropy decoding module can perform entropy decoding on the first bitstream to obtain the first optical flow super-prior information after entropy decoding of the current frame; and output the first optical flow super-prior information after entropy decoding of the current frame to the first super-prior decoding network. Afterwards, the first super-prior decoding network can decode the first optical flow super-prior information after entropy decoding of the current frame (it should be noted that the encoding performed by the first super-prior decoding network is essentially feature recovery) to obtain the second optical flow super-prior information of the current frame (wherein the second optical flow super-prior information can also be called the second optical flow super-prior feature); and output the second optical flow super-prior information of the current frame to the first fusion module. In other words, the "decoding" based on the first bitstream decoding in S305 includes two steps: entropy decoding and feature recovery.

[0202] For example, the prior information of the first optical flow of the current frame can also be input into the first fusion module. Then, the first fusion module can perform probability estimation based on the super prior information of the second optical flow of the current frame and the prior information of the first optical flow of the current frame to obtain the optical flow probability distribution of the current frame. The optical flow probability distribution of the current frame is input into the first entropy coding module. The optical flow probability distribution of the current frame is also the probability distribution of the feature information of the first optical flow of the current frame.

[0203] S306 , performing entropy coding on feature information of the first optical flow of the current frame based on the optical flow probability distribution of the current frame to obtain a second code stream.

[0204] S307: Perform inter-frame prediction on the current frame to obtain prediction information of the current frame.

[0205] Exemplarily, inter-frame prediction may be performed based on the second optical flow of the current frame to obtain prediction information of the current frame; wherein the second optical flow may be obtained based on decoding of the second code stream.

[0206] It should be noted that the "decoding" in the decoding based on the second code stream may include two steps: entropy decoding and feature recovery. For details, please refer to the description in the embodiment of Figure 2A above, which will not be repeated here.

[0207] S308 : Determine residual information of the current frame based on the prediction information of the current frame and the original image of the current frame.

[0208] S309: Encode the residual information of the current frame to obtain a third bit stream.

[0209] For example, S306 to S309 can refer to the description in the embodiment of FIG2A , which will not be repeated here. It should be noted that the residual information of the current frame in S308 and S309 refers to the first residual information of the current frame in the embodiment of FIG2A .

[0210] Exemplarily, S309 may include: determining a residual probability distribution of the current frame; and then performing entropy coding on residual information of the current frame according to the residual probability distribution of the current frame to obtain a third bitstream.

[0211] Exemplarily, the entropy coding module in the AI ​​video coding framework may also include a fifth entropy coding module and a fifth entropy decoding module; the second entropy estimation module includes a third super-prior coding network, a third super-prior decoding network and a third fusion module. Specifically, the process of determining the residual probability distribution of the current frame can be as follows: the residual information of the current frame can be input into the third super-prior coding network to obtain the third residual super-prior information of the current frame (wherein the third residual super-prior information can also be called the third residual super-prior feature); the third residual super-prior information of the current frame is input into the fifth entropy coding module. Then, the fifth entropy coding module performs entropy coding on the third residual super-prior information of the current frame to obtain a fourth bitstream; on the one hand, the fourth bitstream is sent to the decoding end (or server) or saved locally; on the other hand, the fourth bitstream is output to the fifth entropy decoding module. Afterwards, the fifth entropy decoding module can perform entropy decoding on the fourth bitstream to obtain the third residual super-prior information after entropy decoding of the current frame; and output the third residual super-prior information after entropy decoding of the current frame to the third super-prior coding network. Then, the third super-prior coding network performs feature recovery on the third residual super-prior information after entropy decoding of the current frame to obtain fourth residual super-prior information of the current frame (wherein the fourth residual super-prior information may also be referred to as a fourth residual super-prior feature); and outputs the fourth residual super-prior information of the current frame to the third fusion module. Furthermore, the prior information of the residual information of the current frame may be input to the third fusion module, which performs probability estimation based on the prior information of the residual information of the current frame and the fourth residual super-prior information of the current frame to determine the residual probability distribution of the current frame.

[0212] Figure 4 is a schematic diagram illustrating an exemplary decoding process. The decoding process in the embodiment of Figure 4 corresponds to the encoding process in the embodiment of Figure 3B.

[0213] S401, receiving code streams; wherein the code streams include a first code stream, a second code stream, and a third code stream.

[0214] S402: Decode the first code stream to obtain the second optical flow super-prior information of the current frame.

[0215] Referring to Figure 3A , for example, the received first bitstream can be input into the third entropy decoding module, which can perform entropy decoding on the first bitstream to obtain first optical flow super-prior information after entropy decoding of the current frame; and output the first optical flow super-prior information after entropy decoding of the current frame to the first super-prior decoding network. The first super-prior decoding network can then perform feature recovery on the first optical flow super-prior information after entropy decoding of the current frame to obtain second optical flow super-prior information of the current frame; and output the second optical flow super-prior information of the current frame to the first fusion module.

[0216] S403 , performing probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain an optical flow probability distribution of the current frame.

[0217] Exemplarily, at the decoding end, the prior information of the first optical flow of the current frame may refer to information that can be obtained before entropy decoding the second bitstream (or before obtaining feature information of the first optical flow after entropy decoding of the current frame).

[0218] Exemplarily, the prior information of the first optical flow of the current frame can also be input into the first fusion module; then, the first fusion module can perform probability estimation based on the super prior information of the second optical flow of the current frame and the prior information of the first optical flow of the current frame to obtain the optical flow probability distribution of the current frame; and input the optical flow probability distribution of the current frame into the first entropy coding module.

[0219] S404 , performing entropy decoding on the second bitstream based on the optical flow probability distribution of the current frame to obtain feature information of the second optical flow of the current frame.

[0220] S405 , performing feature recovery on feature information of the second optical flow of the current frame to obtain the second optical flow of the current frame.

[0221] S406 , performing inter-frame prediction based on the second optical flow of the current frame to obtain prediction information of the current frame; wherein the second optical flow of the current frame is obtained by performing feature recovery based on feature information of the second optical flow of the current frame.

[0222] S407: Obtain decoded residual information of the current frame based on the third bitstream.

[0223] S408 , reconstructing based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

[0224] For example, S405 to S408 may refer to the description of the embodiment of FIG2B , which will not be repeated here. It should be noted that the decoded residual information in S407 and S408 may refer to the second residual information of the current frame in the embodiment of FIG2B .

[0225] Exemplarily, the code stream received by the decoding end may also include a fourth code stream; and the entropy decoding module in the AI ​​video decoding framework may also include a fifth entropy decoding module. Based on the third code stream, the process of obtaining the decoded residual information of the current frame may be as follows: the fourth code stream may be input into the fifth entropy decoding module, and the fifth entropy decoding module may perform entropy decoding on the fourth code stream to obtain the third residual super-prior information after entropy decoding of the current frame; and the third residual super-prior information after entropy decoding of the current frame is output to the third super-prior coding network. Then, the third super-prior coding network performs feature recovery on the third residual super-prior information after entropy decoding of the current frame to obtain the fourth residual super-prior information of the current frame; and the fourth residual super-prior information of the current frame is output to the third fusion module. Furthermore, the prior information of the residual information of the current frame may be input into the third fusion module, and the third fusion module performs probability estimation based on the prior information of the residual information of the current frame and the fourth residual super-prior information of the current frame to determine the residual probability distribution of the current frame. Afterwards, the third bitstream may be entropy decoded according to the residual probability distribution of the current frame to obtain decoded residual information of the current frame.

[0226] It should be noted that, in the embodiment of Figure 3A and the embodiment of Figure 4, the first code stream may refer to the code stream of the first optical flow super-prior information, and the second code stream may refer to the code stream of the optical flow (corresponding to the optical flow code stream in Figures 2A and 2B); the third code stream may be the residual code stream (corresponding to the residual code stream in Figures 2A and 2B), and the fourth code stream may refer to the code stream of the first residual super-prior information.

[0227] FIG5A is a schematic structural diagram of an exemplary second entropy estimation module.

[0228] 5A , illustratively, the second entropy estimation module further includes a second super-prior encoding network, a second super-prior decoding network, and a second fusion module. Furthermore, the entropy encoding module in the AI ​​video encoding framework may further include a fourth entropy encoding module, and the entropy decoding modules in the AI ​​video encoding framework and the AI ​​video decoding framework may further include a fourth entropy decoding module.

[0229] The following describes the encoding and decoding process of video data based on FIG. 2A , FIG. 2B and FIG. 5A .

[0230] Fig. 5B is a schematic diagram of an exemplary encoding process. In the embodiment of Fig. 5B, inter-frame decoding of an image frame in video data is taken as an example for description.

[0231] S501, performing inter-frame prediction on the current frame to obtain prediction information of the current frame.

[0232] Exemplarily, the inter-frame prediction process in S501 can refer to the description in the embodiment of Figure 2A above, and will not be repeated here. It should be noted that S501 can be performed by the inter-frame prediction module in the AI ​​video coding framework, wherein the inter-frame prediction module in the AI ​​video coding framework can include the motion estimation module, optical flow encoding module, first entropy estimation module, optical flow decoding module and prediction module in Figure 2A above.

[0233] Exemplarily, exemplary, and the entropy coding module in the AI ​​video coding framework may further include a sixth entropy coding module and a sixth entropy decoding module; the first entropy estimation network also includes a fourth super-prior coding network, a fourth super-prior decoding network and a fourth fusion module. Specifically, the process of determining the optical flow probability distribution can be as follows: the feature information of the first optical flow of the current frame can be input into the fourth super-prior coding network to obtain the third optical flow super-prior information of the current frame (wherein the third optical flow super-prior information can also be referred to as the third optical flow super-prior feature); the third optical flow super-prior information of the current frame is input into the sixth entropy coding module. Then, the sixth entropy coding module entropy encodes the third optical flow super-prior information of the current frame to obtain a third code stream; on the one hand, the third code stream is sent to the decoding end (or server) or saved locally; on the other hand, the third code stream is output to the sixth entropy decoding module. Afterwards, the sixth entropy decoding module can entropy decode the third code stream to obtain the third optical flow super-prior information after entropy decoding of the current frame; and output the third optical flow super-prior information after entropy decoding of the current frame to the fourth super-prior coding network. Then, the fourth super-prior encoding network performs feature recovery on the third optical flow super-prior information after entropy decoding of the current frame to obtain the fourth optical flow super-prior information of the current frame (wherein the fourth optical flow super-prior information may also be referred to as the fourth optical flow super-prior feature); and outputs the fourth optical flow super-prior information of the current frame to the fourth fusion module. Furthermore, the prior information of the first optical flow of the current frame may be input to the fourth fusion module, which performs probability estimation based on the prior information of the first optical flow of the previous frame and the fourth optical flow super-prior information of the current frame to determine the optical flow probability distribution of the current frame.

[0234] Afterwards, entropy coding may be performed on the feature information of the first optical flow of the current frame based on the optical flow probability distribution of the current frame to obtain a fourth code stream.

[0235] S502 : Determine residual information of the current frame based on the prediction information of the current frame and the original image of the current frame.

[0236] For example, S502 may refer to the description in the embodiment of FIG. 2A , which will not be repeated here.

[0237] S503 : Determine first residual super-prior information of the current frame based on the residual information of the current frame and the prior information of the residual information of the current frame.

[0238] 5A , illustratively, the residual information of the current frame and the prior information of the residual information of the current frame can be input into a second super-prior coding network, which performs feature extraction on the residual information of the current frame and the prior information of the residual information of the current frame to obtain first residual super-prior information of the current frame (wherein the first residual super-prior information may also be referred to as a first residual super-prior feature). Subsequently, the second super-prior coding network may output the first residual super-prior information of the current frame to a fourth entropy coding module.

[0239] For example, at the encoding end, the prior information of the residual information of the current frame may refer to the empirical information (or historical information) that can be obtained before entropy encoding the residual information of the current frame, which can be used for subsequent inference and decision-making (such as used to determine the probability distribution of the residual information of K3 pixels in the current frame (i.e., the residual probability distribution)).

[0240] Exemplarily, the second super-prior coding network can determine the prior information of the residual information of the current frame, which can be used to determine the probability distribution of the residual information of which pixels in the current frame; then, it can determine the information in the residual information of the current frame that can be used to determine the probability distribution of the residual information of other pixels in the current frame, perform feature extraction on this information, and obtain the first residual super-prior information of the current frame. In other words, the first residual super-prior information of the current frame obtained by feature extraction is used to determine the probability distribution of the residual information of K4 pixels in the current frame. In this way, the amount of data used to determine the residual probability distribution (i.e., the first residual super-prior information) can be reduced, thereby reducing the bitrate overhead.

[0241] Among them, the current frame can include K pixels, K is a positive integer, K3 and K4 are both positive integers less than K, and the union of K3 pixels and K4 pixels is the K pixels included in the current frame. It should be noted that, in one possible way, K3 pixels and K4 pixels have no intersection, that is, K3 pixels and K4 pixels can be complementary to each other, K=K3+K4; for example, assuming that the current frame includes 100 pixels, K3=K4=50, then using the prior information of the residual information of the current frame alone, the residual probability distribution of 50 pixels in the 100 pixels can be determined, and using the first residual information super prior information of the current frame alone, the residual probability distribution of the other 50 pixels in the 100 pixels can be determined. In one possible approach, K3 pixels and K4 pixels intersect, and K3+K4 is greater than K; for example, assuming that the current frame includes 100 pixels, K3=50, K4=60, then using the prior information of the residual information of the current frame alone, the residual probability distribution of 40 pixels among the 100 pixels can be determined; using the first residual information super prior information of the current frame alone, the residual probability distribution of another 50 pixels among the 100 pixels can be determined; and combining the prior information of the residual information of the current frame and the first residual super prior information of the current frame, the residual probability distribution of another 10 pixels among the 100 pixels can be determined.

[0242] Exemplarily, the prior information of the residual information of the current frame includes at least one of the time domain prior information of the residual information of the current frame or the spatial domain prior information of the residual information of the current frame. In other words, the prior information of the residual information of the current frame may include only the time domain prior information of the residual information of the current frame; or, the prior information of the residual information of the current frame may include only the spatial domain prior information of the residual information of the current frame; or, the prior information of the residual information of the current frame may include both the spatial domain prior information of the residual information of the current frame and the time domain prior information of the residual information of the current frame.

[0243] Exemplarily, the time domain prior information of the residual information of the current frame may refer to information related to the residual information of the current frame in the time domain; the spatial domain prior information of the residual information of the current frame may refer to information related to the residual information of the current frame in the spatial domain.

[0244] Exemplarily, the time domain prior information of the residual information of the current frame includes residual information of n frames before the current frame, where n is a positive integer. When n is 1, the time domain prior information of the residual information of the current frame may be residual information of a reference frame corresponding to the current frame.

[0245] Exemplarily, the spatial domain prior information of the residual information of the current frame includes prediction information of the current frame.

[0246] S504 , encode the first residual super-prior information of the current frame to obtain a first bitstream.

[0247] Exemplarily, the fourth entropy coding module may perform entropy coding on the first residual super-prior information of the current frame to obtain a first bitstream. On the one hand, the fourth entropy coding module may send the first bitstream to a decoding end (or server) or store it locally; on the other hand, the fourth entropy coding module may output the first bitstream to a fourth entropy decoding module.

[0248] S505 , performing probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame; wherein the second residual super-prior information of the current frame is obtained based on decoding the first bitstream.

[0249] Next, the fourth entropy decoding module can perform entropy decoding on the first bitstream to obtain the first residual super-prior information after entropy decoding of the current frame; and output the first residual super-prior information after entropy decoding of the current frame to the first super-prior decoding network. Afterwards, the first super-prior decoding network can perform feature recovery on the first residual super-prior information after entropy decoding of the current frame to obtain the second residual super-prior information of the current frame (wherein the second residual super-prior information can also be referred to as the second residual super-prior feature); and output the second residual super-prior information of the current frame to the second fusion module. In other words, the "decoding" based on the decoding of the first bitstream in S505 includes two steps: entropy decoding and feature recovery.

[0250] For example, the prior information of the residual information of the current frame may also be input into the second fusion module. Then, the second fusion module may perform probability estimation based on the second residual super prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame. The residual probability distribution of the current frame is input into the second entropy coding module. The residual probability distribution of the current frame is also the probability distribution of the residual information of the current frame.

[0251] S506 , performing entropy coding on the residual information of the current frame based on the residual probability distribution of the current frame to obtain a second bitstream.

[0252] For example, S506 may refer to the description of the embodiment of FIG. 2A above, and will not be described in detail here.

[0253] It should be noted that the residual information of the current frame in S502 , S503 , S505 and S506 is the first residual information of the current frame in FIG. 2A .

[0254] Fig. 6 is a schematic diagram illustrating an exemplary decoding process, which corresponds to the encoding process of Fig. 5B.

[0255] S601, receiving a code stream; wherein the code stream includes a first code stream and a second code stream.

[0256] S602: Decode the first code stream to obtain second residual super-prior information of the current frame.

[0257] For example, the first bitstream can be input into a fourth entropy decoding module, which can perform entropy decoding on the first bitstream to obtain first residual super-prior information of the current frame after entropy decoding. The fourth entropy decoding module can then output the first residual super-prior information of the current frame after entropy decoding to a second super-prior decoding network. The second super-prior decoding network can then perform feature recovery on the first residual super-prior information of the current frame after entropy decoding to obtain second residual super-prior information of the current frame. The second residual super-prior information of the current frame can then be output to a second fusion module.

[0258] S603 , performing probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame.

[0259] For example, the prior information of the residual information of the current frame may also be input into the second fusion module. Then, the second fusion module may perform probability estimation based on the second residual super prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame. The residual probability distribution of the current frame is input into the second entropy coding module. The residual probability distribution of the current frame is also the probability distribution of the residual information of the current frame.

[0260] S604 , performing entropy decoding on the second bitstream based on the residual probability distribution of the current frame to obtain decoded residual information of the current frame.

[0261] Afterwards, the second entropy coding module may perform entropy decoding on the second bitstream based on the residual probability distribution of the current frame to obtain decoded residual information of the current frame.

[0262] S605: Perform inter-frame prediction on the current frame to obtain prediction information of the current frame.

[0263] Exemplarily, the code stream received by the decoding end further includes: a third code stream and a fourth code stream. And the entropy decoding module in the AI ​​video decoding framework may further include a sixth entropy decoding module.

[0264] Exemplarily, the third code stream can be output to the sixth entropy decoding module, which can perform entropy decoding on the third code stream to obtain the third optical flow super-prior information after entropy decoding of the current frame; and output the third optical flow super-prior information after entropy decoding of the current frame to the fourth super-prior coding network. Then, the fourth super-prior coding network performs feature recovery on the third optical flow super-prior information after entropy decoding of the current frame to obtain the fourth optical flow super-prior information of the current frame; and output the fourth optical flow super-prior information of the current frame to the fourth fusion module. In addition, the prior information of the feature information of the first optical flow of the current frame can be input to the fourth fusion module, and the fourth fusion module performs probability estimation based on the prior information of the feature information of the first optical flow of the current frame and the fourth optical flow super-prior information of the current frame to determine the optical flow probability distribution of the current frame; and the optical flow probability distribution of the current frame is input to the first entropy decoding module. The first entropy decoding module then entropy decodes the fourth bitstream to obtain feature information of the first optical flow of the current frame after entropy decoding. This feature information is then input into the optical flow decoding module, which performs feature recovery on the feature information of the first optical flow of the current frame after entropy decoding to obtain the second optical flow of the current frame. A prediction is then performed based on the second optical flow of the current frame and the image information of the reference frame to obtain prediction information for the current frame. For details, refer to the description in FIG. 2B above and will not be repeated here.

[0265] Exemplarily, S605 can be performed by an inter-frame prediction module in the AI ​​video decoding framework, wherein the inter-frame prediction module in the AI ​​video decoding framework can include the first entropy estimation module, the optical flow decoding module and the prediction module in Figure 2B.

[0266] S606 , reconstructing based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

[0267] For example, S606 may refer to the description in FIG2B , which will not be repeated here. It should be noted that the decoded residual information in S603 , S604 and S606 is the second residual information of the current frame in FIG2B .

[0268] It should be noted that, in the embodiments of Figure 5A and Figure 6, the first code stream may refer to the code stream of the first residual super-prior information, and the second code stream may refer to the residual code stream (corresponding to the residual code stream in Figures 2A and 2B); the third code stream may be the code stream of the first optical flow super-prior information, and the fourth code stream may refer to the optical flow code stream (corresponding to the optical flow code stream in Figures 2A and 2B).

[0269] FIG7 is a schematic diagram illustrating an exemplary encoding process.

[0270] S701 , performing motion estimation on the current frame to obtain a first optical flow of the current frame.

[0271] S702: Perform feature extraction on the first optical flow of the current frame to obtain feature information of the first optical flow of the current frame.

[0272] S703 : Determine super-prior information of the first optical flow of the current frame based on the feature information of the first optical flow of the current frame and the prior information of the first optical flow of the current frame.

[0273] S704 , encode the first optical flow super-prior information of the current frame to obtain a first code stream.

[0274] S705 , performing probability estimation based on the second optical flow super prior information of the current frame and the first optical flow prior information of the current frame to obtain an optical flow probability distribution of the current frame; wherein the second optical flow super prior information of the current frame is obtained based on decoding of the first bitstream.

[0275] S706 , performing entropy coding on feature information of the first optical flow of the current frame based on the optical flow probability distribution to obtain a second code stream.

[0276] S707: Perform inter-frame prediction on the current frame to obtain prediction information of the current frame.

[0277] S708 : Determine residual information of the current frame based on the prediction information of the current frame and the original image of the current frame.

[0278] S709 : Determine first residual super-prior information of the current frame based on the residual information of the current frame and the prior information of the residual information of the current frame.

[0279] S710 , encode the first residual super-prior information of the current frame to obtain a fourth bitstream.

[0280] S711, perform probability estimation based on the second residual super priori information of the current frame and the priori information of the residual information of the current frame to obtain the residual probability distribution of the current frame; wherein the second residual super priori information of the current frame is obtained based on decoding the fourth bitstream.

[0281] S712 , performing entropy coding on the residual information of the current frame based on the residual probability distribution of the current frame to obtain a third bitstream.

[0282] For example, S701 to S712 may refer to the description in the embodiment of FIG3B and FIG5B , and will not be described in detail here.

[0283] FIG8 is a schematic diagram showing an exemplary decoding process. The decoding process in the embodiment of FIG8 corresponds to the encoding process in the embodiment of FIG7.

[0284] S801, receiving code streams; wherein the code streams include a first code stream, a second code stream, and a third code stream.

[0285] S802: Decode the first code stream to obtain the second optical flow super-prior information of the current frame.

[0286] S803 , performing probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain an optical flow probability distribution of the current frame.

[0287] S804 , performing entropy decoding on the second bitstream based on the optical flow probability distribution of the current frame to obtain feature information of the second optical flow of the current frame.

[0288] S805 , performing feature recovery on feature information of the second optical flow of the current frame to obtain the second optical flow of the current frame.

[0289] S806 , performing inter-frame prediction based on the second optical flow of the current frame to obtain prediction information of the current frame; wherein the second optical flow of the current frame is obtained by performing feature recovery based on feature information of the second optical flow of the current frame.

[0290] S807 , decoding the fourth bit stream to obtain second residual super-prior information of the current frame.

[0291] S808 , performing probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame.

[0292] S809 , performing entropy decoding on the third bitstream based on the residual probability distribution of the current frame to obtain decoded residual information of the current frame.

[0293] S810 , reconstructing based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

[0294] For example, S801 to S810 may refer to the description in the embodiment of FIG. 4 and the embodiment of FIG. 6 , and will not be described in detail here.

[0295] It should be noted that, in the embodiments of Figures 7 and 8, the first code stream may refer to the code stream of the first optical flow super-prior information, and the second code stream may refer to the code stream of the optical flow (corresponding to the optical flow code stream in Figures 2A and 2B); the third code stream may be the residual code stream (corresponding to the residual code stream in Figures 2A and 2B), and the fourth code stream may refer to the code stream of the first residual super-prior information.

[0296] The following test set is used to test the encoding method of the embodiment of Figure 7 and the decoding method of the embodiment of Figure 8, the traditional encoding and decoding method of the prior art (such as H266) and the AI ​​video encoding and decoding method of the prior art, and compare the compression performance of the encoding method of the embodiment of Figure 7 and the decoding method of the embodiment of Figure 8 of this application and the AI ​​video encoding and decoding method of the prior art.

[0297] For example, the test set includes 5: two 4k test sets (of which one 4K (A1) test set has a complex motion type and one 4K test set (A2) has a simple motion type), a 1080p test set, an 832*480 test set (target object motion), and a 720p test set (such as conference video); the peak signal-to-noise ratio (PSNR) of the encoding method of the embodiment of FIG. 7 and the decoding method of the embodiment of FIG. 8 relative to H266, as well as the PSNR of the AI ​​video encoding and decoding method of the prior art relative to H266, can be shown in Table 1 below:

[0298] Table 1

[0299] Referring to Table 1, Y-PSNR represents the PSNR of the Y component, U-PSNR represents the PSNR of the U component, and V-PSNR represents the PSNR of the V component.

[0300] It should be noted that the video data used for testing in the test set contains YUV420 images, where the Y component: U component: V component = 8:1:1; therefore, Table 1 focuses on the Y-PSNR. Referring to Table 1, compared with the AI ​​video encoding method using the prior art, encoding and decoding using the encoding method of the embodiment of Figure 7 and the decoding method of the embodiment of Figure 8 of this application can improve the compression effect. That is, at the same bit rate, the reconstructed image quality is higher, or in other words, at the same quality, the bit rate is lower.

[0301] In addition, since the feature information, residual information, and super-prior coding network of the first optical flow are of small scale, the increased computational complexity is very small, and the decoding complexity at the decoding end is not increased.

[0302] It should be noted that in the above embodiment, the fourth code stream, the third code stream, the second code stream, and the first code stream can be four code streams. In one possible embodiment, the fourth code stream, the third code stream, the second code stream, and the first code stream are four parts of the same code stream. That is, one code stream simultaneously carries residual information, the first optical stream, the first optical stream super-prior information, and the first residual super-prior information. In one possible embodiment, any two of the fourth code stream, the third code stream, the second code stream, and the first code stream can form a code stream. In this way, the fourth code stream, the third code stream, the second code stream, and the first code stream can form two code streams. In one possible embodiment, any three of the fourth code stream, the third code stream, the second code stream, and the first code stream can form a code stream. In this way, the fourth code stream, the third code stream, the second code stream, and the first code stream can form two code streams.

[0303] In an example, FIG9 shows a schematic block diagram of a device 900 according to an embodiment of the present application. The device 900 may include: a processor 901 and a transceiver / transceiver pin 902 , and optionally, a memory 903 .

[0304] The various components of the device 900 are coupled together via a bus 904, wherein the bus 904 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all buses are referred to as bus 904 in the figure.

[0305] Optionally, the memory 903 may be used to store instructions in the aforementioned method embodiment. The processor 901 may be used to execute the instructions in the memory 903 and control the receiving pin to receive a signal and control the transmitting pin to send a signal.

[0306] The apparatus 900 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0307] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0308] The present application also provides a chip including one or more interface circuits and one or more processors. The one or more processors receive or send data via the one or more interface circuits. When the one or more processors execute computer instructions, the steps of the above-mentioned related methods are implemented. The interface circuit is a transceiver / transceiver pin 902.

[0309] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0310] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0311] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the methods in the above-mentioned method embodiments.

[0312] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0313] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0314] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0315] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0316] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0317] Any content of each embodiment of this application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0318] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0319] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0320] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0321] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A coding method, characterized in that: The method comprises: Performing motion estimation on the current frame to obtain a first optical flow of the current frame; Determine the super-prior information of the first optical flow of the current frame based on the feature information of the first optical flow of the current frame and the prior information of the first optical flow of the current frame; Encoding the first optical flow super-prior information of the current frame to obtain a first bitstream; Probability estimation is performed based on the second optical flow super prior information of the current frame and the first optical flow prior information of the current frame to obtain the optical flow probability distribution of the current frame; wherein the second optical flow super prior information of the current frame is obtained based on decoding of the first bitstream; Performing entropy coding on feature information of a first optical flow of the current frame based on an optical flow probability distribution of the current frame to obtain a second bitstream; Performing inter-frame prediction on the current frame to obtain prediction information of the current frame; Determining residual information of the current frame based on the prediction information of the current frame and the original image of the current frame; Encode the residual information of the current frame to obtain a third bit stream.

2. The method according to claim 1, characterized in that The prior information of the first optical flow of the current frame includes at least one of the time domain prior information of the first optical flow of the current frame or the space domain prior information of the first optical flow of the current frame.

3. The method according to claim 2, characterized in that The time domain prior information of the first optical flow of the current frame includes feature information of the second optical flow of m frames before the current frame, where m is a positive integer; The spatial domain prior information of the first optical flow of the current frame includes feature information of a reconstructed image of a reference frame corresponding to the current frame.

4. The method according to any one of claims 1 to 3, characterized in that: The encoding of the residual information of the current frame to obtain a third bitstream includes: Determine first residual super prior information of the current frame based on the residual information of the current frame and the prior information of the residual information of the current frame; Encoding the first residual super-prior information of the current frame to obtain a fourth bit stream; Probability estimation is performed based on the second residual super prior information of the current frame and the prior information of the residual information of the current frame to obtain the residual probability distribution of the current frame; wherein the second residual super prior information of the current frame is obtained based on the decoding of the fourth bitstream; The residual information of the current frame is entropy encoded based on the residual probability distribution of the current frame to obtain the third code stream.

5. The method according to claim 4, characterized in that The prior information of the residual information of the current frame includes at least one of the time domain prior information of the residual information of the current frame or the spatial domain prior information of the residual information of the current frame.

6. The method according to claim 5, characterized in that The time domain prior information of the residual information of the current frame includes the residual information of n frames before the current frame, where n is a positive integer; The spatial domain prior information of the residual information of the current frame includes prediction information of the current frame.

7. The method according to any one of claims 1 to 6, characterized in that: The method is applied to an AI video coding framework, which includes a motion estimation module, an entropy estimation module, an entropy coding module, a prediction module and a residual coding module, wherein: The motion estimation module is used to perform motion estimation on the current frame to obtain a first optical flow of the current frame; The entropy estimation module is used to determine the super prior information of the first optical flow of the current frame based on the feature information of the first optical flow of the current frame and the prior information of the first optical flow of the current frame; and output the super prior information of the first optical flow of the current frame to the entropy coding module; The entropy coding module is used to encode the first optical flow super-prior information of the current frame to obtain the first bit stream; The entropy estimation module is further used to perform probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain the optical flow probability distribution of the current frame; and output the optical flow probability distribution to the entropy coding module; The entropy coding module is further used to perform entropy coding on feature information of the first optical flow of the current frame based on the optical flow probability distribution of the current frame to obtain the second code stream; The prediction module is used to perform inter-frame prediction on the current frame to obtain prediction information of the current frame; and output the prediction information of the current frame to the residual coding module; The residual coding module is used to determine the residual information of the current frame based on the prediction information of the current frame and the original image of the current frame; and output the residual information of the current frame to the entropy coding module; The entropy coding module is further used to encode the residual information of the current frame to obtain the third code stream.

8. A decoding method, characterized in that: The method comprises: Receive a code stream; wherein the code stream includes a first code stream, a second code stream and a third code stream; Decoding the first bitstream to obtain super-prior information of the second optical flow of the current frame; Performing probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain an optical flow probability distribution of the current frame; Performing entropy decoding on the second bitstream based on the optical flow probability distribution of the current frame to obtain feature information of the second optical flow of the current frame; Performing inter-frame prediction based on the second optical flow of the current frame to obtain prediction information of the current frame; wherein the second optical flow of the current frame is obtained by performing feature recovery based on feature information of the second optical flow of the current frame; Based on the third bitstream, obtaining decoded residual information of the current frame; Reconstruction is performed based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

9. The method according to claim 8, characterized in that The prior information of the first optical flow of the current frame includes at least one of the time domain prior information of the first optical flow of the current frame or the space domain prior information of the first optical flow of the current frame.

10. The method according to claim 9, characterized in that The time domain prior information of the first optical flow of the current frame includes feature information of the second optical flow of m frames before the current frame, where m is a positive integer; The spatial domain prior information of the first optical flow of the current frame includes feature information of a reconstructed image of a reference frame corresponding to the current frame.

11. The method according to any one of claims 8 to 10, characterized in that: The bitstream also includes a fourth bitstream; and obtaining the residual information of the current frame based on the third bitstream includes: Decoding the fourth bit stream to obtain second residual super-prior information of the current frame; Probability estimation is performed based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame; The third bitstream is entropy decoded based on the residual probability distribution of the current frame to obtain decoded residual information of the current frame.

12. The method according to claim 11, characterized in that The prior information of the residual information of the current frame includes at least one of time domain prior information of the decoded residual information of the current frame or space domain prior information of the decoded residual information of the current frame.

13. The method according to claim 12, characterized in that The time domain prior information of the decoded residual information of the current frame includes the decoded residual information of n frames before the current frame, where n is a positive integer; The spatial domain prior information of the decoded residual information of the current frame includes prediction information of the current frame.

14. The method according to any one of claims 8 to 13, characterized in that The method is applied to an AI video decoding framework, which includes a motion estimation module, an entropy estimation module, an entropy decoding module, a prediction module and a reconstruction module, wherein: The entropy decoding module is used to decode the first bit stream to obtain the second optical flow super-prior information of the current frame; and output the second optical flow super-prior information of the current frame to the entropy estimation module; The entropy estimation module is used to perform probability estimation based on the second optical flow super-prior information of the current frame and the prior information of the first optical flow of the current frame to obtain the optical flow probability distribution of the current frame; and output the optical flow probability distribution to the entropy decoding module; The entropy decoding module is further used to perform entropy decoding on the second bitstream based on the optical flow probability distribution of the current frame to obtain feature information of the second optical flow of the current frame; The prediction module is used to perform inter-frame prediction based on the second optical flow of the current frame to obtain prediction information of the current frame; and output the prediction information of the current frame to the reconstruction module; The entropy decoding module is used to obtain the decoded residual information of the current frame based on the third bit stream; and output the decoded residual information of the current frame to the reconstruction module; The reconstruction module is used to reconstruct based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

15. A coding method, characterized in that: The method comprises: Performing inter-frame prediction on the current frame to obtain prediction information of the current frame; Determining residual information of the current frame based on the prediction information of the current frame and the original image of the current frame; Determine first residual super prior information of the current frame based on the residual information of the current frame and the prior information of the residual information of the current frame; Encoding the first residual super-prior information of the current frame to obtain a first bit stream; Probability estimation is performed based on the second residual super prior information of the current frame and the prior information of the residual information of the current frame to obtain the residual probability distribution of the current frame; wherein the second residual super prior information of the current frame is obtained based on decoding of the first bitstream; Perform entropy coding on the residual information of the current frame based on the residual probability distribution of the current frame to obtain a second code stream.

16. The method according to claim 11, characterized in that The prior information of the residual information of the current frame includes at least one of the time domain prior information of the residual information of the current frame or the spatial domain prior information of the residual information of the current frame.

17. The method according to claim 16, characterized in that The time domain prior information of the residual information of the current frame includes the residual information of n frames before the current frame, where n is a positive integer; The spatial domain prior information of the residual information of the current frame includes prediction information of the current frame.

18. The method according to any one of claims 15 to 17, characterized in that The method is applied to an AI video coding framework, which includes an entropy estimation module, an entropy coding module, an inter-frame prediction module and a residual coding module, wherein: The inter-frame prediction module is used to perform inter-frame prediction on the current frame to obtain prediction information of the current frame; and output the prediction information of the current frame to the residual coding module; The residual coding module is used to determine the residual information of the current frame based on the prediction information of the current frame and the original image of the current frame; and output the residual information of the current frame to the entropy estimation module; The entropy estimation module is used to determine the first residual super prior information of the current frame based on the residual information of the current frame and the prior information of the residual information of the current frame; and output the first residual super prior information of the current frame to the entropy coding module; The entropy coding module is used to encode the first residual super-prior information of the current frame to obtain a first bit stream; The entropy estimation module is further used to perform probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain the residual probability distribution of the current frame; and output the residual probability distribution to the entropy coding module; The entropy coding module is further used to perform entropy coding on the residual information of the current frame based on the residual probability distribution of the current frame to obtain the second code stream.

19. A decoding method, characterized in that: The method comprises: Receive a code stream; wherein the code stream includes a first code stream and a second code stream; Decoding the first bitstream to obtain second residual super-prior information of the current frame; Probability estimation is performed based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain a residual probability distribution of the current frame; Performing entropy decoding on the second bitstream based on the residual probability distribution of the current frame to obtain decoded residual information of the current frame; Performing inter-frame prediction on the current frame to obtain prediction information of the current frame; Reconstruction is performed based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

20. The method according to claim 19, characterized in that The prior information of the residual information of the current frame includes at least one of time domain prior information of the decoded residual information of the current frame or space domain prior information of the decoded residual information of the current frame.

21. The method according to claim 20, characterized in that The time domain prior information of the decoded residual information of the current frame includes the decoded residual information of n frames before the current frame, where n is a positive integer; The spatial domain prior information of the decoded residual information of the current frame includes prediction information of the current frame.

22. The method according to any one of claims 19 to 21, characterized in that The method is applied to an AI video decoding framework, which includes an inter-frame prediction module, an entropy estimation module, an entropy decoding module, a prediction module and a reconstruction module, wherein: The entropy decoding module is used to decode the first bit stream to obtain the second residual super-prior information of the current frame; and output the second residual super-prior information of the current frame to the entropy estimation module; The entropy estimation module is used to perform probability estimation based on the second residual super-prior information of the current frame and the prior information of the residual information of the current frame to obtain the residual probability distribution of the current frame; and output the residual probability distribution to the entropy coding module; The entropy coding module is used to perform entropy decoding on the second bit stream based on the residual probability distribution of the current frame to obtain the decoded residual information of the current frame; and output the decoded residual information of the current frame to the reconstruction module; The inter-frame prediction module is used to perform inter-frame prediction on the current frame to obtain prediction information of the current frame; and output the prediction information of the current frame to the reconstruction module; The reconstruction module is used to reconstruct based on the prediction information of the current frame and the decoded residual information of the current frame to obtain a reconstructed image of the current frame.

23. A coding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the encoding device executes the method according to any one of claims 1 to 7 or any one of claims 15 to 18.

24. A decoding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the decoding device executes the method according to any one of claims 8 to 14 or any one of claims 19 to 22.

25. A decoding device, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method as described in any one of claims 1 to 7 or any one of claims 15 to 18 are executed.

26. A decoding device, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method as described in any one of claims 8 to 14 or any one of claims 19 to 22 are executed.

27. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 22.

28. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 22 are performed.

Images