Encoding and decoding method and electronic equipment

Abstract

The embodiment of the invention provides a coding and decoding method and electronic equipment. The method comprises the following steps: performing inter-frame prediction on a coding unit in a current frame to obtain first prediction information of the coding unit; determining residual information of the coding unit according to the first prediction information and an original image of the coding unit; determining a preset parameter of the coding unit, the preset parameter being used for fusing the reconstructed image information of the reference unit and the reconstructed image information of the coding unit to obtain a target reconstructed image of the coding unit, the reconstructed image information of the reference unit comprising a target reconstructed image of the reference unit or feature information of the target reconstructed image of the reference unit, the reconstructed image information of the coding unit comprises an initial reconstructed image of the coding unit or feature information of the initial reconstructed image of the coding unit; afterwards, encoding residual error information; and encoding the preset parameter. Therefore, accumulative errors can be reduced, and time sequence stability is ensured.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of coding and decoding, and in particular to a coding and decoding method and an electronic device. Background Art

[0002] As videos evolve from high-definition to ultra-high-definition, people have higher and higher requirements for video quality. At the same time, high-definition videos have higher and higher requirements for bandwidth and storage. Accordingly, due to cost control considerations for bandwidth, transmission delay, and storage, the demand for video encoding, or video compression, has become more and more urgent.

[0003] Among them, the artificial intelligence (AI) video compression (or codec) algorithm is based on deep learning and has better compression effect than traditional video compression technologies (such as H265 and H266).

[0004] Since there is temporal continuity between video frames, the change between the previous frame and the next frame is relatively small, that is, there is temporal redundancy between video frames; thus, inter-frame coding can be used to reduce the bitrate overhead. In the inter-frame coding process of AI video coding, many steps such as prediction and entropy coding of residual information will introduce errors; thus, in the process of encoding a group of pictures (GOP), the cumulative error will continue to increase, and as the cumulative error increases, the compression performance will also decrease. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a coding method and electronic device. The method can reduce the cumulative error to a certain extent and ensure the timing stability (i.e., ensure the stability of the compression performance of each frame in each GOP (GOP refers to a group of images between I frames, and I frame is Intra frame, that is, intra-frame coded frame)).

[0006] In a first aspect, an embodiment of the present application provides a coding method, the method comprising: first, performing inter-frame prediction on a coding unit in a current frame to obtain first prediction information of the coding unit; then, determining residual information of the coding unit based on the first prediction information and the original image of the coding unit; thereafter, determining preset parameters of the coding unit; wherein the preset parameters are used to fuse reconstructed image information of a reference unit and reconstructed image information of the coding unit to obtain a target reconstructed image of the coding unit, the reconstructed image information of the reference unit includes the target reconstructed image of the reference unit or feature information of the target reconstructed image of the reference unit, the reconstructed image information of the coding unit includes an initial reconstructed image of the coding unit or feature information of the initial reconstructed image of the coding unit, the initial reconstructed image of the coding unit and the feature information of the initial reconstructed image of the coding unit are reconstructed based on the first prediction information and the residual information, the reference unit is a unit in a reference frame corresponding to the current frame; subsequently, encoding the residual information; and encoding the preset parameters.

[0007] Among them, the preset parameters can be used to describe the motion complexity of the image of the coding unit relative to the image of the reference unit; specifically, the motion complexity can refer to the difference or similarity; that is, the motion complexity of the image of the coding unit relative to the image of the reference unit can refer to the similarity between the image of the coding unit and the image of the reference unit, or can refer to the difference between the image of the coding unit and the image of the reference unit. For example, the smaller the difference or the greater the similarity between the image of the coding unit and the image of the reference unit, the lower the motion complexity of the image of the coding unit relative to the image of the reference unit; the greater the difference or the smaller the similarity between the image of the coding unit and the image of the reference unit, the higher the motion complexity of the image of the coding unit relative to the image of the reference unit.

[0008] The present application generates a target reconstructed image of the coding unit by fusing the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to preset parameters; that is, the reconstructed image information of the coding unit and the reconstructed image information of the reference unit are fused according to a fusion rule (or fusion strategy or fusion method) corresponding to the motion complexity of the image of the coding unit relative to the image of the reference unit to generate the target reconstructed image of the coding unit. In this way, for a coding unit whose image motion complexity is low relative to the reference unit, the error introduced by the inter-frame prediction and the coding residual information in the process of encoding the coding unit can be compensated to a certain extent, so that the error of the coding unit can be reduced to a certain extent, and then the error of the current frame can be reduced to a certain extent, and the compression performance of the current frame can be improved; thereby, the accumulated error of other subsequent frames can be reduced to a certain extent, the compression performance of other subsequent frames can be improved, and the timing stability can be guaranteed. For a coding unit whose image motion complexity is high relative to the reference unit, the quality of the target reconstructed image of the coding unit can be guaranteed, and then the quality of the target reconstructed image of the current frame can be guaranteed.

[0009] Exemplarily, the cumulative error is explained by taking the i-th frame as an example: the cumulative error of the reconstructed image of the i-th frame may refer to the error introduced in the process of encoding the i-th frame original image by the error between the reconstructed image of the i-th frame corresponding to the reference frame and the original image of the i-th frame corresponding to the reference frame. For example, in a GOP, the second frame original image is encoded with reference to the first frame reconstructed image. Since some steps in the encoding process may introduce errors, there is an error 1 between the second frame reconstructed image and the second frame original image. The third frame original image is encoded with reference to the second frame reconstructed image. There is an error 2 between the third frame reconstructed image and the third frame original image; wherein, error 2 includes error 3 introduced by error 1 in the process of encoding the third frame original image and error 4 introduced by some steps in encoding the third frame original image; error 3 may be called the cumulative error.

[0010] Exemplarily, the residual information of the coding unit may be the first residual of the coding unit (wherein the first residual may refer to the residual between the feature information of the original image of the coding unit (or the original image of the coding unit) and the first prediction information of the coding unit), or may be the feature information of the first residual of the coding unit (wherein the feature information of the first residual may also be referred to as the compressed feature of the first residual, which may be obtained by feature extraction of the first residual), and the present application does not impose any limitation on this. The present application takes the residual information of the coding unit as the feature information of the first residual of the coding unit as an example for explanation.

[0011] Exemplarily, the residual bitstream obtained by encoding the residual information may be first decoded to obtain the decoded residual information; then, reconstruction may be performed based on the first prediction information and the decoded residual information to obtain the initial reconstructed image of the coding unit and the feature information of the initial reconstructed image of the coding unit. The feature information of the initial reconstructed image of the coding unit may be restored to obtain the initial reconstructed image of the coding unit.

[0012] Exemplarily, the feature information of the target reconstructed image of the reference unit may be obtained by performing feature extraction on the target reconstructed image of the reference unit.

[0013] In a possible manner, the preset parameters of the encoding unit may be directly encoded to obtain a preset parameter bitstream; in this way, the decoding end may decode the preset parameter of the current frame from the preset parameter bitstream.

[0014] In one possible manner, feature extraction can be performed on the preset parameters of the coding unit to obtain feature information of the preset parameters of the coding unit; then, the feature information of the preset parameters of the coding unit is encoded to obtain a preset parameter bitstream. In this way, the decoding end can perform feature recovery based on the information decoded from the preset parameter bitstream to obtain the preset parameters of the current frame.

[0015] Exemplarily, the current frame can be divided into multiple coding units, and the encoding method of the present application can be executed for each coding unit; wherein the coding unit can include at least one pixel point, and when the coding unit includes multiple pixel points, the multiple pixel points included in the coding unit share a preset parameter.

[0016] Exemplarily, the encoding method of the present application may refer to an AI video encoding method, which can be used to perform inter-frame prediction on image frames in video data.

[0017] Exemplarily, residual information is encoded to obtain a residual bitstream.

[0018] It should be noted that in one possible manner, the residual bitstream and the preset parameter bitstream are two bitstreams. In one possible manner, the residual bitstream and the preset parameter bitstream may be two parts of the same bitstream, that is, the same bitstream carries the residual information and the preset parameters.

[0019] It should be noted that the present application does not limit the order of encoding residual information and encoding preset parameters.

[0020] According to the first aspect, encoding the residual information includes: performing adaptive rate encoding on the residual information according to preset parameters, so as to reduce the rate overhead.

[0021] That is to say, the bit rate of residual information encoding is adaptively adjusted according to the preset parameters of the encoding unit. Since the reconstructed image information of the reference unit and the reconstructed image information of the encoding unit are fused during the reconstruction process to generate the target reconstructed image of the encoding unit, even if the residual information of the current frame is encoded at a low bit rate, the quality of the target reconstructed image of the encoding unit can be guaranteed, thereby reducing the bit rate overhead to a certain extent under the same quality.

[0022] According to the first aspect, or any implementation of the first aspect above, inter-frame prediction is performed on the coding unit in the current frame to obtain first prediction information of the coding unit, including: determining the first optical flow of the coding unit according to the original image of the coding unit and the target reconstructed image of the reference unit; determining the first prediction information according to the first optical flow and the reconstructed image information of the reference unit; the method also includes: adaptively encoding the first optical flow according to preset parameters. In this way, the bit rate overhead can be reduced.

[0023] That is to say, the bit rate of the first optical flow encoding is adaptively adjusted according to the preset parameters of the encoding unit. Since the reconstructed image information of the reference unit and the reconstructed image information of the encoding unit are fused during the reconstruction process to generate the target reconstructed image of the encoding unit, even if the feature information of the first optical flow of the current frame is encoded at a low bit rate, the quality of the target reconstructed image of the encoding unit can be guaranteed, and thus the bit rate overhead can be reduced to a certain extent under the same quality.

[0024] Exemplarily, according to preset parameters, performing adaptive rate encoding on the first optical flow may include: determining characteristic information of the first optical flow; according to the preset parameters, performing adaptive rate encoding on the characteristic information of the first optical flow; wherein encoding the characteristic information of the first optical flow may obtain an optical flow code stream.

[0025] It should be noted that, in one possible manner, the optical stream code stream, the residual code stream and the preset parameter code stream are three code streams. In one possible manner, the optical stream code stream, the residual code stream and the preset parameter code stream can be three parts of the same code stream, that is, the same code stream carries the residual information, the preset parameters and the first optical stream. In one possible manner, the residual code stream and the preset parameter code stream can be two parts of the same code stream, and the optical stream code stream is another code stream. In one possible manner, the residual code stream and the optical stream code stream can be two parts of the same code stream, and the preset parameter code stream is another code stream. In one possible manner, the preset parameter code stream and the optical stream code stream can be two parts of the same code stream, and the residual code stream is another code stream; this application does not impose any restrictions on this.

[0026] It should also be noted that the present application does not limit the order of encoding the first optical flow, encoding the residual information, and encoding the preset parameters.

[0027] According to the first aspect, or any implementation method of the first aspect above, determining the preset parameters of the coding unit includes: determining the similarity between the image of the coding unit and the image of the reference unit; wherein the image of the coding unit includes the original image of the coding unit and / or the initial reconstructed image of the coding unit, and the image of the reference unit includes the original image of the reference unit and / or the target reconstructed image of the reference unit; and determining the preset parameters based on the similarity.

[0028] Wherein, when the similarity is greater than or equal to the similarity threshold, it means that the similarity between the image of the coding unit and the image of the reference unit is high. At this time, it can be determined that the motion complexity of the image of the coding unit relative to the image of the reference unit is low, and the value of the preset parameter can be set to the first preset value; that is, the first preset value can be used to indicate that the motion complexity of the image of the coding unit relative to the image of the reference unit is low complexity. When the similarity is less than the similarity threshold, it means that the similarity between the image of the coding unit and the image of the reference unit is low. At this time, it can be determined that the motion complexity of the image of the coding unit relative to the image of the reference unit is high, and the value of the preset parameter can be set to the second preset value; that is, the second preset value can be used to indicate that the motion complexity of the image of the coding unit relative to the image of the reference unit is high complexity. Wherein, the similarity threshold can be set according to demand. Assuming that the similarity when two images are exactly the same is 1, the similarity threshold can be a decimal less than 1 and greater than 0.

[0029] It should be noted that the first preset value and the second preset value are different, the first preset value may be greater than the second preset value, or the first preset value may be less than the second preset value; it may be set specifically as required, and this application does not limit this. In one possible manner, the first preset value and the second preset value may be numbers between 0 and 1 (the values ​​of the first preset value and the second preset value may both be 0 or 1). Optionally, the sum of the first preset value and the second preset value may be 1. For example, the first preset value is 1, and the second preset value is 0; or, the first preset value is 0, and the second preset value is 1. For another example, the first preset value is 0.8, and the second preset value is 0.2; or, the first preset value is 0.2, and the second preset value is 0.8.

[0030] In one possible manner, the error between the image of the coding unit and the image of the reference unit may be calculated, and the similarity between the image of the coding unit and the image of the reference unit may be determined based on the error between the image of the coding unit and the image of the reference unit. Assuming that the similarity when the two images are exactly the same is 1, the difference between 1 and the error between the image of the coding unit and the image of the reference unit may be used as the similarity.

[0031] In one possible way, the third optical flow of the coding unit can be obtained; the second prediction information of the coding unit is determined according to the third optical flow of the coding unit and the target reconstructed image of the reference unit; the similarity between the image of the coding unit and the image of the reference unit is determined according to the error between the second prediction information of the coding unit and the image of the coding unit. Among them, the initial reconstructed image of the coding unit and the target reconstructed image of the reference unit can be input into the motion estimation module, and the motion estimation module performs motion estimation and outputs the third optical flow of the coding unit. Alternatively, the feature information of the first optical flow of the coding unit is input into the optical flow decoding model, and the optical flow decoding model decodes the feature information of the first optical flow (that is, feature recovery) and outputs the third optical flow of the coding unit.

[0032] It should be understood that there are many ways to determine the similarity, for example, calculating the error and determining the similarity based on the error; for example, using a similarity model to determine the similarity calculation, etc., which is not limited in this application. In addition, other methods can be used to calculate the error, which is not limited in this application.

[0033] According to the first aspect, or any implementation method of the first aspect above, the reconstructed image information of the reference unit and the reconstructed image information of the coding unit are fused to obtain the target reconstructed image of the coding unit, including: determining the weight of the reconstructed image information of the reference unit and the weight of the reconstructed image information of the coding unit according to preset parameters; performing weighted calculation on the reconstructed image information of the reference unit and the reconstructed image information of the coding unit according to the weight of the reconstructed image information of the reference unit and the weight of the reconstructed image information of the coding unit to obtain the target reconstructed image of the coding unit.

[0034] Exemplarily, when the value of the preset parameter is a first preset value, the weight of the reconstructed image information of the coding unit may be smaller than the weight of the reconstructed image information of the reference unit; when the value of the preset parameter is a second preset value, the weight of the reconstructed image information of the coding unit may be greater than the weight of the reconstructed image information of the reference unit.

[0035] Wherein, if the first preset value is greater than the second preset value, then when the first preset value and the second preset value are numbers between 0 and 1 (the first preset value and the second preset value can both be 0 or 1), the value of the preset parameter of the coding unit can be used as the weight of the reconstructed image information of the reference unit, and the difference between 1 and the value of the preset parameter can be used as the weight of the reconstructed image information of the coding unit. When the first preset value and the second preset value are both numbers greater than 1, the first preset value and the second preset value can be transformed into numbers between 0 and 1, and then the value of the preset parameter after the transformation of the coding unit can be used as the weight of the reconstructed image information of the reference unit, and the difference between 1 and the value of the preset parameter after the transformation of the coding unit can be used as the weight of the reconstructed image information of the coding unit. If the first preset value is less than the second preset value, then vice versa.

[0036] When the motion complexity of the image of the coding unit relative to the image of the reference unit is low complexity and the first preset value is 1, the reconstructed image information of the reference unit and the reconstructed image information of the coding unit are fused, and the reconstructed image information of the reference unit is copied to the coding unit to obtain the target reconstructed image of the coding unit. In this way, the error introduced by the inter-frame prediction and the entropy coding of the residual information can be compensated, the error of the current frame can be reduced, and the accumulated error of the subsequent frames can be reduced to ensure the temporal stability.

[0037] When the motion complexity of the image of the coding unit relative to the image of the reference unit is high complexity and the second preset value is 0, the target reconstructed image of the coding unit obtained by fusing the reconstructed image information of the reference unit and the reconstructed image information of the coding unit is substantially the initial reconstructed image of the coding unit; since the motion complexity of the image of the coding unit relative to the image of the reference unit is high complexity, if the reconstructed image information of the reference unit is copied to the coding unit, the quality of the target reconstructed image of the coding unit obtained is low; therefore, the initial reconstructed image of the coding unit obtained by reconstruction can be used as the initial reconstructed image of the coding unit; in this way, the quality of the reconstructed image of the coding unit can be guaranteed.

[0038] When the motion complexity of the image of the coding unit relative to the image of the reference unit is low complexity, and the first preset value and the second preset value are other values ​​except 0 and 1, in the target image of the coding unit obtained by fusing the reconstructed image information of the reference unit and the reconstructed image information of the coding unit, the reconstructed image information of the coding unit and the reconstructed image information of the reference unit both occupy a certain proportion; in this way, the error introduced by the inter-frame prediction and the entropy coding of the residual information can be compensated to a certain extent, and the error of the current frame can be reduced; thereby, the accumulated error of the subsequent frames can be reduced to a certain extent, and the temporal stability can be ensured. And the quality of the target reconstructed image of the coding unit can also be guaranteed to a certain extent.

[0039] According to the first aspect, or any implementation of the first aspect above, the reconstructed image information of the reference unit and the reconstructed image information of the coding unit are fused to obtain the target reconstructed image of the coding unit, including: inputting preset parameters, the reconstructed image information of the reference unit and the reconstructed image information of the coding unit into a fusion model to obtain the target reconstructed image of the coding unit. In this case, the size of the first preset value and the second preset value may not be limited; wherein the fusion model is pre-trained and has the ability to generate high-quality information (such as high-quality images or high-quality feature information). In this way, the error of the current frame coding unit can also be reduced, thereby reducing the cumulative error of subsequent frames and ensuring timing stability. And the quality of the target reconstructed image of the coding unit can also be guaranteed to a certain extent.

[0040] According to the first aspect, or any implementation manner of the first aspect above, adaptive rate encoding is performed on the residual information according to a preset parameter, including: when the value of the preset parameter is a first preset value, encoding the residual information according to the first rate parameter; when the value of the preset parameter is a second preset value, encoding the residual information according to the second rate parameter; wherein a target bit rate of a code stream obtained by encoding the residual information according to the second rate parameter is greater than a target bit rate of a code stream obtained by encoding the residual information according to the first rate parameter.

[0041] Since when the value of the preset parameter is the first preset value, the weight of the reconstructed image information of the coding unit is less than the weight of the reconstructed image information of the reference unit; when the value of the preset parameter is the second preset value, the weight of the reconstructed image information of the coding unit is greater than the weight of the reconstructed image information of the reference unit; therefore, when the motion complexity of the image of the coding unit is lower than that of the image of the reference unit (i.e., the value of the preset parameter is the first preset value), the residual information can be encoded at a low bit rate, so that the bit rate overhead can be reduced without reducing the quality of the target reconstructed image of the coding unit. When the motion complexity of the image of the coding unit is higher than that of the image of the reference unit (i.e., the value of the preset parameter is the second preset value), the residual information can be encoded at a high bit rate, so that the quality of the target reconstructed image of the coding unit can be guaranteed.

[0042] Exemplarily, when the value of the preset parameter of the encoding unit is a first preset value, the characteristic information of the first optical flow of the encoding unit can be encoded at a low bit rate to obtain an optical flow code stream; when the value of the preset parameter of the encoding unit is a second preset value, the characteristic information of the first optical flow of the encoding unit can be encoded at a high bit rate to obtain an optical flow code stream.

[0043] Specifically, when the value of the preset parameter is a first preset value, the characteristic information of the first optical flow is encoded according to the third bit rate parameter; when the value of the preset parameter is a second preset value, the characteristic information of the first optical flow is encoded according to the fourth bit rate parameter; wherein the target bit rate of the bit stream obtained by encoding the first optical flow according to the fourth bit rate parameter is greater than the target bit rate of the bit stream obtained by encoding the first optical flow according to the third bit rate parameter.

[0044] According to the first aspect, or any implementation of the first aspect above, encoding preset parameters includes: performing entropy estimation based on at least one item of the residual information or the second optical flow of the encoding unit to determine the probability distribution of the preset parameters; wherein the second optical flow is obtained based on the code stream obtained by encoding the first optical flow; and entropy encoding the preset parameters according to the probability distribution of the preset parameters.

[0045] According to the first aspect, or any implementation of the first aspect above, the method further includes: according to preset parameters, fusing the first prediction information and the reconstructed image information of the reference unit to obtain the second prediction information; according to the first prediction information and the original image of the coding unit, determining the residual information of the coding unit, including: according to the second prediction information and the original image of the coding unit, determining the residual information of the coding unit. In this way, the accuracy of the inter-frame prediction can be improved, thereby reducing the error introduced by the inter-frame prediction, further reducing the error of the current frame, and thus reducing the accumulated error of further subsequent frames.

[0046] According to the first aspect, or any implementation of the first aspect above, the method is applied to an AI video coding framework, the AI ​​video coding framework includes: an inter-frame prediction module, a residual coding module, an entropy coding module, a preset parameter determination module and a fusion module, wherein:

[0047] The inter-frame prediction module is used to perform inter-frame prediction on the coding unit to obtain first prediction information; and output the first prediction information to the residual coding module;

[0048] A residual coding module, used to determine residual information according to the first prediction information and the original image of the coding unit; and output the residual information to the entropy coding module;

[0049] A preset parameter determination module is used to determine the preset parameters; output the preset parameters to the entropy coding module and the fusion module;

[0050] An entropy coding module, used to encode residual information and preset parameters;

[0051] The fusion module is used to fuse the reconstructed image information of the reference unit and the reconstructed image information of the coding unit according to preset parameters to obtain a target reconstructed image of the coding unit.

[0052] Exemplarily, the inter-frame prediction module, the residual coding module, the entropy coding module, the preset parameter determination module and the fusion module may be implemented by a neural network or by an algorithm, and the present application does not impose any limitation on this.

[0053] Exemplarily, the AI ​​video coding framework may further include a residual decoding network, which may input the first prediction information and the original image of the coding unit (or the feature information of the original image of the coding unit) into the residual coding network to obtain residual information; and input the result obtained by entropy decoding the encoding result of the residual information into the residual decoding network to obtain decoded residual information.

[0054] Exemplarily, the first prediction information and the original image of the coding unit can be directly input into the residual coding network to obtain the residual information of the coding unit; or the features of the original image of the coding unit can be extracted first to obtain the feature information of the original image of the coding unit, and then the first prediction information and the feature information of the original image of the coding unit can be input into the residual coding network to obtain the residual information of the coding unit; this application is not limited to this.

[0055] According to the first aspect, or any implementation of the first aspect above, the inter-frame prediction module includes a motion estimation network, an optical flow processing module and a prediction module, wherein: the motion estimation network is used to perform motion estimation based on the original image of the coding unit and the target reconstructed image of the reference unit to obtain the first optical flow of the coding unit; the first optical flow is output to the optical flow processing module; the optical flow processing module is used to perform processing based on the first optical flow to determine the second optical flow; the second optical flow is output to the prediction module; the prediction module is used to perform prediction based on the reconstructed image information of the reference unit and the second optical flow to obtain the first prediction information.

[0056] In a second aspect, an embodiment of the present application provides a decoding method, which includes: first, receiving a code stream; then, obtaining decoded residual information of a coding unit in a current frame according to the code stream, and obtaining preset parameters of the coding unit according to the code stream; performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit; then, reconstructing according to the decoded residual information and the first prediction information to determine reconstructed image information of the coding unit; wherein the reconstructed image information of the coding unit includes an initial reconstructed image of the coding unit or feature information of the initial reconstructed image of the coding unit; thereafter, fusing the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to preset parameters to obtain a target reconstructed image of the coding unit; wherein the reconstructed image information of the reference unit includes a target reconstructed image of the reference unit or feature information of the target reconstructed image of the reference unit, and the reference unit is a unit in a reference frame corresponding to the current frame.

[0057] According to the second aspect, the reconstructed image information of the coding unit and the reconstructed image information of the reference unit are fused according to preset parameters to obtain a target reconstructed image of the coding unit, including: determining the weight of the reconstructed image information of the coding unit and the weight of the reconstructed image information of the reference unit according to the preset parameters; performing weighted calculation on the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to the weight of the reconstructed image information of the coding unit and the weight of the reconstructed image information of the reference unit to obtain the target reconstructed image of the coding unit.

[0058] According to the second aspect, or any implementation method of the second aspect above, the reconstructed image information of the coding unit and the reconstructed image information of the reference unit are fused according to preset parameters to obtain a target reconstructed image of the coding unit, including: inputting the preset parameters, the reconstructed image information of the coding unit and the reconstructed image information of the reference unit into a fusion model to obtain the target reconstructed image of the coding unit.

[0059] According to the second aspect, or any implementation of the second aspect above, the bitstream includes a preset parameter bitstream and an optical stream bitstream, and the preset parameters of the coding unit in the current frame are obtained according to the bitstream, including: performing entropy estimation based on at least one of the decoded residual information or the second optical stream of the coding unit to determine the probability distribution of the preset parameters; wherein the second optical stream is obtained based on the optical stream bitstream; according to the probability distribution of the preset parameters, entropy decoding the preset parameters from the preset parameter bitstream.

[0060] The preset parameter code stream may refer to a code stream obtained by encoding the preset parameters. The optical flow code stream may refer to a code stream obtained by encoding the feature information of the first optical flow.

[0061] According to the second aspect, or any implementation method of the above second aspect, the bitstream includes a residual bitstream, and obtaining the decoded residual information of the coding unit according to the bitstream includes: performing adaptive bitrate decoding on the residual bitstream according to preset parameters; and determining the decoded residual information of the coding unit according to information obtained by adaptive bitrate decoding the residual bitstream.

[0062] The residual bitstream may refer to a bitstream obtained by encoding the residual information.

[0063] According to the second aspect, or any implementation of the second aspect above, the code stream includes an optical flow code stream, and the method further includes: performing adaptive code rate decoding on the optical flow code stream according to preset parameters; and determining the second optical flow of the encoding unit according to information obtained by adaptive code rate decoding the optical flow code stream.

[0064] According to the second aspect, or any implementation of the second aspect above, the method is applied to an AI video decoding framework, the AI ​​video decoding framework includes: a decoding module, an inter-frame prediction module, a reconstruction module and a fusion module, wherein:

[0065] A decoding module, used to obtain preset parameters according to the bit stream, and obtain decoded residual information according to the bit stream; output the preset parameters to the fusion module, and output the decoded residual information to the reconstruction module;

[0066] An inter-frame prediction module, used to perform inter-frame prediction on the coding unit to obtain first prediction information; and output the first prediction information to the reconstruction module;

[0067] A reconstruction module, configured to reconstruct according to the decoded residual information and the first prediction information, determine the reconstructed image information of the coding unit; and output the reconstructed image information of the coding unit to the fusion module;

[0068] The fusion module is used to fuse the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to preset parameters to obtain a target reconstructed image of the coding unit.

[0069] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0070] In the third aspect, an embodiment of the present application also provides a coding method, which includes: first, determining preset parameters of a coding unit in a current frame; then, performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit; subsequently, fusing the first prediction information and the reconstructed image information of the reference unit according to the preset parameters to obtain second prediction information of the coding unit; wherein the reconstructed image information of the reference unit includes a target reconstructed image of the reference unit or feature information of the target reconstructed image of the reference unit, and the reference unit is a unit in a reference frame corresponding to the current frame; thereafter, determining the residual information of the coding unit according to the second prediction information and the original image of the coding unit; and then encoding the residual information, as well as the encoding preset parameters.

[0071] After obtaining the first prediction information through inter-frame prediction, the present application fuses the reconstructed image information of the reference unit and the first prediction information according to preset parameters to generate the final prediction information (i.e., the second prediction information); that is, according to the fusion rule (or fusion strategy or fusion method) corresponding to the motion complexity of the image of the coding unit relative to the image of the reference unit, the reconstructed image information of the reference unit and the first prediction information are fused to generate the final prediction information. In this way, for coding units with low image motion complexity relative to the reference unit, the accuracy of inter-frame prediction can be improved to a certain extent, and the error of inter-frame prediction can be reduced, so that the error of the coding unit can be reduced to a certain extent, and then the error of the current frame can be reduced to a certain extent, and the compression performance of the current frame can be improved; thereby, the cumulative error of other subsequent frames can be reduced to a certain extent, the compression performance of other subsequent frames can be improved, and the timing stability can be guaranteed. For coding units with high image motion complexity relative to the reference unit, the accuracy of inter-frame prediction can be guaranteed, and then the quality of the target reconstructed image of the coding unit can be guaranteed, and then the quality of the target reconstructed image of the current frame can be guaranteed.

[0072] According to the third aspect, or any implementation of the third aspect above, encoding the residual information includes: performing adaptive rate encoding on the residual information according to preset parameters.

[0073] According to the third aspect, or any implementation method of the third aspect above, inter-frame prediction is performed on the coding unit to obtain first prediction information of the coding unit, including: determining the first optical flow of the coding unit based on the original image of the coding unit and the target reconstructed image of the reference unit; determining the first prediction information based on the first optical flow and the reconstructed image information of the reference unit; the method also includes: performing adaptive bit rate encoding on the first optical flow according to preset parameters.

[0074] According to the third aspect, or any implementation of the third aspect above, the method also includes: fusing the reconstructed image information of the reference unit and the reconstructed image information of the coding unit according to preset parameters to obtain a target reconstructed image of the coding unit; wherein the reconstructed image information of the coding unit includes an initial reconstructed image of the coding unit or feature information of the initial reconstructed image of the coding unit, and the initial reconstructed image of the coding unit and the feature information of the initial reconstructed image of the coding unit are reconstructed based on the second prediction information and the residual information.

[0075] According to the third aspect, or any implementation of the third aspect above, the method is applied to an AI video coding framework, the AI ​​video coding framework includes: an inter-frame prediction module, a residual coding module, an entropy coding module, a preset parameter determination module and a fusion module, wherein:

[0076] A preset parameter determination module is used to determine the preset parameters; output the preset parameters to the fusion module and the entropy coding module;

[0077] An inter-frame prediction module, used to perform inter-frame prediction on the coding unit to obtain first prediction information; and output the first prediction information to the fusion module;

[0078] A fusion module, used to fuse the first prediction information and the reconstructed image information of the reference unit according to preset parameters to obtain second prediction information; and output the second prediction information to the residual coding module;

[0079] A residual coding module, used to determine residual information according to the second prediction information and the original image of the coding unit; and output the residual information to the entropy coding module;

[0080] The entropy coding module is used to encode residual information and preset parameters.

[0081] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0082] In a fourth aspect, the present application proposes a decoding method, which includes: receiving a code stream; obtaining decoded residual information of a coding unit in a current frame according to the code stream, and obtaining preset parameters of the coding unit according to the code stream; performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit; fusing the first prediction information and the reconstructed image information of the reference unit according to the preset parameters to obtain second prediction information of the coding unit; wherein the reconstructed image information of the reference unit includes a target reconstructed image of the reference unit or feature information of the target reconstructed image of the reference unit, and the reference unit is a unit in a reference frame corresponding to the current frame; reconstructing according to the decoded residual information and the second prediction information to determine the reconstructed image information of the coding unit; wherein the reconstructed image information of the coding unit includes an initial reconstructed image of the coding unit or feature information of the initial reconstructed image of the coding unit, and the feature information of the initial reconstructed image of the coding unit is used to obtain the initial reconstructed image of the coding unit.

[0083] Exemplarily, the initial reconstructed image of the coding unit may be obtained by performing feature transformation (or feature recovery) on feature information of the initial reconstructed image of the coding unit.

[0084] According to a fourth aspect, the method further comprises:

[0085] The reconstructed image information of the coding unit and the reconstructed image information of the reference unit are fused according to preset parameters to obtain a target reconstructed image of the coding unit.

[0086] According to the fourth aspect, or any implementation of the fourth aspect, the bitstream includes a preset parameter bitstream and an optical stream bitstream, and obtaining the preset parameters of the encoding unit according to the bitstream includes:

[0087] Perform entropy estimation based on at least one of the decoded residual information and the second optical flow of the coding unit to determine the probability distribution of the preset parameters; wherein the second optical flow is obtained based on the optical flow code stream;

[0088] The preset parameters are entropy decoded from the preset parameter bitstream according to the probability distribution of the preset parameters.

[0089] According to the fourth aspect, or any implementation of the fourth aspect above, the method is applied to an AI video decoding framework, and the AI ​​video decoding framework includes: a decoding module, an inter-frame prediction module, a reconstruction module, and a fusion module, wherein:

[0090] A decoding module, used to obtain decoded residual information according to the bit stream, and obtain preset parameters according to the bit stream; output the decoded residual information to the reconstruction module, and output the preset parameters to the fusion module;

[0091] An inter-frame prediction module, used to perform inter-frame prediction on the coding unit to obtain first prediction information; and output the first prediction information to the fusion module;

[0092] A fusion module, used to fuse the first prediction information and the reconstructed image information of the reference unit according to preset parameters to obtain second prediction information; and output the second prediction information to the reconstruction module;

[0093] The reconstruction module is used to reconstruct according to the decoded residual information and the second prediction information to determine the reconstructed image information of the coding unit.

[0094] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0095] In a fifth aspect, an embodiment of the present application provides a coding device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, and when the program instructions are executed by the processor, the coding device executes the coding method in the first aspect or any possible implementation of the first aspect; or, executes the coding method in the third aspect or any possible implementation of the third aspect.

[0096] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively; or correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or can refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, which will not be repeated here.

[0097] In a sixth aspect, an embodiment of the present application provides a decoding device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, and when the program instructions are executed by the processor, the decoding device executes the decoding method in the second aspect or any possible implementation of the second aspect; or, executes the decoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0098] The sixth aspect and any implementation of the sixth aspect correspond to the second aspect and any implementation of the second aspect, respectively; or correspond to the fourth aspect and any implementation of the fourth aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or can refer to the technical effects corresponding to the above-mentioned fourth aspect and any implementation of the fourth aspect, which will not be repeated here.

[0099] In the seventh aspect, an embodiment of the present application provides a decoding device, comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the first aspect or any implementation of the first aspect, or the method in the third aspect or any implementation of the third aspect is executed.

[0100] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively; or correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, or can refer to the above-mentioned third aspect and any implementation of the third aspect, which will not be repeated here.

[0101] In an eighth aspect, an embodiment of the present application provides a decoding device, comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the second aspect or any implementation of the second aspect, or the method in the fourth aspect or any implementation of the fourth aspect is executed.

[0102] The eighth aspect and any implementation of the eighth aspect correspond to the second aspect and any implementation of the second aspect, respectively; or correspond to the fourth aspect and any implementation of the fourth aspect, respectively. The technical effects corresponding to the eighth aspect and any implementation of the eighth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or can refer to the above-mentioned fourth aspect and any implementation of the fourth aspect, which will not be repeated here.

[0103] In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect; or, executes the encoding method in the third aspect or any possible implementation of the third aspect.

[0104] The ninth aspect and any implementation of the ninth aspect correspond to the first aspect and any implementation of the first aspect, respectively; or correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the ninth aspect and any implementation of the ninth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or can refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, which will not be repeated here.

[0105] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the encoding method in the second aspect or any possible implementation of the second aspect; or, executes the decoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0106] The tenth aspect and any implementation of the tenth aspect correspond to the second aspect and any implementation of the second aspect, respectively; or correspond to the fourth aspect and any implementation of the fourth aspect, respectively. The technical effects corresponding to the tenth aspect and any implementation of the tenth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or can refer to the above-mentioned fourth aspect and any implementation of the fourth aspect, which will not be repeated here.

[0107] In the eleventh aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect, or executes the encoding method in the third aspect or any possible implementation of the third aspect.

[0108] The eleventh aspect and any implementation of the eleventh aspect correspond to the first aspect and any implementation of the first aspect, respectively; or correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the eleventh aspect and any implementation of the eleventh aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, or can refer to the technical effects corresponding to the above-mentioned third aspect and any implementation of the third aspect, which will not be repeated here.

[0109] In the twelfth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the decoding method in the second aspect or any possible implementation of the second aspect, or executes the decoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0110] The twelfth aspect and any implementation of the twelfth aspect correspond to the second aspect and any implementation of the second aspect, respectively; or correspond to the fourth aspect and any implementation of the fourth aspect, respectively. The technical effects corresponding to the twelfth aspect and any implementation of the twelfth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or can refer to the technical effects corresponding to the above-mentioned fourth aspect and any implementation of the fourth aspect, which will not be repeated here.

[0111] In a thirteenth aspect, an embodiment of the present application provides a device for storing a code stream, the device comprising: a receiver and at least one storage medium, the receiver being used to receive the code stream; the at least one storage medium being used to store the code stream; the code stream is generated according to the first aspect and any one of the implementation methods of the first aspect, or, generated according to the third aspect and any one of the implementation methods of the third aspect.

[0112] The thirteenth aspect and any implementation of the thirteenth aspect respectively correspond to the first aspect and any implementation of the first aspect; or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the thirteenth aspect and any implementation of the thirteenth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, or can refer to the technical effects corresponding to the above-mentioned third aspect and any implementation of the third aspect, which will not be repeated here.

[0113] In a fourteenth aspect, an embodiment of the present application provides a device for transmitting a code stream, the device comprising: a transmitter and at least one storage medium, the at least one storage medium is used to store the code stream, the code stream is generated according to the first aspect and any one of the implementation methods of the first aspect, or, is generated according to the third aspect and any one of the implementation methods of the third aspect; the transmitter is used to obtain the code stream from the storage medium and send the code stream to the terminal side device through the transmission medium.

[0114] The fourteenth aspect and any implementation of the fourteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively; or correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the fourteenth aspect and any implementation of the fourteenth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, or can refer to the technical effects corresponding to the above-mentioned third aspect and any implementation of the third aspect, which will not be repeated here.

[0115] In a fifteenth aspect, an embodiment of the present application provides a system for distributing code streams, the system comprising: at least one storage medium for storing at least one code stream, the at least one code stream being generated according to the first aspect and any one of the implementation methods of the first aspect, or generated according to the third aspect and any one of the implementation methods of the third aspect, a streaming media device for obtaining a target code stream from the at least one storage medium and sending the target code stream to a terminal side device, wherein the streaming media device comprises a content server or a content distribution server.

[0116] The fifteenth aspect and any implementation of the fifteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively; or correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the fifteenth aspect and any implementation of the fifteenth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, or can refer to the technical effects corresponding to the above-mentioned third aspect and any implementation of the third aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0117] Figure 1A is a schematic diagram of an application framework shown as an example;

[0118] Figure 1B A schematic diagram of a storage framework is shown as an example;

[0119] Figure 1C is a schematic diagram of an exemplary transmission framework;

[0120] Figure 2A Schematic diagram of the structure of an AI video encoding framework shown as an example;

[0121] Figure 2B Schematic diagram of the structure of an AI video decoding framework shown as an example;

[0122] Figure 3A Schematic diagram of an AI video encoding process shown as an example;

[0123] Figure 3B is a schematic diagram of an exemplary preset parameter encoding process;

[0124] Figure 3C is a schematic diagram of an exemplary encoding process;

[0125] Figure 3D is a schematic diagram of an exemplary encoding process;

[0126] Figure 4A is a schematic diagram of an exemplary decoding process;

[0127] Figure 4B is a schematic diagram of an exemplary preset parameter encoding process;

[0128] Figure 4C is a schematic diagram showing the reconstruction quality of a test video for example;

[0129] Figure 5A is a schematic diagram of an exemplary encoding process;

[0130] Figure 5B is a schematic diagram of an exemplary encoding process;

[0131] Figure 6 is a schematic diagram of an exemplary decoding process;

[0132] Fig. 7A is a schematic diagram of an exemplary encoding process;

[0133] Figure 7B is a schematic diagram of an exemplary encoding process;

[0134] Figure 8 is a schematic diagram of an exemplary decoding process;

[0135] Fig. 9 is a schematic diagram showing the reconstruction quality of a test video for example;

[0136] Fig. 10A is a schematic diagram of an exemplary encoding process;

[0137] Fig. 10B is a schematic diagram of an exemplary encoding process;

[0138] Fig.11 is a schematic diagram of an exemplary decoding process;

[0139] Fig.12 Schematic diagram of the structure of the device shown as an example. DETAILED DESCRIPTION

[0140] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0141] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0142] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0143] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0144] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0145] Figure 1A The following is a schematic diagram of an application framework shown as an example.

[0146] Reference Figure 1A For example, a camera can capture video and obtain video data; then, an AI video encoding framework can be used to perform AI video encoding on the video data to obtain a bitstream. Afterwards, the bitstream can be stored locally or transmitted to a remote device.

[0147] Continue to refer to Figure 1A In one possible method, after the bit stream is stored locally, when the video needs to be played or edited, the AI ​​video decoding framework can be used to perform AI video decoding on the bit stream to obtain reconstructed video data; then, the reconstructed video data can be played or edited.

[0148] Continue to refer to Figure 1A In one possible way, after the remote device receives the bit stream, when it needs to play or edit the video, it can use the AI ​​video decoding framework to perform AI video decoding on the bit stream to obtain reconstructed video data; then, the reconstructed video data can be played or edited.

[0149] Figure 1B A schematic diagram of a storage framework is shown as an example.

[0150] Reference Figure 1B , exemplarily, the AI ​​video encoding framework may include an AI encoding module (also referred to as an AI encoding unit) and an entropy encoding module (also referred to as an entropy encoding module), and the AI ​​video decoding framework may include an AI decoding module (also referred to as an AI decoding unit) and an entropy decoding module (also referred to as an entropy decoding module); it should be understood that, Figure 1B This is only an example of the present application. The AI ​​video encoding framework and the AI ​​video decoding framework may include Figure 1B More modules are shown, but this application does not limit this.

[0151] Exemplarily, the AI ​​encoding module and the AI ​​decoding module can be set in an embedded neural network processor (Neural network Processing Unit, NPU) or a graphics processor (Graphics Processing Unit, GPU). Exemplarily, the entropy encoding module and the entropy decoding module can be set in a central processing unit (Central Processing Unit, CPU). Exemplarily, the file saving module and the file loading module can be set in the CPU.

[0152] Reference Figure 1B , exemplarily, after the Camera captures the video data, it can be input into the AI ​​encoding module. Then, the AI ​​encoding module can perform the following processing for each frame of the video data: perform prediction (including intra-frame prediction and inter-frame prediction, this application takes inter-frame prediction as an example), determine the prediction information; and determine the residual information based on the prediction information and the original image, and determine the probability distribution of the residual information; then, input the residual information and the probability distribution of the residual information into the entropy encoding module. Afterwards, the entropy encoding module can entropy encode the residual information according to the probability distribution of the residual information to obtain a code stream; and input the code stream into the file saving module for saving to obtain a file.

[0153] Continue to refer to Figure 1B For example, when a video needs to be played or edited, the file loading module can load the file, and then input the bitstream in the file into the entropy decoding module. Then, the entropy decoding module can obtain the probability distribution of the residual information from the AI ​​decoding module, and according to the probability distribution of the residual information, entropy decode the residual information from the bitstream and output the residual information to the AI ​​decoding module. After that, the AI ​​decoding module can perform inter-frame prediction to obtain prediction information; then, reconstruct according to the prediction information and the residual information to obtain a reconstructed image; and then, reconstructed video data can be obtained.

[0154] Figure 1C FIG. 1 is a schematic diagram of an exemplary transmission framework.

[0155] Reference Figure 1C , exemplarily, the encoding process of the encoding end may be as follows:

[0156] Exemplarily, after the Camera captures the video data, it can be input into the AI ​​encoding module. Then, the AI ​​encoding module can perform the following processing for each frame of the video data: perform prediction (including intra-frame prediction and inter-frame prediction, this application takes inter-frame prediction as an example), determine the prediction information; and determine the residual information based on the prediction information and the original image, and determine the probability distribution of the residual information; then, input the residual information and the probability distribution of the residual information into the entropy encoding module. Afterwards, the entropy encoding module can entropy encode the residual information according to the probability distribution of the residual information to obtain a bitstream; and send the bitstream to a server in the cloud.

[0157] Afterwards, the server can send the code stream to the decoding end. The server can be a single server or a server cluster, which is not limited in this application.

[0158] Continue to refer to Figure 1C , for example, after the decoding end receives the code stream sent by the server, the decoding process can be as follows:

[0159] Exemplarily, the file loading module can load the file and then send the code stream in the file to the entropy decoding module. Then, the AI ​​decoding network can perform inter-frame prediction to obtain prediction information; after that, the entropy decoding module can obtain the probability distribution of residual information from the AI ​​decoding module, and according to the probability distribution of residual information, entropy decode the residual information from the code stream and output the residual information to the AI ​​decoding module. Subsequently, reconstruction is performed based on the prediction information and residual information to obtain a reconstructed image; and then, reconstructed video data can be obtained.

[0160] It should be understood that the encoding end can also directly send the code stream to the decoding end without forwarding through the server, and this application does not impose any restrictions on this.

[0161] Figure 2A FIG. 1 is a schematic diagram of the structure of an AI video encoding framework. Figure 2A In the embodiment, the inter-frame coding process is shown.

[0162] Reference Figure 2A , exemplarily, the AI ​​encoding module may include a motion estimation module, an optical flow encoding module, a first entropy estimation module, an optical flow decoding module, a prediction module, a feature extraction module, a residual encoding module, a second entropy estimation module, a residual decoding module and a reconstruction module; it should be understood that the AI ​​encoding module may also include Figure 2A The present application is not limited to more or fewer modules shown.

[0163] Reference Figure 2A , exemplarily, the entropy coding module may include a first entropy coding module and a second entropy coding module.

[0164] Reference Figure 2A , exemplarily, the AI ​​video encoding framework may further include an entropy decoding module, and the entropy decoding module may include a first entropy decoding module and a second entropy decoding module.

[0165] It should be noted that Figure 2A The feature extraction module in the dotted box is an optional module.

[0166] It should be noted that the various modules included in the AI ​​video coding framework can be implemented only by a neural network, or by a combination of an algorithm and a neural network; or, some modules included in the AI ​​video coding framework can be implemented only by an algorithm, and this application does not impose any restrictions on this.

[0167] Exemplarily, the original image of the current frame (that is, the image of the current frame in the image sequence of the original video (also referred to as the video to be encoded)) can be divided into multiple coding units (it should be noted that the coding unit can refer to the smallest coding unit, and a coding unit can include at least one pixel); then each coding unit of the current frame is encoded.

[0168] Reference Figure 2A , taking inter-frame coding of a coding unit in an image frame of video data as an example for explanation; wherein the AI ​​video coding process can be as follows:

[0169] Exemplarily, the original image of the coding unit and the reconstructed image of the reference unit in the reference frame corresponding to the current frame can be input into the motion estimation module. Then, the motion estimation module can perform motion estimation based on the original image of the coding unit and the reconstructed image of the reference unit, determine the optical flow of the coding unit (hereinafter referred to as the first optical flow); and output the first optical flow of the coding unit to the optical flow encoding module. The optical flow encoding module (also referred to as the optical flow feature extraction module) can encode the first optical flow of the coding unit (it should be noted that the encoding performed by the optical flow encoding module is essentially feature extraction), obtain the feature information of the first optical flow of the coding unit (also referred to as the compression feature of the first optical flow of the coding unit) and output it to the first entropy estimation module.

[0170] Exemplarily, the first entropy estimation module can perform probability estimation on the feature information of the first optical flow of the coding unit, and determine the probability distribution of the feature information of the first optical flow of the coding unit (hereinafter referred to as the optical flow probability distribution of the coding unit); then, the feature information of the first optical flow of the coding unit and the optical flow probability distribution of the coding unit are output to the first entropy coding module.

[0171] Exemplarily, the first entropy coding module may perform entropy coding on the feature information of the first optical flow of the coding unit according to the optical flow probability distribution of the coding unit to obtain an optical flow code stream.

[0172] Exemplarily, on the one hand, the optical flow code stream can be sent / saved (for example, the optical flow code stream is sent to a server or a decoding end). On the other hand, the optical flow code stream can be output to the first entropy decoding module; in addition, the first entropy estimation module can also output the optical flow probability distribution of the coding unit to the first entropy decoding module.

[0173] Exemplarily, the first entropy decoding module can entropy decode the optical flow code stream according to the optical flow probability distribution of the coding unit to obtain the characteristic information of the first optical flow after decoding of the coding unit (which may also be called the characteristic information of the second optical flow of the coding unit); and output the characteristic information of the second optical flow of the coding unit to the optical flow decoding module.

[0174] Exemplarily, the optical flow decoding module can decode the feature information of the second optical flow of the coding unit (it should be noted that the decoding performed by the optical flow decoding module is essentially feature recovery), obtain the second optical flow of the coding unit and output the second optical flow of the coding unit to the prediction module. The second optical flow can be generally described as being determined by the optical flow processing module based on the first optical flow; the optical flow processing module may include an optical flow encoding module and an optical flow decoding module.

[0175] In one possible manner, when the AI ​​encoding module includes a feature extraction module, the reconstructed image of the reference unit can be input into the feature extraction module, and the feature extraction module extracts the features of the reconstructed image of the reference unit to obtain the feature information of the reconstructed image of the reference unit; and the feature information of the reconstructed image of the reference unit is output to the prediction module. The prediction module can make a prediction based on the second optical flow of the encoding unit and the feature information of the reconstructed image of the reference unit to obtain the prediction information of the encoding unit (hereinafter referred to as the first prediction information); and the first prediction information of the encoding unit is output to the residual encoding module. Correspondingly, the original image of the encoding unit can be input into the feature extraction module, and the feature extraction module extracts the features of the original image of the encoding unit to obtain the feature information of the original image of the encoding unit, and the feature information of the original image of the encoding unit is output to the residual encoding module. The residual encoding module outputs the residual information of the encoding unit (hereinafter referred to as the first residual information of the encoding unit) to the second entropy estimation module based on the feature information of the original image of the encoding unit and the first prediction information of the encoding unit. Among them, the first residual information of the coding unit can be the first residual of the coding unit (wherein, the first residual can refer to the residual between the feature information of the original image of the coding unit and the prediction information of the coding unit, or it can be the feature information of the first residual of the coding unit (wherein, the feature information of the first residual can also be called the compression feature of the first residual, which can be obtained by encoding the first residual of the residual coding module (it should be noted that the encoding performed by the residual coding module is essentially feature extraction)), and the present application does not limit this. The present application takes the first residual information of the coding unit as the feature information of the first residual of the coding unit as an example for explanation.

[0176] In one possible manner, when the AI ​​encoding module does not include a feature extraction module, the reconstructed image of the reference unit can be directly input into the prediction module, and the prediction module performs prediction based on the second optical flow of the encoding unit and the reconstructed image of the reference unit to obtain the first prediction information of the encoding unit; and the first prediction information of the encoding unit is output to the residual encoding module. Correspondingly, the original image of the encoding unit can be directly input into the residual encoding module; the residual encoding module can output the first residual information of the encoding unit to the second entropy estimation module based on the original image of the encoding unit and the first prediction information of the encoding unit.

[0177] Exemplarily, the second entropy estimation module can perform probability estimation on the first residual information of the coding unit to determine the probability distribution of the first residual information of the coding unit (hereinafter referred to as the residual probability distribution of the coding unit); then, the first residual information of the coding unit and the residual probability distribution of the coding unit are output to the second entropy coding module.

[0178] Exemplarily, the second entropy coding module may perform entropy coding on the first residual information of the coding unit according to the residual probability distribution of the coding unit to obtain a residual code stream.

[0179] Exemplarily, on the one hand, the residual code stream can be sent / saved (for example, the residual code stream is sent to a server or a decoding end). On the other hand, the residual code stream can be output to the second entropy decoding module; in addition, the second entropy estimation module can also output the residual probability distribution of the coding unit to the second entropy decoding module.

[0180] Exemplarily, the second entropy decoding module may entropy decode the residual code stream according to the residual probability distribution of the coding unit to obtain the first residual information after entropy decoding of the coding unit. When the first residual information of the coding unit is the characteristic information of the first residual of the coding unit, the first residual information after entropy decoding of the coding unit may be output to the residual decoding module.

[0181] Exemplarily, the residual decoding module can decode the first residual information after entropy decoding of the coding unit (it should be noted that the decoding performed by the residual decoding module is essentially feature recovery), obtain the second residual information of the coding unit (that is, the decoded residual information) and output the second residual information of the coding unit to the reconstruction module. It should be noted that when the first residual information of the coding unit is the first residual of the coding unit, the AI ​​video coding framework may not include a residual decoding module; at this time, the second residual information of the coding unit is the decoded first residual information obtained by entropy decoding of the coding unit. In addition, it can also be said that the second residual information of the coding unit is obtained by decoding the encoding result (that is, the residual code stream) of the first residual information of the coding unit; when the first residual information of the coding unit is the feature information of the first residual of the coding unit, the decoding in this step includes entropy decoding and feature recovery; when the first residual information of the coding unit is the first residual of the coding unit, the decoding in this step is entropy decoding.

[0182] Exemplarily, the prediction module may also output the first prediction information of the coding unit to the reconstruction module. When the AI ​​coding module includes a feature extraction module, the reconstruction module may first reconstruct according to the second residual information of the coding unit and the first prediction information of the coding unit to obtain the feature information of the reconstructed image of the coding unit; then, the reconstruction module may perform feature transformation on the feature information of the reconstructed image of the coding unit to obtain the reconstructed image of the coding unit. When the AI ​​coding module does not include a feature extraction module, the reconstruction module may reconstruct according to the second residual information of the coding unit and the first prediction information of the coding unit to obtain the reconstructed image of the coding unit.

[0183] Optionally, the present application may also encode information for determining the optical flow probability distribution of the coding unit, and send it to the decoding end; so that the decoding end determines the optical flow probability distribution to obtain the second optical flow of the coding unit; and encode information for determining the residual probability distribution of the coding unit, and send it to the decoding end; so that the decoding end determines the residual probability distribution to obtain the second residual information of the coding unit.

[0184] Figure 2B The figure is a schematic diagram of the structure of an AI video decoding framework shown as an example.

[0185] Reference Figure 2B , exemplarily, the AI ​​decoding module may include: a first entropy estimation module, a second entropy estimation module, an optical flow decoding module, a prediction module, a feature extraction module, a residual decoding module and a reconstruction module; it should be understood that the AI ​​encoding module may also include Figure 2B The present application is not limited to more or fewer modules shown.

[0186] Reference Figure 2B , exemplarily, the entropy decoding module may include a first entropy decoding module and a second entropy encoding module.

[0187] It should be noted that Figure 2B The feature extraction module in the dotted box is an optional module.

[0188] It should be noted that the various modules included in the AI ​​video decoding framework can be implemented only by a neural network, or by a combination of an algorithm and a neural network; or, some modules included in the AI ​​video decoding framework can be implemented only by an algorithm, and this application does not impose any restrictions on this.

[0189] Continue to refer to Figure 2B , taking inter-frame decoding of a coding unit in an image frame of video data as an example for explanation; wherein the AI ​​video decoding process can be as follows:

[0190] Exemplarily, the decoding end may receive a code stream, which may include an optical flow code stream and a residual code stream. When the code stream received by the decoding end does not include a code stream containing information for determining the optical flow probability distribution of the coding unit, the first entropy estimation module may perform probability estimation based on the reconstructed image of the reference unit or the first prediction information of the reference unit or the feature information of the reconstructed image of the reference unit, etc., determine the optical flow probability distribution of the coding unit and output it to the first entropy decoding module. When the received code stream includes a code stream containing information for determining the optical flow probability distribution of the coding unit, the first entropy decoding module may entropy decode the information for determining the optical flow probability distribution of the coding unit from the code stream and output it to the first entropy estimation module; then, the first entropy estimation module may perform probability estimation based on the information for determining the optical flow probability distribution of the coding unit, determine the optical flow probability distribution of the coding unit frame and output it to the first entropy decoding module.

[0191] Exemplarily, the first entropy decoding module may perform entropy decoding on the optical flow code stream according to the optical flow probability distribution of the coding unit, obtain feature information of the second optical flow of the coding unit, and output the feature information to the optical flow decoding module.

[0192] Exemplarily, the optical flow decoding module may perform feature recovery on feature information of the second optical flow of the coding unit, obtain the second optical flow of the coding unit, and output the second optical flow of the coding unit to the prediction module.

[0193] Exemplarily, when the received bitstream does not include a bitstream containing information for determining the residual probability distribution of the coding unit, the second entropy estimation module may perform probability estimation based on the reconstructed image of the reference unit or the first prediction information of the reference unit or the feature information of the reconstructed image of the reference unit, etc., determine the residual probability distribution of the coding unit and output it to the second entropy decoding module. When the received bitstream includes a bitstream containing information for determining the residual probability distribution of the coding unit, the second entropy decoding module may entropy decode the information for determining the residual probability distribution of the coding unit from the bitstream; then, the second entropy estimation module may perform probability estimation based on the information for determining the residual probability distribution of the coding unit, determine the residual probability distribution of the coding unit and output it to the second entropy decoding module.

[0194] Exemplarily, the second entropy decoding module can entropy decode the code stream of information used to determine the residual probability distribution, obtain information used to determine the residual probability distribution of the coding unit, and output the information used to determine the residual probability distribution of the coding unit to the second entropy estimation module; then, the second entropy estimation module performs probability estimation based on the information for determining the residual probability distribution of the coding unit, determines the residual probability distribution of the coding unit and outputs it to the second entropy decoding module.

[0195] Exemplarily, the second entropy decoding module may perform entropy decoding on the residual code stream according to the residual probability distribution of the coding unit, obtain the first residual information after entropy decoding of the coding unit, and output it to the residual decoding module.

[0196] Exemplarily, the residual decoding module may perform feature recovery on the first residual information of the coding unit after entropy decoding, obtain the second residual information of the coding unit, and output the second residual information of the coding unit to the reconstruction module.

[0197] In one possible manner, when the AI ​​decoding module includes a feature extraction module, the reconstructed image of the reference unit can be input into the feature extraction module, and the feature extraction module extracts the features of the reconstructed image of the reference unit to obtain the feature information of the reconstructed image of the reference unit; and the feature information of the reconstructed image of the reference unit is output to the prediction module. The prediction module can make a prediction based on the second optical flow of the coding unit and the feature information of the reconstructed image of the reference unit to obtain the first prediction information of the coding unit; and the first prediction information of the coding unit is output to the reconstruction module. Then, the reconstruction module can first reconstruct based on the second residual information of the coding unit and the first prediction information of the coding unit to obtain the feature information of the reconstructed image of the coding unit; thereafter, the reconstruction module can perform feature transformation (such as feature recovery) on the feature information of the reconstructed image of the coding unit to obtain the reconstructed image of the coding unit.

[0198] In one possible manner, when the AI ​​decoding module does not include a feature extraction module, the reconstructed image of the reference unit can be directly input into the prediction module, and the prediction module predicts based on the second optical flow of the coding unit and the reconstructed image of the reference unit to obtain the first prediction information of the coding unit; and outputs the first prediction information of the coding unit to the reconstruction module. Then, the reconstruction module reconstructs based on the second residual information of the coding unit and the first prediction information of the coding unit to obtain the reconstructed image of the coding unit.

[0199] It should be understood that the present invention is Figure 2A The AI ​​video encoding framework shown in FIG. 1 includes the names of the various modules, and Figure 2B The names of the modules included in the AI ​​video decoding framework shown in the figure are not limited; the modules included in the AI ​​video encoding framework can also be called units; and the modules included in the AI ​​video decoding framework can also be called units.

[0200] It should be noted that in one possible manner, the residual code stream and the optical stream code stream are two code streams. In one possible manner, the residual code stream and the optical stream code stream may be two parts of the same code stream, that is, the same code stream carries the residual information and the first optical stream.

[0201] The following is Figure 2ABased on this, the AI ​​video encoding and decoding process is explained.

[0202] In the following Figure 3A and Figure 4A In the embodiment of Figure 2A On the basis of, the reconstructed image of the reconstructed coding unit is post-processed to reduce the accumulated error.

[0203] Figure 3A FIG. 1 is a schematic diagram of an exemplary AI video encoding process. Figure 3A In the above, a coding unit for encoding a frame of an image in an original video is taken as an example for explanation.

[0204] S301, performing inter-frame prediction on a coding unit in a current frame to obtain first prediction information of the coding unit.

[0205] Exemplarily, S301 may be performed by an inter-frame prediction module in an AI video coding framework, wherein the inter-frame prediction module may include Figure 3A The specific process of inter-frame prediction for the coding unit can be referred to Figure 2A The description in the embodiments will not be repeated here.

[0206] Furthermore, the motion estimation module may be a motion estimation network.

[0207] S302: Determine residual information of the coding unit according to the first prediction information and the original image of the coding unit.

[0208] Exemplarily, S302 may be performed by a residual coding module in an AI video coding framework, and specific reference may be made to the above Figure 2A The description in the embodiment will not be repeated here. It should be noted that the residual information of the coding unit determined in S302 corresponds to Figure 2A The first residual information of the encoding unit in the embodiment.

[0209] S303: Encode residual information of the coding unit.

[0210] Exemplarily, S303 may be performed by the second entropy coding module to obtain a residual bit stream; Figure 2A The description in the embodiment will not be repeated here. It should be noted that the residual information of the coding unit in S303 corresponds to Figure 2A The first residual information of the encoding unit in the embodiment.

[0211] S304: Determine preset parameters of the encoding unit.

[0212] Exemplarily, the AI ​​video encoding framework may further include a preset parameter determination module for executing S304.

[0213] Exemplarily, the preset parameters of the coding unit can be determined based on the image of the coding unit and the image of the reference unit; wherein the preset parameters are used to describe the motion complexity of the image of the coding unit relative to the image of the reference unit. Exemplarily, the image of the coding unit includes the original image of the coding unit and / or the initial reconstructed image of the coding unit, and the image of the reference unit includes the original image of the reference unit in the reference frame and / or the target reconstructed image of the reference unit. The initial reconstructed image of the coding unit may refer to the initial reconstructed image of the coding unit according to Figure 2A After the encoding process in the embodiment is encoded, the reconstruction module outputs a reconstructed image (that is, a reconstructed image obtained by reconstructing according to the second residual information of the encoding unit and the first prediction information of the encoding unit).

[0214] Exemplarily, motion complexity may refer to difference or similarity; that is, the motion complexity of an image of a coding unit relative to an image of a reference unit may refer to the similarity between the image of the coding unit and the image of the reference unit, or may refer to the difference between the image of the coding unit and the image of the reference unit.

[0215] In one possible manner, the similarity between the image of the coding unit and the image of the reference unit can be determined, and the preset parameters of the coding unit can be determined according to the similarity. Wherein, when the similarity is greater than or equal to the similarity threshold, it means that the similarity between the image of the coding unit and the image of the reference unit is high. At this time, it can be determined that the motion complexity of the image of the coding unit relative to the image of the reference unit is low, and the value of the preset parameter can be set to a first preset value; that is, the first preset value can be used to indicate that the motion complexity of the image of the coding unit relative to the image of the reference unit is low complexity. When the similarity is less than the similarity threshold, it means that the similarity between the image of the coding unit and the image of the reference unit is low. At this time, it can be determined that the motion complexity of the image of the coding unit relative to the image of the reference unit is high, and the value of the preset parameter can be set to a second preset value; that is, the second preset value can be used to indicate that the motion complexity of the image of the coding unit relative to the image of the reference unit is high complexity. Wherein, the similarity threshold can be set according to demand. Assuming that the similarity when the two images are exactly the same is 1, the similarity threshold can be a decimal less than 1 and greater than 0.

[0216] It should be noted that the first preset value and the second preset value are different, the first preset value may be greater than the second preset value, or the first preset value may be less than the second preset value; it may be set specifically as required, and this application does not limit this. In one possible manner, the first preset value and the second preset value may be numbers between 0 and 1 (the values ​​of the first preset value and the second preset value may both be 0 or 1). Optionally, the sum of the first preset value and the second preset value may be 1. For example, the first preset value is 1, and the second preset value is 0; or, the first preset value is 0, and the second preset value is 1. For another example, the first preset value is 0.8, and the second preset value is 0.2; or, the first preset value is 0.2, and the second preset value is 0.8.

[0217] Exemplarily, there are multiple ways to determine similarity, for example, calculating an error and determining similarity based on the error; for another example, using a similarity model to determine similarity calculation, etc., which is not limited in this application. This application takes calculating an error to determine similarity as an example for explanation.

[0218] In one possible manner, the error between the image of the coding unit and the image of the reference unit may be calculated, and the similarity between the image of the coding unit and the image of the reference unit may be determined based on the error between the image of the coding unit and the image of the reference unit. Assuming that the similarity when the two images are exactly the same is 1, the difference between 1 and the error between the image of the coding unit and the image of the reference unit may be used as the similarity.

[0219] For example, the value of the preset parameter can be determined according to the following formula (1):

[0220]

[0221] In formula (1), mask represents a preset parameter, error( ) represents a calculated error, X1 represents an image of a coding unit, X2 represents an image of a reference unit, ε represents an error threshold, and the error threshold may be a difference between 1 and a similarity threshold.

[0222] That is, in formula (1), when the error between the image of the coding unit and the image of the reference unit is less than or equal to the error threshold, it is equivalent to that the similarity between the image of the coding unit and the image of the reference unit is greater than or equal to the similarity threshold; at this time, the value of the preset parameter is set to 1. Else in formula (1) means that the error between the image of the coding unit and the image of the reference unit is greater than the error threshold, which is equivalent to that the similarity between the image of the coding unit and the image of the reference unit is less than the similarity threshold; at this time, the value of the preset parameter is set to 0.

[0223] For example, X1 in formula (1) is the initial reconstructed image of the coding unit, and X2 is the target reconstructed image of the reference unit.

[0224] For example, X1 in formula (1) is the original image of the coding unit, and X2 is the original image of the reference unit.

[0225] For example, X1 in formula (1) is the initial reconstructed image of the coding unit, and X2 is the original image of the reference unit.

[0226] For example, X1 in formula (1) is the original image of the coding unit, and X2 is the target reconstructed image of the reference unit.

[0227] In one possible method, a third optical flow of the coding unit can be obtained; second prediction information of the coding unit can be determined based on the third optical flow of the coding unit and a target reconstructed image of a reference unit; and similarity between the image of the coding unit and the image of the reference unit can be determined based on an error between the second prediction information of the coding unit and the image of the coding unit.

[0228] In a possible implementation, the initial reconstructed image of the coding unit and the target reconstructed image of the reference unit may be input into a motion estimation module, which performs motion estimation and outputs the third optical flow of the coding unit; refer to formula (2):

[0229] flow = FLOW(X 12 ,X 22 ) (2)

[0230] In formula (2), flow represents the third optical flow of the coding unit, FLOW() represents motion estimation, and X 12 represents the initial reconstructed image of the coding unit, X 22 Represents the target reconstructed image of the reference cell.

[0231] In one possible approach, the feature information of the first optical flow of the coding unit can be input into an optical flow decoding model (the optical flow decoding model is pre-trained and Figure 2A The optical flow decoding module is different in the first optical flow decoding module), and the optical flow decoding model decodes the feature information of the first optical flow (that is, feature recovery) and outputs the third optical flow of the coding unit; refer to formula (3):

[0232] flow = Decflow (feature information of the first optical flow of the coding unit) (3)

[0233] In formula (3), flow represents the third optical flow of the coding unit, and Decflow ( ) represents decoding (or feature recovery).

[0234] Next, the second prediction information of the coding unit can be determined according to formula (4):

[0235] F=wrap(X 22,flow,'neareast') (4)

[0236] In formula (4), F represents the second prediction information of the coding unit, X 22 It represents the target reconstructed image of the reference unit, flow represents the third optical flow of the coding unit, nearest represents the nearest neighbor interpolation method, and wrap() represents image transformation.

[0237] Afterwards, the preset parameters of the encoding unit can be determined by referring to formula (5):

[0238]

[0239] In formula (5), mask represents the preset parameters of the coding unit, error( ) represents the calculation error, and X 11 represents the original image of the coding unit, F represents the second prediction information of the coding unit, ε is the error threshold, and the error threshold can be the difference between 1 and the similarity threshold.

[0240] That is, in formula (5), when the error between the second prediction information and the original image of the coding unit is less than or equal to the error threshold, it is equivalent to that the similarity between the second prediction information and the original image of the coding unit is greater than or equal to the similarity threshold; at this time, the value of the preset parameter is set to 1. The else in formula (5) means that the error between the second prediction information and the original image of the coding unit is greater than the error threshold, which is equivalent to that the similarity between the second prediction information and the original image of the coding unit is less than the similarity threshold. At this time, the value of the preset parameter is set to 0.

[0241] It should be understood that other methods may be used to calculate the error, and this application does not limit this.

[0242] S305, encoding preset parameters of the encoding unit.

[0243] It should be noted that the present application does not limit the order of encoding residual information and encoding preset parameters, that is, the present application does not limit the execution order of S303 and S305.

[0244] Exemplarily, both the AI ​​video encoding framework and the AI ​​video encoding framework may also include a fusion module; wherein the fusion module may be an independent module or may be integrated into the reconstruction module, and this application does not impose any restrictions on this.

[0245] Exemplarily, after obtaining the preset parameters of the coding unit, on the one hand, the preset parameters of the coding unit can be input into a fusion module, and the fusion module can fuse the reconstructed image information of the reference unit and the reconstructed image information of the coding unit according to the preset parameters of the coding unit to obtain the target reconstructed image of the coding unit. On the other hand, S305 can be executed to encode the preset parameters of the coding unit to obtain a preset parameter bitstream; the preset parameters of the coding unit can be transmitted to the decoding end and applied to the fusion module of the decoding end, so that the fusion module of the decoding end can also fuse the reconstructed image information of the reference unit and the reconstructed image information of the coding unit according to the preset parameters of the coding unit to obtain the target reconstructed image of the coding unit.

[0246] The reconstructed image information of the reference unit may include the target reconstructed image of the reference unit or feature information of the target reconstructed image of the reference unit, and the reconstructed image information of the coding unit may include the initial reconstructed image of the coding unit or feature information of the initial reconstructed image of the coding unit.

[0247] Exemplarily, the entropy coding module of the AI ​​video coding framework may further include a third entropy coding module and the AI ​​video coding framework may further include a third entropy estimation module, and the third entropy coding module may execute S305.

[0248] In one possible approach, the preset parameters of the coding unit can be directly encoded to obtain a preset parameter bitstream; in this way, the decoding end can decode the preset parameters of the current frame from the preset parameter bitstream. For example, the preset parameters of the coding unit can be first entropy estimated to obtain a probability distribution of the preset parameters; then, based on the probability distribution of the preset parameters, the preset parameters of the coding unit can be entropy encoded to obtain a preset parameter bitstream. Specifically, refer to Figure 3B The third entropy estimation module can determine the probability distribution of the preset parameters of the coding unit based on the optical stream code stream and / or the residual code stream. It should be understood that other methods can also be used to encode the preset parameters of the coding unit, and this application does not limit this.

[0249] In one possible manner, feature extraction may be performed on the preset parameters of the coding unit to obtain feature information of the preset parameters of the coding unit; then, the feature information of the preset parameters of the coding unit is encoded to obtain a preset parameter bitstream. In this way, the decoding end may perform feature recovery based on the information decoded from the preset parameter bitstream to obtain the preset parameters of the current frame. The method of extracting the feature information of the preset parameters of the coding unit is similar to the method of encoding the preset parameters of the coding unit, and will not be described in detail here.

[0250] Exemplarily, one way to fuse the reconstructed image information of the reference unit and the reconstructed image information of the coding unit to obtain the target reconstructed image of the coding unit may be to determine the weight of the reconstructed image information of the reference unit and the weight of the reconstructed image information of the coding unit according to preset parameters of the coding unit; and perform weighted calculation on the reconstructed image information of the reference unit and the reconstructed image information of the coding unit according to the weight of the reconstructed image information of the reference unit and the weight of the reconstructed image information of the coding unit to obtain the target reconstructed image of the coding unit.

[0251] In this case, if the first preset value is greater than the second preset value, when the first preset value and the second preset value are numbers between 0 and 1 (the first preset value and the second preset value can both be 0 or 1), the value of the preset parameter of the coding unit can be used as the weight of the reconstructed image information of the reference unit, and the difference between 1 and the value of the preset parameter can be used as the weight of the reconstructed image information of the coding unit. When the first preset value and the second preset value are both numbers greater than 1, the first preset value and the second preset value can be transformed into numbers between 0 and 1, and then the value of the preset parameter after the transformation of the coding unit can be used as the weight of the reconstructed image information of the reference unit, and the difference between 1 and the value of the preset parameter after the transformation of the coding unit can be used as the weight of the reconstructed image information of the coding unit. At this time, the fusion can be performed with reference to the following formula (6):

[0252] G=mask*Y1+(1-mask)*Y2 (6)

[0253] If the second preset value is greater than the first preset value, the fusion can be performed according to the following formula (7):

[0254] G=mask*Y2+(1-mask)*Y1 (7)

[0255] In formula (6) and formula (7), G represents the target reconstructed image of the coding unit or the characteristic information of the target reconstructed image of the coding unit, Y1 represents the target reconstructed image of the reference unit or the characteristic information of the target reconstructed image of the reference unit, and Y2 represents the initial reconstructed image of the coding unit or the characteristic information of the initial reconstructed image of the coding unit.

[0256] It should be understood that when Y1 is the target reconstructed image of the reference unit, Y2 is the initial reconstructed image of the coding unit. When Y1 is the feature information of the target reconstructed image of the reference unit, Y2 is the feature information of the initial reconstructed image of the coding unit.

[0257] When the motion complexity of the image of the coding unit relative to the image of the reference unit is low complexity, and the mask in formula (6) is 1, and the mask in formula (7) is 0, the reconstructed image information of the reference unit and the reconstructed image information of the coding unit are fused, and the reconstructed image information of the reference unit is copied to the coding unit to obtain the target reconstructed image of the coding unit. In this way, the error introduced by inter-frame prediction and entropy coding of residual information can be compensated, the error of the current frame can be reduced, and then the accumulated error of subsequent frames can be reduced, and the timing stability can be ensured.

[0258] When the motion complexity of the image of the coding unit relative to the image of the reference unit is high complexity, and the mask in formula (6) is 0, and the mask in formula (7) is 1, the target reconstructed image of the coding unit obtained by fusing the reconstructed image information of the reference unit and the reconstructed image information of the coding unit is essentially the initial reconstructed image of the coding unit. Since the motion complexity of the image of the coding unit relative to the image of the reference unit is high complexity, if the reconstructed image information of the reference unit is copied to the coding unit, the quality of the target reconstructed image of the coding unit is low. Therefore, the initial reconstructed image of the coding unit obtained by reconstruction can be used as the initial reconstructed image of the coding unit. In this way, the quality of the reconstructed image of the coding unit can be guaranteed.

[0259] When the motion complexity of the image of the coding unit relative to the image of the reference unit is low complexity, and mask in formula (6) and formula (7) is a value other than 0 and 1 (that is, the first preset value and the second preset value are values ​​other than 0 and 1), in the target image of the coding unit obtained by fusing the reconstructed image information of the reference unit and the reconstructed image information of the coding unit, the reconstructed image information of the coding unit and the reconstructed image information of the reference unit both occupy a certain proportion; in this way, the error introduced by inter-frame prediction and entropy coding of residual information can be compensated to a certain extent, and the error of the current frame can be reduced; thereby, the accumulated error of subsequent frames can be reduced to a certain extent, and the temporal stability can be ensured. In addition, the quality of the target reconstructed image of the coding unit can be guaranteed to a certain extent.

[0260] Exemplarily, one way to fuse the reconstructed image information of the reference unit and the reconstructed image information of the coding unit to obtain the target reconstructed image of the coding unit may be to input the preset parameters of the coding unit, the original image of the coding unit of the coding unit and the reconstructed image information of the reference unit into the fusion model to obtain the target reconstructed image of the coding unit. In this case, the size of the first preset value and the second preset value is not limited; wherein the fusion model is pre-trained and has the ability to generate high-quality information (such as high-quality images or high-quality feature information). In this way, the error of the current frame coding unit can also be reduced, thereby reducing the cumulative error of subsequent frames and ensuring timing stability. And the quality of the target reconstructed image of the coding unit can also be guaranteed to a certain extent.

[0261] It should be noted that, when the image of the coding unit in S304 does not include the initial reconstructed image of the coding unit, the present application does not limit the execution order of S304 and any of the above steps S301 to S303. When the image of the coding unit in S304 includes the initial reconstructed image of the coding unit, S304 is executed after S303 in the present application.

[0262] Figure 3C FIG. 1 is a schematic diagram of an exemplary encoding process. Figure 3C The current frame is used as an example for explanation; Figure 3C In the example, the first preset value is 1, the second preset value is 0; and the reconstructed image information of the reference unit used for fusion is the target reconstructed image of the reference frame, and the reconstructed image information of the coding unit used for fusion is the initial reconstructed image of the current frame.

[0263] Figure 3D FIG. 1 is a schematic diagram of an exemplary encoding process. Figure 3D The current frame is used as an example for explanation; Figure 3D In the example, the first preset value is 1, the second preset value is 0, and the reconstructed image information of the reference unit used for fusion is the characteristic information of the target reconstructed image of the reference frame, and the reconstructed image information of the coding unit used for fusion is the characteristic information of the initial reconstructed image of the current frame.

[0264] Reference Figure 3C and Figure 3D , exemplarily, the encoding process in the dotted box with thick lines on the upper left side corresponds to S301 to S303, which can be referred to Figure 2A The processing process in the dotted box with thick lines on the lower right side corresponds to S304 to S305. Figure 3C and Figure 3D In the preset parameters of the current frame, a black dot indicates that the value of the preset parameter of the corresponding encoding unit is 0, and a white dot indicates that the value of the preset parameter of the corresponding encoding unit is 1.

[0265] It should be noted that in one possible manner, the residual bitstream and the preset parameter bitstream are two bitstreams. In one possible manner, the residual bitstream and the preset parameter bitstream may be two parts of the same bitstream, that is, the same bitstream carries the residual information and the preset parameters.

[0266] Figure 4A FIG. 4 is a schematic diagram showing an exemplary decoding process. Figure 4A The decoding process in the embodiment is the same as Figure 3A The encoding process in the embodiment corresponds.

[0267] S401, receiving a code stream.

[0268] Exemplarily, the bitstream received by the decoding end may include an optical stream bitstream, a residual bitstream, and a preset parameter bitstream.

[0269] S402: Obtain preset parameters of the coding unit in the current frame according to the code stream.

[0270] In a possible manner, the preset parameters of the coding unit may be directly decoded from the preset parameter bitstream.

[0271] For example, the encoding end determines the probability distribution of the preset parameters of the encoding unit based on the optical stream and / or the residual stream; then, based on the probability distribution of the preset parameters, entropy encoding is performed on the preset parameters of the encoding unit to obtain the preset parameter stream; Figure 4B The decoding end can determine the probability distribution of the preset parameters of the coding unit based on the optical stream code stream and / or the residual code stream; then, the preset parameter code stream can be entropy decoded based on the probability distribution of the preset parameters to obtain the preset parameters of the coding unit.

[0272] In a possible manner, the preset parameters of the coding unit may be obtained by processing information decoded from the preset parameter bitstream.

[0273] It should be understood that when S402 needs to use the residual code stream to determine the probability distribution of the preset parameters of the coding unit, S403 can be performed first, and then S402. And when S402 needs to use the optical flow code stream to determine the probability distribution of the preset parameters of the coding unit, the second optical flow of the coding unit can be obtained based on the optical flow code stream, and then S402 is performed.

[0274] S403: Obtain decoded residual information of the coding unit according to the bit stream.

[0275] For example, S403 can refer to the above Figure 2B The description of the decoding process in S403 is not repeated here. It should be noted that the residual information decoded by the coding unit in S403 corresponds to Figure 2B The second residual information of the encoding unit in the embodiment.

[0276] It should be noted that S402 and S403 may be executed by a decoding module in the AI ​​video decoding framework, wherein the decoding module may include Figure 2B A second entropy estimation module, a second entropy decoding module and a residual decoding module (these three modules are used to execute S403); and also includes a third entropy decoding module (the third entropy decoding module belongs to the entropy decoding module in the AI ​​video decoding framework, and the third entropy decoding module is used to execute S402).

[0277] S404, performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit.

[0278] For example, S404 can refer to the above Figure 2B The description of the inter-frame prediction process in the AI ​​video decoding framework is not repeated here. It should be noted that S404 can be performed by the inter-frame prediction module in the AI ​​video decoding framework, and the inter-frame prediction module may include Figure 2B The optical flow decoding module, the first entropy estimation module and the prediction module in.

[0279] It should be noted that the present application does not limit the execution order of S404 and S403. When S403 needs to be implemented based on the first prediction information of the coding unit, S404 can be executed first and then S403.

[0280] S405 , reconstructing according to the decoded residual information of the coding unit and the first prediction information to determine the reconstructed image information of the coding unit.

[0281] For example, S405 can refer to the above Figure 2B The description of the reconstruction process performed by the reconstruction module in will not be repeated here.

[0282] S406 , fusing the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to preset parameters to obtain a target reconstructed image of the coding unit.

[0283] Exemplarily, the fusion process of S406 is similar to the above Figure 3A The fusion process in the embodiment is similar and will not be described again here.

[0284] The following test set is used to test the AI ​​video codec method of the present application, the traditional codec method of the prior art (such as H266) and the AI ​​video codec method of the prior art, and compares the effects of the AI ​​video codec method of the present application and the AI ​​video codec method of the prior art.

[0285] For example, the test set includes 5: two 4k test sets (one 4K (A1) test set has a complex motion type, and one 4K test set (A2) has a simple motion type), a 1080p test set, an 832*480 test set (target object motion) and a 770p test set (conference video); the peak signal-to-noise ratio (PSNR) of the AI ​​video encoding and decoding method of the present application relative to H266, and the PSNR of the AI ​​video encoding and decoding method of the prior art relative to H266 can be shown in Table 1 below:

[0286] Table 1

[0287]

[0288] Referring to Table 1, Y-PSNR represents the PSNR of the Y component, U-PSNR represents the PSNR of the U component, and V-PSNR represents the PSNR of the V component.

[0289] It should be noted that the video data used for testing in the test set contains images that are YUV420 images, in which Y component: U component: V component = 8:1:1; therefore, in Table 1, the focus is on Y-PSNR. For the above five test sets, the PSNR of the AI ​​video codec method of the present application is better than the PSNR of the AI ​​video codec method of the prior art; in particular, for the 720p test set, the PSNR of the AI ​​video codec method of the present application is improved by 7.54% relative to the PSNR of the AI ​​video codec method of the prior art. In other words, the coding and decoding method of the present application can reduce the cumulative error of videos with low motion complexity (i.e., the motion complexity of the image of the coding unit relative to the image of the reference unit is low complexity) while ensuring or improving the rate-distortion (RD) performance of video data with high motion complexity (i.e., the motion complexity of the image of the coding unit relative to the image of the reference unit is high complexity) to a certain extent.

[0290] Figure 4C FIG. 1 is a schematic diagram showing the reconstruction quality of a test video. Figure 4C In , the horizontal axis is the frame number of the image frame, and the vertical axis is the Y-PSNR. Figure 4B The Y-PNSR of each frame image in two image groups (each image group includes 50 frames of images) is shown.

[0291] Reference Figure 4CIn the figure, curve 1 is the Y-PSNR of the reconstructed image obtained by encoding using the AI ​​video encoding and decoding method of the present application; curve 2 is the Y-PSNR of the reconstructed image obtained by encoding using the AI ​​video encoding and decoding method of the prior art; by comparing curve 1 and curve 2, it can be seen that, compared with the prior art, the quality of the reconstructed image of the present application is more stable, and the cumulative error is smaller, that is, the reconstructed image quality is high.

[0292] For example, in Figure 3A Based on the embodiment, the residual information of the coding unit may be adaptively encoded at a rate according to preset parameters of the coding unit to reduce the rate overhead.

[0293] Figure 5A FIG. 1 is a schematic diagram of an exemplary encoding process.

[0294] S501, performing inter-frame prediction on a coding unit in a current frame to obtain first prediction information of the coding unit.

[0295] S502: Determine residual information of the coding unit according to the first prediction information and the original image of the coding unit.

[0296] S503: Determine preset parameters of the encoding unit.

[0297] For example, S501 to S503 may refer to the description of S301, S302 and S304 above, which will not be repeated here. It should be noted that the residual information of the coding unit in S502 corresponds to Figure 2A The first residual information of the encoding unit in the embodiment.

[0298] S504: Perform adaptive rate encoding on the residual information of the coding unit according to preset parameters.

[0299] Exemplarily, when the value of the preset parameter of the coding unit is the first preset value, the residual information of the coding unit can be encoded at a low bit rate to obtain a residual code stream; when the value of the preset parameter of the coding unit is the second preset value, the residual information of the coding unit can be encoded at a high bit rate to obtain a residual code stream. In other words, the bit rate of the residual information encoding is adaptively adjusted according to the preset parameters of the coding unit. Since the reconstructed image information of the reference unit and the reconstructed image information of the coding unit are fused during the reconstruction process to generate the target reconstructed image of the coding unit; therefore, even if the residual information of the current frame is encoded at a low bit rate, the quality of the target reconstructed image of the coding unit can be guaranteed; and further, Figure 5A The implementation scheme can reduce the bit rate overhead to a certain extent while maintaining the same quality.

[0300] Specifically, when the value of the preset parameter is a first preset value, the residual information is encoded according to the first bit rate parameter; when the value of the preset parameter is a second preset value, the residual information is encoded according to the second bit rate parameter; wherein a target bit rate of the bit stream obtained by encoding the residual information according to the second bit rate parameter is greater than the target bit rate of the bit stream obtained by encoding the residual information according to the first bit rate parameter.

[0301] S505, encoding preset parameters of the encoding unit.

[0302] Figure 5B FIG. 1 is a schematic diagram of an exemplary encoding process. Figure 5B is Figure 3B As shown on the basis; wherein, the second entropy coding module can perform adaptive rate coding on the residual information of the coding unit according to the residual probability distribution and the preset parameters of the coding unit to obtain a residual code stream.

[0303] Figure 6 FIG. 4 is a schematic diagram showing an exemplary decoding process. Figure 6 The decoding process in the embodiment is the same as Figure 5A The encoding process in the embodiment corresponds.

[0304] S601, receiving a code stream.

[0305] S602: Obtain preset parameters of the coding unit in the current frame according to the code stream.

[0306] Illustratively, S601 to S602 may refer to the description of S401 to S402 above, which will not be repeated here.

[0307] S603: Perform adaptive bit rate decoding on the residual bit stream according to preset parameters of the encoding unit.

[0308] Specifically, when the value of the preset parameter of the encoding unit is a first preset value, the residual bitstream is decoded according to the first bitrate parameter; when the value of the preset parameter of the encoding unit is a second preset value, the residual bitstream is decoded according to the second bitrate parameter.

[0309] S604, determining decoded residual information of the coding unit according to information obtained by adaptive bit rate decoding of the residual bit stream.

[0310] Exemplarily, the information obtained by adaptively decoding the residual code stream may be input into a residual decoding module, which processes the information and outputs the decoded residual information of the coding unit.

[0311] S605, performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit.

[0312] S606: Reconstruct according to the decoded residual information of the coding unit and the first prediction information to determine the reconstructed image information of the coding unit.

[0313] S607 , fusing the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to preset parameters to obtain a target reconstructed image of the coding unit.

[0314] Exemplarily, S605 to S607 may refer to the description of S404 to S406 above, which will not be repeated here.

[0315] It should be noted that Figure 6 In the embodiment, the decoded residual information of the encoding unit corresponds to Figure 2B The second residual information of the encoding unit in the embodiment.

[0316] For example, in Figure 3A On this basis, adaptive rate encoding can be performed on the first optical flow of the encoding unit according to preset parameters of the encoding unit to reduce the bit rate overhead.

[0317] Fig. 7A FIG. 1 is a schematic diagram of an exemplary encoding process.

[0318] S701, performing motion estimation on a coding unit in a current frame to obtain a first optical flow of the coding unit.

[0319] S702, predicting the coding unit in the current frame according to the first optical flow of the coding unit to obtain first prediction information of the coding unit.

[0320] S703: Determine residual information of the coding unit according to the first prediction information and the original image of the coding unit.

[0321] S704, encode residual data of the coding unit.

[0322] S705: Determine preset parameters of the encoding unit.

[0323] Illustratively, S701 to S705 may refer to the description of S301 to S304 above, which will not be repeated here.

[0324] S706: Perform adaptive rate encoding on the first optical flow of the encoding unit according to preset parameters of the encoding unit.

[0325] Exemplarily, the adaptive rate encoding of the first optical flow of the coding unit in S706 may be adaptive rate encoding of feature information of the first optical flow of the coding unit.

[0326] Exemplarily, when the value of the preset parameter of the encoding unit is a first preset value, the characteristic information of the first optical flow of the encoding unit can be encoded at a low bit rate to obtain an optical flow code stream; when the value of the preset parameter of the encoding unit is a second preset value, the characteristic information of the first optical flow of the encoding unit can be encoded at a high bit rate to obtain an optical flow code stream. In other words, the bit rate of the first optical flow encoding is adaptively adjusted according to the preset parameters of the encoding unit. Since the reconstructed image information of the reference unit and the reconstructed image information of the encoding unit are fused during the reconstruction process to generate the target reconstructed image of the encoding unit; therefore, even if the characteristic information of the first optical flow of the current frame is encoded at a low bit rate, the quality of the target reconstructed image of the encoding unit can be guaranteed; and further, Fig. 7A The implementation scheme can reduce the bit rate overhead to a certain extent while maintaining the same quality.

[0327] Specifically, when the value of the preset parameter is a first preset value, the characteristic information of the first optical flow is encoded according to the third bit rate parameter; when the value of the preset parameter is a second preset value, the characteristic information of the first optical flow is encoded according to the fourth bit rate parameter; wherein the target bit rate of the bit stream obtained by encoding the first optical flow according to the fourth bit rate parameter is greater than the target bit rate of the bit stream obtained by encoding the first optical flow according to the third bit rate parameter.

[0328] S707, encode preset parameters of the encoding unit.

[0329] It should be noted that Fig. 7A The residual information of the coding unit in the embodiment corresponds to Figure 2A The first residual information of the coding unit in .

[0330] Figure 7B FIG. 1 is a schematic diagram of an exemplary encoding process. Figure 7B is Figure 3B As shown on the basis; wherein, the first entropy coding module can perform adaptive code rate coding on the feature information of the first optical flow of the coding unit according to the optical flow probability distribution and the preset parameters of the coding unit to obtain an optical flow code stream.

[0331] It should be noted that the present application does not limit the order of encoding the first optical flow, encoding the residual information, and encoding the preset parameters, that is, the present application does not limit the execution order of S704, S706, and S707.

[0332] It should be noted that, in one possible manner, the optical stream code stream, the residual code stream and the preset parameter code stream are three code streams. In one possible manner, the optical stream code stream, the residual code stream and the preset parameter code stream can be three parts of the same code stream, that is, the same code stream carries the residual information, the preset parameters and the first optical stream. In one possible manner, the residual code stream and the preset parameter code stream can be two parts of the same code stream, and the optical stream code stream is another code stream. In one possible manner, the residual code stream and the optical stream code stream can be two parts of the same code stream, and the preset parameter code stream is another code stream. In one possible manner, the preset parameter code stream and the optical stream code stream can be two parts of the same code stream, and the residual code stream is another code stream; this application does not impose any restrictions on this.

[0333] Figure 8 FIG. 4 is a schematic diagram showing an exemplary decoding process. Figure 8 The decoding process in the embodiment is the same as Fig. 7A The encoding process in the embodiment corresponds.

[0334] S801, receiving a code stream.

[0335] S802: Obtain preset parameters of the coding unit in the current frame according to the code stream.

[0336] S803: Obtain decoded residual information of the coding unit according to the bit stream.

[0337] Illustratively, S801 to S803 may refer to the description of S401 to S403 above, which will not be repeated here.

[0338] S804: Adaptively rate decode the optical stream according to preset parameters of the encoding unit.

[0339] Specifically, when the value of the preset parameter of the encoding unit is the first preset value, the optical stream is decoded according to the third rate parameter; when the value of the preset parameter of the encoding unit is the second preset value, the optical stream is decoded according to the fourth rate parameter.

[0340] S805 , determining a second optical flow of the coding unit according to information obtained by adaptively decoding the optical flow code stream.

[0341] Exemplarily, information obtained by adaptively decoding the optical flow code stream may be input into an optical flow decoding module, which processes the information and outputs the second optical flow of the encoding unit.

[0342] S806: Predict the coding unit according to the second optical flow of the coding unit to obtain first prediction information of the coding unit.

[0343] S807, reconstruct according to the decoded residual information of the coding unit and the first prediction information to determine the reconstructed image information of the coding unit.

[0344] S808 , fusing the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to preset parameters to obtain a target reconstructed image of the coding unit.

[0345] Exemplarily, S805 to S808 may refer to the description of S404 to S406 above, which will not be repeated here.

[0346] It should be noted that Figure 8 In the embodiment, the decoded residual information of the encoding unit corresponds to Figure 2B The second residual information of the encoding unit in the embodiment.

[0347] It should be understood that in Figure 3A On the basis of, according to the preset parameters of the coding unit, the first optical flow of the coding unit can be adaptively encoded at a code rate and the residual information of the coding unit can be adaptively encoded at a code rate; in this way, the code rate overhead can be further reduced; specifically, Figure 5A Examples and Fig. 7A The description in the embodiment will not be repeated here. And based on FIG4, according to the preset parameters of the encoding unit, the optical stream code stream is adaptively decoded and the residual code stream is adaptively decoded. Specifically, Figure 6 Examples and Figure 8 The description in the embodiment will not be repeated here. Figure 5A Examples and Fig. 7A The video encoding method combined with the embodiment, and Figure 6 Examples and Figure 8 The video decoding method combined with the embodiment can refer to the reconstruction quality of the reconstructed video data obtained by Fig. 9 .

[0348] Fig. 9 FIG. 1 is a schematic diagram showing the reconstruction quality of a test video. Fig. 9 In , the horizontal axis is the frame number of the image frame, and the vertical axis is the Y-PSNR. Fig. 9 The Y-PNSR of each frame image in two image groups (each image group includes 50 frames of images) is shown.

[0349] Reference Fig. 9 In the figure, curve 1 is the Y-PSNR of the reconstructed image obtained by encoding using the AI ​​video encoding and decoding method of the present application; curve 2 is the Y-PSNR of the reconstructed image obtained by encoding using the AI ​​video encoding and decoding method of the prior art; by comparing curve 1 and curve 2, it can be seen that, compared with the prior art, the quality of the reconstructed image of the present application is more stable and the cumulative error is smaller.

[0350] In the following Fig. 10A and Fig.11 In the embodiment of Figure 2A On the basis of, the first prediction information obtained by the inter-frame prediction is post-processed to reduce the accumulated error.

[0351] Fig. 10A FIG. 1 is a schematic diagram of an exemplary encoding process.

[0352] S1001, determining preset parameters of a coding unit in a current frame.

[0353] Exemplarily, S1001 may be performed by a preset parameter determination module of an AI video encoding framework.

[0354] Exemplarily, the preset parameters of the coding unit can be determined based on the original image of the coding unit and the image of the reference unit; wherein the image of the reference unit includes the original image of the reference unit in the reference frame and / or the target reconstructed image of the reference unit. The specific process of determining the preset parameters of the coding unit is similar to that described in S304 above and will not be repeated here. The difference between S1001 and S304 is that since S1001 is performed before inter-frame prediction, the information used to determine the preset parameters of the coding unit in S1001 does not include the initial reconstructed image of the coding unit.

[0355] The preset parameters may be used to describe the motion complexity of the original image of the coding unit relative to the image of the reference unit.

[0356] S1002, performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit.

[0357] Illustratively, S1002 may refer to the description of S301 above, which will not be repeated here.

[0358] Exemplarily, S1002 may be performed by an inter-frame prediction module in an AI video coding framework, wherein the inter-frame prediction module may include Figure 3A The invention comprises a motion estimation module, an optical flow processing module (including an optical flow encoding module and an optical flow decoding module), a first entropy estimation module and a prediction module.

[0359] S1003, fusing the first prediction information and the reconstructed image information of the reference unit according to preset parameters to obtain second prediction information of the coding unit.

[0360] Exemplarily, S1003 can be executed by a fusion module in the AI ​​video encoding framework; wherein the fusion module can be an independent module or integrated in the reconstruction module, which is not limited in this application.

[0361] Exemplarily, one way to fuse the reconstructed image information of a reference unit and the first prediction information of a coding unit to obtain the second prediction information of the coding unit may be to determine the weight of the reconstructed image information of the reference unit and the weight of the first prediction information of the coding unit according to preset parameters; and to perform weighted calculation on the reconstructed image information of the reference unit and the first prediction information of the coding unit according to the weight of the reconstructed image information of the reference unit and the weight of the first prediction information of the coding unit to obtain the second prediction information of the coding unit.

[0362] In this case, if the first preset value is greater than the second preset value, when the first preset value and the second preset value are numbers between 0 and 1 (the first preset value and the second preset value can both be 0 or 1), the value of the preset parameter of the coding unit can be used as the weight of the reconstructed image information of the reference unit; the difference between 1 and the value of the preset parameter of the coding unit can be used as the weight of the first prediction information of the coding unit. When the first preset value and the second preset value are numbers greater than 1, the first preset value and the second preset value can be transformed into numbers between 0 and 1, and then the value of the preset parameter of the transformed coding unit can be used as the weight of the reconstructed image information of the reference unit, and the difference between 1 and the value of the preset parameter of the transformed coding unit can be used as the weight of the first prediction information of the coding unit. At this time, the fusion can be performed with reference to the following formula (8):

[0363] H1=mask*P +(1-mask)*H2 (8)

[0364] If the second preset value is greater than the first preset value, the fusion can be performed according to the following formula (9):

[0365] H1=mask*H2+(1-mask)*P (9)

[0366] In formula (8) and formula (9), H1 represents the second prediction information of the coding unit, P is the target reconstructed image of the reference unit or the feature information of the target reconstructed image of the reference unit, and H2 is the first prediction information of the coding unit.

[0367] When the motion complexity of the original image of the coding unit relative to the image of the reference unit is low complexity, and the mask in formula (8) is 1, and the mask in formula (9) is 0, the first prediction information of the coding unit and the reconstructed image information of the reference unit are fused, and the reconstructed image information of the reference unit is copied to the coding unit to obtain the second prediction information of the coding unit. In this way, the accuracy of inter-frame prediction can be improved, thereby reducing the error introduced by inter-frame prediction, reducing the error of the current frame, thereby reducing the cumulative error of subsequent frames, and ensuring timing stability.

[0368] When the motion complexity of the original image of the coding unit relative to the image of the reference unit is high complexity, and the mask in formula (8) is 0 and the mask in formula (9) is 1, the second prediction information of the coding unit obtained by fusing the first prediction information of the coding unit and the reconstructed image information of the reference unit is substantially the first prediction information of the coding unit. Since the motion complexity of the original image of the coding unit relative to the image of the reference unit is high complexity, if the reconstructed image information of the reference unit is copied to the coding unit, the accuracy of the second prediction information of the coding unit is low. Therefore, the first prediction information of the coding unit obtained by inter-frame prediction can be used as the second prediction information of the coding unit, which can ensure the accuracy of the inter-frame prediction and thus ensure the quality of the target reconstructed image of the coding unit.

[0369] When the motion complexity of the original image of the coding unit relative to the image of the reference unit is low complexity, and the mask in formula (8) and formula (9) is a value other than 0 and 1 (that is, the first preset value and the second preset value are values ​​other than 0 and 1), in the second prediction information of the coding unit obtained by fusing the first prediction information of the coding unit and the reconstructed image information of the reference unit, the first prediction information of the coding unit and the reconstructed image information of the reference unit both occupy a certain proportion, which can also improve the accuracy of inter-frame prediction to a certain extent and reduce the error of the current frame; thereby reducing the cumulative error of subsequent frames to a certain extent and ensuring temporal stability. And it can also ensure the quality of the target reconstructed image of the coding unit to a certain extent.

[0370] Exemplarily, one way to fuse the reconstructed image information of the reference unit and the first prediction information to obtain the second prediction information of the coding unit may be to input the preset parameters of the coding unit, the first prediction information of the coding unit, and the reconstructed image information of the reference unit into the fusion model (this is similar to the above Figure 3A The fusion model in the embodiment is not the same model), and the second prediction information of the coding unit is obtained. In this case, the size of the first preset value and the second preset value is not limited; wherein the fusion model is pre-trained and has the ability to generate more accurate prediction information. In this way, the error of the current frame coding unit can also be reduced, thereby reducing the cumulative error of subsequent frames and ensuring timing stability.

[0371] S1004: Determine residual information of the coding unit according to the second prediction information and the original image of the coding unit.

[0372] Exemplarily, S1104 may be performed by a residual coding module in an AI video coding framework; Figure 2A The description in the embodiment will not be repeated here. It should be noted that the residual information of the coding unit determined in S1104 corresponds to Figure 2AThe first residual information of the encoding unit in the embodiment.

[0373] S1005, encode residual information of the coding unit.

[0374] Exemplarily, S1005 may be executed by the second entropy coding module to obtain a residual bit stream, which will not be described in detail here. It should be noted that the residual information of the coding unit in S1005 corresponds to Figure 2A The first residual information of the encoding unit in the embodiment.

[0375] S1006, encoding preset parameters of the encoding unit.

[0376] Illustratively, S1006 may refer to the description of S305 in the above embodiment, which will not be repeated here.

[0377] Fig. 10B FIG. 1 is a schematic diagram of an exemplary encoding process. Fig. 10B The current frame is used as an example for explanation; Fig. 10B In the example, the first preset value is 1, the second preset value is 0; and the reconstructed image information of the reference unit used for fusion is the target reconstructed image of the reference frame.

[0378] Refer to the figure Fig. 10B , exemplarily, the encoding process in the dotted box with thick lines on the upper left side corresponds to S1002, S1004~S1006, which can be referred to Figure 2A The processing process of the dotted box with thick lines on the lower right side can correspond to steps S1001 and S1003. Fig. 10B In the preset parameters of the current frame, a black dot indicates that the value of the preset parameter of the corresponding encoding unit is 0, and a white dot indicates that the value of the preset parameter of the corresponding encoding unit is 1.

[0379] Fig.11 FIG. 4 is a schematic diagram showing an exemplary decoding process. Fig.11 The decoding process and Fig. 10A The decoding process corresponds to .

[0380] S1101, receiving a code stream.

[0381] S1102: Obtain preset parameters of the encoding unit according to the bit stream.

[0382] S1103, obtaining decoded residual information of the coding unit in the current frame according to the bit stream.

[0383] S1104, performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit.

[0384] Illustratively, S1101 to S1104 may refer to the description of S401 to S404 above, which will not be repeated here.

[0385] It should be noted that the residual information decoded by the coding unit in S1103 corresponds to Figure 2B The second residual information of the encoding unit in the embodiment.

[0386] It should be noted that S1102 and S1103 can be executed by a decoding module in the AI ​​video decoding framework, wherein the decoding module may include Figure 2B A second entropy estimation module, a second entropy decoding module and a residual decoding module (these three modules are used to execute S1103); and also includes a third entropy decoding module (the third entropy decoding module belongs to the entropy decoding module in the AI ​​video decoding framework, and the third entropy decoding module is used to execute S1102).

[0387] S1105 , fusing the first prediction information and the reconstructed image information of the reference unit according to preset parameters to obtain second prediction information of the coding unit.

[0388] Exemplarily, the fusion process of S1105 is similar to the above Fig. 10A The fusion process in S1003 in the embodiment is similar and will not be repeated here.

[0389] Exemplarily, the AI ​​video decoding framework may include a fusion module, and S1005 may be executed by the fusion module in the AI ​​video encoding framework; wherein, the fusion module may be an independent module or may be integrated in a reconstruction module, and this application does not impose any restrictions on this.

[0390] S1106, reconstructing according to the decoded residual information and the second prediction information, and determining a target reconstructed image of the coding unit.

[0391] For example, S1106 can refer to the above Figure 2B The description of the reconstruction process in the embodiment will not be repeated here.

[0392] It should be noted that Fig.11 In the embodiment, the decoded residual information corresponds to Figure 2B The second residual information in the embodiment.

[0393] It should be understood that the following two fusion steps can be performed simultaneously during the encoding and decoding process: fusing the first prediction information and the reconstructed image information of the reference unit according to the preset parameters of the encoding unit to obtain the second prediction information of the encoding unit, and fusing the reconstructed image information of the reference unit and the reconstructed image information of the encoding unit according to the preset parameters of the encoding unit to obtain the target reconstructed image of the encoding unit. Fig. 10A Examples and Figure 3AThe embodiment is encoded, and combined with Fig.11 Examples and Figure 4A This can further reduce the accumulated error.

[0394] It should be understood that in Fig. 10A Based on the embodiment, the residual information of the coding unit may be adaptively encoded at a rate according to preset parameters of the coding unit to reduce the rate overhead.

[0395] It should be understood that in Fig. 10A On this basis, adaptive rate encoding can be performed on the first optical flow of the encoding unit according to preset parameters of the encoding unit to reduce the bit rate overhead.

[0396] It should be understood that in Fig. 10A On this basis, adaptive rate encoding can be performed on the residual information of the coding unit and the first optical flow of the coding unit according to preset parameters of the coding unit to further reduce the bit rate overhead.

[0397] It should be understood that it is also possible to Fig. 10A Example and Figure 5A Embodiment combination, and Fig.11 Example and Figure 6 The embodiment is combined to further reduce the accumulated error and reduce the code rate overhead.

[0398] It should be understood that it is also possible to Fig. 10A Example and Fig. 7A Embodiment combination, and Fig.11 Example and Figure 8 The embodiment is combined to further reduce the accumulated error and reduce the code rate overhead.

[0399] It should be understood that it is also possible to Fig. 10A Example Figure 5A Examples and Fig. 7A Embodiment combination, and Fig.11 Example Figure 6 Examples and Figure 8 The embodiments are combined to further reduce the accumulated error and reduce the code rate overhead.

[0400] In one example, Fig.12 A schematic block diagram of a device 1200 according to an embodiment of the present application is shown. The device 1200 may include: a processor 1201 and a transceiver / transceiver pin 1202 , and optionally, a memory 1203 .

[0401] The various components of the device 1200 are coupled together via a bus 1204, wherein the bus 1204 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, various buses are referred to as bus 1204 in the figure.

[0402] Optionally, the memory 1203 may be used to store instructions in the aforementioned method embodiment. The processor 1201 may be used to execute instructions in the memory 1203, and control the receiving pin to receive a signal, and control the sending pin to send a signal.

[0403] The device 1200 may be an electronic device (such as an encoding device, a decoding device, a decoding device, etc.) or a chip of an electronic device in the above method embodiment.

[0404] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0405] The embodiment of the present application also provides a chip, including one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the above-mentioned related method steps implement the steps of the method in the above-mentioned embodiment. The interface circuit is a transceiver / transceiver pin 1202.

[0406] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0407] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0408] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the methods in the above-mentioned method embodiments.

[0409] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0410] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0411] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0412] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0413] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0414] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0415] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0416] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (ReadOnly Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disk (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0417] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0418] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A coding method, characterized in that: The method comprises: Performing inter-frame prediction on a coding unit in a current frame to obtain first prediction information of the coding unit; Determine residual information of the coding unit according to the first prediction information and the original image of the coding unit; Determine preset parameters of the coding unit; wherein the preset parameters are used to fuse the reconstructed image information of the reference unit and the reconstructed image information of the coding unit to obtain a target reconstructed image of the coding unit, the reconstructed image information of the reference unit includes the target reconstructed image of the reference unit or feature information of the target reconstructed image of the reference unit, the reconstructed image information of the coding unit includes the initial reconstructed image of the coding unit or feature information of the initial reconstructed image of the coding unit, the initial reconstructed image of the coding unit and the feature information of the initial reconstructed image of the coding unit are reconstructed according to the first prediction information and the residual information, and the reference unit is a unit in the reference frame corresponding to the current frame; The residual information is encoded, and the preset parameter is encoded.

2. The method according to claim 1, characterized in that The encoding of the residual information comprises: According to the preset parameters, adaptive rate encoding is performed on the residual information.

3. The method according to claim 1 or 2, characterized in that: The performing inter-frame prediction on the coding unit in the current frame to obtain first prediction information of the coding unit includes: Determining a first optical flow of the coding unit according to the original image of the coding unit and the target reconstructed image of the reference unit; Determining the first prediction information according to the first optical flow and the reconstructed image information of the reference unit; The method further comprises: According to the preset parameters, adaptive rate encoding is performed on the first optical flow.

4. The method according to any one of claims 1 to 3, characterized in that: The determining of the preset parameters of the encoding unit includes: Determine the similarity between the image of the coding unit and the image of the reference unit; wherein the image of the coding unit includes the original image of the coding unit and / or the initial reconstructed image of the coding unit, and the image of the reference unit includes the original image of the reference unit and / or the target reconstructed image of the reference unit; The preset parameters are determined according to the similarity.

5. The method according to any one of claims 1 to 4, characterized in that: The fusing the reconstructed image information of the reference unit and the reconstructed image information of the coding unit to obtain a target reconstructed image of the coding unit includes: Determining, according to the preset parameters, a weight of the reconstructed image information of the reference unit and a weight of the reconstructed image information of the encoding unit; According to the weight of the reconstructed image information of the reference unit and the weight of the reconstructed image information of the coding unit, weighted calculation is performed on the reconstructed image information of the reference unit and the reconstructed image information of the coding unit to obtain a target reconstructed image of the coding unit.

6. The method according to any one of claims 1 to 4, characterized in that: The fusing the reconstructed image information of the reference unit and the reconstructed image information of the coding unit to obtain a target reconstructed image of the coding unit includes: The preset parameters, the reconstructed image information of the reference unit and the reconstructed image information of the coding unit are input into a fusion model to obtain a target reconstructed image of the coding unit.

7. The method according to claim 2, characterized in that The performing adaptive rate encoding on the residual information according to the preset parameters includes: When the value of the preset parameter is a first preset value, encoding the residual information according to a first bit rate parameter; When the value of the preset parameter is a second preset value, encoding the residual information according to a second coding rate parameter; The target bit rate of the bit stream obtained by encoding the residual information according to the second bit rate parameter is greater than the target bit rate of the bit stream obtained by encoding the residual information according to the first bit rate parameter.

8. The method according to any one of claims 1 to 7, characterized in that: The encoding of the preset parameters includes: Performing entropy estimation according to at least one of the residual information or the second optical flow of the encoding unit to determine the probability distribution of the preset parameter; wherein the second optical flow is obtained according to the code stream obtained by encoding the first optical flow; The preset parameters are entropy encoded according to the probability distribution of the preset parameters.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: According to the preset parameters, the first prediction information and the reconstructed image information of the reference unit are fused to obtain second prediction information; The determining, according to the first prediction information and the original image of the coding unit, the residual information of the coding unit includes: Residual information of the coding unit is determined according to the second prediction information and an original image of the coding unit.

10. The method according to any one of claims 1 to 9, characterized in that: The method is applied to an AI video coding framework, which includes: an inter-frame prediction module, a residual coding module, an entropy coding module, a preset parameter determination module and a fusion module, wherein: The inter-frame prediction module is used to perform inter-frame prediction on the coding unit to obtain the first prediction information; and output the first prediction information to the residual coding module; The residual coding module is used to determine the residual information according to the first prediction information and the original image of the coding unit; and output the residual information to the entropy coding module; The preset parameter determination module is used to determine the preset parameters; output the preset parameters to the entropy coding module and the fusion module; The entropy coding module is used to encode the residual information and the preset parameters; The fusion module is used to fuse the reconstructed image information of the reference unit and the reconstructed image information of the encoding unit according to the preset parameters to obtain a target reconstructed image of the encoding unit.

11. The method according to claim 10, characterized in that The inter-frame prediction module includes a motion estimation network, an optical flow processing module and a prediction module, wherein: The motion estimation network is used to perform motion estimation according to the original image of the coding unit and the target reconstructed image of the reference unit to obtain a first optical flow of the coding unit; and output the first optical flow to the optical flow processing module; The optical flow processing module is used to process based on the first optical flow to determine a second optical flow; and output the second optical flow to the prediction module; The prediction module is used to perform prediction based on the reconstructed image information of the reference unit and the second optical flow to obtain the first prediction information.

12. A decoding method, characterized in that: The method comprises: Receive code stream; Obtaining decoded residual information of a coding unit in a current frame according to the code stream, and obtaining preset parameters of the coding unit according to the code stream; Performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit; Reconstructing according to the decoded residual information and the first prediction information to determine the reconstructed image information of the coding unit; wherein the reconstructed image information of the coding unit includes the initial reconstructed image of the coding unit or feature information of the initial reconstructed image of the coding unit; The reconstructed image information of the encoding unit and the reconstructed image information of the reference unit are fused according to the preset parameters to obtain a target reconstructed image of the encoding unit; wherein the reconstructed image information of the reference unit includes the target reconstructed image of the reference unit or feature information of the target reconstructed image of the reference unit, and the reference unit is a unit in a reference frame corresponding to the current frame.

13. The method according to claim 12, characterized in that The fusing the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to the preset parameters to obtain a target reconstructed image of the coding unit includes: Determining, according to the preset parameters, a weight of the reconstructed image information of the encoding unit and a weight of the reconstructed image information of the reference unit; According to the weight of the reconstructed image information of the coding unit and the weight of the reconstructed image information of the reference unit, weighted calculation is performed on the reconstructed image information of the coding unit and the reconstructed image information of the reference unit to obtain a target reconstructed image of the coding unit.

14. The method according to claim 12, characterized in that The fusing the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to the preset parameters to obtain a target reconstructed image of the coding unit includes: The preset parameters, the reconstructed image information of the encoding unit and the reconstructed image information of the reference unit are input into a fusion model to obtain a target reconstructed image of the encoding unit.

15. The method according to any one of claims 12 to 14, characterized in that The code stream includes a preset parameter code stream and an optical stream code stream, and obtaining the preset parameters of the coding unit in the current frame according to the code stream includes: Perform entropy estimation according to at least one of the decoded residual information or the second optical flow of the encoding unit to determine the probability distribution of the preset parameter; wherein the second optical flow is obtained according to the optical flow code stream; The preset parameters are obtained by entropy decoding from the preset parameter bitstream according to the probability distribution of the preset parameters.

16. The method according to any one of claims 12 to 15, characterized in that The code stream includes a residual code stream, and obtaining the decoded residual information of the coding unit according to the code stream includes: According to the preset parameters, adaptive bit rate decoding is performed on the residual bit stream; Determine the decoded residual information of the encoding unit according to the information obtained by decoding the residual bitstream at the adaptive bit rate.

17. The method according to any one of claims 12 to 16, characterized in that: The code stream includes an optical code stream, and the method further includes: According to the preset parameters, adaptive bit rate decoding is performed on the bit stream of the optical flow; The second optical flow of the encoding unit is determined according to information obtained by decoding the optical flow code stream at the adaptive code rate.

18. The method according to any one of claims 12 to 17, characterized in that The method is applied to an AI video decoding framework, which includes: a decoding module, an inter-frame prediction module, a reconstruction module and a fusion module, wherein: The decoding module is used to obtain the preset parameters according to the bit stream, and obtain the decoded residual information according to the bit stream; output the preset parameters to the fusion module, and output the decoded residual information to the reconstruction module; The inter-frame prediction module is used to perform inter-frame prediction on the coding unit to obtain the first prediction information; and output the first prediction information to the reconstruction module; The reconstruction module is used to reconstruct according to the decoded residual information and the first prediction information, determine the reconstructed image information of the coding unit; and output the reconstructed image information of the coding unit to the fusion module; The fusion module is used to fuse the reconstructed image information of the encoding unit and the reconstructed image information of the reference unit according to the preset parameters to obtain the target reconstructed image of the encoding unit.

19. A coding method, characterized in that: The method comprises: Determining preset parameters of a coding unit in a current frame; Performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit; fusing the first prediction information and the reconstructed image information of the reference unit according to the preset parameters to obtain the second prediction information of the coding unit; wherein the reconstructed image information of the reference unit includes the target reconstructed image of the reference unit or the feature information of the target reconstructed image of the reference unit, and the reference unit is a unit in the reference frame corresponding to the current frame; Determine residual information of the coding unit according to the second prediction information and the original image of the coding unit; The residual information is encoded, and the preset parameter is encoded.

20. The method according to claim 19, characterized in that The encoding of the residual information comprises: According to the preset parameters, adaptive rate encoding is performed on the residual information.

21. The method according to claim 19 or 20, characterized in that The performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit includes: Determining a first optical flow of the coding unit according to the original image of the coding unit and the target reconstructed image of the reference unit; Determining the first prediction information according to the first optical flow and the reconstructed image information of the reference unit; The method further comprises: According to the preset parameters, adaptive rate encoding is performed on the first optical flow.

22. The method according to any one of claims 19 to 21, characterized in that The method further comprises: fusing the reconstructed image information of the reference unit and the reconstructed image information of the encoding unit according to the preset parameters to obtain a target reconstructed image of the encoding unit; Among them, the reconstructed image information of the coding unit includes the initial reconstructed image of the coding unit or the characteristic information of the initial reconstructed image of the coding unit, and the initial reconstructed image of the coding unit and the characteristic information of the initial reconstructed image of the coding unit are reconstructed based on the second prediction information and the residual information.

23. The method according to any one of claims 19 to 22, characterized in that The method is applied to an AI video coding framework, which includes: an inter-frame prediction module, a residual coding module, an entropy coding module, a preset parameter determination module and a fusion module, wherein: The preset parameter determination module is used to determine the preset parameters; output the preset parameters to the fusion module and the entropy coding module; The inter-frame prediction module is used to perform inter-frame prediction on the coding unit to obtain the first prediction information; and output the first prediction information to the fusion module; The fusion module is used to fuse the first prediction information and the reconstructed image information of the reference unit according to the preset parameters to obtain the second prediction information; and output the second prediction information to the residual coding module; The residual coding module is used to determine the residual information according to the second prediction information and the original image of the coding unit; and output the residual information to the entropy coding module; The entropy coding module is used to encode the residual information and the preset parameters.

24. A decoding method, characterized in that: The method comprises: Receive code stream; Obtaining decoded residual information of a coding unit in a current frame according to the code stream, and obtaining preset parameters of the coding unit according to the code stream; Performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit; fusing the first prediction information and the reconstructed image information of the reference unit according to the preset parameters to obtain the second prediction information of the coding unit; wherein the reconstructed image information of the reference unit includes the target reconstructed image of the reference unit or the feature information of the target reconstructed image of the reference unit, and the reference unit is a unit in the reference frame corresponding to the current frame; Reconstruction is performed according to the decoded residual information and the second prediction information to determine the reconstructed image information of the coding unit; wherein the reconstructed image information of the coding unit includes the initial reconstructed image of the coding unit or the characteristic information of the initial reconstructed image of the coding unit, and the characteristic information of the initial reconstructed image of the coding unit is used to obtain the initial reconstructed image of the coding unit.

25. The method according to claim 24, characterized in that The method further comprises: The reconstructed image information of the encoding unit and the reconstructed image information of the reference unit are fused according to the preset parameters to obtain a target reconstructed image of the encoding unit.

26. The method according to claim 24 or 25, characterized in that The code stream includes a preset parameter code stream and an optical stream code stream, and obtaining the preset parameters of the encoding unit according to the code stream includes: Perform entropy estimation according to at least one of the decoded residual information and the second optical flow of the encoding unit to determine the probability distribution of the preset parameter; wherein the second optical flow is obtained according to the optical flow code stream; The preset parameters are obtained by entropy decoding from the preset parameter bitstream according to the probability distribution of the preset parameters.

27. The method according to any one of claims 24 to 26, characterized in that The method is applied to an AI video decoding framework, which includes: a decoding module, an inter-frame prediction module, a reconstruction module and a fusion module, wherein: The decoding module is used to obtain the decoded residual information according to the code stream, and obtain the preset parameters according to the code stream; output the decoded residual information to the reconstruction module, and output the preset parameters to the fusion module; The inter-frame prediction module is used to perform inter-frame prediction on the coding unit to obtain the first prediction information; and output the first prediction information to the fusion module; The fusion module is used to fuse the first prediction information and the reconstructed image information of the reference unit according to the preset parameters to obtain the second prediction information; and output the second prediction information to the reconstruction module; The reconstruction module is used to reconstruct according to the decoded residual information and the second prediction information to determine the reconstructed image information of the coding unit.

28. An encoding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the encoding device executes the method according to any one of claims 1 to 11 or any one of claims 19 to 23.

29. A decoding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the decoding device performs the method according to any one of claims 12 to 18 or any one of claims 24 to 27.

30. A decoding device, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method as described in any one of claims 1 to 11 or any one of claims 19 to 23 are executed.

31. A decoding device, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method as described in any one of claims 12 to 18 or any one of claims 24 to 27 are executed.

32. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 27.

33. A computer program product, characterized in that The computer program product comprises computer instructions, which, when executed by a computer or a processor, cause the steps of the method according to any one of claims 1 to 27 to be performed.

Images