Encoding method, decoding method, and electronic device

Abstract

Embodiments of the present application provide an encoding method, a decoding method, and an electronic device. The encoding method comprises: performing inter-frame prediction on an encoding unit in the current frame to obtain first prediction information of the encoding unit; on the basis of the first prediction information and an original image of the encoding unit, determining residual information of the encoding unit; determining a preset parameter of the encoding unit, wherein the preset parameter is used for fusing reconstructed image information of a reference unit and reconstructed image information of the encoding unit to obtain a target reconstructed image of the encoding unit, the reconstructed image information of the reference unit comprises a target reconstructed image of the reference unit or feature information of the target reconstructed image of the reference unit, and the reconstructed image information of the encoding unit comprises an initial reconstructed image of the encoding unit or feature information of the initial reconstructed image of the encoding unit; then encoding the residual information; and encoding the preset parameter. In this way, cumulative errors can be reduced, and the temporal stability is ensured.

Description

Coding and decoding method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 15, 2023, with application number 202311030599.9 and application name “Encoding and Decoding Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of coding and decoding, and in particular to a coding and decoding method and an electronic device. Background Art

[0003] As video evolves from HD to UHD, people's expectations for video quality are growing. At the same time, HD video places increasing demands on bandwidth and storage. Consequently, the need for video encoding, or compression, is becoming increasingly urgent to control bandwidth, transmission latency, and storage costs.

[0004] Among them, the artificial intelligence (AI) video compression (or codec) algorithm is based on deep learning and has better compression effect than traditional video compression technologies (such as H265 and H266).

[0005] Because video frames are temporally continuous, the difference between the previous and next frames is relatively small, which means there is temporal redundancy between video frames. This allows inter-frame coding to reduce bitrate overhead. During the inter-frame coding process of AI video coding, many steps, such as prediction and entropy coding of residual information, introduce errors. Consequently, when encoding a group of pictures (GOP), the cumulative error increases, and as the cumulative error increases, compression performance decreases.

[0006] Summary of the Invention

[0007] To address the above technical issues, the present application provides a coding and decoding method and electronic device. This method can reduce cumulative error to a certain extent and ensure temporal stability (i.e., ensure the stability of the compression performance of each frame in each GOP (GOP refers to a group of images between I frames, where an I frame is an intra frame, i.e., an intra-frame coded frame)).

[0008] In a first aspect, an embodiment of the present application provides a coding method, which includes: first, performing inter-frame prediction on a coding unit in a current frame to obtain first prediction information of the coding unit; then, determining the residual information of the coding unit based on the first prediction information and the original image of the coding unit; thereafter, determining the preset parameters of the coding unit; wherein the preset parameters are used to fuse the reconstructed image information of the reference unit and the reconstructed image information of the coding unit to obtain a target reconstructed image of the coding unit, the reconstructed image information of the reference unit includes the target reconstructed image of the reference unit or the feature information of the target reconstructed image of the reference unit, the reconstructed image information of the coding unit includes the initial reconstructed image of the coding unit or the feature information of the initial reconstructed image of the coding unit, the initial reconstructed image of the coding unit and the feature information of the initial reconstructed image of the coding unit are reconstructed based on the first prediction information and the residual information, and the reference unit is a unit in the reference frame corresponding to the current frame; subsequently, encoding the residual information; and encoding the preset parameters.

[0009] The preset parameters may be used to describe the motion complexity of the image of the coding unit relative to the image of the reference unit; specifically, the motion complexity may refer to difference or similarity; that is, the motion complexity of the image of the coding unit relative to the image of the reference unit may refer to the similarity between the image of the coding unit and the image of the reference unit, or may refer to the difference between the image of the coding unit and the image of the reference unit. For example, the smaller the difference or the greater the similarity between the image of the coding unit and the image of the reference unit, the lower the motion complexity of the image of the coding unit relative to the image of the reference unit; the greater the difference or the smaller the similarity between the image of the coding unit and the image of the reference unit, the higher the motion complexity of the image of the coding unit relative to the image of the reference unit.

[0010] The present application generates a target reconstructed image of the coding unit by fusing the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to preset parameters; that is, according to a fusion rule (or fusion strategy or fusion method) corresponding to the motion complexity of the image of the coding unit relative to the image of the reference unit, the reconstructed image information of the coding unit and the reconstructed image information of the reference unit are fused to generate the target reconstructed image of the coding unit. In this way, for coding units with low image motion complexity relative to the reference unit, the error introduced by the inter-frame prediction and the coding residual information in the process of encoding the coding unit can be compensated to a certain extent, thereby reducing the error of the coding unit to a certain extent, and then reducing the error of the current frame to a certain extent, improving the compression performance of the current frame; thereby reducing the cumulative error of subsequent frames to a certain extent, improving the compression performance of subsequent frames and ensuring temporal stability. For coding units with high image motion complexity relative to the reference unit, the quality of the target reconstructed image of the coding unit can be guaranteed, thereby ensuring the quality of the target reconstructed image of the current frame.

[0011] Exemplarily, the cumulative error is explained by taking the i-th frame as an example: the cumulative error of the reconstructed image of the i-th frame may refer to the error introduced in the process of encoding the i-th frame original image by the error between the reconstructed image of the reference frame corresponding to the i-th frame and the original image of the reference frame corresponding to the i-th frame. For example, in a GOP, the second frame original image is encoded with reference to the first frame reconstructed image. Since some steps in the encoding process may introduce errors, there is an error 1 between the second frame reconstructed image and the second frame original image. The third frame original image is encoded with reference to the second frame reconstructed image. There is an error 2 between the third frame reconstructed image and the third frame original image; wherein, error 2 includes error 3 introduced by error 1 in the process of encoding the third frame original image and error 4 introduced by some steps in encoding the third frame original image; error 3 may be called a cumulative error.

[0012] Exemplarily, the residual information of the coding unit may be the first residual of the coding unit (wherein the first residual may refer to the residual between the feature information of the original image of the coding unit (or the original image of the coding unit) and the first prediction information of the coding unit), or may be the feature information of the first residual of the coding unit (wherein the feature information of the first residual may also be referred to as the compressed feature of the first residual, which may be obtained by feature extraction of the first residual). This application does not impose any restrictions on this. This application is described by taking the residual information of the coding unit as the feature information of the first residual of the coding unit as an example.

[0013] For example, a residual bitstream obtained by encoding the residual information may be first decoded to obtain decoded residual information; then, reconstruction may be performed based on the first prediction information and the decoded residual information to obtain an initial reconstructed image of the coding unit and feature information of the initial reconstructed image of the coding unit. Feature recovery may be performed on the feature information of the initial reconstructed image of the coding unit to obtain the initial reconstructed image of the coding unit.

[0014] Exemplarily, the feature information of the target reconstructed image of the reference unit can be obtained by performing feature extraction on the target reconstructed image of the reference unit.

[0015] In one possible approach, the preset parameters of the encoding unit may be directly encoded to obtain a preset parameter bitstream; in this way, the decoding end may decode the preset parameter bitstream to obtain the preset parameters of the current frame.

[0016] In one possible approach, feature extraction can be performed on the preset parameters of the coding unit to obtain feature information of the preset parameters of the coding unit. This feature information of the preset parameters of the coding unit can then be encoded to obtain a preset parameter bitstream. In this way, the decoding end can perform feature recovery based on the information decoded from the preset parameter bitstream to obtain the preset parameters of the current frame.

[0017] Exemplarily, the current frame can be divided into multiple coding units, and the encoding method of the present application can be executed for each coding unit; wherein, the coding unit can include at least one pixel point, and when the coding unit includes multiple pixel points, the multiple pixel points included in the coding unit share a preset parameter.

[0018] Exemplarily, the encoding method of the present application may refer to an AI video encoding method, which can be used to perform inter-frame prediction on image frames in video data.

[0019] Exemplarily, residual information is encoded to obtain a residual bitstream.

[0020] It should be noted that in one possible approach, the residual bitstream and the preset parameter bitstream are two bitstreams. In one possible approach, the residual bitstream and the preset parameter bitstream can be two parts of the same bitstream, that is, the same bitstream carries the residual information and the preset parameters.

[0021] It should be noted that this application does not limit the order of encoding residual information and encoding preset parameters.

[0022] According to the first aspect, encoding the residual information includes: performing adaptive rate encoding on the residual information according to preset parameters, thereby reducing the bit rate overhead.

[0023] In other words, the bitrate of the residual information encoding is adaptively adjusted based on the preset parameters of the coding unit. Because the reconstruction process uses the reconstructed image information of the reference unit and the reconstructed image information of the coding unit to generate the target reconstructed image of the coding unit, even if the residual information of the current frame is encoded at a low bitrate, the quality of the target reconstructed image of the coding unit can be guaranteed, thereby reducing the bitrate overhead to a certain extent while maintaining the same quality.

[0024] According to the first aspect, or any implementation of the first aspect above, inter-frame prediction is performed on a coding unit in a current frame to obtain first prediction information for the coding unit, including: determining a first optical flow for the coding unit based on an original image of the coding unit and a target reconstructed image of a reference unit; determining the first prediction information based on the first optical flow and the reconstructed image information of the reference unit; and further comprising: performing adaptive rate encoding on the first optical flow based on preset parameters. In this way, bitrate overhead can be reduced.

[0025] In other words, the bitrate of the first optical flow encoding is adaptively adjusted based on the preset parameters of the encoding unit. Because the reconstruction process fuses the reconstructed image information of the reference unit with the reconstructed image information of the encoding unit to generate the encoding unit's target reconstructed image, even if the feature information of the first optical flow of the current frame is encoded at a low bitrate, the quality of the encoding unit's target reconstructed image can be guaranteed, thereby reducing the bitrate overhead to a certain extent while maintaining the same quality.

[0026] Exemplarily, performing adaptive rate encoding on the first optical flow according to preset parameters may include: determining characteristic information of the first optical flow; performing adaptive rate encoding on the characteristic information of the first optical flow according to the preset parameters; wherein encoding the characteristic information of the first optical flow may obtain an optical flow code stream.

[0027] It should be noted that, in one possible manner, the optical stream code stream, the residual code stream, and the preset parameter code stream are three code streams. In one possible manner, the optical stream code stream, the residual code stream, and the preset parameter code stream can be three parts of the same code stream, that is, the same code stream carries residual information, preset parameters, and the first optical stream. In one possible manner, the residual code stream and the preset parameter code stream can be two parts of the same code stream, and the optical stream code stream is another code stream. In one possible manner, the residual code stream and the optical stream code stream can be two parts of the same code stream, and the preset parameter code stream is another code stream. In one possible manner, the preset parameter code stream and the optical stream code stream can be two parts of the same code stream, and the residual code stream is another code stream; this application does not impose any restrictions on this.

[0028] It should also be noted that the present application does not limit the order of encoding the first optical flow, encoding the residual information, and encoding the preset parameters.

[0029] According to the first aspect, or any implementation of the first aspect above, determining the preset parameters of the coding unit includes: determining the similarity between the image of the coding unit and the image of the reference unit; wherein the image of the coding unit includes the original image of the coding unit and / or the initial reconstructed image of the coding unit, and the image of the reference unit includes the original image of the reference unit and / or the target reconstructed image of the reference unit; and determining the preset parameters based on the similarity.

[0030] When the similarity is greater than or equal to the similarity threshold, it indicates that the image of the coding unit and the image of the reference unit have a high similarity. In this case, it can be determined that the motion complexity of the image of the coding unit relative to the image of the reference unit is low, and the value of the preset parameter can be set to a first preset value; that is, the first preset value can be used to indicate that the motion complexity of the image of the coding unit relative to the image of the reference unit is low. When the similarity is less than the similarity threshold, it indicates that the image of the coding unit and the image of the reference unit have a low similarity. In this case, it can be determined that the motion complexity of the image of the coding unit relative to the image of the reference unit is high, and the value of the preset parameter can be set to a second preset value; that is, the second preset value can be used to indicate that the motion complexity of the image of the coding unit relative to the image of the reference unit is high. The similarity threshold can be set as required. Assuming that the similarity when two images are exactly the same is 1, the similarity threshold can be a decimal less than 1 and greater than 0.

[0031] It should be noted that the first preset value and the second preset value are different, the first preset value may be greater than the second preset value, or the first preset value may be less than the second preset value; it can be set specifically as required, and this application does not impose any restrictions on this. In one possible way, the first preset value and the second preset value may be numbers between 0 and 1 (the values ​​of the first preset value and the second preset value may both be 0 or 1). Optionally, the sum of the first preset value and the second preset value may be 1. For example, the first preset value is 1, and the second preset value is 0; or, the first preset value is 0, and the second preset value is 1. For another example, the first preset value is 0.8, and the second preset value is 0.2; or, the first preset value is 0.2, and the second preset value is 0.8.

[0032] In one possible approach, an error between the image of the coding unit and the image of the reference unit may be calculated, and based on the error between the image of the coding unit and the image of the reference unit, a similarity between the image of the coding unit and the image of the reference unit may be determined. Assuming that the similarity is 1 when two images are exactly the same, the difference between 1 and the error between the image of the coding unit and the image of the reference unit may be used as the similarity.

[0033] In one possible approach, a third optical flow of the coding unit can be obtained; the second prediction information of the coding unit can be determined based on the third optical flow of the coding unit and the target reconstructed image of the reference unit; and the similarity between the image of the coding unit and the image of the reference unit can be determined based on the error between the second prediction information of the coding unit and the image of the coding unit. The initial reconstructed image of the coding unit and the target reconstructed image of the reference unit can be input into a motion estimation module, which performs motion estimation and outputs the third optical flow of the coding unit. Alternatively, the feature information of the first optical flow of the coding unit can be input into an optical flow decoding model, which decodes the feature information of the first optical flow (i.e., recovers the features) and outputs the third optical flow of the coding unit.

[0034] It should be understood that there are many ways to determine similarity, such as calculating an error and determining similarity based on the error; or using a similarity model to determine similarity calculation, etc., which are not limited in this application. In addition, other methods can also be used to calculate the error, which are not limited in this application.

[0035] According to the first aspect, or any implementation method of the first aspect above, the reconstructed image information of the reference unit and the reconstructed image information of the coding unit are fused to obtain the target reconstructed image of the coding unit, including: determining the weight of the reconstructed image information of the reference unit and the weight of the reconstructed image information of the coding unit according to preset parameters; and performing weighted calculation on the reconstructed image information of the reference unit and the reconstructed image information of the coding unit according to the weight of the reconstructed image information of the reference unit and the weight of the reconstructed image information of the coding unit to obtain the target reconstructed image of the coding unit.

[0036] Exemplarily, when the value of the preset parameter is a first preset value, the weight of the reconstructed image information of the coding unit may be smaller than the weight of the reconstructed image information of the reference unit; when the value of the preset parameter is a second preset value, the weight of the reconstructed image information of the coding unit may be greater than the weight of the reconstructed image information of the reference unit.

[0037] If the first preset value is greater than the second preset value, then when the first preset value and the second preset value are numbers between 0 and 1 (the first preset value and the second preset value can both be 0 or 1), the value of the preset parameter of the coding unit can be used as the weight of the reconstructed image information of the reference unit, and the difference between 1 and the value of the preset parameter can be used as the weight of the reconstructed image information of the coding unit. When the first preset value and the second preset value are both numbers greater than 1, the first preset value and the second preset value can be converted to numbers between 0 and 1, and then the value of the preset parameter after the conversion of the coding unit can be used as the weight of the reconstructed image information of the reference unit, and the difference between 1 and the value of the preset parameter after the conversion of the coding unit can be used as the weight of the reconstructed image information of the coding unit. If the first preset value is less than the second preset value, the opposite is true.

[0038] When the motion complexity of the image of the coding unit relative to the image of the reference unit is low, and the first preset value is 1, fusing the reconstructed image information of the reference unit with the reconstructed image information of the coding unit may involve copying the reconstructed image information of the reference unit to the coding unit to obtain the target reconstructed image of the coding unit. This can compensate for errors introduced by inter-frame prediction and entropy coding of residual information, reduce the error of the current frame, and further reduce the accumulated error of subsequent frames, thereby ensuring temporal stability.

[0039] When the motion complexity of the image of the coding unit relative to the image of the reference unit is high complexity, and the second preset value is 0, the target reconstructed image of the coding unit obtained by fusing the reconstructed image information of the reference unit and the reconstructed image information of the coding unit is essentially the initial reconstructed image of the coding unit; since the motion complexity of the image of the coding unit relative to the image of the reference unit is high complexity, if the reconstructed image information of the reference unit is copied to the coding unit, the quality of the target reconstructed image of the coding unit obtained is low; therefore, the reconstructed initial reconstructed image of the coding unit can be used as the initial reconstructed image of the coding unit; in this way, the quality of the reconstructed image of the coding unit can be guaranteed.

[0040] When the motion complexity of the image of the coding unit relative to the image of the reference unit is low, and the first preset value and the second preset value are values ​​other than 0 and 1, the target image of the coding unit obtained by fusing the reconstructed image information of the reference unit and the reconstructed image information of the coding unit contains a certain amount of the reconstructed image information of the coding unit and the reconstructed image information of the reference unit. This can compensate for errors introduced by inter-frame prediction and entropy coding of residual information to a certain extent, reducing the error of the current frame. This can also reduce the accumulated error of subsequent frames to a certain extent, ensuring temporal stability. Furthermore, the quality of the target reconstructed image of the coding unit can be guaranteed to a certain extent.

[0041] According to the first aspect, or any implementation of the first aspect above, fusing the reconstructed image information of the reference unit and the reconstructed image information of the coding unit to obtain the target reconstructed image of the coding unit includes: inputting preset parameters, the reconstructed image information of the reference unit, and the reconstructed image information of the coding unit into a fusion model to obtain the target reconstructed image of the coding unit. In this case, the size of the first preset value and the second preset value may not be restricted; wherein the fusion model is pre-trained and has the ability to generate high-quality information (such as high-quality images or high-quality feature information). In this way, the error of the coding unit of the current frame can also be reduced, thereby reducing the cumulative error of subsequent frames and ensuring temporal stability. It can also ensure the quality of the target reconstructed image of the coding unit to a certain extent.

[0042] According to the first aspect, or any implementation of the first aspect above, adaptive rate encoding is performed on the residual information according to a preset parameter, including: when the value of the preset parameter is a first preset value, encoding the residual information according to the first rate parameter; when the value of the preset parameter is a second preset value, encoding the residual information according to the second rate parameter; wherein a target bit rate of the code stream obtained by encoding the residual information according to the second rate parameter is greater than the target bit rate of the code stream obtained by encoding the residual information according to the first rate parameter.

[0043] Because when the value of the preset parameter is the first preset value, the weight of the reconstructed image information of the coding unit is less than the weight of the reconstructed image information of the reference unit; when the value of the preset parameter is the second preset value, the weight of the reconstructed image information of the coding unit is greater than the weight of the reconstructed image information of the reference unit; therefore, when the motion complexity of the image of the coding unit is low relative to the image of the reference unit (i.e., the value of the preset parameter is the first preset value), the residual information can be encoded at a low bit rate, thereby reducing the bit rate overhead without reducing the quality of the target reconstructed image of the coding unit. When the motion complexity of the image of the coding unit is high relative to the image of the reference unit (i.e., the value of the preset parameter is the second preset value), the residual information can be encoded at a high bit rate, thereby ensuring the quality of the target reconstructed image of the coding unit.

[0044] Exemplarily, when the value of the preset parameter of the encoding unit is a first preset value, the characteristic information of the first optical flow of the encoding unit can be encoded at a low bit rate to obtain an optical flow code stream; when the value of the preset parameter of the encoding unit is a second preset value, the characteristic information of the first optical flow of the encoding unit can be encoded at a high bit rate to obtain an optical flow code stream.

[0045] Specifically, when the value of the preset parameter is a first preset value, the characteristic information of the first optical flow is encoded according to the third code rate parameter; when the value of the preset parameter is a second preset value, the characteristic information of the first optical flow is encoded according to the fourth code rate parameter; wherein, the target code rate of the code stream obtained by encoding the first optical flow according to the fourth code rate parameter is greater than the target code rate of the code stream obtained by encoding the first optical flow according to the third code rate parameter.

[0046] According to the first aspect, or any implementation manner of the first aspect above, encoding the preset parameters includes: performing entropy estimation based on at least one of the residual information or the second optical flow of the coding unit to determine the probability distribution of the preset parameters; wherein the second optical flow is obtained based on the code stream obtained by encoding the first optical flow; and entropy encoding the preset parameters according to the probability distribution of the preset parameters.

[0047] According to the first aspect, or any implementation of the first aspect above, the method further includes: fusing the first prediction information and the reconstructed image information of the reference unit according to preset parameters to obtain second prediction information; and determining residual information of the coding unit according to the first prediction information and the original image of the coding unit, including: determining the residual information of the coding unit according to the second prediction information and the original image of the coding unit. In this way, the accuracy of inter-frame prediction can be improved, thereby reducing the error introduced by inter-frame prediction, further reducing the error of the current frame, and thus reducing the accumulated error of subsequent frames.

[0048] According to the first aspect, or any implementation of the first aspect above, the method is applied to an AI video coding framework, the AI ​​video coding framework including: an inter-frame prediction module, a residual coding module, an entropy coding module, a preset parameter determination module, and a fusion module, wherein,

[0049] An inter-frame prediction module, configured to perform inter-frame prediction on the coding unit to obtain first prediction information; and output the first prediction information to the residual coding module;

[0050] A residual coding module, configured to determine residual information based on the first prediction information and the original image of the coding unit; and output the residual information to the entropy coding module;

[0051] The preset parameter determination module is used to determine the preset parameters; output the preset parameters to the entropy coding module and the fusion module;

[0052] Entropy coding module, used to encode residual information and preset parameters;

[0053] The fusion module is used to fuse the reconstructed image information of the reference unit and the reconstructed image information of the coding unit according to preset parameters to obtain the target reconstructed image of the coding unit.

[0054] Exemplarily, the inter-frame prediction module, residual coding module, entropy coding module, preset parameter determination module and fusion module can be implemented by a neural network or an algorithm, and this application does not impose any restrictions on this.

[0055] Exemplarily, the AI ​​video coding framework may further include a residual decoding network, which may input the first prediction information and the original image of the coding unit (or the feature information of the original image of the coding unit) into the residual coding network to obtain residual information; and input the result obtained by entropy decoding the encoding result of the residual information into the residual decoding network to obtain decoded residual information.

[0056] Exemplarily, the first prediction information and the original image of the coding unit can be directly input into the residual coding network to obtain the residual information of the coding unit; or the features of the original image of the coding unit can be extracted first to obtain the feature information of the original image of the coding unit, and then the first prediction information and the feature information of the original image of the coding unit can be input into the residual coding network to obtain the residual information of the coding unit; this application does not impose any restrictions on this.

[0057] According to the first aspect, or any implementation of the first aspect above, the inter-frame prediction module includes a motion estimation network, an optical flow processing module and a prediction module, wherein: the motion estimation network is used to perform motion estimation based on the original image of the coding unit and the target reconstructed image of the reference unit to obtain the first optical flow of the coding unit; the first optical flow is output to the optical flow processing module; the optical flow processing module is used to process based on the first optical flow to determine the second optical flow; the second optical flow is output to the prediction module; the prediction module is used to perform prediction based on the reconstructed image information of the reference unit and the second optical flow to obtain first prediction information.

[0058] In a second aspect, an embodiment of the present application provides a decoding method, which includes: first, receiving a code stream; then, obtaining decoded residual information of a coding unit in a current frame according to the code stream, and obtaining preset parameters of the coding unit according to the code stream; performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit; then, reconstructing according to the decoded residual information and the first prediction information to determine reconstructed image information of the coding unit; wherein the reconstructed image information of the coding unit includes an initial reconstructed image of the coding unit or feature information of the initial reconstructed image of the coding unit; thereafter, fusing the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to preset parameters to obtain a target reconstructed image of the coding unit; wherein the reconstructed image information of the reference unit includes the target reconstructed image of the reference unit or feature information of the target reconstructed image of the reference unit, and the reference unit is a unit in a reference frame corresponding to the current frame.

[0059] According to the second aspect, the reconstructed image information of the coding unit and the reconstructed image information of the reference unit are fused according to preset parameters to obtain a target reconstructed image of the coding unit, including: determining the weight of the reconstructed image information of the coding unit and the weight of the reconstructed image information of the reference unit according to the preset parameters; performing weighted calculation on the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to the weight of the reconstructed image information of the coding unit and the weight of the reconstructed image information of the reference unit to obtain the target reconstructed image of the coding unit.

[0060] According to the second aspect, or any implementation method of the second aspect above, the reconstructed image information of the coding unit and the reconstructed image information of the reference unit are fused according to preset parameters to obtain the target reconstructed image of the coding unit, including: inputting the preset parameters, the reconstructed image information of the coding unit and the reconstructed image information of the reference unit into a fusion model to obtain the target reconstructed image of the coding unit.

[0061] According to the second aspect, or any implementation of the second aspect above, the bitstream includes a preset parameter bitstream and an optical stream bitstream, and obtaining the preset parameters of the coding unit in the current frame based on the bitstream includes: performing entropy estimation based on at least one of the decoded residual information or the second optical stream of the coding unit to determine the probability distribution of the preset parameters; wherein the second optical stream is obtained based on the optical stream bitstream; and entropy decoding the preset parameters from the preset parameter bitstream based on the probability distribution of the preset parameters.

[0062] The preset parameter code stream may refer to a code stream obtained by encoding the preset parameters, and the optical flow code stream may refer to a code stream obtained by encoding the feature information of the first optical flow.

[0063] According to the second aspect, or any implementation of the second aspect above, the bitstream includes a residual bitstream, and obtaining the decoded residual information of the coding unit according to the bitstream includes: performing adaptive rate decoding on the residual bitstream according to preset parameters; and determining the decoded residual information of the coding unit based on information obtained by adaptive rate decoding the residual bitstream.

[0064] The residual code stream may refer to a code stream obtained by encoding the residual information.

[0065] According to the second aspect, or any implementation of the second aspect above, the code stream includes an optical flow code stream, and the method further includes: performing adaptive code rate decoding on the optical flow code stream according to preset parameters; and determining the second optical flow of the coding unit based on information obtained by adaptive code rate decoding the optical flow code stream.

[0066] According to the second aspect, or any implementation of the second aspect above, the method is applied to an AI video decoding framework, the AI ​​video decoding framework including: a decoding module, an inter-frame prediction module, a reconstruction module, and a fusion module, wherein:

[0067] A decoding module, configured to obtain preset parameters according to the bit stream, and obtain decoded residual information according to the bit stream; output the preset parameters to the fusion module, and output the decoded residual information to the reconstruction module;

[0068] An inter-frame prediction module, configured to perform inter-frame prediction on the coding unit to obtain first prediction information; and output the first prediction information to the reconstruction module;

[0069] A reconstruction module, configured to reconstruct according to the decoded residual information and the first prediction information, determine reconstructed image information of the coding unit, and output the reconstructed image information of the coding unit to the fusion module;

[0070] The fusion module is used to fuse the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to preset parameters to obtain a target reconstructed image of the coding unit.

[0071] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0072] In a third aspect, an embodiment of the present application also provides a coding method, which includes: first, determining the preset parameters of the coding unit in the current frame; then, performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit; then, fusing the first prediction information and the reconstructed image information of the reference unit according to the preset parameters to obtain second prediction information of the coding unit; wherein the reconstructed image information of the reference unit includes the target reconstructed image of the reference unit or the feature information of the target reconstructed image of the reference unit, and the reference unit is a unit in the reference frame corresponding to the current frame; thereafter, determining the residual information of the coding unit based on the second prediction information and the original image of the coding unit; then encoding the residual information, as well as the encoding preset parameters.

[0073] After obtaining the first prediction information through inter-frame prediction, the present application fuses the reconstructed image information of the reference unit and the first prediction information according to preset parameters to generate the final prediction information (i.e., the second prediction information); that is, according to the fusion rule (or fusion strategy or fusion method) corresponding to the motion complexity of the image of the coding unit relative to the image of the reference unit, the reconstructed image information of the reference unit and the first prediction information are fused to generate the final prediction information. In this way, for coding units whose image motion complexity is low relative to the reference unit, the accuracy of inter-frame prediction can be improved to a certain extent, and the error of inter-frame prediction can be reduced, so that the error of the coding unit can be reduced to a certain extent, and then the error of the current frame can be reduced to a certain extent, and the compression performance of the current frame can be improved; thereby, the cumulative error of subsequent other frames can be reduced to a certain extent, the compression performance of subsequent other frames can be improved, and temporal stability can be guaranteed. For coding units whose image motion complexity is high relative to the reference unit, the accuracy of inter-frame prediction can be guaranteed, and then the quality of the target reconstructed image of the coding unit can be guaranteed, and thus the quality of the target reconstructed image of the current frame can be guaranteed.

[0074] According to the third aspect, or any implementation of the third aspect above, encoding the residual information includes: performing adaptive rate encoding on the residual information according to preset parameters.

[0075] According to the third aspect, or any implementation method of the third aspect above, inter-frame prediction is performed on the coding unit to obtain first prediction information of the coding unit, including: determining the first optical flow of the coding unit based on the original image of the coding unit and the target reconstructed image of the reference unit; determining the first prediction information based on the first optical flow and the reconstructed image information of the reference unit; the method also includes: performing adaptive bit rate encoding on the first optical flow according to preset parameters.

[0076] According to the third aspect, or any implementation of the third aspect above, the method also includes: fusing the reconstructed image information of the reference unit and the reconstructed image information of the coding unit according to preset parameters to obtain a target reconstructed image of the coding unit; wherein the reconstructed image information of the coding unit includes the initial reconstructed image of the coding unit or the feature information of the initial reconstructed image of the coding unit, and the initial reconstructed image of the coding unit and the feature information of the initial reconstructed image of the coding unit are reconstructed based on the second prediction information and the residual information.

[0077] According to the third aspect, or any implementation of the third aspect above, the method is applied to an AI video coding framework, the AI ​​video coding framework including: an inter-frame prediction module, a residual coding module, an entropy coding module, a preset parameter determination module, and a fusion module, wherein,

[0078] The preset parameter determination module is used to determine the preset parameters; output the preset parameters to the fusion module and the entropy coding module;

[0079] An inter-frame prediction module, configured to perform inter-frame prediction on the coding unit to obtain first prediction information; and output the first prediction information to the fusion module;

[0080] a fusion module, configured to fuse the first prediction information and the reconstructed image information of the reference unit according to preset parameters to obtain second prediction information; and output the second prediction information to the residual coding module;

[0081] A residual coding module, configured to determine residual information based on the second prediction information and the original image of the coding unit; and output the residual information to the entropy coding module;

[0082] The entropy coding module is used to encode residual information and preset parameters.

[0083] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0084] In a fourth aspect, the present application proposes a decoding method, which includes: receiving a code stream; obtaining decoded residual information of a coding unit in a current frame according to the code stream, and obtaining preset parameters of the coding unit according to the code stream; performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit; fusing the first prediction information and the reconstructed image information of the reference unit according to the preset parameters to obtain second prediction information of the coding unit; wherein the reconstructed image information of the reference unit includes a target reconstructed image of the reference unit or feature information of the target reconstructed image of the reference unit, and the reference unit is a unit in a reference frame corresponding to the current frame; reconstructing according to the decoded residual information and the second prediction information to determine the reconstructed image information of the coding unit; wherein the reconstructed image information of the coding unit includes an initial reconstructed image of the coding unit or feature information of the initial reconstructed image of the coding unit, and the feature information of the initial reconstructed image of the coding unit is used to obtain the initial reconstructed image of the coding unit.

[0085] Exemplarily, the initial reconstructed image of the coding unit may be obtained by performing feature transformation (or feature recovery) on feature information of the initial reconstructed image of the coding unit.

[0086] According to a fourth aspect, the method further comprises:

[0087] The reconstructed image information of the coding unit and the reconstructed image information of the reference unit are fused according to preset parameters to obtain a target reconstructed image of the coding unit.

[0088] According to the fourth aspect, or any implementation of the fourth aspect above, the code stream includes a preset parameter code stream and an optical stream code stream, and obtaining the preset parameters of the coding unit according to the code stream includes:

[0089] Performing entropy estimation based on at least one of the decoded residual information and the second optical flow of the coding unit to determine a probability distribution of a preset parameter; wherein the second optical flow is obtained based on the optical flow code stream;

[0090] The preset parameters are obtained by entropy decoding from the preset parameter bitstream according to the probability distribution of the preset parameters.

[0091] According to the fourth aspect, or any implementation of the fourth aspect above, the method is applied to an AI video decoding framework, and the AI ​​video decoding framework includes: a decoding module, an inter-frame prediction module, a reconstruction module, and a fusion module, wherein:

[0092] A decoding module, configured to obtain decoded residual information according to the bit stream and obtain preset parameters according to the bit stream; output the decoded residual information to the reconstruction module, and output the preset parameters to the fusion module;

[0093] An inter-frame prediction module, configured to perform inter-frame prediction on the coding unit to obtain first prediction information; and output the first prediction information to the fusion module;

[0094] a fusion module, configured to fuse the first prediction information and the reconstructed image information of the reference unit according to preset parameters to obtain second prediction information; and output the second prediction information to the reconstruction module;

[0095] The reconstruction module is used to reconstruct according to the decoded residual information and the second prediction information to determine the reconstructed image information of the coding unit.

[0096] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0097] In a fifth aspect, an embodiment of the present application provides a coding device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the coding device to execute the coding method in the first aspect or any possible implementation of the first aspect; or, to execute the coding method in the third aspect or any possible implementation of the third aspect.

[0098] The fifth aspect and any implementation of the fifth aspect respectively correspond to the first aspect and any implementation of the first aspect; or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and will not be repeated here.

[0099] In the sixth aspect, an embodiment of the present application provides a decoding device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the decoding device to execute the decoding method in the second aspect or any possible implementation of the second aspect; or, to execute the decoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0100] The sixth aspect and any implementation thereof respectively correspond to the second aspect and any implementation thereof; or respectively correspond to the fourth aspect and any implementation thereof. The technical effects corresponding to the sixth aspect and any implementation thereof may refer to the technical effects corresponding to the second aspect and any implementation thereof, or to the technical effects corresponding to the fourth aspect and any implementation thereof, and are not further elaborated here.

[0101] In a seventh aspect, an embodiment of the present application provides a decoding device comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the first aspect or any implementation of the first aspect, or the method in the third aspect or any implementation of the third aspect is executed.

[0102] The seventh aspect and any implementation of the seventh aspect respectively correspond to the first aspect and any implementation of the first aspect; or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and will not be repeated here.

[0103] In an eighth aspect, an embodiment of the present application provides a decoding device comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the second aspect or any implementation of the second aspect, or the method in the fourth aspect or any implementation of the fourth aspect is executed.

[0104] The eighth aspect and any implementation thereof respectively correspond to the second aspect and any implementation thereof; or respectively correspond to the fourth aspect and any implementation thereof. The technical effects corresponding to the eighth aspect and any implementation thereof may refer to the technical effects corresponding to the second aspect and any implementation thereof, or to the technical effects corresponding to the fourth aspect and any implementation thereof, and are not further elaborated here.

[0105] In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, it enables the computer or the processor to execute the encoding method in the first aspect or any possible implementation of the first aspect; or, execute the encoding method in the third aspect or any possible implementation of the third aspect.

[0106] The ninth aspect and any implementation thereof respectively correspond to the first aspect and any implementation thereof; or respectively correspond to the third aspect and any implementation thereof. The technical effects corresponding to the ninth aspect and any implementation thereof may refer to the technical effects corresponding to the first aspect and any implementation thereof, or to the technical effects corresponding to the third aspect and any implementation thereof, and are not further elaborated here.

[0107] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, it enables the computer or the processor to execute the encoding method in the second aspect or any possible implementation of the second aspect; or, execute the decoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0108] The tenth aspect and any implementation of the tenth aspect respectively correspond to the second aspect and any implementation of the second aspect; or respectively correspond to the fourth aspect and any implementation of the fourth aspect. The technical effects corresponding to the tenth aspect and any implementation of the tenth aspect may refer to the technical effects corresponding to the second aspect and any implementation of the second aspect, or refer to the technical effects corresponding to the fourth aspect and any implementation of the fourth aspect, and will not be repeated here.

[0109] In the eleventh aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect, or executes the encoding method in the third aspect or any possible implementation of the third aspect.

[0110] The eleventh aspect and any implementation thereof respectively correspond to the first aspect and any implementation thereof; or respectively correspond to the third aspect and any implementation thereof. The technical effects corresponding to the eleventh aspect and any implementation thereof may refer to the technical effects corresponding to the first aspect and any implementation thereof, or to the technical effects corresponding to the third aspect and any implementation thereof, and are not further elaborated here.

[0111] In the twelfth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the decoding method in the second aspect or any possible implementation of the second aspect, or executes the decoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0112] The twelfth aspect and any implementation of the twelfth aspect respectively correspond to the second aspect and any implementation of the second aspect; or respectively correspond to the fourth aspect and any implementation of the fourth aspect. The technical effects corresponding to the twelfth aspect and any implementation of the twelfth aspect may refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or refer to the technical effects corresponding to the above-mentioned fourth aspect and any implementation of the fourth aspect, and will not be repeated here.

[0113] In a thirteenth aspect, an embodiment of the present application provides an apparatus for storing a code stream, the apparatus comprising: a receiver and at least one storage medium, the receiver being configured to receive the code stream; the at least one storage medium being configured to store the code stream; the code stream being generated according to the first aspect and any one of the implementations of the first aspect, or according to the third aspect and any one of the implementations of the third aspect.

[0114] The thirteenth aspect and any implementation of the thirteenth aspect respectively correspond to the first aspect and any implementation of the first aspect; or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the thirteenth aspect and any implementation of the thirteenth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and will not be repeated here.

[0115] In a fourteenth aspect, an embodiment of the present application provides a device for transmitting a code stream, the device comprising: a transmitter and at least one storage medium, the at least one storage medium being used to store the code stream, the code stream being generated according to the first aspect and any one of the implementation methods of the first aspect, or generated according to the third aspect and any one of the implementation methods of the third aspect; the transmitter being used to obtain the code stream from the storage medium and send the code stream to the end-side device through the transmission medium.

[0116] The fourteenth aspect and any implementation of the fourteenth aspect respectively correspond to the first aspect and any implementation of the first aspect; or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the fourteenth aspect and any implementation of the fourteenth aspect may refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and will not be repeated here.

[0117] In a fifteenth aspect, an embodiment of the present application provides a system for distributing code streams, the system comprising: at least one storage medium for storing at least one code stream, the at least one code stream being generated according to the first aspect and any one of the implementations of the first aspect, or generated according to the third aspect and any one of the implementations of the third aspect; a streaming media device for obtaining a target code stream from the at least one storage medium and sending the target code stream to an end-side device, wherein the streaming media device comprises a content server or a content distribution server.

[0118] The fifteenth aspect and any implementation of the fifteenth aspect respectively correspond to the first aspect and any implementation of the first aspect; or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the fifteenth aspect and any implementation of the fifteenth aspect may refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] FIG1A is a schematic diagram illustrating an exemplary application framework;

[0120] FIG1B is a schematic diagram of an exemplary storage framework;

[0121] FIG1C is a schematic diagram illustrating an exemplary transmission framework;

[0122] FIG2A is a schematic diagram illustrating the structure of an AI video encoding framework;

[0123] FIG2B is a schematic diagram illustrating the structure of an AI video decoding framework;

[0124] FIG3A is a schematic diagram illustrating an exemplary AI video encoding process;

[0125] FIG3B is a schematic diagram illustrating an exemplary process of encoding preset parameters;

[0126] FIG3C is a schematic diagram illustrating an exemplary encoding process;

[0127] FIG3D is a schematic diagram illustrating an exemplary encoding process;

[0128] FIG4A is a schematic diagram illustrating an exemplary decoding process;

[0129] FIG4B is a schematic diagram illustrating an exemplary process of encoding preset parameters;

[0130] FIG4C is a schematic diagram illustrating exemplary reconstruction quality of a test video;

[0131] FIG5A is a schematic diagram illustrating an exemplary encoding process;

[0132] FIG5B is a schematic diagram illustrating an exemplary encoding process;

[0133] FIG6 is a schematic diagram illustrating an exemplary decoding process;

[0134] FIG7A is a schematic diagram illustrating an exemplary encoding process;

[0135] FIG7B is a schematic diagram illustrating an exemplary encoding process;

[0136] FIG8 is a schematic diagram illustrating an exemplary decoding process;

[0137] FIG9 is a schematic diagram showing an exemplary reconstruction quality of a test video;

[0138] FIG10A is a schematic diagram illustrating an exemplary encoding process;

[0139] FIG10B is a schematic diagram illustrating an exemplary encoding process;

[0140] FIG11 is a schematic diagram illustrating an exemplary decoding process;

[0141] FIG12 is a schematic structural diagram of an exemplary device. DETAILED DESCRIPTION

[0142] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0143] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0144] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0145] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0146] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0147] FIG1A is a schematic diagram illustrating an exemplary application framework.

[0148] Referring to Figure 1A , for example, a camera can capture video and obtain video data. Then, an AI video encoding framework can be used to perform AI video encoding on the video data to generate a bitstream. This bitstream can then be stored locally or transmitted to a remote device.

[0149] Continuing with reference to FIG1A , in one possible approach, after the code stream is stored locally, when a video needs to be played or edited, the AI ​​video decoding framework can be used to perform AI video decoding on the code stream to obtain reconstructed video data; subsequently, the reconstructed video data can be played or edited.

[0150] Continuing with reference to Figure 1A, in one possible approach, after the remote device receives the code stream, when it needs to play or edit a video, it can use the AI ​​video decoding framework to perform AI video decoding on the code stream to obtain reconstructed video data; then, the reconstructed video data can be played or edited.

[0151] FIG. 1B is a schematic diagram illustrating an exemplary storage framework.

[0152] 1B , exemplarily, the AI ​​video encoding framework may include an AI encoding module (also referred to as an AI encoding unit) and an entropy encoding module (also referred to as an entropy encoding unit), and the AI ​​video decoding framework may include an AI decoding module (also referred to as an AI decoding unit) and an entropy decoding module (also referred to as an entropy decoding unit); it should be understood that FIG1B is only an example of the present application, and the AI ​​video encoding framework and the AI ​​video decoding framework may include more modules than shown in FIG1B , and the present application does not limit this.

[0153] Exemplarily, the AI ​​encoding module and the AI ​​decoding module can be set in an embedded neural network processor (NPU) or a graphics processing unit (GPU). Exemplarily, the entropy encoding module and the entropy decoding module can be set in a central processing unit (CPU). Exemplarily, the file saving module and the file loading module can be set in a CPU.

[0154] Referring to Figure 1B, illustratively, after the camera captures the video data, it can be input into the AI ​​encoding module. Then, the AI ​​encoding module can perform the following processing for each frame image in the video data: perform prediction (including intra-frame prediction and inter-frame prediction, this application takes inter-frame prediction as an example to illustrate), determine the prediction information; and determine the residual information based on the prediction information and the original image, and determine the probability distribution of the residual information; then, input the residual information and the probability distribution of the residual information into the entropy coding module. Afterwards, the entropy coding module can entropy encode the residual information according to the probability distribution of the residual information to obtain a code stream; and input the code stream into the file saving module for saving to obtain a file.

[0155] Continuing with Figure 1B, for example, when a video needs to be played or edited, the file loading module can load the file and then input the code stream in the file into the entropy decoding module. Next, the entropy decoding module can obtain the probability distribution of the residual information from the AI ​​decoding module, and based on the probability distribution of the residual information, entropy decode the residual information from the code stream and output the residual information to the AI ​​decoding module. Afterwards, the AI ​​decoding module can perform inter-frame prediction to obtain prediction information; then, reconstruct based on the prediction information and residual information to obtain a reconstructed image; and then, reconstructed video data can be obtained.

[0156] FIG1C is a schematic diagram illustrating an exemplary transmission framework.

[0157] 1C , illustratively, the encoding process at the encoding end may be as follows:

[0158] For example, after the camera collects the video data, it can be input into the AI ​​encoding module. Then, the AI ​​encoding module can perform the following processing for each frame image in the video data: perform prediction (including intra-frame prediction and inter-frame prediction, this application takes inter-frame prediction as an example), determine the prediction information; and determine the residual information based on the prediction information and the original image, and determine the probability distribution of the residual information; then, the residual information and the probability distribution of the residual information are input into the entropy coding module. Afterwards, the entropy coding module can entropy encode the residual information according to the probability distribution of the residual information to obtain a code stream; and send the code stream to the server in the cloud.

[0159] Afterwards, the server can send the code stream to the decoding end. Among them, the server can be a single server or a server cluster, which is not limited in this application.

[0160] Continuing with FIG1C , illustratively, after the decoding end receives the code stream sent by the server, the decoding process may be as follows:

[0161] For example, the file loading module can load a file and then send the bitstream in the file to the entropy decoding module. The AI ​​decoding network can then perform inter-frame prediction to obtain prediction information. The entropy decoding module can then obtain the probability distribution of residual information from the AI ​​decoding module. Based on the probability distribution of residual information, it entropy decodes the residual information from the bitstream and outputs the residual information to the AI ​​decoding module. Subsequently, reconstruction is performed based on the prediction information and residual information to obtain a reconstructed image, thereby obtaining reconstructed video data.

[0162] It should be understood that the encoding end can also directly send the code stream to the decoding end without forwarding it through the server, and this application does not impose any restrictions on this.

[0163] Figure 2A is a schematic diagram of the structure of an exemplary AI video coding framework. In the embodiment of Figure 2A, an inter-frame coding process is shown.

[0164] 2A , exemplarily, the AI ​​encoding module may include a motion estimation module, an optical flow encoding module, a first entropy estimation module, an optical flow decoding module, a prediction module, a feature extraction module, a residual encoding module, a second entropy estimation module, a residual decoding module and a reconstruction module; it should be understood that the AI ​​encoding module may also include more or fewer modules than shown in FIG. 2A , and this application does not impose any restrictions on this.

[0165] 2A , exemplarily, the entropy coding module may include a first entropy coding module and a second entropy coding module.

[0166] 2A , exemplarily, the AI ​​video encoding framework may further include an entropy decoding module, and the entropy decoding module may include a first entropy decoding module and a second entropy decoding module.

[0167] It should be noted that the feature extraction module in the dotted box in FIG2A is an optional module.

[0168] It should be noted that the various modules included in the AI ​​video coding framework can be implemented only by a neural network, or by a combination of an algorithm and a neural network; or, some modules included in the AI ​​video coding framework can be implemented only by an algorithm, and this application does not impose any restrictions on this.

[0169] Exemplarily, the original image of the current frame (that is, the image of the current frame in the image sequence of the original video (also referred to as the video to be encoded)) can be divided into multiple coding units (it should be noted that the coding unit can refer to the minimum coding unit, and a coding unit can include at least one pixel); then each coding unit of the current frame is encoded.

[0170] 2A , inter-frame coding of a coding unit in an image frame of video data is used as an example for description. The AI ​​video coding process may be as follows:

[0171] Exemplarily, the original image of the coding unit and the reconstructed image of the reference unit in the reference frame corresponding to the current frame can be input into the motion estimation module. Then, the motion estimation module can perform motion estimation based on the original image of the coding unit and the reconstructed image of the reference unit to determine the optical flow of the coding unit (hereinafter referred to as the first optical flow); and output the first optical flow of the coding unit to the optical flow encoding module. The optical flow encoding module (also referred to as the optical flow feature extraction module) can encode the first optical flow of the coding unit (it should be noted that the encoding performed by the optical flow encoding module is essentially feature extraction), obtain the feature information of the first optical flow of the coding unit (also referred to as the compressed feature of the first optical flow of the coding unit) and output it to the first entropy estimation module.

[0172] Exemplarily, the first entropy estimation module can perform probability estimation on the feature information of the first optical flow of the coding unit, and determine the probability distribution of the feature information of the first optical flow of the coding unit (hereinafter referred to as the optical flow probability distribution of the coding unit); then, the feature information of the first optical flow of the coding unit and the optical flow probability distribution of the coding unit are output to the first entropy coding module.

[0173] Exemplarily, the first entropy coding module may perform entropy coding on the feature information of the first optical flow of the coding unit according to the optical flow probability distribution of the coding unit to obtain an optical flow code stream.

[0174] For example, on one hand, the optical flow code stream can be sent / saved (for example, the optical flow code stream is sent to a server or a decoding end). On the other hand, the optical flow code stream can be output to the first entropy decoding module; in addition, the first entropy estimation module can also output the optical flow probability distribution of the coding unit to the first entropy decoding module.

[0175] Exemplarily, the first entropy decoding module can entropy decode the optical flow code stream according to the optical flow probability distribution of the coding unit to obtain the feature information of the first optical flow after decoding of the coding unit (which can also be called the feature information of the second optical flow of the coding unit); and output the feature information of the second optical flow of the coding unit to the optical flow decoding module.

[0176] Exemplarily, the optical flow decoding module can decode the feature information of the second optical flow of the coding unit (it should be noted that the decoding performed by the optical flow decoding module is essentially feature recovery), obtain the second optical flow of the coding unit, and output the second optical flow of the coding unit to the prediction module. The second optical flow can be generally described as being determined by the optical flow processing module based on the first optical flow; the optical flow processing module may include an optical flow encoding module and an optical flow decoding module.

[0177] In one possible manner, when the AI ​​encoding module includes a feature extraction module, the reconstructed image of the reference unit can be input into the feature extraction module, and the feature extraction module extracts the features of the reconstructed image of the reference unit to obtain the feature information of the reconstructed image of the reference unit; and the feature information of the reconstructed image of the reference unit is output to the prediction module. The prediction module can perform prediction based on the second optical flow of the encoding unit and the feature information of the reconstructed image of the reference unit to obtain the prediction information of the encoding unit (hereinafter referred to as the first prediction information); and output the first prediction information of the encoding unit to the residual coding module. Correspondingly, the original image of the encoding unit can be input into the feature extraction module, and the feature extraction module extracts the features of the original image of the encoding unit to obtain the feature information of the original image of the encoding unit, and output the feature information of the original image of the encoding unit to the residual coding module. The residual coding module outputs the residual information of the encoding unit (hereinafter referred to as the first residual information of the encoding unit) to the second entropy estimation module based on the feature information of the original image of the encoding unit and the first prediction information of the encoding unit. Among them, the first residual information of the coding unit can be the first residual of the coding unit (wherein, the first residual can refer to the residual between the feature information of the original image of the coding unit and the prediction information of the coding unit, or it can be the feature information of the first residual of the coding unit (wherein, the feature information of the first residual can also be called the compression feature of the first residual, which can be obtained by encoding the first residual of the residual coding module (it should be noted that the encoding performed by the residual coding module is essentially feature extraction)). This application does not impose any restrictions on this. This application takes the first residual information of the coding unit as the feature information of the first residual of the coding unit as an example for explanation.

[0178] In one possible approach, when the AI ​​encoding module does not include a feature extraction module, the reconstructed image of the reference unit can be directly input into the prediction module. The prediction module performs prediction based on the second optical flow of the coding unit and the reconstructed image of the reference unit to obtain first prediction information for the coding unit; and outputs the first prediction information for the coding unit to the residual coding module. Alternatively, the original image of the coding unit can be directly input into the residual coding module; the residual coding module can output the first residual information of the coding unit to the second entropy estimation module based on the original image of the coding unit and the first prediction information of the coding unit.

[0179] Exemplarily, the second entropy estimation module can perform probability estimation on the first residual information of the coding unit to determine the probability distribution of the first residual information of the coding unit (hereinafter referred to as the residual probability distribution of the coding unit); then, the first residual information of the coding unit and the residual probability distribution of the coding unit are output to the second entropy coding module.

[0180] Exemplarily, the second entropy coding module may perform entropy coding on the first residual information of the coding unit according to the residual probability distribution of the coding unit to obtain a residual code stream.

[0181] For example, on the one hand, the residual bitstream can be sent / saved (for example, the residual bitstream can be sent to a server or a decoding end). On the other hand, the residual bitstream can be output to the second entropy decoding module; in addition, the second entropy estimation module can also output the residual probability distribution of the coding unit to the second entropy decoding module.

[0182] Exemplarily, the second entropy decoding module may entropy decode the residual code stream according to the residual probability distribution of the coding unit to obtain first residual information after entropy decoding of the coding unit. When the first residual information of the coding unit is characteristic information of the first residual of the coding unit, the first residual information after entropy decoding of the coding unit may be output to the residual decoding module.

[0183] Exemplarily, the residual decoding module can decode the first residual information after entropy decoding of the coding unit (it should be noted that the decoding performed by the residual decoding module is essentially feature recovery), obtain the second residual information of the coding unit (that is, the decoded residual information) and output the second residual information of the coding unit to the reconstruction module. It should be noted that when the first residual information of the coding unit is the first residual of the coding unit, the AI ​​video coding framework may not include a residual decoding module; in this case, the second residual information of the coding unit is the decoded first residual information obtained by entropy decoding of the coding unit. In addition, it can also be said that the second residual information of the coding unit is obtained by decoding the encoding result (that is, the residual code stream) of the first residual information of the coding unit; when the first residual information of the coding unit is the feature information of the first residual of the coding unit, the decoding in this step includes entropy decoding and feature recovery; when the first residual information of the coding unit is the first residual of the coding unit, the decoding in this step is entropy decoding.

[0184] Exemplarily, the prediction module may also output the first prediction information of the coding unit to the reconstruction module. When the AI ​​coding module includes a feature extraction module, the reconstruction module may first reconstruct based on the second residual information of the coding unit and the first prediction information of the coding unit to obtain feature information of the reconstructed image of the coding unit; then, the reconstruction module may perform feature transformation on the feature information of the reconstructed image of the coding unit to obtain the reconstructed image of the coding unit. When the AI ​​coding module does not include a feature extraction module, the reconstruction module may reconstruct based on the second residual information of the coding unit and the first prediction information of the coding unit to obtain the reconstructed image of the coding unit.

[0185] Optionally, the present application may also encode information for determining the optical flow probability distribution of the coding unit and send it to the decoding end; so that the decoding end determines the optical flow probability distribution to obtain the second optical flow of the coding unit; and encode information for determining the residual probability distribution of the coding unit and send it to the decoding end; so that the decoding end determines the residual probability distribution to obtain the second residual information of the coding unit.

[0186] FIG2B is a schematic diagram showing the structure of an exemplary AI video decoding framework.

[0187] 2B , exemplarily, the AI ​​decoding module may include: a first entropy estimation module, a second entropy estimation module, an optical flow decoding module, a prediction module, a feature extraction module, a residual decoding module and a reconstruction module; it should be understood that the AI ​​encoding module may also include more or fewer modules than shown in FIG. 2B , and this application does not impose any restrictions on this.

[0188] 2B , exemplarily, the entropy decoding module may include a first entropy decoding module and a second entropy encoding module.

[0189] It should be noted that the feature extraction module in the dotted box in FIG2B is an optional module.

[0190] It should be noted that the various modules included in the AI ​​video decoding framework can be implemented only by a neural network, or by a combination of an algorithm and a neural network; or, some modules included in the AI ​​video decoding framework can be implemented only by an algorithm, and this application does not impose any restrictions on this.

[0191] 2B , inter-frame decoding of a coding unit in an image frame of video data is used as an example for description. The AI ​​video decoding process may be as follows:

[0192] Exemplarily, the decoding end may receive a code stream, which may include an optical flow code stream and a residual code stream. When the code stream received by the decoding end does not include a code stream containing information for determining the optical flow probability distribution of the coding unit, the first entropy estimation module may perform probability estimation based on the reconstructed image of the reference unit, the first prediction information of the reference unit, or the feature information of the reconstructed image of the reference unit, etc., determine the optical flow probability distribution of the coding unit, and output it to the first entropy decoding module. When the received code stream includes a code stream containing information for determining the optical flow probability distribution of the coding unit, the first entropy decoding module may entropy decode the information for determining the optical flow probability distribution of the coding unit from the code stream and output it to the first entropy estimation module; then, the first entropy estimation module may perform probability estimation based on the information for determining the optical flow probability distribution of the coding unit, determine the optical flow probability distribution of the coding unit frame, and output it to the first entropy decoding module.

[0193] Exemplarily, the first entropy decoding module may perform entropy decoding on the optical flow code stream according to the optical flow probability distribution of the coding unit, obtain feature information of the second optical flow of the coding unit, and output the feature information to the optical flow decoding module.

[0194] Exemplarily, the optical flow decoding module may perform feature recovery on feature information of the second optical flow of the coding unit, obtain the second optical flow of the coding unit, and output the second optical flow of the coding unit to the prediction module.

[0195] Exemplarily, when the received bitstream does not include information for determining the residual probability distribution of the coding unit, the second entropy estimation module may perform probability estimation based on the reconstructed image of the reference unit, the first prediction information of the reference unit, or feature information of the reconstructed image of the reference unit, etc., determine the residual probability distribution of the coding unit, and output the information to the second entropy decoding module. When the received bitstream includes information for determining the residual probability distribution of the coding unit, the second entropy decoding module may entropy decode the information for determining the residual probability distribution of the coding unit from the bitstream; then, the second entropy estimation module may perform probability estimation based on the information for determining the residual probability distribution of the coding unit, determine the residual probability distribution of the coding unit, and output the information to the second entropy decoding module.

[0196] Exemplarily, the second entropy decoding module can perform entropy decoding on the code stream of information used to determine the residual probability distribution, obtain information used to determine the residual probability distribution of the coding unit, and output the information used to determine the residual probability distribution of the coding unit to the second entropy estimation module; then, the second entropy estimation module performs probability estimation based on the information for determining the residual probability distribution of the coding unit, determines the residual probability distribution of the coding unit and outputs it to the second entropy decoding module.

[0197] Exemplarily, the second entropy decoding module may perform entropy decoding on the residual code stream according to the residual probability distribution of the coding unit, obtain the first residual information after entropy decoding of the coding unit, and output it to the residual decoding module.

[0198] Exemplarily, the residual decoding module may perform feature recovery on the first residual information of the coding unit after entropy decoding, obtain the second residual information of the coding unit, and output the second residual information of the coding unit to the reconstruction module.

[0199] In one possible approach, when the AI ​​decoding module includes a feature extraction module, the reconstructed image of the reference unit can be input into the feature extraction module, and the feature extraction module extracts the features of the reconstructed image of the reference unit to obtain feature information of the reconstructed image of the reference unit; and the feature information of the reconstructed image of the reference unit is output to the prediction module. The prediction module can perform prediction based on the second optical flow of the coding unit and the feature information of the reconstructed image of the reference unit to obtain first prediction information of the coding unit; and output the first prediction information of the coding unit to the reconstruction module. Then, the reconstruction module can first reconstruct based on the second residual information of the coding unit and the first prediction information of the coding unit to obtain feature information of the reconstructed image of the coding unit; thereafter, the reconstruction module can perform feature transformation (such as feature recovery) on the feature information of the reconstructed image of the coding unit to obtain the reconstructed image of the coding unit.

[0200] In one possible approach, when the AI ​​decoding module does not include a feature extraction module, the reconstructed image of the reference unit can be directly input into the prediction module. The prediction module performs prediction based on the second optical flow of the coding unit and the reconstructed image of the reference unit to obtain first prediction information for the coding unit. The first prediction information for the coding unit is then output to the reconstruction module. The reconstruction module then reconstructs the coding unit based on the second residual information and the first prediction information to obtain a reconstructed image for the coding unit.

[0201] It should be understood that the present invention does not limit the names of the modules included in the AI ​​video encoding framework shown in Figure 2A, and the names of the modules included in the AI ​​video decoding framework shown in Figure 2B; and the modules included in the AI ​​video encoding framework can also be called units; and the modules included in the AI ​​video decoding framework can also be called units.

[0202] It should be noted that in one possible approach, the residual code stream and the optical stream code stream are two code streams. In one possible approach, the residual code stream and the optical stream code stream can be two parts of the same code stream, that is, the same code stream carries the residual information and the first optical stream.

[0203] The following is an explanation of the AI ​​video encoding and decoding process based on Figure 2A.

[0204] In the embodiments of FIG. 3A and FIG. 4A described below, based on FIG. 2A , the reconstructed image of the coding unit is post-processed to reduce the accumulated error.

[0205] Figure 3A is a schematic diagram illustrating an exemplary AI video encoding process. In Figure 3A, an example of encoding a coding unit of a frame of an original video is used for illustration.

[0206] S301 , performing inter-frame prediction on a coding unit in a current frame to obtain first prediction information of the coding unit.

[0207] Exemplarily, S301 may be performed by an inter-frame prediction module in the AI ​​video coding framework, wherein the inter-frame prediction module may include the motion estimation module, the optical flow processing module (including the optical flow encoding module and the optical flow decoding module), the first entropy estimation module, and the prediction module in FIG3A . The specific process of performing inter-frame prediction on the coding unit can be referred to the description in the embodiment of FIG2A and will not be repeated here.

[0208] Furthermore, the motion estimation module may be a motion estimation network.

[0209] S302: Determine residual information of the coding unit according to the first prediction information and the original image of the coding unit.

[0210] Exemplarily, S302 can be performed by the residual coding module in the AI ​​video coding framework. For details, please refer to the description of the embodiment of Figure 2A above, which will not be repeated here. It should be noted that the residual information of the coding unit determined by S302 corresponds to the first residual information of the coding unit in the embodiment of Figure 2A.

[0211] S303: Encode residual information of the coding unit.

[0212] Exemplarily, S303 may be performed by the second entropy coding module to obtain a residual bitstream; for details, please refer to the description of the embodiment of FIG. 2A above, which will not be repeated here. It should be noted that the residual information of the coding unit in S303 corresponds to the first residual information of the coding unit in the embodiment of FIG. 2A.

[0213] S304: Determine preset parameters of the encoding unit.

[0214] Exemplarily, the AI ​​video encoding framework may further include a preset parameter determination module for executing S304.

[0215] Exemplarily, preset parameters of the coding unit can be determined based on the image of the coding unit and the image of the reference unit; wherein the preset parameters are used to describe the motion complexity of the image of the coding unit relative to the image of the reference unit. Exemplarily, the image of the coding unit includes the original image of the coding unit and / or the initial reconstructed image of the coding unit, and the image of the reference unit includes the original image of the reference unit in the reference frame and / or the target reconstructed image of the reference unit. The initial reconstructed image of the coding unit may refer to the reconstructed image output by the reconstruction module after the coding unit is encoded according to the encoding process in the embodiment of Figure 2A (that is, the reconstructed image obtained by reconstructing according to the second residual information of the coding unit and the first prediction information of the coding unit).

[0216] Exemplarily, motion complexity may refer to difference or similarity; that is, the motion complexity of the image of the coding unit relative to the image of the reference unit may refer to the similarity between the image of the coding unit and the image of the reference unit, or, may refer to the difference between the image of the coding unit and the image of the reference unit.

[0217] In one possible approach, the similarity between the image of the coding unit and the image of the reference unit can be determined, and a preset parameter for the coding unit can be determined based on the similarity. When the similarity is greater than or equal to a similarity threshold, it indicates that the image of the coding unit and the image of the reference unit are highly similar. In this case, it can be determined that the motion complexity of the image of the coding unit relative to the image of the reference unit is low, and the value of the preset parameter can be set to a first preset value; that is, the first preset value can be used to indicate that the motion complexity of the image of the coding unit relative to the image of the reference unit is low. When the similarity is less than the similarity threshold, it indicates that the image of the coding unit and the image of the reference unit are relatively similar. In this case, it can be determined that the motion complexity of the image of the coding unit relative to the image of the reference unit is high, and the value of the preset parameter can be set to a second preset value; that is, the second preset value can be used to indicate that the motion complexity of the image of the coding unit relative to the image of the reference unit is high. The similarity threshold can be set as required. Assuming that the similarity when two images are exactly the same is 1, the similarity threshold can be a decimal less than 1 and greater than 0.

[0218] It should be noted that the first preset value and the second preset value are different, the first preset value may be greater than the second preset value, or the first preset value may be less than the second preset value; it can be set specifically as required, and this application does not impose any restrictions on this. In one possible way, the first preset value and the second preset value may be numbers between 0 and 1 (the values ​​of the first preset value and the second preset value may both be 0 or 1). Optionally, the sum of the first preset value and the second preset value may be 1. For example, the first preset value is 1, and the second preset value is 0; or, the first preset value is 0, and the second preset value is 1. For another example, the first preset value is 0.8, and the second preset value is 0.2; or, the first preset value is 0.2, and the second preset value is 0.8.

[0219] Exemplary methods for determining similarity include various methods, such as calculating an error and determining similarity based on the error; or using a similarity model to determine similarity calculation, etc., which are not limited in this application. This application uses calculating an error to determine similarity as an example for explanation.

[0220] In one possible approach, an error between the image of the coding unit and the image of the reference unit may be calculated, and based on the error between the image of the coding unit and the image of the reference unit, a similarity between the image of the coding unit and the image of the reference unit may be determined. Assuming that the similarity is 1 when two images are exactly the same, the difference between 1 and the error between the image of the coding unit and the image of the reference unit may be used as the similarity.

[0221] For example, the value of the preset parameter can be determined according to the following formula (1):

[0222] In formula (1), mask represents a preset parameter, error() represents a calculated error, X1 represents the image of the coding unit, X2 represents the image of the reference unit, and ε is an error threshold, which can be the difference between 1 and the similarity threshold.

[0223] That is, in formula (1), when the error between the image of the coding unit and the image of the reference unit is less than or equal to the error threshold, it is equivalent to that the similarity between the image of the coding unit and the image of the reference unit is greater than or equal to the similarity threshold; in this case, the value of the preset parameter is set to 1. The else in formula (1) indicates that the error between the image of the coding unit and the image of the reference unit is greater than the error threshold, which is equivalent to that the similarity between the image of the coding unit and the image of the reference unit is less than the similarity threshold; in this case, the value of the preset parameter is set to 0.

[0224] For example, X1 in formula (1) is the initial reconstructed image of the coding unit, and X2 is the target reconstructed image of the reference unit.

[0225] For example, X1 in formula (1) is the original image of the coding unit, and X2 is the original image of the reference unit.

[0226] For example, X1 in formula (1) is the initial reconstructed image of the coding unit, and X2 is the original image of the reference unit.

[0227] For example, X1 in formula (1) is the original image of the coding unit, and X2 is the target reconstructed image of the reference unit.

[0228] In one possible approach, a third optical flow of the coding unit may be obtained; second prediction information of the coding unit may be determined based on the third optical flow of the coding unit and a target reconstructed image of the reference unit; and similarity between the image of the coding unit and the image of the reference unit may be determined based on an error between the second prediction information of the coding unit and the image of the coding unit.

[0229] In a possible implementation, the initial reconstructed image of the coding unit and the target reconstructed image of the reference unit can be input into the motion estimation module, which performs motion estimation and outputs the third optical flow of the coding unit; refer to formula (2): flow = FLOW (X 12 ,X 22 ) (2)

[0230] In formula (2), flow represents the third optical flow of the coding unit, FLOW() represents motion estimation, and X 12 represents the initial reconstructed image of the coding unit, X 22 represents the target reconstructed image of the reference cell.

[0231] In one possible approach, the feature information of the first optical flow of the coding unit can be input into an optical flow decoding model (the optical flow decoding model is pre-trained and is different from the optical flow decoding module in FIG2A ), and the optical flow decoding model decodes the feature information of the first optical flow (i.e., recovers the feature) and outputs the third optical flow of the coding unit; refer to formula (3): flow = Decflow (feature information of the first optical flow of the coding unit) (3)

[0232] In formula (3), flow represents the third optical flow of the coding unit, and Decflow() represents decoding (or feature recovery).

[0233] Then, the second prediction information of the coding unit can be determined according to formula (4): F = wrap(X 22 ,flow,'neareast') (4)

[0234] In formula (4), F represents the second prediction information of the coding unit, X 22 It represents the target reconstructed image of the reference unit, flow represents the third optical flow of the coding unit, nearest represents the nearest neighbor interpolation method, and wrap() represents image transformation.

[0235] Afterwards, the preset parameters of the encoding unit can be determined by referring to formula (5):

[0236] In formula (5), mask represents the preset parameters of the coding unit, error() represents the calculation error, and X 11 represents the original image of the coding unit, F represents the second prediction information of the coding unit, ε is the error threshold, and the error threshold can be the difference between 1 and the similarity threshold.

[0237] That is, in formula (5), when the error between the second prediction information and the original image of the coding unit is less than or equal to the error threshold, it is equivalent to that the similarity between the second prediction information and the original image of the coding unit is greater than or equal to the similarity threshold; in this case, the value of the preset parameter is set to 1. The else in formula (5) indicates that the error between the second prediction information and the original image of the coding unit is greater than the error threshold, which is equivalent to that the similarity between the second prediction information and the original image of the coding unit is less than the similarity threshold. In this case, the value of the preset parameter is set to 0.

[0238] It should be understood that other methods may be used to calculate the error, and this application does not limit this.

[0239] S305: Encode the preset parameters of the encoding unit.

[0240] It should be noted that the present application does not limit the order of encoding residual information and encoding preset parameters, that is, the present application does not limit the execution order of S303 and S305.

[0241] Exemplarily, both the AI ​​video encoding framework and the AI ​​video encoding framework may also include a fusion module; wherein the fusion module may be an independent module or integrated into the reconstruction module, and this application does not impose any restrictions on this.

[0242] Exemplarily, after obtaining the preset parameters of the coding unit, on the one hand, the preset parameters of the coding unit can be input into a fusion module, which can fuse the reconstructed image information of the reference unit and the reconstructed image information of the coding unit according to the preset parameters of the coding unit to obtain a target reconstructed image of the coding unit. On the other hand, S305 can be executed to encode the preset parameters of the coding unit to obtain a preset parameter bitstream; the preset parameters of the coding unit can be transmitted to the decoding end and applied to the fusion module of the decoding end, so that the fusion module of the decoding end can also fuse the reconstructed image information of the reference unit and the reconstructed image information of the coding unit according to the preset parameters of the coding unit to obtain the target reconstructed image of the coding unit.

[0243] The reconstructed image information of the reference unit may include the target reconstructed image of the reference unit or the characteristic information of the target reconstructed image of the reference unit, and the reconstructed image information of the coding unit may include the initial reconstructed image of the coding unit or the characteristic information of the initial reconstructed image of the coding unit.

[0244] Exemplarily, the entropy coding module of the AI ​​video coding framework may further include a third entropy coding module and the AI ​​video coding framework may further include a third entropy estimation module, and the third entropy coding module may execute S305.

[0245] In one possible approach, the preset parameters of the coding unit can be directly encoded to obtain a preset parameter bitstream; in this way, the decoding end can decode the preset parameters of the current frame from the preset parameter bitstream. For example, the preset parameters of the coding unit can first be entropy estimated to obtain a probability distribution of the preset parameters; then, based on the probability distribution of the preset parameters, the preset parameters of the coding unit can be entropy encoded to obtain the preset parameter bitstream. Specifically, referring to Figure 3B, the probability distribution of the preset parameters of the coding unit can be determined by a third entropy estimation module based on the optical stream bitstream and / or the residual stream. It should be understood that other methods can also be used to encode the preset parameters of the coding unit, and this application does not limit this.

[0246] In one possible approach, feature extraction can be performed on the preset parameters of the coding unit to obtain feature information of the preset parameters of the coding unit. This feature information of the preset parameters of the coding unit can then be encoded to obtain a preset parameter bitstream. In this way, the decoding end can perform feature recovery based on the information decoded from the preset parameter bitstream to obtain the preset parameters of the current frame. The method for extracting the feature information of the preset parameters of the coding unit is similar to the method for encoding the preset parameters of the coding unit and will not be further described here.

[0247] Exemplarily, one way to fuse the reconstructed image information of the reference unit and the reconstructed image information of the coding unit to obtain the target reconstructed image of the coding unit may be to determine the weight of the reconstructed image information of the reference unit and the weight of the reconstructed image information of the coding unit according to preset parameters of the coding unit; and perform weighted calculation on the reconstructed image information of the reference unit and the reconstructed image information of the coding unit according to the weight of the reconstructed image information of the reference unit and the weight of the reconstructed image information of the coding unit to obtain the target reconstructed image of the coding unit.

[0248] In this case, if the first preset value is greater than the second preset value, then when the first preset value and the second preset value are numbers between 0 and 1 (the first preset value and the second preset value can both be 0 or 1), the value of the preset parameter of the coding unit can be used as the weight of the reconstructed image information of the reference unit, and the difference between 1 and the value of the preset parameter can be used as the weight of the reconstructed image information of the coding unit. When the first preset value and the second preset value are both numbers greater than 1, the first preset value and the second preset value can be transformed into numbers between 0 and 1, and then the value of the preset parameter after the transformation of the coding unit can be used as the weight of the reconstructed image information of the reference unit, and the difference between 1 and the value of the preset parameter after the transformation of the coding unit can be used as the weight of the reconstructed image information of the coding unit. At this time, the fusion can be performed with reference to the following formula (6): G = mask*Y1 +(1-mask)*Y2 (6)

[0249] If the second preset value is greater than the first preset value, the fusion can be performed according to the following formula (7): G = mask*Y2 + (1-mask)*Y1 (7)

[0250] In formula (6) and formula (7), G represents the target reconstructed image of the coding unit or the characteristic information of the target reconstructed image of the coding unit, Y1 is the target reconstructed image of the reference unit or the characteristic information of the target reconstructed image of the reference unit, and Y2 is the initial reconstructed image of the coding unit or the characteristic information of the initial reconstructed image of the coding unit.

[0251] It should be understood that when Y1 is the target reconstructed image of the reference unit, Y2 is the initial reconstructed image of the coding unit. When Y1 is the feature information of the target reconstructed image of the reference unit, Y2 is the feature information of the initial reconstructed image of the coding unit.

[0252] When the motion complexity of the image of the coding unit relative to the image of the reference unit is low, and the mask in formula (6) is 1, and the mask in formula (7) is 0, the reconstructed image information of the reference unit and the reconstructed image information of the coding unit are fused, and the reconstructed image information of the reference unit is copied to the coding unit to obtain the target reconstructed image of the coding unit. In this way, the errors introduced by inter-frame prediction and entropy coding of residual information can be compensated, the error of the current frame can be reduced, and the accumulated error of subsequent frames can be reduced, thereby ensuring temporal stability.

[0253] When the motion complexity of the image of the coding unit relative to the image of the reference unit is high, and the mask in formula (6) is 0, and the mask in formula (7) is 1, the target reconstructed image of the coding unit obtained by fusing the reconstructed image information of the reference unit and the reconstructed image information of the coding unit is essentially the initial reconstructed image of the coding unit; since the motion complexity of the image of the coding unit relative to the image of the reference unit is high, if the reconstructed image information of the reference unit is copied to the coding unit, the quality of the target reconstructed image of the coding unit is low; therefore, the reconstructed initial reconstructed image of the coding unit can be used as the initial reconstructed image of the coding unit; in this way, the quality of the reconstructed image of the coding unit can be guaranteed.

[0254] When the motion complexity of the image of the coding unit relative to the image of the reference unit is low, and the mask in formula (6) and formula (7) is a value other than 0 and 1 (that is, the first preset value and the second preset value are values ​​other than 0 and 1), in the target image of the coding unit obtained by fusing the reconstructed image information of the reference unit and the reconstructed image information of the coding unit, the reconstructed image information of the coding unit and the reconstructed image information of the reference unit both occupy a certain proportion; in this way, the error introduced by inter-frame prediction and entropy coding of residual information can be compensated to a certain extent, and the error of the current frame can be reduced; further, the accumulated error of subsequent frames can be reduced to a certain extent, ensuring temporal stability. In addition, the quality of the target reconstructed image of the coding unit can be guaranteed to a certain extent.

[0255] Exemplarily, one way to fuse the reconstructed image information of the reference unit and the reconstructed image information of the coding unit to obtain the target reconstructed image of the coding unit may be to input the preset parameters of the coding unit, the original image of the coding unit of the coding unit, and the reconstructed image information of the reference unit into a fusion model to obtain the target reconstructed image of the coding unit. In this case, there is no limit on the size of the first preset value and the second preset value; wherein, the fusion model is pre-trained and has the ability to generate high-quality information (such as high-quality images or high-quality feature information). In this way, the error of the coding unit of the current frame can also be reduced, thereby reducing the cumulative error of subsequent frames and ensuring temporal stability. It can also ensure the quality of the target reconstructed image of the coding unit to a certain extent.

[0256] It should be noted that when the image of the coding unit in S304 does not include the initial reconstructed image of the coding unit, this application does not limit the execution order of S304 and any of the above steps S301 to S303. When the image of the coding unit in S304 includes the initial reconstructed image of the coding unit, in this application, S304 is executed after S303.

[0257] Figure 3C is a schematic diagram illustrating an exemplary encoding process. Figure 3C uses the current frame as an example. In Figure 3C , the first preset value is 1, the second preset value is 0, and the reconstructed image information of the reference unit used for fusion is the target reconstructed image of the reference frame, while the reconstructed image information of the encoding unit used for fusion is the initial reconstructed image of the current frame.

[0258] Figure 3D is a schematic diagram illustrating an exemplary encoding process. Figure 3D uses the current frame as an example for illustration. In Figure 3D , the first preset value is 1, the second preset value is 0, and the reconstructed image information of the reference unit used for fusion is the feature information of the target reconstructed image of the reference frame, while the reconstructed image information of the encoding unit used for fusion is the feature information of the initial reconstructed image of the current frame.

[0259] 3C and 3D , for example, the encoding process in the dotted box with bold lines on the upper left corresponds to S301 to S303, and the description of FIG2A can be referred to, and will not be repeated here. The processing process in the dotted box with bold lines on the lower right corresponds to S304 to S305. In FIG3C and FIG3D , a black dot in the preset parameters of the current frame indicates that the value of the preset parameters of the corresponding encoding unit is 0, and a white dot indicates that the value of the preset parameters of the corresponding encoding unit is 1.

[0260] It should be noted that in one possible approach, the residual bitstream and the preset parameter bitstream are two bitstreams. In one possible approach, the residual bitstream and the preset parameter bitstream can be two parts of the same bitstream, that is, the same bitstream carries the residual information and the preset parameters.

[0261] Fig. 4A is a schematic diagram illustrating an exemplary decoding process. The decoding process in the embodiment of Fig. 4A corresponds to the encoding process in the embodiment of Fig. 3A.

[0262] S401, receiving a code stream.

[0263] Exemplarily, the code stream received by the decoding end may include an optical stream code stream, a residual code stream, and a preset parameter code stream.

[0264] S402: Obtain preset parameters of the coding unit in the current frame according to the code stream.

[0265] In one possible approach, the preset parameters of the coding unit may be directly decoded from the preset parameter bitstream.

[0266] For example, the encoding end determines the probability distribution of the preset parameters of the coding unit based on the optical stream code stream and / or the residual code stream; then, based on the probability distribution of the preset parameters, entropy encoding is performed on the preset parameters of the coding unit to obtain a preset parameter code stream; with reference to Figure 4B, the decoding end can determine the probability distribution of the preset parameters of the coding unit based on the optical stream code stream and / or the residual code stream; then, entropy decoding can be performed on the preset parameter code stream based on the probability distribution of the preset parameters to obtain the preset parameters of the coding unit.

[0267] In one possible manner, information decoded from the preset parameter bitstream may be processed to obtain the preset parameters of the coding unit.

[0268] It should be understood that when S402 requires using the residual code stream to determine the probability distribution of the preset parameters of the coding unit, S403 can be performed first, and then S402. And when S402 requires using the optical flow code stream to determine the probability distribution of the preset parameters of the coding unit, the second optical flow of the coding unit can be obtained based on the optical flow code stream before S402 is performed.

[0269] S403: Obtain decoded residual information of the coding unit according to the code stream.

[0270] For example, S403 can refer to the description of the decoding process in FIG2B , which will not be repeated here. It should be noted that the decoded residual information of the coding unit in S403 corresponds to the second residual information of the coding unit in the embodiment of FIG2B .

[0271] It should be noted that S402 and S403 can be executed by the decoding module in the AI ​​video decoding framework, wherein the decoding module may include the second entropy estimation module, the second entropy decoding module and the residual decoding module in Figure 2B (these three modules are used to execute S403); and also include a third entropy decoding module (the third entropy decoding module belongs to the entropy decoding module in the AI ​​video decoding framework, and the third entropy decoding module is used to execute S402).

[0272] S404: Perform inter-frame prediction on the coding unit to obtain first prediction information of the coding unit.

[0273] For example, S404 can refer to the description of the inter-frame prediction process in FIG2B above, which will not be repeated here. It should be noted that S404 can be performed by the inter-frame prediction module in the AI ​​video decoding framework, and the inter-frame prediction module can include the optical flow decoding module, the first entropy estimation module, and the prediction module in FIG2B.

[0274] It should be noted that the present application does not limit the execution order of S404 and S403. When S403 needs to be implemented based on the first prediction information of the coding unit, S404 can be executed first and then S403.

[0275] S405 , reconstructing according to the decoded residual information of the coding unit and the first prediction information to determine reconstructed image information of the coding unit.

[0276] Exemplarily, S405 may refer to the description of the reconstruction process performed by the reconstruction module in FIG. 2B , which will not be described in detail here.

[0277] S406 , fusing the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to preset parameters to obtain a target reconstructed image of the coding unit.

[0278] Illustratively, the fusion process of S406 is similar to the fusion process in the embodiment of FIG. 3A , and is not described again here.

[0279] The following test set is used to test the AI ​​video encoding and decoding method of this application, the traditional encoding and decoding method of the prior art (such as H266) and the AI ​​video encoding and decoding method of the prior art, and compare the effects of the AI ​​video encoding and decoding method of this application and the AI ​​video encoding and decoding method of the prior art.

[0280] For example, the test set includes 5: two 4k test sets (of which one 4K (A1) test set has a complex motion type and one 4K test set (A2) has a simple motion type), a 1080p test set, an 832*480 test set (target object motion), and a 770p test set (conference video); the peak signal-to-noise ratio (PSNR) of the AI ​​video encoding and decoding method of the present application relative to H266, and the PSNR of the AI ​​video encoding and decoding method of the prior art relative to H266 can be shown in Table 1 below:

[0281] Table 1

[0282] Referring to Table 1, Y-PSNR represents the PSNR of the Y component, U-PSNR represents the PSNR of the U component, and V-PSNR represents the PSNR of the V component.

[0283] It should be noted that the video data used for testing in the test set contains images that are YUV420 images, in which Y component: U component: V component = 8:1:1; therefore, in Table 1, the focus is on Y-PSNR. For the above five test sets, the PSNR of the AI ​​video encoding and decoding method of the present application is better than the PSNR of the AI ​​video encoding and decoding method of the prior art; in particular, for the 720p test set, the PSNR of the AI ​​video encoding and decoding method of the present application is improved by 7.54% relative to the PSNR of the AI ​​video encoding and decoding method of the prior art. In other words, the encoding and decoding method of the present application can reduce the cumulative error of videos with low motion complexity (that is, the motion complexity of the image of the encoding unit relative to the image of the reference unit is low complexity) while ensuring or improving to a certain extent the rate-distortion (RD) performance of video data with high motion complexity (that is, the motion complexity of the image of the encoding unit relative to the image of the reference unit is high complexity).

[0284] Figure 4C is a schematic diagram illustrating the reconstruction quality of a test video. In Figure 4C, the horizontal axis represents the frame number of the image frame, and the vertical axis represents the Y-PSNR. Figure 4B shows the Y-PSNR of each frame in two image groups (each image group includes 50 frames).

[0285] Referring to Figure 4C, curve 1 is the Y-PSNR of the reconstructed image obtained by encoding using the AI ​​video encoding and decoding method of the present application; curve 2 is the Y-PSNR of the reconstructed image obtained by encoding using the AI ​​video encoding and decoding method of the prior art; by comparing curve 1 and curve 2, it can be seen that, compared with the prior art, the quality of the reconstructed image of the present application is more stable, and the cumulative error is smaller, that is, the reconstructed image quality is high.

[0286] Exemplarily, based on the embodiment of FIG. 3A , adaptive rate encoding may be performed on the residual information of the coding unit according to preset parameters of the coding unit to reduce the bit rate overhead.

[0287] FIG5A is a schematic diagram illustrating an exemplary encoding process.

[0288] S501 , performing inter-frame prediction on a coding unit in a current frame to obtain first prediction information of the coding unit.

[0289] S502 : Determine residual information of the coding unit according to the first prediction information and the original image of the coding unit.

[0290] S503: Determine preset parameters of the encoding unit.

[0291] For example, S501 to S503 may refer to the description of S301, S302 and S304 above, which will not be repeated here. It should be noted that the residual information of the coding unit in S502 corresponds to the first residual information of the coding unit in the embodiment of FIG2A.

[0292] S504: Perform adaptive rate coding on the residual information of the coding unit according to preset parameters.

[0293] Exemplarily, when the value of the preset parameter of the coding unit is a first preset value, the residual information of the coding unit can be encoded at a low bit rate to obtain a residual code stream; when the value of the preset parameter of the coding unit is a second preset value, the residual information of the coding unit can be encoded at a high bit rate to obtain a residual code stream. In other words, the bit rate of the residual information encoding is adaptively adjusted according to the preset parameters of the coding unit. Since the reconstructed image information of the reference unit and the reconstructed image information of the coding unit are fused during the reconstruction process to generate the target reconstructed image of the coding unit, even if the residual information of the current frame is encoded at a low bit rate, the quality of the target reconstructed image of the coding unit can be guaranteed. Furthermore, the embodiment of FIG5A can reduce the bit rate overhead to a certain extent while maintaining the same quality.

[0294] Specifically, when the value of the preset parameter is a first preset value, the residual information is encoded according to the first bit rate parameter; when the value of the preset parameter is a second preset value, the residual information is encoded according to the second bit rate parameter; wherein the target bit rate of the code stream obtained by encoding the residual information according to the second bit rate parameter is greater than the target bit rate of the code stream obtained by encoding the residual information according to the first bit rate parameter.

[0295] S505: Encode the preset parameters of the encoding unit.

[0296] FIG5B is a schematic diagram illustrating an exemplary encoding process. FIG5B is based on FIG3B ; the second entropy encoding module can perform adaptive rate encoding on the residual information of the coding unit based on the residual probability distribution and preset parameters of the coding unit to obtain a residual bitstream.

[0297] Figure 6 is a schematic diagram illustrating an exemplary decoding process. The decoding process in the embodiment of Figure 6 corresponds to the encoding process in the embodiment of Figure 5A.

[0298] S601, receiving a code stream.

[0299] S602: Obtain preset parameters of coding units in the current frame according to the code stream.

[0300] For example, S601 to S602 may refer to the description of S401 to S402 above, which will not be repeated here.

[0301] S603 : Perform adaptive rate decoding on the residual bit stream according to preset parameters of the encoding unit.

[0302] Specifically, when the value of the preset parameter of the coding unit is a first preset value, the residual code stream is decoded according to the first code rate parameter; when the value of the preset parameter of the coding unit is a second preset value, the residual code stream is decoded according to the second code rate parameter.

[0303] S604 , determining decoded residual information of the coding unit according to information obtained by adaptively decoding the residual bitstream.

[0304] For example, information obtained by adaptively decoding the residual code stream may be input into a residual decoding module, which processes the information and outputs the decoded residual information of the coding unit.

[0305] S605 , performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit.

[0306] S606 , reconstructing according to the decoded residual information of the coding unit and the first prediction information to determine reconstructed image information of the coding unit.

[0307] S607 , fusing the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to preset parameters to obtain a target reconstructed image of the coding unit.

[0308] For example, S605 to S607 may refer to the description of S404 to S406 above, which will not be repeated here.

[0309] It should be noted that the decoded residual information of the coding unit in the embodiment of FIG. 6 corresponds to the second residual information of the coding unit in the embodiment of FIG. 2B .

[0310] Exemplarily, based on FIG3A , adaptive rate encoding may be performed on the first optical flow of the coding unit according to preset parameters of the coding unit to reduce the bit rate overhead.

[0311] FIG7A is a schematic diagram illustrating an exemplary encoding process.

[0312] S701 , performing motion estimation on a coding unit in a current frame to obtain a first optical flow of the coding unit.

[0313] S702 : Predict the coding unit in the current frame according to the first optical flow of the coding unit to obtain first prediction information of the coding unit.

[0314] S703 : Determine residual information of the coding unit according to the first prediction information and the original image of the coding unit.

[0315] S704: Encode the residual data of the coding unit.

[0316] S705: Determine preset parameters of the encoding unit.

[0317] For example, S701 to S705 may refer to the description of S301 to S304 above, which will not be repeated here.

[0318] S706 , performing adaptive rate encoding on the first optical flow of the encoding unit according to preset parameters of the encoding unit.

[0319] Exemplarily, the performing of adaptive rate coding on the first optical flow of the coding unit in S706 may be performing adaptive rate coding on feature information of the first optical flow of the coding unit.

[0320] Exemplarily, when the value of the preset parameter of the encoding unit is a first preset value, the characteristic information of the first optical flow of the encoding unit can be encoded at a low bit rate to obtain an optical flow code stream; when the value of the preset parameter of the encoding unit is a second preset value, the characteristic information of the first optical flow of the encoding unit can be encoded at a high bit rate to obtain an optical flow code stream. In other words, the bit rate of the first optical flow encoding is adaptively adjusted according to the preset parameters of the encoding unit. Since the reconstructed image information of the reference unit and the reconstructed image information of the encoding unit are fused during the reconstruction process to generate the target reconstructed image of the encoding unit; therefore, even if the characteristic information of the first optical flow of the current frame is encoded at a low bit rate, the quality of the target reconstructed image of the encoding unit can be guaranteed; and further, the embodiment of FIG7A can reduce the bit rate overhead to a certain extent while maintaining the same quality.

[0321] Specifically, when the value of the preset parameter is a first preset value, the characteristic information of the first optical flow is encoded according to the third code rate parameter; when the value of the preset parameter is a second preset value, the characteristic information of the first optical flow is encoded according to the fourth code rate parameter; wherein, the target code rate of the code stream obtained by encoding the first optical flow according to the fourth code rate parameter is greater than the target code rate of the code stream obtained by encoding the first optical flow according to the third code rate parameter.

[0322] S707, encode the preset parameters of the encoding unit.

[0323] It should be noted that the residual information of the coding unit in the embodiment of FIG. 7A corresponds to the first residual information of the coding unit in FIG. 2A .

[0324] FIG7B is a schematic diagram illustrating an exemplary encoding process. FIG7B is based on FIG3B ; the first entropy encoding module can adaptively rate-encode the feature information of the first optical flow of the encoding unit according to the optical flow probability distribution and preset parameters of the encoding unit to obtain an optical flow code stream.

[0325] It should be noted that the present application does not limit the order of encoding the first optical flow, encoding the residual information, and encoding the preset parameters, that is, the present application does not limit the execution order of S704, S706, and S707.

[0326] It should be noted that, in one possible manner, the optical stream code stream, the residual code stream, and the preset parameter code stream are three code streams. In one possible manner, the optical stream code stream, the residual code stream, and the preset parameter code stream can be three parts of the same code stream, that is, the same code stream carries residual information, preset parameters, and the first optical stream. In one possible manner, the residual code stream and the preset parameter code stream can be two parts of the same code stream, and the optical stream code stream is another code stream. In one possible manner, the residual code stream and the optical stream code stream can be two parts of the same code stream, and the preset parameter code stream is another code stream. In one possible manner, the preset parameter code stream and the optical stream code stream can be two parts of the same code stream, and the residual code stream is another code stream; this application does not impose any restrictions on this.

[0327] FIG8 is a schematic diagram illustrating an exemplary decoding process. The decoding process in the embodiment of FIG8 corresponds to the encoding process in the embodiment of FIG7A.

[0328] S801, receiving a code stream.

[0329] S802: Obtain preset parameters of the coding unit in the current frame according to the code stream.

[0330] S803: Obtain decoded residual information of the coding unit according to the code stream.

[0331] For example, S801 to S803 may refer to the description of S401 to S403 above, which will not be repeated here.

[0332] S804: Perform adaptive bit rate decoding on the optical stream according to preset parameters of the encoding unit.

[0333] Specifically, when the value of the preset parameter of the encoding unit is the first preset value, the optical stream is decoded according to the third rate parameter; when the value of the preset parameter of the encoding unit is the second preset value, the optical stream is decoded according to the fourth rate parameter.

[0334] S805 , determining a second optical flow of the coding unit according to information obtained by adaptively decoding the optical flow code stream.

[0335] Exemplarily, information obtained by adaptively decoding the optical flow code stream may be input into an optical flow decoding module, which processes the information and outputs the second optical flow of the encoding unit.

[0336] S806 , predict the coding unit according to the second optical flow of the coding unit to obtain first prediction information of the coding unit.

[0337] S807 , reconstructing according to the decoded residual information of the coding unit and the first prediction information to determine reconstructed image information of the coding unit.

[0338] S808 , fusing the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to preset parameters to obtain a target reconstructed image of the coding unit.

[0339] For example, S805 to S808 may refer to the description of S404 to S406 above, which will not be repeated here.

[0340] It should be noted that the decoded residual information of the coding unit in the embodiment of FIG. 8 corresponds to the second residual information of the coding unit in the embodiment of FIG. 2B .

[0341] It should be understood that, based on FIG3A , adaptive rate encoding can also be performed on the first optical flow of the encoding unit and adaptive rate encoding can be performed on the residual information of the encoding unit according to the preset parameters of the encoding unit; in this way, the rate overhead can be further reduced; specifically, this can be combined with the description in the embodiment of FIG5A and the embodiment of FIG7A, which will not be repeated here. And based on FIG4 , adaptive rate decoding can be performed on the optical flow code stream and adaptive rate decoding can be performed on the residual code stream according to the preset parameters of the encoding unit, which can be combined with the description in the embodiment of FIG6 and the embodiment of FIG8, which will not be repeated here. The reconstruction quality of the reconstructed video data obtained by using the video encoding method combining the embodiment of FIG5A and the embodiment of FIG7A, and the video decoding method combining the embodiment of FIG6 and the embodiment of FIG8 can be referred to FIG9 .

[0342] Figure 9 is a schematic diagram illustrating the reconstruction quality of a test video. In Figure 9, the horizontal axis represents the frame number of the image frame, and the vertical axis represents the Y-PSNR. Figure 9 shows the Y-PSNR of each frame in two image groups (each image group includes 50 frames).

[0343] Referring to Figure 9, curve 1 is the Y-PSNR of the reconstructed image obtained by encoding using the AI ​​video encoding and decoding method of the present application; curve 2 is the Y-PSNR of the reconstructed image obtained by encoding using the AI ​​video encoding and decoding method of the prior art; by comparing curve 1 and curve 2, it can be seen that compared with the prior art, the quality of the reconstructed image of the present application is more stable and the cumulative error is smaller.

[0344] In the embodiments of FIG. 10A and FIG. 11 described below, the first prediction information obtained by inter-frame prediction may be post-processed based on FIG. 2A to reduce the accumulated error.

[0345] FIG10A is a schematic diagram illustrating an exemplary encoding process.

[0346] S1001: Determine preset parameters of coding units in a current frame.

[0347] Exemplarily, S1001 may be performed by a preset parameter determination module of an AI video encoding framework.

[0348] Exemplarily, the preset parameters of the coding unit can be determined based on the original image of the coding unit and the image of the reference unit; wherein the image of the reference unit includes the original image of the reference unit in the reference frame and / or the target reconstructed image of the reference unit. The specific process of determining the preset parameters of the coding unit is similar to that described in S304 above and will not be repeated here. The difference between S1001 and S304 is that since S1001 is performed before inter-frame prediction, the information used to determine the preset parameters of the coding unit in S1001 does not include the initial reconstructed image of the coding unit.

[0349] The preset parameters may be used to describe the motion complexity of the original image of the coding unit relative to the image of the reference unit.

[0350] S1002: Perform inter-frame prediction on the coding unit to obtain first prediction information of the coding unit.

[0351] For example, S1002 may refer to the description of S301 above, which will not be repeated here.

[0352] Exemplarily, S1002 can be performed by an inter-frame prediction module in the AI ​​video coding framework, wherein the inter-frame prediction module can include the motion estimation module, the optical flow processing module (including the optical flow encoding module and the optical flow decoding module), the first entropy estimation module and the prediction module in Figure 3A.

[0353] S1003 , fusing the first prediction information and the reconstructed image information of the reference unit according to preset parameters to obtain second prediction information of the coding unit.

[0354] Exemplarily, S1003 can be executed by a fusion module in the AI ​​video encoding framework; wherein, the fusion module can be an independent module or integrated into the reconstruction module, and this application does not impose any restrictions on this.

[0355] Exemplarily, one way to fuse the reconstructed image information of the reference unit and the first prediction information of the coding unit to obtain the second prediction information of the coding unit may be to determine the weight of the reconstructed image information of the reference unit and the weight of the first prediction information of the coding unit according to preset parameters; and perform weighted calculation on the reconstructed image information of the reference unit and the first prediction information of the coding unit according to the weight of the reconstructed image information of the reference unit and the weight of the first prediction information of the coding unit to obtain the second prediction information of the coding unit.

[0356] In this case, if the first preset value is greater than the second preset value, then when the first preset value and the second preset value are numbers between 0 and 1 (the first preset value and the second preset value can both be 0 or 1), the value of the preset parameter of the coding unit can be used as the weight of the reconstructed image information of the reference unit; and the difference between 1 and the value of the preset parameter of the coding unit can be used as the weight of the first prediction information of the coding unit. When the first preset value and the second preset value are numbers greater than 1, the first preset value and the second preset value can be transformed into numbers between 0 and 1, and then the value of the preset parameter of the coding unit after the transformation can be used as the weight of the reconstructed image information of the reference unit, and the difference between 1 and the value of the preset parameter of the coding unit after the transformation can be used as the weight of the first prediction information of the coding unit. At this time, the fusion can be performed with reference to the following formula (8): H1=mask*P +(1-mask)*H2 (8)

[0357] If the second preset value is greater than the first preset value, the fusion can be performed according to the following formula (9): H1 = mask*H2 + (1-mask)*P (9)

[0358] In formula (8) and formula (9), H1 represents the second prediction information of the coding unit, P is the target reconstructed image of the reference unit or the feature information of the target reconstructed image of the reference unit, and H2 is the first prediction information of the coding unit.

[0359] When the motion complexity of the original image of the coding unit relative to the image of the reference unit is low, and the mask in formula (8) is 1 and the mask in formula (9) is 0, fusing the first prediction information of the coding unit and the reconstructed image information of the reference unit can be performed by copying the reconstructed image information of the reference unit to the coding unit to obtain the second prediction information of the coding unit. In this way, the accuracy of inter-frame prediction can be improved, thereby reducing the error introduced by inter-frame prediction, reducing the error of the current frame, and thus reducing the cumulative error of subsequent frames, thereby ensuring temporal stability.

[0360] When the motion complexity of the original image of the coding unit relative to the image of the reference unit is high, and the mask in formula (8) is 0 and the mask in formula (9) is 1, the second prediction information of the coding unit obtained by fusing the first prediction information of the coding unit and the reconstructed image information of the reference unit is actually the first prediction information of the coding unit; since the motion complexity of the original image of the coding unit relative to the image of the reference unit is high, if the reconstructed image information of the reference unit is copied to the coding unit, the accuracy of the second prediction information of the coding unit obtained is low; therefore, the first prediction information of the coding unit obtained by inter-frame prediction can be used as the second prediction information of the coding unit, which can ensure the accuracy of inter-frame prediction and thus ensure the quality of the target reconstructed image of the coding unit.

[0361] When the motion complexity of the original image of the coding unit relative to the image of the reference unit is low complexity, and the mask in formula (8) and formula (9) is a value other than 0 and 1 (that is, the first preset value and the second preset value are values ​​other than 0 and 1), the first prediction information of the coding unit and the reconstructed image information of the reference unit are fused to obtain the second prediction information of the coding unit. The first prediction information of the coding unit and the reconstructed image information of the reference unit both have a certain weight, which can also improve the accuracy of inter-frame prediction to a certain extent and reduce the error of the current frame; thereby reducing the cumulative error of subsequent frames to a certain extent and ensuring temporal stability. It can also ensure the quality of the target reconstructed image of the coding unit to a certain extent.

[0362] Exemplarily, one way to fuse the reconstructed image information of the reference unit and the first prediction information to obtain the second prediction information of the coding unit may be to input the preset parameters of the coding unit, the first prediction information of the coding unit, and the reconstructed image information of the reference unit into a fusion model (this is not the same model as the fusion model in the embodiment of FIG3A above) to obtain the second prediction information of the coding unit. In this case, there is no limit on the size of the first preset value and the second preset value; wherein, the fusion model is pre-trained and has the ability to generate more accurate prediction information. In this way, the error of the coding unit of the current frame can also be reduced, thereby reducing the cumulative error of subsequent frames and ensuring timing stability.

[0363] S1004 : Determine residual information of the coding unit according to the second prediction information and the original image of the coding unit.

[0364] Exemplarily, S1104 can be performed by the residual coding module in the AI ​​video coding framework; specifically, refer to the description of the embodiment of Figure 2A above, which will not be repeated here. It should be noted that the residual information of the coding unit determined by S1104 corresponds to the first residual information of the coding unit in the embodiment of Figure 2A.

[0365] S1005: Encode residual information of the coding unit.

[0366] Exemplarily, S1005 can be performed by the second entropy coding module to obtain a residual bitstream, which will not be described in detail here. It should be noted that the residual information of the coding unit in S1005 corresponds to the first residual information of the coding unit in the embodiment of FIG. 2A .

[0367] S1006, encoding preset parameters of the encoding unit.

[0368] For example, S1006 may refer to the description of S305 in the above embodiment, which will not be repeated here.

[0369] Figure 10B is a schematic diagram illustrating an exemplary encoding process. Figure 10B is illustrated using the current frame as an example. In Figure 10B , the first preset value is 1, the second preset value is 0, and the reconstructed image information of the reference unit used for fusion is the target reconstructed image of the reference frame.

[0370] Referring to FIG10B , for example, the encoding process in the dotted box with bold lines on the upper left corresponds to steps S1002 and S1004 to S1006, and the description of FIG2A can be referred to, which will not be repeated here. The processing process in the dotted box with bold lines on the lower right corresponds to steps S1001 and S1003. In FIG10B , a black dot in the preset parameters of the current frame indicates that the value of the preset parameters of the corresponding encoding unit is 0, and a white dot indicates that the value of the preset parameters of the corresponding encoding unit is 1.

[0371] Figure 11 is a schematic diagram illustrating an exemplary decoding process, which corresponds to the decoding process of Figure 10A .

[0372] S1101, receiving a code stream.

[0373] S1102: Obtain preset parameters of the coding unit according to the code stream.

[0374] S1103: Obtain decoded residual information of the coding unit in the current frame according to the code stream.

[0375] S1104 , performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit.

[0376] For example, S1101 to S1104 may refer to the description of S401 to S404 above, which will not be repeated here.

[0377] It should be noted that the decoded residual information of the coding unit in S1103 corresponds to the second residual information of the coding unit in the embodiment of FIG. 2B .

[0378] It should be noted that S1102 and S1103 can be executed by the decoding module in the AI ​​video decoding framework, wherein the decoding module may include the second entropy estimation module, the second entropy decoding module and the residual decoding module in Figure 2B (these three modules are used to execute S1103); and also include a third entropy decoding module (the third entropy decoding module belongs to the entropy decoding module in the AI ​​video decoding framework, and the third entropy decoding module is used to execute S1102).

[0379] S1105 , fusing the first prediction information and the reconstructed image information of the reference unit according to preset parameters to obtain second prediction information of the coding unit.

[0380] Illustratively, the fusion process of S1105 is similar to the fusion process of S1003 in the embodiment of FIG. 10A , and is not described again here.

[0381] Exemplarily, the AI ​​video decoding framework may include a fusion module, and S1005 may be executed by the fusion module in the AI ​​video encoding framework; wherein, the fusion module may be an independent module or integrated into the reconstruction module, and this application does not impose any restrictions on this.

[0382] S1106 , reconstructing according to the decoded residual information and the second prediction information to determine a target reconstructed image of the coding unit.

[0383] For example, S1106 may refer to the description of the reconstruction process in the embodiment of FIG. 2B , which will not be repeated here.

[0384] It should be noted that the decoded residual information in the embodiment of FIG. 11 corresponds to the second residual information in the embodiment of FIG. 2B .

[0385] It should be understood that during the encoding and decoding process, the following two fusion steps can be performed simultaneously: fusing the first prediction information and the reconstructed image information of the reference unit according to the preset parameters of the encoding unit to obtain the second prediction information of the encoding unit, and fusing the reconstructed image information of the reference unit and the reconstructed image information of the encoding unit according to the preset parameters of the encoding unit to obtain the target reconstructed image of the encoding unit. In other words, encoding is performed in combination with the embodiment of FIG10A and the embodiment of FIG3A, and decoding is performed in combination with the embodiment of FIG11 and the embodiment of FIG4A; this can further reduce the cumulative error.

[0386] It should be understood that, based on the embodiment of FIG. 10A , adaptive rate encoding may be performed on the residual information of the coding unit according to preset parameters of the coding unit to reduce the bit rate overhead.

[0387] It should be understood that, based on FIG. 10A , adaptive rate encoding may be performed on the first optical flow of the coding unit according to preset parameters of the coding unit to reduce the bit rate overhead.

[0388] It should be understood that, based on FIG10A , adaptive rate encoding may be performed on the residual information of the coding unit and the first optical flow of the coding unit according to preset parameters of the coding unit to further reduce the bit rate overhead.

[0389] It should be understood that the embodiment of FIG. 10A may be combined with the embodiment of FIG. 5A , and the embodiment of FIG. 11 may be combined with the embodiment of FIG. 6 , so as to further reduce the accumulated error and reduce the code rate overhead.

[0390] It should be understood that the embodiment of FIG. 10A may be combined with the embodiment of FIG. 7A , and the embodiment of FIG. 11 may be combined with the embodiment of FIG. 8 , so as to further reduce the accumulated error and reduce the code rate overhead.

[0391] It should be understood that the embodiment of FIG. 10A , the embodiment of FIG. 5A , and the embodiment of FIG. 7A may be combined, as well as the embodiment of FIG. 11 , the embodiment of FIG. 6 , and the embodiment of FIG. 8 may be combined to further reduce the cumulative error and reduce the code rate overhead.

[0392] In an example, FIG12 shows a schematic block diagram of a device 1200 according to an embodiment of the present application. The device 1200 may include: a processor 1201 and a transceiver / transceiver pin 1202 , and optionally, a memory 1203 .

[0393] The various components of the device 1200 are coupled together via a bus 1204, wherein the bus 1204 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all buses are referred to as bus 1204 in the figure.

[0394] Optionally, the memory 1203 may be used to store instructions in the aforementioned method embodiment. The processor 1201 may be used to execute the instructions in the memory 1203 and control the receiving pin to receive a signal and control the transmitting pin to send a signal.

[0395] The device 1200 may be an electronic device (such as an encoding device, a decoding device, a decoding device, etc.) or a chip of an electronic device in the above method embodiment.

[0396] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0397] The present application also provides a chip including one or more interface circuits and one or more processors. The one or more processors receive or send data via the one or more interface circuits. When the one or more processors execute computer instructions, the steps of the above-mentioned related methods are implemented. The interface circuit is a transceiver / transceiver pin 1202.

[0398] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0399] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0400] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the methods in the above-mentioned method embodiments.

[0401] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0402] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0403] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0404] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0405] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0406] Any content of each embodiment of this application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0407] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0408] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0409] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0410] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A coding method, characterized in that: The method comprises: Performing inter-frame prediction on a coding unit in a current frame to obtain first prediction information of the coding unit; Determine residual information of the coding unit according to the first prediction information and the original image of the coding unit; Determine preset parameters of the coding unit; wherein the preset parameters are used to fuse the reconstructed image information of the reference unit and the reconstructed image information of the coding unit to obtain a target reconstructed image of the coding unit, the reconstructed image information of the reference unit includes the target reconstructed image of the reference unit or feature information of the target reconstructed image of the reference unit, the reconstructed image information of the coding unit includes the initial reconstructed image of the coding unit or feature information of the initial reconstructed image of the coding unit, the initial reconstructed image of the coding unit and the feature information of the initial reconstructed image of the coding unit are reconstructed according to the first prediction information and the residual information, and the reference unit is a unit in the reference frame corresponding to the current frame; The residual information is encoded, and the preset parameter is encoded.

2. The method according to claim 1, characterized in that The encoding of the residual information comprises: According to the preset parameters, adaptive rate encoding is performed on the residual information.

3. The method according to claim 1 or 2, characterized in that: The performing inter-frame prediction on the coding unit in the current frame to obtain first prediction information of the coding unit includes: Determining a first optical flow of the coding unit according to the original image of the coding unit and the target reconstructed image of the reference unit; Determining the first prediction information according to the first optical flow and the reconstructed image information of the reference unit; The method further comprises: According to the preset parameters, adaptive rate encoding is performed on the first optical flow.

4. The method according to any one of claims 1 to 3, characterized in that: The determining of the preset parameters of the encoding unit includes: Determine the similarity between the image of the coding unit and the image of the reference unit; wherein the image of the coding unit includes the original image of the coding unit and / or the initial reconstructed image of the coding unit, and the image of the reference unit includes the original image of the reference unit and / or the target reconstructed image of the reference unit; The preset parameters are determined according to the similarity.

5. The method according to any one of claims 1 to 4, characterized in that: The fusing the reconstructed image information of the reference unit and the reconstructed image information of the coding unit to obtain a target reconstructed image of the coding unit includes: Determining, according to the preset parameters, a weight of the reconstructed image information of the reference unit and a weight of the reconstructed image information of the encoding unit; According to the weight of the reconstructed image information of the reference unit and the weight of the reconstructed image information of the coding unit, weighted calculation is performed on the reconstructed image information of the reference unit and the reconstructed image information of the coding unit to obtain a target reconstructed image of the coding unit.

6. The method according to any one of claims 1 to 4, characterized in that: The fusing the reconstructed image information of the reference unit and the reconstructed image information of the coding unit to obtain a target reconstructed image of the coding unit includes: The preset parameters, the reconstructed image information of the reference unit and the reconstructed image information of the coding unit are input into a fusion model to obtain a target reconstructed image of the coding unit.

7. The method according to claim 2, characterized in that The performing adaptive rate encoding on the residual information according to the preset parameters includes: When the value of the preset parameter is a first preset value, encoding the residual information according to a first bit rate parameter; When the value of the preset parameter is a second preset value, encoding the residual information according to a second coding rate parameter; The target bit rate of the bit stream obtained by encoding the residual information according to the second bit rate parameter is greater than the target bit rate of the bit stream obtained by encoding the residual information according to the first bit rate parameter.

8. The method according to any one of claims 1 to 7, characterized in that: The encoding of the preset parameters includes: Performing entropy estimation according to at least one of the residual information or the second optical flow of the encoding unit to determine the probability distribution of the preset parameter; wherein the second optical flow is obtained according to a code stream obtained by encoding the first optical flow; The preset parameters are entropy encoded according to the probability distribution of the preset parameters.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: According to the preset parameters, the first prediction information and the reconstructed image information of the reference unit are fused to obtain second prediction information; The determining, according to the first prediction information and the original image of the coding unit, the residual information of the coding unit includes: Residual information of the coding unit is determined according to the second prediction information and an original image of the coding unit.

10. The method according to any one of claims 1 to 9, characterized in that: The method is applied to an AI video coding framework, which includes: an inter-frame prediction module, a residual coding module, an entropy coding module, a preset parameter determination module and a fusion module, wherein: The inter-frame prediction module is used to perform inter-frame prediction on the coding unit to obtain the first prediction information; and output the first prediction information to the residual coding module; The residual coding module is used to determine the residual information according to the first prediction information and the original image of the coding unit; and output the residual information to the entropy coding module; The preset parameter determination module is used to determine the preset parameters; output the preset parameters to the entropy coding module and the fusion module; The entropy coding module is used to encode the residual information and the preset parameters; The fusion module is used to fuse the reconstructed image information of the reference unit and the reconstructed image information of the encoding unit according to the preset parameters to obtain a target reconstructed image of the encoding unit.

11. The method according to claim 10, characterized in that The inter-frame prediction module includes a motion estimation network, an optical flow processing module and a prediction module, wherein: The motion estimation network is used to perform motion estimation according to the original image of the coding unit and the target reconstructed image of the reference unit to obtain a first optical flow of the coding unit; and output the first optical flow to the optical flow processing module; The optical flow processing module is used to process based on the first optical flow to determine a second optical flow; and output the second optical flow to the prediction module; The prediction module is used to perform prediction based on the reconstructed image information of the reference unit and the second optical flow to obtain the first prediction information.

12. A decoding method, characterized in that: The method comprises: Receive code stream; Obtaining decoded residual information of a coding unit in a current frame according to the code stream, and obtaining preset parameters of the coding unit according to the code stream; Performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit; Reconstructing according to the decoded residual information and the first prediction information to determine the reconstructed image information of the coding unit; wherein the reconstructed image information of the coding unit includes the initial reconstructed image of the coding unit or feature information of the initial reconstructed image of the coding unit; The reconstructed image information of the encoding unit and the reconstructed image information of the reference unit are fused according to the preset parameters to obtain a target reconstructed image of the encoding unit; wherein the reconstructed image information of the reference unit includes the target reconstructed image of the reference unit or feature information of the target reconstructed image of the reference unit, and the reference unit is a unit in a reference frame corresponding to the current frame.

13. The method according to claim 12, characterized in that The fusing the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to the preset parameters to obtain a target reconstructed image of the coding unit includes: Determining, according to the preset parameters, a weight of the reconstructed image information of the encoding unit and a weight of the reconstructed image information of the reference unit; According to the weight of the reconstructed image information of the coding unit and the weight of the reconstructed image information of the reference unit, weighted calculation is performed on the reconstructed image information of the coding unit and the reconstructed image information of the reference unit to obtain a target reconstructed image of the coding unit.

14. The method according to claim 12, characterized in that The fusing the reconstructed image information of the coding unit and the reconstructed image information of the reference unit according to the preset parameters to obtain a target reconstructed image of the coding unit includes: The preset parameters, the reconstructed image information of the encoding unit and the reconstructed image information of the reference unit are input into a fusion model to obtain a target reconstructed image of the encoding unit.

15. The method according to any one of claims 12 to 14, characterized in that The code stream includes a preset parameter code stream and an optical stream code stream, and obtaining the preset parameters of the coding unit in the current frame according to the code stream includes: Perform entropy estimation according to at least one of the decoded residual information or the second optical flow of the encoding unit to determine the probability distribution of the preset parameter; wherein the second optical flow is obtained according to the optical flow code stream; The preset parameters are obtained by entropy decoding from the preset parameter bitstream according to the probability distribution of the preset parameters.

16. The method according to any one of claims 12 to 15, characterized in that The code stream includes a residual code stream, and obtaining the decoded residual information of the coding unit according to the code stream includes: According to the preset parameters, adaptive bit rate decoding is performed on the residual bit stream; Determine the decoded residual information of the encoding unit according to the information obtained by decoding the residual bitstream at the adaptive bit rate.

17. The method according to any one of claims 12 to 16, characterized in that: The code stream includes an optical code stream, and the method further includes: According to the preset parameters, adaptive bit rate decoding is performed on the bit stream of the optical flow; The second optical flow of the encoding unit is determined according to information obtained by decoding the optical flow code stream at the adaptive code rate.

18. The method according to any one of claims 12 to 17, characterized in that The method is applied to an AI video decoding framework, which includes: a decoding module, an inter-frame prediction module, a reconstruction module and a fusion module, wherein: The decoding module is used to obtain the preset parameters according to the bit stream, and obtain the decoded residual information according to the bit stream; output the preset parameters to the fusion module, and output the decoded residual information to the reconstruction module; The inter-frame prediction module is used to perform inter-frame prediction on the coding unit to obtain the first prediction information; and output the first prediction information to the reconstruction module; The reconstruction module is used to reconstruct according to the decoded residual information and the first prediction information, determine the reconstructed image information of the coding unit; and output the reconstructed image information of the coding unit to the fusion module; The fusion module is used to fuse the reconstructed image information of the encoding unit and the reconstructed image information of the reference unit according to the preset parameters to obtain the target reconstructed image of the encoding unit.

19. A coding method, characterized in that: The method comprises: Determining preset parameters of a coding unit in a current frame; Performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit; fusing the first prediction information and the reconstructed image information of the reference unit according to the preset parameters to obtain the second prediction information of the coding unit; wherein the reconstructed image information of the reference unit includes the target reconstructed image of the reference unit or the feature information of the target reconstructed image of the reference unit, and the reference unit is a unit in the reference frame corresponding to the current frame; Determine residual information of the coding unit according to the second prediction information and the original image of the coding unit; The residual information is encoded, and the preset parameter is encoded.

20. The method according to claim 19, characterized in that The encoding of the residual information comprises: According to the preset parameters, adaptive rate encoding is performed on the residual information.

21. The method according to claim 19 or 20, characterized in that The performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit includes: Determining a first optical flow of the coding unit according to the original image of the coding unit and the target reconstructed image of the reference unit; Determining the first prediction information according to the first optical flow and the reconstructed image information of the reference unit; The method further comprises: According to the preset parameters, adaptive rate encoding is performed on the first optical flow.

22. The method according to any one of claims 19 to 21, characterized in that The method further comprises: fusing the reconstructed image information of the reference unit and the reconstructed image information of the encoding unit according to the preset parameters to obtain a target reconstructed image of the encoding unit; Among them, the reconstructed image information of the coding unit includes the initial reconstructed image of the coding unit or the characteristic information of the initial reconstructed image of the coding unit, and the initial reconstructed image of the coding unit and the characteristic information of the initial reconstructed image of the coding unit are reconstructed based on the second prediction information and the residual information.

23. The method according to any one of claims 19 to 22, characterized in that The method is applied to an AI video coding framework, which includes: an inter-frame prediction module, a residual coding module, an entropy coding module, a preset parameter determination module and a fusion module, wherein: The preset parameter determination module is used to determine the preset parameters; output the preset parameters to the fusion module and the entropy coding module; The inter-frame prediction module is used to perform inter-frame prediction on the coding unit to obtain the first prediction information; and output the first prediction information to the fusion module; The fusion module is used to fuse the first prediction information and the reconstructed image information of the reference unit according to the preset parameters to obtain the second prediction information; and output the second prediction information to the residual coding module; The residual coding module is used to determine the residual information according to the second prediction information and the original image of the coding unit; and output the residual information to the entropy coding module; The entropy coding module is used to encode the residual information and the preset parameters.

24. A decoding method, characterized in that: The method comprises: Receive code stream; Obtaining decoded residual information of a coding unit in a current frame according to the code stream, and obtaining preset parameters of the coding unit according to the code stream; Performing inter-frame prediction on the coding unit to obtain first prediction information of the coding unit; fusing the first prediction information and the reconstructed image information of the reference unit according to the preset parameters to obtain the second prediction information of the coding unit; wherein the reconstructed image information of the reference unit includes the target reconstructed image of the reference unit or the feature information of the target reconstructed image of the reference unit, and the reference unit is a unit in the reference frame corresponding to the current frame; Reconstruction is performed according to the decoded residual information and the second prediction information to determine the reconstructed image information of the coding unit; wherein the reconstructed image information of the coding unit includes the initial reconstructed image of the coding unit or the characteristic information of the initial reconstructed image of the coding unit, and the characteristic information of the initial reconstructed image of the coding unit is used to obtain the initial reconstructed image of the coding unit.

25. The method according to claim 24, characterized in that The method further comprises: The reconstructed image information of the encoding unit and the reconstructed image information of the reference unit are fused according to the preset parameters to obtain a target reconstructed image of the encoding unit.

26. The method according to claim 24 or 25, characterized in that The code stream includes a preset parameter code stream and an optical stream code stream, and obtaining the preset parameters of the encoding unit according to the code stream includes: Perform entropy estimation according to at least one of the decoded residual information and the second optical flow of the encoding unit to determine the probability distribution of the preset parameter; wherein the second optical flow is obtained according to the optical flow code stream; The preset parameters are obtained by entropy decoding from the preset parameter bitstream according to the probability distribution of the preset parameters.

27. The method according to any one of claims 24 to 26, characterized in that The method is applied to an AI video decoding framework, which includes: a decoding module, an inter-frame prediction module, a reconstruction module and a fusion module, wherein: The decoding module is used to obtain the decoded residual information according to the code stream, and obtain the preset parameters according to the code stream; output the decoded residual information to the reconstruction module, and output the preset parameters to the fusion module; The inter-frame prediction module is used to perform inter-frame prediction on the coding unit to obtain the first prediction information; and output the first prediction information to the fusion module; The fusion module is used to fuse the first prediction information and the reconstructed image information of the reference unit according to the preset parameters to obtain the second prediction information; and output the second prediction information to the reconstruction module; The reconstruction module is used to reconstruct according to the decoded residual information and the second prediction information to determine the reconstructed image information of the coding unit.

28. An encoding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the encoding device executes the method according to any one of claims 1 to 11 or any one of claims 19 to 23.

29. A decoding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the decoding device performs the method according to any one of claims 12 to 18 or any one of claims 24 to 27.

30. A decoding device, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method as described in any one of claims 1 to 11 or any one of claims 19 to 23 are executed.

31. A decoding device, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method as described in any one of claims 12 to 18 or any one of claims 24 to 27 are executed.

32. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 27.

33. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 27 are performed.

Images