Decoding and coding method, device and equipment

Abstract

The invention provides a decoding and coding method, device and equipment, and the decoding method comprises the steps: carrying out the analysis of a prediction information code stream corresponding to a current image block, and obtaining the prediction information corresponding to the current image block; performing prediction information decoding on the prediction information to obtain a prediction mode feature and a prediction confidence coefficient corresponding to the current image block; determining a target prediction feature corresponding to the current image block based on the prediction mode feature and the prediction confidence; analyzing the residual information code stream corresponding to the current image block to obtain residual information corresponding to the current image block; performing residual decoding on the residual information to obtain a target residual feature corresponding to the current image block; and determining a reconstructed image block based on the target residual feature and the target prediction feature. Through the scheme of the invention, the decoding performance can be improved.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202411145013.8 (22) Application Date 2024.08.19 (71) Applicant Hangzhou Hikvision Digital Technology Co., Ltd. Address 555 Qianmo Road, Binjiang District, Hangzhou City, Zhejiang Province, 310051 (72) Inventors Sun Yucheng, Liu Yunzhuo, Chen Fangdong, Wu Xiaoyang (74) Patent Agency Beijing Bosijia Intellectual Property Agency Co., Ltd. 11415 Patent Attorney Yang Chunxiang (51) Int.Cl. H04N 19 / 109 (2014.01) H04N 19 / 176 (2014.01) H04N 19 / 503 (2014.01) (54) Invention Title: A Decoding and Encoding Method, Apparatus, and Device Thereof (57) Abstract: This application provides a decoding and encoding method, apparatus, and device thereof. The decoding method includes: parsing the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block; decoding the prediction information to obtain the prediction mode features and prediction confidence corresponding to the current image block; determining the target prediction features corresponding to the current image block based on the prediction mode features and prediction confidence; parsing the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; decoding the residual information to obtain the target residual features corresponding to the current image block; and determining the reconstructed image block based on the target residual features and target prediction features. The solution of this application can improve decoding performance. Claims 5 pages, Description 54 pages, Drawings 12 pages, CN 121603654 A 2026.03.03 CN 1 21 60 36 54 A 1. A decoding method, characterized in that it is applied at a decoding end, the method comprising: parsing a prediction information bitstream corresponding to a current image block to obtain prediction information corresponding to the current image block; decoding the prediction information to obtain prediction mode features and prediction confidence corresponding to the current image block; determining target prediction features corresponding to the current image block based on the prediction mode features and the prediction confidence; wherein the prediction confidence is obtained based on information in the prediction information bitstream, and the prediction confidence represents the accuracy of the target prediction features; parsing a residual information bitstream corresponding to the current image block to obtain residual information corresponding to the current image block; decoding the residual information to obtain target residual features corresponding to the current image block; determining a reconstructed image block corresponding to the current image block based on the target residual features and the target prediction features. 2. The method according to claim 1, wherein determining the target prediction features corresponding to the current image patch based on the prediction pattern features and the prediction confidence includes:Based on the predicted pattern features, the predicted confidence level, and historical temporal reference features, temporal information mining is performed to obtain the initial predicted features corresponding to the current image patch; wherein, the information buffer stores image reference information that the current image patch can refer to, and the historical temporal reference features are derived through the image reference information; the target predicted features corresponding to the current image patch are determined based on the initial predicted features. 3. The method according to claim 2, wherein the step of performing temporal information mining based on the predicted pattern features, the predicted confidence level, and historical temporal reference features to obtain the initial predicted features corresponding to the current image patch includes: performing a compensation operation on the historical temporal reference features based on the predicted pattern features to obtain temporal compensated features; and determining the initial predicted features based on the temporal compensated features and the predicted confidence level. 4. The method according to claim 2, characterized in that: the prediction pattern feature includes a first prediction pattern feature and a second prediction pattern feature; the historical time-domain reference feature includes a first historical time-domain reference feature and a second historical time-domain reference feature; the prediction confidence includes a first prediction confidence and a second prediction confidence; and the initial prediction feature includes a first initial prediction feature and a second initial prediction feature; the step of performing time-domain information mining based on the prediction pattern feature, the prediction confidence, and the historical time-domain reference feature to obtain the initial prediction feature corresponding to the current image block includes: performing a compensation operation on the first historical time-domain reference feature based on the first prediction pattern feature to obtain a first time-domain compensated feature; determining the first initial prediction feature based on the first time-domain compensated feature and the first prediction confidence; performing a compensation operation on the second historical time-domain reference feature based on the second prediction pattern feature to obtain a second time-domain compensated feature; and determining the second initial prediction feature based on the second time-domain compensated feature and the second prediction confidence. 5. The method according to claim 3 or 4, characterized in that, determining the initial prediction feature based on the temporal compensation feature and the prediction confidence includes: performing a weighted operation based on the temporal compensation feature and the prediction confidence to obtain a post-operation feature; performing a first type of operation based on the post-operation feature to obtain the initial prediction feature; or, performing a second type of operation based on the temporal compensation feature to obtain a post-operation feature; performing a weighted operation based on the post-operation feature and the prediction confidence to obtain the initial prediction feature. 6. The method according to claim 5, characterized in that, performing a first type of operation based on the post-operation feature to obtain the initial prediction feature includes: sequentially performing a convolution operation and a residual block operation on the post-operation feature, and adding the feature after the residual block operation to the post-operation feature to obtain the initial prediction feature;The second type of operation based on the temporal compensation feature to obtain the post-operation feature includes: sequentially performing convolution and residual block operations on the temporal compensation feature, and adding the feature after the residual block operation with the temporal compensation feature to obtain the post-operation feature. 7. The method according to claim 2, wherein determining the target prediction feature corresponding to the current image block based on the initial prediction feature includes: determining the initial prediction feature as the target prediction feature; or, performing a transformation operation on the initial prediction feature to obtain the target prediction feature; or, performing a weighted operation on the initial prediction feature and the residual confidence to obtain the target prediction feature; wherein the residual confidence is obtained when performing residual decoding on the residual information; wherein the residual confidence is obtained based on information in the residual information bitstream, and the residual confidence represents the accuracy of the target prediction feature; or, performing a weighted operation on the initial prediction feature and the residual confidence to obtain the weighted feature, and performing a transformation operation on the weighted feature to obtain the target prediction feature. 8. The method according to claim 1, characterized in that, determining the target prediction feature corresponding to the current image patch based on the prediction mode feature and the prediction confidence score includes: performing temporal information mining based on the prediction mode feature and historical temporal reference features to obtain the initial prediction feature corresponding to the current image patch; wherein, the information buffer stores image reference information that the current image patch can refer to, and the historical temporal reference feature is derived through the image reference information; determining the target confidence score corresponding to the initial prediction feature based on the prediction confidence score; wherein, the target confidence score is obtained based on information in the prediction information bitstream and / or information in the residual information bitstream, and the target confidence score represents the accuracy of the target prediction feature; performing a weighted operation on the initial prediction feature and the target confidence score to obtain the target prediction feature; or, performing a weighted operation on the initial prediction feature and the target confidence score to obtain a weighted feature, and performing a transformation operation on the weighted feature to obtain the target prediction feature. 9. The method according to claim 8, wherein determining the target confidence level corresponding to the initial prediction feature based on the prediction confidence level comprises: determining the prediction confidence level as the target confidence level; or, obtaining the target confidence level by performing a transformation operation on the prediction confidence level; or, generating the target confidence level based on the prediction confidence level and the residual confidence level. 10. The method according to any one of claims 1-9, wherein parsing the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block comprises:The prediction edge information bitstream corresponding to the current image block is parsed to obtain prediction prior information; temporal prediction prior information is obtained from the prediction latent buffer, which is the prediction information that the current image block can refer to; a probability distribution of prediction information is generated based on the prediction prior information and the temporal prediction prior information, and the prediction information bitstream is parsed using the probability distribution to obtain the prediction information corresponding to the current image block, and the prediction information is stored in the prediction latent buffer for temporal prediction prior information of subsequent image blocks. Claims 2 / 5, page 3, CN 121603654, A 11. The method according to any one of claims 1-9, characterized in that, the step of parsing the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block includes: parsing the residual edge information bitstream corresponding to the current image block to obtain residual super-prior information; obtaining residual prior information from the residual latent buffer, wherein the residual prior information is residual information that can be referenced by the current image block; performing spatial prior processing based on the initial prediction features corresponding to the current image block to obtain spatial prior information; or, performing spatial prior processing based on the target prediction features corresponding to the current image block to obtain spatial prior information; generating a probability distribution of residual information based on the residual super-prior information, the residual prior information, and the spatial prior information, using the probability distribution to parse the residual information bitstream to obtain the residual information corresponding to the current image block, and storing the residual information in the residual latent buffer for residual prior information of subsequent image blocks. 12. An encoding method, characterized in that it is applied at an encoding end, the method comprising: parsing a prediction information bitstream corresponding to a current image patch to obtain prediction information corresponding to the current image patch; decoding the prediction information to obtain prediction mode features and prediction confidence corresponding to the current image patch; determining a target prediction feature corresponding to the current image patch based on the prediction mode features and the prediction confidence; wherein the prediction confidence is obtained based on information in the prediction information bitstream, and the prediction confidence represents the accuracy of the target prediction feature; parsing a residual information bitstream corresponding to the current image patch to obtain residual information corresponding to the current image patch; decoding the residual information to obtain a target residual feature corresponding to the current image patch; determining a reconstructed image patch corresponding to the current image patch based on the target residual feature and the target prediction feature. 13. A decoding method, characterized in that it is applied at a decoding end, the method comprising: parsing a prediction information bitstream corresponding to a current image patch to obtain prediction information corresponding to the current image patch; decoding the prediction information to obtain prediction mode features corresponding to the current image patch; determining a target prediction feature based on the prediction mode features and the prediction confidence ...The prediction features corresponding to the current image block are determined by using prediction mode features and historical image reference information; the residual information bitstream corresponding to the current image block is parsed to obtain the residual information corresponding to the current image block; the residual information is residual decoded to obtain the target residual features corresponding to the current image block; based on the target residual features and the prediction features, the initial image reference information and the reconstructed image block corresponding to the current image block are determined; the reference confidence level corresponding to the initial image reference information is obtained; wherein, the reference confidence level represents the accuracy of the initial image reference information; based on the reference confidence level and the initial image reference information, the target image reference information corresponding to the current image block is determined, and the target image reference information is used as the historical image reference information for subsequent image blocks. 14. The method according to claim 13, wherein the reference confidence is obtained when performing residual decoding on the residual information; or, the reference confidence is obtained when performing prediction information decoding on the prediction information; or, a first reference feature is obtained when performing residual decoding on the residual information, and a second reference feature is obtained when performing prediction information decoding on the prediction information, wherein the reference confidence is obtained based on the first reference feature and the second reference feature. 15. An encoding method, characterized in that it is applied at an encoding end, the method comprising: parsing a prediction information bitstream corresponding to a current image block to obtain prediction information corresponding to the current image block; decoding the prediction information to obtain prediction mode features corresponding to the current image block; determining prediction features corresponding to the current image block based on the prediction mode features and historical image reference information; parsing a residual information bitstream corresponding to the current image block to obtain residual information corresponding to the current image block; decoding the residual information to obtain target residual features corresponding to the current image block; determining initial image reference information and reconstructed image blocks corresponding to the current image block based on the target residual features and the prediction features; obtaining a reference confidence level corresponding to the initial image reference information; wherein the reference confidence level represents the accuracy of the initial image reference information; determining target image reference information corresponding to the current image block based on the reference confidence level and the initial image reference information, wherein the target image reference information is used for historical image reference information of subsequent image blocks. 16. A decoding apparatus, characterized in that it is applied at a decoding end, the apparatus comprising: a prediction module, configured to parse a prediction information bitstream corresponding to a current image block to obtain prediction information corresponding to the current image block; and to decode the prediction information to obtain prediction mode features corresponding to the current image block.The system comprises: a prediction confidence level; determining the target prediction feature corresponding to the current image patch based on the prediction mode feature and the prediction confidence level; wherein the prediction confidence level is obtained based on information in the prediction information bitstream, and the prediction confidence level represents the accuracy of the target prediction feature; a residual decoding module, used to parse the residual information bitstream corresponding to the current image patch to obtain the residual information corresponding to the current image patch; performing residual decoding on the residual information to obtain the target residual feature corresponding to the current image patch; and determining the reconstructed image patch corresponding to the current image patch based on the target residual feature and the target prediction feature. 17. An encoding apparatus, characterized in that it is applied at an encoding end, the apparatus comprising: a prediction module, configured to parse a prediction information bitstream corresponding to a current image patch to obtain prediction information corresponding to the current image patch; to decode the prediction information to obtain prediction mode features and prediction confidence corresponding to the current image patch; and to determine target prediction features corresponding to the current image patch based on the prediction mode features and the prediction confidence; wherein the prediction confidence is obtained based on information in the prediction information bitstream, and the prediction confidence represents the accuracy of the target prediction features; and a residual encoding module, configured to parse a residual information bitstream corresponding to the current image patch to obtain residual information corresponding to the current image patch; to perform residual decoding on the residual information to obtain target residual features corresponding to the current image patch; and to determine a reconstructed image patch corresponding to the current image patch based on the target residual features and the target prediction features. 18. A decoding apparatus, characterized in that it is applied at a decoding end, the apparatus comprising: a prediction module, configured to parse a prediction information bitstream corresponding to a current image block to obtain prediction information corresponding to the current image block; perform prediction information decoding on the prediction information to obtain prediction mode features corresponding to the current image block; determine prediction features corresponding to the current image block based on the prediction mode features and historical image reference information; a residual decoding module, configured to parse a residual information bitstream corresponding to the current image block to obtain residual information corresponding to the current image block; perform residual decoding on the residual information to obtain target residual features corresponding to the current image block; determine initial image reference information and reconstructed image blocks corresponding to the current image block based on the target residual features and the prediction features; obtain a reference confidence level corresponding to the initial image reference information; wherein the reference confidence level represents the accuracy of the initial image reference information; determine target image reference information corresponding to the current image block based on the reference confidence level and the initial image reference information, the target image reference information being used for historical image reference information of subsequent image blocks.19. An encoding apparatus, characterized in that it is applied at an encoding end, the apparatus comprising: a prediction module, configured to parse a prediction information bitstream corresponding to a current image block to obtain prediction information corresponding to the current image block; to decode the prediction information to obtain prediction mode features corresponding to the current image block; and to determine prediction features corresponding to the current image block based on the prediction mode features and historical image reference information; a residual encoding module, configured to parse a residual information bitstream corresponding to the current image block to obtain residual information corresponding to the current image block; to perform residual decoding on the residual information to obtain target residual features corresponding to the current image block; to determine initial image reference information and a reconstructed image block corresponding to the current image block based on the target residual features and the prediction features; to obtain a reference confidence level corresponding to the initial image reference information; wherein the reference confidence level represents the accuracy of the initial image reference information; and to determine target image reference information corresponding to the current image block based on the reference confidence level and the initial image reference information, wherein the target image reference information is used as historical image reference information for subsequent image blocks. 20. A decoding device, characterized in that the decoding device comprises: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor; the processor is configured to execute the machine-executable instructions to implement the method according to any one of claims 1-11; or, the processor is configured to execute the machine-executable instructions to implement the method according to any one of claims 13-14. 21. An encoding device, characterized in that the encoding device comprises: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor; the processor is configured to execute the machine-executable instructions to implement the method according to claim 12 or 15. 22. A machine-readable storage medium, characterized in that the machine-readable storage medium stores a plurality of computer instructions, which, when executed by a processor, implement the method according to any one of claims 1-11, 13-14, or, when executed by a processor, implement the method according to claim 12 or 15. Claims 5 / 5 Page 6 CN 121603654 A Decoding and Encoding Method, Apparatus and Device Technical Field

[0001] This application relates to the field of encoding and decoding technology, and in particular to a decoding and encoding method, apparatus and device. Background Art

[0002] In order to save space, video images are encoded before transmission. Complete video encoding can include prediction, transformation, quantization, entropy coding, filtering and other processes. For the prediction process, the prediction process can include intra-frame predictionInter-frame prediction refers to using the correlation in the temporal domain of the video to predict the current pixel using pixels of neighboring encoded images, so as to effectively remove temporal redundancy in the video. Intra-frame prediction refers to using the correlation in the spatial domain of the video to predict the current pixel using pixels of the encoded blocks of the current frame image, so as to remove spatial redundancy in the video.

[0003] With the rapid development of deep learning, deep learning has achieved success in many high-level computer vision problems, such as image classification and object detection. Deep learning has also gradually begun to be applied in the field of encoding and decoding, that is, neural networks can be used to encode and decode images. Although neural network-based encoding and decoding methods have shown great performance potential, they still have problems such as poor encoding performance, poor decoding performance, and high complexity. Summary of the Invention

[0004] In view of this, this application provides a decoding and encoding method, apparatus, and device to improve encoding and decoding performance.

[0005] This application provides a decoding method applied to a decoding end, the method comprising:

[0006] parsing the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block; decoding the prediction information to obtain the prediction mode feature and prediction confidence level corresponding to the current image block; determining the target prediction feature corresponding to the current image block based on the prediction mode feature and the prediction confidence level; wherein, the prediction confidence level is obtained based on information in the prediction information bitstream, and the prediction confidence level represents the accuracy of the target prediction feature;

[0007] parsing the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; decoding the residual information to obtain the target residual feature corresponding to the current image block;

[0008] determining the reconstructed image block corresponding to the current image block based on the target residual feature and the target prediction feature.

[0009] This application provides an encoding method applied to an encoding end, the method comprising:

[0010] parsing the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block; decoding the prediction information to obtain the prediction mode feature and prediction confidence level corresponding to the current image block; determining the target prediction feature corresponding to the current image block based on the prediction mode feature and the prediction confidence level; wherein, the prediction confidence level is obtained based on information in the prediction information bitstream, and the prediction confidence level represents the accuracy of the target prediction feature;

[0011] parsing the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; decoding the residual information to obtain the target residual feature corresponding to the current image block;

[0012] determining the reconstructed image feature corresponding to the current image block based on the target residual feature and the target prediction feature.Page 1 / 54, 7 CN 121603654 A image block.

[0013] This application provides a decoding method applied to a decoding end, the method comprising:

[0014] parsing the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block; decoding the prediction information to obtain the prediction mode feature corresponding to the current image block; determining the prediction feature corresponding to the current image block based on the prediction mode feature and historical image reference information;

[0015] parsing the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; decoding the residual information to obtain the target residual feature corresponding to the current image block; determining the initial image reference information and the reconstructed image block corresponding to the current image block based on the target residual feature and the prediction feature; obtaining the reference confidence level corresponding to the initial image reference information; wherein, the reference confidence level represents the accuracy of the initial image reference information; determining the target image reference information corresponding to the current image block based on the reference confidence level and the initial image reference information, the target image reference information being used for the historical image reference information of subsequent image blocks.

[0016] This application provides an encoding method applied to an encoding end, the method comprising:

[0017] parsing the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block; decoding the prediction information to obtain the prediction mode feature corresponding to the current image block; determining the prediction feature corresponding to the current image block based on the prediction mode feature and historical image reference information;

[0018] parsing the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; decoding the residual information to obtain the target residual feature corresponding to the current image block; determining the initial image reference information and the reconstructed image block corresponding to the current image block based on the target residual feature and the prediction feature; obtaining the reference confidence level corresponding to the initial image reference information; wherein, the reference confidence level represents the accuracy of the initial image reference information; determining the target image reference information corresponding to the current image block based on the reference confidence level and the initial image reference information, the target image reference information being used for the historical image reference information of subsequent image blocks.

[0019] This application provides a decoding device applied at a decoding end. The device includes: a prediction module, configured to parse the prediction information bitstream corresponding to the current image block to obtain prediction information corresponding to the current image block; decode the prediction information to obtain prediction mode features and prediction confidence corresponding to the current image block; and determine target prediction features corresponding to the current image block based on the prediction mode features and the prediction confidence; wherein the prediction...The confidence level is obtained based on the information in the prediction information bitstream, and the prediction confidence level represents the accuracy of the target prediction feature;

[0020] The residual decoding module is used to parse the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; perform residual decoding on the residual information to obtain the target residual feature corresponding to the current image block; and determine the reconstructed image block corresponding to the current image block based on the target residual feature and the target prediction feature.

[0021] This application provides an encoding device applied at an encoding end. The device includes: a prediction module, used to parse the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block; to decode the prediction information to obtain the prediction mode feature and prediction confidence of the current image block; and to determine the target prediction feature corresponding to the current image block based on the prediction mode feature and the prediction confidence; wherein the prediction confidence is obtained based on the information in the prediction information bitstream, and the prediction confidence represents the accuracy of the target prediction feature; Specification 2 / 54 pages 8 CN 121603654 A

[0022] A residual encoding module, used to parse the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; to decode the residual information to obtain the target residual feature corresponding to the current image block; and to determine the reconstructed image block corresponding to the current image block based on the target residual feature and the target prediction feature.

[0023] This application provides a decoding device applied to a decoding end, the device comprising:

[0024] a prediction module, configured to parse the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block; to perform prediction information decoding on the prediction information to obtain the prediction mode feature corresponding to the current image block; to determine the prediction feature corresponding to the current image block based on the prediction mode feature and historical image reference information; a residual decoding module, configured to parse the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; to perform residual decoding on the residual information to obtain the target residual feature corresponding to the current image block; to determine the initial image reference information and the reconstructed image block corresponding to the current image block based on the target residual feature and the prediction feature; to obtain the reference confidence level corresponding to the initial image reference information; wherein, the reference confidence level represents the accuracy of the initial image reference information; and to determine the target image reference information corresponding to the current image block based on the reference confidence level and the initial image reference information, the target image reference information being used as historical image reference information for subsequent image blocks.

[0025] This application provides an encoding device applied at an encoding end, the device comprising:

[0026] A prediction module is used to parse the prediction information bitstream corresponding to the current image patch to obtain the prediction information corresponding to the current image patch; decode the prediction information to obtain the prediction mode feature corresponding to the current image patch; and determine the prediction feature corresponding to the current image patch based on the prediction mode feature and historical image reference information. A residual coding module is used to parse the residual information bitstream corresponding to the current image patch to obtain the residual information corresponding to the current image patch; perform residual decoding on the residual information to obtain the target residual feature corresponding to the current image patch; determine the initial image reference information and reconstructed image patch corresponding to the current image patch based on the target residual feature and the prediction feature; obtain the reference confidence level corresponding to the initial image reference information; wherein the reference confidence level represents the accuracy of the initial image reference information; and determine the target image reference information corresponding to the current image patch based on the reference confidence level and the initial image reference information, the target image reference information being used as historical image reference information for subsequent image patches.

[0027] This application provides a decoding end device, the decoding end device comprising: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor;

[0028] the processor is used to execute the machine-executable instructions to implement the above-described decoding method.

[0029] This application provides an encoding end device, the decoding end device comprising: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor;

[0030] the processor is used to execute the machine-executable instructions to implement the above-described encoding method.

[0031] This application provides an electronic device comprising: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor; the processor is used to execute the machine-executable instructions to implement the above-described decoding method; or, the processor is used to execute the machine-executable instructions to implement the above-described encoding method.

[0032] This application provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, implement the above-described decoding method; or, implement the above-described encoding method.

[0033] This application provides a computer program product comprising a computer program, which, when executed by a processor, implements the above-described decoding method; or, implements the above-described encoding method.

[0034] As can be seen from the above technical solutions, in the embodiments of this application, an end-to-end video image compression method is proposed, which implements video image encoding and decoding based on a neural network, enabling the neural network to effectively maintain low complexity while achieving high performance.This approach ensures the quality of the reconstructed image, thereby improving encoding and decoding performance while reducing complexity. It can utilize confidence levels for feature enhancement, thereby improving compression efficiency, enhancing objective image quality, and achieving good prediction results.

[0035] FIG1A is a flowchart illustrating a decoding method in one embodiment of the present application;

[0036] FIG1B is a flowchart illustrating an encoding method in one embodiment of the present application;

[0037] FIG2A is a flowchart illustrating a decoding method in one embodiment of the present application;

[0038] FIG2B is a flowchart illustrating an encoding method in one embodiment of the present application;

[0039] FIG3A is a schematic diagram illustrating the processing procedure of the encoding end in one embodiment of the present application;

[0040] FIG3B is a schematic diagram illustrating the processing procedure of the decoding end in one embodiment of the present application;

[0041] FIG4A is a schematic diagram illustrating the processing procedure of the encoding end in one embodiment of the present application;

[0042] FIG4B is a schematic diagram illustrating the processing procedure of the decoding end in one embodiment of the present application;

[0043] FIG5A is a schematic diagram illustrating the processing procedure of the encoding end in one embodiment of the present application;

[0044] FIG5B is a schematic diagram illustrating the processing procedure of the decoding end in one embodiment of the present application;

[0045] Figures 6A and 6B are schematic diagrams of the processing of the time-domain information mining module in one embodiment of this application;

[0046] Figure 7A is a schematic diagram of the processing of the encoding end in one embodiment of this application;

[0047] Figure 7B is a schematic diagram of the processing of the decoding end in one embodiment of this application;

[0048] Figure 8A is a schematic diagram of the processing of the encoding end in one embodiment of this application;

[0049] Figure 8B is a schematic diagram of the processing of the decoding end in one embodiment of this application;

[0050] Figures 9A-9D are schematic diagrams of the processing of the time-domain information mining module in one embodiment of this application;

[0051] Figure 10A is a schematic diagram of the processing of the encoding end in one embodiment of this application;

[0052] Figure 10B is a schematic diagram of the processing of the decoding end in one embodiment of this application;

[0053] Figure 11A is a schematic diagram of the residual confidence level output by the residual decoder in one embodiment of this application;

[0054] Figures 11B and 11C are schematic diagrams of the reference confidence level output by the residual decoder in one embodiment of this application;

[0055] Figure 12A is a hardware structure diagram of a decoding end device in one embodiment of this application;

[0056] Figure 12B is a hardware structure diagram of an encoding end device in one embodiment of this application. Detailed Embodiments

[0057] This application proposes a decoding method and an encoding method, which may involve the following concepts:

[0058] Entropy Encoding: Entropy encoding means that no information is lost during the encoding process according to the entropy principle.Encoding, information entropy is the average information content (a measure of uncertainty) of a source. Entropy coding methods can include, but are not limited to: Shannon coding, Huffman coding, and arithmetic coding.

[0059] Neural Network (NN): A neural network refers to an artificial neural network. A neural network is a computational model composed of a large number of interconnected nodes (or neurons). In a neural network, neuron processing units can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. The types of processing units in a neural network can be divided into three categories: input units, output units, and hidden units. Input units receive signals and data from the external world; Output units output the processing results; Hidden units are units located between input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units, and the representation and processing of information are reflected in the connection relationships between processing units. Neural networks are a non-programmed, brain-like information processing method. The essence of neural networks is to obtain a parallel distributed information processing function through the transformation and dynamic behavior of neural networks, and to imitate the information processing function of the human brain's nervous system to different degrees and levels. In the field of video processing, commonly used neural networks can include, but are not limited to: Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Fully Connected Networks.

[0060] Convolutional Neural Network (CNN): A convolutional neural network is a feedforward neural network and one of the most representative network structures in deep learning technology. The artificial neurons of a convolutional neural network can respond to a portion of the surrounding units within its coverage area, and it performs well in large-scale image processing. The basic structure of a convolutional neural network includes two layers. The first is the feature extraction layer (also called the convolutional layer), where the input of each neuron is connected to the local receptive field of the previous layer and extracts the local features. Once the local features are extracted, their positional relationship with other features is also determined. The second is the feature mapping layer (also called the activation layer). Each computational layer of the neural network consists of multiple feature maps. Each feature map is a plane, and all neurons on the plane have equal weights. For example, the feature mapping structure can use the Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, GDN function, etc. as the activation function of the convolutional network. Furthermore, since neurons on a mapping plane share weights, the number of free parameters in the network is reduced.

[0061] For example, one of the advantages of convolutional neural networks compared to image processing algorithms is that they avoid complex image processing...The complex preprocessing process (extracting artificial features, etc.) can be directly input into the original image for end-to-end learning. One of the advantages of convolutional neural networks compared to ordinary neural networks is that ordinary neural networks all use a fully connected approach, that is, all neurons from the input layer to the hidden layer are connected. This will result in a huge number of parameters, making network training time-consuming or even difficult. Convolutional neural networks avoid this difficulty through local connections, weight sharing, and other methods.

[0062] Deconvolution layer: The deconvolution layer is also called the transposed convolution layer. The working process of the deconvolution layer is very similar to that of the convolution layer. The main difference is that the deconvolution layer will use padding to make the output larger than the input (of course, it can also keep them the same). If the stride is 1, it means that the output size is equal to the input size; if the stride is N, it means that the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0063] Generalization Ability: Generalization ability refers to the ability of a machine learning algorithm to adapt to new samples. The purpose of learning is to learn the patterns hidden behind data pairs. For data outside the learning set that have the same pattern, the trained network can also give appropriate outputs. This ability can be called generalization ability.

[0064] Feature: The feature involved in this application is a three-dimensional feature matrix or tensor of size C*W*H. In the three-dimensional feature matrix, C can represent the number of channels, H can represent the feature height, and W can represent the feature width. The three-dimensional feature matrix can be the input of the neural network or the output of the neural network.

[0065] Rate-Distortion Optimized: There are two main indicators for evaluating coding efficiency: bit rate and PSNR (Peak Signal to Noise Ratio). The smaller the bit stream, the greater the compression ratio. The larger the PSNR, the better the quality of the reconstructed image. In mode selection, the discrimination formula is essentially a comprehensive evaluation of the two. For example, the cost corresponding to a mode is: J(mode) = D + λ*R, where D represents Distortion, which is usually measured using the SSE metric. SSE refers to the sum of the mean squares of the differences between the reconstructed image patch and the source image. To consider the cost, the SAD metric can also be used, which refers to the sum of the absolute values ​​of the differences between the reconstructed image patch and the source image. λ is a Lagrange multiplier, and R is the actual number of bits required for encoding the image patch in this mode, including the total number of bits required for encoding mode information, motion information, residuals, etc. When selecting a mode, if the rate-distortion principle is used to compare and decide on the encoding modes, the optimal encoding performance can usually be guaranteed.

[0066] Numerous encoding tools have been proposed for various modules of the encoding end, and each tool often has multiple modes. For different video sequences, the encoding tool that can achieve the best encoding performance is often different. Therefore, in the encoding process, RDO (Rate-Distortion Optimization) is usually used to compare the encoding performance of different tools or modes to select the best mode. After determining the optimal tool or mode, the decision information of the tool or mode is transmitted by encoding the marker information in the bit stream. Although this method brings higher encoding complexity, it can adaptively select the optimal mode combination for different content to obtain the best encoding performance. The decoding end can obtain the relevant mode information by directly parsing the marker information, and the complexity has a small impact.

[0067] The decoding method and encoding method in the embodiments of this application will be described in detail below with reference to several specific embodiments.

[0068] Example 1: This application proposes a decoding method. Referring to Figure 1A, which is a flowchart of the decoding method, this method can be applied to the decoding end (also known as a video decoder). The method may include:

[0069] Step 111: Parse the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block; decode the prediction information to obtain the prediction mode feature and prediction confidence corresponding to the current image block; determine the target prediction feature corresponding to the current image block based on the prediction mode feature and the prediction confidence. Wherein, the prediction confidence is obtained based on the information in the prediction information bitstream, and the prediction confidence represents the accuracy of the target prediction feature.

[0070] Step 112: Parse the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; decode the residual information to obtain the target residual feature corresponding to the current image block.

[0071] Step 113: Determine the reconstructed image block corresponding to the current image block based on the target residual feature and the target prediction feature.

[0072] For example, determining the target prediction feature corresponding to the current image patch based on the prediction pattern features and prediction confidence may include: performing temporal information mining based on the prediction pattern features, prediction confidence, and historical temporal reference features to obtain the initial prediction feature corresponding to the current image patch. The information buffer stores image reference information that can be referenced by the current image patch, and derives historical temporal reference features through the image reference information; determining the target prediction feature corresponding to the current image patch based on the initial prediction feature.

[0073] For example, performing temporal information mining based on the prediction pattern features, prediction confidence, and historical temporal reference features to obtain the initial prediction feature corresponding to the current image patch may include: performing temporal information mining based on the prediction pattern features and historical temporal reference features to obtain the initial prediction feature corresponding to the current image patch.The features are compensated to obtain temporal compensated features; based on the temporal compensated features and the prediction confidence, the initial prediction features are determined.

[0074] For example, the prediction pattern features may include a first prediction pattern feature and a second prediction pattern feature, the historical temporal reference features may include a first historical temporal reference feature and a second historical temporal reference feature, the prediction confidence may include a first prediction confidence and a second prediction confidence, and the initial prediction features may include a first initial prediction feature and a second initial prediction feature; based on the prediction pattern features, the prediction confidence and the historical temporal reference features, temporal information mining is performed to obtain the initial prediction features corresponding to the current image block, which may include: performing a compensation operation on the first historical temporal reference features based on the first prediction pattern features to obtain a first temporal compensated feature; determining the first initial prediction feature based on the first temporal compensated feature and the first prediction confidence; performing a compensation operation on the second historical temporal reference features based on the second prediction pattern features to obtain a second temporal compensated feature; and determining the second initial prediction feature based on the second temporal compensated feature and the second prediction confidence.

[0075] For example, determining the initial prediction features based on temporal compensated features and prediction confidence may include, but is not limited to: performing a weighted operation on the temporal compensated features and prediction confidence to obtain the post-operated features; performing a first type of operation on the post-operated features to obtain the initial prediction features; or performing a second type of operation on the temporal compensated features to obtain the post-operated features; and performing a weighted operation on the post-operated features and prediction confidence to obtain the initial prediction features.

[0076] For example, performing a first type of operation on the post-operated features to obtain the initial prediction features may include, but is not limited to: sequentially performing convolution and residual block operations on the post-operated features, and adding the features after the residual block operation to the post-operated features to obtain the initial prediction features. Furthermore, performing a second type of operation on the temporal compensated features to obtain the post-operated features may include, but is not limited to: sequentially performing convolution and residual block operations on the temporal compensated features, and adding the features after the residual block operation to the temporal compensated features to obtain the post-operated features.

[0077] For example, determining the target prediction feature corresponding to the current image block based on the initial prediction feature may include: determining the initial prediction feature as the target prediction feature; or, obtaining the target prediction feature after performing a transformation operation on the initial prediction feature; or, obtaining the target prediction feature by performing a weighted operation on the initial prediction feature and the residual confidence; wherein, the residual confidence is obtained when performing residual decoding on the residual information; wherein, the residual confidence is obtained based on the information in the residual information bitstream, and the residual confidence represents the accuracy of the target prediction feature; or, performing a weighted operation on the initial prediction feature and the residual confidence to obtain the weighted feature, and obtaining the target prediction feature after performing a transformation operation on the weighted feature.

[0078] For example, determining the target prediction feature corresponding to the current image patch based on the prediction mode features and prediction confidence may include: performing temporal information mining based on the prediction mode features and historical temporal reference features to obtain initial prediction features; wherein, the information buffer stores image reference information that can be referenced by the current image patch, and the historical temporal reference features are derived through the image reference information; determining the target confidence corresponding to the initial prediction feature based on the prediction confidence; wherein, the target confidence is obtained based on the information in the prediction information bitstream and / or the information in the residual information bitstream, and the target confidence represents the accuracy of the target prediction feature. On this basis, the initial prediction feature and the target confidence are weighted to obtain the target prediction feature; or, the initial prediction feature and the target confidence are weighted to obtain the weighted feature, and the weighted feature is transformed to obtain the target prediction feature.

[0079] For example, determining the target confidence level corresponding to the initial prediction feature based on the prediction confidence level may include, but is not limited to: determining the prediction confidence level as the target confidence level; or, obtaining the target confidence level after performing a transformation operation on the prediction confidence level; or, generating the target confidence level based on the prediction confidence level and the residual confidence level.

[0080] For example, parsing the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block may include, but is not limited to: parsing the prediction side information bitstream corresponding to the current image block to obtain prediction prior information; obtaining temporal prediction prior information from the prediction latent buffer, wherein the temporal prediction prior information may be prediction information that can be referenced by the current image block; generating a probability distribution of prediction information based on the prediction prior information and the temporal prediction prior information, using the probability distribution to parse the prediction information bitstream to obtain the prediction information corresponding to the current image block, and storing the prediction information in the prediction latent buffer for temporal prediction prior information of subsequent image blocks.

[0081] For example, parsing the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block may include, but is not limited to: parsing the residual edge information bitstream corresponding to the current image block to obtain residual prior information; obtaining residual prior information from the residual latent buffer, wherein the residual prior information may be residual information that can be referenced by the current image block; performing spatial prior processing based on the initial prediction features corresponding to the current image block to obtain spatial prior information; or performing spatial prior processing based on the target prediction features corresponding to the current image block to obtain spatial prior information. Based on this, a probability distribution of residual information can be generated based on the residual prior information, the residual prior information, and the spatial prior information. The residual information bitstream corresponding to the current image block can be parsed using this probability distribution to obtain the residual information corresponding to the current image block, and the residual information can be stored in the residual latent buffer for the residual prior information of subsequent image blocks.

[0082] This application also proposes an encoding method, as shown in Figure 1B, which is a flowchart of the encoding method. This method can be applied to the encoding end (also known as a video encoder). The method may include:

[0083] Step 121: Parse the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block; decode the prediction information to obtain the prediction mode features and prediction confidence corresponding to the current image block; determine the target prediction features corresponding to the current image block based on the prediction mode features and prediction confidence. Wherein, the prediction confidence is obtained based on the information in the prediction information bitstream, and the prediction confidence represents the accuracy of the target prediction features.

[0084] Step 122: Parse the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; decode the residual information to obtain the target residual features corresponding to the current image block.

[0085] Step 123: Determine the reconstructed image block corresponding to the current image block based on the target residual feature and the target prediction feature.

[0086] For example, the processing procedure at the encoding end is similar to that at the decoding end, and the similarities will not be repeated. The processing procedure at the decoding end can be applied to the encoding end, that is, the encoding end adopts the same processing method.

[0087] For example, the above execution order is only an example given for the convenience of description. In actual applications, the execution order between steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the method may include more or fewer steps than described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps in other embodiments; multiple steps described in this specification may also be combined into a single step in other embodiments.

[0088] As can be seen from the above technical solutions, in this application embodiment, an end-to-end video image compression method is proposed, which realizes the encoding and decoding of video images based on neural networks, so that the neural network can effectively guarantee the quality of the reconstructed image while maintaining low complexity, thereby achieving the purpose of improving encoding performance and decoding performance and reducing complexity. It can use confidence to perform feature enhancement, thereby improving compression efficiency, improving the objective quality of the image, and the prediction effect is good.

[0089] Embodiment 2: In this application embodiment, a decoding method is proposed. See Figure 2A, which is a flowchart of the decoding method. This method can be applied to the decoding end (also known as a video decoder). The method may include:

[0090] Step 211: Parse the prediction information bitstream corresponding to the current image block to obtain the prediction information bitstream corresponding to the current image block.

[0091] Step 212: Parse the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; decode the residual information to obtain the target residual feature corresponding to the current image block.

[0092] Step 213: Determine the initial image reference information and reconstructed image block corresponding to the current image block based on the target residual feature and the prediction feature.

[0093] Step 214: Obtain the reference confidence level corresponding to the initial image reference information. Wherein, the reference confidence level represents the accuracy of the initial image reference information.

[0094] Step 215: Determine the target image reference information corresponding to the current image block based on the reference confidence level and the initial image reference information. The target image reference information is used for the historical image reference information of subsequent image blocks.

[0095] For example, when obtaining the reference confidence level corresponding to the initial image reference information, the reference confidence level is obtained when performing residual decoding on the residual information; or, the reference confidence level is obtained when performing prediction information decoding on the prediction information; or, a first reference feature is obtained when performing residual decoding on the residual information, and a second reference feature is obtained when performing prediction information decoding on the prediction information, and the reference confidence level is obtained based on the first reference feature and the second reference feature.

[0096] An encoding method is also proposed in this application embodiment. Referring to Figure 2B, which is a flowchart of the encoding method, page 8 / 54 of the specification, 14 CN 121603654 A, the method can be applied to the encoding end (also known as a video encoder). The method may include:

[0097] Step 221: Parse the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block; perform prediction information decoding on the prediction information to obtain the prediction mode feature corresponding to the current image block; determine the prediction feature corresponding to the current image block based on the prediction mode feature and historical image reference information.

[0098] Step 222: Parse the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; perform residual decoding on the residual information to obtain the target residual features corresponding to the current image block.

[0099] Step 223: Determine the initial image reference information and reconstructed image block corresponding to the current image block based on the target residual features and prediction features.

[0100] Step 224: Obtain the reference confidence level corresponding to the initial image reference information. The reference confidence level represents the accuracy of the initial image reference information.

[0101] Step 225: Determine the target image reference information corresponding to the current image block based on the reference confidence level and the initial image reference information. The target image reference information is used for the historical image reference information of subsequent image blocks.

[0102] For example, the processing process at the encoding end is similar to that at the decoding end, and the similarities will not be repeated. The processing process at the decoding end can be applied to the encoding end, that is, the encoding end adopts the same processing method.

[0103] For example, the above execution order is only an example given for the convenience of description. In actual applications, the execution order between steps can also be changed, and there is no restriction on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the method may include more or fewer steps than described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps in other embodiments; multiple steps described in this specification may also be combined into a single step in other embodiments.

[0104] As can be seen from the above technical solutions, in the embodiments of this application, an end-to-end video image compression method is proposed, which realizes the encoding and decoding of video images based on neural networks, so that the neural network can effectively guarantee the quality of the reconstructed image while maintaining low complexity, thereby achieving the purpose of improving encoding performance and decoding performance and reducing complexity. It can utilize confidence to enhance features, thereby improving compression efficiency, improving the objective quality of the image, and achieving good prediction results.

[0105] Example 3: The intelligent video coding framework may include a preprocessing module, a prediction module, a residual coding module, an entropy coding module, an in-loop filtering module (i.e., a filtering module), and a postprocessing module. Among them, the preprocessing module is used to process the input data to make the input data meet the coding requirements. For example, it can perform enhancement processing and / or smoothing processing, upsampling processing, downsampling processing, channel cropping processing, feature extraction processing, etc. The input data may be images, videos, features, etc.

[0106] The prediction module is used to derive prediction information using the reconstructed features and / or image information. The prediction information may include one or more of the multi-dimensional information such as spatial domain and temporal domain. For example, temporal domain prediction information may be optical flow information or motion vector information. For example, spatial domain prediction information may be context information, texture information, intra-frame prediction information, etc.

[0107] Residual coding module: Used to process the difference between the predicted information (i.e., the predicted information exported by the prediction module) and the true information. This difference can be in various forms, such as the difference in the feature domain or the difference in the pixel domain. Of course, it can also include the difference in the pixel domain and the difference in the feature domain at the same time, without limitation.

[0108] Entropy coding module: Used to encode the predicted information losslessly if the predicted information exported by the prediction module exists, and to encode the residual information losslessly if the residual information exported by the residual coding module exists. The encoded data needs to be informed to the decoding end.

[0109] In-loop filtering module: Used to process the output data so that the output data meets the subsequent prediction requirements. For example,The output data is enhanced and / or smoothed, upsampled, downsampled, clipped, and feature extracted. The output data can be images, videos, features, etc. The output of the loop filtering module can be input to the prediction module.

[0110] Post-processing module: used to process the output data to make it meet the requirements of subsequent applications. For example, the output data is enhanced and / or smoothed, upsampled, downsampled, clipped, and feature extracted. The output data can be images, videos, features, etc. For example, the output of the post-processing module should not be input to the prediction module, which means that the incremental information of the post-processing module will not be perceived by the prediction module.

[0111] It should be noted that some modules of the pre-processing module, prediction module, residual coding module, entropy coding module, in-loop filtering module and post-processing module are optional. For example, the pre-processing module, in-loop filtering module and post-processing module are optional. Multiple modules can exist, for example, there can be two in-loop filtering modules.

[0112] Each module can be in the form of a neural network or a non-neural network. For example, a preprocessing module can be implemented using a neural network or a non-neural network. A prediction module can be implemented using a neural network or a non-neural network. A residual coding module can be implemented using a neural network or a non-neural network. An entropy coding module can be implemented using a neural network or a non-neural network. An in-loop filtering module can be implemented using a neural network or a non-neural network, such as performing non-neural network filtering (e.g., ALF filtering) on ​​the output image. A post-processing module can be implemented using a neural network or a non-neural network.

[0113] Based on the structure of the neural network, some modules can be fused. For example, a module can have both prediction and residual coding functions, that is, the functions of the prediction module and the residual coding module can be integrated into the same module.

[0114] For intelligent video coding frameworks, this application proposes an end-to-end video coding exploration experiment model. Referring to Figure 3A, which is a schematic diagram of the video coding exploration experiment model, this model may include a lower prediction module and an upper residual coding module. The prediction module can perform temporal prediction, and the residual coding module can perform spatial prediction. The processing procedure of this video coding exploration experiment model is described below.

[0115] 1. After obtaining the current image block xt, the encoding end can input the current image block xt to the feature extractor.The feature extractor extracts the current image features Ft corresponding to the current image block xt, i.e., the image domain is transformed to the feature domain.

[0116] After the encoder obtains the current frame image (the current frame image is the original image, i.e., the input image), the current frame image can be divided into one image block or multiple image blocks. If the current frame image is divided into one image block, then the current image block xt is the current frame image, i.e., the current frame image is processed. If the current frame image is divided into multiple image blocks, then each image block of the current frame image is processed separately, and the processing method of each image block is the same. Taking the current image block xt as an example.

[0117] The feature extractor is used to extract the image features of the current image block xt. Therefore, after inputting the current image block xt to the feature extractor, the current image features Ft corresponding to the current image block xt can be obtained.

[0118] 2. The information buffer (Decoded Feature & Picture Buffer, also known as the decoded image / feature buffer) stores the image reference information that can be referenced by the current image block xt. The historical temporal reference features are derived from the image reference information.

[0119] The image reference information can be the image reference information of the image block before the current image block xt (such as the historical image block, which can be located in the same frame or in a different frame as the current image block). The image reference information can also be the image reference information of the image frame before the current frame. See the following steps for the method of obtaining the image reference information.

[0120] The image reference information can be input to the feature extractor to obtain the historical temporal reference features. The image reference information can also be directly used as the historical temporal reference features. There are no restrictions on the method of obtaining the historical temporal reference features, as long as the historical temporal reference features can be derived from the image reference information.

[0121] 3. The information buffer (Decoded Feature & Picture Buffer) stores historical prior prediction features. Historical prior prediction features are the prior prediction features that the current image block xt can refer to. The method for obtaining these features is described in subsequent steps.

[0122] Historical prior prediction features can be the prior prediction features of image blocks preceding the current image block xt (such as historical image blocks, which can be located in the same frame as the current image block or in different frames). Historical prior prediction features can also be the prior prediction features of image frames preceding the current frame. There are no restrictions on this. In addition, historical prior prediction features can also be called historical prior motion features.

[0123] 4. The encoder performs motion estimation on the current image block xt and the historical image block xt-1 to obtain optical flow information vt. The historical image block xt-1 can be an image block preceding the current image block xt (this image block can be in the same frame as the current image block, or it can be in a different frame), or it can be an image from an image frame preceding the current frame.

[0124] 5. The encoder inputs the current image feature Ft, optical flow information vt, historical temporal reference features, and historical prior motion features to the predictive encoder (Motion Encoder). The predictive encoder encodes based on the current image feature Ft, optical flow information vt, historical temporal reference features, and historical prior motion features to obtain unquantized predictive latent features mt.

[0125] For example, the predictive encoder can also be called a motion encoder. The predictive encoder can be a neural network used to implement predictive information encoding (predictive information encoding can also be called motion information encoding). Based on this, after inputting the current image features Ft, optical flow information vt, historical temporal reference features, and historical prior motion features into the predictive encoder, the predictive encoder can process these features. There are no restrictions on the structure and processing method of the predictive encoder, and the predictive latent features mt are obtained. For example, the predictive latent features mt can also be called motion latent features mt.

[0126] 6. The encoder performs quantization (i.e., Q operation) on the predictive latent features mt to obtain the prediction information, i.e., the prediction information features. There are no restrictions on this quantization operation process. For example, the prediction information can also be called motion information features.

[0127] 7. The Motion Latent Buffer (prediction latent buffer, which can also be called motion latent buffer) stores temporal prediction prior information (temporal prediction prior information can also be called temporal motion prior information).

[0128] The temporal prediction prior information can be the prediction information of the image blocks preceding the current image block xt (such as historical image blocks, which may be located in the same frame as the current image block or in different frames). The temporal prediction prior information can also be the prediction information of the image frames preceding the current frame.

[0129] The prediction latent feature mt and the temporal prediction prior information are input to the Hyper Enc (hyper-encoder). After obtaining the prediction latent feature mt and the temporal prediction prior information, the hyper-encoder encodes the prediction latent feature mt and the temporal prediction prior information to obtain the prediction side information bitstream (the prediction side information bitstream can also be called the motion side information bitstream). After obtaining the prediction side information bitstream, the encoding end can also transmit the prediction side information bitstream to the Hyper Enc at the decoding end.Regarding the processing at the decoding end, please refer to the following embodiments. Specification 11 / 54 pages 17 CN 121603654 A

[0130] 8. The Hyper Dec (hyper-prior decoder) at the encoding end can obtain the prediction side information bitstream and parse it to obtain the prediction hyper-prior information (Hyper Prior, such as motion hyper-prior information).

[0131] The prediction hyper-prior information (Hyper Prior) and the temporal prediction prior information (Temporal Prior) are input to the Motion Entropy Model (also called the motion entropy model). The prediction entropy model generates a probability distribution of the prediction information based on the prediction hyper-prior information and the temporal prediction prior information. This probability distribution is used to guide encoding and decoding.

[0132] 9. The encoding end encodes the prediction information based on the probability distribution of the prediction information (i.e., AE operation) to obtain the prediction information bitstream (also called the motion information bitstream). After obtaining the prediction information bitstream, the encoding end transmits the prediction information bitstream to the decoding end. The processing of the prediction information bitstream by the decoding end is described in the following process.

[0133] For example, a probability distribution model can be determined based on the probability distribution of the prediction information (such as probability distribution parameters), and then the prediction information is encoded based on the probability distribution model to obtain the prediction information bitstream.

[0134] 10. The encoding end parses the prediction information bitstream based on the probability distribution of the prediction information (i.e., AD operation) to obtain the prediction information. The prediction information can also be dequantized to obtain the prediction latent feature mt.

[0135] For example, after obtaining the prediction information bitstream, a probability distribution model can be determined based on the probability distribution of the prediction information (such as probability distribution parameters), and then the prediction information bitstream is parsed based on the probability distribution model.

[0136] For example, the prediction information may include one or more of multi-dimensional information such as spatial domain and temporal domain. For example, temporal domain prediction information may be optical flow information or motion vector information. For example, spatial domain prediction information may be context information, texture information, intra-frame prediction information, etc. There are no restrictions on this prediction information.

[0137] 11. After obtaining the prediction information (i.e., parsing the prediction information bitstream to obtain the prediction information), the encoding end stores the prediction information in the Motion Latent Buffer for use as prior information for temporal prediction of subsequent image blocks.

[0138] 12. The encoding end inputs the prediction information to the prediction decoder (Motion Decoder), which can also be called a motion decoder. The prediction decoder decodes the prediction information based on the prediction information (such as performing motion information decoding).The structure and decoding method of the predictive decoder are not restricted, and the predictive mode features and prior predictive features are obtained. The predictive mode features can also be called optical flow features, and the prior predictive features can also be called prior motion features.

[0139] The prior predictive features are stored in the information buffer (Decoded Feature & Picture Buffer), and the prior predictive features can be used as the historical prior predictive features of subsequent image blocks. Its function has been introduced above.

[0140] For example, a motion decoder may include multiple network layers. When the motion decoder decodes the prediction information based on the prediction information, the features output by the penultimate network layer of the motion decoder can be used as prior prediction features. Of course, the above is just an example, and the output features of other network layers (such as the penultimate network layer) can also be used as prior prediction features. The features output by the last network layer of the motion decoder can be used as prediction mode features. Of course, the above is just an example, and the output features of other network layers can also be used as prediction mode features.

[0141] 13. The encoding end inputs the prediction mode features and historical temporal reference features into the temporal residual mining module. The temporal residual mining module can be implemented by a neural network or by a non-neural network. The temporal residual mining module can perform temporal residual mining based on the prediction mode features and historical temporal reference features to obtain prediction features, which are used for residual conditional coding. For example, the temporal residual mining module can obtain a predicted feature and output the predicted feature. The temporal residual mining module can also obtain two predicted features and output the predicted features. In Figure 3A, the output of the predicted features is taken as an example.

[0142] 14. The encoding end inputs the current image feature FT and the predicted feature (such as the predicted feature and the predicted feature) to the residual encoder. The residual encoder can perform residual encoding based on the current image feature FT and the predicted feature (such as the predicted feature and the predicted feature) to obtain the unquantized residual information feature yT.

[0143] For example, the residual encoder can be a neural network used to implement residual encoding. After the current image feature FT and the predicted feature are input to the residual encoder, the residual encoder can process based on the current image feature FT and the predicted feature. The structure and processing method of the residual encoder are not limited to obtain the unquantized residual information feature yT.

[0144] 15. The encoding end quantizes the unquantized residual information feature yt (i.e., Q operation) to obtain the residual information, i.e., the quantized residual information feature yt, as the residual information.It can also be called residual latent features

[0145] 16. The encoder inputs the predicted features (predicted features) to the Spatial Prior Enc. The Spatial Prior Enc processes the predicted features without any restrictions, and obtains the Spatial Prior information.

[0146] 17. The Contextual Latent Buffer stores the residual prior information. The residual prior information is the residual latent features (i.e., residual information) of the image blocks preceding the current image block xt (such as historical image blocks, which can be located in the same frame as the current image block or in different frames). The residual prior information can also be the residual latent features (i.e., residual information) of the image frames preceding the current frame.

[0147] The encoder can input the residual information features yt, the Spatial Prior information, and the residual prior information to the Hyper Enc (Hyper Encoder). The hyper-prior encoder encodes the residual information feature yt, the spatial prior information, and the residual prior information to obtain the residual side information bitstream. After obtaining the residual side information bitstream, the encoder can also send the residual side information bitstream to the decoder. The processing of the decoder is described in the following process.

[0148] 18. After obtaining the residual side information bitstream, the Hyper Dec (hyper-prior decoder) of the encoder can parse the residual side information bitstream to obtain the residual hyper-prior information.

[0149] 19. The encoding end inputs the residual hyperprior information, spatial prior information, and residual temporary prior information into the Contextual Entropy Model. The residual entropy model generates a probability distribution of residual latent features based on the residual hyperprior information, spatial prior information, and residual temporary prior information. The probability distribution of residual latent features is used to guide encoding and decoding.

[0150] 20. The encoding end encodes the residual latent features based on the probability distribution of residual latent features (i.e., AE operation) to obtain the residual information bitstream. There are no restrictions on this encoding process. After obtaining the residual information bitstream, the encoding end can also transmit the residual information bitstream to the decoding end. The processing process of the residual information bitstream by the decoding end is described in the following process.

[0151] For example, the encoding end can determine the probability based on the probability distribution of residual latent features (such as probability distribution parameters).The probability distribution model is then used to encode the residual latent features to obtain the residual information code stream.

[0152] 21. The encoding end parses the residual information code stream based on the probability distribution of the residual latent features (i.e., AD operation) to obtain the residual latent features. The residual latent features can also be dequantized to obtain the residual information features yt. Instruction manual, pages 13 / 54, 19 CN 121603654 A

[0153] For example, a probability distribution model can be determined based on the probability distribution of the residual latent features (such as probability distribution parameters), and then the residual information bitstream can be parsed based on the probability distribution model to obtain the residual latent features

[0154] 22. After obtaining the residual latent features (also called residual information), the encoder can store the residual latent features in the Contextual Latent Buffer for use as residual prior information for subsequent image blocks

[0155] 23. The encoder inputs the residual latent features and prediction features (such as prediction features) to the residual decoder. The residual decoder decodes the residual information based on the residual latent features and prediction features (such as prediction features). The structure and function of the residual decoder are not restricted, and the target residual features and residual confidence α are obtained. For example, the residual confidence α can represent the accuracy of the prediction features (also called confidence level). Since the residual confidence α is obtained by decoding the residual information (residual latent features), and the residual information is parsed from the residual information bitstream, the residual confidence is based on the information in the residual information bitstream.

[0156] For example, as shown in Figure 3A, after the residual latent features are input to the residual decoder, the residual decoder can have two outputs, one output as the target residual feature and the other output as the residual confidence α.

[0157] 24. The encoding end performs a weighted operation (such as multiplication) on the predicted features (e.g., predicted features) and the residual confidence α, and performs an addition operation on the weighted features and the target residual features. The added features are then input to the frame generator, which generates image reference information and stores it in the information buffer (Decoded Feature & Picture Buffer) for use as image reference information for subsequent image blocks. Alternatively, the encoding end can process the image reference information (e.g., feature extraction) and store the processed features in the information buffer for use as image reference information for subsequent image blocks. The image reference information can be image domain information or feature domain information.

[0158] For example, if the prediction effect of the prediction module is poor, the predicted features (e.g., predicted features) output by the prediction module are not processed.The prediction is inaccurate, meaning the accuracy of the predicted features is poor. When determining image reference information based on the predicted features and the target residual features, the image reference information is inaccurate, resulting in a large residual in the residual coding module and poor overall compression efficiency.

[0159] In response to the above findings, when the prediction of the prediction module is inaccurate, in order to explicitly reduce the dependence on the prediction, after inputting the residual latent features to the residual decoder, the residual decoder can output the target residual features and the residual confidence α. The target residual features are used to represent the residual information, and the residual confidence α is used to represent the confidence in the prediction information and the reference information.

[0160] Based on this, the features input to the frame generator can be represented by the following expression: f represents the features input to the frame generator (i.e., the features after addition), represents the weighted operation on the predicted features and the residual confidence α, and represents the addition operation on the weighted features and the target residual features.

[0161] As can be seen from the above expression, for regions where the prediction module predicts accurately, i.e., the predicted features are accurate, the residual confidence α is close to 1, making the features input to the frame generator close to the sum of the predicted features and the target residual features. For regions where the prediction module predicts with low accuracy, i.e., the predicted features are inaccurate, the residual confidence α is close to 0, making the features input to the frame generator close to the target residual features. The residual coding degenerates into I-frame coding. Based on the above processing, the accuracy of the image reference information can be improved, the residual of the residual coding module can be reduced, and the overall compression efficiency can be improved.

[0162] 25. The encoding end generates reconstructed image blocks based on the features output by the frame generator. Specification 14 / 54 pages 20 CN 121603654 A

[0163] Thus, the processing of the encoding end is completed, and the reconstructed image block corresponding to the current image block xt can be obtained.

[0164] In one possible implementation, the above processing of the encoding end can be executed by a deep learning model or a neural network model, thereby realizing the end-to-end image compression and encoding process. There are no restrictions on this encoding process. The above-mentioned deep learning model or neural network model can be called an end-to-end video coding exploration experiment model. The framework of the video coding exploration experiment model is shown in Figure 3A. Based on the video coding exploration experiment model, the image is transformed into the feature domain.

[0165] For example, after dividing the current frame image into multiple image blocks, after performing the above processing on each image block of the current frame image, the reconstructed image block corresponding to each image block can be obtained, and the reconstructed image blocks corresponding to all image blocks constitute the reconstructed image corresponding to the current frame image. In addition, when storing image reference information in the information buffer, after storing the image reference information corresponding to all image blocks, it is actually storing the image corresponding to the current frame image.Reference information (i.e., image reference information of the entire frame), that is, the image reference information of the entire frame is stored in the information buffer.

[0166] For the intelligent video decoding framework, this application embodiment proposes an end-to-end video decoding exploration experiment model. See Figure 3B, which is a schematic diagram of the structure of the video decoding exploration experiment model. The video decoding exploration experiment model may include a prediction module in the lower half and a residual decoding module in the upper half. The prediction module can realize temporal prediction, and the residual decoding module can realize spatial prediction. The processing process of the video decoding exploration experiment model is described below.

[0167] 1. The decoding end obtains the prediction side information bitstream (also called motion side information bitstream).

[0168] 2. The Hyper Dec (hyper-prior decoder) of the decoding end can obtain the prediction side information bitstream and parse the prediction side information bitstream to obtain the prediction hyper-prior information (Hyper Prior, such as motion hyper-prior information).

[0169] The decoding end inputs the prediction super-prior information and the temporal prediction prior information to the Motion Entropy Model (also known as the motion entropy model). The prediction entropy model generates a probability distribution of prediction information based on the prediction super-prior information and the temporal prediction prior information. This probability distribution is used to guide the decoding.

[0170] For example, the Motion Latent Buffer (also known as the motion latent buffer) stores temporal prediction prior information (also known as temporal motion prior information). The temporal prediction prior information can be the prediction information of the image block before the current image block xt (such as a historical image block, which can be in the same frame as the current image block or in a different frame). The temporal prediction prior information can also be the prediction information of the image frame before the current frame.

[0171] 3. The decoding end obtains the prediction information bitstream (also known as the motion information bitstream).

[0172] 4. The decoding end parses the predicted information bitstream based on the probability distribution of the predicted information (i.e., AD operation) to obtain the predicted information. The predicted information can also be called the predicted information feature or motion information feature. The predicted information can also be dequantized to obtain the predicted latent feature mt, which can also be called the motion latent feature mt.

[0173] For example, after obtaining the predicted information bitstream, a probability distribution model can be determined based on the probability distribution of the predicted information (such as probability distribution parameters), and then the predicted information bitstream can be parsed based on the probability distribution model.

[0174] For example, the predicted information may include one or more of multi-dimensional information such as spatial domain and temporal domain. For exampleIn other words, temporal prediction information can be optical flow information or motion vector information. For example, spatial prediction information can be context information, texture information, intra-frame prediction information, etc. There are no restrictions on the prediction information. Instruction manual, pages 15 / 54, 21 CN 121603654 A

[0175] 5. After obtaining the prediction information (i.e., parsing the prediction information bitstream to obtain the prediction information), the decoding end stores the prediction information in the Motion Latent Buffer for the temporal prediction prior information of subsequent image blocks.

[0176] 6. The decoding end inputs the prediction information to the prediction decoder (Motion Decoder), which can also be called a motion decoder. The prediction decoder decodes the prediction information based on the prediction information (such as decoding motion information). There are no restrictions on the structure and decoding method of the prediction decoder. The prediction mode features and prior prediction features are obtained. The prediction mode features can also be called optical flow features. The prior prediction features can also be called prior motion features.

[0177] For example, the prior prediction features can be stored in the information buffer, and the prior prediction features can be used as the historical prior prediction features of subsequent image blocks. Of course, the prediction decoder may not output the prior prediction features.

[0178] For example, the prediction decoder (Motion Decoder) The decoder can include multiple network layers. When the predictive decoder decodes the predictive information based on the predictive information, the features output by the penultimate network layer of the predictive decoder can be used as prior predictive features. Of course, the above is just an example. The output features of other network layers (such as the penultimate network layer, etc.) can also be used as prior predictive features. The features output by the last network layer of the predictive decoder can be used as predictive pattern features. Of course, the above is just an example. The output features of other network layers can also be used as predictive pattern features.

[0179] 7. The decoding end inputs the predictive pattern features and historical temporal reference features into the temporal residual mining module. The temporal residual mining module can be implemented by a neural network or by a non-neural network. The temporal residual mining module can perform temporal residual mining based on the predictive pattern features and historical temporal reference features to obtain predictive features. These predictive features are used for residual conditional decoding. For example, the temporal residual mining module can obtain one predictive feature and output the predictive feature. The temporal residual mining module can also obtain two predictive features and output the predictive features and predictive features. In Figure 3B, the output predictive features and predictive features are taken as an example.

[0180] For example, the information buffer (Decoded Feature & Picture Buffer, also known as the decoded image / feature buffer) stores image reference information that the current image block xt can refer to.Historical temporal reference features can be derived from image reference information. Image reference information can be the image reference information of the image block preceding the current image block xt (such as a historical image block, which can be in the same frame or a different frame as the current image block). Image reference information can also be the image reference information of the image frame preceding the current frame. Image reference information can be input to the feature extractor to obtain historical temporal reference features. Alternatively, image reference information can be directly used as historical temporal reference features, as long as historical temporal reference features can be derived from image reference information.

[0181] 8. The decoding end obtains the residual side information bitstream.

[0182] 9. After obtaining the residual side information bitstream, the Hyper Dec (hyper-prior decoder) at the decoding end can parse the residual side information bitstream to obtain residual hyper-prior information.

[0183] 10. The decoding end inputs the predicted features (predicted features) to the Spatial Prior Encoder. The Spatial Prior Encoder processes the predicted features without restriction, and obtains the Spatial Prior information. (See specification 16 / 54 pages, 22 CN 121603654 A)

[0184] 11. The Contextual Latent Buffer stores residual prior information. The residual prior information is the residual potential features (i.e., residual information) of the image blocks preceding the current image block xt (such as historical image blocks, which can be in the same frame or different frames). The residual prior information can also be the residual potential features (i.e., residual information) of the image frames preceding the current frame.

[0185] 12. The decoding end inputs the residual hyperprior information, spatial prior information, and residual temporary prior information into the Contextual Entropy Model. The residual entropy model generates a probability distribution of residual latent features based on the residual hyperprior information, spatial prior information, and residual temporary prior information. The probability distribution of residual latent features is used to guide the decoding.

[0186] 13. The decoding end obtains the residual information bitstream.

[0187] For example, the prediction side information bitstream, prediction information bitstream, residual side information bitstream, and residual information bitstream all correspond to the current image block xt. For the current frame image (the current frame image is the original image, i.e., the input image), the current frame...An image can be divided into one image block or multiple image blocks. If the current frame image is divided into one image block, then the current image block xt is the current frame image, that is, the current frame image is processed. If the current frame image is divided into multiple image blocks, then each image block of the current frame image is processed separately, and the processing method of each image block is the same. Taking the current image block xt as an example.

[0188] 14. The decoding end parses the residual information bitstream based on the probability distribution of the residual latent features (i.e., AD operation) to obtain the residual latent features. The residual latent features can also be dequantized to obtain the residual information features yt.

[0189] For example, a probability distribution model can be determined based on the probability distribution of the residual latent features (such as probability distribution parameters), and then the residual information bitstream can be parsed based on the probability distribution model to obtain the residual latent features.

[0190] 15. After obtaining the residual latent features (also called residual information), the decoding end can store the residual latent features in the Contextual Latent Buffer for use as residual prior information for subsequent image blocks.

[0191] 16. The decoding end inputs the residual latent features and prediction features (such as prediction features) to the residual decoder. The residual decoder decodes the residual information based on the residual latent features and prediction features (such as prediction features). The structure of the residual decoder is not restricted, and the target residual features and residual confidence α are obtained. For example, the residual confidence α can represent the accuracy of the prediction features (also called confidence level). Since the residual confidence α is obtained by decoding the residual information (residual latent features), and the residual information is parsed from the residual information bitstream, the residual confidence is based on the information in the residual information bitstream.

[0192] For example, as shown in Figure 3B, after the residual latent features are input to the residual decoder, the residual decoder can have two outputs: one output as the target residual feature and the other output as the residual confidence α.

[0193] 17. The decoding end performs a weighted operation (such as multiplication) on the predicted features (e.g., the predicted features) and the residual confidence α, and performs an addition operation on the weighted features and the target residual features. The added features are then input to the frame generator, which generates image reference information and stores it in the information buffer (Decoded Feature & Picture Buffer). The image reference information is used as the image reference information for subsequent image blocks. Alternatively, the decoding end can also process the image reference information (e.g., feature extraction), store the processed features in the information buffer, and use the processed features as the image reference information for subsequent image blocks.

[0194] For example, the features input to the frame generator can be represented by the following expression: f can be represented by the features input to the frame generator (i.e., the features after addition operation), can be represented by the weighted operation of the predicted features and the residual confidence α, and can be represented by the addition operation of the weighted features and the target residual features.

[0195] 18. The decoding end generates reconstructed image blocks based on the features input to the frame generator

[0196] Thus, the processing of the decoding end is completed, and the reconstructed image block corresponding to the current image block xt can be obtained

[0197] In one possible implementation, the above-mentioned processing of the decoding end can be executed by a deep learning model or a neural network model, thereby realizing the end-to-end image compression and decoding process, and there is no limitation on this decoding process. The above-mentioned deep learning model or neural network model can be called the end-to-end video decoding exploration experiment model. The framework of the video decoding exploration experiment model is shown in Figure 3B. Based on the video decoding exploration experiment model, the image is transformed into the feature domain.

[0198] For example, after dividing the current frame image into multiple image blocks, after performing the above processing on each image block of the current frame image, a reconstructed image block corresponding to each image block can be obtained, and the reconstructed image blocks corresponding to all image blocks constitute the reconstructed image corresponding to the current frame image. In addition, when storing image reference information in the information buffer, after storing the image reference information corresponding to all image blocks, it is actually storing the image reference information corresponding to the current frame image (i.e., the image reference information of the entire frame image), that is, the information buffer stores the image reference information of the entire frame image.

[0199] Example 4: In Example 3, the residual confidence α is output by the residual coding module (or residual decoding module), while the predicted feature (such as the predicted feature) is output by the prediction module. When the residual confidence α represents the accuracy of the predicted feature, the residual confidence α may not accurately reflect the accuracy of the predicted feature. Based on this, considering that confidence is essentially an inhibition or enhancement of the prediction module, and the prediction module can directly perceive its own prediction efficiency (i.e., the accuracy of the predicted features), the confidence obtained from the prediction module is more accurate. Therefore, this application proposes a neural network video encoding / decoding method. Considering the initial design intent of confidence, which is an adjustment of inter-frame reference features, intuitively, this information directly comes from the prediction module (motion information encoding module), which can improve the accuracy of the confidence. Therefore, the confidence can be derived from the prediction module. For example, the confidence can be derived from the prediction decoder (Motion Decoder) of the prediction module, or from the temporal residual mining module (Temporal Context) of the prediction module.The following section will use the prediction decoder (Motion Decoder) as an example.

[0200] This application proposes an encoding method. The processing of the encoding end can be seen in Figure 4A. Of course, Figure 4A is just an example of the processing of the encoding end, and the processing of the encoding end is not limited.

[0201] 1. After obtaining the current image block xt, the encoding end can input the current image block xt to the feature extractor to obtain the current image feature Ft corresponding to the current image block xt, that is, the image domain is converted to the feature domain.

[0202] 2. The information buffer (Decoded Feature & Picture Buffer) stores the image reference information that the current image block xt can refer to. The historical time domain reference features are derived from the image reference information. For example, the image reference information is input to the feature extractor to obtain the historical time domain reference features, or the image reference information is directly used as the historical time domain reference features. The historical time domain reference features can be derived from the image reference information.

[0203] 3. The information buffer (Decoded Feature & Picture Buffer) stores historical prior prediction features (historical prior motion features). The historical prior prediction features are the prior prediction features that the current image block xt can refer to.

[0204] 4. The encoder performs motion estimation on the current image block xt and the historical image block xt-1 to obtain optical flow information vt. Specification 18 / 54 pages 24 CN 121603654 A

[0205] 5. The encoder inputs the current image feature Ft, optical flow information vt, historical temporal reference features and historical prior motion features to the predictive encoder (Motion Encoder). The predictive encoder encodes based on the current image feature Ft, optical flow information vt, historical temporal reference features and historical prior motion features to obtain unquantized predictive latent features mt.

[0206] 6. Quantize the predicted latent feature mt (Q operation) to obtain the predicted information, i.e., the motion information feature.

[0207] 7. The Motion Latent Buffer (prediction latent buffer, i.e., motion latent buffer) stores time-domain prediction prior information (time-domain motion prior information). The predicted latent feature mt and the time-domain prediction prior information are input to the Hyper Enc (super prior encoder). The super prior encoder encodes the predicted latent feature mt and the time-domain prediction prior information to obtain the prediction side information bitstream (i.e., motion side information bitstream), and transmits the prediction side information bitstream to the Hyper Dec at the decoding end.

[0208] 8. The Hyper at the encoding endThe Dec (hyper-prior decoder) can obtain the prediction side information bitstream, parse the prediction side information bitstream, and obtain the prediction hyper-prior information (Hyper Prior, such as motion hyper-prior information).

[0209] The prediction hyper-prior information (Hyper Prior) and the temporal prediction prior information (Temporal Prior) are input to the Motion Entropy Model (also known as the motion entropy model). The prediction entropy model generates a probability distribution of the prediction information based on the prediction hyper-prior information and the temporal prediction prior information. This probability distribution is used to guide encoding and decoding.

[0210] 9. The encoding end encodes the prediction information based on the probability distribution of the prediction information (i.e., AE operation) to obtain the prediction information bitstream (also known as the motion information bitstream), and transmits the prediction information bitstream to the decoding end.

[0211] 10. The encoding end parses the prediction information bitstream based on the probability distribution of the prediction information (i.e., AD operation) to obtain the prediction information. The prediction information can also be dequantized to obtain the prediction latent feature mt.

[0212] 11. After obtaining the prediction information, the encoding end stores the prediction information in the Motion Latent Buffer.

[0213] 12. The encoding end inputs the prediction information to the prediction decoder (Motion Decoder), which decodes the prediction information based on the prediction information (such as decoding motion information) to obtain the prediction mode features (also known as optical flow features), prior prediction features (also known as prior motion features), and prediction confidence αt. The prediction confidence αt can represent the accuracy of the prediction features. The prior prediction features are stored in the information buffer (Decoded Feature & Picture Buffer) as the historical prior prediction features of subsequent image blocks.

[0214] Since the prediction confidence αt is obtained by decoding the prediction information, and the prediction information is parsed from the prediction information bitstream, the prediction confidence αt can be obtained based on the information in the prediction information bitstream.

[0215] For example, the predictive decoder includes multiple network layers. When decoding the predictive information based on the predictive information, the features output by the penultimate network layer of the predictive decoder can be used as prior predictive features or the output features of other network layers (such as the penultimate network layer) can be used as prior predictive features. The features output by the last network layer of the predictive decoder can be used as predictive pattern features or the output features of other network layers can be used as predictive pattern features.

[0216] In addition to outputting prior predictive features and predictive pattern features, the predictive decoder can also output prediction confidence αt. The prediction confidence αt is obtained by processing the output features of a certain network layer of the predictive decoder. There are no restrictions on this process.

[0217] In summary, the encoder can parse the prediction edge information bitstream corresponding to the current image block to obtain the prediction super-prior information (step 8). The encoder can obtain the temporal prediction prior information from the prediction latent buffer (step 7), which is the prediction information that the current image block can refer to. The encoder can generate a probability distribution of the prediction information based on the prediction super-prior information and the temporal prediction prior information (step 8). The encoder can use the probability distribution to parse the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block (step 10), and store the prediction information in the prediction latent buffer for the temporal prediction prior information of subsequent image blocks (step 11). The encoder can decode the prediction information to obtain the prediction mode features and prediction confidence corresponding to the current image block (step 12).

[0218] 13. The encoding end inputs the predicted pattern features and historical temporal reference features into the temporal context mining module. The temporal context mining module can be implemented using a neural network or a non-neural network. The temporal context mining module can perform temporal context mining based on the predicted pattern features and historical temporal reference features to obtain initial predicted features, which are used for residual conditional coding. In summary, the encoding end can perform temporal information mining based on the predicted pattern features and historical temporal reference features to obtain initial predicted features; wherein, the information buffer stores image reference information that can be referenced by the current image block, and historical temporal reference features are derived through the image reference information.

[0219] For example, the temporal residual mining module can obtain an initial prediction feature and output the initial prediction feature. The temporal residual mining module can also obtain two initial prediction features and output the initial prediction feature sum.

[0220] 14. The encoding end inputs the current image feature Ft and the initial prediction feature (such as prediction feature and prediction feature) to the residual encoder. The residual encoder performs residual encoding based on the current image feature Ft and the initial prediction feature (such as prediction feature and prediction feature) to obtain the unquantized residual information feature yt.

[0221] 15. The encoding end quantizes the unquantized residual information feature yt (i.e., Q operation) to obtain residual information.

[0222] 16. The encoding end inputs the initial prediction feature (prediction feature) to the spatial prior encoder. The spatial prior encoder processes the initial prediction feature to obtain spatial prior information.

[0223] 17. Contextual LatentThe Buffer (Residual Potential Buffer) stores residual prior information. The residual prior information is the residual information of the image block (or the image frame before the current image block xt)

[0224] . The encoder inputs the residual information feature yt, the spatial prior information, and the residual prior information to the Hyper Enc (Super Prior Encoder). The Super Prior Encoder encodes the residual information feature yt, the spatial prior information, and the residual prior information to obtain the residual side information bitstream, and sends the residual side information bitstream to the decoder.

[0225] 18. After obtaining the residual side information bitstream, the Hyper Dec (Super Prior Decoder) of the encoder parses the residual side information bitstream to obtain the residual super prior information (Hyper Prior).

[0226] 19. The encoding end inputs the residual hyperprior information, spatial prior information, and residual temporary prior information into the Contextual Entropy Model. The residual entropy model generates a probability distribution of residual information (i.e., residual latent features) based on the residual hyperprior information, spatial prior information, and residual temporary prior information. The probability distribution of residual latent features is used to guide encoding and decoding.

[0227] 20. The encoding end encodes the residual latent features based on the probability distribution of residual latent features (i.e., AE operation) to obtain the residual information bitstream. The encoding end transmits the residual information bitstream to the decoding end. Specification 20 / 54 pages 26 CN 121603654 A

[0228] 21. The encoding end parses the residual information bitstream based on the probability distribution of residual latent features (i.e., AD operation) to obtain the residual latent features. The residual latent features can also be dequantized to obtain the residual information feature yt. The encoder stores the residual latent features in the Contextual Latent Buffer for residual prior information of subsequent image blocks

[0229] 22. The encoder inputs the residual latent features and initial prediction features (such as initial prediction features) to the residual decoder. The residual decoder decodes the residual information based on the residual latent features and initial prediction features to obtain the target residual features

[0230] In summary, the encoder can parse the residual edge information bitstream corresponding to the current image block to obtain residual prior information (step 18). The encoder can obtain residual prior information from the residual latent buffer, which can be the residual information that can be referenced by the current image block (step 17). The encoder can perform spatial prior processing based on the initial prediction features (prediction features) corresponding to the current image block to obtain spatial prior information (step 16). The encoder can perform spatial prior processing based on the residual latent features (prediction features) corresponding to the current image block to obtain spatial prior information (step 16).The probability distribution of residual information generated from the prior information of the difference, the prior information of the residual, and the prior information of the spatial information is used (step 19). The residual information bitstream corresponding to the current image block is parsed using the probability distribution to obtain the residual information corresponding to the current image block (step 21). The residual information is stored in the residual latent buffer for the residual prior information of subsequent image blocks. The encoder can perform residual decoding on the residual information to obtain the target residual feature corresponding to the current image block (step 22).

[0231] 23. The encoder determines the target confidence corresponding to the initial prediction feature (such as the initial prediction feature) based on the prediction confidence αt. The initial prediction feature and the target confidence are weighted (such as multiplication) to obtain the target prediction feature; or, the initial prediction feature and the target confidence are weighted (such as multiplication) to obtain the weighted feature, and the weighted feature is transformed to obtain the target prediction feature. For example, similar to the function of the prediction confidence α, the target confidence can represent the accuracy of the prediction feature.

[0232] For example, the predicted confidence level αt can be determined as the target confidence level, that is, the predicted confidence level αt is directly used as the target confidence level. Alternatively, the predicted confidence level αt can be transformed to obtain the target confidence level. For example, a certain network can be used to process the predicted confidence level αt, and the processed confidence level can be used as the target confidence level. For example, a certain algorithm can be used to process the predicted confidence level αt, and the processed confidence level can be used as the target confidence level. Of course, the above are just examples and are not limited thereto, as long as the predicted confidence level αt is transformed to obtain the target confidence level.

[0233] The target confidence level can be obtained based on the residual confidence level α, the predicted confidence level αt, or both the residual confidence level α and the predicted confidence level αt. Since the residual confidence level α can be obtained based on information in the residual information bitstream, and the predicted confidence level αt can be obtained based on information in the predicted information bitstream, the target confidence level can be obtained based on information in the residual information bitstream and / or information in the predicted information bitstream. In this embodiment, since the target confidence level is determined based on the predicted confidence level αt, the target confidence level can be obtained based on information in the predicted information bitstream.

[0234] For example, weighting the initial predicted feature and the target confidence (such as multiplication) may include: the initial predicted feature may be a three-dimensional feature matrix of C*H*W, where C represents the number of channels, H represents the feature height, and W represents the feature width; the target confidence may include C confidence values; the feature matrix of the first channel (H*W feature values) is weighted with the first confidence value, that is, H*W feature values ​​correspond to the same confidence value; the feature matrix of the second channel is weighted with the second confidence value, and so on.

[0235] Alternatively, the target confidence level may include H*W confidence values. The feature matrix of the first channel (H*W feature values, see page 21 / 54 of the specification, CN 121603654 A) is weighted and calculated with the H*W confidence values, that is, the H*W feature values ​​correspond one-to-one with the H*W confidence values, and each feature value corresponds to one confidence value. The feature matrix of the second channel (H*W feature values) is weighted and calculated with the H*W confidence values, and so on. That is, the feature matrices of all C channels correspond to the same target confidence level.

[0236] Alternatively, the target confidence level may include C*H*W confidence values. The feature matrix of the first channel (H*W feature values) is weighted and calculated with the H*W confidence values ​​of the first channel, that is, the H*W feature values ​​correspond one-to-one with the H*W confidence values. The feature matrix (H*W eigenvalues) of the second channel is weighted and the H*W confidence values ​​of the second channel are weighted, and so on. That is, the feature matrix of different channels corresponds to different target confidences.

[0237] For example, when the initial prediction feature and the target confidence are weighted and the weighted feature is obtained, the weighted feature can be directly used as the target prediction feature. Alternatively, the weighted feature can be transformed and the transformed feature can be used as the target prediction feature. For example, a certain network can be used to process the weighted feature and the processed feature can be used as the target prediction feature. For example, a certain algorithm can be used to process the weighted feature and the processed feature can be used as the target prediction feature. Of course, the above are just examples and are not limited. As long as the weighted feature is transformed to obtain the target prediction feature, it is acceptable.

[0238] 24. The encoding end determines the image reference information corresponding to the current image block based on the target residual feature and the target prediction feature. For example, the target prediction features and target residual features are added together, and the added features are input to the frame generator. The frame generator obtains image reference information based on the added features and stores the image reference information in the information buffer (Decoded Feature & Picture Buffer). The image reference information is used as the image reference information for subsequent image blocks. Alternatively, the encoding end can also process the image reference information (such as feature extraction), and store the processed features in the information buffer. The processed features are used as the image reference information for subsequent image blocks.

[0239] For example, the features input to the frame generator can be represented by the following expression: f represents the features input to the frame generator (i.e., the added features), represents the weighted operation of the initial prediction features and the target confidence α, i.e., the target prediction features, and represents the weighted operation of the target prediction features and the target residual features.Perform addition operations.

[0240] As can be seen from the above expression, for regions where the prediction module predicts accurately, i.e., the initial prediction features are accurate, the target confidence α is close to 1, making the features input to the frame generator close to the sum of the initial prediction features and the target residual features. For regions where the prediction module predicts with low accuracy, i.e., the initial prediction features are inaccurate, the target confidence α is close to 0, making the features input to the frame generator close to the target residual features. The residual coding degenerates into I-frame coding.

[0241] 25. Generate the reconstructed image block corresponding to the current image block xt at the encoding end

[0242] Thus, the processing at the encoding end is completed, and the reconstructed image block corresponding to the current image block xt can be obtained.

[0243] In one possible implementation, the above processing at the encoding end can be executed by a deep learning model or a neural network model, thereby realizing the end-to-end image compression and encoding process. There are no restrictions on this encoding process.

[0244] This application embodiment proposes a decoding method. The processing procedure at the decoding end can be seen in Figure 4B. Of course, Figure 4B is only an example of the decoding end's processing procedure, and this processing procedure is not limited.

[0245] 1. The decoding end obtains the prediction side information bitstream (also called the motion side information bitstream).

[0246] 2. The Hyper Dec (hyper-prior decoder) at the decoding end obtains the prediction side information bitstream, parses the prediction side information bitstream, and obtains the prediction hyper-prior information (Hyper Prior, such as motion hyper-prior information). The decoding end inputs the prediction hyper-prior information and the temporal prediction prior information (Temporal Prior) from page 22 / 54 of the prediction specification (CN 121603654 A) into the Motion Entropy Model. The prediction entropy model generates the probability distribution of the prediction information based on the prediction hyper-prior information and the temporal prediction prior information.

[0247] For example, the Motion Latent Buffer (prediction latent buffer / motion latent buffer) stores time-domain prediction prior information (time-domain motion prior information). The time-domain prediction prior information can be the prediction information of the image blocks preceding the current image block xt or the prediction information of the image frames preceding the current frame.

[0248] 3. The decoding end obtains the prediction information bitstream (the prediction information bitstream can also be called the motion information bitstream).

[0249] 4. The decoding end parses the prediction information bitstream based on the probability distribution of the prediction information (i.e., AD operation) to obtain the prediction information. The prediction information can also be called motion information features or prediction information features. The prediction information can also be dequantized to obtain the prediction latent feature mt, which can also be called the motion latent feature mt. The decoding end stores the prediction information in the Motion Latent Buffer.Buffer, used for temporal prediction prior information of subsequent image blocks

[0250] 5. The decoding end inputs the prediction information to the prediction decoder (Motion Decoder), which decodes the prediction information based on the prediction information to obtain the prediction mode feature (also known as optical flow feature), prior prediction feature (also known as prior motion feature) and prediction confidence αt. The prediction confidence αt can represent the accuracy of the prediction feature. The prior prediction feature is stored in the information buffer (Decoded Feature & Picture Buffer) as the historical prior prediction feature of subsequent image blocks. Of course, the prediction decoder may not output the prior prediction feature

[0251] Since the prediction confidence αt is obtained by decoding the prediction information, and the prediction information is parsed from the prediction information bitstream, the prediction confidence αt can be obtained based on the information in the prediction information bitstream.

[0252] For example, the prediction decoder includes multiple network layers. When decoding the prediction information based on the prediction information, the features output by the penultimate network layer of the prediction decoder can be used as prior prediction features, or the output features of other network layers (such as the penultimate network layer) can be used as prior prediction features. The features output by the last network layer of the prediction decoder can be used as prediction mode features, or the output features of other network layers can be used as prediction mode features.

[0253] In addition to outputting prior prediction features and prediction mode features, the prediction decoder can also output prediction confidence αt. The prediction confidence αt is obtained by processing the output features of a certain network layer of the prediction decoder. There are no restrictions on this process.

[0254] In summary, the decoding end can parse the prediction side information bitstream corresponding to the current image block to obtain the prediction super-prior information, and obtain the temporal prediction prior information from the prediction latent buffer. This temporal prediction prior information is the prediction information that the current image block can refer to. Based on the prediction prior information and the temporal prediction prior information, a probability distribution of prediction information is generated. This probability distribution is used to parse the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block. The prediction information is then stored in the prediction latent buffer for the temporal prediction prior information of subsequent image blocks. The prediction information is then decoded to obtain the prediction mode features and prediction confidence corresponding to the current image block.

[0255] 6. The decoding end inputs the prediction mode features and historical temporal reference features into the temporal residual mining module. The temporal residual mining module can perform temporal residual mining based on the prediction mode features and historical temporal reference features to obtain the initial prediction features. In summary, the decoding end can perform temporal residual mining based on the prediction mode features. (See page 23 / 54 of the specification, 29 CN 121603654 A)Temporal information mining is performed on historical temporal reference features to obtain initial prediction features; wherein, the information buffer stores the image reference information that the current image block xt can refer to. Historical temporal reference features are derived from the image reference information.

[0256] For example, the temporal residual mining module can obtain an initial prediction feature and output the initial prediction feature. The temporal residual mining module can also obtain two initial prediction features and output the sum of the initial prediction features.

[0257] 7. The decoding end obtains the residual side information bitstream. The Hyper Dec (super prior decoder) of the decoding end parses the residual side information bitstream to obtain residual super prior information (Hyper Prior).

[0258] 8. The decoding end inputs the prediction feature (prediction feature) to the spatial prior encoder (Spatial Prior Enc). The spatial prior encoder processes the prediction feature to obtain spatial prior information (Spatial Prior).

[0259] 9. The Contextual Latent Buffer stores residual prior information. The residual prior information is the residual latent feature (i.e., residual information) of the image blocks preceding the current image block xt. The residual prior information can also be the residual latent feature (i.e., residual information) of the image frames preceding the current frame.

[0260] 10. The decoder inputs the residual super-prior information, spatial prior information, and residual prior information (Temporal Prior) to the Contextual Entropy Model. The residual entropy model generates a probability distribution of residual latent features based on the residual super-prior information, spatial prior information, and residual prior information. This probability distribution is used to guide decoding.

[0261] 11. The decoder obtains the residual information bitstream.

[0262] 12. The decoder parses the residual information bitstream (i.e., AD operation) based on the probability distribution of the residual latent features to obtain the residual latent features. The residual latent features can also be dequantized to obtain the residual information feature yt.

[0263] 13. After obtaining the residual latent features (also known as residual information), the decoding end can store the residual latent features in the Contextual Latent Buffer for use as residual prior information for subsequent image blocks.

[0264] 14. The decoding end inputs the residual latent features and the initial prediction features (such as the initial prediction features) to the residual decoder. The residual decoder decodes the residual information based on the residual latent features and the initial prediction features to obtain the target residual features.

[0265] In summary, the decoding end parses the residual edge information bitstream corresponding to the current image block to obtain the residual prior information and obtains the residual prior information from the residual latent buffer. This residual prior information can be a reference for the current image block.The residual information is obtained by performing spatial prior processing based on the initial prediction feature (prediction feature) corresponding to the current image block. Based on the residual prior information, the residual prior information and the spatial prior information, a probability distribution of residual information is generated. The residual information bitstream corresponding to the current image block is parsed using the probability distribution to obtain the residual information corresponding to the current image block. The residual information is then subjected to residual decoding to obtain the target residual feature corresponding to the current image block.

[0266] 15. The decoding end determines the target confidence corresponding to the initial prediction feature (such as the initial prediction feature) based on the prediction confidence αt. The initial prediction feature and the target confidence are weighted (such as multiplication) to obtain the target prediction feature; or, the initial prediction feature and the target confidence are weighted (such as multiplication) to obtain the weighted feature, and the weighted feature is transformed to obtain the target prediction feature. For example, similar to the function of prediction confidence α, the target confidence can represent the accuracy of the prediction feature.

[0267] For example, the predicted confidence level αt can be determined as the target confidence level, that is, the predicted confidence level αt is directly used as the target confidence level. Alternatively, the predicted confidence level αt can be transformed to obtain the target confidence level.

[0268] The target confidence level can be obtained based on the residual confidence level α, or based on the predicted confidence level αt, or based on both the residual confidence level α and the predicted confidence level αt. Based on this, since the residual confidence level α can be obtained based on the information in the residual information bitstream, and the predicted confidence level αt can be obtained based on the information in the predicted information bitstream, the target confidence level can be obtained based on the information in the residual information bitstream and / or the information in the predicted information bitstream. In this embodiment, since the target confidence level is determined based on the predicted confidence level αt, the target confidence level can be obtained based on the information in the predicted information bitstream.

[0269] For example, the weighted operation (such as multiplication) on the initial predicted feature and the target confidence may include, but is not limited to: the initial predicted feature may be a three-dimensional feature matrix of C*H*W, the target confidence may include C confidence values, the feature matrix of the first channel (H*W feature values) is weighted with the first confidence value, the feature matrix of the second channel is weighted with the second confidence value, and so on. Alternatively, the target confidence may include H*W confidence values, the feature matrix of the first channel (H*W feature values) is weighted with H*W confidence values, the feature matrix of the second channel (H*W feature values) is weighted with H*W confidence values, and so on. Alternatively, the target confidence may include C*H*W confidence values, the feature matrix of the first channel (H*W feature values) is weighted with the first confidence value, the feature matrix of the second channel (H*W feature values) is weighted with the second confidence value, and so on.The H*W confidence values ​​of the first channel are weighted, and the feature matrix of the second channel (H*W feature values) is weighted with the H*W confidence values ​​of the second channel, and so on.

[0270] For example, when the initial prediction feature and the target confidence are weighted to obtain the weighted feature, the weighted feature can be directly used as the target prediction feature. Alternatively, the weighted feature can be transformed and used as the target prediction feature.

[0271] 16. The decoding end determines the image reference information corresponding to the current image block based on the target residual feature and the target prediction feature. For example, the target prediction features and target residual features are added together, and the added features are input to the frame generator. The frame generator obtains the image reference information and stores the image reference information in the information buffer. The image reference information is used as the image reference information for subsequent image blocks. Alternatively, the encoding end can also process the image reference information (such as feature extraction), store the processed features in the information buffer, and use the processed features as the image reference information for subsequent image blocks.

[0272] 17. The decoding end generates the reconstructed image block corresponding to the current image block xt.

[0273] Thus, the processing of the decoding end is completed, and the reconstructed image block corresponding to the current image block xt can be obtained.

[0274] In one possible implementation, the above-mentioned processing of the decoding end can be executed by a deep learning model or a neural network model to realize the end-to-end image compression and decoding process. There are no restrictions on this decoding process.

[0275] Example 5: In Example 3, the residual confidence α is output by the residual encoding module (or residual decoding module), while the predicted feature (such as the predicted feature) is output by the prediction module. When the residual confidence α represents the accuracy of the predicted feature, the residual confidence α may not accurately reflect the accuracy of the predicted feature. Based on this, considering that the confidence is essentially an inhibition or enhancement of the prediction module, the prediction module can directly perceive the prediction efficiency it provides, that is, the prediction module can directly perceive the accuracy of the predicted feature. Therefore, the confidence obtained from the prediction module is more accurate. On this basis, considering the original design intention of the confidence, this information directly comes from the prediction module (motion information encoding module), which can improve the accuracy of the confidence. Therefore, the confidence can be derived from the prediction module. For example, the confidence level can be derived from the prediction decoder (Motion Decoder) of the prediction module, or it can be derived from the temporal residual mining module specification (page 25 / 54, CN 121603654 A (Temporal Context Mining)). The following example will focus on deriving it from the prediction decoder (Motion Decoder).

[0276] This application proposes an encoding method. The processing procedure of the encoding end can be seen in Figure 5A. Of course, Figure 5A is just an example of the processing procedure of the encoding end, and the processing procedure of the encoding end is not limited.

[0277] 1. The encoding end inputs the current image block xt to the feature extractor to obtain the current image feature Ft.

[0278] 2. The information buffer (Decoded Feature & Picture Buffer) stores the image reference information that the current image block xt can refer to. The historical temporal reference feature is derived through the image reference information.

[0279] 3. The information buffer (Decoded Feature & Picture Buffer) stores the historical prior prediction features.

[0280] 4. The encoding end performs motion estimation on the current image block xt and the historical image block xt-1 to obtain the optical flow information vt.

[0281] 5. The encoding end inputs the current image feature Ft, optical flow information vt, historical temporal reference features, and historical prior motion features to the predictive encoder (Motion Encoder) to obtain unquantized predictive latent features mt.

[0282] 6. Quantize the predictive latent features mt (Q operation) to obtain the predicted information, i.e., motion information features.

[0283] 7. The Motion Latent Buffer stores the temporal predictive prior information, predictive latent features mt, and temporal predictive prior information is input to the Hyper Enc (super prior encoder) to obtain the predictive side information bitstream (i.e., motion side information bitstream). The encoding end transmits the predictive side information bitstream to the Hyper Dec at the decoding end. The Hyper Dec (super prior decoder) at the encoding end parses the predictive side information bitstream to obtain the predictive super prior information (Hyper Prior).

[0284] 8. The Hyper Prior and Temporal Prior are input into the Motion Entropy Model to obtain the probability distribution of the prediction information.

[0285] 9. The encoder encodes the prediction information based on the probability distribution (i.e., AE operation) to obtain the prediction information bitstream (also called motion information bitstream), and transmits the prediction information bitstream to the decoder.

[0286] 10. The encoder parses the prediction information bitstream based on the probability distribution (i.e., AD operation) to obtain the prediction information. The prediction information can also be dequantized to obtain the prediction latent feature mt.

[0287] 11. After obtaining the prediction information, the encoder stores the prediction information in the Motion Latent Buffer.

[0288] 12. The encoder transmits the prediction information...The input is given to the prediction decoder (Motion Decoder) to obtain the prediction pattern features (also known as optical flow features), prior prediction features (also known as prior motion features), and prediction confidence αt. The prediction confidence αt can represent the accuracy of the prediction features. The prior prediction features are stored in the information buffer.

[0289] For example, in addition to outputting the prior prediction features and prediction pattern features, the prediction decoder can also output the prediction confidence αt. The output features of a certain network layer of the prediction decoder are processed to obtain the prediction confidence αt.

[0290] 13. The encoding end inputs the prediction pattern features and historical temporal reference features into the temporal residual mining module. The temporal residual mining module can perform temporal residual mining based on the prediction pattern features and historical temporal reference features to obtain the initial prediction features. The temporal residual mining module can obtain an initial prediction feature and output the initial prediction feature. The temporal residual mining module can also obtain two initial prediction features and output the initial prediction feature sum.

[0291] 14. The encoder inputs the current image feature Ft and the initial prediction feature (such as prediction feature and prediction feature) to the residual encoder to obtain the unquantized residual information feature yt.

[0292] 15. The encoder quantizes the unquantized residual information feature yt (i.e., Q operation) to obtain the residual information. Specification 26 / 54 pages 32 CN 121603654 A

[0293] 16. The encoder inputs the initial prediction feature (prediction feature) to the spatial prior encoder to obtain the spatial prior information.

[0294] 17. The residual prior information is stored in the Contextual Latent Buffer. The encoder inputs the residual information feature yt, the spatial prior information, and the residual prior information to the Hyper Enc to obtain the residual side information bitstream. The encoder sends the residual side information bitstream to the decoder.

[0295] 18. The Hyper Dec of the encoder parses the residual side information bitstream to obtain the residual hyper prior information.

[0296] 19. The encoder inputs the residual hyper prior information, the spatial prior information, and the residual prior information to the Contextual Entropy Model to obtain the probability distribution of the residual information (i.e., the residual latent feature).

[0297] 20. The encoder encodes the residual latent feature based on the probability distribution of the residual latent feature (i.e., AE operation) to obtain the residual information bitstream. The encoder transmits the residual information bitstream to the decoder.

[0298] 21. The encoding end parses the residual information bitstream based on the probability distribution of the residual latent features (i.e., AD operation).The residual latent features can also be dequantized to obtain residual information features yt. The encoder stores the residual latent features in the Contextual Latent Buffer for the residual prior information of subsequent image blocks.

[0299] 22. The encoder inputs the residual latent features and the initial prediction features to the residual decoder. The residual decoder decodes the residual information based on the residual latent features and the initial prediction features to obtain the target residual features and the residual confidence α.

[0300] 23. The encoder generates the target confidence corresponding to the initial prediction features (such as the initial prediction features) based on the prediction confidence αt and the residual confidence α. The encoder performs a weighted operation (such as multiplication) on the initial prediction features and the target confidence to obtain the target prediction features; or, the encoder performs a weighted operation (such as multiplication) on the initial prediction features and the target confidence to obtain the weighted features, and performs a transformation operation on the weighted features to obtain the target prediction features.

[0301] For example, the target confidence level can be obtained based on the residual confidence level α, or based on the predicted confidence level αt, or based on both the residual confidence level α and the predicted confidence level αt. Since the residual confidence level α can be obtained based on information in the residual information stream, and the predicted confidence level αt can be obtained based on information in the predicted information stream, the target confidence level can be obtained based on information in the residual information stream and / or information in the predicted information stream. In this embodiment, since the target confidence level corresponding to the initial predicted feature is generated based on the predicted confidence level αt and the residual confidence level α, the target confidence level can be obtained based on information in the residual information stream and information in the predicted information stream.

[0302] For example, the predicted confidence level αt and the residual confidence level α can be input to the network layer C, and the network layer C processes them based on the predicted confidence level αt and the residual confidence level α to obtain the target confidence level. In this embodiment, the structure of the network layer C is not limited, as long as the network layer C can process the predicted confidence αt and the residual confidence α to obtain the target confidence. For example, the predicted confidence and the residual confidence can be concatenated to obtain the concatenated features. The concatenated features are then convolved (e.g., through several convolutional layers) to obtain the target confidence.

[0303] For example, when performing a weighted operation (e.g., multiplication) on the initial predicted features and the target confidence, the initial predicted features can be a three-dimensional feature matrix of C*H*W, and the target confidence can include C confidence values, or the target confidence can include H*W confidence values, or the target confidence can include C*H*W confidence values.

[0304] 24. The encoding end determines the image corresponding to the current image block based on the target residual features and the target predicted features. (See page 27 / 54 of the specification)33 CN 121603654 A Information. For example, the target prediction features and target residual features are added together, and the added features are input to the frame generator to obtain image reference information and store the image reference information in the information buffer.

[0305] 25. The encoding end generates the reconstructed image block corresponding to the current image block xt

[0306] In one possible implementation, the above-mentioned encoding end processing can be executed by a deep learning model or a neural network model to realize the end-to-end image compression and encoding process. There are no restrictions on this encoding process.

[0307] In this application embodiment, a decoding method is proposed. Regarding the decoding end processing, please refer to Figure 5B. Of course, Figure 5B is just an example of the decoding end processing. There are no restrictions on this decoding end processing.

[0308] 1. The decoding end obtains the prediction side information bitstream (also called the motion side information bitstream).

[0309] 2. The Hyper Dec of the decoding end parses the prediction side information bitstream to obtain the prediction prior information. The decoding end inputs the prediction prior information and the temporal prediction prior information into the Motion Entropy Model to obtain the probability distribution of the prediction information. For example, the Motion Latent Buffer stores the temporal prediction prior information

[0310] 3. The decoding end obtains the prediction information bitstream (the prediction information bitstream can also be called the motion information bitstream).

[0311] 4. The decoding end parses the prediction information bitstream based on the probability distribution of the prediction information (i.e., AD operation) to obtain the prediction information. The prediction information can also be dequantized to obtain the prediction latent feature mt. The decoding end stores the prediction information into the Motion Latent Buffer for the temporal prediction prior information of subsequent image blocks

[0312] 5. The decoding end inputs the prediction information into the prediction decoder, which decodes the prediction information based on the prediction information to obtain the prediction mode feature (also called optical flow feature), the prior prediction feature (also called the prior motion feature) and the prediction confidence αt. The prediction confidence αt can represent the accuracy of the prediction feature.

[0313] For example, in addition to outputting prior prediction features and prediction pattern features, the prediction decoder can also output prediction confidence αt. The output features of a certain network layer of the prediction decoder are processed to obtain the prediction confidence αt.

[0314] 6. The decoding end inputs the prediction pattern features and historical temporal reference features into the temporal residual mining module. The temporal residual mining module can perform temporal residual mining based on the prediction pattern features and historical temporal reference features to obtain initial prediction features. The temporal residual mining module can obtain one initial prediction feature and output the initial prediction feature. The temporal residual mining module can also obtain two initial prediction features and output the sum of the initial prediction features.

[0315] 7. The decoding end obtains the residual side information bitstream. The Hyper Dec (super prior decoder) of the decoding end parses the residual side information bitstream to obtain the residual super prior information (Hyper Prior). The decoding end inputs the predicted features (predicted features) to the Spatial Prior Encoder (Spatial Prior) to obtain the Spatial Prior information (Spatial Prior). The Contextual Latent Buffer (Residual Latent Buffer) stores the residual prior information (i.e., residual information).

[0316] 8. The decoding end inputs the residual super prior information, spatial prior information and residual prior information to the Contextual Entropy Model (Residual Entropy Model) to obtain the probability distribution of the residual latent features. This probability distribution is used to guide the decoding.

[0317] 9. The decoding end obtains the residual information bitstream.

[0318] 10. The decoding end parses the residual information bitstream based on the probability distribution of the residual latent features (i.e., AD operation) to obtain the residual latent features. The residual latent features can also be dequantized to obtain the residual information features yt. The decoding end can store the residual latent features in the Contextual Latent Buffer for the residual prior information of subsequent image blocks.

[0319] 11. The decoding end inputs the residual latent features and the initial prediction features to the residual decoder. The residual decoder decodes the residual information based on the residual latent features and the initial prediction features to obtain the target residual features and the residual confidence α.

[0320] 12. The decoding end generates a target confidence level corresponding to the initial prediction feature (e.g., the initial prediction feature) based on the prediction confidence level αt and the residual confidence level αt. The initial prediction feature and the target confidence level are weighted (e.g., multiplication) to obtain the target prediction feature; or, the initial prediction feature and the target confidence level are weighted (e.g., multiplication) to obtain the weighted feature, and the weighted feature is transformed to obtain the target prediction feature.

[0321] For example, the target confidence level can be obtained based on the residual confidence level αt, or based on the prediction confidence level αt, or based on both the residual confidence level αt and the prediction confidence level αt. Based on this, since the residual confidence level αt can be obtained based on the information in the residual information bitstream, and the prediction confidence level αt can be obtained based on the information in the prediction information bitstream, the target confidence level can be obtained based on the information in the residual information bitstream and / or the information in the prediction information bitstream. In this embodiment, since the target confidence level corresponding to the initial predicted feature is generated based on the predicted confidence level αt and the residual confidence level α, therefore...Therefore, the target confidence can be obtained based on the information in the residual information bitstream and the information in the prediction information bitstream.

[0322] For example, the prediction confidence αt and the residual confidence α are input to the network layer C, and the network layer C processes the prediction confidence αt and the residual confidence α to obtain the target confidence. For example, the prediction confidence and the residual confidence can be channel-connected to obtain the channel-connected features. The channel-connected features are then convolved to obtain the target confidence.

[0323] For example, when the initial prediction features and the target confidence are weighted (e.g., multiplication), the initial prediction features can be a three-dimensional feature matrix of C*H*W, and the target confidence can include C confidence values, or the target confidence can include H*W confidence values, or the target confidence can include C*H*W confidence values.

[0324] 13. The decoding end determines the image reference information corresponding to the current image block based on the target residual features and the target prediction features. For example, the target prediction features and target residual features are added together, and the added features are input to the frame generator to obtain image reference information and the image reference information is stored in the information buffer.

[0325] 14. The decoding end generates the reconstructed image block corresponding to the current image block xt

[0326] In one possible implementation, the above-mentioned decoding end processing can be executed by a deep learning model or a neural network model to realize the end-to-end image compression and decoding process. There are no restrictions on this decoding process.

[0327] Example 6: In Examples 4 and 5, the temporal residual mining module can perform temporal residual mining based on the prediction mode features and historical temporal reference features to obtain the initial prediction features. The following describes this process.

[0328] In one possible implementation, see Figure 6A, which is a schematic diagram of the structure of the temporal residual mining module (temporal residual mining network). The prediction mode features (denoted as) and historical temporal reference features (denoted as) can be input into the temporal residual mining module. The temporal residual mining module can perform compensation operations (i.e., Align operations) on historical temporal reference features based on the prediction pattern features to obtain temporal compensated features (such as optical flow features). Then, convolution operations (i.e., Cov operations) and residual block operations (i.e., ResBlock operations) are performed on the temporal compensated features in sequence, and the features after the residual block operation are added to the temporal compensated features to obtain the initial prediction features. In this case, the temporal residual mining module can obtain an initial prediction feature and output the initial prediction feature. For example, the residual block operation refers to processing the input features through a residual block network. In this embodiment, the network structure of the residual block network is not limited.

[0329] In one possible implementation, as shown in Figure 6B, the temporal residual mining module (temporal residual mining network) is a temporal residual mining module (temporal residual mining network).A schematic diagram of the network structure. Prediction mode features may include first prediction mode features and second prediction mode features. Specification 29 / 54 Page 35 CN 121603654 A Historical time domain reference features may include first historical time domain reference features and second historical time domain reference features. Initial prediction features may include first initial prediction features and second initial prediction features. For example, the feature dimension of the first prediction mode feature is greater than the feature dimension of the second prediction mode feature, such as the width of is greater than the width of , and / or, the height of is greater than the height of . The feature dimension of the first historical time domain reference feature is greater than the feature dimension of the second historical time domain reference feature, such as the width of is greater than the width of , and / or, the height of is greater than the height of . The feature dimension of the first initial prediction feature is greater than the feature dimension of the second initial prediction feature, such as the width of is greater than the width of , and / or, the height of is greater than the height of . For another example, the feature dimension of the first prediction mode feature is less than the feature dimension of the second prediction mode feature, such as the width of is less than the width of , and / or, the height of is less than the height of . The feature dimension of the first historical time domain reference feature is less than the feature dimension of the second historical time domain reference feature, such as the width of is less than the width of , and / or, the height of is less than the height of . The feature dimension of the first initial prediction feature is smaller than the feature dimension of the second initial prediction feature, such as the width of is smaller than the width of, and / or the height of is smaller than the height of.

[0330] The first prediction mode feature, the second prediction mode feature, the first historical time domain reference feature, and the second historical time domain reference feature are input into the time domain residual mining module. The time domain residual mining module can perform a compensation operation (i.e., an Align operation) on the second historical time domain reference feature based on the second prediction mode feature to obtain the second time domain compensated feature. Then, the second time domain compensated feature is sequentially subjected to a convolution operation (i.e., a Cov operation) and a residual block operation (i.e., a ResBlock operation), and the feature after the residual block operation is added to the second time domain compensated feature to obtain the second initial prediction feature.

[0331] The time domain residual mining module can perform a compensation operation (i.e., an Align operation) on the first historical time domain reference feature based on the first prediction mode feature to obtain the first time domain compensated feature. The second time domain compensated feature is subjected to an upsampling operation (Up2) to obtain the sampled time domain compensated feature. For example, if the feature dimension of the first prediction mode feature is greater than the feature dimension of the second prediction mode feature, then an upsampling operation is performed; if the feature dimension of the first prediction mode feature is less than the feature dimension of the second prediction mode feature, then a downsampling operation is performed on the second temporal compensation feature to obtain the sampled temporal compensation feature.

[0332] The first temporal compensation feature and the sampled temporal compensation feature are concatenated (C), i.e., the C operation is performed.However, other methods can also be used to operate on the first temporal compensation feature and the sampled temporal compensation feature, and there are no restrictions on the operation method. Then, convolution operation (i.e., Cov operation) and residual block operation (i.e., ResBlock operation) are performed on the channel-connected feature (i.e., the channel-connected temporal compensation feature) in sequence, and the feature after the residual block operation is added to the first temporal compensation feature to obtain the first initial prediction feature. In summary, the temporal residual mining module can obtain the first initial prediction feature and the second initial prediction feature and output these two initial prediction features.

[0333] Example 7: In Example 3, the residual confidence α is output by the residual encoding module (residual decoding module), while the prediction feature is output by the prediction module. When the residual confidence α represents the accuracy of the prediction feature, the residual confidence α may not accurately reflect the accuracy of the prediction feature. Based on this, the confidence obtained from the prediction module is more accurate. Compared with Example 5 and Example 6, instead of extracting the confidence from the prediction module, the confidence is directly applied to the prediction module. For example, the confidence level can be applied to the prediction decoder (Motion Decoder) of the prediction module, or it can be applied to the temporal context mining module of the prediction module. The following example is from page 30 / 54 of the specification of the temporal context mining module, CN 121603654 A.

[0334] An encoding method is proposed in this application embodiment. Regarding the processing of the encoding end, please refer to Figure 7A. Of course, Figure 7A is only an example of the processing of the encoding end, and the processing of the encoding end is not limited.

[0335] 1. The encoding end inputs the current image block xt to the feature extractor to obtain the current image feature Ft.

[0336] 2. The information buffer (Decoded Feature & Picture Buffer) stores the image reference information that can be referenced by the current image block xt. The historical temporal reference features can be derived from the image reference information.

[0337] 3. The information buffer (Decoded Feature & Picture Buffer) stores the historical prior prediction features.

[0338] 4. The encoder performs motion estimation on the current image block xt and the historical image block xt-1 to obtain the optical flow information vt.

[0339] 5. The encoder inputs the current image feature Ft, the optical flow information vt, the historical temporal reference features, and the historical prior motion features to the predictive encoder (Motion Encoder) to obtain the unquantized predictive latent features mt.

[0340] 6. The predictive latent features mt are quantized (Q operation) to obtain the prediction information, i.e., the motion information features.

[0341] 7. Motion LatentThe Buffer stores the temporal prediction prior information, the latent feature mt, and the temporal prediction prior information is input to the Hyper Enc (super prior encoder) to obtain the prediction side information bitstream (i.e., the motion side information bitstream).

[0342] The encoder transmits the prediction side information bitstream to the decoder Hyper Dec. The encoder Hyper Dec (super prior decoder) parses the prediction side information bitstream to obtain the prediction super prior information (Hyper Prior).

[0343] 8. The prediction super prior information (Hyper Prior) and the temporal prediction prior information (Temporal Prior) are input to the Motion Entropy Model (prediction entropy model) to obtain the probability distribution of the prediction information.

[0344] 9. The encoder encodes the prediction information based on the probability distribution of the prediction information (i.e., AE operation) to obtain the prediction information bitstream (also called the motion information bitstream) and transmits the prediction information bitstream to the decoder.

[0345] 10. The encoder parses the prediction information bitstream based on the probability distribution of the prediction information (i.e., AD operation) to obtain the prediction information. The prediction information can also be dequantized to obtain the prediction latent feature mt.

[0346] 11. After obtaining the prediction information, the encoder stores the prediction information in the Motion Latent Buffer.

[0347] 12. The encoder inputs the prediction information to the prediction decoder (Motion Decoder) to obtain the prediction mode feature (also known as optical flow feature), prior prediction feature (also known as prior motion feature), and prediction confidence αt. The prediction confidence αt can represent the accuracy of the prediction feature. The prior prediction feature is stored in the information buffer.

[0348] For example, in addition to outputting the prior prediction feature and prediction mode feature, the prediction decoder can also output the prediction confidence αt. The output feature of a certain network layer of the prediction decoder is processed to obtain the prediction confidence αt.

[0349] Since the prediction confidence αt is obtained by decoding the prediction information, and the prediction information is parsed from the prediction information bitstream, the prediction confidence αt can be obtained based on the information in the prediction information bitstream.

[0350] 13. The encoding end inputs the prediction confidence αt, prediction pattern features, and historical temporal reference features into the temporal residual mining module. The temporal residual mining module can perform temporal residual mining based on the prediction confidence αt, prediction pattern features, and historical temporal reference features to obtain the initial prediction features. In summary, the encoding end can perform temporal information mining based on the prediction pattern features, prediction confidence, and historical temporal reference features to obtain the initial prediction features corresponding to the current image block. Specification 31 / 54 pages 37 CN 121603654 A

[0351] For example, the temporal residual mining module can obtain an initial predicted feature and output the initial predicted feature. The temporal residual mining module can also obtain two initial predicted features and output the sum of the initial predicted features.

[0352] In one possible implementation, if the temporal residual mining module outputs an initial predicted feature, it can perform a compensation operation on the historical temporal reference feature based on the predicted pattern feature to obtain a temporal compensated feature; and determine the initial predicted feature based on the temporal compensated feature and the prediction confidence. For example, a weighted operation is performed based on the temporal compensated feature and the prediction confidence to obtain the post-operation feature; and a first type of operation is performed based on the post-operation feature (such as performing a convolution operation and a residual block operation on the post-operation feature in sequence, and adding the feature after the residual block operation to the post-operation feature) to obtain the initial predicted feature. Alternatively, a second type of operation can be performed based on the temporal compensation features (e.g., the second type of operation is to sequentially perform convolution and residual block operations on the temporal compensation features, and then add the features after the residual block operation to the temporal compensation features) to obtain the post-operation features; a weighted operation is performed based on the post-operation features and the prediction confidence to obtain the initial prediction features.

[0353] In one possible implementation, if the temporal residual mining module outputs two initial prediction features, the prediction pattern features may include a first prediction pattern feature and a second prediction pattern feature, the historical temporal reference features may include a first historical temporal reference feature and a second historical temporal reference feature, the prediction confidence may include a first prediction confidence and a second prediction confidence, and the initial prediction features may include a first initial prediction feature and a second initial prediction feature.

[0354] Based on this, a compensation operation can be performed on the first historical temporal reference feature based on the first prediction pattern feature to obtain the first temporal compensation feature; the first initial prediction feature is determined based on the first temporal compensation feature and the first prediction confidence.

[0355] The second historical temporal reference feature can be compensated based on the second prediction mode feature to obtain the second temporal compensated feature; the second initial prediction feature is determined based on the second temporal compensated feature and the second prediction confidence.

[0356] 14. The encoder determines the target prediction feature corresponding to the current image block based on the initial prediction feature. For example, the initial prediction feature can be directly determined as the target prediction feature. Alternatively, the initial prediction feature can be transformed to obtain the target prediction feature, that is, the prediction feature after the transformation is used as the target prediction feature. For example, the initial prediction feature can be processed by a certain algorithm, and the processed feature is used as the target prediction feature. Of course, the above is just an example and is not limited thereto, as long as the initial prediction feature is transformed to obtain the target prediction feature.

[0357] 15. The encoder inputs the current image feature Ft and the target prediction feature (such as the target prediction feature and the target prediction feature) to the residual encoder.The encoder obtains the unquantized residual information feature yt. The encoder quantizes the unquantized residual information feature yt (i.e., Q operation) to obtain the residual information, i.e., the residual latent feature.

[0358] 16. The encoder inputs the target prediction feature (such as prediction feature) to the spatial prior encoder to obtain the spatial prior information.

[0359] 17. The residual prior information is stored in the Contextual Latent Buffer. The encoder inputs the residual information feature yt, the spatial prior information and the residual prior information to the Hyper Enc to obtain the residual side information bitstream. The encoder sends the residual side information bitstream to the decoder.

[0360] 18. The Hyper Dec of the encoder parses the residual side information bitstream to obtain the residual hyper prior information.

[0361] 19. The encoding end inputs the residual prior information, spatial prior information, and residual prior information into the Contextual Entropy Model to obtain the probability distribution of the residual information (i.e., the residual latent features). (See manual, pages 32 / 54, 38 CN 121603654 A)

[0362] 20. The encoding end encodes the residual latent features based on the probability distribution of the residual latent features (i.e., AE operation) to obtain the residual information bitstream. The encoding end then transmits the residual information bitstream to the decoding end.

[0363] 21. The encoding end parses the residual information bitstream based on the probability distribution of the residual latent features (i.e., AD operation) to obtain the residual latent features. The residual latent features can also be dequantized to obtain the residual information feature yt. The encoding end stores the residual latent features in the Contextual Latent Buffer for residual prior information of subsequent image blocks

[0364] 22. The encoding end inputs the residual latent features and target prediction features (such as target prediction features and / or target prediction features, taking target prediction features as an example) to the residual decoder, and the residual decoder decodes the residual information based on the residual latent features and target prediction features (such as target prediction features) to obtain the target residual features

[0365] 23. The encoding end determines the image reference information corresponding to the current image block based on the target residual features and target prediction features (such as target prediction features). For example, the target prediction features and target residual features are added together, and the added features are input to the frame generator to obtain the image reference information and the image reference information is stored in the information buffer.

[0366] 24. The encoding end generates the reconstructed image block corresponding to the current image block xt

[0367] In one possible implementation, the above processing of the encoding end can be performed by a deep learning model or a neural network.The end-to-end image compression and encoding process is achieved through the network model, and no restrictions are placed on this encoding process.

[0368] In this embodiment of the application, a decoding method is proposed. Regarding the processing process of the decoding end, please refer to Figure 7B. Of course, Figure 7B is only an example of the processing process of the decoding end, and no restrictions are placed on this processing process of the decoding end.

[0369] 1. The decoding end obtains the prediction side information bitstream (also known as the motion side information bitstream).

[0370] 2. The Hyper Dec of the decoding end parses the prediction side information bitstream to obtain the prediction prior information. The decoding end inputs the prediction prior information and the temporal prediction prior information to the Motion Entropy Model to obtain the probability distribution of the prediction information. For example, the Motion Latent Buffer stores the temporal prediction prior information.

[0371] 3. The decoding end obtains the prediction information bitstream (the prediction information bitstream can also be called the motion information bitstream).

[0372] 4. The decoding end parses the prediction information bitstream based on the probability distribution of the prediction information (i.e., AD operation) to obtain the prediction information. The prediction information can also be dequantized to obtain the prediction latent feature mt. The decoding end stores the prediction information in the Motion Latent Buffer for the prior information of the temporal prediction of subsequent image blocks.

[0373] 5. The decoding end inputs the prediction information to the prediction decoder, which decodes the prediction information based on the prediction information to obtain the prediction mode feature (also known as optical flow feature), prior prediction feature (also known as prior motion feature) and prediction confidence αt. The prediction confidence αt can represent the accuracy of the prediction feature. The prediction decoder can also decode the prediction information based on the prediction information to obtain the prediction mode feature and prediction confidence αt.

[0374] For example, in addition to outputting the prior prediction feature and prediction mode feature, the prediction decoder can also output the prediction confidence αt. The output feature of a certain network layer of the prediction decoder is processed to obtain the prediction confidence αt.

[0375] Since the prediction confidence αt is obtained by decoding the prediction information, and the prediction information is parsed from the prediction information bitstream, the prediction confidence αt can be obtained based on the information in the prediction information bitstream.

[0376] 6. The decoding end inputs the prediction confidence αt, prediction mode features, and historical temporal reference features into the temporal residual mining module. The temporal residual mining module can perform temporal residual mining based on the prediction confidence αt, prediction mode features, and historical temporal reference features to obtain the initial prediction features. In summary, the decoding end can perform temporal information mining based on the prediction mode features, prediction confidence, and historical temporal reference features to obtain the initial prediction features corresponding to the current image block.

[0377] For example, the temporal residual mining module can obtain one initial prediction feature and output the initial prediction feature; the temporal residual mining module can also obtain two initial prediction features and output the initial prediction feature sum.

[0378] In one possible implementation, if the temporal residual mining module outputs one initial prediction feature, then a compensation operation is performed on the historical temporal reference feature based on the prediction pattern feature to obtain a temporal compensated feature; based on the temporal compensated feature and the prediction confidence, the initial prediction feature is determined. Alternatively, if the temporal residual mining module outputs two initial prediction features, then the prediction pattern feature includes a first prediction pattern feature and a second prediction pattern feature, the historical temporal reference feature includes a first historical temporal reference feature and a second historical temporal reference feature, the prediction confidence includes a first prediction confidence and a second prediction confidence, and the initial prediction feature includes a first initial prediction feature and a second initial prediction feature. A compensation operation is performed on the first historical temporal reference feature based on the first prediction pattern feature to obtain a first temporal compensated feature; based on the first temporal compensated feature and the first prediction confidence, the first initial prediction feature is determined. The second historical temporal reference features are compensated based on the second prediction mode features to obtain the second temporal compensated features; the second initial prediction features are determined based on the second temporal compensated features and the second prediction confidence.

[0379] 7. The decoding end determines the target prediction features corresponding to the current image block based on the initial prediction features. For example, the initial prediction features are directly determined as the target prediction features. Or, the initial prediction features are transformed to obtain the target prediction features.

[0380] 8. The decoding end obtains the residual side information bitstream. The Hyper Dec (super prior decoder) of the decoding end parses the residual side information bitstream to obtain the residual super prior information (Hyper Prior). The decoding end inputs the target prediction features (such as the target prediction features) to the Spatial Prior Enc to obtain the Spatial Prior information (Spatial Prior). The Contextual Latent Buffer (residual latent buffer) stores the residual prior information (i.e., residual information).

[0381] 9. The decoding end inputs the residual prior information, spatial prior information, and residual prior information into the Contextual Entropy Model to obtain the probability distribution of the residual latent features. This probability distribution is used to guide the decoding.

[0382] 10. The decoding end obtains the residual information bitstream. Based on the probability distribution of the residual latent features, the residual information bitstream is parsed (AD operation) to obtain the residual latent features. The residual latent features can also be dequantized to obtain the residual information features yt. The residual latent features are stored in the Contextual Latent Buffer for the residual prior information of subsequent image blocks.

[0383] 11. The decoding end inputs the residual latent features and target prediction features (such as target prediction features and / or target prediction features, taking target prediction features as an example) to the residual decoder. The residual decoder decodes the residual information based on the residual latent features and target prediction features (such as target prediction features) to obtain the target residual features.

[0384] 12. The decoding end determines the image reference information corresponding to the current image block based on the target residual features and target prediction features (such as target prediction features). For example, the target prediction features and target residual features are added together, and the added features are input to the frame generator to obtain the image reference information and store the image reference information in the information buffer.

[0385] 13. The decoding end generates the reconstructed image block corresponding to the current image block xt. (Page 34 / 54 of the specification, 40 CN 121603654 A)

[0386] In one possible implementation, the above-mentioned decoding end processing can be executed by a deep learning model or a neural network model to realize the end-to-end image compression and decoding process. There are no restrictions on this decoding process.

[0387] Example 8: In Example 3, the residual confidence α is output by the residual coding module (residual decoding module), while the predicted feature is output by the prediction module. When the residual confidence α represents the accuracy of the predicted feature, the residual confidence α may not accurately reflect the accuracy of the predicted feature. Based on this, the confidence obtained from the prediction module is more accurate. Compared with Examples 5 and 6, instead of extracting the confidence from the prediction module, the confidence is directly applied to the prediction module. For example, the confidence can be applied to the prediction decoder (Motion Decoder) of the prediction module, or it can be applied to the temporal residual mining module (Temporal Context Mining) of the prediction module. The temporal residual mining module will be used as an example in the following.

[0388] An encoding method is proposed in this application embodiment. Regarding the processing of the encoding end, please refer to Figure 8A. Of course, Figure 8A is only an example of the processing of the encoding end, and the processing of the encoding end is not limited.

[0389] 1. The encoding end inputs the current image block xt to the feature extractor to obtain the current image feature Ft.

[0390] 2. The information buffer (Decoded Feature & Picture Buffer) stores the image reference information that the current image block xt can refer to. Historical temporal reference features can be derived from this image reference information.

[0391] 3. The information buffer (Decoded Feature & Picture Buffer) stores historical prior prediction features.

[0392] 4. The encoder performs motion estimation on the current image block xt and the historical image block xt-1 to obtain optical flow information vt.

[0393] 5. The encoder inputs the current image feature Ft, optical flow information vt, historical temporal reference features, and historical prior motion features to the predictive encoder (Motion Encoder) to obtain unquantized predictive latent features mt.

[0394] 6. The predictive latent features mt are quantized (Q operation) to obtain predictive information, i.e., motion information features.

[0395] 7. The Motion Latent Buffer stores temporal predictive prior information, predictive latent features mt, and temporal predictive prior information is input to the Hyper Enc (super prior encoder) to obtain the predictive side information bitstream (i.e., motion side information bitstream).

[0396] The encoder transmits the predictive side information bitstream to the Hyper Dec at the decoder. The Hyper Dec (super prior decoder) at the encoder parses the predictive side information bitstream to obtain predictive super prior information (Hyper Prior).

[0397] 8. The Hyper Prior and Temporal Prior are input into the Motion Entropy Model to obtain the probability distribution of the prediction information.

[0398] 9. The encoder encodes the prediction information based on the probability distribution (i.e., AE operation) to obtain the prediction information bitstream (also called motion information bitstream) and transmits the prediction information bitstream to the decoder.

[0399] 10. The encoder parses the prediction information bitstream based on the probability distribution (i.e., AD operation) to obtain the prediction information. The prediction information can also be dequantized to obtain the prediction latent feature mt.

[0400] 11. After obtaining the prediction information, the encoder stores the prediction information in the Motion Latent Buffer.

[0401] 12. The encoding end inputs the prediction information to the prediction decoder (Motion Decoder) to obtain the prediction mode features (also known as optical flow features), prior prediction features (also known as prior motion features), and prediction confidence αt. The prediction confidence αt can represent the accuracy of the prediction features. The prior prediction features are stored in the information buffer.

[0402] For example, in addition to outputting the prior prediction features and prediction mode features, the prediction decoder can also output the prediction confidence αt. The output features of a certain network layer of the prediction decoder are processed to obtain the prediction confidence αt.

[0403] 13. The encoding end inputs the prediction confidence αt, prediction mode features, and historical temporal reference features into the temporal residual.The temporal residual mining module can perform temporal residual mining based on the prediction confidence αt, prediction pattern features, and historical temporal reference features to obtain initial prediction features. In summary, the encoding end can perform temporal information mining based on prediction pattern features, prediction confidence, and historical temporal reference features to obtain the initial prediction features corresponding to the current image block.

[0404] For example, the temporal residual mining module can obtain one initial prediction feature and output the initial prediction feature. The temporal residual mining module can also obtain two initial prediction features and output the sum of the initial prediction features.

[0405] In one possible implementation, if the temporal residual mining module outputs one initial prediction feature, then the historical temporal reference features are compensated based on the prediction pattern features to obtain temporal compensated features; the initial prediction features are determined based on the temporal compensated features and the prediction confidence. Alternatively, if the temporal residual mining module outputs two initial prediction features, then the prediction pattern features include the first prediction pattern feature and the second prediction pattern feature, the historical temporal reference features include the first historical temporal reference feature and the second historical temporal reference feature, the prediction confidence includes the first prediction confidence and the second prediction confidence, and the initial prediction features include the first initial prediction feature and the second initial prediction feature. A compensation operation is performed on the first historical temporal reference feature based on the first prediction pattern feature to obtain the first temporal compensated feature; based on the first temporal compensated feature and the first prediction confidence, the first initial prediction feature is determined. A compensation operation is performed on the second historical temporal reference feature based on the second prediction pattern feature to obtain the second temporal compensated feature; based on the second temporal compensated feature and the second prediction confidence, the second initial prediction feature is determined.

[0406] 14. The encoding end inputs the current image feature Ft and the initial prediction features (such as the initial prediction feature and the initial prediction feature) to the residual encoder to obtain the unquantized residual information feature yt. The encoder quantizes the unquantized residual information feature yt (i.e., Q operation) to obtain the residual information, i.e., the residual latent feature.

[0407] 15. The encoder inputs the initial prediction feature (such as the prediction feature) to the spatial prior encoder to obtain the spatial prior information.

[0408] 16. The residual prior information is stored in the Contextual Latent Buffer. The encoder inputs the residual information feature yt, the spatial prior information, and the residual prior information to the Hyper Enc to obtain the residual side information bitstream. The encoder sends the residual side information bitstream to the decoder.

[0409] 17. The Hyper Dec of the encoder parses the residual side information bitstream to obtain the residual hyper prior information.

[0410] 18. The encoding end inputs the residual prior information, spatial prior information, and residual prior information into the Contextual Entropy Model to obtain the probability distribution of the residual information (i.e., residual latent features).

[0411] 19. The encoding end encodes the residual latent features based on the probability distribution of the residual latent features (i.e., AE operation) to obtain the residual information bitstream, and transmits the residual information bitstream to the decoding end.

[0412] 20. The encoding end parses the residual information bitstream based on the probability distribution of the residual latent features (i.e., AD operation) to obtain the residual latent features. The residual latent features can also be dequantized to obtain the residual information feature yt. The encoding end stores the residual latent features in the Contextual Latent Buffer for the residual prior information of subsequent image blocks. Instruction manual, pages 36 / 54, 42 CN 121603654 A

[0413] 21. The encoder inputs the residual latent features and initial prediction features (such as initial prediction features and / or initial prediction features as an example) to the residual decoder. The residual decoder decodes the residual information based on the residual latent features and initial prediction features (such as initial prediction features) to obtain the target residual features and residual confidence α. The residual confidence α can represent the accuracy of the prediction features, and the residual confidence α is obtained based on the information in the residual information bitstream.

[0414] 22. The encoder determines the target prediction features corresponding to the current image block based on the initial prediction features and residual confidence α.

[0415] For example, the initial prediction features and residual confidence can be weighted to obtain the target prediction features. For example, the initial prediction features and residual confidence can be weighted (such as multiplication) to obtain the weighted features, and the weighted features are used as the target prediction features. Alternatively, the initial predicted features and residual confidence scores can be weighted to obtain weighted features, and the target predicted features can be obtained by performing a transformation operation on the weighted features.

[0416] For example, when performing a weighted operation (such as multiplication) on the initial predicted features and residual confidence scores, the initial predicted features can be a three-dimensional feature matrix of C*H*W, and the residual confidence scores can include C confidence scores, or H*W confidence scores, or C*H*W confidence scores.

[0417] For example, when performing a transformation operation on the weighted features, a certain network can be used to process the weighted features, and the processed features can be used as the target predicted features. Alternatively, a certain algorithm can be used to process the weighted features, and the processed features can be used as the target predicted features. Of course, the above are just examples.No restrictions are imposed, as long as the weighted features are transformed to obtain the target prediction features.

[0418] 23. The encoding end determines the image reference information corresponding to the current image block based on the target residual features and the target prediction features (such as the target prediction features and / or the target prediction features, which are used as an example in this embodiment).

[0419] For example, the target prediction features and the target residual features are added together, and the added features are input to the frame generator to obtain the image reference information and store the image reference information in the information buffer.

[0420] 24. The encoding end generates the reconstructed image block corresponding to the current image block xt.

[0421] In one possible implementation, the above-mentioned processing of the encoding end can be executed by a deep learning model or a neural network model to realize the end-to-end image compression and encoding process. There are no restrictions on this encoding process.

[0422] In this embodiment, a decoding method is proposed. Regarding the processing of the decoding end, please refer to Figure 8B. Of course, Figure 8B is only an example of the processing of the decoding end, and there are no restrictions on this processing of the decoding end.

[0423] 1. The decoder obtains the prediction side information bitstream (also known as the motion side information bitstream).

[0424] 2. The decoder's Hyper Dec parses the prediction side information bitstream to obtain the prediction prior information. The decoder inputs the prediction prior information and the temporal prediction prior information into the Motion Entropy Model to obtain the probability distribution of the prediction information. For example, the Motion Latent Buffer stores the temporal prediction prior information.

[0425] 3. The decoder obtains the prediction information bitstream (the prediction information bitstream can also be called the motion information bitstream).

[0426] 4. The decoder parses the prediction information bitstream based on the probability distribution of the prediction information (i.e., AD operation) to obtain the prediction information. The prediction information can also be dequantized to obtain the prediction latent feature mt. The decoder stores the prediction information in the Motion Latent Buffer for the temporal prediction prior information of subsequent image blocks.

[0427] 5. The decoding end inputs the prediction information to the prediction decoder, which decodes the prediction information based on the prediction information (page 37 / 54 of the specification, 43 CN 121603654 A) to obtain the prediction mode features (also known as optical flow features), prior prediction features (also known as prior motion features), and prediction confidence αt. The prediction confidence αt can represent the accuracy of the prediction features. The prediction decoder can also decode the prediction information based on the prediction information to obtain the prediction mode features and prediction confidence αt.

[0428] 6. The decoding end inputs the prediction confidence αt, prediction mode features, and historical time-domain reference features into the time-domain residual.The mining module, the temporal residual mining module can perform temporal residual mining based on the prediction confidence αt, prediction mode features and historical temporal reference features to obtain initial prediction features. In summary, the decoding end can perform temporal information mining based on the prediction mode features, prediction confidence and historical temporal reference features to obtain the initial prediction features corresponding to the current image block.

[0429] For example, the temporal residual mining module can obtain one initial prediction feature and output the initial prediction feature. The temporal residual mining module can also obtain two initial prediction features and output the sum of the initial prediction features.

[0430] 7. The decoding end obtains the residual side information bitstream. The Hyper Dec (super prior decoder) of the decoding end parses the residual side information bitstream to obtain residual super prior information (Hyper Prior). The decoding end inputs the initial prediction features (such as the initial prediction features) to the Spatial Prior Enc to obtain the Spatial Prior information (Spatial Prior). The Contextual Latent Buffer stores residual prior information (i.e., residual information).

[0431] 8. The decoder inputs the residual super-prior information, spatial prior information, and residual prior information into the Contextual Entropy Model to obtain the probability distribution of the residual latent features. This probability distribution is used to guide the decoding.

[0432] 9. The decoder obtains the residual information bitstream. Based on the probability distribution of the residual latent features, the residual information bitstream is parsed (AD operation) to obtain the residual latent features. The residual latent features can also be dequantized to obtain the residual information feature yt. The residual latent features are stored in the Contextual Latent Buffer for residual prior information of subsequent image blocks.

[0433] 10. The decoding end inputs the residual latent features and the initial prediction features (such as the initial prediction features and / or the initial prediction features are taken as an example) to the residual decoder. The residual decoder decodes the residual information based on the residual latent features and the initial prediction features (such as the initial prediction features) to obtain the target residual features and the residual confidence α. The residual confidence α can represent the accuracy of the prediction features, and the residual confidence α is obtained based on the information in the residual information bitstream.

[0434] 11. The decoding end determines the target prediction features corresponding to the current image block based on the initial prediction features and the residual confidence α.

[0435] For example, the initial prediction features and the residual confidence can be weighted to obtain the target prediction features. For example, the initial prediction features and the residual confidence can be weighted (such as multiplication) to obtain the weighted features, and the weighted features are used as the target prediction features. Alternatively, the initial prediction features and residual confidence levels can be further refined.Perform weighted operations to obtain the weighted features, and perform transformation operations on the weighted features to obtain the target prediction features.

[0436] For example, when performing weighted operations (such as multiplication) on the initial prediction features and the residual confidence, the initial prediction features can be a three-dimensional feature matrix of C*H*W, and the residual confidence can include C confidence values, or the residual confidence can include H*W confidence values, or the residual confidence can include C*H*W confidence values. Specification 38 / 54 pages 44 CN 121603654 A

[0437] 12. The decoding end determines the image reference information corresponding to the current image block based on the target residual features and the target prediction features (such as the target prediction features and / or the target prediction features are used as an example in this embodiment).

[0438] For example, the target prediction feature and the target residual feature are added together, and the added feature is input to the frame generator to obtain image reference information and the image reference information is stored in the information buffer.

[0439] 13. The decoding end generates the reconstructed image block corresponding to the current image block xt

[0440] In one possible implementation, the above-mentioned decoding end processing can be executed by a deep learning model or a neural network model to realize the end-to-end image compression and decoding process. There are no restrictions on this decoding process.

[0441] Example 9: In Examples 7 and 8, the temporal residual mining module can perform temporal residual mining based on the prediction confidence αt, the prediction pattern feature and the historical temporal reference feature to obtain the initial prediction feature (or the target prediction feature, which will be used as an example in the following). The processing of the temporal residual mining module is described below.

[0442] In one possible implementation, see Figure 9A, which is a schematic diagram of the structure of the temporal residual mining module (temporal residual mining network). The predicted pattern features (denoted as), historical temporal reference features (denoted as), and prediction confidence αt (denoted as) can be input into the temporal residual mining module. The temporal residual mining module can perform a compensation operation (i.e., an alignment operation) on the historical temporal reference features based on the predicted pattern features to obtain temporal compensated features (such as optical flow features). Then, a weighted operation is performed based on the temporal compensated features and the prediction confidence to obtain the processed features (denoted as feature A). When performing the weighted operation, the temporal compensated features can be a three-dimensional feature matrix of C*H*W, and the prediction confidence can include C confidence values, or H*W confidence values, or C*H*W confidence values.

[0443] Convolution operation (i.e., Cov operation) and residual block operation (i.e., ResBlock operation) are performed on feature A in sequence, and the features after the residual block operation are added to feature A to obtain the initial predicted features. In this case, the temporal residual mining module...The mining module can obtain an initial prediction feature and output the initial prediction feature. For example, the residual block operation refers to processing the input feature through the residual block network. In this embodiment, the network structure of the residual block network is not limited.

[0444] In one possible implementation, see Figure 9B, which is a schematic diagram of the structure of the temporal residual mining module (temporal residual mining network). The prediction mode feature (denoted as), the historical temporal reference feature (denoted as), and the prediction confidence αt (denoted as) can be input into the temporal residual mining module. The temporal residual mining module can perform a compensation operation (i.e., Align operation) on the historical temporal reference feature based on the prediction mode feature to obtain temporal compensation features (such as optical flow features).

[0445] The temporal compensation features are sequentially subjected to convolution operation (i.e., Cov operation) and residual block operation (i.e., ResBlock operation), and the feature after the residual block operation is added to the temporal compensation feature to obtain the feature after the operation (denoted as feature B).

[0446] Based on feature B and prediction confidence, a weighted operation is performed to obtain the initial prediction feature. That is, the weighted feature is used as the initial prediction feature, or the weighted feature is transformed and then used as the initial prediction feature. In this case, the temporal residual mining module can obtain an initial prediction feature and output the initial prediction feature. When performing the weighted operation, feature B can be a three-dimensional feature matrix of C*H*W, and the prediction confidence can include C confidence values, or H*W confidence values, or C*H*W confidence values.

[0447] In one possible implementation, see Figure 9C, which is a schematic diagram of the structure of the temporal residual mining module (temporal residual mining network). In Figure 9C, it is a schematic diagram of confidence pre-positioning through the output adjustment of the alignment module (Align). Instruction manual, pages 39 / 54, 45, CN 121603654 A

[0448] : Prediction mode features may include first prediction mode features and second prediction mode features; historical time domain reference features may include first historical time domain reference features and second historical time domain reference features; prediction confidence αt may include first prediction confidence and second prediction confidence; initial prediction features may include first initial prediction features and second initial prediction features. For example, the feature dimension of the first prediction mode feature is greater than the feature dimension of the second prediction mode feature, such as the width of αt being greater than the width of αt, and / or the height of αt being greater than the height of αt, and / or the number of channels of αt being greater than the number of channels of αt. The feature dimension of the first historical time domain reference feature is greater than the feature dimension of the second historical time domain reference feature, such as the width of αt being greater than the width of αt, and / or the height of αt being greater than the height of αt, and / or the number of channels of αt being greater than the number of channels of αt. The feature dimension of the first initial prediction feature is greater than the feature dimension of the second initial prediction feature, such as αt being greater than the width of αt, and / or the height of αt being greater than the height of αt, and / or the number of channels of αt being greater than the number of channels of αt.The width of is greater than the width of , and / or, the height of is greater than the height of , and / or, the number of channels of is greater than the number of channels of . The dimension of the first prediction confidence is greater than the dimension of the second prediction confidence, such as the width of is greater than the width of , and / or, the height of is greater than the height of , and / or, the number of channels of is greater than the number of channels of .

[0449] In this case, the prediction confidence output by the prediction decoder (Motion Decoder) can be a first prediction confidence. By downsampling the first prediction confidence, a second prediction confidence can be obtained. Alternatively, the first prediction confidence can be used as the second prediction confidence. Or, the prediction confidence output by the prediction decoder can be a second prediction confidence. By upsampling the second prediction confidence, a first prediction confidence can be obtained. Alternatively, the second prediction confidence can be used as the first prediction confidence. For example, if the prediction confidence includes C confidence values, then the first prediction confidence and the second prediction confidence can be the same. If the prediction confidence includes H*W confidence values, or C*H*W confidence values, then the width of is greater than the width of, and / or the height of is greater than the height of.

[0450] For another example, the feature dimension of the first prediction mode feature can be smaller than the feature dimension of the second prediction mode feature, such as the width of is less than the width of, and / or the height of is less than the height of, and / or the number of channels of is less than the number of channels of. The feature dimension of the first historical time-domain reference feature can be smaller than that of the second historical time-domain reference feature, such as the width of being smaller than the width of , and / or the height of being smaller than the height of , and / or the number of channels of being smaller than the number of channels of . The feature dimension of the first initial prediction feature can be smaller than that of the second initial prediction feature, such as the width of being smaller than the width of , and / or the height of being smaller than the height of , and / or the number of channels of being smaller than the number of channels of . The dimension of the first prediction confidence can be smaller than that of the second prediction confidence, such as the width of being smaller than the width of , and / or the height of being smaller than the height of , and / or the number of channels of being smaller than the number of channels of .

[0451] In this case, the prediction confidence output by the prediction decoder (Motion Decoder) can be a first prediction confidence. By upsampling the first prediction confidence, a second prediction confidence can be obtained. Alternatively, the first prediction confidence can be used as the second prediction confidence. Or, the prediction confidence output by the prediction decoder can be a second prediction confidence. By downsampling the second prediction confidence, a first prediction confidence can be obtained. Alternatively, the second prediction confidence can be used as the first prediction confidence. For example, if the prediction confidence includes C confidence values, then the first prediction confidence and the second prediction confidence can be the same. If the prediction confidence includes H*W confidence values, or C*H*W confidence values, then the width is less than...The width and / or the height is less than the height. Specification 40 / 54 pages 46 CN 121603654 A

[0452] The first prediction mode feature, the second prediction mode feature, the first historical time-domain reference feature, the second historical time-domain reference feature, the first prediction confidence, and the second prediction confidence are input into the time-domain residual mining module. The time-domain residual mining module performs a compensation operation (Align operation) on the second historical time-domain reference feature based on the second prediction mode feature to obtain the second time-domain compensated feature. A weighted operation is performed based on the second time-domain compensated feature and the second prediction confidence to obtain the processed feature (denoted as feature C). During the weighted operation, the second time-domain compensated feature is a three-dimensional feature matrix of C*H*W, and the second prediction confidence includes C confidence values, or H*W confidence values, or C*H*W confidence values.

[0453] Then, convolution operation (i.e., Cov operation) and residual block operation (i.e., ResBlock operation) are performed on feature C in sequence, and the feature after residual block operation is added to feature C to obtain the second initial prediction feature.

[0454] The temporal residual mining module can perform compensation operation (i.e., Align operation) on the first historical temporal reference feature based on the first prediction mode feature to obtain the first temporal compensated feature. The calculated feature (denoted as feature D) is obtained by weighting the first temporal compensated feature and the first prediction confidence. The first temporal compensated feature is a three-dimensional feature matrix of C*H*W, and the first prediction confidence includes C confidence values, or H*W confidence values, or C*H*W confidence values.

[0455] The feature C is upsampled (Up2) to obtain the sampled temporal compensated feature. For example, if the feature dimension of the first prediction mode feature is greater than the feature dimension of the second prediction mode feature, then an upsampling operation is performed. If the feature dimension of the first prediction mode feature is less than the feature dimension of the second prediction mode feature, then a downsampling operation is performed on feature C to obtain the sampled temporal compensation feature. Channel concatenation (C) is performed on feature D and the sampled temporal compensation feature. Of course, other methods can also be used to operate on feature D and the sampled temporal compensation feature.

[0456] Then, convolution operation (Cov operation) and residual block operation (ResBlock operation) are performed on the channel concatenated feature (i.e., the feature after C operation, such as the channel concatenated temporal compensation feature) in sequence, and the feature after the residual block operation is added to feature D to obtain the first initial prediction feature. In summary, the temporal residual mining module can obtain the first initial prediction feature and the second initial prediction feature and output these two initial prediction features.

[0457] In one possible implementation, as shown in Figure 9D, the temporal residual mining module (temporal residual mining network) is...Figure 9D is a schematic diagram of the structure of the network. In Figure 9D, it is a schematic diagram of the confidence level being adjusted by the output of the prediction module.

[0458] The prediction mode features include the first prediction mode features and the second prediction mode features. The historical time domain reference features include the first historical time domain reference features and the second historical time domain reference features. The prediction confidence αt includes the first prediction confidence and the second prediction confidence. The initial prediction features include the first initial prediction features and the second initial prediction features.

[0459] The first prediction mode features, the second prediction mode features, the first historical time domain reference features, the second historical time domain reference features, the first prediction confidence and the second prediction confidence are input into the time domain residual mining module. The time domain residual mining module performs a compensation operation (Align operation) on the second historical time domain reference features based on the second prediction mode features to obtain the second time domain compensated features. Then, the second time domain compensated features are sequentially subjected to convolution operation (i.e., Cov operation) and residual block operation (i.e., ResBlock operation), and the features after the residual block operation are added to the second time domain compensated features to obtain the operated features (denoted as feature D). The second initial prediction feature is obtained by weighting the feature D and the second prediction confidence. For example, the weighted feature is used as the second initial prediction feature, or the weighted feature is transformed and then used as the second initial prediction feature. When performing the weighting operation, the second prediction confidence may include C confidence values, or H*W confidence values, or C*H*W confidence values.

[0460] The temporal residual mining module performs a compensation operation (i.e., Align operation) on the first historical temporal reference feature based on the first prediction mode feature to obtain the first temporal compensated feature. The second temporal compensated feature is upsampled (Up2) to obtain the sampled temporal compensated feature. The first temporal compensated feature and the sampled temporal compensated feature are connected (Concat), i.e., C operation. Other methods can also be used to operate on the first temporal compensated feature and the sampled temporal compensated feature.

[0461] Then, convolution (i.e., Cov operation) and residual block operation (i.e., ResBlock operation) can be performed sequentially on the channel-connected features (i.e., the features after C operation, such as the temporal compensation features after channel connection), and the features after the residual block operation are added to the first temporal compensation features to obtain the operated features (denoted as feature E).

[0462] Then, a weighted operation is performed based on feature E and the first prediction confidence to obtain the first initial prediction feature. For example, the weighted feature is used as the first initial prediction feature, or the weighted feature is transformed and then used as the first initial prediction feature. When performing the weighted operation, the first initial prediction feature...It can include C confidence values, or H*W confidence values, or C*H*W confidence values.

[0463] Of course, Figures 9A-9D are just a few examples of the structure of the temporal residual mining module. In this embodiment, the structure of this temporal residual mining module is not limited, as long as the temporal residual mining module can output the initial prediction features.

[0464] Example 10: Based on Examples 3-9, the confidence can be applied to more locations to improve compression efficiency. For example, in the inter-frame prediction process, not all regions of the current frame are suitable for reference. The confidence can be introduced into the image reference information for differentiation, which helps the network learn features worth referencing and suppresses features that are not suitable for reference.

[0465] For example, for the encoding end and the decoding end, the prediction information bitstream corresponding to the current image block can be parsed to obtain the prediction information corresponding to the current image block. The prediction information is decoded to obtain the prediction mode features (also known as optical flow features) corresponding to the current image block. Based on the predicted pattern features and historical image reference information, the predicted features (such as predicted features and predicted features) corresponding to the current image patch are determined. For example, the information buffer (Decoded Feature & Picture Buffer) stores image reference information that the current image patch xt can refer to. This image reference information is called historical image reference information. Historical temporal reference features can be derived from historical image reference information. For example, historical image reference information can be input into the feature extractor to obtain historical temporal reference features, or historical image reference information can be directly used as historical temporal reference features. Based on this, the predicted features corresponding to the current image patch can be determined based on the predicted pattern features and historical temporal reference features.

[0466] For the encoding and decoding ends, the residual information bitstream corresponding to the current image block can be parsed to obtain the residual information corresponding to the current image block. The residual information can be residual decoded to obtain the target residual feature corresponding to the current image block.

[0467] The initial image reference information corresponding to the current image block can be determined based on the target residual feature and the prediction feature. For example, the prediction feature and the target residual feature can be added together, and the feature after addition can be input to the frame generator. The frame generator outputs the image reference information and uses this image reference information as the initial image reference information.

[0468] After obtaining the initial image reference information, instead of directly storing the initial image reference information in the information buffer specification (pages 42 / 54, CN 121603654 A), the reference confidence corresponding to the initial image reference information is obtained. The reference confidence can represent the initial image reference.The accuracy of the reference information. Based on this, the target image reference information corresponding to the current image block is determined based on the reference confidence and the initial image reference information. The target image reference information is used as the historical image reference information for subsequent image blocks. For example, the target image reference information can be stored in the information buffer, or, after performing a transformation operation on the target image reference information, the transformed features are stored in the information buffer, so as to serve as the image reference information that can be referenced by subsequent image blocks. The image reference information can be called historical image reference information.

[0469] For example, for embodiment 3, when performing residual decoding on the residual information, the target residual feature and the first reference feature can be obtained (that is, when the residual decoder performs residual decoding on the residual information, it can output two features, one of which is called the target residual feature, and the other is called the first reference feature. The first reference feature is the output feature related to the residual confidence α). Based on this, the residual confidence α can be determined based on the first reference feature, and the reference confidence α′ can be determined based on the residual confidence α. For example, the residual confidence α can be used as the reference confidence α′, or the residual confidence α can be transformed to obtain the reference confidence α′. Alternatively, the reference confidence α′ can be obtained by performing a third type of operation based on the first reference feature. For example, the first reference feature can be convolved to obtain the convolved feature, and the convolved feature can be transformed to obtain the reference confidence α′. Alternatively, the first reference feature can be residual block operated to obtain the residual block feature, and the residual block feature can be transformed to obtain the reference confidence α′. In summary, the reference confidence α′ is the confidence obtained when performing residual decoding on the residual information.

[0470] Based on embodiment 3, after obtaining the reference confidence α′ corresponding to the initial image reference information, the target image reference information corresponding to the current image block is determined based on the reference confidence α′ and the initial image reference information. For example, a weighted operation (such as multiplication) is performed based on the reference confidence α′ and the initial image reference information to obtain the post-operation features. Then, an addition operation is performed based on the post-operation features and the initial image reference information to obtain the target image reference information.

[0471] When performing the weighted operation, the initial image reference information can be a three-dimensional feature matrix of C*H*W, and the reference confidence α′ can include C confidence values, or H*W confidence values, or C*H*W confidence values.

[0472] For example, for embodiment 4, when decoding the prediction information, a prediction mode feature and a second reference feature can be obtained (that is, when decoding the prediction information, two features can be output, one output feature is called the prediction mode feature, and the other output feature is called the second reference feature, the second reference feature is related to the prediction mode feature).(Output features related to confidence αt). Based on this, the prediction confidence αt can be determined based on the second reference feature, and the reference confidence α′ can be determined based on the prediction confidence αt. For example, the prediction confidence αt can be used as the reference confidence α′, or the prediction confidence αt can be transformed to obtain the reference confidence α′. Alternatively, the reference confidence α′ can be obtained by performing a third type of operation based on the second reference feature. For example, the convolution operation can be performed on the second reference feature to obtain the convolutional feature, and the transformation operation can be performed on the convolutional feature to obtain the reference confidence α′. Alternatively, the residual block operation can be performed on the second reference feature to obtain the residual block feature, and the transformation operation can be performed on the residual block feature to obtain the reference confidence α′. In summary, the reference confidence α′ is the confidence obtained when decoding the prediction information.

[0473] Based on Example 4, after obtaining the reference confidence α′ corresponding to the initial image reference information, the target image reference information corresponding to the current image block is determined based on the reference confidence α′ and the initial image reference information. For example, a weighted operation (such as multiplication) is performed based on the reference confidence α′ and the initial image reference information to obtain the post-operation features. Then, an addition operation is performed based on the post-operation features and the initial image reference information to obtain the target image reference information. Specification 43 / 54 pages 49 CN 121603654 A

[0474] For example, for embodiment 5, when performing residual decoding on the residual information, target residual features and first reference features can be obtained. Based on this, the residual confidence α can be determined based on the first reference feature, and the reference confidence α′ can be determined based on the residual confidence α. For example, the residual confidence α can be used as the reference confidence α′, or the residual confidence α can be transformed to obtain the reference confidence α′. Alternatively, a third type of operation can be performed based on the first reference feature to obtain the reference confidence α′. For example, a convolution operation is performed on the first reference feature to obtain the post-convolution features, and a transformation operation is performed on the post-convolution features to obtain the reference confidence α′. Alternatively, a residual block operation can be performed on the first reference feature to obtain a feature after the residual block operation, and a transformation operation can be performed on the feature after the residual block operation to obtain a reference confidence α′.

[0475] Alternatively, when decoding the prediction information, a prediction pattern feature and a second reference feature can be obtained. Based on this, a prediction confidence αt can be determined based on the second reference feature, and a reference confidence α′ can be determined based on the prediction confidence αt. For example, the prediction confidence αt can be used as the reference confidence α′, or a transformation operation can be performed on the prediction confidence αt to obtain the reference confidence α′. Alternatively, a third type of operation can be performed based on the second reference feature to obtain the reference confidence α′. For example, a convolution operation can be performed on the second reference feature to obtain a feature after the convolution operation, and a transformation operation can be performed on the feature after the convolution operation.A transformation operation is performed to obtain a reference confidence α′. Alternatively, a residual block operation is performed on the second reference feature to obtain a feature after the residual block operation, and a transformation operation is performed on the feature after the residual block operation to obtain a reference confidence α′.

[0476] Alternatively, when performing residual decoding on the residual information, a target residual feature and a first reference feature can be obtained. When performing prediction information decoding on the prediction information, a prediction pattern feature and a second reference feature can be obtained. Based on this, a residual confidence can be determined based on the first reference feature, a prediction confidence can be determined based on the second reference feature, and a reference confidence can be generated based on the residual confidence and the prediction confidence. For example, the prediction confidence αt and the residual confidence α are input to the network layer C, and the network layer C processes the prediction confidence αt and the residual confidence α to obtain a reference confidence α′. For example, the prediction confidence and the residual confidence can be concatted to obtain a feature after concatting. A convolution operation (e.g., through several convolutional layers) is performed on the channel-connected features to obtain a reference confidence α′.

[0477] Alternatively, after obtaining the first and second reference features, a third reference feature is generated based on the first and second reference features. For example, the first and second reference features are input into network layer C, and network layer C processes them based on the first and second reference features to obtain the third reference feature. For example, the first and second reference features are concatenated to obtain the channel-connected features. A convolution operation (e.g., through several convolutional layers) is performed on the channel-connected features to obtain the third reference feature.

[0478] After obtaining the third reference feature, a third type of operation can be performed based on the third reference feature to obtain a reference confidence. For example, a convolution operation can be performed on the third reference feature to obtain the convolutional features, and a transformation operation can be performed on the convolutional features to obtain the reference confidence α′. Alternatively, a residual block operation can be performed on the third reference feature to obtain the feature after the residual block operation, and a transformation operation can be performed on the feature after the residual block operation to obtain the reference confidence α′.

[0479] In summary, it can be seen that the first reference feature is obtained when the residual information is decoded, and the second reference feature is obtained when the prediction information is decoded. The reference confidence is obtained based on the first reference feature and the second reference feature.

[0480] Based on embodiment 5, after obtaining the reference confidence α′ corresponding to the initial image reference information, the target image reference information corresponding to the current image block is determined based on the reference confidence α′ and the initial image reference information. For example, a weighted operation (such as multiplication) is performed based on the reference confidence α′ and the initial image reference information to obtain the feature after the operation. Then, an addition operation is performed based on the feature after the operation and the initial image reference information to obtain the target image reference information. Specification 44 / 54 pages 50CN 121603654 A

[0481] For example, in embodiment 7, when decoding the prediction information, a prediction pattern feature and a second reference feature can be obtained. Based on this, a prediction confidence αt can be determined based on the second reference feature, and a reference confidence α′ can be determined based on the prediction confidence αt. Alternatively, a third type of operation can be performed based on the second reference feature to obtain the reference confidence α′. Based on embodiment 7, after obtaining the reference confidence α′ corresponding to the initial image reference information, the target image reference information corresponding to the current image block is determined based on the reference confidence α′ and the initial image reference information.

[0482] For example, in embodiment 8, when decoding the residual information, a target residual feature and a first reference feature can be obtained. Based on this, a residual confidence α can be determined based on the first reference feature, and a reference confidence α′ can be determined based on the residual confidence α. Alternatively, a third type of operation can be performed based on the first reference feature to obtain the reference confidence α′.

[0483] Alternatively, when decoding the prediction information, a prediction pattern feature and a second reference feature can be obtained. Based on this, a prediction confidence αt can be determined based on the second reference feature, and a reference confidence α′ can be determined based on the prediction confidence αt. Alternatively, a third type of operation can be performed based on the second reference feature to obtain the reference confidence α′.

[0484] Alternatively, when decoding the residual information, a target residual feature and a first reference feature can be obtained. When decoding the prediction information, a prediction pattern feature and a second reference feature can be obtained.

[0485] Based on this, a residual confidence can be determined based on the first reference feature, a prediction confidence can be determined based on the second reference feature, and a reference confidence can be determined based on the residual confidence and the prediction confidence. Alternatively, after obtaining the first reference feature and the second reference feature, a third reference feature is generated based on the first reference feature and the second reference feature, and a third type of operation is performed based on the third reference feature to obtain the reference confidence.

[0486] Based on Embodiment 8, after obtaining the reference confidence α′ corresponding to the initial image reference information, the target image reference information corresponding to the current image block is determined based on the reference confidence α′ and the initial image reference information.

[0487] In one possible implementation, taking the application of the reference confidence α′ to the initial image reference information to obtain the target image reference information based on Embodiment 3 as an example. Based on other embodiments, the reference confidence α′ can also be applied to the initial image reference information to obtain the target image reference information, and the implementation process is similar, so it will not be repeated here.

[0488] An encoding method is proposed in the embodiments of this application. Regarding the processing process of the encoding end, please refer to Figure 10A. Of course, Figure 10A is only an example of the processing process of the encoding end, and the processing process of the encoding end is not limited.

[0489] 1. The encoder inputs the current image block xt to the feature extractor to obtain the current image feature Ft.

[0490] 2. The information buffer (Decoded Feature & Picture Buffer) stores the image reference information that the current image block xt can refer to, i.e., the historical image reference information. The historical temporal reference features can be derived from this image reference information.

[0491] 3. The information buffer (Decoded Feature & Picture Buffer) stores the historical prior prediction features.

[0492] 4. The encoder performs motion estimation on the current image block xt and the historical image block xt-1 to obtain the optical flow information vt.

[0493] 5. The encoder inputs the current image feature Ft, the optical flow information vt, the historical temporal reference features, and the historical prior motion features to the predictive encoder (Motion Encoder) to obtain the unquantized predictive latent features mt.

[0494] 6. Quantize the predicted latent feature mt (Q operation) to obtain the predicted information, i.e., motion information feature specification 45 / 54 pages 51 CN 121603654 A

[0495] 7. The Motion Latent Buffer stores the temporal prediction prior information. The predicted latent feature mt and the temporal prediction prior information are input to the Hyper Enc (super prior encoder) to obtain the predicted side information bitstream (i.e., motion side information bitstream).

[0496] The encoding end transmits the predicted side information bitstream to the decoding end's Hyper Dec. The encoding end's Hyper Dec (super prior decoder) parses the predicted side information bitstream to obtain the predicted super prior information (Hyper Prior).

[0497] 8. The predicted super prior information (Hyper Prior) and the temporal prediction prior information (Temporal Prior) are input to the Motion Entropy Model (prediction entropy model) to obtain the probability distribution of the predicted information.

[0498] 9. The encoding end encodes the prediction information based on the probability distribution of the prediction information (i.e., AE operation) to obtain the prediction information bitstream (also called motion information bitstream), and transmits the prediction information bitstream to the decoding end.

[0499] 10. The encoding end parses the prediction information bitstream based on the probability distribution of the prediction information (i.e., AD operation) to obtain the prediction information. The prediction information can also be dequantized to obtain the prediction latent feature mt.

[0500] 11. After obtaining the prediction information, the encoding end stores the prediction information in the Motion Latent Buffer.

[0501] 12. The encoding end inputs the prediction information to the prediction decoder (Motion Decoder), which then...The prediction information is decoded to obtain the prediction pattern features (also known as optical flow features) and prior prediction features (also known as prior motion features). For example, the prior prediction features are stored in the information buffer.

[0502] 13. The encoding end inputs the prediction pattern features and historical temporal reference features into the temporal residual mining module. The temporal residual mining module performs temporal residual mining based on the prediction pattern features and historical temporal reference features to obtain the prediction features.

[0503] For example, the temporal residual mining module can obtain one prediction feature and output the prediction feature, or the temporal residual mining module can also obtain two prediction features and output the prediction feature sum.

[0504] 14. The encoding end inputs the current image feature Ft and the prediction features (such as prediction feature and prediction feature) to the residual encoder to obtain the unquantized residual information feature yt. The encoding end quantizes the unquantized residual information feature yt (i.e., Q operation) to obtain the residual information. The residual information can be called the residual latent feature.

[0505] 15. The encoder inputs the predicted features (such as the predicted features) to the spatial prior encoder to obtain spatial prior information.

[0506] 16. The residual prior information is stored in the Contextual Latent Buffer. The encoder inputs the residual information features yt, the spatial prior information, and the residual prior information to the Hyper Enc to obtain the residual side information bitstream. The encoder sends the residual side information bitstream to the decoder.

[0507] 17. The Hyper Dec of the encoder parses the residual side information bitstream to obtain the residual hyper prior information.

[0508] 18. The encoder inputs the residual hyper prior information, the spatial prior information, and the residual prior information to the Contextual Entropy Model to obtain the probability distribution of the residual information (i.e., the residual latent features).

[0509] 19. The encoding end encodes the residual latent features based on the probability distribution of the residual latent features (i.e., AE operation) to obtain the residual information bitstream. The encoding end can transmit the residual information bitstream to the decoding end.

[0510] 20. The encoding end parses the residual information bitstream based on the probability distribution of the residual latent features (i.e., AD operation) to obtain the residual latent features. The residual latent features can also be dequantized to obtain the residual information feature yt. Encoding manual 46 / 54 pages 52 CN 121603654 A end stores the residual latent features in Contextual Latent Buffer for residual prior information of subsequent image blocks

[0511] 21. The encoding end stores the residual latent featuresThe predicted features (e.g.) are input to the residual decoder, which decodes the residual information based on the residual latent features and the predicted features to obtain the target residual features and the residual confidence α.

[0512] 22. The encoder performs a weighted operation on the predicted features (e.g., the predicted features) and the residual confidence α, and performs an addition operation on the weighted features and the target residual features, and inputs the added features to the frame generator, which obtains the initial image reference information.

[0513] 23. The encoder generates the reconstructed image block corresponding to the current image block xt.

[0514] 24. The encoder obtains the reference confidence α′ corresponding to the initial image reference information. The reference confidence α′ is used to represent the accuracy of the initial image reference information. Based on the reference confidence α′ and the initial image reference information, the target image reference information corresponding to the current image block is determined, and the target image reference information is stored in the information buffer (Decoded Feature & Picture Buffer). Alternatively, the encoding end can also process the target image reference information (such as feature extraction) and store the processed features in the information buffer. There are no restrictions on this.

[0515] In one possible implementation, the above-mentioned processing at the encoding end can be executed by a deep learning model or a neural network model to realize the end-to-end image compression and encoding process. There are no restrictions on this encoding process.

[0516] In this embodiment of the application, a decoding method is proposed. Regarding the processing at the decoding end, please refer to Figure 10B. Of course, Figure 10B is only an example of the processing at the decoding end, and there are no restrictions on this processing at the decoding end.

[0517] 1. The decoding end obtains the prediction side information bitstream (also called the motion side information bitstream).

[0518] 2. The Hyper Dec of the decoding end parses the prediction side information bitstream to obtain the prediction prior information. The decoder inputs the prediction prior information and temporal prediction prior information to the Motion Entropy Model to obtain the probability distribution of the prediction information. For example, the Motion Latent Buffer stores the temporal prediction prior information

[0519] 3. The decoder obtains the prediction information bitstream (the prediction information bitstream can also be called the motion information bitstream).

[0520] 4. The decoder parses the prediction information bitstream based on the probability distribution of the prediction information (i.e., AD operation) to obtain the prediction information. The prediction information can also be dequantized to obtain the prediction latent feature mt. The decoder stores the prediction information in the Motion Latent Buffer for the temporal prediction prior information of subsequent image blocks

[0521] 5. The decoder inputs the prediction information to the prediction decoder (Motion Decoder), which then uses the prediction information to obtain the prediction information.Decoding the prediction information yields prediction mode features (also known as optical flow features) and prior prediction features (also known as prior motion features). For example, the prior prediction features are stored in an information buffer.

[0522] 6. The decoding end inputs the prediction mode features and historical temporal reference features into the temporal residual mining module. The temporal residual mining module performs temporal residual mining based on the prediction mode features and historical temporal reference features to obtain prediction features.

[0523] For example, the temporal residual mining module can obtain one prediction feature and output the prediction feature, or the temporal residual mining module can obtain two prediction features and output the prediction feature sum.

[0524] 7. The decoding end obtains the residual side information bitstream. The Hyper Dec (super prior decoder) of the decoding end parses the residual side information bitstream to obtain residual super prior information (Hyper Prior). The decoder inputs the predicted features (such as the predicted features, see page 47 / 54 of the specification, CN 121603654 A) to the Spatial Prior Encoder to obtain the Spatial Prior information. The Contextual Latent Buffer stores the residual prior information (i.e., residual information).

[0525] 8. The decoder inputs the residual prior information, spatial prior information, and residual prior information to the Contextual Entropy Model to obtain the probability distribution of the residual latent features. This probability distribution is used to guide the decoding.

[0526] 9. The decoder obtains the residual information bitstream. Based on the probability distribution of the residual latent features, the residual information bitstream is parsed (AD operation) to obtain the residual latent features. The residual latent features can also be dequantized to obtain the residual information feature yt. The residual latent features are stored in the Contextual Latent Buffer for residual prior information of subsequent image blocks

[0527] 10. The decoder inputs the residual latent features and predicted features (e.g.) to the residual decoder, which decodes the residual information based on the residual latent features and predicted features to obtain the target residual features and residual confidence α.

[0528] 11. The decoder performs a weighted operation on the predicted features (e.g., predicted features) and residual confidence α, and performs an addition operation on the weighted features and the target residual features, and inputs the added features to the frame generator, which obtains the initial image reference information

[0529] 12. The decoder generates the reconstructed image block corresponding to the current image block xt

[0530] 13. The decoder obtains the reference confidence α′ corresponding to the initial image reference information. Based on the reference confidence α′ andThe initial image reference information determines the target image reference information corresponding to the current image block, and stores the target image reference information in the information buffer (Decoded Feature & Picture Buffer). Alternatively, the encoding end can also process the target image reference information (such as feature extraction) and store the processed features in the information buffer. There are no restrictions on this.

[0531] In one possible implementation, the above-mentioned decoding end processing can be executed by a deep learning model or a neural network model to realize the end-to-end image compression and decoding process. There are no restrictions on this decoding process.

[0532] Example 11: In Example 10, the reference confidence α′ corresponding to the initial image reference information can be obtained, and the target image reference information can be determined based on the reference confidence α′ and the initial image reference information. The following describes this process.

[0533] Referring to Figure 11A, it is a schematic diagram of the residual confidence α output by the residual decoder. The residual latent features can be input to the residual decoder, and the residual decoder outputs the target residual features and the residual confidence α based on the residual latent features. For example, after inputting the latent residual features into the residual decoder, the residual decoder outputs the target residual features and the first reference features. For example, the residual decoder includes multiple network layers. When the residual decoder processes based on the latent residual features, the feature output by the last network layer of the residual decoder can be used as the target residual feature. Of course, the above is just an example, and the output features of other network layers (such as the second to last network layer) can also be used as the target residual feature. The feature output by the second to last network layer of the residual decoder can be used as the first reference feature. Of course, the above is just an example, and the output features of other network layers can also be used as the first reference feature.

[0534] After obtaining the first reference feature, a transformation operation (such as an operation) can be performed on the first reference feature to obtain the residual confidence α. For example, the transformation operation can be an activation operation (such as using an activation function to activate the first reference feature), or it can be other types of operations. There are no restrictions on this transformation operation.

[0535] After obtaining the residual confidence α, a weighted operation is performed on the predicted features (such as the predicted features) and the residual confidence α, and an addition operation is performed on the weighted features and the target residual features. The added features (such as feature f) are then input to the frame generator, which outputs the initial image reference information.

[0536] Based on this, this embodiment considers various ways to introduce the confidence into other parts. For example, the confidence can be introduced by convolution. See Figure 11B, which shows the residual decoder.A schematic diagram of the output reference confidence α′ of the Decoder. The residual latent features can be input to the residual decoder, and the residual decoder outputs the target residual features and the first reference features. After obtaining the first reference features, a transformation operation (such as operation) can be performed on the first reference features to obtain the residual confidence α. This transformation operation can be an activation operation or other types of operation, and there is no restriction on this transformation operation. After obtaining the residual confidence α, the predicted features (such as the predicted features) and the residual confidence α are weighted, and the weighted features and the target residual features are added. The added features (such as feature f) are input to the frame generator, and the frame generator outputs the initial image reference information

[0537] . After obtaining the first reference features, a convolution operation (such as a Cov operation, which can be performed using at least one convolutional layer) is performed on the first reference features to obtain the convolutional features. The convolutional features are then transformed (such as operation) to obtain the reference confidence α′. For example, the transformation operation can be an activation operation (such as activating the features using an activation function) or other types of operations, and there are no restrictions on this transformation operation.

[0538] Then, a weighted operation (such as a multiplication operation) is performed based on the reference confidence α′ and the initial image reference information to obtain the processed features. Then, an addition operation is performed based on the processed features and the initial image reference information to obtain the target image reference information. For example, when performing the weighted operation, the initial image reference information can be a three-dimensional feature matrix of C*H*W, and the reference confidence α′ can include C confidence values, or H*W confidence values, or C*H*W confidence values.

[0539] For example, the confidence can be introduced by using the ResBlock method (i.e., processing the input features through a residual block network). See Figure 11C, which is a schematic diagram of the reference confidence α′ output by the residual decoder.

[0540] For example, the latent residual features can be input to the residual decoder, which outputs the target residual features and the first reference features. After obtaining the first reference features, a transformation operation (such as an operation) can be performed on the first reference features to obtain the residual confidence α. This transformation operation can be an activation operation or other types of operations, and there are no restrictions on this transformation operation. After obtaining the residual confidence α, a weighted operation is performed on the predicted features (such as the predicted features) and the residual confidence α, and an addition operation is performed on the weighted features and the target residual features. The added features (such as feature f) are then input to the frame generator, which outputs the initial image reference information.

[0541] After obtaining the first reference feature, a residual block operation (such as a ResBlock operation, which can be performed using at least one residual block network) is performed on the first reference feature to obtain the residual block operation feature. A transformation operation (such as an operation) is then performed on the residual block operation feature to obtain the reference confidence α′. For example, the transformation operation can be an activation operation (such as activating the feature using an activation function), or other types of operations. There are no restrictions on this transformation operation.

[0542] Then, a weighted operation (such as a multiplication operation) is performed based on the reference confidence α′ and the initial image reference information to obtain the operation feature. Then, an addition operation is performed based on the operation feature and the initial image reference information to obtain the target image reference information. For example, when performing the weighted operation, the initial image reference information can be a three-dimensional feature matrix of C*H*W, and the reference confidence α′ can include C confidence values, or H*W confidence values, or C*H*W confidence values.

[0543] Example 12: Since the structure of the encoding framework (decoding framework) differs at different code rate points in one application, this example indicates that the framework at some code rate points can be implemented in a pre-implementation manner, without needing to apply it to the framework at every code rate point, that is, it is not necessary to implement the framework in a pre-implementation manner for all code rate points. Specification 49 / 54 pages 55 CN 121603654 A

[0544] For the encoding end, multiple encoding frameworks can be pre-configured. The multiple encoding frameworks can be divided into a first type of encoding framework and a second type of encoding framework. The first type of encoding framework can be any of the encoding frameworks in Examples 3-11, as shown in Figures 3A, 4A, 5A, 7A, 8A, and 10A. The second type of encoding framework is the remaining encoding frameworks other than the first type of encoding framework.

[0545] If the activation conditions of the first type of encoding framework are met, the encoding end can use the first type of encoding framework to implement the encoding method, that is, use the above-described embodiment to implement the encoding method. Alternatively, if the activation conditions of the second type of coding framework are met, the encoding end can use the second type of coding framework to implement the encoding method. In this embodiment, this encoding method is not restricted.

[0546] For example, if the current bitrate point is the bitrate point corresponding to the first type of coding framework, the activation conditions of the first type of coding framework are met. If the current bitrate point is not the bitrate point corresponding to the first type of coding framework, the activation conditions of the first type of coding framework are not met. Of course, the above are just examples of activation conditions. In this embodiment, this activation condition is not restricted.

[0547] For the decoding end, multiple decoding frameworks can be pre-configured. The multiple decoding frameworks can be divided into the first type of decoding framework and the second type of decoding framework. The first type of decoding framework can be any of the decoding frameworks in Embodiments 3 to 11, as shown in Figures 3B, 4B, 5B, 7B, 8B, and 10B. The second type of decoding framework is any other decoding framework besides the first type of decoding framework.

[0548] If the activation conditions of the first type of decoding framework are met, the decoding end can use the first type of decoding framework to implement the decoding method, that is, to implement the decoding method using the above embodiment. Alternatively, if the activation conditions of the second type of decoding framework are met, the decoding end can use the second type of decoding framework to implement the decoding method. This embodiment does not limit this decoding method.

[0549] For example, if the current bitrate point is the bitrate point corresponding to the first type of decoding framework, then the activation conditions of the first type of decoding framework are met; if the current bitrate point is not the bitrate point corresponding to the first type of decoding framework, then the activation conditions of the first type of decoding framework are not met. For example, the bitstream carries a flag bit, which indicates whether the activation conditions of the first type of decoding framework are met. For example, if the flag bit is a first value, it indicates that the activation conditions of the first type of decoding framework are met; if the flag bit is a second value, it indicates that the activation conditions of the first type of decoding framework are not met. Of course, the above are just examples of activation conditions and are not limited thereto.

[0550] For example, the above embodiments can be implemented individually or in combination. For instance, each embodiment in Embodiments 1-12 can be implemented individually, and at least two embodiments in Embodiments 1-12 can be implemented in combination. For each embodiment, the content of that embodiment can also be applied to other embodiments (any embodiment).

[0551] For example, in the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.

[0552] Based on the same application concept as the above method, this application also proposes a decoding device, which is applied to the decoding end. The device may include: a memory configured to store video data; and a decoder configured to implement the decoding method in Embodiments 1-12 above, that is, the processing flow of the decoding end.

[0553] Based on the same application concept as the above method, this application also proposes an encoding device, which is applied to the encoding end. The device may include: a memory configured to store video data; and an encoder configured to implement the encoding methods in embodiments 1-12 above, i.e., the processing flow of the encoding end.

[0554] Based on the same application concept as the above method, the decoding end device (also called a video decoder) provided in this application embodiment, from a hardware perspective, its hardware architecture schematic diagram can be specifically seen in Figure 12A. It includes: a processor 1211 and a machine-readable storage medium 1212. The machine-readable storage medium 1212 stores machine-executable instructions that can be executed by the processor 1211; the processor 1211 is used to execute the machine-executable instructions to implement the decoding methods in embodiments 1-12 above. Specification 50 / 54 pages 56 CN 121603654 A

[0555] Based on the same application concept as the above method, the encoding end device (also called a video encoder) provided in this application embodiment, from a hardware perspective, its hardware architecture schematic diagram can be specifically referred to as Figure 12B. It includes: a processor 1221 and a machine-readable storage medium 1222, the machine-readable storage medium 1222 storing machine-executable instructions that can be executed by the processor 1221; the processor 1221 is used to execute the machine-executable instructions to implement the encoding methods of embodiments 1-12 of this application.

[0556] Based on the same application concept as the above method, this application embodiment provides an electronic device. It includes: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the decoding method or encoding method of embodiments 1-12 of this application.

[0557] Based on the same application concept as the above method, this application embodiment also provides a machine-readable storage medium, the machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the methods disclosed in the above examples of this application, such as the decoding method or encoding method in the above embodiments.

[0558] Based on the same application concept as the above method, this application embodiment also provides a computer application, which, when executed by a processor, can implement the decoding method or encoding method disclosed in the above examples of this application.

[0559] Based on the same application concept as the above method, this application embodiment also proposes a decoding device that can be applied to a decoding end (also called a video decoder). The decoding device may include: a prediction module, used to parse the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block; to perform prediction information decoding on the prediction information to obtain the prediction mode feature and prediction confidence of the current image block; and to determine the target prediction feature corresponding to the current image block based on the prediction mode feature and the prediction confidence; wherein the prediction confidence is obtained based on the information in the prediction information bitstream, and the prediction confidence represents the accuracy of the target prediction feature; and a residual decoding module, used to parse the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; to perform residual decoding on the residual information to obtain the target residual feature corresponding to the current image block; and to determine the reconstructed image block corresponding to the current image block based on the target residual feature and the target prediction feature.

[0560] For example, when the prediction module determines the target prediction feature corresponding to the current image patch based on the prediction pattern feature and the prediction confidence, it specifically performs temporal information mining based on the prediction pattern feature, the prediction confidence, and historical temporal reference features to obtain the initial prediction feature corresponding to the current image patch; wherein, the information buffer...The overlay region stores image reference information that the current image block can refer to, and the historical temporal reference features are derived through the image reference information; the target prediction features corresponding to the current image block are determined based on the initial prediction features.

[0561] For example, when the prediction module performs temporal information mining based on the prediction mode features, the prediction confidence and the historical temporal reference features to obtain the initial prediction features corresponding to the current image block, it is specifically used to: perform a compensation operation on the historical temporal reference features based on the prediction mode features to obtain temporal compensation features; and determine the initial prediction features based on the temporal compensation features and the prediction confidence.

[0562] For example, the prediction pattern feature includes a first prediction pattern feature and a second prediction pattern feature, the historical time domain reference feature includes a first historical time domain reference feature and a second historical time domain reference feature, the prediction confidence includes a first prediction confidence and a second prediction confidence, and the initial prediction feature includes a first initial prediction feature and a second initial prediction feature; when the prediction module performs time domain information mining based on the prediction pattern feature, the prediction confidence and the historical time domain reference feature to obtain the initial prediction feature corresponding to the current image block, it is specifically used for: performing a compensation operation on the first historical time domain reference feature based on the first prediction pattern feature to obtain a first time domain compensation feature; determining the first initial prediction feature based on the first time domain compensation feature and the first prediction confidence; performing a compensation operation on the second historical time domain reference feature based on the second prediction pattern feature as described on pages 51 / 54 of the specification (CN 121603654 A) to obtain a second time domain compensation feature; and determining the second initial prediction feature based on the second time domain compensation feature and the second prediction confidence.

[0563] For example, when the prediction module determines the initial prediction feature based on the temporal compensation feature and the prediction confidence, it is specifically used to: perform a weighted operation based on the temporal compensation feature and the prediction confidence to obtain the post-operation feature; perform a first type of operation based on the post-operation feature to obtain the initial prediction feature; or, perform a second type of operation based on the temporal compensation feature to obtain the post-operation feature; and perform a weighted operation based on the post-operation feature and the prediction confidence to obtain the initial prediction feature.

[0564] For example, when the prediction module performs a first type of operation based on the post-operation feature to obtain the initial prediction feature, it is specifically used to: sequentially perform convolution operation and residual block operation on the post-operation feature, and add the feature after the residual block operation to the post-operation feature to obtain the initial prediction feature; when the prediction module performs a second type of operation based on the temporal compensation feature to obtain the post-operation feature, it is specifically used to: sequentially perform convolution operation and residual block operation on the post-operation feature, and add the feature after the residual block operation to the post-operation feature to obtain the initial prediction feature; when the prediction module performs a second type of operation based on the temporal compensation feature to obtain the post-operation feature, it is specifically used to: sequentially perform convolution operation and residual block operation on the post-operation feature, and perform a weighted operation on the feature after the temporal compensation feature.Perform convolution and residual block operations, and add the features after the residual block operation to the temporal compensation features to obtain the features after the operation.

[0565] For example, when the prediction module determines the target prediction feature corresponding to the current image block based on the initial prediction feature, it is specifically used to: determine the initial prediction feature as the target prediction feature; or, perform a transformation operation on the initial prediction feature to obtain the target prediction feature; or, perform a weighted operation on the initial prediction feature and the residual confidence to obtain the target prediction feature; wherein, the residual confidence is obtained when performing residual decoding on the residual information; wherein, the residual confidence is obtained based on the information in the residual information bitstream, and the residual confidence represents the accuracy of the target prediction feature; or, perform a weighted operation on the initial prediction feature and the residual confidence to obtain the weighted feature, and perform a transformation operation on the weighted feature to obtain the target prediction feature.

[0566] For example, when the prediction module determines the target prediction feature corresponding to the current image block based on the prediction mode feature and the prediction confidence, it is specifically used to: perform temporal information mining based on the prediction mode feature and historical temporal reference feature to obtain the initial prediction feature corresponding to the current image block; wherein, the information buffer stores image reference information that the current image block can refer to, and the historical temporal reference feature is derived through the image reference information; determine the target confidence corresponding to the initial prediction feature based on the prediction confidence; wherein, the target confidence is obtained based on the information in the prediction information bitstream and / or the information in the residual information bitstream, and the target confidence represents the accuracy of the target prediction feature; perform a weighted operation on the initial prediction feature and the target confidence to obtain the target prediction feature; or, perform a weighted operation on the initial prediction feature and the target confidence to obtain the weighted feature, and perform a transformation operation on the weighted feature to obtain the target prediction feature.

[0567] For example, when the prediction module determines the target confidence corresponding to the initial prediction feature based on the prediction confidence, it is specifically used to: determine the prediction confidence as the target confidence; or, obtain the target confidence after performing a transformation operation on the prediction confidence; or, generate the target confidence based on the prediction confidence and the residual confidence.

[0568] For example, when the prediction module parses the prediction information bitstream corresponding to the current image patch to obtain the prediction information corresponding to the current image patch, it is specifically used to: parse the prediction side information bitstream corresponding to the current image patch to obtain prediction prior information; obtain temporal prediction prior information from the prediction latent buffer, wherein the temporal prediction prior information is the prediction information that the current image patch can refer to; and based on the prediction prior information and the temporal prediction prior...The probability distribution of the information generation prediction information is used to parse the prediction information bitstream to obtain the prediction information corresponding to the image block on page 52 / 54 of the current specification, CN 121603654 A. The prediction information is stored in the prediction latent buffer for the temporal prediction prior information of subsequent image blocks.

[0569] For example, when the residual decoding module parses the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block, it specifically performs the following: parses the residual edge information bitstream corresponding to the current image block to obtain residual prior information; obtains residual prior information from the residual latent buffer, wherein the residual prior information is residual information that the current image block can refer to; performs spatial prior processing based on the initial prediction features corresponding to the current image block to obtain spatial prior information; or performs spatial prior processing based on the target prediction features corresponding to the current image block to obtain spatial prior information; generates a probability distribution of residual information based on the residual prior information, the residual prior information, and the spatial prior information, uses the probability distribution to parse the residual information bitstream to obtain the residual information corresponding to the current image block, and stores the residual information in the residual latent buffer for residual prior information of subsequent image blocks.

[0570] Based on the same application concept as the above method, this application embodiment also proposes a decoding device that can be applied to a decoding end (also called a video decoder). The decoding device may include: a prediction module, used to parse the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block; to perform prediction information decoding on the prediction information to obtain the prediction mode feature corresponding to the current image block; to determine the prediction feature corresponding to the current image block based on the prediction mode feature and historical image reference information; a residual decoding module, used to parse the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; to perform residual decoding on the residual information to obtain the target residual feature corresponding to the current image block; to determine the initial image reference information and the reconstructed image block corresponding to the current image block based on the target residual feature and the prediction feature; to obtain the reference confidence level corresponding to the initial image reference information; wherein, the reference confidence level represents the accuracy of the initial image reference information; and to determine the target image reference information corresponding to the current image block based on the reference confidence level and the initial image reference information, wherein the target image reference information is used for the historical image reference information of subsequent image blocks.

[0571] For example, the reference confidence level is obtained when performing residual decoding on the residual information; or, the reference confidence level is obtained when performing prediction information decoding on the prediction information; or, the residual information is obtained by performing residual decoding on the residual information.When encoding, a first reference feature is obtained, and when the prediction information is decoded, a second reference feature is obtained. The reference confidence is obtained based on the first reference feature and the second reference feature.

[0572] Based on the same application concept as the above method, this application embodiment also proposes an encoding device, applied to the encoding end (also called a video encoder). The encoding device includes: a prediction module, used to parse the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block; to decode the prediction information to obtain the prediction mode feature and prediction confidence of the current image block; and to determine the target prediction feature corresponding to the current image block based on the prediction mode feature and the prediction confidence. The prediction confidence is obtained based on the information in the prediction information bitstream, and the prediction confidence represents the accuracy of the target prediction feature. A residual encoding module is used to parse the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; to decode the residual information to obtain the target residual feature corresponding to the current image block; and to determine the reconstructed image block corresponding to the current image block based on the target residual feature and the target prediction feature.

[0573] Based on the same concept as the above method, this application also proposes an encoding device applied at an encoding end (also called a video encoder). The encoding device includes: a prediction module, used to parse the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block; to decode the prediction information to obtain the prediction mode feature corresponding to the current image block; and to perform prediction information decoding on the prediction information to obtain the prediction mode feature corresponding to the current image block; based on the prediction mode feature and historical image reference information, the device performs prediction information decoding on the current image block to obtain the prediction mode feature. The information determines the predicted features corresponding to the current image block; the residual coding module is used to parse the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; the residual information is residual decoded to obtain the target residual features corresponding to the current image block; the initial image reference information and the reconstructed image block corresponding to the current image block are determined based on the target residual features and the predicted features; the reference confidence level corresponding to the initial image reference information is obtained; wherein, the reference confidence level represents the accuracy of the initial image reference information; the target image reference information corresponding to the current image block is determined based on the reference confidence level and the initial image reference information, and the target image reference information is used for the historical image reference information of subsequent image blocks.

[0574] For those skilled in the art, this application can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application. Specification 54 / 54 pages 60 CN121603654 A Figure 1A Figure 1B Figure 2A Instruction Manual Drawings 1 / 12 Page 61 CN 121603654 A Figure 2B Figure 3A Instruction Manual Drawings 2 / 12 Page 62 CN 121603654 A Figure 3B Figure 4A Instruction Manual Drawings 3 / 12 Page 63 CN 121603654 A Figure 4B Figure 5A Instruction Manual Drawings 4 / 12 Page 64 CN 121603654 A Figure 5B Figure 6A Figure 6B Instruction Manual Drawings 5 / 12 Page 65 CN 121603654 A Figure 7A Figure 7B Instruction Manual Drawings 6 / 12 Page 66 CN 121603654 A Figure 8A Figure 8B Figure 9A Instruction Manual Drawings 7 / 12 Page 67 CN 121603654 A Figure 9B Figure 9C Figure 9D Instruction Manual Drawings 8 / 12 Page 68 CN 121603654 Figure 10A Figure 10B Appendix to the Specification 9 / 12 Page 69 CN 121603654 A Figure 11A Figure 11B Figure 11C Appendix to the Specification 10 / 12 Page 70 CN 121603654 A Figure 12A Appendix to the Specification 11 / 12 Page 71 CN 121603654 A Figure 12B Appendix to the Specification 12 / 12 Page 72 CN 121603654 A Abstract The invention provides a decoding and coding method, device and equipment, and the decoding method comprises the steps: carrying out the analysis of a prediction information code stream corresponding to a current image block, and obtaining the prediction information corresponding to the current image block; performing prediction information decoding on the prediction information to obtain a prediction mode feature and a predictionconfidence coefficient corresponding to the current image block; determining a target prediction feature corresponding to the current image block based on the prediction mode feature and the prediction confidence; analyzing the residual information code stream corresponding to the current image block to obtain residual information corresponding to the current image block; performing residual decoding on the residual information to obtain a target residual feature corresponding to the current image block; and determining a reconstructed image block based on the target residual feature and the target prediction feature. Through the scheme of the invention, the decoding performance can be improved.

Claims

1. A decoding method, characterized in that, Applied to the decoding end, the method includes: The prediction information bitstream corresponding to the current image patch is parsed to obtain the prediction information corresponding to the current image patch; the prediction information is decoded to obtain the prediction mode features and prediction confidence corresponding to the current image patch; the target prediction features corresponding to the current image patch are determined based on the prediction mode features and the prediction confidence; wherein, the prediction confidence is obtained based on the information in the prediction information bitstream, and the prediction confidence represents the accuracy of the target prediction features; The residual information bitstream corresponding to the current image block is parsed to obtain the residual information corresponding to the current image block; the residual information is then residual decoded to obtain the target residual feature corresponding to the current image block. The reconstructed image block corresponding to the current image block is determined based on the target residual features and the target prediction features.

2. The method according to claim 1, characterized in that, Determining the target prediction features corresponding to the current image patch based on the prediction pattern features and the prediction confidence includes: Based on the predicted pattern features, the predicted confidence level, and the historical temporal reference features, temporal information mining is performed to obtain the initial predicted features corresponding to the current image patch; wherein, the information buffer stores image reference information that the current image patch can refer to, and the historical temporal reference features are derived through the image reference information; The target prediction features corresponding to the current image patch are determined based on the initial prediction features.

3. The method according to claim 2, characterized in that, The step of mining temporal information based on the predicted pattern features, the predicted confidence level, and historical temporal reference features to obtain the initial predicted features corresponding to the current image patch includes: Based on the predicted pattern features, a compensation operation is performed on the historical time-domain reference features to obtain time-domain compensated features; The initial prediction features are determined based on the time-domain compensation features and the prediction confidence.

4. The method according to claim 2, characterized in that, The prediction pattern features include a first prediction pattern feature and a second prediction pattern feature; the historical time domain reference features include a first historical time domain reference feature and a second historical time domain reference feature; the prediction confidence includes a first prediction confidence and a second prediction confidence; and the initial prediction features include a first initial prediction feature and a second initial prediction feature. The step of mining temporal information based on the predicted pattern features, the predicted confidence level, and historical temporal reference features to obtain the initial predicted features corresponding to the current image patch includes: The first historical time-domain reference features are compensated based on the first prediction pattern features to obtain the first time-domain compensated features; the first initial prediction features are determined based on the first time-domain compensated features and the first prediction confidence. The second historical time-domain reference features are compensated based on the second prediction mode features to obtain the second time-domain compensated features; the second initial prediction features are determined based on the second time-domain compensated features and the second prediction confidence.

5. The method according to claim 3 or 4, characterized in that, The step of determining the initial prediction features based on the time-domain compensation features and the prediction confidence includes: A weighted operation is performed based on the time-domain compensated features and the prediction confidence to obtain the post-operated features; a first type of operation is performed based on the post-operated features to obtain the initial prediction features; or, A second type of operation is performed based on the time-domain compensation features to obtain post-operation features; a weighted operation is performed based on the post-operation features and the prediction confidence to obtain the initial prediction features.

6. The method according to claim 5, characterized in that, The first type of operation based on the processed features to obtain the initial prediction features includes: The features after the operation are sequentially subjected to convolution and residual block operations, and the features after the residual block operation are added to the features after the operation to obtain the initial prediction features; The second type of operation based on the time-domain compensation features, to obtain the post-operation features, includes: The temporal compensated features are sequentially subjected to convolution and residual block operations, and the features after the residual block operation are added to the temporal compensated features to obtain the operated features.

7. The method according to claim 2, characterized in that, Determining the target prediction features corresponding to the current image patch based on the initial prediction features includes: The initial prediction feature is determined as the target prediction feature; or... The target prediction features are obtained by performing a transformation operation on the initial prediction features; or, The target predicted features are obtained by weighting the initial predicted features and the residual confidence scores; wherein the residual confidence scores are obtained by performing residual decoding on the residual information; wherein the residual confidence scores are based on information in the residual information bitstream, and the residual confidence scores represent the accuracy of the target predicted features; or, The initial predicted features and residual confidence scores are weighted to obtain weighted features, and the weighted features are then transformed to obtain the target predicted features.

8. The method according to claim 1, characterized in that, Determining the target prediction features corresponding to the current image patch based on the prediction pattern features and the prediction confidence includes: Based on the predicted pattern features and historical temporal reference features, temporal information mining is performed to obtain the initial predicted features corresponding to the current image patch; wherein, the information buffer stores image reference information that the current image patch can refer to, and the historical temporal reference features are derived through the image reference information; The target confidence level corresponding to the initial predicted feature is determined based on the predicted confidence level; wherein the target confidence level is obtained based on the information in the predicted information bitstream and / or the information in the residual information bitstream, and the target confidence level represents the accuracy of the target predicted feature; The initial predicted features and the target confidence score are weighted to obtain the target predicted features; Alternatively, the initial predicted features and the target confidence level can be weighted to obtain weighted features, and the weighted features can be transformed to obtain the target predicted features.

9. The method according to claim 8, characterized in that, Determining the target confidence level corresponding to the initial predicted feature based on the predicted confidence level includes: The predicted confidence level is determined as the target confidence level; or... The target confidence level is obtained by performing a transformation operation on the predicted confidence level; or, The target confidence level is generated based on the predicted confidence level and the residual confidence level.

10. The method according to any one of claims 1-9, characterized in that, The step of parsing the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block includes: The prediction edge information bitstream corresponding to the current image block is parsed to obtain the prediction prior information; temporal prediction prior information is obtained from the prediction latent buffer, and the temporal prediction prior information is the prediction information that the current image block can refer to; Based on the predicted prior information and the temporal predicted prior information, a probability distribution for generating predicted information is generated. This probability distribution is used to parse the predicted information bitstream to obtain the predicted information corresponding to the current image block. The predicted information is then stored in the predicted latent buffer for temporal predicted prior information of subsequent image blocks.

11. The method according to any one of claims 1-9, characterized in that, The step of parsing the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block includes: The residual edge information bitstream corresponding to the current image block is parsed to obtain residual prior information; residual prior information is obtained from the residual latent buffer, and the residual prior information is residual information that can be referenced by the current image block; Spatial prior information is obtained by performing spatial prior processing based on the initial prediction features corresponding to the current image patch; or, spatial prior information is obtained by performing spatial prior processing based on the target prediction features corresponding to the current image patch. Based on the residual prior information, the residual prior information, and the spatial prior information, a probability distribution of residual information is generated. This probability distribution is used to parse the residual information bitstream to obtain the residual information corresponding to the current image block. The residual information is then stored in the residual latent buffer for use as residual prior information for subsequent image blocks.

12. An encoding method, characterized in that, Applied to the encoding end, the method includes: The prediction information bitstream corresponding to the current image patch is parsed to obtain the prediction information corresponding to the current image patch; the prediction information is decoded to obtain the prediction mode features and prediction confidence corresponding to the current image patch; the target prediction features corresponding to the current image patch are determined based on the prediction mode features and the prediction confidence; wherein, the prediction confidence is obtained based on the information in the prediction information bitstream, and the prediction confidence represents the accuracy of the target prediction features; The residual information bitstream corresponding to the current image block is parsed to obtain the residual information corresponding to the current image block; the residual information is then residual decoded to obtain the target residual feature corresponding to the current image block. The reconstructed image block corresponding to the current image block is determined based on the target residual features and the target prediction features.

13. A decoding method, characterized in that, Applied to the decoding end, the method includes: The prediction information bitstream corresponding to the current image block is parsed to obtain the prediction information corresponding to the current image block; the prediction information is decoded to obtain the prediction mode feature corresponding to the current image block; the prediction feature corresponding to the current image block is determined based on the prediction mode feature and historical image reference information. The residual information bitstream corresponding to the current image block is parsed to obtain the residual information corresponding to the current image block; the residual information is then residual decoded to obtain the target residual feature corresponding to the current image block. Based on the target residual features and the prediction features, determine the initial image reference information and the reconstructed image block corresponding to the current image block; Obtain the reference confidence level corresponding to the initial image reference information; wherein, the reference confidence level represents the accuracy of the initial image reference information; Based on the reference confidence level and the initial image reference information, the target image reference information corresponding to the current image block is determined, and the target image reference information is used as the historical image reference information for subsequent image blocks.

14. The method according to claim 13, characterized in that, The reference confidence level is obtained by performing residual decoding on the residual information; or... The reference confidence level is obtained when decoding the prediction information; or... When performing residual decoding on the residual information, a first reference feature is obtained; when performing prediction information decoding on the prediction information, a second reference feature is obtained; and the reference confidence is obtained based on the first reference feature and the second reference feature.

15. An encoding method, characterized in that, Applied to the encoding end, the method includes: The prediction information bitstream corresponding to the current image block is parsed to obtain the prediction information corresponding to the current image block; the prediction information is decoded to obtain the prediction mode feature corresponding to the current image block; the prediction feature corresponding to the current image block is determined based on the prediction mode feature and historical image reference information. The residual information bitstream corresponding to the current image block is parsed to obtain the residual information corresponding to the current image block; the residual information is then residual decoded to obtain the target residual feature corresponding to the current image block. Based on the target residual features and the prediction features, determine the initial image reference information and the reconstructed image block corresponding to the current image block; Obtain the reference confidence level corresponding to the initial image reference information; wherein, the reference confidence level represents the accuracy of the initial image reference information; Based on the reference confidence level and the initial image reference information, the target image reference information corresponding to the current image block is determined, and the target image reference information is used as the historical image reference information for subsequent image blocks.

16. A decoding device, characterized in that, The device, applied at the decoding end, includes: The prediction module is used to parse the prediction information bitstream corresponding to the current image patch to obtain the prediction information corresponding to the current image patch; decode the prediction information to obtain the prediction mode features and prediction confidence corresponding to the current image patch; and determine the target prediction features corresponding to the current image patch based on the prediction mode features and the prediction confidence; wherein, the prediction confidence is obtained based on the information in the prediction information bitstream, and the prediction confidence represents the accuracy of the target prediction features; The residual decoding module is used to parse the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; perform residual decoding on the residual information to obtain the target residual feature corresponding to the current image block; and determine the reconstructed image block corresponding to the current image block based on the target residual feature and the target prediction feature.

17. An encoding device, characterized in that, Applied to the encoding end, the device includes: The prediction module is used to parse the prediction information bitstream corresponding to the current image patch to obtain the prediction information corresponding to the current image patch; decode the prediction information to obtain the prediction mode features and prediction confidence corresponding to the current image patch; and determine the target prediction features corresponding to the current image patch based on the prediction mode features and the prediction confidence; wherein, the prediction confidence is obtained based on the information in the prediction information bitstream, and the prediction confidence represents the accuracy of the target prediction features; The residual coding module is used to parse the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; perform residual decoding on the residual information to obtain the target residual feature corresponding to the current image block; and determine the reconstructed image block corresponding to the current image block based on the target residual feature and the target prediction feature.

18. A decoding device, characterized in that, The device, applied at the decoding end, includes: The prediction module is used to parse the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block; decode the prediction information to obtain the prediction mode features corresponding to the current image block; and determine the prediction features corresponding to the current image block based on the prediction mode features and historical image reference information. The residual decoding module is used to parse the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; perform residual decoding on the residual information to obtain the target residual feature corresponding to the current image block; determine the initial image reference information and reconstructed image block corresponding to the current image block based on the target residual feature and the predicted feature; obtain the reference confidence level corresponding to the initial image reference information; wherein, the reference confidence level represents the accuracy of the initial image reference information; determine the target image reference information corresponding to the current image block based on the reference confidence level and the initial image reference information, and the target image reference information is used as the historical image reference information for subsequent image blocks.

19. An encoding device, characterized in that, Applied to the encoding end, the device includes: The prediction module is used to parse the prediction information bitstream corresponding to the current image block to obtain the prediction information corresponding to the current image block; decode the prediction information to obtain the prediction mode features corresponding to the current image block; and determine the prediction features corresponding to the current image block based on the prediction mode features and historical image reference information. The residual coding module is used to parse the residual information bitstream corresponding to the current image block to obtain the residual information corresponding to the current image block; perform residual decoding on the residual information to obtain the target residual feature corresponding to the current image block; determine the initial image reference information and reconstructed image block corresponding to the current image block based on the target residual feature and the predicted feature; obtain the reference confidence level corresponding to the initial image reference information; wherein, the reference confidence level represents the accuracy of the initial image reference information; determine the target image reference information corresponding to the current image block based on the reference confidence level and the initial image reference information, and the target image reference information is used as the historical image reference information for subsequent image blocks.

20. A decoding device, characterized in that, The decoding device includes a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor. The processor is configured to execute machine-executable instructions to implement the method according to any one of claims 1-11; or, the processor is configured to execute machine-executable instructions to implement the method according to any one of claims 13-14.

21. An encoding terminal device, characterized in that, The encoding end device includes: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method of claim 12 or 15.

22. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores a plurality of computer instructions, which, when executed by a processor, implement the method described in any one of claims 1-11 and 13-14, or, when executed by a processor, implement the method described in claim 12 or 15.

Images