Decoding and encoding method, apparatus and device

Abstract

The present invention provides a decoding and encoding method, apparatus, and device thereof, the decoding method including the steps of: obtaining a first feature corresponding to a current image block by a first neural network, and obtaining a feature adjustment factor corresponding to the current image block, the first neural network including at least one convolution layer; determining a target feature based on the first feature and the feature adjustment factor; and obtaining a reconstructed image block corresponding to the current image block by a second neural network based on the target feature, the second neural network including at least one convolution layer. The technical solution of the present invention improves the encoding and decoding performance.

Description

[Technical Field]

[0001] The present invention relates to the field of encoding and decoding technology, and in particular to a decoding and encoding method, apparatus and device thereof. [Background technology]

[0002] To save space, all video images are coded before transmission, and complete video coding may include processes such as prediction, transformation, quantization, entropy coding, filtering, etc. Regarding the prediction process, the prediction process may include intra-frame prediction and inter-frame prediction, where inter-frame prediction utilizes the temporal correlation of video to predict a current pixel using pixels of adjacent coded images, thereby achieving the purpose of effectively removing temporal redundancy in video. Intra-frame prediction utilizes the spatial correlation of video to predict a current pixel using pixels of coded blocks of images of the current frame, thereby achieving the purpose of removing spatial redundancy in video.

[0003] With the rapid development of deep learning, deep learning has achieved results in many high-level computer vision problems, such as image classification and target detection, and deep learning has also gradually begun to be applied in the field of encoding and decoding, that is, it has become possible to encode and decode images using neural networks. Although encoding and decoding methods based on neural networks have shown great potential, they still suffer from problems such as low encoding performance, low decoding performance, and high complexity. Summary of the Invention [Means for solving the problem]

[0004] In view of this, the present invention provides a decoding and encoding method, apparatus, and device therefor that improve the encoding and decoding performance.

[0005] In the first aspect, The present invention 、 obtaining a first feature corresponding to the current image block by a first neural network, and obtaining a feature adjustment factor corresponding to the current image block; performing feature enhancement on the first feature to obtain a second feature; performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature; determining a target feature based on the third feature; and obtaining a reconstructed image block corresponding to the current image block by a second neural network based on the target feature. A decoding method is provided.

[0006] In some embodiments, the step of performing feature enhancement on the first feature to obtain the second feature comprises: determining initial features corresponding to an attention sub-network based on the first features; performing feature enhancement on the initial features using a residual enhancement sub-network to obtain enhanced features; generating weight features corresponding to the initial features using a weight generation sub-network; generating the second feature based on the enhanced feature and the weight feature.

[0007] In some embodiments, determining the target feature based on the third feature comprises: The method includes steps of determining an initial feature corresponding to an attention sub-network based on the first feature, generating a fourth feature based on the initial feature and the third feature, and determining a target feature based on the fourth feature.

[0008] In some embodiments, determining initial features corresponding to an attention sub-network based on the first features comprises: performing feature enhancement on the first features using an initial enhancement sub-network to obtain enhanced features; and performing upsampling convolution on the enhanced features using an upsampling convolution sub-network to obtain initial features corresponding to the attention sub-network.

[0009] In some embodiments, the weight generation sub-network includes a residual block sub-network, a convolutional sub-network, and a feature mapping sub-network, and generating weight features corresponding to the initial features using the weight generation sub-network includes: performing convolutional activation on the initial features using the residual block sub-network to obtain post-convolutional activation features; performing convolution on the convolutional activation features using the convolutional sub-network to obtain convolutional features; and performing feature mapping on the convolved features using the feature mapping sub-network to obtain the weight features.

[0010] In some embodiments, performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature comprises: where the second feature includes C*H*W feature values, where C represents the number of channels, H represents the feature height, and W represents the feature width; determining feature adjustment values ​​corresponding to the C*H*W feature values ​​according to the feature adjustment factor; and performing feature adjustment on the C*H*W feature values ​​according to the feature adjustment values ​​to obtain adjusted feature values ​​corresponding to the C*H*W feature values; and generating the third feature based on adjusted feature values ​​corresponding to the C*H*W feature values.

[0011] In some embodiments, determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the feature adjustment factors comprises: When the feature adjustment factor includes C*H*W feature adjustment values, determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the C*H*W feature adjustment values; or If the feature adjustment factor includes H*W feature adjustment values, determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the H*W feature adjustment values; or If the feature adjustment factor includes C feature adjustment values, the method includes determining feature adjustment values ​​corresponding to the C*H*W feature values, respectively, based on the C feature adjustment values.

[0012] In some embodiments, the method includes, if a feature adjustment mode is on for the current image block, obtaining a feature adjustment factor corresponding to the current image block, and determining that a feature adjustment mode is on for the current image block comprises: Parsing a feature adjustment flag bit from the bitstream and determining that the feature adjustment mode is on for the current image block if the feature adjustment flag bit allows the feature adjustment mode to be on for the current image block; or The method includes analyzing a first value range of feature values ​​from the bitstream, and determining that a feature adjustment mode is on for the current image block if the feature value of the second feature is in the first value range.

[0013] In some embodiments, the first neural network includes a decoding sub-network.

[0014] In some embodiments, the first neural network includes a decoding sub-network, and the step of obtaining a first feature corresponding to the current image block by the first neural network comprises: obtaining a probability distribution parameter based on a bitstream corresponding to the current image block; determining a probability distribution model based on the probability distribution parameters, and decoding a bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features; and determining a first feature corresponding to the current image block based on the decoded image features, or obtaining probability distribution parameters and predicted values ​​based on a bitstream corresponding to the current image block; determining a probability distribution model based on the probability distribution parameters, and decoding a bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features; performing residual reconstruction on the decoded image features to obtain residual features; determining a first feature corresponding to the current image block based on the residual feature and the predicted value.

[0015] In some embodiments, the step of obtaining a first feature corresponding to the current image block by the first neural network comprises: obtaining a probability distribution parameter based on a first bitstream corresponding to the current image block; determining a probability distribution model based on the probability distribution parameters, and decoding a second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features; determining a first feature corresponding to the current image block based on the decoded image feature; The method further includes performing a synthesis transform on the first feature by a synthesis transform network to obtain a reconstructed image block corresponding to the current image block; In a synthesis transformation process, feature enhancement is performed on the first feature to obtain a second feature, feature adjustment is performed on the second feature based on the feature adjustment factor to obtain a third feature, the target feature is determined based on the third feature, and a reconstructed image block corresponding to the current image block is determined based on the target feature.

[0016] In some embodiments, the method further comprises obtaining a prediction value based on a bitstream corresponding to the current image block; determining a first feature corresponding to the current image block based on the decoded image feature, performing residual reconstruction on the decoded image features to obtain residual features; determining a first feature corresponding to the current image block based on the residual feature and the predicted value.

[0017] In some embodiments, the step of obtaining a feature adjustment factor corresponding to the current image block comprises: The step includes decoding a bitstream corresponding to the current image block to obtain a feature adjustment factor corresponding to the current image block, or determining a fixed parameter value as the feature adjustment factor corresponding to the current image block.

[0018] In some embodiments, the second neural network includes a reconstruction decoding sub-network, and the step of obtaining a reconstructed image block corresponding to the current image block by the second neural network based on the target feature includes: processing the target features by a reconstruction decoding sub-network to obtain the reconstructed image block; The reconstruction decoding subnetwork includes an upsampling convolution subnetwork, which performs upsampling convolution on the target features to obtain the reconstructed image block, or the reconstruction decoding subnetwork includes an upsampling convolution subnetwork and a color space conversion subnetwork, which performs upsampling convolution on the target features to obtain upsampling convolved features by the upsampling convolution subnetwork, and performs color space conversion on the upsampling convolved features by the color space conversion subnetwork to obtain the reconstructed image block.

[0019] Correspondingly, the present invention provides: obtaining a first feature corresponding to the current image block by a first neural network; obtaining a feature adjustment factor corresponding to the current image block based on the first feature, and encoding the feature adjustment factor into a bitstream corresponding to the current image block; performing feature enhancement on the first feature to obtain a second feature; performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature; determining a target feature based on the third feature; and obtaining a reconstructed image block corresponding to the current image block by a second neural network based on the target feature.

[0020] In a second aspect, the present invention provides a method for producing a method of manufacturing a semiconductor device comprising: obtaining a first feature corresponding to the current image block by a first neural network, and obtaining a feature adjustment factor corresponding to the current image block; determining initial features corresponding to an attention sub-network based on the first features; performing feature enhancement on the initial features using a residual enhancement sub-network to obtain enhanced features; generating weight features corresponding to the initial features using a weight generation sub-network; and generating second features based on the enhanced features and the weight features; performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature; determining a target feature based on the third feature; and obtaining a reconstructed image block corresponding to the current image block by a second neural network based on the target feature.

[0021] In some embodiments, determining initial features corresponding to an attention sub-network based on the first features comprises: performing feature enhancement on the first features using an initial enhancement sub-network to obtain enhanced features; and performing upsampling convolution on the enhanced features using an upsampling convolution sub-network to obtain initial features corresponding to the attention sub-network.

[0022] In some embodiments, the weight generation sub-network includes a residual block sub-network, a convolutional sub-network, and a feature mapping sub-network, and generating weight features corresponding to the initial features using the weight generation sub-network includes: performing convolutional activation on the initial features using the residual block sub-network to obtain post-convolutional activation features; performing convolution on the convolutional activation features using the convolutional sub-network to obtain convolutional features; and performing feature mapping on the convolved features using the feature mapping sub-network to obtain the weight features.

[0023] In some embodiments, the first neural network includes a decoding sub-network, and the step of obtaining a first feature corresponding to the current image block by the first neural network comprises: obtaining a probability distribution parameter based on a first bitstream corresponding to the current image block; determining a probability distribution model based on the probability distribution parameters, and decoding a second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features; and determining a first feature corresponding to the current image block based on the decoded image features, or obtaining a probability distribution parameter and a predicted value according to a first bitstream corresponding to the current image block; determining a probability distribution model based on the probability distribution parameters, and decoding a second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features; performing residual reconstruction on the decoded image features to obtain residual features; determining a first feature corresponding to the current image block based on the residual feature and the predicted value.

[0024] In some embodiments, the step of obtaining the feature adjustment factor corresponding to the current image block includes the step of decoding a bitstream corresponding to the current image block to obtain the feature adjustment factor corresponding to the current image block, or determining a fixed parameter value as the feature adjustment factor corresponding to the current image block.

[0025] In some embodiments, the second neural network includes a reconstruction decoding sub-network, and the step of obtaining a reconstructed image block corresponding to the current image block by the second neural network based on the target feature includes: processing the target features by a reconstruction decoding sub-network to obtain the reconstructed image block; The reconstruction decoding subnetwork includes an upsampling convolution subnetwork, which performs upsampling convolution on the target features to obtain the reconstructed image block, or the reconstruction decoding subnetwork includes an upsampling convolution subnetwork and a color space conversion subnetwork, which performs upsampling convolution on the target features using the upsampling convolution subnetwork and performs color space conversion on the features after the upsampling convolution using the color space conversion subnetwork to obtain the reconstructed image block.

[0026] In some embodiments, the method further comprises, if a feature adjustment mode is on for the current image block, obtaining a feature adjustment factor corresponding to the current image block, and determining that a feature adjustment mode is on for the current image block comprises: Parsing a feature adjustment flag bit from the bitstream and determining that the feature adjustment mode is on for the current image block if the feature adjustment flag bit allows the feature adjustment mode to be on for the current image block; or The method includes analyzing a first value range of feature values ​​from the bitstream, and determining that a feature adjustment mode is on for the current image block if the feature value of the second feature is in the first value range.

[0027] Correspondingly, the present invention provides: obtaining a first feature corresponding to the current image block by a first neural network; obtaining a feature adjustment factor corresponding to the current image block based on the first feature, and encoding the feature adjustment factor into a bitstream corresponding to the current image block; determining initial features corresponding to an attention sub-network based on the first features; performing feature enhancement on the initial features using a residual enhancement sub-network to obtain enhanced features; generating weight features corresponding to the initial features using a weight generation sub-network; and generating second features based on the enhanced features and the weight features; performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature; determining a target feature based on the third feature; and obtaining a reconstructed image block corresponding to the current image block by a second neural network based on the target feature.

[0028] In a third aspect, the present invention provides a method for producing a method of manufacturing a semiconductor device comprising: obtaining a first feature corresponding to the current image block by a first neural network, and obtaining a feature adjustment factor corresponding to the current image block; performing feature enhancement on the first feature to obtain a second feature; performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature, where if the second feature includes C*H*W feature values, C representing a channel number, H representing a feature height, and W representing a feature width, determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the feature adjustment factor, performing feature adjustment on the C*H*W feature values ​​based on the feature adjustment values ​​to obtain adjusted feature values ​​corresponding to the C*H*W feature values, and generating the third feature based on the adjusted feature values ​​corresponding to the C*H*W feature values; determining a target feature based on the third feature; and obtaining a reconstructed image block corresponding to the current image block by a second neural network based on the target feature.

[0029] In some embodiments, determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the feature adjustment factors comprises: When the feature adjustment factor includes C*H*W feature adjustment values, determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the C*H*W feature adjustment values; or If the feature adjustment factor includes H*W feature adjustment values, determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the H*W feature adjustment values; or If the feature adjustment factor includes C feature adjustment values, the method includes determining feature adjustment values ​​corresponding to the C*H*W feature values, respectively, based on the C feature adjustment values.

[0030] In some embodiments, the first neural network includes a decoding sub-network, and the step of obtaining a first feature corresponding to the current image block by the first neural network comprises: obtaining a probability distribution parameter based on a first bitstream corresponding to the current image block; determining a probability distribution model based on the probability distribution parameters, and decoding a second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features; and determining a first feature corresponding to the current image block based on the decoded image features, or obtaining a probability distribution parameter and a predicted value according to a first bitstream corresponding to the current image block; determining a probability distribution model based on the probability distribution parameters, and decoding a second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features; performing residual reconstruction on the decoded image features to obtain residual features; determining a first feature corresponding to the current image block based on the residual feature and the predicted value.

[0031] In some embodiments, the step of obtaining the feature adjustment factor corresponding to the current image block includes the step of decoding a bitstream corresponding to the current image block to obtain the feature adjustment factor corresponding to the current image block, or determining a fixed parameter value as the feature adjustment factor corresponding to the current image block.

[0032] In some embodiments, the second neural network includes a reconstruction decoding sub-network, and the step of obtaining a reconstructed image block corresponding to the current image block by the second neural network based on the target feature includes: processing the target features by a reconstruction decoding sub-network to obtain the reconstructed image block; The reconstruction decoding subnetwork includes an upsampling convolution subnetwork, which performs upsampling convolution on the target features to obtain the reconstructed image block, or the reconstruction decoding subnetwork includes an upsampling convolution subnetwork and a color space conversion subnetwork, which performs upsampling convolution on the target features using the upsampling convolution subnetwork and performs color space conversion on the features after the upsampling convolution using the color space conversion subnetwork to obtain the reconstructed image block.

[0033] In some embodiments, the method further comprises, if a feature adjustment mode is on for the current image block, obtaining a feature adjustment factor corresponding to the current image block, and determining that a feature adjustment mode is on for the current image block comprises: Parsing a feature adjustment flag bit from the bitstream and determining that the feature adjustment mode is on for the current image block if the feature adjustment flag bit allows the feature adjustment mode to be on for the current image block; or The method includes analyzing a first value range of feature values ​​from the bitstream, and determining that a feature adjustment mode is on for the current image block if the feature value of the second feature is in the first value range.

[0034] Correspondingly, the present invention provides: obtaining a first feature corresponding to the current image block by a first neural network; obtaining a feature adjustment factor corresponding to the current image block based on the first feature, and encoding the feature adjustment factor into a bitstream corresponding to the current image block; performing feature enhancement on the first feature to obtain a second feature; performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature, where if the second feature includes C*H*W feature values, C representing a channel number, H representing a feature height, and W representing a feature width, determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the feature adjustment factor, performing feature adjustment on the C*H*W feature values ​​based on the feature adjustment values ​​to obtain adjusted feature values ​​corresponding to the C*H*W feature values, and generating the third feature based on the adjusted feature values ​​corresponding to the C*H*W feature values; determining a target feature based on the third feature; and obtaining a reconstructed image block corresponding to the current image block by a second neural network based on the target feature.

[0035] In a fourth aspect, the present invention provides a method for producing a method of manufacturing a semiconductor device comprising: obtaining a first feature corresponding to the current image block by a first neural network, and obtaining a feature adjustment factor corresponding to the current image block; performing feature enhancement on the first feature to obtain a second feature; performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature; determining an initial feature corresponding to an attention sub-network based on the first feature, generating a fourth feature based on the initial feature and the third feature, and determining a target feature based on the fourth feature; and obtaining a reconstructed image block corresponding to the current image block by a second neural network based on the target feature.

[0036] In some embodiments, determining initial features corresponding to an attention sub-network based on the first features comprises: performing feature enhancement on the first features using an initial enhancement sub-network to obtain enhanced features; and performing upsampling convolution on the enhanced features using an upsampling convolution sub-network to obtain initial features corresponding to the attention sub-network.

[0037] In some embodiments, the first neural network includes a decoding sub-network, and the step of obtaining a first feature corresponding to the current image block by the first neural network comprises: obtaining a probability distribution parameter based on a first bitstream corresponding to the current image block; determining a probability distribution model based on the probability distribution parameters, and decoding a second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features; and determining a first feature corresponding to the current image block based on the decoded image features, or obtaining a probability distribution parameter and a predicted value according to a first bitstream corresponding to the current image block; determining a probability distribution model based on the probability distribution parameters, and decoding a second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features; performing residual reconstruction on the decoded image features to obtain residual features; determining a first feature corresponding to the current image block based on the residual feature and the predicted value.

[0038] In some embodiments, the step of obtaining the feature adjustment factor corresponding to the current image block includes the step of decoding a bitstream corresponding to the current image block to obtain the feature adjustment factor corresponding to the current image block, or determining a fixed parameter value as the feature adjustment factor corresponding to the current image block.

[0039] In some embodiments, the second neural network includes a reconstruction decoding sub-network, and the step of obtaining a reconstructed image block corresponding to the current image block by the second neural network based on the target feature includes: processing the target features by a reconstruction decoding sub-network to obtain the reconstructed image block; The reconstruction decoding subnetwork includes an upsampling convolution subnetwork, which performs upsampling convolution on the target features to obtain the reconstructed image block, or the reconstruction decoding subnetwork includes an upsampling convolution subnetwork and a color space conversion subnetwork, which performs upsampling convolution on the target features using the upsampling convolution subnetwork and performs color space conversion on the features after the upsampling convolution using the color space conversion subnetwork to obtain the reconstructed image block.

[0040] In some embodiments, the method further comprises, if a feature adjustment mode is on for the current image block, obtaining a feature adjustment factor corresponding to the current image block, and determining that a feature adjustment mode is on for the current image block comprises: Parsing a feature adjustment flag bit from the bitstream and determining that the feature adjustment mode is on for the current image block if the feature adjustment flag bit allows the feature adjustment mode to be on for the current image block; or The method includes analyzing a first value range of feature values ​​from the bitstream, and determining that a feature adjustment mode is on for the current image block if the feature value of the second feature is in the first value range.

[0041] Correspondingly, the present invention provides: obtaining a first feature corresponding to the current image block by a first neural network; obtaining a feature adjustment factor corresponding to the current image block based on the first feature, and encoding the feature adjustment factor into a bitstream corresponding to the current image block; performing feature enhancement on the first feature to obtain a second feature; performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature; determining an initial feature corresponding to an attention sub-network based on the first feature, generating a fourth feature based on the initial feature and the third feature, and determining a target feature based on the fourth feature; and obtaining a reconstructed image block corresponding to the current image block by a second neural network based on the target feature.

[0042] In a fifth aspect, the present invention provides a method for producing a method of manufacturing a semiconductor device comprising: obtaining a probability distribution parameter according to a bitstream corresponding to a current image block; determining a probability distribution model according to the probability distribution parameter; decoding the bitstream corresponding to the current image block according to the probability distribution model to obtain decoded image features; and determining a first feature corresponding to the current image block according to the decoded image features; obtaining a feature adjustment factor corresponding to the current image block; performing a synthesis transformation on the first feature by a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block; An image decoding method is provided, in which, in a synthesis transformation process, feature enhancement is performed on the first feature to obtain a second feature, feature adjustment is performed on the second feature based on the feature adjustment factor to obtain a third feature, the target feature is determined based on the third feature, and a reconstructed image block corresponding to the current image block is determined based on the target feature.

[0043] In some embodiments, the method further comprises obtaining a prediction value based on a bitstream corresponding to the current image block; determining a first feature corresponding to the current image block based on the decoded image feature, performing residual reconstruction on the decoded image features to obtain residual features; determining a first feature corresponding to the current image block based on the residual feature and the predicted value.

[0044] In some embodiments, the step of obtaining a feature adjustment factor corresponding to the current image block comprises: The step includes decoding a bitstream corresponding to the current image block to obtain a feature adjustment factor corresponding to the current image block, or determining a fixed parameter value as the feature adjustment factor corresponding to the current image block.

[0045] In some embodiments, determining a reconstructed image block corresponding to the current image block based on the target feature comprises: processing the target features by a reconstruction decoding sub-network to obtain the reconstructed image block; The reconstruction decoding subnetwork includes an upsampling convolution subnetwork, which performs upsampling convolution on the target features to obtain the reconstructed image block, or the reconstruction decoding subnetwork includes an upsampling convolution subnetwork and a color space conversion subnetwork, which performs upsampling convolution on the target features using the upsampling convolution subnetwork and performs color space conversion on the features after the upsampling convolution using the color space conversion subnetwork to obtain the reconstructed image block.

[0046] In some embodiments, the method further comprises, if a feature adjustment mode is on for the current image block, obtaining a feature adjustment factor corresponding to the current image block, and determining that a feature adjustment mode is on for the current image block comprises: Parsing a feature adjustment flag bit from the bitstream and determining that the feature adjustment mode is on for the current image block if the feature adjustment flag bit allows the feature adjustment mode to be on for the current image block; or The method includes analyzing a first value range of feature values ​​from the bitstream, and determining that a feature adjustment mode is on for the current image block if the feature value of the second feature is in the first value range.

[0047] Correspondingly, the present invention provides: obtaining a probability distribution parameter according to a bitstream corresponding to a current image block; determining a probability distribution model according to the probability distribution parameter; decoding the bitstream corresponding to the current image block according to the probability distribution model to obtain decoded image features; and determining a first feature corresponding to the current image block according to the decoded image features; obtaining a feature adjustment factor corresponding to the current image block based on the first feature, and encoding the feature adjustment factor into a bitstream corresponding to the current image block; performing a synthesis transformation on the first feature by a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block; The image coding method includes, in a synthesis transformation process, performing feature enhancement on the first feature to obtain a second feature, performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature, determining the target feature based on the third feature, and determining a reconstructed image block corresponding to the current image block based on the target feature.

[0048] The present invention is applied to the decoding side image A decoding device, comprising: a memory configured to store video data; Any of the above image decoding methods and a decoder configured to implement the above.

[0049] The present invention is applied to the coding side image 1. An encoding device, comprising: a memory configured to store video data; Any of the above image encoding methods and an encoder configured to implement the above.

[0050] The present invention provides a decoding device including a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor; The processor executes machine-executable instructions to Either of image A decoding device is provided for use in implementing the decoding method.

[0051] The present invention provides a coding device including a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor; The processor executes machine-executable instructions to Either of image An encoding device is provided for use in implementing the encoding method.

[0052] The present invention provides an electronic device including a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor, the processor executing the machine-executable instructions to perform the above-mentioned Either of image The processor may be used to perform the decoding method, or the processor may execute machine-executable instructions to perform the decoding method. Either of image An electronic device is provided for use in implementing the encoding method.

[0053] The present invention provides a machine-readable storage medium having stored thereon a plurality of computer instructions, which, when executed by a processor, Either of image The decoding method is carried out or Either of imageA machine-readable storage medium is provided on which the encoding method is implemented.

[0054] As can be seen from the above technical solution, in an embodiment of the present invention, a first feature corresponding to a current image block is obtained by a neural network, the first feature is adjusted according to a feature adjustment factor corresponding to the current image block to obtain a target feature, and a reconstructed image block corresponding to the current image block is obtained by the neural network according to the target feature, thereby providing an end-to-end video compression method based on a neural network, which achieves the purpose of improving coding and decoding efficiency by combining feature adjustment factors.By designing the network structure and combining it with an auxiliary bitstream, the quality of the reconstructed image block can be effectively guaranteed while maintaining low complexity of the neural network, thereby achieving the purpose of improving coding and decoding performance and reducing complexity. [Brief explanation of the drawings]

[0055] [Figure 1] FIG. 2 is a schematic diagram of a three-dimensional feature matrix in one embodiment of the present invention. [Figure 2] 1 is a flowchart of a decoding method according to an embodiment of the present invention. [Figure 3] 1 is a flowchart of an encoding method in one embodiment of the present invention. [Figure 4] FIG. 2 is a schematic diagram of the encoding process in one embodiment of the present invention. [Figure 5] FIG. 10 is a schematic diagram of a decoding process according to an embodiment of the present invention. [Figure 6A] 1 is a schematic diagram of a network structure in one embodiment of the present invention; [Figure 6B] 1 is a schematic diagram of a network structure in one embodiment of the present invention; [Figure 6C] 1 is a schematic diagram of a network structure in one embodiment of the present invention; [Figure 6D]1 is a schematic diagram of a network structure in one embodiment of the present invention; [Figure 6E] 1 is a schematic diagram of a network structure in one embodiment of the present invention; [Figure 6F] 1 is a schematic diagram of a network structure in one embodiment of the present invention; [Figure 6G] 1 is a schematic diagram of a network structure in one embodiment of the present invention; [Figure 6H] 1 is a schematic diagram of a network structure in one embodiment of the present invention; [Figure 7A] FIG. 10 is a schematic diagram of a decoding process according to an embodiment of the present invention. [Figure 7B] FIG. 2 is a schematic diagram of the encoding process in one embodiment of the present invention. [Figure 8A] FIG. 2 is a schematic diagram of the structure of a decoding sub-network in one embodiment of the present invention. [Figure 8B] FIG. 2 is a schematic diagram of the structure of a decoding sub-network in one embodiment of the present invention. [Figure 9A] FIG. 2 is a schematic diagram of the structure of an attention sub-network in one embodiment of the present invention. [Figure 9B] FIG. 2 is a schematic diagram of the structure of an attention sub-network in one embodiment of the present invention. [Figure 10A] FIG. 10 is a schematic diagram of the adjustment position of the feature adjustment factor in one embodiment of the present invention. [Figure 10B] FIG. 10 is a schematic diagram of the adjustment position of the feature adjustment factor in one embodiment of the present invention. [Figure 10C] FIG. 10 is a schematic diagram of the adjustment position of the feature adjustment factor in one embodiment of the present invention. [Figure 11A] FIG. 2 is a hardware structural diagram of a decoding device according to an embodiment of the present invention. [Figure 11B] FIG. 2 is a hardware structural diagram of an encoding device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0056] The terms used in the embodiments of the present invention are merely for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the embodiments and claims of the present invention, the singular forms "a," "the," and "the" are intended to include the plural forms unless the context clearly dictates otherwise. The term "and / or" as used herein should be understood to mean including any and all possible combinations of one or more associated listed items. While the embodiments of the present invention may use terms such as "first," "second," and "third" to describe various pieces of information, it should be understood that such information is not limited to these terms. These terms are used only to distinguish between pieces of information of the same type. For example, depending on the context, first information may be referred to as second information, and similarly, second information may be referred to as first information, without departing from the scope of the embodiments of the present invention. Furthermore, the word "if" as used herein may be interpreted as "with," "when," or "in response to a determination."

[0057] Embodiments of the present invention provide decoding and encoding methods and may relate to the following concepts.

[0058] JPEG (Joint Photographic Experts Group): JPEG is a standard for continuous-tone still image compression. Its file extensions may be .jpg or .jpeg. It is a common image file format. JPEG employs a joint coding method involving predictive coding (e.g., differential pulse-code modulation (DPCM), discrete cosine transform (DCT)), and entropy coding to eliminate redundant image and color data. It is a lossy compression format, compressing images into small storage spaces without damaging the image data. In particular, excessive compression can degrade the quality of the decompressed image. Therefore, excessive JPEG compression is undesirable when high-quality images are desired.

[0059] JPEG-AI (Joint Photographic Experts Group Artificial Intelligence): The scope of JPEG-AI is to create a learning-based image coding standard that provides a single-stream, compact, compressed-domain representation, significantly improves compression efficiency over commonly used image coding standards at the same subjective quality, and effectively improves performance in image processing and computer vision tasks. JPEG-AI is suitable for a wide range of applications, including cloud storage, vision management, autonomous driving, image collection, storage and management, real-time management of vision data, and media distribution. The goal of JPEG-AI is to design an encoding and decoding solution that significantly improves compression efficiency at the same subjective quality and provides effective compressed-domain processing for machine learning-based image processing and computer vision tasks. JPEG-AI must achieve hardware- and software-friendly encoding and decoding, support 8-bit and 10-bit depth, and perform efficient encoding and progressive decoding for images with text and graphics.

[0060] Entropy Encoding: Entropy coding is a coding method that does not lose information during the coding process according to the entropy principle, and information entropy is the average amount of information (degree of uncertainty) of the information source. Entropy coding methods may include, but are not limited to, Shannon coding, Huffman coding, and arithmetic coding.

[0061] Neural network (NN): A neural network refers to an artificial neural network. It is a computational model composed of a large number of interconnected nodes (also called neurons). In a neural network, neuron processing units can represent different objects, such as features, alphabets, concepts, or several meaningful abstract modes. The processing units in a neural network are divided into three types: input units, output units, and hidden units. Input units receive external signals and data, output units output the processing results, and hidden units are units located between the input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units, and information representation and processing are reflected in the connection relationships between processing units. A neural network is a non-programmed, brain-like information processing method. The essence of a neural network is to obtain parallel, distributed information processing functions through the transformation and dynamic behavior of neural networks, imitating the information processing functions of the human nervous system to different degrees and levels. In the video processing field, commonly used neural networks may include, but are not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), fully connected networks, etc.

[0062] Convolutional Neural Network (CNN): A convolutional neural network (CNN) is a feedforward neural network and one of the most representative network structures in deep learning technology. Its artificial neurons can respond to peripheral units within a limited coverage area, demonstrating excellent performance in large-scale image processing. The basic structure of a convolutional neural network includes two layers: a feature extraction layer (also called a convolutional layer), in which each neuron's input is connected to the local receptive field of the previous layer to extract local features. Once the local feature is extracted, its positional relationship with other features is also determined. The other layer is a feature mapping layer (also called an activation layer). Each computational layer of the neural network consists of multiple feature mappings, each of which is a plane, with equal weights for all neurons in the plane. The feature mapping structure may use a sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, or GDN function as the activation function for the convolutional network. Also, since neurons in one mapping surface share weights, the number of free parameters in the network is reduced.

[0063] For example, one advantage of convolutional neural networks over image processing algorithms is that they can avoid complex image preprocessing processes (such as artificial feature extraction) and directly input the original image for end-to-end learning. One advantage of convolutional neural networks over conventional neural networks is that, while conventional neural networks use a fully connected method, meaning that all neurons from the input layer to the hidden layer are connected, resulting in a huge number of parameters and making network training time-consuming and difficult, convolutional neural networks avoid this difficulty by using methods such as local connections and weight value sharing.

[0064] Deconvolution layer: Deconvolution layer is also called transposed convolution layer. The operation process of deconvolution layer and convolution layer is similar, the main difference is that deconvolution layer uses padding to make the output larger than the input (it can be the same). A stride of 1 indicates that the output size is equal to the input size, and a stride of N indicates that the width of the output feature is N times the width of the input feature and the height of the output feature is N times the height of the input feature.

[0065] Generalization Ability: Generalization ability may refer to the adaptability of a machine learning algorithm to fresh samples, and the purpose of learning is to learn the rules behind the data. The trained network can give appropriate outputs even for data outside the learning set that has the same rules, and this ability may be called generalization ability.

[0066] Feature: The feature of the present invention is a three-dimensional feature matrix of C*W*H. Figure 1 is a schematic diagram of the three-dimensional feature matrix, where C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be the input or output of a neural network.

[0067] Rate-Distortion Optimization Principle: There are two indices to evaluate coding efficiency: bit rate and PSNR (Peak Signal to Noise Ratio). The smaller the bitstream, the higher the compression rate. The higher the PSNR, the better the quality of the reconstructed image. When selecting a mode, the discriminant is essentially a comprehensive evaluation of both. For example, the cost corresponding to a mode is: J(mode)=D+λ*R, where D represents distortion, which can usually be evaluated using the SSE (Sum of the Squared Errors) index, where SSE refers to the root mean square of the difference between the reconstructed image block and the source image. To consider the cost, the SAD index can also be used, where SAD refers to the sum of the absolute values ​​of the differences between the reconstructed image block and the source image, λ is the Lagrange multiplier, and R is the actual number of bits required to encode the image block in that mode, including the sum of bits required to encode mode information, motion information, residual, etc. During mode selection, using rate-distortion principles to compare and decide between coding modes can usually ensure optimal coding performance.

[0068] A large number of coding tools have been proposed for each module on the encoding side, and each tool typically has many modes. Different coding tools often provide optimal coding performance for different video sequences. Therefore, during the encoding process, a rate-distortion optimization (RDO) is typically used to compare the coding performance of different tools or modes and select the optimal mode. After determining the optimal tool or mode, the tool or mode determination information is transmitted by encoding mark information into the bitstream. While this method increases coding complexity, it enables adaptive selection of optimal mode combinations for different content and optimal coding performance. The decoding side can obtain relevant mode information by directly analyzing mark information, resulting in a low impact on complexity.

[0069] Hereinafter, the decoding method and encoding method in the embodiments of the present invention will be described in detail with reference to some specific examples.

[0070] Embodiment 1: An embodiment of the present invention provides a decoding method, and FIG. 2 is a schematic flowchart of the decoding method, which can be applied to a decoding side (also called a video decoder), and may include steps 201 to 203.

[0071] Step 201: Obtain a first feature corresponding to a current image block by a first neural network, and obtain a feature adjustment factor corresponding to the current image block, where the first neural network may include at least one convolution layer.

[0072] In one possible embodiment, the step of obtaining the first feature corresponding to the current image block by the first neural network may include, but is not limited to, the steps of obtaining a probability distribution parameter based on the first bitstream corresponding to the current image block, determining a probability distribution model based on the probability distribution parameter, decoding the second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features, and determining the first feature corresponding to the current image block based on the decoded image features. As can be seen from the above, in this embodiment, the first neural network is used to realize functions such as obtaining the probability distribution parameter, determining the probability distribution model, decoding the second bitstream corresponding to the current image block, and determining the first feature corresponding to the current image block.

[0073] In another possible embodiment, the step of obtaining the first feature corresponding to the current image block by the first neural network may include, but is not limited to, the steps of obtaining a probability distribution parameter and a predicted value based on the first bitstream corresponding to the current image block, determining a probability distribution model based on the probability distribution parameter, and decoding the second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features, performing residual reconstruction on the decoded image features to obtain residual features, and determining the first feature corresponding to the current image block based on the residual features and the predicted value. As can be seen from the above, in this embodiment, the first neural network is used to realize functions such as obtaining the probability distribution parameter, obtaining the predicted value, determining the probability distribution model, decoding the second bitstream corresponding to the current image block, residual reconstruction, and determining the first feature corresponding to the current image block.

[0074] Exemplarily, the step of obtaining the feature adjustment factor corresponding to the current image block may include, but is not limited to, decoding the auxiliary bitstream corresponding to the current image block to obtain the feature adjustment factor corresponding to the current image block, i.e., analyzing the feature adjustment factor corresponding to the current image block from the auxiliary bitstream, or determining a fixed parameter value as the feature adjustment factor corresponding to the current image block, for example, determining a fixed parameter value of 1 as the feature adjustment factor corresponding to the current image block.

[0075] Step 202: Determine a target feature based on the first feature and the feature adjustment factor.

[0076] Exemplarily, the step of determining a target feature based on the first feature and the feature adjustment factor may include, but is not limited to, the steps of: performing feature enhancement on the first feature to obtain a second feature; after obtaining the second feature, performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature; and, after obtaining the third feature, determining a target feature based on the third feature.

[0077] In one possible embodiment, the step of performing feature enhancement on the first feature to obtain the second feature may include, but is not limited to, determining an initial feature corresponding to the attention sub-network based on the first feature, and performing feature enhancement on the initial feature by the attention sub-network to obtain the second feature. After obtaining the second feature, the step of performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature (i.e., the feature after feature adjustment is the third feature). After obtaining the third feature, the step of determining a target feature based on the third feature may include, but is not limited to, determining the third feature as the target feature.

[0078] Exemplarily, the attention sub-network may include a residual enhancement sub-network and a weight generation sub-network, and the step of performing feature enhancement on the initial features to obtain second features by the attention sub-network may include, but is not limited to, performing feature enhancement on the initial features using the residual enhancement sub-network to obtain enhanced features, generating weight features corresponding to the initial features using the weight generation sub-network, and generating second features based on the initial features, the enhanced features, and the weight features.

[0079] Exemplarily, the weight generation subnetwork may include a residual block subnetwork, a convolutional subnetwork, and a feature mapping subnetwork, and the step of generating weight features corresponding to the initial features using the weight generation subnetwork may include, but is not limited to, performing convolutional activation on the initial features using the residual block subnetwork to obtain post-convolutional activation features; performing convolution on the post-convolutional activation features using the convolutional subnetwork to obtain post-convolutional features; and performing feature mapping on the post-convolutional features using the feature mapping subnetwork to obtain weight features.

[0080] Exemplarily, the step of determining initial features corresponding to an attention sub-network based on a first feature may include, but is not limited to, performing feature enhancement on the first feature using an initial enhancement sub-network to obtain enhanced features; and performing upsampling convolution on the enhanced features using an upsampling convolution sub-network to obtain initial features corresponding to the attention sub-network.

[0081] In another possible embodiment, the step of performing feature enhancement on the first feature to obtain a second feature may include, but is not limited to, determining an initial feature corresponding to the attention sub-network based on the first feature, and performing feature enhancement on the initial feature by a first sub-network in the attention sub-network to obtain a second feature. After obtaining the second feature, the step of performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature (i.e., the feature after feature adjustment is the third feature). After obtaining the third feature, the step of determining a target feature based on the third feature may include, but is not limited to, processing the third feature by a second sub-network in the attention sub-network to obtain a fourth feature, and determining the fourth feature as the target feature.

[0082] For example, performing feature enhancement on the initial features to obtain second features by a first sub-network in the attention sub-network may include, but is not limited to, performing feature enhancement on the initial features to obtain enhanced features by a residual enhancement sub-network, generating weight features corresponding to the initial features by a weight generation sub-network, and generating second features based on the enhanced features and the weight features. Processing the third features to obtain fourth features by a second sub-network in the attention sub-network may include, but is not limited to, generating fourth features based on the initial features and the third features.

[0083] The weight generation subnetwork may include a residual block subnetwork, a convolutional subnetwork, and a feature mapping subnetwork, and generating weight features corresponding to the initial features using the weight generation subnetwork may include, but is not limited to, performing convolutional activation on the initial features using the residual block subnetwork to obtain post-convolutional activation features, performing convolution on the post-convolutional activation features using the convolutional subnetwork to obtain post-convolutional features, and performing feature mapping on the post-convolutional features using the feature mapping subnetwork to obtain weight features.

[0084] For example, performing feature enhancement on the initial features to obtain second features by a first sub-network in the attention sub-network may include, but is not limited to, performing feature enhancement on the initial features to obtain enhanced features using a residual block sub-network, performing convolution on the enhanced features to obtain convolved features using a convolution sub-network, and determining the second features based on the convolved features. Processing the third features to obtain fourth features by a second sub-network in the attention sub-network may include, but is not limited to, performing feature mapping on the third features to obtain weighted features using a feature mapping sub-network, performing feature enhancement on the initial features to obtain enhanced features using a residual enhancement sub-network, and generating a fourth feature based on the initial features, the enhanced features, and the weighted features.

[0085] In the above embodiment, the step of determining initial features corresponding to an attention sub-network based on a first feature may include, but is not limited to, performing feature enhancement on the first feature using an initial enhancement sub-network to obtain enhanced features; and performing upsampling convolution on the enhanced features using an upsampling convolution sub-network to obtain initial features corresponding to the attention sub-network.

[0086] In one possible embodiment, the step of performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature may include, but is not limited to, when the second feature includes C*H*W feature values, where C represents the number of channels, H represents the feature height, and W represents the feature width, determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the feature adjustment factor, and performing feature adjustment on the C*H*W feature values ​​based on the feature adjustment values ​​to obtain adjusted feature values ​​corresponding to the C*H*W feature values, based on which a third feature can be generated based on the adjusted feature values ​​corresponding to the C*H*W feature values.

[0087] For example, determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the feature adjustment factor may include, but is not limited to, determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the C*H*W feature adjustment values ​​when the feature adjustment factor includes C*H*W feature adjustment values; determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the H*W feature adjustment values ​​when the feature adjustment factor includes H*W feature adjustment values; or determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the C feature adjustment values ​​when the feature adjustment factor includes C feature adjustment values.

[0088] Exemplarily, the step of performing feature adjustment on the C*H*W feature values ​​based on the feature adjustment value to obtain adjusted feature values ​​corresponding to the C*H*W feature values ​​may include, when one of the C*H*W feature values ​​corresponds to one feature adjustment value, determining an adjusted feature value corresponding to the feature value using a formula yr=rf*y, where yr represents the adjusted feature value, rf represents the feature adjustment value, and y represents the feature value; or, when one of the C*H*W feature values ​​corresponds to N+1 feature adjustment values, where N is a positive integer, determining an adjusted feature value corresponding to the feature value using a formula yr=rf_0*y 0 +rf_1*y 1 +rf_2*y 2 +…+rf_N*y N where yr represents the adjusted feature value, rf_0, rf_1, rf_2, ..., rf_N represent the N+1 feature adjustment values, and y represents the feature value.

[0089] Step 203: Obtain a reconstructed image block corresponding to the current image block (i.e., the final output reconstructed image block) by a second neural network based on the target feature, where the second neural network may include at least one convolution layer.

[0090] Exemplarily, the second neural network may include a reconstruction decoding subnetwork, and the step of obtaining a reconstructed image block corresponding to the current image block by the second neural network based on the target feature may include, but is not limited to, processing the target feature by the reconstruction decoding subnetwork to obtain a reconstructed image block corresponding to the current image block. For example, the reconstruction decoding subnetwork may include an upsampling convolution subnetwork, and the upsampling convolution subnetwork may perform upsampling convolution on the target feature to obtain a reconstructed image block corresponding to the current image block. Or, for example, the reconstruction decoding subnetwork may include an upsampling convolution subnetwork and a color space conversion subnetwork, and the upsampling convolution subnetwork may perform upsampling convolution on the target feature and perform color space conversion on the upsampling convolved feature by the color space conversion subnetwork to obtain a reconstructed image block corresponding to the current image block.

[0091] In one possible embodiment, if the feature adjustment mode is on for the current image block, obtain the feature adjustment factor corresponding to the current image block, and determine the target feature based on the first feature and the feature adjustment factor; or if the feature adjustment mode is not on for the current image block, determine the target feature based on the first feature, and there is no need to obtain the feature adjustment factor corresponding to the current image block.

[0092] For example, a feature adjustment flag bit may be analyzed from the bitstream, and if the feature adjustment flag bit allows the feature adjustment mode to be turned on for the current image block, it may be determined that the feature adjustment mode is on for the current image block; alternatively, a first value range of feature values ​​may be analyzed from the bitstream, and if the feature value of a second feature is in the first value range, it may be determined that the feature adjustment mode is on for the current image block; alternatively, a second value range of feature change values ​​may be analyzed from the bitstream, and if the feature change value corresponding to the second feature is in the second value range, it may be determined that the feature adjustment mode is on for the current image block.

[0093] For example, if the attribute value corresponding to the second feature is within a value range of a set attribute value, it may be determined that the feature adjustment mode is on for the current image block, and the attribute value may include a variance value.

[0094] For illustrative purposes, the above execution order is merely an example for ease of explanation, and in actual applications, the execution order between steps may be changed and is not limited to this execution order. In addition, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in the present invention, and the method may include more or fewer steps than those described in the present invention. In addition, a single step described in the present invention may be decomposed and described as multiple steps in other embodiments, and multiple steps described in the present invention may be combined and described as a single step in other embodiments.

[0095] As can be seen from the above technical solution, in this embodiment of the present invention, a first feature corresponding to a current image block is obtained by a first neural network, the first feature is adjusted according to a feature adjustment factor corresponding to the current image block to obtain a target feature, and a second neural network is used to obtain a reconstructed image block corresponding to the current image block according to the target feature, thereby realizing video decoding based on neural networks and achieving the purpose of improving decoding efficiency by combining feature adjustment factors.By designing the network structure and combining it with the auxiliary bitstream, the quality of the reconstructed image block can be effectively guaranteed while maintaining low complexity of the neural network, thereby achieving the purpose of improving decoding performance and reducing complexity.

[0096] Embodiment 2: An embodiment of the present invention provides an encoding method, and FIG. 3 is a schematic flowchart of the encoding method, which can be applied to an encoding side (also called a video encoder), and the method may include:

[0097] Step 301: Obtain a first feature corresponding to a current image block by a first neural network, where the first neural network may include at least one convolution layer.

[0098] In one possible embodiment, the step of obtaining the first feature corresponding to the current image block by the first neural network may include, but is not limited to, the steps of obtaining a probability distribution parameter based on the first bitstream corresponding to the current image block, determining a probability distribution model based on the probability distribution parameter, decoding the second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features, and determining the first feature corresponding to the current image block based on the decoded image features. As can be seen from the above, in this embodiment, the first neural network is used to realize functions such as obtaining the probability distribution parameter, determining the probability distribution model, decoding the second bitstream corresponding to the current image block, and determining the first feature corresponding to the current image block.

[0099] In another possible embodiment, the step of obtaining the first feature corresponding to the current image block by the first neural network may include, but is not limited to, the steps of obtaining a probability distribution parameter and a predicted value based on the first bitstream corresponding to the current image block, determining a probability distribution model based on the probability distribution parameter, and decoding the second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features, performing residual reconstruction on the decoded image features to obtain residual features, and determining the first feature corresponding to the current image block based on the residual features and the predicted value. As can be seen from the above, in this embodiment, the first neural network is used to realize functions such as obtaining the probability distribution parameter, obtaining the predicted value, determining the probability distribution model, decoding the second bitstream corresponding to the current image block, residual reconstruction, and determining the first feature corresponding to the current image block.

[0100] Step 302: Obtain a feature adjustment factor corresponding to the current image block based on the first feature.

[0101] Exemplarily, the step of obtaining a feature adjustment factor corresponding to the current image block based on the first feature may include, but is not limited to, the steps of obtaining at least one candidate feature adjustment factor, determining a rate-distortion cost corresponding to each candidate feature adjustment factor based on the first feature, and selecting one candidate feature adjustment factor from all the candidate feature adjustment factors as the feature adjustment factor corresponding to the current image block based on the rate-distortion cost corresponding to each candidate feature adjustment factor.

[0102] Step 303: Encode the feature adjustment factor into the auxiliary bitstream corresponding to the current image block.

[0103] For example, a fixed parameter value may be determined as the feature adjustment factor corresponding to the current image block, for example, a fixed parameter value of 1 may be determined as the feature adjustment factor corresponding to the current image block, in which case the feature adjustment factor does not need to be coded in the auxiliary bitstream corresponding to the current image block.

[0104] For example, a target feature may be determined based on a first feature corresponding to the current image block and a feature adjustment factor corresponding to the current image block, and a reconstructed image block corresponding to the current image block (i.e., the finally output reconstructed image block) may be obtained by a second neural network based on the target feature, and the second neural network may include at least one convolutional layer.

[0105] Exemplarily, the step of determining a target feature based on the first feature and the feature adjustment factor may include, but is not limited to, the steps of: performing feature enhancement on the first feature to obtain a second feature; after obtaining the second feature, performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature; and, after obtaining the third feature, determining a target feature based on the third feature.

[0106] In one possible embodiment, the step of performing feature enhancement on the first feature to obtain the second feature may include, but is not limited to, determining an initial feature corresponding to the attention sub-network based on the first feature, and performing feature enhancement on the initial feature by the attention sub-network to obtain the second feature. After obtaining the second feature, the step of performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature (i.e., the feature after feature adjustment is the third feature). After obtaining the third feature, the step of determining a target feature based on the third feature may include, but is not limited to, determining the third feature as the target feature.

[0107] Exemplarily, the attention sub-network may include a residual enhancement sub-network and a weight generation sub-network, and the step of performing feature enhancement on the initial features to obtain second features by the attention sub-network may include, but is not limited to, performing feature enhancement on the initial features using the residual enhancement sub-network to obtain enhanced features, generating weight features corresponding to the initial features using the weight generation sub-network, and generating second features based on the initial features, the enhanced features, and the weight features.

[0108] Exemplarily, the weight generation subnetwork may include a residual block subnetwork, a convolutional subnetwork, and a feature mapping subnetwork, and the step of generating weight features corresponding to the initial features using the weight generation subnetwork may include, but is not limited to, performing convolutional activation on the initial features using the residual block subnetwork to obtain post-convolutional activation features; performing convolution on the post-convolutional activation features using the convolutional subnetwork to obtain post-convolutional features; and performing feature mapping on the post-convolutional features using the feature mapping subnetwork to obtain weight features.

[0109] Exemplarily, the step of determining initial features corresponding to an attention sub-network based on a first feature may include, but is not limited to, performing feature enhancement on the first feature using an initial enhancement sub-network to obtain enhanced features; and performing upsampling convolution on the enhanced features using an upsampling convolution sub-network to obtain initial features corresponding to the attention sub-network.

[0110] In another possible embodiment, the step of performing feature enhancement on the first feature to obtain a second feature may include, but is not limited to, determining an initial feature corresponding to the attention sub-network based on the first feature, and performing feature enhancement on the initial feature by a first sub-network in the attention sub-network to obtain a second feature. After obtaining the second feature, the step of performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature (i.e., the feature after feature adjustment is the third feature). After obtaining the third feature, the step of determining a target feature based on the third feature may include, but is not limited to, processing the third feature by a second sub-network in the attention sub-network to obtain a fourth feature, and determining the fourth feature as the target feature.

[0111] For example, performing feature enhancement on the initial features to obtain second features by a first sub-network in the attention sub-network may include, but is not limited to, performing feature enhancement on the initial features to obtain enhanced features by a residual enhancement sub-network, generating weight features corresponding to the initial features by a weight generation sub-network, and generating second features based on the enhanced features and the weight features. Processing the third features to obtain fourth features by a second sub-network in the attention sub-network may include, but is not limited to, generating fourth features based on the initial features and the third features.

[0112] The weight generation subnetwork may include a residual block subnetwork, a convolutional subnetwork, and a feature mapping subnetwork, and generating weight features corresponding to the initial features using the weight generation subnetwork may include, but is not limited to, performing convolutional activation on the initial features using the residual block subnetwork to obtain post-convolutional activation features, performing convolution on the post-convolutional activation features using the convolutional subnetwork to obtain post-convolutional features, and performing feature mapping on the post-convolutional features using the feature mapping subnetwork to obtain weight features.

[0113] For example, performing feature enhancement on the initial features to obtain second features by a first sub-network in the attention sub-network may include, but is not limited to, performing feature enhancement on the initial features to obtain enhanced features using a residual block sub-network, performing convolution on the enhanced features to obtain convolved features using a convolution sub-network, and determining the second features based on the convolved features. Processing the third features to obtain fourth features by a second sub-network in the attention sub-network may include, but is not limited to, performing feature mapping on the third features to obtain weighted features using a feature mapping sub-network, performing feature enhancement on the initial features to obtain enhanced features using a residual enhancement sub-network, and generating a fourth feature based on the initial features, the enhanced features, and the weighted features.

[0114] In the above embodiment, the step of determining initial features corresponding to an attention sub-network based on a first feature may include, but is not limited to, performing feature enhancement on the first feature using an initial enhancement sub-network to obtain enhanced features; and performing upsampling convolution on the enhanced features using an upsampling convolution sub-network to obtain initial features corresponding to the attention sub-network.

[0115] In one possible embodiment, the step of performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature may include, but is not limited to, when the second feature includes C*H*W feature values, where C represents the number of channels, H represents the feature height, and W represents the feature width, determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the feature adjustment factor, and performing feature adjustment on the C*H*W feature values ​​based on the feature adjustment values ​​to obtain adjusted feature values ​​corresponding to the C*H*W feature values, based on which a third feature can be generated based on the adjusted feature values ​​corresponding to the C*H*W feature values.

[0116] For example, determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the feature adjustment factor may include, but is not limited to, determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the C*H*W feature adjustment values ​​when the feature adjustment factor includes C*H*W feature adjustment values; determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the H*W feature adjustment values ​​when the feature adjustment factor includes H*W feature adjustment values; or determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the C feature adjustment values ​​when the feature adjustment factor includes C feature adjustment values.

[0117] Exemplarily, the step of performing feature adjustment on the C*H*W feature values ​​based on the feature adjustment values ​​to obtain adjusted feature values ​​corresponding to the C*H*W feature values ​​may include, when one of the C*H*W feature values ​​corresponds to one feature adjustment value, determining an adjusted feature value corresponding to the feature value using a formula yr=rf*y, where yr represents the adjusted feature value, rf represents the feature adjustment value, and y represents the feature value; or, when one of the C*H*W feature values ​​corresponds to N+1 feature adjustment values, where N is a positive integer, determining an adjusted feature value corresponding to the feature value using a formula yr=rf_0*y 0 +rf_1*y 1 +rf_2*y 2+…+rf_N*y N where yr represents the adjusted feature value, rf_0, rf_1, rf_2, ..., rf_N represent the N+1 feature adjustment values, and y represents the feature value.

[0118] Exemplarily, the second neural network may include a reconstruction decoding subnetwork, and the step of obtaining a reconstructed image block corresponding to the current image block by the second neural network based on the target feature may include, but is not limited to, processing the target feature by the reconstruction decoding subnetwork to obtain a reconstructed image block corresponding to the current image block. For example, the reconstruction decoding subnetwork may include an upsampling convolution subnetwork, and the upsampling convolution subnetwork may perform upsampling convolution on the target feature to obtain a reconstructed image block corresponding to the current image block. Or, for example, the reconstruction decoding subnetwork may include an upsampling convolution subnetwork and a color space conversion subnetwork, and the upsampling convolution subnetwork may perform upsampling convolution on the target feature and perform color space conversion on the upsampling convolved feature by the color space conversion subnetwork to obtain a reconstructed image block corresponding to the current image block.

[0119] In one possible embodiment, if the feature adjustment mode is on for the current image block, obtain a feature adjustment factor corresponding to the current image block based on the first feature, and encode the feature adjustment factor into the auxiliary bitstream corresponding to the current image block; or if the feature adjustment mode is not on for the current image block, there is no need to obtain a feature adjustment factor corresponding to the current image block.

[0120] For example, the bitstream may include encoding a feature adjustment flag bit that enables or disables the feature adjustment mode for the current image block, a first value range of feature values, or a second value range of feature change values.

[0121] For illustrative purposes, the above execution order is merely an example for ease of explanation, and in actual applications, the execution order between steps may be changed and is not limited to this execution order. In addition, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in the present invention, and the method may include more or fewer steps than those described in the present invention. In addition, a single step described in the present invention may be decomposed and described as multiple steps in other embodiments, and multiple steps described in the present invention may be combined and described as a single step in other embodiments.

[0122] As can be seen from the above technical solutions, the embodiments of the present invention provide an end-to-end video compression method that can realize video image encoding based on neural networks and achieve the purpose of improving encoding efficiency by combining feature adjustment factors. By combining the design of the network structure and the auxiliary bitstream (used to carry the feature adjustment factors), the quality of the reconstructed image blocks can be effectively guaranteed while maintaining the low complexity of the neural network, thereby achieving the purpose of improving encoding performance and reducing complexity.

[0123] Third Embodiment: For the first and second embodiments, the encoding process is shown in FIG. 4. Of course, FIG. 4 is merely an example of the encoding process, and the encoding process is not limited to this.

[0124] After obtaining a current image block x (the current image block x may be the original image block x, i.e., the input image block), the encoding side may perform analysis transformation on the current image block x using an analysis transformation network (i.e., a neural network) to obtain image features y corresponding to the current image block x. Here, performing feature transformation on the current image block x using the analysis transformation network converts the current image block x into image features y in the latent domain, allowing all subsequent processes to operate in the latent domain.

[0125] The image may be divided into one image block or multiple image blocks, and if the image is divided into one image block, the current image block x may be the image, that is, the encoding and decoding process for the image block may be directly used on the image.

[0126] After obtaining image feature y, the encoding side performs coefficient hyperparameter feature transformation on image feature y to obtain coefficient hyperparameter feature z. For example, image feature y is input to a hyperparameter coding network (i.e., a neural network), and the hyperparameter coding network performs coefficient hyperparameter feature transformation on image feature y to obtain coefficient hyperparameter feature z. The hyperparameter coding network may be a trained neural network, and the training process of this hyperparameter coding network is not limited as long as it can perform coefficient hyperparameter feature transformation on image feature y. Here, after image feature y in the latent domain passes through the hyperparameter coding network, super-prior latent information z is obtained.

[0127] After obtaining the coefficient hyperparameter feature z, the encoding side may quantize the coefficient hyperparameter feature z to obtain a hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, i.e., the Q operation in FIG. 4 is the quantization process. After obtaining the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, the encoding side may encode the hyperparameter quantization feature to obtain Bitstream#1 (i.e., the first bitstream) corresponding to the current image block, i.e., the AE operation in FIG. 4 represents an encoding process such as an entropy encoding process. Alternatively, the encoding side may directly encode the coefficient hyperparameter feature z to obtain Bitstream#1 corresponding to the current image block. The hyperparameter quantization feature or the coefficient hyperparameter feature z included in Bitstream#1 is mainly used to obtain the mean value and parameters of the probability distribution model.

[0128] After obtaining Bitstream #1 corresponding to the current image block, the encoding side may send Bitstream #1 corresponding to the current image block to the decoding side, and the processing process for Bitstream #1 corresponding to the current image block by the decoding side will be described in the subsequent embodiments.

[0129] After obtaining Bitstream#1 corresponding to the current image block, the encoding side may decode Bitstream#1 to obtain hyperparameter quantization features, that is, AD in Figure 4 represents the decoding process, and then dequantize the hyperparameter quantization features to obtain coefficient hyperparameter features z_hat, which may be the same as or different from the coefficient hyperparameter features z, and the IQ operation in Figure 4 is the dequantization process. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the encoding side may decode Bitstream#1 to obtain coefficient hyperparameter features z_hat without performing the dequantization process of the coefficient hyperparameter features z_hat.

[0130] A fixed probability density model encoding method may be adopted for the encoding process of Bitstream#1, and a fixed probability density model decoding method may be adopted for the decoding process of Bitstream#1, and there are no limitations on these encoding and decoding processes.

[0131] After obtaining the coefficient hyperparameter feature z_hat, the encoding side may perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block (the process of determining the residual feature y_hat will be described in subsequent embodiments) to obtain a predicted value mu (i.e., average value mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat are input to a mean value prediction network, and the mean value prediction network determines the predicted value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. This prediction process is not limited. Here, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat, and the two are combined and input to obtain a more accurate predicted value mu. The predicted value mu is used to calculate the difference between the original feature and the residual, which is then added to the decoded residual to obtain a reconstruction y.

[0132] Note that the mean value prediction network is a selectable neural network, i.e., there is no need to have a mean value prediction network, i.e., there is no need to determine the predicted value mu by the mean value prediction network, and the dashed frame in Figure 4 indicates that the mean value prediction network is selectable.

[0133] After obtaining image feature y, the encoding side can determine residual feature r based on image feature y and predicted value mu. For example, the difference between image feature y and predicted value mu is defined as residual feature r. Then, feature processing is performed on residual feature r to obtain image feature s. This feature processing process is not limited and can be any feature processing method. In this case, a mean value prediction network must be installed, and the predicted value mu is provided by the mean value prediction network. Alternatively, after obtaining image feature y, the encoding side can perform feature processing on image feature y to obtain image feature s. This feature processing process is not limited and can be any feature processing method. In this case, there is no need to install a mean value prediction network, and the dashed frame indicates that the residual process is a selectable process.

[0134] After obtaining the image feature s, the encoding side may quantize the image feature s to obtain an image quantization feature corresponding to the image feature s, i.e., the Q operation in Fig. 4 is the quantization process. After obtaining the image quantization feature corresponding to the image feature s, the encoding side may encode the image quantization feature to obtain Bitstream#2 (i.e., the second bitstream) corresponding to the current image block, i.e., the AE operation in Fig. 4 represents an encoding process such as an entropy encoding process. Alternatively, the encoding side may directly encode the image feature s to obtain Bitstream#2 corresponding to the current image block without performing a quantization process on the image feature s.

[0135] After obtaining Bitstream #2 corresponding to the current image block, the encoding side may send Bitstream #2 corresponding to the current image block to the decoding side, and the processing process for Bitstream #2 corresponding to the current image block by the decoding side will be described in the subsequent embodiments.

[0136] After obtaining Bitstream#2 corresponding to the current image block, the encoding side may decode Bitstream#2 to obtain image quantization features, that is, AD in Figure 4 represents the decoding process, and then the encoding side may dequantize the image quantization features to obtain image feature s', where image feature s' may be the same as or different from image feature s, and the IQ operation in Figure 4 is the dequantization process. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the encoding side may decode Bitstream#2 to obtain image feature s' without performing the dequantization process of the image quantization features.

[0137] After obtaining image feature s', the encoding side may perform feature reconstruction (i.e., the inverse process of feature processing) on ​​image feature s'. This feature reconstruction process is not limited and may be any feature reconstruction method. A residual feature r_hat is obtained, and the residual feature r_hat may be the same as or different from the residual feature r. After obtaining the residual feature r_hat, the encoding side determines image feature y_hat based on the residual feature r_hat and the predicted value mu. The image feature y_hat may be the same as or different from the image feature y. For example, the image feature y_hat is the sum of the residual feature r_hat and the predicted value mu. In this case, a mean value prediction network must be implemented, and the predicted value mu is provided by the mean value prediction network. Alternatively, after obtaining image feature s', the encoding side may perform feature reconstruction (i.e., the inverse process of feature processing) on ​​image feature s' to obtain image feature y_hat. The image feature y_hat may be the same as or different from the image feature y. In this case, there is no need to implement a mean value prediction network, and the dashed frame indicates that the residual process is an optional process.

[0138] After obtaining the image feature y_hat, the encoding side may perform a synthesis transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into a synthesis transformation network, and the synthesis transformation network performs a synthesis transformation on the image feature y_hat to obtain a reconstructed image block x_hat. Up to this point, the image reconstruction process is completed.

[0139] In one possible embodiment, when the encoding side encodes the image quantization feature or image feature s to obtain Bitstream#2 corresponding to the current image block, the encoding side must first determine a probability distribution model, and then encode the image quantization feature or image feature s based on the probability distribution model. Also, when the encoding side decodes Bitstream#2, the encoding side must first determine a probability distribution model, and then decode Bitstream#2 based on the probability distribution model.

[0140] To obtain a probability distribution model, as shown in Figure 4, the encoding side obtains a coefficient hyperparameter feature z_hat, and then performs a coefficient hyperparameter feature inverse transform on the coefficient hyperparameter feature z_hat to obtain a probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat may be input to a probability hyperparameter decoding network, and the probability hyperparameter decoding network may perform a coefficient hyperparameter feature inverse transform on the coefficient hyperparameter feature z_hat to obtain a probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model may be generated based on the probability distribution parameter p. Here, the probability hyperparameter decoding network may be a trained neural network, and the training process of this probability hyperparameter decoding network is not limited as long as it can perform a coefficient hyperparameter feature inverse transform on the coefficient hyperparameter feature z_hat.

[0141] In one possible embodiment, the encoding side processing process may be performed by a deep learning model or a neural network model to realize an end-to-end image compression and encoding process, and there is no limitation on this encoding process.

[0142] Embodiment 4: For the embodiments 1 and 2, the processing process on the decoding side is shown in FIG. 5. Of course, FIG. 5 is merely an example of the processing process on the decoding side, and the processing process on the decoding side is not limited to this.

[0143] After obtaining Bitstream#1 corresponding to the current image block, the decoding side may decode Bitstream#1 to obtain hyperparameter quantization features, that is, AD in Figure 5 represents the decoding process, and then dequantize the hyperparameter quantization features to obtain coefficient hyperparameter features z_hat, which may be the same as or different from the coefficient hyperparameter features z, and the IQ operation in Figure 5 is the dequantization process. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the decoding side may decode Bitstream#1 to obtain coefficient hyperparameter features z_hat without performing the dequantization process of the coefficient hyperparameter features z_hat.

[0144] The decoding process of Bitstream#1 may employ a fixed probability density model decoding method, but is not limited thereto.

[0145] The image may be divided into one image block or multiple image blocks, and if the image is divided into one image block, the current image block x may be the image, that is, the decoding process for the image block may be directly used on the image.

[0146] After obtaining the coefficient hyperparameter feature z_hat, the decoding side may perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block (the process of determining the residual feature y_hat will be described in subsequent embodiments) to obtain a predicted value mu (i.e., average value mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat are input to a mean value prediction network, and the mean value prediction network determines the predicted value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. This prediction process is not limited. Here, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat, and the two are combined and input to obtain a more accurate predicted value mu.

[0147] Note that the mean value prediction network is a selectable neural network, i.e., there is no need to have a mean value prediction network, i.e., there is no need to determine the predicted value mu by the mean value prediction network, and the dashed frame in Figure 5 indicates that the mean value prediction network is selectable.

[0148] After obtaining Bitstream#2 corresponding to the current image block, the decoding side may decode Bitstream#2 to obtain image quantization features, that is, AD in Figure 5 represents the decoding process, and then the decoding side may dequantize the image quantization features to obtain image feature s', where image feature s' may be the same as or different from image feature s, and the IQ operation in Figure 5 is the dequantization process. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the decoding side may decode Bitstream#2 to obtain image feature s' without performing the dequantization process of the image quantization features.

[0149] After obtaining image feature s', the decoding side may perform feature reconstruction (i.e., the inverse process of feature processing) on ​​image feature s' to obtain residual feature r_hat, where residual feature r_hat may be the same as or different from residual feature r. After obtaining residual feature r_hat, the decoding side determines image feature y_hat based on residual feature r_hat and predicted value mu, where image feature y_hat may be the same as or different from image feature y; for example, image feature y_hat is the sum of residual feature r_hat and predicted value mu. In this case, a mean value prediction network must be deployed, and the predicted value mu is provided by the mean value prediction network. Alternatively, after obtaining image feature s', the decoding side may perform feature reconstruction on image feature s' to obtain image feature y_hat, where image feature y_hat may be the same as or different from image feature y. In this case, there is no need to deploy a mean value prediction network, and the dashed frame indicates that the residual process is a selectable process.

[0150] After obtaining the image feature y_hat, the decoding side may perform a synthesis transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into a synthesis transformation network, and the synthesis transformation network performs a synthesis transformation on the image feature y_hat to obtain a reconstructed image block x_hat. Up to this point, the image reconstruction process is completed.

[0151] In one possible embodiment, when decoding Bitstream #2, the decoding side must first determine a probability distribution model, and then decode Bitstream #2 based on the probability distribution model. To obtain the probability distribution model, as shown in Figure 5, the decoding side obtains the coefficient hyperparameter feature z_hat, and then performs coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat may be input to a probability hyperparameter decoding network, and the probability hyperparameter decoding network may perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, the probability distribution model may be generated based on the probability distribution parameter p. Here, the probability hyperparameter decoding network may be a trained neural network, and the training process of this probability hyperparameter decoding network is not limited, as long as it can perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p.

[0152] In one possible embodiment, the decoding side processing process may be performed by a deep learning model or a neural network model to realize an end-to-end image compression and encoding process, and there is no limitation on this decoding process.

[0153] Example 5: In Example 1, Example 2, Example 3 and Example 4, an analysis transformation network is involved in the encoding side, and a synthesis transformation network is involved in the encoding side and the decoding side, and both the analysis transformation network and the synthesis transformation network are neural networks. In one possible embodiment, the structure of the analysis transformation network can be seen in Figure 6A, and the structure of the synthesis transformation network can be seen in Figure 6B. Of course, Figures 6A and 6B are merely examples, and this embodiment does not limit the structure of the analysis transformation network and the synthesis transformation network as long as they can realize the analysis transformation function and the synthesis transformation function.

[0154] The analysis-transformation network includes, in order, a padding layer, a convolution layer, a ResAU layer, a padding layer, an RNAB layer, a convolution layer, a ResAU layer, a padding layer, a Conv layer, a ResAU layer, a padding layer, a Conv layer, a Conv layer. The padding layer is a filling layer for realizing feature expansion operations. The convolution layer is a convolution layer for realizing feature convolution operations. C*3*3 represents the use of a 3*3 convolution kernel (C channels) for feature convolution operations, C*1*1 represents the use of a 1*1 convolution kernel (C channels) for feature convolution operations, and the downward arrow of 2 represents 2x downsampling of the features.

[0155] The ResAU layer is an activation layer. The schematic structure of the ResAU layer can be seen in Figure 6C, where LeakyReLU represents the activation operation, Conv1*1 represents using a 1*1 convolution kernel to realize the feature convolution operation, and tanh represents the hyperbolic tangent operation.

[0156] The RNAB layer is a residual non-local attention block. The structure of the RNAB layer can be seen in Figure 6D. The RNAB layer may include an RB layer, a Conv layer, and a sigmoid layer. In the Conv layer, the up arrow "2" indicates that the features are upsampled by a factor of 2. The sigmoid layer performs feature mapping on the features, for example, by using a sigmoid function to map the features between 0 and 1.

[0157] The RB (Residual Blocks) layer is a residual block. The schematic structure of the RB layer can be seen in FIG. 6E. The RB layer may include a Conv layer, a LeakyReLU and a Conv layer. The Conv layer uses a 3*3 convolution kernel.

[0158] The synthesis transformation network includes, in order, a ResBlock layer (i.e., RB layer), a ResBlock layer, a Conv layer, a Cropping layer, a ResAU layer, a Conv layer, a Cropping layer, a ResAU layer, a Conv layer, an RNAB layer, a Cropping layer, a ResAU layer, a Conv layer, and a Cropping layer. Here, the ResBlock layer is the residual block. The schematic structure of the ResBlock layer can be seen in Figure 6E. The Conv layer is a convolution layer for realizing the convolution operation of the features. C*3*3 represents the use of a 3*3 convolution kernel (C channels) to realize the convolution operation of the features. The upward arrow 2 represents 2x upsampling of the features. The Cropping layer is a cropping layer for realizing the cropping operation of the features, i.e., the inverse operation of the padding operation. The ResAU layer is an activation layer. The schematic structure of the ResAU layer can be seen in Figure 6C. The RNAB layer is a residual non-local attention block. The schematic structure of the RNAB layer can be seen in Figure 6D. The RNAB layer may include an RB layer, a Conv layer, and a sigmoid layer.

[0159] Example 6: As shown in FIG. 6B for Example 5, the synthesis transformation network mainly includes four upsampling 3*3 convolution layers, two ResBlock layers (each ResBlock layer includes two 3*3 convolution layers), three activation layers, ResAU (each ResAU includes one 1*1 convolution layer), and one RNAB layer (the RNAB layer includes nine RB blocks and three 3*3 convolution layers, for a total of 21 3*3 convolution layers). As can be seen from the above, the synthesis transformation network has a total of 29 3*3 convolution layers and three 1*1 convolution layers, with the RNAB layer accounting for nearly 70% of the complexity. The complexity of the RNAB layer has considerable room for optimization. Based on this, in this example, the synthesis transformation network can be optimized to design a synthesis transformation network with low complexity and high performance. Alternatively, the analysis transformation network can be optimized to design an analysis transformation network with low complexity and high performance. Of course, other high-complexity network structures may be optimized to design a network structure with low complexity and high performance, and this embodiment is not limited thereto. For convenience of explanation, this embodiment takes the optimization of the composite transformation network as an example, and can optimize a complex network such as the RNAB layer in the composite transformation network. Of course, other network sides in the composite transformation network may be optimized, and this is not limited thereto, as long as the complexity of the composite transformation network can be simplified.

[0160] In one possible embodiment, for the RNAB layer shown in Figure 6D, the RNAB layer may include nine RB layers, three Conv layers, and one sigmoid layer. Based on this, some network layers may be removed to obtain an optimized RNAB layer, such as the three RB layers and two Conv layers shown in Figure 6F. Of course, other network layers may also be removed to obtain an optimized RNAB layer, and the manner of removing this network layer is not limited.

[0161] For example, the RNAB layer after optimization can refer to Figure 6G, and the RNAB layer may include an RB layer (composed of at least one RB block, for example, may be composed of three consecutive RB blocks, although of course the number of RB blocks may be more or less), an RB layer (composed of at least one RB block, for example, may be composed of three consecutive RB blocks, although of course the number of RB blocks may be more or less), one Conv layer (for example, a 3*3 Conv layer), and one sigmoid layer (i.e., a feature mapping layer).

[0162] For example, a 3*3 Conv layer may be replaced with a 1*1 Conv layer or a 5*5 Conv layer. A 1*1 Conv layer is taken as an example, and the optimized RNAB layer can be seen in FIG. 6H. The RNAB layer may include an RB layer, an RB layer, one Conv layer (e.g., a 1*1 Conv layer), and one sigmoid layer (i.e., a feature mapping layer).

[0163] For example, after obtaining the optimized RNAB layer, the optimized RNAB layer can be substituted into Figure 6B to obtain the optimized synthesis transformation network. Based on the optimized synthesis transformation network, the image feature y_hat is input to the synthesis transformation network, and the synthesis transformation network performs synthesis transformation on the image feature y_hat to obtain the reconstructed image block x_hat.

[0164] Example 7: In Example 5, the synthetic transformation network can be optimized to design a synthetic transformation network with low complexity and high performance. Alternatively, the analysis transformation network can be optimized to design an analysis transformation network with low complexity and high performance. Of course, other high-complexity network structures can be optimized to design a network structure with low complexity and high performance. In this example, encoding and decoding can be performed using the method of "simplified network structure + adaptive adjustment factor".

[0165] First, a complex network can be simplified to obtain a simplified network; for example, a synthetic transformation network can be simplified to obtain a simplified synthetic transformation network; for example, the RNAB layer (i.e., a complex network) in the synthetic transformation network can be simplified to obtain a simplified synthetic transformation network; of course, other networks can also be simplified, and this is not limited thereto.

[0166] Then, adjust the features (e.g., important features) of at least one layer of the simplified network. For example, the simplified network is trained based on a large number of images, and the features it generates have a general effect. However, for a specific image of a frame, the features are often not optimal. Therefore, it is necessary to design customized features for the image. To obtain these customized features, it is necessary to introduce feature adjustment factors into a process of the network, and adjust the features based on the feature adjustment factors to obtain the adjusted customized features. For example, the encoding side can adaptively select a feature adjustment factor for the current image block, and encode the feature adjustment factor into a bitstream, and the decoding side can analyze the feature adjustment factor from the bitstream.

[0167] In one possible embodiment, the decoding method of the decoding side can refer to FIG. 7A. The decoding side can receive a main bitstream (i.e., a first bitstream Bitstream #1 and a second bitstream Bitstream #2) for a current image block. After the main bitstream passes through a coefficient decoding and decoding sub-network, a first feature corresponding to the current image block is obtained. The decoding side can decode a feature adjustment factor corresponding to the current image block from the auxiliary bitstream. After obtaining the first feature corresponding to the current image block, a feature enhancement network can perform feature enhancement on the first feature to obtain a second feature, and a feature adjustment factor corresponding to the current image block can be used to perform feature adjustment on the second feature to obtain a third feature. After obtaining the third feature, a target feature corresponding to the current image block is determined based on the third feature, and the target feature is input to a reconstruction decoding network. The reconstruction decoding network processes the target feature to obtain a reconstructed image block corresponding to the current image block.

[0168] For example, the feature adjustment factor may be one decoded from the auxiliary bitstream by the decoding side, and the feature adjustment factor may be a fixed parameter value. When the feature adjustment factor is a fixed parameter value, for example, when the feature adjustment factor is a fixed parameter value of 1, it corresponds to not performing feature adjustment on the second feature, that is, it does not involve a feature adjustment process and corresponds to the method of Example 6.

[0169] For example, the decoding side may decode a feature adjustment factor for the decoding sub-network (which may be the same as or different from the feature adjustment factor for the second feature) from the auxiliary bitstream, and after obtaining the feature adjustment factor for the decoding sub-network, may perform feature adjustment on a feature in the decoding sub-network based on the feature adjustment factor. For example, the decoding sub-network may obtain feature A (feature A is any feature in the decoding sub-network) and then perform feature adjustment on feature A using the feature adjustment factor to obtain an adjusted feature B. The decoding sub-network continues processing based on feature B and finally obtains a first feature. By using the feature adjustment factor in some decoding processes of the decoding sub-network, a first feature with less distortion can be generated.

[0170] In one possible embodiment, for the encoding side, the encoding method of the encoding side can refer to Figure 7B, and after the current image block goes through the encoding network and coefficient encoding, the main bitstream corresponding to the current image block (i.e., the first bitstream Bitstream #1 and the second bitstream Bitstream #2) is obtained, and the encoding side can send the main bitstream of the current image block to the decoding side.

[0171] After the main bitstream passes through coefficient decoding and decoding sub-network, the main stream also obtains a first feature corresponding to the current image block, and may obtain a feature adjustment factor corresponding to the current image block based on the first feature, and encode the feature adjustment factor into the auxiliary bitstream corresponding to the current image block, so that the decoding side can decode the feature adjustment factor corresponding to the current image block from the auxiliary bitstream.

[0172] For example, the encoding side may obtain at least one candidate feature adjustment factor (e.g., a list of candidate feature adjustment factors may be constructed in advance, and all feature adjustment factors in the list may be candidate feature adjustment factors, or at least one candidate feature adjustment factor may be generated using an algorithm, without limitation). A rate-distortion cost corresponding to each candidate feature adjustment factor may be determined based on a first feature. For example, for each candidate feature adjustment factor, a first feature corresponding to the current image block may be obtained, and then a feature enhancement network may perform feature enhancement on the first feature to obtain a second feature, and then the candidate feature adjustment factor may be used to perform feature adjustment on the second feature to obtain a third feature. After the third feature is obtained, a target feature corresponding to the current image block may be determined based on the third feature, and the target feature may be input to a reconstruction decoding network, which processes the target feature to obtain a reconstructed image block corresponding to the current image block. After the reconstructed image block is obtained, a rate-distortion cost corresponding to the candidate feature adjustment factor may be determined based on the loss between the reconstructed image block and the current image block, thereby obtaining a rate-distortion cost corresponding to each candidate feature adjustment factor. After obtaining the rate-distortion cost corresponding to each candidate feature adjustment factor, select one candidate feature adjustment factor from all the candidate feature adjustment factors as the feature adjustment factor corresponding to the current image block based on the rate-distortion cost corresponding to each candidate feature adjustment factor, for example, select the candidate feature adjustment factor with the smallest rate-distortion cost as the feature adjustment factor corresponding to the current image block.

[0173] For example, the encoding side may obtain a feature adjustment factor for a decoding sub-network and perform feature adjustment on a feature in the decoding sub-network based on the feature adjustment factor. For example, the decoding sub-network may obtain feature A (feature A is any feature in the decoding sub-network) and then perform feature adjustment on feature A using the feature adjustment factor to obtain adjusted feature B. The decoding sub-network continues processing based on feature B and finally obtains a first feature. Based on this, the encoding side may encode the feature adjustment factor for the decoding sub-network (which may be the same as or different from the feature adjustment factor for the second feature) into the auxiliary bitstream so that the decoding side can decode the feature adjustment factor for the decoding sub-network from the auxiliary bitstream.

[0174] Example 8: For Examples 1 to 7, on the encoding side and the decoding side, the first feature corresponding to the current image block can be obtained by the first neural network, for example, by obtaining a probability distribution parameter based on the first bit stream corresponding to the current image block, determining a probability distribution model based on the probability distribution parameter, decoding the second bit stream corresponding to the current image block based on the probability distribution model to obtain the decoded image feature, and determining the first feature corresponding to the current image block based on the decoded image feature.

[0175] For example, the first neural network may be a decoding sub-network, and the decoding sub-network will be described as an example. As shown in FIG. 8A, the network within the dashed frame is the decoding sub-network, and y_hat is the first feature output by the decoding sub-network.

[0176] For example, after obtaining Bitstream#1 corresponding to the current image block, the encoding side or the decoding side may decode Bitstream#1 to obtain hyperparameter quantization features and dequantize the hyperparameter quantization features to obtain coefficient hyperparameter features z_hat. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the encoding side or the decoding side may decode Bitstream#1 to obtain coefficient hyperparameter features z_hat without performing the dequantization process of the coefficient hyperparameter features z_hat.

[0177] After obtaining the coefficient hyperparameter feature z_hat, the encoding side or the decoding side may perform a coefficient hyperparameter feature inverse transform on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p, for example, by inputting the coefficient hyperparameter feature z_hat to a probability hyperparameter decoding network and using the probability hyperparameter decoding network to perform a coefficient hyperparameter feature inverse transform on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model may be generated based on the probability distribution parameter p.

[0178] After obtaining Bitstream#2 corresponding to the current image block, the encoding side or the decoding side may decode Bitstream#2 to obtain image quantization features, and then dequantize the image quantization features to obtain image feature s'. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the encoding side or the decoding side may decode Bitstream#2 to obtain image feature s' without performing the dequantization process of the image quantization features. Illustratively, when decoding Bitstream#2, the encoding side or the decoding side may decode Bitstream#2 based on a probability distribution model.

[0179] After obtaining the image feature s', the encoding side or the decoding side may perform feature recovery on the image feature s' to obtain the image feature y_hat, and the image feature y_hat can be the first feature, that is, the decoding sub-network outputs the first feature y_hat.

[0180] Example 9: For Examples 1 to 7, on the encoding side and the decoding side, a first feature corresponding to a current image block can be obtained by a first neural network, for example, by obtaining a probability distribution parameter and a predicted value (e.g., an average value) based on a first bit stream corresponding to the current image block, determining a probability distribution model based on the probability distribution parameter, decoding a second bit stream corresponding to the current image block based on the probability distribution model to obtain decoded image features, performing residual reconstruction on the decoded image features to obtain residual features, and determining a first feature corresponding to the current image block based on the residual features and the predicted value.

[0181] For example, the first neural network may be a decoding sub-network, and the decoding sub-network will be described as an example. As shown in FIG. 8B, the network within the dashed frame is the decoding sub-network, and y_hat is the first feature output by the decoding sub-network.

[0182] For example, after obtaining Bitstream#1 corresponding to the current image block, the encoding side or the decoding side may decode Bitstream#1 to obtain hyperparameter quantization features and dequantize the hyperparameter quantization features to obtain coefficient hyperparameter features z_hat. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the encoding side or the decoding side may decode Bitstream#1 to obtain coefficient hyperparameter features z_hat without performing the dequantization process of the coefficient hyperparameter features z_hat.

[0183] After obtaining the coefficient hyperparameter feature z_hat, the encoding side or the decoding side may perform a coefficient hyperparameter feature inverse transform on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p, for example, by inputting the coefficient hyperparameter feature z_hat to a probability hyperparameter decoding network and using the probability hyperparameter decoding network to perform a coefficient hyperparameter feature inverse transform on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model may be generated based on the probability distribution parameter p.

[0184] After obtaining the coefficient hyperparameter feature z_hat, the encoding side or decoding side may perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block to obtain a predicted value mu (i.e., average value mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat are input to a mean value prediction network, and the mean value prediction network determines the predicted value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. This prediction process is not limited. Here, in the context-based prediction process, the input of the mean value prediction network may include the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat, and the two are combined and input to obtain a more accurate predicted value mu.

[0185] After obtaining Bitstream#2 corresponding to the current image block, the encoding side or the decoding side may decode Bitstream#2 to obtain image quantization features, and then dequantize the image quantization features to obtain image feature s'. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the encoding side or the decoding side may decode Bitstream#2 to obtain image feature s' without performing the dequantization process of the image quantization features. Illustratively, when decoding Bitstream#2, the encoding side or the decoding side may decode Bitstream#2 based on a probability distribution model.

[0186] After obtaining the image feature s', the encoding side or the decoding side performs feature reconstruction (i.e., residual reconstruction, the inverse process of residual processing) on ​​the image feature s' to obtain the residual feature r_hat. After obtaining the residual feature r_hat, the encoding side or the decoding side determines the image feature y_hat based on the residual feature r_hat and the predicted value mu. For example, the sum of the residual feature r_hat and the predicted value mu can be set as the image feature y_hat, and the image feature y_hat can be set as the first feature, that is, the decoding sub-network outputs the first feature y_hat.

[0187] 8B and 8A, the decoding sub-network may not only obtain a probability distribution parameter for the second bitstream based on the first bitstream, but also generate a predicted value mu for the first feature based on the first bitstream, perform decoding based on the second bitstream to obtain a feature residual, perform residual reconstruction to obtain a residual r_hat of the first feature, and then obtain the first feature based on mu and r_hat.

[0188] Example 10: For Examples 1 to 9, on the encoding side and the decoding side, after obtaining a first feature y_hat, a synthesis transformation network can be used to perform synthesis transformation on the first feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. In the synthesis transformation process, feature enhancement can be performed on the first feature y_hat to obtain a second feature, and feature adjustment can be performed on the second feature based on a feature adjustment factor to obtain a third feature. After obtaining the third feature, a target feature can be determined based on the third feature. Next, a reconstructed image block x_hat corresponding to the current image block x is obtained based on the target feature.

[0189] Figure 6B is a schematic diagram of the structure of a synthesis transform network, which may include, in order, a ResBlock layer (i.e., an RB layer), a ResBlock layer, a Conv layer, a Cropping layer, a ResAU layer, a Conv layer, a Cropping layer, a ResAU layer, a Conv layer, an RNAB layer, a Cropping layer, a ResAU layer, a Conv layer, and a Cropping layer. A schematic diagram of the structure of the RB layer can be referred to in Figure 6E, and a schematic diagram of the structure of the ResAU layer can be referred to in Figure 6C.

[0190] The RNAB layer is a residual non-local attention block. A schematic diagram of the structure of the RNAB layer can be seen in FIG. 6D. Alternatively, some network layers in FIG. 6D can be removed to obtain the optimized RNAB layer. As shown in FIG. 6F, three RB layers and two Conv layers can be removed to obtain the optimized RNAB layer. Therefore, a schematic diagram of the structure of the RNAB layer can be seen in FIG. 6G. Alternatively, a schematic diagram of the structure of the RNAB layer can be seen in FIG. 6H. In FIGS. 6G and 6H, the RNAB layer can include an RB layer (composed of at least one RB block, e.g., three consecutive RB blocks), an RB layer (composed of at least one RB block, e.g., three consecutive RB blocks), one Conv layer, and one sigmoid layer (i.e., a feature mapping layer). The convolution layer may be a 3*3 convolution layer, a 5*5 convolution layer, a 1*1 convolution layer, or a convolution layer of other sizes such as a 7*7 convolution layer or a 9*9 convolution layer, and there is no limitation on the size of the convolution layer.

[0191] The synthesis transform network may be divided into a feature enhancement subnetwork and a reconstruction decoding subnetwork. The feature enhancement subnetwork may include an initial enhancement subnetwork, a first upsampling convolution subnetwork, and an attention subnetwork. The reconstruction decoding subnetwork may include a second upsampling convolution subnetwork and a color space conversion subnetwork.

[0192] For example, all networks before the RNAB layer in the synthesis transform network can be an initial enhancement sub-network and a first upsampling convolution sub-network, where the initial enhancement sub-network may include two RB networks, and the first upsampling convolution sub-network may include at least one upsampling convolution network (e.g., three upsampling convolution networks). For example, as shown in Figure 6B, all networks before the RNAB layer are an RB layer, an RB layer, a Conv layer, a Cropping layer, a ResAU layer, a Conv layer, a Cropping layer, a ResAU layer, and a Conv layer, in order. Based on this, the initial enhancement sub-network may include an RB layer, an RB layer, and the first upsampling convolution sub-network may include a Conv layer, a Cropping layer, a ResAU layer, a Conv layer, a Cropping layer, a ResAU layer, and a Conv layer, that is, three upsampling convolution networks (Conv layers), where a 3*3 Conv layer is taken as an example.

[0193] For example, the RNAB layer in the synthesis transformation network can be an attention sub-network (i.e., an attention sub-network), which can effectively reduce complexity by eliminating the up-down sampling convolution network. Figures 6G and 6H are schematic diagrams of the structure of the attention sub-network, which may include an RB layer (composed of at least one RB block), an RB layer (composed of at least one RB block), one Conv layer, and one sigmoid layer (i.e., a feature mapping layer).

[0194] The attention subnetwork may include a residual skip subnetwork, a residual emphasis subnetwork, and a weight generation subnetwork. For the residual skip subnetwork, x_out0 = x_in, where x_in is the input feature of the residual skip subnetwork and x_out0 is the output feature of the residual skip subnetwork. For the residual emphasis subnetwork, x_out1 = RB(x_in), where x_in is the input feature of the residual emphasis subnetwork and x_out1 is the output feature of the residual emphasis subnetwork. For the weight generation subnetwork, the output feature x_out1 of the residual emphasis subnetwork is used to obtain weight feature k. Therefore, the final output of the attention subnetwork is x_out = x_in + k * x_out1.

[0195] 9A and 9B, the residual skip sub-network is used to add the input feature x_in (i.e., x_out0) to the final output feature of the attention sub-network. The residual emphasis sub-network includes at least one RB block (e.g., one RB block or three RB blocks), and the residual emphasis sub-network is used to process the input feature x_in based on the RB block to obtain the output feature x_out1. The weight generation sub-network may include at least one RB block (e.g., one RB block or three RB blocks), one Conv layer (e.g., a 3*3 Conv layer, a 5*5 Conv layer, a 1*1 Conv layer, etc.), and a feature mapping layer (e.g., feature mapping is realized using a sigmoid processing process). The weight generation sub-network processes input features x_in based on the RB block, performs convolution processing on the features processed by the RB block using the Conv layer, and performs feature mapping on the features processed by the Conv layer using the feature mapping layer to obtain weight features k, the dimension of which is the same as the dimension of the output feature x_out1 of the residual emphasis sub-network. After multiplying the output feature x_out1 by the weight feature k, it is added with the output feature x_out0 of the residual skip sub-network. The final output feature of the attention sub-network obtained is x_out=x_in+k*x_out1.

[0196] For example, all networks after the RNAB layer in the synthesis transformation network can be a reconstruction decoding subnetwork, and the reconstruction decoding subnetwork may include a second upsampling convolution subnetwork and a color space conversion subnetwork, that is, all networks after the RNAB layer are a second upsampling convolution subnetwork and a color space conversion subnetwork. The second upsampling convolution subnetwork may include at least one upsampling convolution network (e.g., one upsampling convolution network), and the color space conversion subnetwork is used to realize an image conversion process from the YUV domain to RGBG, and may be a filtering process from the YUV domain to the YUV domain.

[0197] For example, as shown in FIG. 6B, all networks after the RNAB layer are, in order, a Cropping layer, a ResAU layer, a Conv layer, and a Cropping layer. Therefore, the second upsampling convolutional sub-network may include a Cropping layer, a ResAU layer, a Conv layer, and a Cropping layer, i.e., one upsampling convolutional network (Conv layer), taking a 3*3 Conv layer as an example. If there is a color space conversion need, a color space conversion sub-network (not shown in FIG. 6B) may be included to perform the image conversion process from the YUV domain to RGB, or the filtering process from the YUV domain to the YUV domain. If there is no color space conversion need, the color space conversion sub-network may not be included.

[0198] After dividing the synthesis transformation network into an initial enhancement subnetwork, a first upsampling convolution subnetwork, an attention subnetwork and a reconstruction decoding subnetwork (the reconstruction decoding subnetwork includes a second upsampling convolution subnetwork and a color space transformation subnetwork), a synthesis transformation is performed on the first feature y_hat based on the initial enhancement subnetwork, the first upsampling convolution subnetwork, the attention subnetwork and the reconstruction decoding subnetwork to obtain a reconstructed image block x_hat corresponding to the current image block x.

[0199] Example 11: In Example 10, a synthesis transformation can be performed on the first feature y_hat based on the initial enhancement sub-network, the first upsampling convolution sub-network, the attention sub-network and the reconstruction decoding sub-network to obtain a reconstructed image block x_hat corresponding to the current image block x. In one possible embodiment, a feature adjustment factor can be used to adjust the output feature of the feature enhancement sub-network. The adjustment position of the feature adjustment factor can be referred to Figure 10A. In the adjustment method of the feature adjustment factor, a synthesis transformation can be performed on the first feature y_hat using the following method to obtain a reconstructed image block x_hat:

[0200] The initial enhancement sub-network is used to perform feature enhancement on the first feature y_hat to obtain an enhanced feature. For example, the first feature y_hat is input to the initial enhancement sub-network, and the initial enhancement sub-network performs feature enhancement on the first feature y_hat to obtain an enhanced feature. For example, the initial enhancement sub-network may include an RB layer and an RB layer, so that the first feature y_hat can be enhanced by the two RB layers to obtain an enhanced feature; this process is not limited.

[0201] After obtaining the enhanced features, a first upsampling convolutional sub-network is used to perform upsampling convolution on the enhanced features to obtain initial features corresponding to the attention sub-network, i.e., input features x_in corresponding to the attention sub-network. For example, the enhanced features (i.e., output features of the initial enhancement sub-network) are input to the first upsampling convolutional sub-network, and the first upsampling convolutional sub-network performs upsampling convolution on the enhanced features to obtain initial features corresponding to the attention sub-network. For example, the first upsampling convolutional sub-network may include a Conv layer, a Cropping layer, a ResAU layer, a Conv layer, a Cropping layer, a ResAU layer, and a Conv layer, and upsampling convolution on the enhanced features may be performed by these network layers, and this process is not limited.

[0202] After obtaining the initial feature x_in corresponding to the attention sub-network, the attention sub-network may perform feature enhancement on the initial feature x_in to obtain a second feature. For example, the initial feature x_in may be input to the residual skip sub-network, the residual emphasis sub-network, and the weight generation sub-network, respectively. After obtaining the initial feature x_in, the residual skip sub-network adds the initial feature x_in (i.e., x_out0) to the final output feature of the attention sub-network. After obtaining the initial feature x_in, the residual emphasis sub-network performs feature enhancement on the initial feature x_in to obtain an enhanced feature. For example, the residual emphasis sub-network performs feature enhancement on the initial feature x_in based on the RB block to obtain an enhanced feature x_out1. After obtaining the initial feature x_in, the weight generation sub-network generates a weight feature k corresponding to the initial feature x_in, i.e., the weight feature k of the output feature x_out1 of the residual emphasis sub-network.

[0203] For example, the weight generation subnetwork may include a residual block subnetwork, a convolutional subnetwork, and a feature mapping subnetwork, and may use the residual block subnetwork to perform convolutional activation on the initial feature x_in to obtain a convolutional activation feature, the convolutional subnetwork to perform convolution on the convolutional activation feature to obtain a convolutional feature, and the feature mapping subnetwork to perform feature mapping on the convolutional feature to obtain a weight feature k. As shown in FIG. 10A , the weight generation sub-network may include at least one RB block (e.g., one RB block or three RB blocks, etc., i.e., a residual block sub-network), one Conv layer (e.g., a 3*3 Conv layer, a 5*5 Conv layer, a 1*1 Conv layer, etc., i.e., a convolutional sub-network), and a feature mapping layer (e.g., feature mapping is realized using a sigmoid processing process, i.e., a feature mapping sub-network). Based on this, convolutional activation may be performed on the initial feature x_in based on the RB block to obtain a convolutional activation feature, and then convolution may be performed on the convolutional activation feature using the Conv layer to obtain a convolutional feature, and then feature mapping may be performed on the convolutional feature using the feature mapping layer to obtain a weight feature k.

[0204] 10A, after obtaining the enhanced feature x_out1, the weight feature k, and the initial feature x_in (i.e., x_out0), a second feature can be generated based on the initial feature x_in, the enhanced feature x_out1, and the weight feature k, and the second feature can be the final output feature of the attention sub-network. For example, the enhanced feature x_out1 is multiplied by the weight feature k, and then the output feature x_out0 of the residual skip sub-network is added, that is, the second feature is x_out=x_in+k*x_out1.

[0205] After the second feature is obtained, the second feature may be adjusted based on the feature adjustment factor, and the adjusted feature may be set as the third feature. After the third feature is obtained, the third feature may be determined as the target feature.

[0206] After obtaining the target features, the target features may be processed by a reconstruction decoding subnetwork to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the target features may be input to the reconstruction decoding subnetwork, and the target features may be processed by the reconstruction decoding subnetwork to obtain a reconstructed image block x_hat corresponding to the current image block x. Exemplarily, the reconstruction decoding subnetwork may include a second upsampling convolution subnetwork, which may perform upsampling convolution on the target features to obtain a reconstructed image block x_hat corresponding to the current image block x. Alternatively, the reconstruction decoding subnetwork may include a second upsampling convolution subnetwork and a color space conversion subnetwork, which may perform upsampling convolution on the target features by the second upsampling convolution subnetwork, and perform color space conversion on the features after the upsampling convolution by the color space conversion subnetwork to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the second upsampling convolution sub-network may include a Cropping layer, a ResAU layer, a Conv layer, and a Cropping layer, and these network layers may perform upsampling convolution on the target feature, and this process is not limited. The color space conversion sub-network is used to perform an image conversion process from the YUV domain to RGB, or the color space conversion sub-network is used to perform a filtering process from the YUV domain to the YUV domain, and this process is not limited.

[0207] Example 12: In Example 10, a synthesis transformation can be performed on the first feature y_hat based on the initial enhancement sub-network, the first upsampling convolution sub-network, the attention sub-network and the reconstruction decoding sub-network to obtain a reconstructed image block x_hat corresponding to the current image block x. In one possible embodiment, a feature adjustment factor can be used to adjust the residual enhancement output feature of the attention sub-network. The adjustment position of the feature adjustment factor can be referred to Figure 10B. In the adjustment method of the feature adjustment factor, a synthesis transformation can be performed on the first feature y_hat using the following method to obtain a reconstructed image block x_hat.

[0208] The initial enhancement sub-network is used to perform feature enhancement on the first feature y_hat to obtain an enhanced feature. For example, the first feature y_hat is input to the initial enhancement sub-network, and the initial enhancement sub-network performs feature enhancement on the first feature y_hat to obtain an enhanced feature. For example, the initial enhancement sub-network may include an RB layer and an RB layer, so that the first feature y_hat can be enhanced by the two RB layers to obtain an enhanced feature; this process is not limited.

[0209] After obtaining the enhanced features, a first upsampling convolutional sub-network is used to perform upsampling convolution on the enhanced features to obtain initial features corresponding to the attention sub-network, i.e., input features x_in corresponding to the attention sub-network. For example, the enhanced features (i.e., output features of the initial enhancement sub-network) are input to the first upsampling convolutional sub-network, and the first upsampling convolutional sub-network performs upsampling convolution on the enhanced features to obtain initial features corresponding to the attention sub-network. For example, the first upsampling convolutional sub-network may include a Conv layer, a Cropping layer, a ResAU layer, a Conv layer, a Cropping layer, a ResAU layer, and a Conv layer, and upsampling convolution on the enhanced features may be performed by these network layers, and this process is not limited.

[0210] After obtaining the initial feature x_in corresponding to the attention sub-network, the initial feature x_in is input to the residual skip sub-network, the residual emphasis sub-network, and the weight generation sub-network, respectively. After obtaining the initial feature x_in, the residual skip sub-network adds the initial feature x_in (i.e., x_out0) to the final output feature of the attention sub-network. After obtaining the initial feature x_in, the residual emphasis sub-network performs feature enhancement on the initial feature x_in to obtain an enhanced feature. For example, the residual emphasis sub-network performs feature enhancement on the initial feature x_in based on the RB block to obtain an enhanced feature x_out1. After obtaining the initial feature x_in, the weight generation sub-network generates a weight feature k corresponding to the initial feature x_in, i.e., the weight feature k of the output feature x_out1 of the residual emphasis sub-network.

[0211] For example, the weight generation subnetwork may include a residual block subnetwork, a convolutional subnetwork, and a feature mapping subnetwork, where the residual block subnetwork is used to perform convolutional activation on the initial feature x_in to obtain a convolutional activation feature, the convolutional subnetwork is used to perform convolution on the convolutional activation feature to obtain a convolutional feature, and the feature mapping subnetwork is used to perform feature mapping on the convolutional feature to obtain a weight feature k. As shown in FIG. 10B , the weight generation subnetwork may include at least one RB block (i.e., residual block subnetwork), one Convolution layer (i.e., convolutional subnetwork), and a feature mapping layer (i.e., feature mapping subnetwork), where the weight generation subnetwork is used to perform convolutional activation on the initial feature x_in based on the RB block to obtain a convolutional activation feature, the Convolution layer is used to perform convolution on the convolutional activation feature to obtain a convolutional feature, and the feature mapping layer is used to perform feature mapping on the convolutional feature to obtain a weight feature k.

[0212] As shown in FIG. 10B , after obtaining the enhanced feature x_out1 and the weight feature k, a second feature can be generated based on the enhanced feature x_out1 and the weight feature k. For example, the enhanced feature x_out1 can be multiplied by the weight feature k, and the multiplied feature can be the second feature, that is, the second feature can be k*x_out1.

[0213] After obtaining the second feature, the second feature may be subjected to feature adjustment based on the feature adjustment factor, and the feature after the feature adjustment may be the third feature, which may be feature rd as shown in FIG. 10B.

[0214] After obtaining the third feature rd, a fourth feature can be generated based on the initial feature x_in (i.e., the output feature x_out0 of the residual skip sub-network) and the third feature rd, and the fourth feature can be the final output feature of the attention sub-network. For example, by adding the output feature x_out0 of the residual skip sub-network to the third feature rd, the fourth feature obtained is x_out=x_in+rd.

[0215] As can be seen from the above, a first subnetwork in the attention subnetwork (e.g., a residual emphasis subnetwork and a weight generation subnetwork, where the weight generation subnetwork may include a residual block subnetwork, a convolution subnetwork, and a feature mapping subnetwork) can perform feature emphasis on the initial feature x_in to obtain a second feature. After obtaining the second feature, a feature adjustment can be performed on the second feature based on the feature adjustment factor to obtain a third feature rd. After obtaining the third feature rd, a second subnetwork in the attention subnetwork (e.g., a residual skip subnetwork) can process the third feature rd to obtain a fourth feature x_out. After obtaining the fourth feature x_out, the fourth feature x_out can be determined as the target feature.

[0216] After obtaining the target features, the target features may be processed by a reconstruction decoding subnetwork to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the target features may be input to the reconstruction decoding subnetwork, and the target features may be processed by the reconstruction decoding subnetwork to obtain a reconstructed image block x_hat corresponding to the current image block x. Exemplarily, the reconstruction decoding subnetwork may include a second upsampling convolution subnetwork, which may perform upsampling convolution on the target features to obtain a reconstructed image block x_hat corresponding to the current image block x. Alternatively, the reconstruction decoding subnetwork may include a second upsampling convolution subnetwork and a color space conversion subnetwork, which may perform upsampling convolution on the target features by the second upsampling convolution subnetwork, and perform color space conversion on the features after the upsampling convolution by the color space conversion subnetwork to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the second upsampling convolution sub-network may include a Cropping layer, a ResAU layer, a Conv layer, and a Cropping layer, and these network layers may perform upsampling convolution on the target feature, and this process is not limited. The color space conversion sub-network is used to perform an image conversion process from the YUV domain to RGB, or the color space conversion sub-network is used to perform a filtering process from the YUV domain to the YUV domain, and this process is not limited.

[0217] Example 13: In Example 10, a synthesis transformation can be performed on the first feature y_hat based on the initial enhancement sub-network, the first upsampling convolution sub-network, the attention sub-network and the reconstruction decoding sub-network to obtain a reconstructed image block x_hat corresponding to the current image block x. In one possible embodiment, a feature adjustment factor can be used to adjust the weight output feature of the attention sub-network. The adjustment position of the feature adjustment factor can be referred to Figure 10C. In the adjustment method of the feature adjustment factor, a synthesis transformation can be performed on the first feature y_hat using the following method to obtain a reconstructed image block x_hat:

[0218] The initial enhancement sub-network is used to perform feature enhancement on the first feature y_hat to obtain an enhanced feature. For example, the first feature y_hat is input to the initial enhancement sub-network, and the initial enhancement sub-network performs feature enhancement on the first feature y_hat to obtain an enhanced feature. For example, the initial enhancement sub-network may include an RB layer and an RB layer, so that the first feature y_hat can be enhanced by the two RB layers to obtain an enhanced feature; this process is not limited.

[0219] After obtaining the enhanced features, a first upsampling convolutional sub-network is used to perform upsampling convolution on the enhanced features to obtain initial features corresponding to the attention sub-network, i.e., input features x_in corresponding to the attention sub-network. For example, the enhanced features (i.e., output features of the initial enhancement sub-network) are input to the first upsampling convolutional sub-network, and the first upsampling convolutional sub-network performs upsampling convolution on the enhanced features to obtain initial features corresponding to the attention sub-network. For example, the first upsampling convolutional sub-network may include a Conv layer, a Cropping layer, a ResAU layer, a Conv layer, a Cropping layer, a ResAU layer, and a Conv layer, and upsampling convolution on the enhanced features may be performed by these network layers, and this process is not limited.

[0220] After obtaining the initial feature x_in corresponding to the attention sub-network, the initial feature x_in is input to the residual skip sub-network, the residual emphasis sub-network, and the weight generation sub-network, respectively. Exemplarily, after obtaining the initial feature x_in, the residual skip sub-network adds the initial feature x_in (i.e., x_out0) to the final output feature of the attention sub-network. After obtaining the initial feature x_in, the residual emphasis sub-network performs feature emphasis on the initial feature x_in to obtain an enhanced feature. For example, the residual emphasis sub-network performs feature emphasis on the initial feature x_in based on the RB block to obtain an enhanced feature x_out1.

[0221] The weight generation subnetwork may include a residual block subnetwork, a convolutional subnetwork, and a feature mapping subnetwork. As shown in FIG. 10C , the weight generation subnetwork may include at least one RB block (i.e., residual block subnetwork), one Conv layer (i.e., convolutional subnetwork), and a feature mapping layer (i.e., feature mapping subnetwork). Based on this, the initial features x_in may be input to the residual block subnetwork, and the residual block subnetwork may perform convolutional activation on the initial features x_in to obtain post-convolutional activation features. For example, the residual block subnetwork may perform convolutional activation on the initial features x_in based on the RB block to obtain post-convolutional activation features.

[0222] Then, a convolutional sub-network may be used to perform convolution on the convolutional activation features to obtain convolutional features, for example, the convolutional sub-network may use a Conv layer to perform convolution on the convolutional activation features to obtain convolutional features. After obtaining the convolutional features, a second feature may be determined based on the convolutional features, for example, the convolutional features may be the second features.

[0223] As described above, the residual block sub-network is used to perform feature enhancement on the initial feature x_in to obtain an enhanced feature, and the convolution sub-network is used to perform convolution on the enhanced feature to obtain a convolved feature, i.e., a second feature. After obtaining the second feature, feature adjustment is performed on the second feature based on the feature adjustment factor, and the feature after feature adjustment can be used as a third feature.

[0224] As shown in FIG. 10C , after obtaining the third feature, a weight feature k can be obtained by performing feature mapping on the third feature using a feature mapping sub-network. After obtaining the enhanced feature x_out1, the weight feature k, and the initial feature x_in (i.e., x_out0), a fourth feature can be generated based on the initial feature x_in, the enhanced feature x_out1, and the weight feature k, and the fourth feature can be the final output feature of the attention sub-network. For example, the enhanced feature x_out1 is multiplied by the weight feature k, and then the output feature x_out0 of the residual skip sub-network is added, i.e., the fourth feature is x_out=x_in+k*x_out1.

[0225] As can be seen from the above, a first subnetwork in the attention subnetwork (e.g., the residual block subnetwork and the convolution subnetwork in the weight generation subnetwork) can perform feature enhancement on the initial feature x_in to obtain a second feature. After obtaining the second feature, feature adjustment can be performed on the second feature based on the feature adjustment factor to obtain a third feature. After obtaining the third feature, a second subnetwork in the attention subnetwork (e.g., the residual skip subnetwork, the residual enhancement subnetwork, and the feature mapping subnetwork in the weight generation subnetwork) can process the third feature to obtain a fourth feature x_out. For example, the feature mapping subnetwork can be used to perform feature mapping on the third feature to obtain a weight feature k, and the residual enhancement subnetwork can be used to perform feature enhancement on the initial feature x_in to obtain an enhanced feature x_out1. The fourth feature x_out can then be generated based on the initial feature x_in, the enhanced feature x_out1, and the weight feature k. After obtaining the fourth feature x_out, the fourth feature x_out can be determined as the target feature.

[0226] After obtaining the target features, the target features may be processed by a reconstruction decoding subnetwork to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the target features may be input to the reconstruction decoding subnetwork, and the target features may be processed by the reconstruction decoding subnetwork to obtain a reconstructed image block x_hat corresponding to the current image block x. Exemplarily, the reconstruction decoding subnetwork may include a second upsampling convolution subnetwork, which may perform upsampling convolution on the target features to obtain a reconstructed image block x_hat corresponding to the current image block x. Alternatively, the reconstruction decoding subnetwork may include a second upsampling convolution subnetwork and a color space conversion subnetwork, which may perform upsampling convolution on the target features by the second upsampling convolution subnetwork, and perform color space conversion on the features after the upsampling convolution by the color space conversion subnetwork to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the second upsampling convolution sub-network may include a Cropping layer, a ResAU layer, a Conv layer, and a Cropping layer, and these network layers may perform upsampling convolution on the target feature, and this process is not limited. The color space conversion sub-network is used to perform an image conversion process from the YUV domain to RGB, or the color space conversion sub-network is used to perform a filtering process from the YUV domain to the YUV domain, and this process is not limited.

[0227] Example 14: For Examples 1 to 13, on the encoding side and decoding side, a third feature can be obtained by performing feature adjustment on the second feature based on a feature adjustment factor. For example, if the second feature includes C*H*W feature values, where C represents the number of channels, H represents the feature height, and W represents the feature width, a feature adjustment value corresponding to each feature value can be determined based on the feature adjustment factor. For each feature value, feature adjustment is performed on the feature value based on the corresponding feature adjustment value to obtain an adjusted feature value corresponding to the feature value. A third feature is generated based on the adjusted feature value corresponding to each feature value.

[0228] For example, if the second feature includes C*H*W feature values, a feature adjustment value corresponding to each feature value of the second feature can be determined based on the feature adjustment factor, and in this process, the following cases may occur:

[0229] Case 1: The dimension of the feature adjustment factor is the same as the dimension of the second feature, that is, each feature value of the second feature has a different feature adjustment value. For example, if the feature adjustment factor includes C*H*W feature adjustment values, the feature adjustment values ​​corresponding to the C*H*W feature values ​​may be determined based on the C*H*W feature adjustment values.

[0230] For example, for a feature value whose position in the cth channel in the second feature is h*w, the feature adjustment value corresponding to the feature value may be the feature adjustment value whose position in the cth channel in the feature adjustment factor is h*w.

[0231] Case 2: The W and H dimensions of the feature adjustment factor are the same as the dimensions of the second feature, but the C dimension of the feature adjustment factor is 1, i.e., feature values ​​at the same position in each channel of the second feature correspond to the same feature adjustment value. For example, if the feature adjustment factor includes H*W feature adjustment values, feature adjustment values ​​corresponding to the C*H*W feature values ​​may be determined based on the H*W feature adjustment values ​​(i.e., the C dimension of the feature adjustment factor is 1).

[0232] For example, for a feature value whose position in the cth channel in the second feature (i.e., each channel in the second feature) is h*w, the feature adjustment value corresponding to the feature value may be the feature adjustment value whose position in the feature adjustment factor is h*w.

[0233] Case 3: The C dimension of the feature adjustment factor is the same as the C dimension of the second feature, but the W and H dimensions of the feature adjustment factor are both 1, i.e., W*H feature values ​​of the same channel of the second feature correspond to the same feature adjustment value. For example, if the feature adjustment factor includes C feature adjustment values, feature adjustment values ​​corresponding to C*H*W feature values ​​may be determined based on the C feature adjustment values ​​(i.e., the W and H dimensions of the feature adjustment factor are both 1).

[0234] For example, for all feature values ​​of the cth channel in the second feature (ie, feature values ​​at any position), the feature adjustment value corresponding to the feature value may be the feature adjustment value of the cth channel in the feature adjustment factor.

[0235] For example, after obtaining a feature adjustment value corresponding to each feature value, for each feature value, feature adjustment can be performed on the feature value based on the feature adjustment value corresponding to the feature value to obtain an adjusted feature value corresponding to the feature value.

[0236] For example, if the feature value corresponds to only one feature adjustment value, the adjusted feature value corresponding to the feature value may be determined by the formula yr=rf*y, where yr represents the adjusted feature value, rf represents the feature adjustment value, and y represents the feature value. For example, for a feature value y(c,w,h) whose spatial domain position of a certain c-th channel is (w,h), after determining the feature adjustment value rf(c,w,h) corresponding to the feature value y(c,w,h), the adjusted feature value yr(c,w,h) corresponding to the feature value y(c,w,h) is yr(c,w,h)=rf(c,w,h)*y(c,w,h).

[0237] If the feature value corresponds to N+1 feature adjustment values, where N is a positive integer, i.e., corresponds to at least two feature adjustment values, then the formula yr=rf_0*y 0 +rf_1*y 1 +rf_2*y 2 +…+rf_N*y N where yr represents the adjusted feature value, rf_0, rf_1, rf_2, ..., rf_N represent N+1 feature adjustment values, and y represents the feature value. For example, for a feature value y(c,w,h) whose spatial domain position is (w,h) of a certain c-th channel, after determining the feature adjustment value rf_i(c,w,h), i=0,1,2...N corresponding to the feature value y(c,w,h), the adjusted feature value yr(c,w,h) corresponding to the feature value y(c,w,h) can be calculated as follows: yr(c,w,h)=rf_0(c,w,h)*y 0 (c,w,h)+rf_1(c,w,h)*y 1 (c,w,h)+rf_2(c,w,h)*y 2 (c,w,h)+…+rf_N(c,w,h)*y N (c,w,h). If N is 1, then yr(c,w,h)=rf_0(c,w,h)+rf_1(c,w,h)*y(c,w,h). If N is 2, then yr(c,w,h)=rf_0(c,w,h)+rf_1(c,w,h)*y(c,w,h)+rf_2(c,w,h)*y 2 (c, w, h). Similarly, when N is other values, the format is similar and will not be explained here. In the above formula, y k (c,w,h) represents the kth power of y.

[0238] For example, after obtaining the adjusted feature values ​​corresponding to each feature value, a third feature may be generated based on the adjusted feature values ​​corresponding to each feature value, and the third feature may include the adjusted feature values ​​corresponding to each feature value.

[0239] Example 15: In Examples 1 to 14, the encoding side further needs to determine whether the feature adjustment mode is on for the current image block. If the feature adjustment mode is on for the current image block, obtain a feature adjustment factor corresponding to the current image block and encode the feature adjustment factor into the auxiliary bitstream corresponding to the current image block. If the feature adjustment mode is not on for the current image block, there is no need to obtain a feature adjustment factor corresponding to the current image block, and there is no need to encode the feature adjustment factor into the auxiliary bitstream corresponding to the current image block. On the decoding side, there is further needs to determine whether the feature adjustment mode is on for the current image block. If the feature adjustment mode is on for the current image block, decode a feature adjustment factor corresponding to the current image block from the auxiliary bitstream corresponding to the current image block, and determine a target feature based on the first feature and the feature adjustment factor. If the feature adjustment mode is not on for the current image block, there is no need to decode a feature adjustment factor corresponding to the current image block from the auxiliary bitstream corresponding to the current image block.

[0240] Exemplarily, to determine whether the feature adjustment mode is on for the current image block, the following scheme may be adopted.

[0241] In method 1, the encoding side encodes a feature adjustment flag bit into a bitstream, and the decoding side decodes the feature adjustment flag bit from the bitstream. The feature adjustment flag bit enables or disables the feature adjustment mode for the current image block. If the feature adjustment flag bit enables or disables the feature adjustment mode for the current image block, the decoding side determines that the feature adjustment mode is enabled for the current image block. Otherwise, if the feature adjustment flag bit disables or disables the feature adjustment mode for the current image block, the decoding side determines that the feature adjustment mode is disabled for the current image block. For example, if the feature adjustment flag bit is a first value, the feature adjustment flag bit enables or disables the feature adjustment mode for the current image block, and if the feature adjustment flag bit is a second value, the feature adjustment flag bit disables or disables the feature adjustment mode for the current image block.

[0242] For example, the feature adjustment flag bit may be a sequence-level feature adjustment flag bit, i.e., every image block in a sequence corresponds to the feature adjustment flag bit, or the feature adjustment flag bit may be an image-level feature adjustment flag bit, i.e., every image block in an image corresponds to the feature adjustment flag bit, or the feature adjustment flag bit may be a slice-level feature adjustment flag bit, i.e., every image block in a slice corresponds to the feature adjustment flag bit, and this is not limiting.

[0243] Method 2: The encoding side encodes a first value range of feature values ​​into a bitstream, and the decoding side decodes the first value range of feature values ​​from the bitstream. Based on this, if the feature values ​​of the second feature (e.g., all feature values) are within the first value range, the decoding side determines that the feature adjustment mode is on for the current image block. Otherwise, if the feature values ​​of the second feature (e.g., any feature values) are not within the first value range, the decoding side determines that the feature adjustment mode is prohibited from being on for the current image block. For example, after obtaining the second feature, the decoding side can determine whether the feature value of the second feature is within the first value range.

[0244] For example, the first value range may be a first value range at a sequence level, i.e., all image blocks in a sequence correspond to the first value range, or the first value range may be a first value range at an image level, i.e., all image blocks in an image correspond to the first value range, or the first value range may be a first value range at a slice level, i.e., all image blocks in a slice correspond to the first value range, with no limitation on this first value range.

[0245] Method 3: The encoding side encodes a second value range of the feature change value into a bitstream, and the decoding side decodes the second value range of the feature change value from the bitstream. Based on this, if the feature change value corresponding to the second feature (e.g., the feature change values ​​corresponding to all feature values, i.e., each feature value corresponds to one feature change value) is within the second value range, the decoding side determines that the feature adjustment mode is on for the current image block. Otherwise, if the feature change value corresponding to the second feature (e.g., any feature change value) is not within the second value range, the decoding side determines that the feature adjustment mode is prohibited for the current image block. For example, after the decoding side obtains the second feature, it can also determine the feature change value corresponding to the second feature, i.e., the feature change value corresponding to each feature value in the second feature can be determined. The feature change value can indicate a change degree range in the spatial domain or the channel domain, i.e., the change status of the second feature in the spatial domain or the channel domain can be determined. For example, the feature change value may include, but is not limited to, a gradient value.

[0246] For example, the second value range may be a sequence-level second value range, i.e., all image blocks in a sequence correspond to the second value range, or the second value range may be an image-level second value range, i.e., all image blocks in an image correspond to the second value range, or the second value range may be a slice-level second value range, i.e., all image blocks in a slice correspond to the second value range, with no limitation on the second value range.

[0247] Method 4: The encoding side encodes a feature adjustment flag bit and a first value range into a bitstream, and the decoding side decodes the feature adjustment flag bit and the first value range from the bitstream. If the feature adjustment flag bit allows the feature adjustment mode to be turned on for the current image block and the feature values ​​of the second feature (e.g., all feature values) are within the first value range, it is determined that the feature adjustment mode is on for the current image block. Otherwise, it is determined that the feature adjustment flag bit prohibits the feature adjustment mode from being turned on for the current image block and / or that if the feature values ​​of the second feature (e.g., any feature value) are not within the first value range, it is determined that the feature adjustment mode is prohibited from being turned on for the current image block.

[0248] Method 5: The encoding side encodes the feature adjustment flag bit and the second value range into a bitstream, and the decoding side decodes the feature adjustment flag bit and the second value range from the bitstream. If the feature adjustment flag bit enables the feature adjustment mode to be turned on for the current image block and the feature change values ​​corresponding to the second feature (e.g., the feature change values ​​corresponding to all feature values, i.e., each feature value corresponds to one feature change value) are within the second value range, the decoding side determines that the feature adjustment mode is on for the current image block. Otherwise, the feature adjustment flag bit prohibits the feature adjustment mode from being turned on for the current image block, and / or if the feature change values ​​corresponding to the second feature (e.g., any feature change value) are not within the second value range, the decoding side determines that the feature adjustment mode is prohibited from being turned on for the current image block.

[0249] Method 6: The encoding side encodes a first value range and a second value range into a bitstream, and the decoding side decodes the first value range and the second value range from the bitstream. If the feature value of the second feature is in the first value range and the feature change value corresponding to the second feature is in the second value range, the decoding side may determine that the feature adjustment mode is on for the current image block. Otherwise, if the feature value of the second feature is not in the first value range and / or the feature change value corresponding to the second feature is not in the second value range, the decoding side may determine that the feature adjustment mode is prohibited from being on for the current image block.

[0250] Method 7: The encoding side encodes the feature adjustment flag bit, the first value range, and the second value range into a bitstream, and the decoding side decodes the feature adjustment flag bit, the first value range, and the second value range from the bitstream. If the feature adjustment flag bit enables the feature adjustment mode to be turned on for the current image block, the feature value of the second feature is in the first value range, and the feature change value corresponding to the second feature is in the second value range, it is determined that the feature adjustment mode is on for the current image block. Otherwise, if the feature adjustment flag bit disables the feature adjustment mode to be turned on for the current image block, and the feature value of the second feature is not in the first value range and / or the feature change value corresponding to the second feature is not in the second value range, it is determined that the feature adjustment mode is disabled to be turned on for the current image block.

[0251] Method 8: If the attribute value corresponding to the second feature is within the set attribute value range, the decoding side may determine that the feature adjustment mode is on for the current image block. Otherwise, if the attribute value corresponding to the second feature is not within the set attribute value range, the decoding side may determine that the feature adjustment mode is prohibited from being on for the current image block. In Method 8, the decoding side does not need to analyze information about the feature adjustment mode from the bitstream, and determines whether the feature adjustment mode is on for the current image block based on the set attribute value range. In Method 8, the attribute value corresponding to the second feature may include, but is not limited to, a variance value corresponding to the feature value of the second feature, and the variance value is used to indicate the magnitude of the drastic change in the location of the feature value of the second feature.

[0252] Example 16: In Example 14, for each feature value in the second feature, feature adjustment can be performed on the feature value based on the feature adjustment value corresponding to the feature value to obtain an adjusted feature value corresponding to the feature value. For the feature adjustment process, it is first necessary to determine whether a condition is met, and the feature adjustment process is performed only if the condition is met (i.e., feature adjustment can be performed on the feature value based on the feature adjustment value corresponding to the feature value to obtain an adjusted feature value corresponding to the feature value).

[0253] For example, these conditions may include, but are not limited to, the following: 1. Analyzing related information based on the additional bitstream to determine whether to adjust a feature value, which may be flag information indicating whether the region where the current feature value is located (which may be the feature where it is located, or the channel of the feature where it is located, or a slice of the channel where it is located) needs to be adjusted, information on the range of the magnitude of the feature value that needs to be adjusted (determining whether the current feature value is within this range to determine whether it needs to be adjusted), or the range of change in the spatial or channel region of the feature value that needs to be adjusted (calculating the change status (e.g., gradient value) of the current feature value in the spatial or channel region to determine whether the change status is within this range to determine whether it needs to be adjusted). 2. Determining based on an attribute of the feature value, for example, the variance corresponding to the current feature value (indicating the drastic change in the location of the feature value).

[0254] For example, for each feature value in the second feature, the following method may be adopted to determine whether to perform feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value to obtain an adjusted feature value corresponding to the feature value:

[0255] Method 1: The encoding side encodes a feature adjustment flag bit corresponding to the feature value into a bitstream, and the decoding side decodes the feature adjustment flag bit corresponding to the feature value from the bitstream, where the feature adjustment flag bit indicates whether feature adjustment is performed on the feature value or not. If the feature adjustment flag bit indicates that feature adjustment is performed on the feature value, the decoding side performs feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value to obtain an adjusted feature value corresponding to the feature value. Otherwise, if the feature adjustment flag bit indicates that feature adjustment is not performed on the feature value, the decoding side does not perform feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value. For example, if the feature adjustment flag bit is a first value, the feature adjustment flag bit is used to indicate that feature adjustment is performed on the feature value, and if the feature adjustment flag bit is a second value, the feature adjustment flag bit is used to indicate that feature adjustment is not performed on the feature value.

[0256] Method 2: The encoding side encodes a first value range of a feature value into a bitstream, and the decoding side decodes the first value range of the feature value from the bitstream. Based on this, if the feature value is within the first value range, the decoding side performs feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value to obtain an adjusted feature value corresponding to the feature value. Otherwise, if the feature value is not within the first value range, the decoding side does not perform feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value.

[0257] Method 3: The encoding side encodes a second value range of the feature change value into a bitstream, and the decoding side decodes the second value range of the feature change value from the bitstream. Based on this, if the feature change value (e.g., gradient value) corresponding to the feature value is within the second value range, the decoding side performs feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value to obtain an adjusted feature value corresponding to the feature value. Otherwise, if the feature change value (e.g., gradient value) corresponding to the feature value is not within the second value range, the decoding side does not perform feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value.

[0258] Method 4: The encoding side encodes a feature adjustment flag bit and a first value range into a bitstream, and the decoding side decodes the feature adjustment flag bit and the first value range from the bitstream. If the feature adjustment flag bit indicates that feature adjustment is to be performed on the feature value and the feature value is within the first value range, the decoding side performs feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value. Otherwise, the decoding side does not perform feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value.

[0259] Method 5: The encoding side encodes a feature adjustment flag bit and a second value range into a bitstream, and the decoding side decodes the feature adjustment flag bit and the second value range from the bitstream. If the feature adjustment flag bit indicates that feature adjustment is to be performed on the feature value and the feature change value corresponding to the feature value is within the second value range, the decoding side performs feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value. Otherwise, the decoding side does not perform feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value.

[0260] Method 6: The encoding side encodes a first value range and a second value range into a bitstream, and the decoding side decodes the first value range and the second value range from the bitstream. If the feature value is in the first value range and the feature change value corresponding to the feature value is in the second value range, the decoding side performs feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value. Otherwise, the decoding side does not perform feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value.

[0261] Method 7: The encoding side encodes a feature adjustment flag bit, a first value range, and a second value range into a bitstream, and the decoding side decodes the feature adjustment flag bit, the first value range, and the second value range from the bitstream. If the feature adjustment flag bit indicates that feature adjustment is to be performed on the feature value, and the feature value is in the first value range and the feature change value corresponding to the feature value is in the second value range, the decoding side performs feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value. Otherwise, the decoding side does not perform feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value.

[0262] Method 8: If the attribute value corresponding to the feature value is within the set value range of the attribute value, the decoding side performs feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value. Otherwise, the decoding side does not perform feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value. In method 8, the decoding side does not need to analyze information about the feature adjustment mode from the bitstream, and determines whether to perform feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value based on the set value range of the attribute value. In method 8, the attribute value corresponding to the feature value may include, but is not limited to, a variance value corresponding to the feature value, and the variance value is used to indicate the degree of drastic change in the location of the feature value.

[0263] Example 17: Referring to Example 7, the encoding side can obtain a feature adjustment factor for the decoding sub-network, perform feature adjustment on a feature in the decoding sub-network based on the feature adjustment factor, and encode the feature adjustment factor for the decoding sub-network into the auxiliary bitstream. The decoding side can decode the feature adjustment factor for the decoding sub-network from the auxiliary bitstream and perform feature adjustment on a feature in the decoding sub-network based on the feature adjustment factor. For example, the decoding sub-network may obtain feature A (feature A is any feature in the decoding sub-network), and then perform feature adjustment on feature A using the feature adjustment factor to obtain an adjusted feature B. The decoding sub-network continues processing based on feature B, and finally obtains a first feature.

[0264] In one possible embodiment, as shown in Figure 8A of Example 8, the feature adjustment factor for the decoding sub-network may be used in the feature restoration process in the decoding sub-network, that is, in the feature restoration process, the feature adjustment factor is used to perform feature adjustment on a certain feature, and the feature adjustment process can refer to the feature adjustment process of the second feature, and the description is omitted here.

[0265] In one possible embodiment, as shown in Figure 8B of Example 9, the feature adjustment factor for the decoding sub-network may be used in the residual restoration process in the decoding sub-network, that is, in the residual restoration process, the feature adjustment factor is used to perform feature adjustment on a certain feature, and the feature adjustment process can refer to the feature adjustment process of the second feature, and the description is omitted here.

[0266] For example, each of the above embodiments may be realized alone or in combination. For example, each of the embodiments 1 to 17 may be realized alone, and at least two of the embodiments 1 to 17 may be realized in combination.

[0267] For example, in each of the above embodiments, the content of the encoding side may be applied to the decoding side, i.e., may be processed in the same manner by the decoding side, and the content of the decoding side may be applied to the encoding side, i.e., may be processed in the same manner by the encoding side.

[0268] Based on the same idea as the above method, an embodiment of the present invention further provides a decoding device, which is applied to the decoding side, and includes: a memory configured to store video data; and a decoder configured to implement the decoding method in the above embodiments 1 to 17, i.e., the processing process of the decoding side.

[0269] For example, in one possible embodiment, the decoder Obtaining a first feature corresponding to a current image block by a first neural network, and obtaining a feature adjustment factor corresponding to the current image block, wherein the first neural network includes at least one convolution layer; determining a target feature based on the first feature and the feature adjustment factor; and obtaining a reconstructed image block corresponding to the current image block by a second neural network based on the target feature, the second neural network including at least one convolutional layer.

[0270] Based on the same idea as the above method, an embodiment of the present invention further provides an encoding device, which is applied to the encoding side, and includes: a memory configured to store video data; and an encoder configured to implement the encoding method in the above embodiments 1 to 17, i.e., the processing process of the encoding side.

[0271] For example, in one possible embodiment, the encoder Obtaining a first feature corresponding to a current image block by a first neural network, the first neural network including at least one convolutional layer; obtaining a feature adjustment factor corresponding to the current image block based on the first feature; encoding the feature adjustment factor into an auxiliary bitstream corresponding to the current image block.

[0272] Based on the same concept as the above-mentioned method, a decoding device (also called a video decoder) provided by an embodiment of the present invention, from the hardware perspective, may specifically refer to Figure 11A for a schematic hardware architecture diagram thereof, which includes a processor 1101 and a machine-readable storage medium 1102, in which machine-executable instructions executable by the processor 1101 are stored, and the processor 1101 is used to execute the machine-executable instructions to implement the decoding methods of the above-mentioned embodiments 1 to 17 of the present invention. For example, in one possible embodiment, the decoding device may include: Obtaining a first feature corresponding to a current image block by a first neural network, and obtaining a feature adjustment factor corresponding to the current image block, wherein the first neural network includes at least one convolution layer; determining a target feature based on the first feature and the feature adjustment factor; and a step of obtaining a reconstructed image block corresponding to the current image block by a second neural network based on the target feature, the second neural network including at least one convolutional layer.

[0273] Based on the same concept as the above method, an encoding device (also called a video encoder) provided by an embodiment of the present invention, from the hardware perspective, can be specifically referred to in Figure 11B for its schematic hardware architecture diagram. The encoding device includes a processor 1111 and a machine-readable storage medium 1112, and the machine-readable storage medium 1112 stores machine-executable instructions executable by the processor 1111. The processor 1111 is used to execute the machine-executable instructions to implement the encoding methods of the above embodiments 1 to 17 of the present invention. For example, in one possible embodiment, the encoding device: Obtaining a first feature corresponding to a current image block by a first neural network, the first neural network including at least one convolutional layer; obtaining a feature adjustment factor corresponding to the current image block based on the first feature; encoding the feature adjustment factor into an auxiliary bitstream corresponding to the current image block.

[0274] Based on the same idea as the above method, an embodiment of the present invention provides an electronic device, which includes a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being used to execute the machine-executable instructions to implement the decoding method or encoding method of the above embodiments 1 to 17 of the present invention.

[0275] Based on the same idea as the above method, an embodiment of the present invention further provides a machine-readable storage medium having stored thereon some computer instructions, which, when executed by a processor, can implement the methods disclosed in the above examples of the present invention, such as the decoding method or the encoding method in each of the above embodiments.

[0276] Based on the same idea as the above method, an embodiment of the present invention further provides a computer application program, which, when executed by a processor, can implement the decoding method or the encoding method disclosed in the above example of the present invention.

[0277] Based on the same idea as the above method, an embodiment of the present invention further provides a decoding device, which is applied to a decoding side, and may include: an acquisition module for acquiring a first feature corresponding to a current image block by a first neural network and acquiring a feature adjustment factor corresponding to the current image block, where the first neural network includes at least one convolutional layer; and a determination module for determining a target feature based on the first feature and the feature adjustment factor, where the acquisition module is further used to acquire a reconstructed image block corresponding to the current image block by a second neural network based on the target feature, where the second neural network includes at least one convolutional layer.

[0278] Illustratively, when the acquisition module acquires a first feature corresponding to the current image block using the first neural network, the acquisition module is specifically used to acquire probability distribution parameters based on a first bitstream corresponding to the current image block, determine a probability distribution model based on the probability distribution parameters, decode a second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features, and determine the first feature corresponding to the current image block based on the decoded image features.

[0279] Exemplarily, when the acquisition module acquires a first feature corresponding to the current image block using the first neural network, the acquisition module specifically acquires probability distribution parameters and predicted values ​​based on a first bitstream corresponding to the current image block, determines a probability distribution model based on the probability distribution parameters, decodes a second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features, performs residual reconstruction on the decoded image features to obtain residual features, and determines the first feature corresponding to the current image block based on the residual features and the predicted value.

[0280] Illustratively, when the acquisition module acquires the feature adjustment factor corresponding to the current image block, it specifically decodes the auxiliary bitstream corresponding to the current image block to obtain the feature adjustment factor corresponding to the current image block, or it is used to determine a fixed parameter value as the feature adjustment factor corresponding to the current image block.

[0281] Exemplarily, when determining a target feature based on the first feature and the feature adjustment factor, the determination module is specifically used to perform feature enhancement on the first feature to obtain a second feature, perform feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature, and determine the target feature based on the third feature.

[0282] Exemplarily, when performing feature enhancement on the first feature to obtain a second feature, the determination module is specifically used to determine an initial feature corresponding to an attention sub-network based on the first feature, and perform feature enhancement on the initial feature by the attention sub-network to obtain the second feature; exemplarily, when determining the target feature based on the third feature, the determination module is specifically used to determine the third feature as the target feature.

[0283] The attention sub-network includes a residual emphasis sub-network and a weight generation sub-network. When the attention sub-network performs feature emphasis on the initial features to obtain the second features, the determination module specifically uses the residual emphasis sub-network to perform feature emphasis on the initial features to obtain enhanced features, and uses the weight generation sub-network to generate weight features corresponding to the initial features, and generates the second features based on the initial features, the enhanced features, and the weight features.

[0284] Exemplarily, the weight generation subnetwork may include a residual block subnetwork, a convolutional subnetwork, and a feature mapping subnetwork. When generating weight features corresponding to the initial features using the weight generation subnetwork, the determination module specifically performs convolutional activation on the initial features using the residual block subnetwork to obtain post-convolutional activation features, performs convolution on the post-convolutional activation features using the convolutional subnetwork to obtain post-convolutional features, and performs feature mapping on the post-convolutional features using the feature mapping subnetwork to obtain the weight features corresponding to the initial features.

[0285] Exemplarily, when the determination module performs feature enhancement on the first feature to obtain the second feature, it is specifically used to determine an initial feature corresponding to an attention sub-network based on the first feature, and perform feature enhancement on the initial feature by a first sub-network in the attention sub-network to obtain the second feature; and when the determination module determines the target feature based on the third feature, it is specifically used to process the third feature by a second sub-network in the attention sub-network to obtain a fourth feature, and determine the fourth feature as the target feature.

[0286] Exemplarily, when a first sub-network in the attention sub-network performs feature enhancement on the initial features to obtain the second features, the determination module is specifically used to perform feature enhancement on the initial features using a residual enhancement sub-network to obtain enhanced features, generate weight features corresponding to the initial features using a weight generation sub-network, and generate the second features based on the enhanced features and weight features; and when a second sub-network in the attention sub-network processes the third features to obtain fourth features, the determination module is specifically used to generate the fourth features based on the initial features and the third features.

[0287] Exemplarily, when a first sub-network in the attention sub-network performs feature enhancement on the initial features to obtain the second features, the determination module is specifically used to perform feature enhancement on the initial features using a residual block sub-network to obtain enhanced features, perform convolution on the enhanced features using a convolution sub-network to obtain convolutional features, and determine the second features based on the convolutional features; when a second sub-network in the attention sub-network processes the third features to obtain fourth features, the determination module is specifically used to perform feature mapping on the third features using a feature mapping sub-network to obtain weighted features, perform feature enhancement on the initial features using a residual enhancement sub-network to obtain enhanced features, and generate the fourth features based on the initial features, the enhanced features, and the weighted features.

[0288] Exemplarily, when determining an initial feature corresponding to an attention sub-network based on the first feature, the determination module specifically performs feature enhancement on the first feature using an initial enhancement sub-network to obtain an enhanced feature, and then performs upsampling convolution on the enhanced feature using an upsampling convolution sub-network to obtain the initial feature.

[0289] Exemplarily, when the determination module performs feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature, specifically, when the second feature includes C*H*W feature values, where C represents the number of channels, H represents the feature height, and W represents the feature width, the determination module determines a feature adjustment value corresponding to each feature value based on the feature adjustment factor, performs feature adjustment on the feature value based on the corresponding feature adjustment value to obtain an adjusted feature value corresponding to the feature value, and generates the third feature based on the adjusted feature value corresponding to each feature value.

[0290] For example, when determining the feature adjustment values ​​corresponding to each feature value based on the feature adjustment factor, the determination module may be used to determine the feature adjustment values ​​corresponding to each of the C*H*W feature values ​​based on the C*H*W feature adjustment values ​​if the feature adjustment factor includes C*H*W feature adjustment values; or to determine the feature adjustment values ​​corresponding to each of the C*H*W feature values ​​based on the H*W feature adjustment values ​​if the feature adjustment factor includes H*W feature adjustment values; or to determine the feature adjustment values ​​corresponding to each of the C*H*W feature values ​​based on the C feature adjustment values ​​if the feature adjustment factor includes C feature adjustment values.

[0291] For example, when the determination module performs feature adjustment on a feature value based on a feature adjustment value corresponding to the feature value to obtain an adjusted feature value corresponding to the feature value, specifically, when the feature value corresponds to only one feature adjustment value, the determined adjusted feature value is determined by the formula yr=rf*y, where yr represents the adjusted feature value, rf represents the feature adjustment value, and y represents the feature value; or when the feature value corresponds to N+1 feature adjustment values, where N is a positive integer, the determined adjusted feature value is determined by the formula yr=rf_0*y. 0 +rf_1*y 1 +rf_2*y 2 +…+rf_N*y N is used to determine the adjusted feature values, yr represents the adjusted feature values, rf_0, rf_1, rf_2, ..., rf_N represent the N+1 feature adjustment values, and y represents the feature value.

[0292] Exemplarily, the second neural network includes a reconstruction decoding subnetwork, and when the acquisition module obtains a reconstructed image block corresponding to the current image block using the second neural network based on the target feature, the acquisition module is specifically used to process the target feature using the reconstruction decoding subnetwork to obtain the reconstructed image block, and the reconstruction decoding subnetwork includes an upsampling convolution subnetwork, which performs upsampling convolution on the target feature to obtain the reconstructed image block, or the reconstruction decoding subnetwork includes an upsampling convolution subnetwork and a color space conversion subnetwork, which performs upsampling convolution on the target feature using the upsampling convolution subnetwork and performs color space conversion on the features after upsampling convolution using the color space conversion subnetwork to obtain the reconstructed image block.

[0293] Based on the same idea as the above method, an embodiment of the present invention further provides an encoding device, which is applied to an encoding side, and the device obtains a first feature corresponding to a current image block by a first neural network, the first neural network including at least one convolutional layer, and includes: an obtaining module for obtaining a feature adjustment factor corresponding to the current image block based on the first feature; and an encoding module for encoding the feature adjustment factor into an auxiliary bitstream corresponding to the current image block.

[0294] Exemplarily, when the acquisition module acquires a feature adjustment factor corresponding to the current image block based on the first feature, the acquisition module is specifically used to acquire at least one candidate feature adjustment factor, determine a rate-distortion cost corresponding to each candidate feature adjustment factor based on the first feature, and select one candidate feature adjustment factor from all the candidate feature adjustment factors as the feature adjustment factor corresponding to the current image block based on the rate-distortion cost corresponding to each candidate feature adjustment factor.

[0295] It should be understood by those skilled in the art that embodiments of the present invention may be provided as a method, a system, or a computer program product. The present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware. The present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code. The above are merely examples of the present invention and are not intended to limit the present invention.

[0296] Those skilled in the art can make various modifications and variations to the present invention, and any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. obtaining a first feature corresponding to the current image block by a first neural network, and obtaining a feature adjustment factor corresponding to the current image block; performing feature enhancement on the first feature to obtain a second feature; performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature; determining a target feature based on the third feature; and obtaining a reconstructed image block corresponding to the current image block by a second neural network based on the target feature. A decoding method comprising:

2. The step of performing feature enhancement on the first feature to obtain a second feature includes: determining initial features corresponding to an attention sub-network based on the first features; performing feature enhancement on the initial features using a residual enhancement sub-network to obtain enhanced features; generating weight features corresponding to the initial features using a weight generation sub-network; generating the second feature based on the enhanced feature and the weight feature; determining the target feature based on the third feature, processing the third feature by a second sub-network in the attention sub-network to obtain a fourth feature and determining the fourth feature as the target feature, wherein processing the third feature by the second sub-network in the attention sub-network to obtain a fourth feature includes generating the fourth feature based on the initial feature and the third feature.

2. The method of claim 1 .

3. determining initial features corresponding to an attention sub-network based on the first features, performing feature enhancement on the first features using an initial enhancement sub-network to obtain enhanced features; and performing upsampling convolution on the enhanced features using an upsampling convolution sub-network to obtain initial features corresponding to the attention sub-network.

3. The method of claim 2.

4. the weight generation sub-network includes a residual block sub-network, a convolution sub-network, and a feature mapping sub-network, and generating weight features corresponding to the initial features using the weight generation sub-network includes: performing convolutional activation on the initial features using the residual block sub-network to obtain post-convolutional activation features; performing convolution on the convolutional activation features using the convolutional sub-network to obtain convolutional features; performing feature mapping on the convolved features using the feature mapping sub-network to obtain the weight features; 3. The method of claim 2.

5. performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature, where the second feature includes C*H*W feature values, where C represents a channel number, H represents a feature height, and W represents a feature width; determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the feature adjustment factor; and performing feature adjustment on the C*H*W feature values ​​based on the feature adjustment values ​​to obtain adjusted feature values ​​corresponding to the C*H*W feature values; generating the third feature based on adjusted feature values ​​corresponding to the C*H*W feature values; 2. The method of claim 1 .

6. Parsing a feature adjustment flag bit from the bitstream and determining that the feature adjustment mode is on for the current image block if the feature adjustment flag bit allows the feature adjustment mode to be on for the current image block; or analyzing a first value range of feature values ​​from the bitstream, and determining that a feature adjustment mode is on for the current image block if the feature value of the second feature is in the first value range; 2. The method of claim 1 .

7. the first neural network includes a decoding sub-network; 2. The method of claim 1 .

8. The step of obtaining a first feature corresponding to the current image block by the first neural network includes: obtaining a probability distribution parameter based on a first bitstream corresponding to the current image block; determining a probability distribution model based on the probability distribution parameters, and decoding a second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features; determining a first feature corresponding to the current image block based on the decoded image features; 2. The method of claim 1 .

9. The step of obtaining a first feature corresponding to the current image block by the first neural network includes: obtaining a probability distribution parameter and a predicted value based on a first bitstream corresponding to the current image block; determining a probability distribution model based on the probability distribution parameters, and decoding a second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features; performing residual reconstruction on the decoded image features to obtain residual features; determining a first feature corresponding to the current image block based on the residual feature and the predicted value; 2. The method of claim 1 .

10. The step of obtaining a feature adjustment factor corresponding to the current image block includes: decoding an auxiliary bitstream corresponding to the current image block to obtain a feature adjustment factor corresponding to the current image block, or determining a fixed parameter value as the feature adjustment factor corresponding to the current image block; 2. The method of claim 1 .

11. The second neural network includes a reconstruction decoding sub-network, and the step of obtaining a reconstructed image block corresponding to the current image block by the second neural network based on the target feature includes: processing the target features by a reconstruction decoding sub-network to obtain the reconstructed image block; the reconstruction decoding subnetwork includes an upsampling convolution subnetwork, and performs upsampling convolution on the target features to obtain the reconstructed image block by the upsampling convolution subnetwork; or the reconstruction decoding subnetwork includes an upsampling convolution subnetwork and a color space conversion subnetwork, and performs upsampling convolution on the target features by the upsampling convolution subnetwork, and performs color space conversion on the features after the upsampling convolution by the color space conversion subnetwork to obtain the reconstructed image block.

2. The method of claim 1 .

12. obtaining a first feature corresponding to the current image block by a first neural network; obtaining a feature adjustment factor corresponding to the current image block based on the first feature; performing feature enhancement on the first feature to obtain a second feature; performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature; determining a target feature based on the third feature; and obtaining a reconstructed image block corresponding to the current image block by a second neural network based on the target feature.

10. A coding method comprising:

13. encoding the feature adjustment factor into an auxiliary bitstream corresponding to the current image block; 13. The method of claim 12.

14. The step of performing feature enhancement on the first feature to obtain a second feature includes: determining initial features corresponding to an attention sub-network based on the first features; performing feature enhancement on the initial features using a residual enhancement sub-network to obtain enhanced features; generating weight features corresponding to the initial features using a weight generation sub-network; generating the second feature based on the enhanced feature and the weight feature; determining the target feature based on the third feature, processing the third feature by a second sub-network in the attention sub-network to obtain a fourth feature and determining the fourth feature as the target feature, wherein processing the third feature by the second sub-network in the attention sub-network to obtain a fourth feature includes generating the fourth feature based on the initial feature and the third feature.

13. The method of claim 12.

15. determining initial features corresponding to an attention sub-network based on the first features, performing feature enhancement on the first features using an initial enhancement sub-network to obtain enhanced features; and performing upsampling convolution on the enhanced features using an upsampling convolution sub-network to obtain initial features corresponding to the attention sub-network.

15. The method of claim 14.

16. the weight generation sub-network includes a residual block sub-network, a convolution sub-network, and a feature mapping sub-network, and generating weight features corresponding to the initial features using the weight generation sub-network includes: performing convolutional activation on the initial features using the residual block sub-network to obtain post-convolutional activation features; performing convolution on the convolutional activation features using the convolutional sub-network to obtain convolutional features; performing feature mapping on the convolved features using the feature mapping sub-network to obtain the weight features; 15. The method of claim 14.

17. performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature, where the second feature includes C*H*W feature values, where C represents a channel number, H represents a feature height, and W represents a feature width; determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the feature adjustment factor; and performing feature adjustment on the C*H*W feature values ​​based on the feature adjustment values ​​to obtain adjusted feature values ​​corresponding to the C*H*W feature values; generating the third feature based on adjusted feature values ​​corresponding to the C*H*W feature values; 13. The method of claim 12.

18. encoding a feature adjustment flag bit into the bitstream, wherein the feature adjustment flag bit allows turning on a feature adjustment mode for the current image block, or the feature adjustment flag bit prohibits turning on a feature adjustment mode for the current image block; or encoding a first value range of feature values ​​into a bitstream, wherein a feature adjustment mode is enabled for the current image block if the feature value of the second feature is in the first value range; 13. The method of claim 12.

19. the first neural network includes a decoding sub-network; 13. The method of claim 12.

20. a memory configured to store video data; obtaining a first feature corresponding to the current image block by a first neural network, and obtaining a feature adjustment factor corresponding to the current image block; performing feature enhancement on the first feature to obtain a second feature; performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature; determining a target feature based on the third feature; and obtaining a reconstructed image block corresponding to the current image block by a second neural network based on the target feature. A decoding device characterized by:

21. The step of performing feature enhancement on the first feature to obtain a second feature includes: determining initial features corresponding to an attention sub-network based on the first features; performing feature enhancement on the initial features using a residual enhancement sub-network to obtain enhanced features; generating weight features corresponding to the initial features using a weight generation sub-network; generating the second feature based on the enhanced feature and the weight feature; determining the target feature based on the third feature, processing the third feature by a second sub-network in the attention sub-network to obtain a fourth feature and determining the fourth feature as the target feature, wherein processing the third feature by the second sub-network in the attention sub-network to obtain a fourth feature includes generating the fourth feature based on the initial feature and the third feature.

21. The apparatus of claim 20.

22. determining initial features corresponding to an attention sub-network based on the first features, performing feature enhancement on the first features using an initial enhancement sub-network to obtain enhanced features; and performing upsampling convolution on the enhanced features using an upsampling convolution sub-network to obtain initial features corresponding to the attention sub-network.

22. The apparatus of claim 21 .

23. the weight generation sub-network includes a residual block sub-network, a convolution sub-network, and a feature mapping sub-network, and generating weight features corresponding to the initial features using the weight generation sub-network includes: performing convolutional activation on the initial features using the residual block sub-network to obtain post-convolutional activation features; performing convolution on the convolutional activation features using the convolutional sub-network to obtain convolutional features; performing feature mapping on the convolved features using the feature mapping sub-network to obtain the weight features; 22. The apparatus of claim 21 .

24. performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature, where the second feature includes C*H*W feature values, where C represents a channel number, H represents a feature height, and W represents a feature width; determining feature adjustment values ​​corresponding to the C*H*W feature values ​​based on the feature adjustment factor; and performing feature adjustment on the C*H*W feature values ​​based on the feature adjustment values ​​to obtain adjusted feature values ​​corresponding to the C*H*W feature values; generating the third feature based on adjusted feature values ​​corresponding to the C*H*W feature values; 21. The apparatus of claim 20.

25. Parsing a feature adjustment flag bit from the bitstream and determining that the feature adjustment mode is on for the current image block if the feature adjustment flag bit allows the feature adjustment mode to be on for the current image block; or analyzing a first value range of feature values ​​from the bitstream, and determining that a feature adjustment mode is on for the current image block if the feature value of the second feature is in the first value range; 21. The apparatus of claim 20.

26. the first neural network includes a decoding sub-network; 21. The apparatus of claim 20.

27. a memory configured to store video data; obtaining a first feature corresponding to the current image block by a first neural network; obtaining a feature adjustment factor corresponding to the current image block based on the first feature; performing feature enhancement on the first feature to obtain a second feature; performing feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature; determining a target feature based on the third feature; and obtaining a reconstructed image block corresponding to the current image block by a second neural network based on the target feature.

1. An encoding device comprising:

28. A decoding device including a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is adapted to execute machine-executable instructions to perform the method of any one of claims 1 to 11.

10. A decoding device comprising:

29. a coding device including a processor and a machine-readable storage medium having machine-executable instructions stored thereon that are executable by the processor; The processor is adapted to execute machine-executable instructions to perform the method of any one of claims 12 to 19.

10. A coding device comprising:

30. An electronic device including a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is adapted to execute machine-executable instructions to perform the method of any one of claims 1 to 11 or the method of any one of claims 12 to 19. An electronic device characterized by:

31. A machine-readable storage medium having stored thereon a plurality of computer instructions, which, when executed by a processor, perform the method of any one of claims 1 to 11, or which, when executed by a processor, perform the method of any one of claims 12 to 19. A machine-readable storage medium comprising:

Images