Decoding and coding method, device and equipment

Abstract

The invention provides a decoding and coding method, device and equipment, and the decoding method comprises the steps: obtaining a first feature corresponding to a current image block through a first neural network, and obtaining a feature adjustment factor corresponding to the current image block, the first neural network comprising at least one convolution layer; determining a target feature based on the first feature and the feature adjustment factor; and obtaining a reconstructed image block corresponding to the current image block through a second neural network based on the target feature, the second neural network comprising at least one convolutional layer. Through the technical scheme of the invention, the coding performance and the decoding performance can be improved.

Description

Technical Field

[0001] The present application relates to the field of coding and decoding technology, and in particular to a decoding and coding method, device and equipment thereof. Background Art

[0002] In order to save space, video images are transmitted after being encoded. Complete video encoding can include prediction, transformation, quantization, entropy coding, filtering and other processes. For the prediction process, the prediction process can include intra-frame prediction and inter-frame prediction. Inter-frame prediction refers to the use of the correlation in the video time domain to use the pixels of the adjacent encoded images to predict the current pixel, so as to effectively remove the redundancy in the video time domain. Intra-frame prediction refers to the use of the correlation in the video space domain to use the pixels of the encoded blocks of the current frame image to predict the current pixel, so as to remove the redundancy in the video space domain.

[0003] With the rapid development of deep learning, deep learning has achieved success in many high-level computer vision problems, such as image classification and target detection. Deep learning has also gradually begun to be applied in the field of encoding and decoding, that is, neural networks can be used to encode and decode images. Although the encoding and decoding methods based on neural networks have shown great performance potential, they still have problems such as poor encoding performance, poor decoding performance and high complexity. Summary of the invention

[0004] In view of this, the present application provides a decoding and encoding method, device and equipment thereof to improve encoding performance and decoding performance.

[0005] The present application provides a decoding method, which is applied to a decoding end, and the method includes:

[0006] Acquire a first feature corresponding to the current image block through a first neural network, and acquire a feature adjustment factor corresponding to the current image block, wherein the first neural network includes at least one convolutional layer;

[0007] determining a target feature based on the first feature and the feature adjustment factor;

[0008] Based on the target feature, a reconstructed image block corresponding to the current image block is obtained through a second neural network, where the second neural network includes at least one convolutional layer.

[0009] The present application provides an encoding method, which is applied to an encoding end, and the method includes: obtaining a first feature corresponding to a current image block through a first neural network, wherein the first neural network includes at least one convolutional layer;

[0010] Acquire a feature adjustment factor corresponding to the current image block based on the first feature;

[0011] The feature adjustment factor is encoded in an auxiliary code stream corresponding to the current image block.

[0012] The present application provides a decoding device, applied to a decoding end, the device comprising:

[0013] a memory configured to store video data;

[0014] A decoder configured to implement:

[0015] Acquire a first feature corresponding to the current image block through a first neural network, and acquire a feature adjustment factor corresponding to the current image block, wherein the first neural network includes at least one convolutional layer;

[0016] determining a target feature based on the first feature and the feature adjustment factor;

[0017] Based on the target feature, a reconstructed image block corresponding to the current image block is obtained through a second neural network, where the second neural network includes at least one convolutional layer.

[0018] The present application provides an encoding device, applied to an encoding end, the device comprising:

[0019] a memory configured to store video data;

[0020] An encoder configured to: obtain a first feature corresponding to a current image block through a first neural network, wherein the first neural network includes at least one convolutional layer;

[0021] Acquire a feature adjustment factor corresponding to the current image block based on the first feature;

[0022] The feature adjustment factor is encoded in an auxiliary code stream corresponding to the current image block.

[0023] The present application provides a decoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0024] The processor is used to execute machine executable instructions to implement the above decoding method.

[0025] The present application provides an encoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0026] The processor is used to execute machine executable instructions to implement the above encoding method.

[0027] The present application provides an electronic device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-mentioned decoding method; or, the processor is used to execute the machine-executable instructions to implement the above-mentioned encoding method.

[0028] The present application provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the above-mentioned decoding method is implemented; or, the above-mentioned encoding method is implemented.

[0029] It can be seen from the above technical solutions that in the embodiment of the present application, the first feature corresponding to the current image block is obtained through a neural network, and the first feature is adjusted based on the feature adjustment factor corresponding to the current image block to obtain the target feature. Based on the target feature, the reconstructed image block corresponding to the current image block is obtained through a neural network, thereby proposing an end-to-end video image compression method, which can realize the encoding and decoding of video images based on a neural network, and achieve the purpose of improving the encoding efficiency and decoding efficiency by combining the feature adjustment factor. Combined with the network structure design and the auxiliary code stream, the neural network can effectively guarantee the quality of the reconstructed image block while maintaining low complexity, thereby achieving the purpose of improving the encoding performance and decoding performance, and reducing the complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a schematic diagram of a three-dimensional feature matrix in one embodiment of the present application;

[0031] Figure 2 is a flowchart of a decoding method in one embodiment of the present application;

[0032] Figure 3 is a flowchart of an encoding method in one embodiment of the present application;

[0033] Figure 4 is a schematic diagram of a processing process of an encoding end in one implementation manner of the present application;

[0034] Figure 5 is a schematic diagram of a processing process of a decoding end in one embodiment of the present application;

[0035] Figure 6A-6H This is a schematic diagram of a network structure in one implementation of the present application;

[0036] Fig. 7A is a schematic diagram of a processing process of a decoding end in one embodiment of the present application;

[0037] Figure 7B is a schematic diagram of a processing process of an encoding end in one implementation manner of the present application;

[0038] Fig. 8A and Figure 8B is a schematic diagram of the structure of a decoding subnetwork in one embodiment of the present application;

[0039] Fig.9A and Fig. 9B is a schematic diagram of the structure of the attention sub-network in one embodiment of the present application;

[0040] Figure 10A-10C is a schematic diagram of the adjustment position of the characteristic adjustment factor in one embodiment of the present application;

[0041] Fig.11A is a hardware structure diagram of a decoding end device in one embodiment of the present application;

[0042] Fig. 11B It is a hardware structure diagram of an encoding end device in one implementation of the present application. DETAILED DESCRIPTION

[0043] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than for limiting the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items. It should be understood that, although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, depending on the context. In addition, the word "if" used can be interpreted as "at ... ", or "when ... ", or "in response to determination".

[0044] In the embodiments of the present application, a decoding method and an encoding method are proposed, which may involve the following concepts:

[0045] JPEG (Joint Photographic Experts Group): JPEG is a standard for continuous tone static image compression. The file extension can be .jpg or .jpeg. It is a commonly used image file format. JPEG uses a joint coding method of predictive coding (such as DPCM, Differential pulse-code modulation), discrete cosine transform (discrete cosine transform, DCT) and entropy coding to remove redundant image and color data. It is a lossy compression format that can compress images in a very small storage space, which will cause damage to image data to a certain extent. In particular, when using too high a compression ratio, the quality of the image restored after the final decompression will be reduced. If you pursue high-quality images, JPEG should not use too high a compression ratio.

[0046] JPEG-AI (Joint Photographic Experts Group Artificial Intelligence): The scope of JPEG-AI is to create a learning-based image coding standard that provides a single-stream, compact compressed domain representation that significantly improves compression efficiency over commonly used image coding standards at the same subjective quality, effectively improving performance in image processing and computer vision tasks. JPEG-AI targets a wide range of applications such as cloud storage, vision management, self-driving cars and devices, image acquisition storage and management, real-time management of visual data, and media distribution. The goal of JPEG-AI is to design a codec solution that needs to significantly improve compression efficiency at the same subjective quality and provide efficient compressed domain processing for machine learning-based image processing and computer vision tasks. JPEG-AI requires hardware- and software-friendly encoding and decoding, support for 8-bit and 10-bit depths, and efficient encoding and progressive decoding of images with text and graphics.

[0047] Entropy coding: Entropy coding is the coding process that does not lose any information according to the entropy principle. Information entropy is the average amount of information of the source (a measure of uncertainty). Entropy coding methods may include but are not limited to: Shannon coding, Huffman coding, and arithmetic coding.

[0048] Neural Network (NN): Neural network refers to artificial neural network. Neural network is a computing model composed of a large number of nodes (or neurons) connected to each other. In a neural network, neuron processing units can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. The types of processing units in a neural network can be divided into three categories: input units, output units, and hidden units. Input units receive signals and data from the outside world; output units realize the output of processing results; hidden units are units between input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units, and the representation and processing of information are reflected in the connection relationship of processing units. Neural network is a non-programmed, brain-like information processing method. The essence of neural network is to obtain a parallel distributed information processing function through the transformation and dynamic behavior of neural network, and imitate the information processing function of human brain nervous system to different degrees and levels. In the field of video processing, commonly used neural networks may include but are not limited to: convolutional neural network (CNN), recurrent neural network (RNN), fully connected network, etc.

[0049] Convolutional Neural Network (CNN): Convolutional neural network is a feedforward neural network and one of the most representative network structures in deep learning technology. The artificial neurons of convolutional neural network can respond to surrounding units within a certain coverage range and have excellent performance in large-scale image processing. The basic structure of convolutional neural network includes two layers. One is the feature extraction layer (also called convolution layer). The input of each neuron is connected to the local receptive field of the previous layer and extracts the local features. Once the local feature is extracted, its positional relationship with other features is also determined. The second is the feature mapping layer (also called activation layer). Each computing layer of the neural network consists of multiple feature maps. Each feature map is a plane, and the weights of all neurons on the plane are equal. The feature mapping structure can use Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, GDN function, etc. as the activation function of the convolutional network. In addition, since the neurons on a mapping surface share weights, the number of free parameters of the network is reduced.

[0050] For example, one of the advantages of convolutional neural networks over image processing algorithms is that they avoid the complex pre-processing of images (extracting artificial features, etc.), and can directly input the original image for end-to-end learning. One of the advantages of convolutional neural networks over ordinary neural networks is that ordinary neural networks use a fully connected approach, that is, all neurons from the input layer to the hidden layer are connected, which will result in a huge number of parameters, making network training time-consuming or even difficult to train, while convolutional neural networks avoid this difficulty through local connections, weight sharing and other methods.

[0051] Deconvolution layer: Deconvolution layer is also called transposed convolution layer. The working process of deconvolution layer and convolution layer is very similar. The main difference is that deconvolution layer will make the output larger than the input through padding (of course, it can also remain the same). If stride is 1, it means that the output size is equal to the input size; if stride is N, it means that the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0052] Generalization Ability: Generalization ability refers to the ability of a machine learning algorithm to adapt to fresh samples. The purpose of learning is to learn the rules hidden behind the data pairs. For data outside the learning set with the same rules, the trained network can also give appropriate outputs. This ability can be called generalization ability.

[0053] Feature: The feature involved in this application is a three-dimensional feature matrix of C*W*H, see Figure 1 As shown, it is a schematic diagram of a three-dimensional feature matrix. In the three-dimensional feature matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be the input of a neural network or the output of a neural network.

[0054] Rate-Distortion Optimized: There are two major indicators for evaluating coding efficiency: bit rate and PSNR (Peak Signal to Noise Ratio). The smaller the bit stream, the greater the compression rate, and the greater the PSNR, the better the quality of the reconstructed image. When selecting a mode, the discriminant formula is essentially a comprehensive evaluation of the two. For example, the cost corresponding to the mode: J(mode) = D + λ*R, where D represents Distortion, which can usually be measured using the SSE indicator. SSE refers to the mean square sum of the difference between the reconstructed image block and the source image. In order to achieve cost considerations, the SAD indicator can also be used. SAD refers to the sum of the absolute values ​​of the difference between the reconstructed image block and the source image; λ is the Lagrange multiplier, and R is the actual number of bits required for encoding the image block in this mode, including the sum of bits required for encoding mode information, motion information, residuals, etc. When selecting a mode, if the rate-distortion principle is used to make a comparison decision on the encoding mode, the best encoding performance can usually be guaranteed.

[0055] A large number of coding tools have been proposed for each module of the encoding end, and each tool often has multiple modes. For different video sequences, the coding tools that can achieve the best coding performance are often different. Therefore, in the encoding process, RDO (Rate-Distortion Optimize) is usually used to compare the coding performance of different tools or modes to select the best mode. After determining the optimal tool or mode, the decision information of the tool or mode is transmitted by encoding the marker information in the bitstream. Although this method brings higher coding complexity, it can adaptively select the optimal mode combination for different content to obtain the best coding performance. The decoding end can obtain relevant mode information by directly parsing the flag information, and the complexity impact is small.

[0056] The decoding method and encoding method in the embodiments of the present application are described in detail below in combination with several specific embodiments.

[0057] Example 1: A decoding method is proposed in the embodiment of the present application, see Figure 2 As shown, it is a flowchart of the decoding method, which can be applied to a decoding end (also called a video decoder), and the method may include:

[0058] Step 201: obtaining a first feature corresponding to a current image block through a first neural network, and obtaining a feature adjustment factor corresponding to the current image block. The first neural network may include at least one convolutional layer.

[0059] In a possible implementation, obtaining the first feature corresponding to the current image block through the first neural network may include but is not limited to: obtaining probability distribution parameters based on the first code stream corresponding to the current image block; determining a probability distribution model based on the probability distribution parameters, and decoding the second code stream corresponding to the current image block based on the probability distribution model to obtain decoded image features; and determining the first feature corresponding to the current image block based on the decoded image features. In summary, it can be seen that in this implementation, the first neural network is used to realize functions such as obtaining probability distribution parameters, determining a probability distribution model, decoding the second code stream corresponding to the current image block, and determining the first feature corresponding to the current image block.

[0060] In another possible implementation, obtaining the first feature corresponding to the current image block through the first neural network may include but is not limited to: obtaining probability distribution parameters and prediction values ​​based on the first code stream corresponding to the current image block; determining a probability distribution model based on the probability distribution parameters, and decoding the second code stream corresponding to the current image block based on the probability distribution model to obtain decoded image features; performing residual recovery on the decoded image features to obtain residual features; and determining the first feature corresponding to the current image block based on the residual features and the prediction values. In summary, it can be seen that in this implementation, the first neural network is used to realize functions such as obtaining probability distribution parameters, obtaining prediction values, determining a probability distribution model, decoding the second code stream corresponding to the current image block, performing residual recovery, and determining the first feature corresponding to the current image block.

[0061] Exemplarily, obtaining the feature adjustment factor corresponding to the current image block may include, but is not limited to: decoding the auxiliary code stream corresponding to the current image block to obtain the feature adjustment factor corresponding to the current image block, that is, parsing the feature adjustment factor corresponding to the current image block from the auxiliary code stream. Alternatively, a fixed parameter value is determined as the feature adjustment factor corresponding to the current image block, for example, a fixed parameter value of 1 may be determined as the feature adjustment factor corresponding to the current image block.

[0062] Step 202: Determine a target feature based on the first feature and the feature adjustment factor.

[0063] Exemplarily, determining the target feature based on the first feature and the feature adjustment factor may include, but is not limited to: enhancing the first feature to obtain a second feature; after obtaining the second feature, adjusting the second feature based on the feature adjustment factor to obtain a third feature; after obtaining the third feature, determining the target feature based on the third feature.

[0064] In one possible implementation, the first feature is enhanced to obtain the second feature, which may include but is not limited to: determining the initial feature corresponding to the attention subnetwork based on the first feature; and enhancing the initial feature through the attention subnetwork to obtain the second feature. After obtaining the second feature, the second feature can be feature adjusted based on the feature adjustment factor to obtain a third feature (i.e., the feature after feature adjustment is used as the third feature). After obtaining the third feature, the target feature is determined based on the third feature, which may include but is not limited to: determining the third feature as the target feature.

[0065] Exemplarily, the attention subnetwork may include a residual enhancement subnetwork and a weight generation subnetwork. The initial feature is enhanced by the attention subnetwork to obtain a second feature, which may include but is not limited to: using the residual enhancement subnetwork to enhance the initial feature to obtain an enhanced feature, and using the weight generation subnetwork to generate a weight feature corresponding to the initial feature, and generating a second feature based on the initial feature, the enhanced feature and the weight feature.

[0066] Exemplarily, the weight generation subnetwork may include a residual block subnetwork, a convolution subnetwork and a feature mapping subnetwork. The weight generation subnetwork is used to generate weight features corresponding to the initial features, which may include but is not limited to: using the residual block subnetwork to perform convolution activation on the initial features to obtain convolution activated features; using the convolution subnetwork to convolve the convolution activated features to obtain convolution features; using the feature mapping subnetwork to perform feature mapping on the convolution features to obtain weight features.

[0067] Exemplarily, determining the initial feature corresponding to the attention sub-network based on the first feature may include but is not limited to: using an initial enhancement sub-network to perform feature enhancement on the first feature to obtain an enhanced feature, and using an upsampling convolution sub-network to upsample and convolve the enhanced feature to obtain the initial feature corresponding to the attention sub-network.

[0068] In another possible implementation, the first feature is feature enhanced to obtain the second feature, which may include but is not limited to: determining the initial feature corresponding to the attention subnetwork based on the first feature; and performing feature enhancement on the initial feature through the first subnetwork in the attention subnetwork to obtain the second feature. After obtaining the second feature, the second feature can be feature adjusted based on the feature adjustment factor to obtain a third feature (i.e., the feature after feature adjustment is used as the third feature). After obtaining the third feature, the target feature is determined based on the third feature, which may include but is not limited to: processing the third feature through the second subnetwork in the attention subnetwork to obtain a fourth feature, and determining the fourth feature as the target feature.

[0069] Exemplarily, the first sub-network in the attention sub-network performs feature enhancement on the initial feature to obtain the second feature, which may include but is not limited to: using the residual enhancement sub-network to perform feature enhancement on the initial feature to obtain the enhanced feature, and using the weight generation sub-network to generate the weight feature corresponding to the initial feature, and generating the second feature based on the enhanced feature and the weight feature. The third feature is processed by the second sub-network in the attention sub-network to obtain the fourth feature, which may include but is not limited to: generating the fourth feature based on the initial feature and the third feature.

[0070] The weight generation subnetwork may include a residual block subnetwork, a convolution subnetwork and a feature mapping subnetwork. The weight generation subnetwork is used to generate weight features corresponding to the initial features, which may include but is not limited to: using the residual block subnetwork to perform convolution activation on the initial features to obtain convolution activated features; using the convolution subnetwork to convolve the convolution activated features to obtain convolution features; using the feature mapping subnetwork to perform feature mapping on the convolution features to obtain weight features.

[0071] Exemplarily, the first sub-network in the attention sub-network performs feature enhancement on the initial feature to obtain the second feature, which may include but is not limited to: using the residual block sub-network to perform feature enhancement on the initial feature to obtain the enhanced feature, using the convolution sub-network to convolve the enhanced feature to obtain the convolution feature, and determining the second feature based on the convolution feature. The second sub-network in the attention sub-network processes the third feature to obtain the fourth feature, which may include but is not limited to: using the feature mapping sub-network to perform feature mapping on the third feature to obtain the weight feature; using the residual enhancement sub-network to perform feature enhancement on the initial feature to obtain the enhanced feature; generating the fourth feature based on the initial feature, the enhanced feature and the weight feature.

[0072] In the above embodiment, determining the initial feature corresponding to the attention sub-network based on the first feature may include but is not limited to: using the initial enhancement sub-network to perform feature enhancement on the first feature to obtain the enhanced feature, and using the upsampling convolution sub-network to upsample and convolve the enhanced feature to obtain the initial feature corresponding to the attention sub-network.

[0073] In a possible implementation, the second feature is feature adjusted based on the feature adjustment factor to obtain the third feature, which may include but is not limited to: if the second feature includes C*H*W feature values, C represents the number of channels, H represents the feature height, and W represents the feature width, then the feature adjustment values ​​corresponding to the C*H*W feature values ​​are determined based on the feature adjustment factor, and the C*H*W feature values ​​are feature adjusted based on the feature adjustment value to obtain the adjusted feature values ​​corresponding to the C*H*W feature values. On this basis, the third feature can be generated based on the adjusted feature values ​​corresponding to the C*H*W feature values.

[0074] Exemplarily, determining feature adjustment values ​​corresponding to C*H*W eigenvalues ​​based on feature adjustment factors may include but is not limited to: if the feature adjustment factor includes C*H*W feature adjustment values, then determining the feature adjustment values ​​corresponding to the C*H*W eigenvalues ​​respectively based on the C*H*W feature adjustment values; or, if the feature adjustment factor includes H*W feature adjustment values, then determining the feature adjustment values ​​corresponding to the C*H*W eigenvalues ​​respectively based on the H*W feature adjustment values; or, if the feature adjustment factor includes C feature adjustment values, then determining the feature adjustment values ​​corresponding to the C*H*W eigenvalues ​​respectively based on the C feature adjustment values.

[0075] Exemplarily, feature adjustment is performed on C*H*W feature values ​​based on feature adjustment values ​​to obtain adjusted feature values ​​corresponding to the C*H*W feature values, which may include but is not limited to: one feature value among the C*H*W feature values ​​corresponds to a feature adjustment value, and the adjusted feature value corresponding to the feature value may be determined by the following formula: yr=rf*y; wherein yr represents the adjusted feature value, rf represents the feature adjustment value, and y represents the feature value. Alternatively, if one feature value among the C*H*W feature values ​​corresponds to N+1 feature adjustment values, and N is a positive integer, the adjusted feature value corresponding to the feature value may be determined by the following formula: yr=rf_0*y0+rf_1*y1+rf_2*y2+…+rf_N*yN; wherein yr represents the adjusted feature value, rf_0, rf_1, rf_2, …, rf_N represent N+1 feature adjustment values, and y represents the feature value.

[0076] Step 203: Based on the target feature, a reconstructed image block corresponding to the current image block (i.e., the reconstructed image block outputted finally) is obtained through a second neural network, and the second neural network may include at least one convolutional layer.

[0077] Exemplarily, the second neural network may include a reconstruction decoding subnetwork, and based on the target feature, a reconstructed image block corresponding to the current image block is obtained through the second neural network, which may include but is not limited to: processing the target feature through the reconstruction decoding subnetwork to obtain the reconstructed image block corresponding to the current image block. For example, the reconstruction decoding subnetwork may include an upsampling convolution subnetwork, and the target feature may be upsampled and convolved through the upsampling convolution subnetwork to obtain the reconstructed image block corresponding to the current image block. For another example, the reconstruction decoding subnetwork may include an upsampling convolution subnetwork and a color space transformation subnetwork, and the target feature may be upsampled and convolved through the upsampling convolution subnetwork, and the upsampled and convolved feature may be transformed in color space through the color space transformation subnetwork to obtain the reconstructed image block corresponding to the current image block.

[0078] In a possible implementation, if the feature adjustment mode is enabled for the current image block, the feature adjustment factor corresponding to the current image block is obtained, and the target feature is determined based on the first feature and the feature adjustment factor. Alternatively, if the feature adjustment mode is not enabled for the current image block, the target feature is determined based on the first feature, and the feature adjustment factor corresponding to the current image block does not need to be obtained.

[0079] Exemplarily, a feature adjustment flag may be parsed from the code stream. If the feature adjustment flag allows the current image block to enable the feature adjustment mode, it may be determined that the current image block enables the feature adjustment mode. Alternatively, a first value range of a feature value may be parsed from the code stream. If the feature value of a second feature is within the first value range, it may be determined that the current image block enables the feature adjustment mode. Alternatively, a second value range of a feature change value may be parsed from the code stream. If the feature change value corresponding to the second feature is within the second value range, it may be determined that the current image block enables the feature adjustment mode.

[0080] Exemplarily, if the attribute value corresponding to the second feature is within the configured attribute value range, it can be determined that the feature adjustment mode is enabled for the current image block; wherein the attribute value may include a variance value.

[0081] Exemplarily, the above execution order is only for the convenience of describing the examples given. In practical applications, the execution order between the steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0082] It can be seen from the above technical solutions that in the embodiment of the present application, the first feature corresponding to the current image block is obtained through the first neural network, and the first feature is adjusted based on the feature adjustment factor corresponding to the current image block to obtain the target feature. Based on the target feature, the reconstructed image block corresponding to the current image block is obtained through the second neural network, thereby proposing an end-to-end video image compression method, which can realize the decoding of video images based on neural networks, and achieve the purpose of improving decoding efficiency by combining feature adjustment factors. Combined with network structure design and auxiliary code stream, the neural network can effectively guarantee the quality of the reconstructed image block while maintaining low complexity, thereby achieving the purpose of improving decoding performance and reducing complexity.

[0083] Example 2: This application proposes an encoding method, see Figure 3As shown, it is a schematic diagram of the process of the encoding method, which can be applied to an encoding end (also called a video encoder), and the method may include:

[0084] Step 301: Acquire a first feature corresponding to a current image block through a first neural network, where the first neural network may include at least one convolutional layer.

[0085] In a possible implementation, obtaining the first feature corresponding to the current image block through the first neural network may include but is not limited to: obtaining probability distribution parameters based on the first code stream corresponding to the current image block; determining a probability distribution model based on the probability distribution parameters, and decoding the second code stream corresponding to the current image block based on the probability distribution model to obtain decoded image features; and determining the first feature corresponding to the current image block based on the decoded image features. In summary, it can be seen that in this implementation, the first neural network is used to realize functions such as obtaining probability distribution parameters, determining a probability distribution model, decoding the second code stream corresponding to the current image block, and determining the first feature corresponding to the current image block.

[0086] In another possible implementation, obtaining the first feature corresponding to the current image block through the first neural network may include but is not limited to: obtaining probability distribution parameters and prediction values ​​based on the first code stream corresponding to the current image block; determining a probability distribution model based on the probability distribution parameters, and decoding the second code stream corresponding to the current image block based on the probability distribution model to obtain decoded image features; performing residual recovery on the decoded image features to obtain residual features; and determining the first feature corresponding to the current image block based on the residual features and the prediction values. In summary, it can be seen that in this implementation, the first neural network is used to realize functions such as obtaining probability distribution parameters, obtaining prediction values, determining a probability distribution model, decoding the second code stream corresponding to the current image block, performing residual recovery, and determining the first feature corresponding to the current image block.

[0087] Step 302: Obtain a feature adjustment factor corresponding to the current image block based on the first feature.

[0088] Exemplarily, obtaining a feature adjustment factor corresponding to the current image block based on the first feature may include, but is not limited to: obtaining at least one candidate feature adjustment factor, and determining a rate-distortion cost corresponding to each candidate feature adjustment factor based on the first feature. Based on the rate-distortion cost corresponding to each candidate feature adjustment factor, a candidate feature adjustment factor may be selected from all candidate feature adjustment factors as the feature adjustment factor corresponding to the current image block.

[0089] Step 303: Encode the feature adjustment factor in the auxiliary bitstream corresponding to the current image block.

[0090] Exemplarily, a fixed parameter value may be determined as the feature adjustment factor corresponding to the current image block, for example, a fixed parameter value of 1 may be determined as the feature adjustment factor corresponding to the current image block. In this case, the feature adjustment factor may not be encoded in the auxiliary bitstream corresponding to the current image block.

[0091] Exemplarily, the target feature can also be determined based on the first feature corresponding to the current image block and the feature adjustment factor corresponding to the current image block. Based on the target feature, the reconstructed image block corresponding to the current image block (i.e., the reconstructed image block finally output) can be obtained through a second neural network. The second neural network can include at least one convolutional layer.

[0092] Exemplarily, determining the target feature based on the first feature and the feature adjustment factor may include, but is not limited to: enhancing the first feature to obtain a second feature; after obtaining the second feature, adjusting the second feature based on the feature adjustment factor to obtain a third feature; after obtaining the third feature, determining the target feature based on the third feature.

[0093] In one possible implementation, the first feature is enhanced to obtain the second feature, which may include but is not limited to: determining the initial feature corresponding to the attention subnetwork based on the first feature; and enhancing the initial feature through the attention subnetwork to obtain the second feature. After obtaining the second feature, the second feature can be feature adjusted based on the feature adjustment factor to obtain a third feature (i.e., the feature after feature adjustment is used as the third feature). After obtaining the third feature, the target feature is determined based on the third feature, which may include but is not limited to: determining the third feature as the target feature.

[0094] Exemplarily, the attention subnetwork may include a residual enhancement subnetwork and a weight generation subnetwork. The initial feature is enhanced by the attention subnetwork to obtain a second feature, which may include but is not limited to: using the residual enhancement subnetwork to enhance the initial feature to obtain an enhanced feature, and using the weight generation subnetwork to generate a weight feature corresponding to the initial feature, and generating a second feature based on the initial feature, the enhanced feature and the weight feature.

[0095] Exemplarily, the weight generation subnetwork may include a residual block subnetwork, a convolution subnetwork and a feature mapping subnetwork. The weight generation subnetwork is used to generate weight features corresponding to the initial features, which may include but is not limited to: using the residual block subnetwork to perform convolution activation on the initial features to obtain convolution activated features; using the convolution subnetwork to convolve the convolution activated features to obtain convolution features; using the feature mapping subnetwork to perform feature mapping on the convolution features to obtain weight features.

[0096] Exemplarily, determining the initial feature corresponding to the attention sub-network based on the first feature may include but is not limited to: using an initial enhancement sub-network to perform feature enhancement on the first feature to obtain an enhanced feature, and using an upsampling convolution sub-network to upsample and convolve the enhanced feature to obtain the initial feature corresponding to the attention sub-network.

[0097] In another possible implementation, the first feature is feature enhanced to obtain the second feature, which may include but is not limited to: determining the initial feature corresponding to the attention subnetwork based on the first feature; and performing feature enhancement on the initial feature through the first subnetwork in the attention subnetwork to obtain the second feature. After obtaining the second feature, the second feature can be feature adjusted based on the feature adjustment factor to obtain a third feature (i.e., the feature after feature adjustment is used as the third feature). After obtaining the third feature, the target feature is determined based on the third feature, which may include but is not limited to: processing the third feature through the second subnetwork in the attention subnetwork to obtain a fourth feature, and determining the fourth feature as the target feature.

[0098] Exemplarily, the first sub-network in the attention sub-network performs feature enhancement on the initial feature to obtain the second feature, which may include but is not limited to: using the residual enhancement sub-network to perform feature enhancement on the initial feature to obtain the enhanced feature, and using the weight generation sub-network to generate the weight feature corresponding to the initial feature, and generating the second feature based on the enhanced feature and the weight feature. The third feature is processed by the second sub-network in the attention sub-network to obtain the fourth feature, which may include but is not limited to: generating the fourth feature based on the initial feature and the third feature.

[0099] The weight generation subnetwork may include a residual block subnetwork, a convolution subnetwork and a feature mapping subnetwork. The weight generation subnetwork is used to generate weight features corresponding to the initial features, which may include but is not limited to: using the residual block subnetwork to perform convolution activation on the initial features to obtain convolution activated features; using the convolution subnetwork to convolve the convolution activated features to obtain convolution features; using the feature mapping subnetwork to perform feature mapping on the convolution features to obtain weight features.

[0100] Exemplarily, the first sub-network in the attention sub-network performs feature enhancement on the initial feature to obtain the second feature, which may include but is not limited to: using the residual block sub-network to perform feature enhancement on the initial feature to obtain the enhanced feature, using the convolution sub-network to convolve the enhanced feature to obtain the convolution feature, and determining the second feature based on the convolution feature. The second sub-network in the attention sub-network processes the third feature to obtain the fourth feature, which may include but is not limited to: using the feature mapping sub-network to perform feature mapping on the third feature to obtain the weight feature; using the residual enhancement sub-network to perform feature enhancement on the initial feature to obtain the enhanced feature; generating the fourth feature based on the initial feature, the enhanced feature and the weight feature.

[0101] In the above embodiment, determining the initial feature corresponding to the attention sub-network based on the first feature may include but is not limited to: using the initial enhancement sub-network to perform feature enhancement on the first feature to obtain the enhanced feature, and using the upsampling convolution sub-network to upsample and convolve the enhanced feature to obtain the initial feature corresponding to the attention sub-network.

[0102] In a possible implementation, the second feature is feature adjusted based on the feature adjustment factor to obtain the third feature, which may include but is not limited to: if the second feature includes C*H*W feature values, C represents the number of channels, H represents the feature height, and W represents the feature width, then the feature adjustment values ​​corresponding to the C*H*W feature values ​​are determined based on the feature adjustment factor, and the C*H*W feature values ​​are feature adjusted based on the feature adjustment value to obtain the adjusted feature values ​​corresponding to the C*H*W feature values. On this basis, the third feature can be generated based on the adjusted feature values ​​corresponding to the C*H*W feature values.

[0103] Exemplarily, determining feature adjustment values ​​corresponding to C*H*W eigenvalues ​​based on feature adjustment factors may include but is not limited to: if the feature adjustment factor includes C*H*W feature adjustment values, then determining the feature adjustment values ​​corresponding to the C*H*W eigenvalues ​​respectively based on the C*H*W feature adjustment values; or, if the feature adjustment factor includes H*W feature adjustment values, then determining the feature adjustment values ​​corresponding to the C*H*W eigenvalues ​​respectively based on the H*W feature adjustment values; or, if the feature adjustment factor includes C feature adjustment values, then determining the feature adjustment values ​​corresponding to the C*H*W eigenvalues ​​respectively based on the C feature adjustment values.

[0104] Exemplarily, feature adjustment is performed on C*H*W feature values ​​based on feature adjustment values ​​to obtain adjusted feature values ​​corresponding to the C*H*W feature values, which may include but is not limited to: if one feature value among the C*H*W feature values ​​corresponds to one feature adjustment value, the adjusted feature value corresponding to the feature value may be determined by the following formula: yr=rf*y; wherein yr represents the adjusted feature value, rf represents the feature adjustment value, and y represents the feature value. Alternatively, if one feature value among the C*H*W feature values ​​corresponds to N+1 feature adjustment values, and N is a positive integer, the adjusted feature value corresponding to the feature value may be determined by the following formula: yr=rf_0*y0+rf_1*y1+rf_2*y2+…+rf_N*yN; wherein yr represents the adjusted feature value, rf_0, rf_1, rf_2, …, rf_N represent N+1 feature adjustment values, and y represents the feature value.

[0105] Exemplarily, the second neural network may include a reconstruction decoding subnetwork, and based on the target feature, a reconstructed image block corresponding to the current image block is obtained through the second neural network, which may include but is not limited to: processing the target feature through the reconstruction decoding subnetwork to obtain the reconstructed image block corresponding to the current image block. For example, the reconstruction decoding subnetwork may include an upsampling convolution subnetwork, and the target feature may be upsampled and convolved through the upsampling convolution subnetwork to obtain the reconstructed image block corresponding to the current image block. For another example, the reconstruction decoding subnetwork may include an upsampling convolution subnetwork and a color space transformation subnetwork, and the target feature may be upsampled and convolved through the upsampling convolution subnetwork, and the upsampled and convolved feature may be transformed in color space through the color space transformation subnetwork to obtain the reconstructed image block corresponding to the current image block.

[0106] In a possible implementation, if the feature adjustment mode is enabled for the current image block, a feature adjustment factor corresponding to the current image block is obtained based on the first feature, and the feature adjustment factor is encoded in an auxiliary bitstream corresponding to the current image block. Alternatively, if the feature adjustment mode is not enabled for the current image block, it is not necessary to obtain the feature adjustment factor corresponding to the current image block.

[0107] Exemplarily, a feature adjustment flag may be encoded in the code stream, the feature adjustment flag allowing the current image block to enable the feature adjustment mode, or the feature adjustment flag prohibiting the current image block from enabling the feature adjustment mode. Alternatively, a first value range of the feature value may be encoded in the code stream. Alternatively, a second value range of the feature change value may be encoded in the code stream.

[0108] Exemplarily, the above execution order is only for the convenience of describing the examples given. In practical applications, the execution order between the steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0109] As can be seen from the above technical solutions, in the embodiment of the present application, an end-to-end video image compression method is proposed, which can realize the encoding of video images based on a neural network, and achieve the purpose of improving the encoding efficiency by combining the feature adjustment factor. Combined with the network structure design and the auxiliary code stream (used to carry the feature adjustment factor), the neural network can effectively ensure the quality of the reconstructed image block while maintaining low complexity, thereby achieving the purpose of improving the encoding performance and reducing the complexity.

[0110] Embodiment 3: For Embodiment 1 and Embodiment 2, the processing process of the encoding end can be seen in Figure 4 As shown, of course, Figure 4 This is just an example of the processing process at the encoding end, and there is no limitation on the processing process at the encoding end.

[0111] After obtaining the current image block x (the current image block x may be the original image block x, i.e., the input image block), the encoder may analyze and transform the current image block x through an analysis transformation network (i.e., a neural network) to obtain an image feature y corresponding to the current image block x. Among them, performing feature transformation on the current image block x through the analysis transformation network means: transforming the current image block x to an image feature y in the latent domain, thereby facilitating all subsequent processes to operate on the latent domain.

[0112] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the encoding and decoding process for the image block can also be directly used for the image.

[0113] After obtaining the image feature y, the encoding end performs coefficient hyperparameter feature transformation on the image feature y to obtain coefficient hyperparameter feature z. For example, the image feature y can be input into a hyperparameter encoding network (i.e., a neural network), and the hyperparameter encoding network performs coefficient hyperparameter feature transformation on the image feature y to obtain coefficient hyperparameter feature z. The hyperparameter encoding network can be a trained neural network, and there is no restriction on the training process of the hyperparameter encoding network, as long as the coefficient hyperparameter feature transformation can be performed on the image feature y. The image feature y in the latent domain obtains the super prior latent information z after passing through the hyperparameter encoding network.

[0114] After obtaining the coefficient hyperparameter feature z, the encoder can quantize the coefficient hyperparameter feature z to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, that is, Figure 4 The Q operation in is the quantization process. After obtaining the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, the hyperparameter quantization feature is encoded to obtain Bitstream#1 (i.e., the first bitstream) corresponding to the current image block, i.e. Figure 4 The AE operation in represents the encoding process, such as the entropy encoding process. Alternatively, the encoding end can also directly encode the coefficient hyperparameter feature z to obtain the Bitstream#1 corresponding to the current image block. Among them, the hyperparameter quantization feature or coefficient hyperparameter feature z carried in Bitstream#1 is mainly used to obtain the parameters of the mean and probability distribution model.

[0115] After obtaining the Bitstream#1 corresponding to the current image block, the encoder may send the Bitstream#1 corresponding to the current image block to the decoder. For the processing process of the decoder for the Bitstream#1 corresponding to the current image block, refer to the subsequent embodiments.

[0116] After obtaining the Bitstream#1 corresponding to the current image block, the encoder can also decode Bitstream#1 to obtain the hyperparameter quantization feature, that is, Figure 4 AD in it represents the decoding process. Then, the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat can be the same as or different from the coefficient hyperparameter feature z. Figure 4 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the encoder can also decode the Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0117] For the encoding process of Bitstream#1, the encoding method of the fixed probability density model can be adopted, and for the decoding process of Bitstream#1, the decoding method of the fixed probability density model can be adopted. There is no restriction on this encoding and decoding process.

[0118] After obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block (the determination process of the residual feature y_hat is described in the subsequent embodiments) to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat are input to the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu. The prediction value mu is used to obtain the residual by subtracting from the original feature and to obtain the reconstructed y by adding the decoded residual.

[0119] It should be noted that the mean prediction network is an optional neural network, that is, there is no mean prediction network, that is, there is no need to determine the predicted value mu through the mean prediction network. Figure 4 The dashed box in indicates that the mean prediction network is optional.

[0120] After obtaining the image feature y, the encoder can determine the residual feature r based on the image feature y and the prediction value mu, such as taking the difference between the image feature y and the prediction value mu as the residual feature r. Then, the residual feature r is feature processed to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, it is necessary to deploy a mean prediction network, and the mean prediction network provides the prediction value mu. Alternatively, after obtaining the image feature y, the encoder can perform feature processing on the image feature y to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, there is no need to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0121] After obtaining the image feature s, the encoder can quantize the image feature s to obtain the image quantization feature corresponding to the image feature s, that is, Figure 4 The Q operation in is the quantization process. After obtaining the image quantization feature corresponding to the image feature s, the encoding end can encode the image quantization feature to obtain Bitstream#2 (i.e., the second bitstream) corresponding to the current image block, i.e. Figure 4The AE operation in represents an encoding process, such as an entropy encoding process. Alternatively, the encoding end may directly encode the image feature s to obtain Bitstream#2 corresponding to the current image block without involving the quantization process of the image feature s.

[0122] After obtaining Bitstream#2 corresponding to the current image block, the encoder may send Bitstream#2 corresponding to the current image block to the decoder. For the processing of Bitstream#2 corresponding to the current image block by the decoder, refer to the subsequent embodiments.

[0123] After obtaining the Bitstream#2 corresponding to the current image block, the encoder can also decode Bitstream#2 to obtain the image quantization feature, that is, Figure 4 AD in the figure represents the decoding process. Then, the encoder can dequantize the image quantization feature to obtain the image feature s'. The image feature s' can be the same as or different from the image feature s. Figure 4 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the encoder can also decode the Bitstream#2 to obtain the image feature s' without involving the inverse quantization process of the image quantization feature.

[0124] After obtaining the image feature s', the encoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing). There is no restriction on this feature recovery process, and it can be any feature recovery method to obtain the residual feature r_hat. The residual feature r_hat and the residual feature r can be the same or different. After obtaining the residual feature r_hat, the encoder determines the image feature y_hat based on the residual feature r_hat and the prediction value mu. The image feature y_hat and the image feature y can be the same or different. For example, the sum of the residual feature r_hat and the prediction value mu is used as the image feature y_hat. In this case, it is necessary to deploy a mean prediction network, and the mean prediction network provides the prediction value mu. Alternatively, after obtaining the image feature s', the encoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing) to obtain the image feature y_hat. The image feature y_hat and the image feature y can be the same or different. In this case, it is not necessary to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0125] After obtaining the image feature y_hat, the encoding end can perform a synthetic transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

[0126] In a possible implementation, when the encoding end encodes the image quantization feature or the image feature s to obtain the Bitstream#2 corresponding to the current image block, the encoding end needs to first determine the probability distribution model, and then encode the image quantization feature or the image feature s based on the probability distribution model. In addition, when the encoding end decodes Bitstream#2, it also needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model.

[0127] To obtain the probability distribution model, continue to refer to Figure 4 As shown, after obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input to the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probability hyperparameter decoding network can be a trained neural network, and there is no restriction on the training process of this probability hyperparameter decoding network, and it is sufficient to be able to perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat.

[0128] In a possible implementation, the processing process at the encoding end may be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process, without any limitation on the encoding process.

[0129] Embodiment 4: For Embodiment 1 and Embodiment 2, the processing process of the decoding end can be seen in Figure 5 As shown, of course, Figure 5 This is just an example of the processing process at the decoding end, and there is no limitation on the processing process at the decoding end.

[0130] After obtaining the Bitstream#1 corresponding to the current image block, the decoding end can also decode Bitstream#1 to obtain the hyperparameter quantization feature, that is, Figure 5 AD in it represents the decoding process. Then, the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat can be the same as or different from the coefficient hyperparameter feature z. Figure 5 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the decoder can also decode the Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0131] For the decoding process of Bitstream#1, a decoding method of a fixed probability density model may be used, and there is no restriction on this.

[0132] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the decoding process for the image block can also be directly used for the image.

[0133] After obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block (the determination process of the residual feature y_hat is described in the subsequent embodiments) to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat are input to the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu.

[0134] It should be noted that the mean prediction network is an optional neural network, that is, there is no mean prediction network, that is, there is no need to determine the predicted value mu through the mean prediction network. Figure 5 The dashed box in indicates that the mean prediction network is optional.

[0135] After obtaining the Bitstream#2 corresponding to the current image block, the decoding end can also decode Bitstream#2 to obtain the image quantization feature, that is, Figure 5 AD in the figure represents the decoding process. Then, the decoding end can dequantize the image quantization feature to obtain the image feature s'. The image feature s' can be the same as or different from the image feature s. Figure 5 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the decoder can also decode the Bitstream#2 to obtain the image feature s' without involving the inverse quantization process of the image quantization feature.

[0136] After obtaining the image feature s', the decoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing) to obtain the residual feature r_hat, which is the same as or different from the residual feature r. After obtaining the residual feature r_hat, the decoder determines the image feature y_hat based on the residual feature r_hat and the prediction value mu, and the image feature y_hat is the same as or different from the image feature y, such as taking the sum of the residual feature r_hat and the prediction value mu as the image feature y_hat. In this case, it is necessary to deploy a mean prediction network, and the prediction value mu is provided by the mean prediction network. Alternatively, after obtaining the image feature s', the decoder can perform feature recovery on the image feature s' to obtain the image feature y_hat, which is the same as or different from the image feature y. In this case, it is not necessary to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0137] After obtaining the image feature y_hat, the decoding end can perform a synthetic transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

[0138] In a possible implementation, when decoding Bitstream#2, the decoding end needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model. To obtain the probability distribution model, continue to refer to Figure 5 As shown, after obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input into the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probability hyperparameter decoding network can be a trained neural network, and there is no restriction on the training process of this probability hyperparameter decoding network. It can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p.

[0139] In a possible implementation, the processing process at the above decoding end can be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process, without limiting the decoding process.

[0140] Example 5: For Example 1, Example 2, Example 3 and Example 4, the encoding end involves an analysis transformation network, and the encoding end and the decoding end involve a synthesis transformation network, and both the analysis transformation network and the synthesis transformation network are neural networks. In a possible implementation, the structural diagram of the analysis transformation network can be seen in Fig. 6A As shown, the structural diagram of the synthetic transformation network can be found in Figure 6B As shown, of course, Fig. 6A and Figure 6B This is just an example. In this embodiment, there is no limitation on the structure of the analysis transformation network and the synthesis transformation network, as long as the analysis transformation function and the synthesis transformation function can be realized.

[0141] For the analysis transformation network, it includes Padding layer, Conv layer, ResAU layer, Padding layer, RNAB layer, Conv layer, ResAU layer, Padding layer, Conv layer, ResAU layer, Padding layer, Conv layer, Conv layer. Among them, Padding layer is a padding layer used to implement the feature edge expansion operation. Conv layer is a convolution layer used to implement the feature convolution operation. C*3*3 means that a 3*3 convolution kernel (C channels) is used to implement the feature convolution operation. C*1*1 means that a 1*1 convolution kernel (C channels) is used to implement the feature convolution operation. The downward arrow of 2 means that the feature is downsampled by 2 times.

[0142] The ResAU layer is an activation layer. The structural diagram of the ResAU layer can be found in Figure 6C As shown in the figure, LeakyReLU represents the activation operation, Conv 1*1 represents the convolution operation using a 1*1 convolution kernel to implement the feature, and tanh represents the hyperbolic tangent operation.

[0143] The RNAB layer is a residual non-local attention block. The structural diagram of the RNAB layer can be found in Fig.6D As shown, the RNAB layer may include a RB layer, a Conv layer, and a sigmoid layer. In the Conv layer, the upward arrow of 2 indicates that the feature is upsampled by a factor of 2. The sigmoid layer is used to perform feature mapping on the feature, such as using a sigmoid function to map the feature to between 0 and 1.

[0144] The RB (Residual Blocks) layer is a residual block. The structural diagram of the RB layer can be seen in Fig. 6E As shown, the RB layer may include a Conv layer, a LeakyReLU and a Conv layer, and a 3*3 convolution kernel is used in the Conv layer.

[0145] For the synthetic transformation network, it includes ResBlock layer (RB layer), ResBlock layer, Conv layer, Cropping layer, ResAU layer, Conv layer, Cropping layer, ResAU layer, Conv layer, RNAB layer, Cropping layer, ResAU layer, Conv layer, Cropping layer. Among them, the ResBlock layer is a residual block. The structural diagram of the ResBlock layer can be seen in Fig. 6E As shown in the figure. The Conv layer is a convolution layer, which is used to implement the convolution operation of the feature. C*3*3 means that a 3*3 convolution kernel (C channels) is used to implement the convolution operation of the feature. The upper arrow of 2 means that the feature is upsampled by 2 times. The Cropping layer is a cropping layer, which is used to implement the cropping operation of the feature, that is, the inverse operation of the Padding operation. The ResAU layer is an activation layer. The structural diagram of the ResAU layer can be seen in Figure 6C The RNAB layer is a residual non-local attention block. The structural diagram of the RNAB layer can be found in Fig.6D As shown, the RNAB layer may include a RB layer, a Conv layer, and a sigmoid layer.

[0146] Embodiment 6: For embodiment 5, from Figure 6B It can be seen that the synthetic transformation network mainly includes four up-sampled 3*3 convolution layers, 2 ResBlock layers (ResBlock layers include 2 3*3 convolution layers), three activation layers ResAU (ResAU includes 1 1*1 convolution layer), and 1 RNAB layer (RNAB layer includes 9 RB blocks, 3 3*3 convolution layers, that is, a total of 21 3*3 convolution layers). In summary, it can be seen that the synthetic transformation network has a total of 29 3*3 convolution layers and 3 1*1 convolution layers, and the RNAB layer accounts for nearly 70% of the complexity, that is, the complexity of the RNAB layer has a large optimization space. Based on this, in this embodiment, the synthetic transformation network can be optimized to design a low-complexity, high-performance synthetic transformation network. Alternatively, the analysis transformation network can also be optimized to design a low-complexity, high-performance analysis transformation network. Of course, other high-complexity network structures can also be optimized to design low-complexity, high-performance network structures, which is not limited in this embodiment. For the convenience of description, this embodiment takes the optimization of the synthetic transformation network as an example, for example, a complex network similar to the RNAB layer in the synthetic transformation network can be optimized. Of course, other network sides in the synthetic transformation network can also be optimized, and there is no restriction on this, as long as the complexity of the synthetic transformation network can be simplified.

[0147] In a possible implementation, for Fig.6DThe RNAB layer shown in the figure can include 9 RB layers, 3 Conv layers and 1 sigmoid layer. On this basis, some network layers can be removed to obtain the optimized RNAB layer. Fig. 6F As shown, three RB layers and two Conv layers can be removed to obtain an optimized RNAB layer. Of course, other network layers can also be removed to obtain an optimized RNAB layer, and there is no restriction on the removal method of this network layer.

[0148] For example, the optimized RNAB layer can be found in Figure 6G As shown, the RNAB layer may include a RB layer (consisting of at least one RB block, such as 3 consecutive RB blocks, and of course, the number of RB blocks may be more or less), a RB layer (consisting of at least one RB block, such as 3 consecutive RB blocks, and of course, the number of RB blocks may be more or less), 1 Conv layer (such as a 3*3 Conv layer) and 1 sigmoid layer (i.e., a feature mapping layer).

[0149] For another example, the 3*3 Conv layer can be replaced with a 1*1 Conv layer or a 5*5 Conv layer. The 1*1 Conv layer is used as an example. The optimized RNAB layer can be found in Figure 6H As shown, the RNAB layer may include a RB layer, a RB layer, a Conv layer (such as a 1*1 Conv layer) and a sigmoid layer (ie, a feature mapping layer).

[0150] Exemplarily, after obtaining the optimized RNAB layer, the optimized RNAB layer can be substituted into Figure 6B , and obtain the optimized synthesis transformation network. Based on the optimized synthesis transformation network, the image feature y_hat can be input into the synthesis transformation network, and the synthesis transformation network performs synthesis transformation on the image feature y_hat to obtain the reconstructed image block x_hat.

[0151] Embodiment 7: With respect to Embodiment 5, the synthesis transformation network can be optimized to design a low-complexity, high-performance synthesis transformation network. Alternatively, the analysis transformation network can be optimized to design a low-complexity, high-performance analysis transformation network. Of course, other high-complexity network structures can also be optimized to design low-complexity, high-performance network structures. In this embodiment, the method of "simplified network structure + adaptive adjustment factor" can be used for encoding and decoding.

[0152] First, simplify the complex network to obtain a simplified network. For example, the synthetic transformation network can be simplified to obtain a simplified synthetic transformation network. For example, the RNAB layer (i.e., the complex network) in the synthetic transformation network can be simplified to obtain a simplified synthetic transformation network. Of course, other networks can also be simplified, and there is no limitation to this.

[0153] Then, at least one layer of features (such as key features) of the simplified network is adjusted. For example, the simplified network is obtained based on a large number of image training, and the features it generates have a universal effect. However, for a specific frame of an image, its features are often not optimal. Therefore, it is necessary to design customized features for the image. In order to obtain this customized feature, it is necessary to introduce a feature adjustment factor in a certain process of the network, and adjust the features based on the feature adjustment factor to obtain the adjusted customized feature. Exemplarily, the encoder adaptively selects a feature adjustment factor for the current image block and encodes the feature adjustment factor in the bitstream, and the decoder can parse the feature adjustment factor from the bitstream.

[0154] In a possible implementation manner, for the decoding end, the decoding method of the decoding end can be referred to Fig. 7A As shown, the decoding end can receive the main code stream (i.e., the first code stream Bitstream#1 and the second code stream Bitstream#2) for the current image block. After the main code stream passes through coefficient decoding and the decoding sub-network, the first feature corresponding to the current image block can be obtained. The decoding end can decode the feature adjustment factor corresponding to the current image block from the auxiliary code stream. After obtaining the first feature corresponding to the current image block, the first feature can be enhanced through the feature enhancement network to obtain the second feature, and the feature adjustment factor corresponding to the current image block can be used to adjust the second feature to obtain the third feature. After obtaining the third feature, the target feature corresponding to the current image block can be determined based on the third feature, and the target feature can be input into the reconstruction decoding network, and the reconstruction decoding network processes the target feature to obtain the reconstructed image block corresponding to the current image block.

[0155] Exemplarily, the feature adjustment factor can be decoded by the decoding end from the auxiliary code stream, and the feature adjustment factor can also be a fixed parameter value. When the feature adjustment factor is a fixed parameter value, such as when the feature adjustment factor is a fixed parameter value of 1, it is equivalent to not performing feature adjustment on the second feature, that is, it does not involve a feature adjustment process, which is equivalent to the solution of Example 6.

[0156] Exemplarily, the decoding end can also decode a feature adjustment factor for the decoding subnetwork from the auxiliary bitstream (the same as or different from the feature adjustment factor for the second feature). After obtaining the feature adjustment factor for the decoding subnetwork, a feature in the decoding subnetwork can be adjusted based on the feature adjustment factor. For example, after the decoding subnetwork obtains feature A (feature A is any feature in the decoding subnetwork), the feature adjustment factor can be used to adjust feature A to obtain the adjusted feature B. The decoding subnetwork continues to process based on feature B and finally obtains the first feature. By using the feature adjustment factor in certain decoding processes of the decoding subnetwork, a first feature with less distortion can be generated.

[0157] In a possible implementation manner, for the encoding end, the encoding method of the encoding end can be referred to Figure 7B As shown, after the current image block passes through the coding network and coefficient coding, the main code stream corresponding to the current image block (i.e., the first code stream Bitstream#1 and the second code stream Bitstream#2) can be obtained, and the encoder can send the main code stream of the current image block to the decoder.

[0158] After the main code stream passes through coefficient decoding and the decoding sub-network, the first feature corresponding to the current image block can also be obtained. The feature adjustment factor corresponding to the current image block can be obtained based on the first feature, and the feature adjustment factor can be encoded in the auxiliary code stream corresponding to the current image block, so that the decoding end can decode the feature adjustment factor corresponding to the current image block from the auxiliary code stream.

[0159] Exemplarily, the encoding end may obtain at least one candidate feature adjustment factor (for example, a list of candidate feature adjustment factors may be pre-constructed, and all feature adjustment factors in the list of candidate feature adjustment factors may be used as candidate feature adjustment factors, or at least one candidate feature adjustment factor may be generated by a certain algorithm, without limitation). The rate-distortion cost corresponding to each candidate feature adjustment factor may be determined based on the first feature. For example, for each candidate feature adjustment factor, after obtaining the first feature corresponding to the current image block, the first feature may be enhanced by a feature enhancement network to obtain a second feature, and the second feature may be feature adjusted by the candidate feature adjustment factor to obtain a third feature. After obtaining the third feature, the target feature corresponding to the current image block may be determined based on the third feature, and the target feature may be input into a reconstruction decoding network, and the reconstruction decoding network may process the target feature to obtain a reconstructed image block corresponding to the current image block. After obtaining the reconstructed image block, the rate-distortion cost corresponding to the candidate feature adjustment factor may be determined based on the loss between the reconstructed image block and the current image block, thereby obtaining the rate-distortion cost corresponding to each candidate feature adjustment factor. After obtaining the rate-distortion cost corresponding to each candidate feature adjustment factor, based on the rate-distortion cost corresponding to each candidate feature adjustment factor, one candidate feature adjustment factor is selected from all the candidate feature adjustment factors as the feature adjustment factor corresponding to the current image block, such as the candidate feature adjustment factor with the smallest rate-distortion cost is selected as the feature adjustment factor corresponding to the current image block.

[0160] Exemplarily, the encoding end can also obtain a feature adjustment factor for the decoding subnetwork, and perform feature adjustment on a certain feature in the decoding subnetwork based on the feature adjustment factor. For example, after the decoding subnetwork obtains feature A (feature A is any feature in the decoding subnetwork), the feature adjustment factor can be used to perform feature adjustment on feature A to obtain the adjusted feature B. The decoding subnetwork continues to process based on feature B and finally obtains the first feature. On this basis, the encoding end can also encode the feature adjustment factor for the decoding subnetwork in the auxiliary code stream (the same as or different from the feature adjustment factor for the second feature), so that the decoding end can decode the feature adjustment factor for the decoding subnetwork from the auxiliary code stream.

[0161] Embodiment 8: For Embodiment 1 to Embodiment 7, for the encoding end and the decoding end, the first feature corresponding to the current image block can be obtained through the first neural network. For example, probability distribution parameters are obtained based on the first code stream corresponding to the current image block, and a probability distribution model is determined based on the probability distribution parameters, and the second code stream corresponding to the current image block is decoded based on the probability distribution model to obtain decoded image features; and the first feature corresponding to the current image block is determined based on the decoded image features.

[0162] Exemplarily, the first neural network may be a decoding subnetwork, and the decoding subnetwork is taken as an example for explanation, see Fig. 8A As shown, the network in the dotted box is the decoding subnetwork, and y_hat is the first feature output by the decoding subnetwork.

[0163] For example, after obtaining Bitstream#1 corresponding to the current image block, the encoder or decoder can decode Bitstream#1 to obtain the hyperparameter quantization feature, dequantize the hyperparameter quantization feature, and obtain the coefficient hyperparameter feature z_hat. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the encoder or decoder can also decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the dequantization process of the coefficient hyperparameter feature z_hat.

[0164] After obtaining the coefficient hyperparameter feature z_hat, the encoder or decoder can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input to the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p.

[0165] After obtaining Bitstream#2 corresponding to the current image block, the encoder or decoder may also decode Bitstream#2 to obtain image quantization features, and dequantize the image quantization features to obtain image features s'. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the encoder or decoder may also decode Bitstream#2 to obtain image features s' without involving the dequantization process of the image quantization features. Exemplarily, when decoding Bitstream#2, the encoder or decoder may decode Bitstream#2 based on a probability distribution model.

[0166] After obtaining the image feature s', the encoding end or the decoding end can perform feature recovery on the image feature s' to obtain the image feature y_hat, and the image feature y_hat can be used as the first feature, that is, the decoding subnetwork outputs the first feature y_hat.

[0167] Embodiment 9: For Embodiment 1 to Embodiment 7, for the encoding end and the decoding end, the first feature corresponding to the current image block can be obtained through the first neural network. For example, the probability distribution parameters and the predicted value (such as the mean) are obtained based on the first code stream corresponding to the current image block; the probability distribution model is determined based on the probability distribution parameters, and the second code stream corresponding to the current image block is decoded based on the probability distribution model to obtain the decoded image features; the decoded image features are subjected to residual recovery to obtain the residual features; based on the residual features and the predicted value, the first feature corresponding to the current image block is determined.

[0168] Exemplarily, the first neural network may be a decoding subnetwork, and the decoding subnetwork is taken as an example for explanation, see Figure 8B As shown, the network in the dotted box is the decoding subnetwork, and y_hat is the first feature output by the decoding subnetwork.

[0169] For example, after obtaining Bitstream#1 corresponding to the current image block, the encoder or decoder can decode Bitstream#1 to obtain the hyperparameter quantization feature, dequantize the hyperparameter quantization feature, and obtain the coefficient hyperparameter feature z_hat. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the encoder or decoder can also decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the dequantization process of the coefficient hyperparameter feature z_hat.

[0170] After obtaining the coefficient hyperparameter feature z_hat, the encoder or decoder can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input to the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p.

[0171] After obtaining the coefficient hyperparameter feature z_hat, the encoder or decoder can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat are input to the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input of the mean prediction network can include the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu.

[0172] After obtaining Bitstream#2 corresponding to the current image block, the encoder or decoder may also decode Bitstream#2 to obtain image quantization features, and dequantize the image quantization features to obtain image features s'. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the encoder or decoder may also decode Bitstream#2 to obtain image features s' without involving the dequantization process of the image quantization features. Exemplarily, when decoding Bitstream#2, the encoder or decoder may decode Bitstream#2 based on a probability distribution model.

[0173] After obtaining the image feature s', the encoder or decoder performs feature recovery (i.e., residual recovery, which is the inverse process of residual processing) on ​​the image feature s' to obtain the residual feature r_hat. After obtaining the residual feature r_hat, the encoder or decoder determines the image feature y_hat based on the residual feature r_hat and the predicted value mu. For example, the sum of the residual feature r_hat and the predicted value mu is used as the image feature y_hat. The image feature y_hat can be used as the first feature, that is, the decoding subnetwork outputs the first feature y_hat.

[0174] For example, Figure 8B and Fig. 8A In contrast, in the decoding subnetwork, in addition to obtaining the probability distribution parameters for the second bitstream based on the first bitstream, the prediction value mu for the first feature can also be generated based on the first bitstream. The feature residual is obtained based on the decoding of the second bitstream, and the residual r_hat of the first feature is obtained by residual recovery. The first feature can be obtained based on mu and r_hat.

[0175] Embodiment 10: For Embodiments 1 to 9, for the encoding end and the decoding end, after obtaining the first feature y_hat, the first feature y_hat can be synthetically transformed through a synthetic transformation network to obtain a reconstructed image block x_hat corresponding to the current image block x. During the synthetic transformation process, the first feature y_hat can be feature enhanced to obtain a second feature; the second feature can be feature adjusted based on a feature adjustment factor to obtain a third feature; after obtaining the third feature, the target feature is determined based on the third feature. Then, the reconstructed image block x_hat corresponding to the current image block x is obtained based on the target feature.

[0176] See also Figure 6BAs shown in FIG. 1 , it is a schematic diagram of the structure of the synthetic transformation network. The synthetic transformation network may include a ResBlock layer (i.e., a RB layer), a ResBlock layer, a Conv layer, a Cropping layer, a ResAU layer, a Conv layer, a Cropping layer, a ResAU layer, a Conv layer, an RNAB layer, a Cropping layer, a ResAU layer, a Conv layer, and a Cropping layer in sequence. The schematic diagram of the structure of the RB layer may be referred to in FIG. Fig. 6E As shown, the structural diagram of the ResAU layer can be found in Figure 6C shown.

[0177] The RNAB layer is a residual non-local attention block. The structural diagram of the RNAB layer can be found in Fig.6D As shown, or, you can remove Fig.6D , and obtain the optimized RNAB layer. Fig. 6F As shown in Figure 2, the three RB layers and the two Conv layers can be removed to obtain the optimized RNAB layer. Therefore, the schematic diagram of the RNAB layer can be found in Figure 2. Figure 6G Alternatively, the schematic diagram of the structure of the RNAB layer can be found in Figure 6H As shown. Figure 6G and Figure 6H In the example, the RNAB layer may include an RB layer (composed of at least one RB block, such as 3 consecutive RB blocks), an RB layer (composed of at least one RB block, such as 3 consecutive RB blocks), a Conv layer, and a sigmoid layer (i.e., a feature mapping layer). For the Conv layer, it may be a 3*3 Conv layer, a 5*5 Conv layer, a 1*1 Conv layer, or a Conv layer of other sizes, such as a 7*7 Conv layer, a 9*9 Conv layer, and there is no limitation on the size of the Conv layer.

[0178] For the above-mentioned synthesis transformation network, the synthesis transformation network can be divided into a feature enhancement subnetwork and a reconstruction decoding subnetwork. The feature enhancement subnetwork may include an initial enhancement subnetwork, a first upsampling convolution subnetwork, and an attention subnetwork. The reconstruction decoding subnetwork may include a second upsampling convolution subnetwork and a color space transformation subnetwork.

[0179] For example, all networks before the RNAB layer in the synthetic transformation network can be used as the initial enhancement subnetwork and the first upsampling convolution subnetwork, the initial enhancement subnetwork can include 2 RB networks, and the first upsampling convolution subnetwork can include at least one upsampling convolution network (such as 3 upsampling convolution networks). For example, see Figure 6BAs shown, all networks before the RNAB layer are RB layer, RB layer, Conv layer, Cropping layer, ResAU layer, Conv layer, Cropping layer, ResAU layer, and Conv layer, respectively. Based on this, the initial enhancement sub-network can include RB layer and RB layer, and the first upsampling convolution sub-network can include Conv layer, Cropping layer, ResAU layer, Conv layer, Cropping layer, ResAU layer, and Conv layer, that is, involving 3 upsampling convolutional networks (Conv layers). Here, the 3*3 Conv layer is taken as an example.

[0180] For example, the RNAB layer in the synthetic transformation network can be used as an attention subnetwork (i.e., attention subnetwork). The attention subnetwork can effectively reduce the complexity by removing the up- and down-sampling convolutional network, see Figure 6G and Figure 6H As shown, it is a structural diagram of the attention sub-network. The attention sub-network may include a RB layer (consisting of at least one RB block), a RB layer (consisting of at least one RB block), 1 Conv layer and 1 sigmoid layer (i.e., feature mapping layer).

[0181] The attention subnetwork may include a residual skip subnetwork, a residual enhancement subnetwork, and a weight generation subnetwork. For the residual skip subnetwork, x_out0 = x_in, x_in is the input feature of the residual skip subnetwork, and x_out0 is the output feature of the residual skip subnetwork. For the residual enhancement subnetwork, x_out1 = RB (x_in), x_in is the input feature of the residual enhancement subnetwork, and x_out1 is the output feature of the residual enhancement subnetwork. For the weight generation subnetwork, it is used to obtain the weight feature k of the output feature x_out1 of the residual enhancement subnetwork. Therefore, the final output of the attention subnetwork is: x_out = x_in + k*x_out1.

[0182] See also Fig.9A and Fig. 9BAs shown, the residual skip subnetwork is used to superimpose the input feature x_in (i.e., x_out0) onto the final output feature of the attention subnetwork. The residual enhancement subnetwork includes at least one RB block (such as one RB block or three RB blocks, etc.), and the residual enhancement subnetwork is used to process the input feature x_in based on the RB block to obtain the output feature x_out1. The weight generation subnetwork may include at least 1 RB block (such as 1 RB block or 3 RB blocks, etc.), 1 Conv layer (such as 3*3 Conv layer, 5*5 Conv layer, 1*1 Conv layer, etc.) and a feature mapping layer (such as using a sigmoid processing process to implement feature mapping). The weight generation subnetwork is used to process the input feature x_in based on the RB block, and then use the Conv layer to perform convolution processing on the features processed by the RB block. Then, the feature mapping layer is used to perform feature mapping on the features processed by the Conv layer to obtain a weight feature k. The dimension of the weight feature k is the same as the dimension of the output feature x_out1 of the residual enhancement subnetwork. After multiplying the output feature x_out1 by the weight feature k, it is added to the output feature x_out0 of the residual skip subnetwork to obtain the final output feature of the attention subnetwork: x_out=x_in+k*x_out1.

[0183] Exemplarily, all networks after the RNAB layer in the synthetic transformation network can be used as a reconstruction decoding subnetwork, and the reconstruction decoding subnetwork can include a second upsampling convolution subnetwork and a color space transformation subnetwork, that is, all networks after the RNAB layer are used as a second upsampling convolution subnetwork and a color space transformation subnetwork. The second upsampling convolution subnetwork can include at least one upsampling convolution network (such as one upsampling convolution network), and the color space transformation subnetwork is used to implement the image transformation process from YUV domain to RBG, and can also be a filtering process from YUV domain to YUV domain.

[0184] For example, see Figure 6B As shown in the figure, all networks after the RNAB layer are Cropping layer, ResAU layer, Conv layer, Cropping layer in order. Therefore, the second upsampling convolution subnetwork can include Cropping layer, ResAU layer, Conv layer, Cropping layer, that is, it involves one upsampling convolution network (Conv layer). Here, a 3*3 Conv layer is taken as an example. If there is a need for color space transformation, a color space transformation subnetwork ( Figure 6B (not shown in the figure), the color space conversion subnetwork performs the image conversion process from the YUV domain to the RBG domain, or the color space conversion subnetwork performs the filtering process from the YUV domain to the YUV domain. If there is no color space conversion requirement, the color space conversion subnetwork may not be included.

[0185] After the synthetic transformation network is divided into an initial enhancement subnetwork, a first upsampling convolution subnetwork, an attention subnetwork, and a reconstruction decoding subnetwork (the reconstruction decoding subnetwork includes a second upsampling convolution subnetwork and a color space transformation subnetwork), the first feature y_hat can be synthetically transformed based on the initial enhancement subnetwork, the first upsampling convolution subnetwork, the attention subnetwork, and the reconstruction decoding subnetwork to obtain a reconstructed image block x_hat corresponding to the current image block x.

[0186] Example 11: In Example 10, the first feature y_hat can be synthetically transformed based on the initial enhancement sub-network, the first upsampling convolution sub-network, the attention sub-network and the reconstruction decoding sub-network to obtain a reconstructed image block x_hat corresponding to the current image block x. In a possible implementation, the output feature of the feature enhancement sub-network can be adjusted using a feature adjustment factor. The adjustment position of the feature adjustment factor can be found in Fig. 10A As shown, under the adjustment method of the feature adjustment factor, the first feature y_hat can be synthetically transformed in the following manner to obtain the reconstructed image block x_hat.

[0187] The initial enhancement sub-network is used to enhance the first feature y_hat to obtain an enhanced feature. For example, the first feature y_hat can be input to the initial enhancement sub-network, and the initial enhancement sub-network performs feature enhancement on the first feature y_hat to obtain an enhanced feature. For example, the initial enhancement sub-network may include an RB layer and an RB layer, so the first feature y_hat can be enhanced through two RB layers to obtain an enhanced feature, and there is no limitation on this process.

[0188] After obtaining the enhanced features, the first upsampling convolutional subnetwork is used to upsample and convolve the enhanced features to obtain the initial features corresponding to the attention subnetwork, that is, the input features x_in corresponding to the attention subnetwork. For example, the enhanced features (that is, the output features of the initial enhancement subnetwork) can be input to the first upsampling convolutional subnetwork, and the first upsampling convolutional subnetwork performs upsample and convolve on the enhanced features to obtain the initial features corresponding to the attention subnetwork. For example, the first upsampling convolutional subnetwork may include Conv layer, Cropping layer, ResAU layer, Conv layer, Cropping layer, ResAU layer, Conv layer, and the enhanced features may be upsampled and convoluted through these network layers, and there is no restriction on this process.

[0189] After obtaining the initial feature x_in corresponding to the attention subnetwork, the initial feature x_in can be enhanced through the attention subnetwork to obtain the second feature. For example, the initial feature x_in can be input to the residual skip subnetwork, the residual enhancement subnetwork and the weight generation subnetwork respectively. After obtaining the initial feature x_in, the residual skip subnetwork superimposes the initial feature x_in (i.e., x_out0) on the final output feature of the attention subnetwork. After obtaining the initial feature x_in, the residual enhancement subnetwork performs feature enhancement on the initial feature x_in to obtain the enhanced feature, such as the residual enhancement subnetwork performs feature enhancement on the initial feature x_in based on the RB block to obtain the enhanced feature x_out1. After obtaining the initial feature x_in, the weight generation subnetwork generates the weight feature corresponding to the initial feature x_in, i.e., the weight feature k of the output feature x_out1 of the residual enhancement subnetwork.

[0190] For example, the weight generation subnetwork may include a residual block subnetwork, a convolution subnetwork, and a feature mapping subnetwork. The residual block subnetwork may be used to perform convolution activation on the initial feature x_in to obtain the convolution activated feature. The convolution subnetwork may be used to perform convolution on the convolution activated feature to obtain the convolution feature. The feature mapping subnetwork may be used to perform feature mapping on the convolution feature to obtain the weight feature k. Fig. 10A As shown, the weight generation subnetwork may include at least one RB block (such as one RB block or three RB blocks, etc., i.e., a residual block subnetwork), one Conv layer (such as a 3*3 Conv layer, a 5*5 Conv layer, a 1*1 Conv layer, etc., i.e., a convolution subnetwork) and a feature mapping layer (such as using a sigmoid processing process to implement feature mapping, i.e., a feature mapping subnetwork). On this basis, the initial feature x_in can be convolutionally activated based on the RB block to obtain a convolution-activated feature. Then, the Conv layer can be used to convolve the convolution-activated feature to obtain a convolution-post feature. Then, the feature mapping layer can be used to feature map the convolution-post feature to obtain a weight feature k.

[0191] See also Fig. 10A As shown, after obtaining the enhanced feature x_out1, the weight feature k and the initial feature x_in (i.e., x_out0), the second feature can be generated based on the initial feature x_in, the enhanced feature x_out1 and the weight feature k. The second feature can be the final output feature of the attention sub-network. For example, after multiplying the enhanced feature x_out1 with the weight feature k, it is added to the output feature x_out0 of the residual skip sub-network, that is, the second feature is: x_out = x_in + k*x_out1.

[0192] After the second feature is obtained, the second feature can be feature adjusted based on the feature adjustment factor, and the feature after feature adjustment is used as the third feature. After the third feature is obtained, the third feature can be determined as the target feature.

[0193] After obtaining the target feature, the target feature can be processed by the reconstruction decoding subnetwork to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the target feature can be input to the reconstruction decoding subnetwork, and the reconstruction decoding subnetwork processes the target feature to obtain the reconstructed image block x_hat corresponding to the current image block x. Exemplarily, the reconstruction decoding subnetwork may include a second upsampling convolution subnetwork, and the target feature can be upsampled and convolved by the second upsampling convolution subnetwork to obtain the reconstructed image block x_hat corresponding to the current image block x. Alternatively, the reconstruction decoding subnetwork may include a second upsampling convolution subnetwork and a color space transformation subnetwork, and the target feature can be upsampled and convolved by the second upsampling convolution subnetwork, and the color space transformation of the upsampled and convolved feature can be performed by the color space transformation subnetwork to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the second upsampling convolution subnetwork may include a Cropping layer, a ResAU layer, a Conv layer, and a Cropping layer, and the target feature can be upsampled and convolved by these network layers, and there is no limitation on this process. The color space transformation subnetwork is used to perform the image transformation process from YUV domain to RBG, or the color space transformation subnetwork is used to perform the filtering process from YUV domain to YUV domain, and there is no restriction on this process.

[0194] Example 12: In Example 10, the first feature y_hat can be synthetically transformed based on the initial enhancement subnetwork, the first upsampling convolution subnetwork, the attention subnetwork and the reconstruction decoding subnetwork to obtain a reconstructed image block x_hat corresponding to the current image block x. In a possible implementation, the residual enhancement output feature of the attention subnetwork can be adjusted using a feature adjustment factor. The adjustment position of the feature adjustment factor can be found in Fig. 10B As shown, under the adjustment method of the feature adjustment factor, the first feature y_hat can be synthetically transformed in the following manner to obtain the reconstructed image block x_hat.

[0195] The initial enhancement sub-network is used to enhance the first feature y_hat to obtain an enhanced feature. For example, the first feature y_hat can be input to the initial enhancement sub-network, and the initial enhancement sub-network performs feature enhancement on the first feature y_hat to obtain an enhanced feature. For example, the initial enhancement sub-network may include an RB layer and an RB layer, so the first feature y_hat can be enhanced through two RB layers to obtain an enhanced feature, and there is no limitation on this process.

[0196] After obtaining the enhanced features, the first upsampling convolutional subnetwork is used to upsample and convolve the enhanced features to obtain the initial features corresponding to the attention subnetwork, that is, the input features x_in corresponding to the attention subnetwork. For example, the enhanced features (that is, the output features of the initial enhancement subnetwork) can be input to the first upsampling convolutional subnetwork, and the first upsampling convolutional subnetwork performs upsample and convolve on the enhanced features to obtain the initial features corresponding to the attention subnetwork. For example, the first upsampling convolutional subnetwork may include Conv layer, Cropping layer, ResAU layer, Conv layer, Cropping layer, ResAU layer, Conv layer, and the enhanced features may be upsampled and convoluted through these network layers, and there is no restriction on this process.

[0197] After obtaining the initial feature x_in corresponding to the attention subnetwork, the initial feature x_in is input to the residual skip subnetwork, the residual enhancement subnetwork and the weight generation subnetwork respectively. After obtaining the initial feature x_in, the residual skip subnetwork superimposes the initial feature x_in (i.e., x_out0) onto the final output feature of the attention subnetwork. After obtaining the initial feature x_in, the residual enhancement subnetwork performs feature enhancement on the initial feature x_in to obtain the enhanced feature. For example, the residual enhancement subnetwork performs feature enhancement on the initial feature x_in based on the RB block to obtain the enhanced feature x_out1. After obtaining the initial feature x_in, the weight generation subnetwork generates the weight feature corresponding to the initial feature x_in, i.e., the weight feature k of the output feature x_out1 of the residual enhancement subnetwork.

[0198] For example, the weight generation subnetwork includes a residual block subnetwork, a convolution subnetwork, and a feature mapping subnetwork. The residual block subnetwork is used to perform convolution activation on the initial feature x_in to obtain the convolution activated feature. The convolution subnetwork is used to perform convolution on the convolution activated feature to obtain the convolution feature. The feature mapping subnetwork is used to perform feature mapping on the convolution feature to obtain the weight feature k. Fig. 10B As shown, the weight generation subnetwork includes at least one RB block (i.e., residual block subnetwork), one Conv layer (i.e., convolution subnetwork) and a feature mapping layer (i.e., feature mapping subnetwork). On this basis, the initial feature x_in is convolutionally activated based on the RB block to obtain the convolution activated feature, the convolution activated feature is convolved by the Conv layer to obtain the convolution feature, and the convolution feature is feature mapped by the feature mapping layer to obtain the weight feature k.

[0199] See also Fig. 10BAs shown, after obtaining the enhanced feature x_out1 and the weight feature k, the second feature can be generated based on the enhanced feature x_out1 and the weight feature k. For example, the enhanced feature x_out1 can be multiplied by the weight feature k, and the multiplied feature can be used as the second feature. That is, the second feature can be: k*x_out1.

[0200] After obtaining the second feature, the second feature can be adjusted based on the feature adjustment factor, and the adjusted feature is used as the third feature. Fig. 10B As shown, the third feature may be feature rd.

[0201] After obtaining the third feature rd, the fourth feature can be generated based on the initial feature x_in (i.e., the output feature x_out0 of the residual skip subnetwork) and the third feature rd. The fourth feature can be the final output feature of the attention subnetwork. For example, the third feature rd is added to the output feature x_out0 of the residual skip subnetwork to obtain the fourth feature: x_out = x_in + rd.

[0202] In summary, it can be seen that the initial feature x_in can be enhanced by the first subnetwork in the attention subnetwork (such as the residual enhancement subnetwork and the weight generation subnetwork, the weight generation subnetwork can include the residual block subnetwork, the convolution subnetwork and the feature mapping subnetwork) to obtain the second feature. After obtaining the second feature, the second feature can be feature adjusted based on the feature adjustment factor to obtain the third feature rd. After obtaining the third feature rd, the third feature rd can be processed by the second subnetwork in the attention subnetwork (such as the residual skip subnetwork) to obtain the fourth feature x_out. After obtaining the fourth feature x_out, the fourth feature x_out can be determined as the target feature.

[0203] After obtaining the target feature, the target feature can be processed by the reconstruction decoding subnetwork to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the target feature can be input to the reconstruction decoding subnetwork, and the reconstruction decoding subnetwork processes the target feature to obtain the reconstructed image block x_hat corresponding to the current image block x. Exemplarily, the reconstruction decoding subnetwork may include a second upsampling convolution subnetwork, and the target feature can be upsampled and convolved by the second upsampling convolution subnetwork to obtain the reconstructed image block x_hat corresponding to the current image block x. Alternatively, the reconstruction decoding subnetwork may include a second upsampling convolution subnetwork and a color space transformation subnetwork, and the target feature can be upsampled and convolved by the second upsampling convolution subnetwork, and the color space transformation of the upsampled and convolved feature can be performed by the color space transformation subnetwork to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the second upsampling convolution subnetwork may include a Cropping layer, a ResAU layer, a Conv layer, and a Cropping layer, and the target feature can be upsampled and convolved by these network layers, and there is no limitation on this process. The color space transformation subnetwork is used to perform the image transformation process from YUV domain to RBG, or the color space transformation subnetwork is used to perform the filtering process from YUV domain to YUV domain, and there is no restriction on this process.

[0204] Example 13: In Example 10, the first feature y_hat can be synthetically transformed based on the initial enhancement subnetwork, the first upsampling convolution subnetwork, the attention subnetwork and the reconstruction decoding subnetwork to obtain a reconstructed image block x_hat corresponding to the current image block x. In a possible implementation, the weight output feature of the attention subnetwork can be adjusted using a feature adjustment factor. The adjustment position of the feature adjustment factor can be found in Fig. 10C As shown, under the adjustment method of the feature adjustment factor, the first feature y_hat can be synthetically transformed in the following manner to obtain the reconstructed image block x_hat.

[0205] The initial enhancement sub-network is used to enhance the first feature y_hat to obtain an enhanced feature. For example, the first feature y_hat can be input to the initial enhancement sub-network, and the initial enhancement sub-network performs feature enhancement on the first feature y_hat to obtain an enhanced feature. For example, the initial enhancement sub-network may include an RB layer and an RB layer, so the first feature y_hat can be enhanced through two RB layers to obtain an enhanced feature, and there is no limitation on this process.

[0206] After obtaining the enhanced features, the first upsampling convolutional subnetwork is used to upsample and convolve the enhanced features to obtain the initial features corresponding to the attention subnetwork, that is, the input features x_in corresponding to the attention subnetwork. For example, the enhanced features (that is, the output features of the initial enhancement subnetwork) can be input to the first upsampling convolutional subnetwork, and the first upsampling convolutional subnetwork performs upsample and convolve on the enhanced features to obtain the initial features corresponding to the attention subnetwork. For example, the first upsampling convolutional subnetwork may include Conv layer, Cropping layer, ResAU layer, Conv layer, Cropping layer, ResAU layer, Conv layer, and the enhanced features may be upsampled and convoluted through these network layers, and there is no restriction on this process.

[0207] After obtaining the initial feature x_in corresponding to the attention subnetwork, the initial feature x_in is input to the residual skip subnetwork, the residual enhancement subnetwork and the weight generation subnetwork respectively. Exemplarily, after obtaining the initial feature x_in, the residual skip subnetwork superimposes the initial feature x_in (i.e., x_out0) onto the final output feature of the attention subnetwork. After obtaining the initial feature x_in, the residual enhancement subnetwork performs feature enhancement on the initial feature x_in to obtain the enhanced feature, such as the residual enhancement subnetwork performs feature enhancement on the initial feature x_in based on the RB block to obtain the enhanced feature x_out1.

[0208] The weight generation subnetwork can include a residual block subnetwork, a convolution subnetwork, and a feature mapping subnetwork, see Fig. 10C As shown, the weight generation subnetwork may include at least one RB block (i.e., residual block subnetwork), one Conv layer (i.e., convolution subnetwork) and a feature mapping layer (i.e., feature mapping subnetwork). On this basis, the initial feature x_in may be input to the residual block subnetwork, and the residual block subnetwork may be used to perform convolution activation on the initial feature x_in to obtain the convolution activated feature. For example, the residual block subnetwork performs convolution activation on the initial feature x_in based on the RB block to obtain the convolution activated feature.

[0209] Then, the convolution subnetwork may be used to convolve the convolution activated features to obtain the convolution features. For example, the convolution subnetwork may use the Conv layer to convolve the convolution activated features to obtain the convolution features. After obtaining the convolution features, the second feature may be determined based on the convolution features. For example, the convolution features may be used as the second feature.

[0210] In summary, the residual block subnetwork can be used to enhance the initial feature x_in to obtain the enhanced feature, and the convolution subnetwork can be used to convolve the enhanced feature to obtain the convolution feature, that is, the second feature. After obtaining the second feature, the second feature can be adjusted based on the feature adjustment factor, and the adjusted feature is used as the third feature.

[0211] See also Fig. 10C As shown, after obtaining the third feature, the feature mapping subnetwork can be used to perform feature mapping on the third feature to obtain the weight feature k. After obtaining the enhanced feature x_out1, the weight feature k and the initial feature x_in (i.e., x_out0), the fourth feature can be generated based on the initial feature x_in, the enhanced feature x_out1 and the weight feature k. The fourth feature can be the final output feature of the attention subnetwork. For example, after multiplying the enhanced feature x_out1 with the weight feature k, it is added to the output feature x_out0 of the residual skip subnetwork, that is, the fourth feature is: x_out = x_in + k*x_out1.

[0212] In summary, it can be seen that the first subnetwork in the attention subnetwork (such as the residual block subnetwork and the convolution subnetwork in the weight generation subnetwork) can be used to enhance the initial feature x_in to obtain the second feature. After obtaining the second feature, the second feature can be feature adjusted based on the feature adjustment factor to obtain the third feature. After obtaining the third feature, the third feature can be processed by the second subnetwork in the attention subnetwork (such as the residual skip subnetwork, the residual enhancement subnetwork, and the feature mapping subnetwork in the weight generation subnetwork) to obtain the fourth feature x_out. For example, the feature mapping subnetwork is used to feature map the third feature to obtain the weight feature k; the residual enhancement subnetwork is used to feature enhance the initial feature x_in to obtain the enhanced feature x_out1; the fourth feature x_out is generated based on the initial feature x_in, the enhanced feature x_out1 and the weight feature k. After obtaining the fourth feature x_out, the fourth feature x_out can be determined as the target feature.

[0213] After obtaining the target feature, the target feature can be processed by the reconstruction decoding subnetwork to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the target feature can be input to the reconstruction decoding subnetwork, and the reconstruction decoding subnetwork processes the target feature to obtain the reconstructed image block x_hat corresponding to the current image block x. Exemplarily, the reconstruction decoding subnetwork may include a second upsampling convolution subnetwork, and the target feature can be upsampled and convolved by the second upsampling convolution subnetwork to obtain the reconstructed image block x_hat corresponding to the current image block x. Alternatively, the reconstruction decoding subnetwork may include a second upsampling convolution subnetwork and a color space transformation subnetwork, and the target feature can be upsampled and convolved by the second upsampling convolution subnetwork, and the color space transformation of the upsampled and convolved feature can be performed by the color space transformation subnetwork to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the second upsampling convolution subnetwork may include a Cropping layer, a ResAU layer, a Conv layer, and a Cropping layer, and the target feature can be upsampled and convolved by these network layers, and there is no limitation on this process. The color space transformation subnetwork is used to perform the image transformation process from YUV domain to RBG, or the color space transformation subnetwork is used to perform the filtering process from YUV domain to YUV domain, and there is no restriction on this process.

[0214] Example 14: For Example 1 to Example 13, for the encoding end and the decoding end, the second feature can be feature adjusted based on the feature adjustment factor to obtain the third feature. For example, if the second feature includes C*H*W feature values, C represents the number of channels, H represents the feature height, and W represents the feature width, then the feature adjustment value corresponding to each feature value can be determined based on the feature adjustment factor. For each feature value, the feature adjustment is performed on the feature value based on the feature adjustment value corresponding to the feature value to obtain the adjusted feature value corresponding to the feature value. The third feature is generated based on the adjusted feature value corresponding to each feature value.

[0215] Exemplarily, if the second feature includes C*H*W eigenvalues, then the feature adjustment value corresponding to each eigenvalue of the second feature can be determined based on the feature adjustment factor. For this process, the following situations can be adopted:

[0216] Case 1: The dimension of the feature adjustment factor is the same as the dimension of the second feature, that is, each feature value of the second feature has a different feature adjustment value. For example, if the feature adjustment factor can include C*H*W feature adjustment values, then the feature adjustment values ​​corresponding to the C*H*W feature values ​​can be determined based on the C*H*W feature adjustment values.

[0217] For example, for the feature value whose position of the c-th channel in the second feature is h*w, the feature adjustment value corresponding to the feature value may be: the feature adjustment value whose position of the c-th channel in the feature adjustment factor is h*w.

[0218] Case 2: The W dimension and H dimension of the feature adjustment factor are the same as the dimension of the second feature, but the C dimension of the feature adjustment factor is 1, that is, the feature values ​​at the same position of each channel of the second feature correspond to the same feature adjustment value. For example, if the feature adjustment factor can include H*W feature adjustment values, then the feature adjustment values ​​corresponding to the C*H*W feature values ​​can be determined based on the H*W feature adjustment values ​​(that is, the C dimension of the feature adjustment factor is 1).

[0219] For example, for the eigenvalue of the cth channel in the second feature (ie, each channel in the second feature) at position h*w, the feature adjustment value corresponding to the eigenvalue may be: the feature adjustment value at position h*w in the feature adjustment factor.

[0220] Case 3: The C dimension of the feature adjustment factor is the same as the C dimension of the second feature, but the W dimension and H dimension of the feature adjustment factor are both 1, that is, the W*H eigenvalues ​​of the same channel of the second feature correspond to the same feature adjustment value. For example, if the feature adjustment factor can include C feature adjustment values, then the feature adjustment values ​​corresponding to the C*H*W eigenvalues ​​can be determined based on the C feature adjustment values ​​(that is, the W dimension and H dimension of the feature adjustment factor are both 1).

[0221] For example, for all eigenvalues ​​of the cth channel in the second feature (ie, eigenvalues ​​at any position), the feature adjustment value corresponding to the eigenvalue may be: the feature adjustment value of the cth channel in the feature adjustment factor.

[0222] Exemplarily, after obtaining the feature adjustment value corresponding to each eigenvalue, for each eigenvalue, feature adjustment can be performed on the eigenvalue based on the feature adjustment value corresponding to the eigenvalue to obtain an adjusted eigenvalue corresponding to the eigenvalue.

[0223] For example, if the eigenvalue corresponds to only one feature adjustment value, the adjusted eigenvalue corresponding to the eigenvalue can be determined by the following formula: yr = rf * y; where yr represents the adjusted eigenvalue, rf represents the feature adjustment value, and y represents the eigenvalue. For example, for a eigenvalue y(c, w, h) whose spatial position of the c-th channel is (w, h), after determining the feature adjustment value rf(c, w, h) corresponding to the eigenvalue y(c, w, h), the adjusted eigenvalue yr(c, w, h) corresponding to the eigenvalue y(c, w, h) is: yr(c, w, h) = rf(c, w, h) * y(c, w, h).

[0224] If the eigenvalue corresponds to N+1 eigenvalues, where N is a positive integer, i.e., corresponds to at least two eigenvalues, then the adjusted eigenvalue corresponding to the eigenvalue can be determined using the following formula: yr = rf_0*y 0 +rf_1*y 1 +rf_2*y 2 +…+rf_N*y N ; Wherein, yr represents the adjusted eigenvalue, rf_0, rf_1, rf_2, ..., rf_N represent N+1 feature adjustment values, and y represents the eigenvalue. For example, for a eigenvalue y(c, w, h) with a spatial position of (w, h) in the cth channel, after determining the feature adjustment value rf_i(c, w, h) corresponding to the eigenvalue y(c, w, h), i = 0, 1, 2 ... N, then the adjusted eigenvalue yr(c, w, h) corresponding to the eigenvalue y(c, w, h) is: yr(c, w, h) = rf_0(c, w, h)*y 0 (c,w,h)+rf_1(c,w,h)*y 1 (c,w,h)+rf_2(c,w,h)*y 2 (c,w,h)+…+rf_N(c,w,h)*y N (c, w, h). When N is 1, yr(c,w,h)=rf_0(c,w,h)+rf_1(c,w,h)*y(c,w,h). When N is 2, yr(c,w,h)=rf_0(c,w,h)+rf_1(c,w,h)*y(c,w,h)+rf_2(c,w,h)*y 2 (c, w, h). Similarly, when N is other values, the form is similar and will not be repeated here. In the above formula, y k (c, w, h) represents y to the power of k.

[0225] Exemplarily, after obtaining the adjusted eigenvalue corresponding to each eigenvalue, a third feature may be generated based on the adjusted eigenvalue corresponding to each eigenvalue. The third feature may include the adjusted eigenvalue corresponding to each eigenvalue.

[0226] Example 15: For Example 1 to Example 14, for the encoding end, it is also necessary to determine whether the current image block enables the feature adjustment mode. If the current image block enables the feature adjustment mode, the feature adjustment factor corresponding to the current image block is obtained, and the feature adjustment factor is encoded in the auxiliary code stream corresponding to the current image block. If the feature adjustment mode is not enabled for the current image block, it is not necessary to obtain the feature adjustment factor corresponding to the current image block, nor is it necessary to encode the feature adjustment factor in the auxiliary code stream corresponding to the current image block. For the decoding end, it is also necessary to determine whether the current image block enables the feature adjustment mode. If the current image block enables the feature adjustment mode, the feature adjustment factor corresponding to the current image block is decoded from the auxiliary code stream corresponding to the current image block, and the target feature is determined based on the first feature and the feature adjustment factor. If the current image block does not enable the feature adjustment mode, it is not necessary to decode the feature adjustment factor corresponding to the current image block from the auxiliary code stream corresponding to the current image block.

[0227] Exemplarily, in order to determine whether the feature adjustment mode is enabled for the current image block, the following method may be used:

[0228] Method 1: The encoder encodes a feature adjustment flag in the bitstream, and the decoder decodes the feature adjustment flag from the bitstream. The feature adjustment flag allows the current image block to enable the feature adjustment mode, or the feature adjustment flag prohibits the current image block from enabling the feature adjustment mode. If the feature adjustment flag allows the current image block to enable the feature adjustment mode, the decoder determines that the current image block enables the feature adjustment mode. Otherwise, if the feature adjustment flag prohibits the current image block from enabling the feature adjustment mode, the decoder determines that the current image block prohibits enabling the feature adjustment mode. For example, if the feature adjustment flag is a first value, the feature adjustment flag allows the current image block to enable the feature adjustment mode, and if the feature adjustment flag is a second value, the feature adjustment flag prohibits the current image block from enabling the feature adjustment mode.

[0229] Exemplarily, the feature adjustment flag may be a sequence-level feature adjustment flag, i.e., all image blocks in the sequence correspond to the feature adjustment flag; or, the feature adjustment flag may be an image-level feature adjustment flag, i.e., all image blocks in the image correspond to the feature adjustment flag; or, the feature adjustment flag may be a slice-level feature adjustment flag, i.e., all image blocks in the slice correspond to the feature adjustment flag, and there is no limitation on this.

[0230] Method 2: The encoding end encodes the first value range of the characteristic value in the code stream, and the decoding end decodes the first value range of the characteristic value from the code stream. Based on this, if the characteristic value of the second feature (such as all characteristic values) is within the first value range, the decoding end determines that the characteristic adjustment mode is enabled for the current image block. Otherwise, if the characteristic value of the second feature (such as any characteristic value) is not within the first value range, the decoding end determines that the characteristic adjustment mode is prohibited for the current image block. For example, after obtaining the second feature, the decoding end can determine whether the characteristic value of the second feature is within the first value range.

[0231] Exemplarily, the first value range may be a first value range at the sequence level, i.e., all image blocks in the sequence correspond to the first value range; or, the first value range may be a first value range at the image level, i.e., all image blocks in the image correspond to the first value range; or, the first value range may be a first value range at the slice level, i.e., all image blocks in the slice correspond to the first value range, and there is no limitation on this first value range.

[0232] Method 3, the encoder encodes the second value range of the feature change value in the code stream, and the decoder decodes the second value range of the feature change value from the code stream. Based on this, if the feature change value corresponding to the second feature (such as the feature change value corresponding to all feature values, that is, each feature value corresponds to a feature change value) is within the second value range, the decoder determines that the feature adjustment mode is enabled for the current image block. Otherwise, if the feature change value corresponding to the second feature (such as any feature change value) is not within the second value range, the decoder determines that the feature adjustment mode is prohibited for the current image block. For example, after obtaining the second feature, the decoder can also determine the feature change value corresponding to the second feature, that is, determine the feature change value corresponding to each feature value in the second feature. The feature change value can represent the range of the degree of change in the spatial domain or channel domain, that is, determine the change of the second feature in the spatial domain or channel domain. For example, the feature change value may include but is not limited to the gradient value.

[0233] Exemplarily, the second value range may be a second value range at the sequence level, i.e., all image blocks in the sequence correspond to the second value range; or, the second value range may be a second value range at the image level, i.e., all image blocks in the image correspond to the second value range; or, the second value range may be a second value range at the slice level, i.e., all image blocks in the slice correspond to the second value range, and there is no limitation on this second value range.

[0234] Mode 4: The encoder encodes the feature adjustment flag and the first value range in the code stream, and the decoder decodes the feature adjustment flag and the first value range from the code stream. If the feature adjustment flag allows the current image block to enable the feature adjustment mode, and the feature value of the second feature (such as all feature values) is within the first value range, it is determined that the current image block enables the feature adjustment mode. Otherwise, if the feature adjustment flag prohibits the current image block from enabling the feature adjustment mode, and / or the feature value of the second feature (such as any feature value) is not within the first value range, it is determined that the current image block prohibits enabling the feature adjustment mode.

[0235] Method 5: The encoder encodes the feature adjustment flag and the second value range in the code stream, and the decoder decodes the feature adjustment flag and the second value range from the code stream. If the feature adjustment flag allows the current image block to enable the feature adjustment mode, and the feature change value corresponding to the second feature (such as the feature change value corresponding to all feature values, that is, each feature value corresponds to a feature change value) is within the second value range, the decoder determines that the current image block enables the feature adjustment mode. Otherwise, if the feature adjustment flag prohibits the current image block from enabling the feature adjustment mode, and / or the feature change value corresponding to the second feature (such as any feature change value) is not within the second value range, it is determined that the current image block prohibits enabling the feature adjustment mode.

[0236] Method 6: The encoding end encodes the first value range and the second value range in the code stream, and the decoding end decodes the first value range and the second value range from the code stream. If the feature value of the second feature is within the first value range, and the feature change value corresponding to the second feature is within the second value range, the decoding end can determine that the feature adjustment mode is enabled for the current image block. Otherwise, if the feature value of the second feature is not within the first value range, and / or the feature change value corresponding to the second feature is not within the second value range, the decoding end can determine that the feature adjustment mode is prohibited for the current image block.

[0237] Method 7: The encoding end encodes the feature adjustment flag, the first value range, and the second value range in the code stream, and the decoding end decodes the feature adjustment flag, the first value range, and the second value range from the code stream. If the feature adjustment flag allows the current image block to enable the feature adjustment mode, and the feature value of the second feature is within the first value range, and the feature change value corresponding to the second feature is within the second value range, it is determined that the current image block enables the feature adjustment mode. Otherwise, if the feature adjustment flag prohibits the current image block from enabling the feature adjustment mode, the feature value of the second feature is not within the first value range, and / or the feature change value corresponding to the second feature is not within the second value range, it is determined that the current image block prohibits enabling the feature adjustment mode.

[0238] Method 8: If the attribute value corresponding to the second feature is within the configured attribute value range, the decoding end can determine that the feature adjustment mode is enabled for the current image block. Otherwise, if the attribute value corresponding to the second feature is not within the configured attribute value range, the decoding end can determine that the feature adjustment mode is prohibited for the current image block. In Method 8, the decoding end does not need to parse the information related to the feature adjustment mode from the bitstream, but determines whether the feature adjustment mode is enabled for the current image block based on the configured attribute value range. In Method 8, the attribute value corresponding to the second feature may include, but is not limited to, the variance value corresponding to the feature value in the second feature, and the variance value is used to indicate the magnitude of the degree of change in the position of the feature value of the second feature.

[0239] Embodiment 16: In Embodiment 14, for each feature value in the second feature, feature adjustment can be performed on the feature value based on the feature adjustment value corresponding to the feature value to obtain the adjusted feature value corresponding to the feature value. For the above feature adjustment process, it is necessary to first determine whether the conditions are met. If the conditions are met, the feature adjustment process is performed (that is, the feature value can be feature adjusted based on the feature adjustment value corresponding to the feature value to obtain the adjusted feature value corresponding to the feature value).

[0240] Exemplarily, these conditions may include but are not limited to: 1. Based on the additional code stream parsing related information to determine whether to adjust the eigenvalue, this information may be flag information indicating whether the area where the current eigenvalue is located (it may be the feature, the channel of the feature, or a slice of the channel) needs to be adjusted, or range information of the size of the eigenvalue that needs to be adjusted (determine whether the current eigenvalue is within the range to determine whether adjustment is required), or the range of degree of change in the spatial domain or channel domain of the eigenvalue that needs to be adjusted (calculate the change of the current eigenvalue in the spatial domain or channel domain (such as the gradient value), and determine whether the change is within the range to determine whether adjustment is required). 2. Make a judgment based on a certain attribute of the eigenvalue, such as the variance corresponding to the current eigenvalue (characterizing the degree of change in the location of the eigenvalue).

[0241] Exemplarily, for each eigenvalue in the second feature, in order to determine whether to perform feature adjustment on the eigenvalue based on the feature adjustment value corresponding to the eigenvalue to obtain the adjusted eigenvalue corresponding to the eigenvalue, the following method may be used:

[0242] Method 1: The encoding end encodes the feature adjustment flag corresponding to the feature value in the code stream, and the decoding end decodes the feature adjustment flag corresponding to the feature value from the code stream. The feature adjustment flag indicates that the feature value is adjusted, or the feature value is not adjusted. If the feature adjustment flag indicates that the feature value is adjusted, then the decoding end performs feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value to obtain the adjusted feature value corresponding to the feature value. Otherwise, if the feature adjustment flag indicates that the feature value is not adjusted, then the decoding end does not perform feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value. For example, if the feature adjustment flag is a first value, then the feature adjustment flag is used to indicate that the feature value is adjusted, and if the feature adjustment flag is a second value, then the feature adjustment flag is used to indicate that the feature value is not adjusted.

[0243] Mode 2: The encoding end encodes the first value range of the characteristic value in the code stream, and the decoding end decodes the first value range of the characteristic value from the code stream. Based on this, if the characteristic value is within the first value range, the decoding end performs characteristic adjustment on the characteristic value based on the characteristic adjustment value corresponding to the characteristic value to obtain the adjusted characteristic value corresponding to the characteristic value. Otherwise, if the characteristic value is not within the first value range, the decoding end does not perform characteristic adjustment on the characteristic value based on the characteristic adjustment value corresponding to the characteristic value.

[0244] Mode 3: The encoder encodes the second value range of the feature change value in the code stream, and the decoder decodes the second value range of the feature change value from the code stream. Based on this, if the feature change value (such as the gradient value) corresponding to the feature value is within the second value range, the decoder performs feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value to obtain the adjusted feature value corresponding to the feature value. Otherwise, if the feature change value (such as the gradient value) corresponding to the feature value is not within the second value range, the decoder does not perform feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value.

[0245] Mode 4: The encoder encodes the feature adjustment flag and the first value range in the bitstream, and the decoder decodes the feature adjustment flag and the first value range from the bitstream. If the feature adjustment flag indicates that the feature value is to be adjusted, and the feature value is within the first value range, the decoder performs feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value. Otherwise, the decoder does not perform feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value.

[0246] Mode 5: The encoder encodes the feature adjustment flag and the second value range in the bitstream, and the decoder decodes the feature adjustment flag and the second value range from the bitstream. If the feature adjustment flag indicates that the feature value is to be adjusted, and the feature change value corresponding to the feature value is within the second value range, the decoder performs feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value. Otherwise, the feature value is not adjusted based on the feature adjustment value corresponding to the feature value.

[0247] Mode 6: The encoding end encodes the first value range and the second value range in the code stream, and the decoding end decodes the first value range and the second value range from the code stream. If the eigenvalue is within the first value range, and the eigenvalue corresponding to the eigenvalue is within the second value range, the decoding end performs feature adjustment on the eigenvalue based on the feature adjustment value corresponding to the eigenvalue. Otherwise, the decoding end does not perform feature adjustment on the eigenvalue based on the feature adjustment value corresponding to the eigenvalue.

[0248] Mode 7: The encoding end encodes the feature adjustment flag, the first value range, and the second value range in the code stream, and the decoding end decodes the feature adjustment flag, the first value range, and the second value range from the code stream. If the feature adjustment flag indicates that the feature value is to be adjusted, and the feature value is in the first value range, and the feature change value corresponding to the feature value is in the second value range, then the decoding end performs feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value. Otherwise, the decoding end does not perform feature adjustment on the feature value based on the feature adjustment value corresponding to the feature value.

[0249] Mode 8: If the attribute value corresponding to the eigenvalue is within the configured attribute value range, the decoding end performs feature adjustment on the eigenvalue based on the feature adjustment value corresponding to the eigenvalue. Otherwise, the decoding end does not perform feature adjustment on the eigenvalue based on the feature adjustment value corresponding to the eigenvalue. In Mode 8, the decoding end does not need to parse out information related to the feature adjustment mode from the bitstream, but determines whether to perform feature adjustment on the eigenvalue based on the feature adjustment value corresponding to the eigenvalue based on the configured attribute value range. In Mode 8, the attribute value corresponding to the eigenvalue may include, but is not limited to, the variance value corresponding to the eigenvalue, and the variance value is used to indicate the magnitude of the degree of drastic change in the position of the eigenvalue.

[0250] Embodiment 17: Referring to Embodiment 7, the encoding end may obtain a feature adjustment factor for the decoding subnetwork, and perform feature adjustment on a feature in the decoding subnetwork based on the feature adjustment factor, and encode the feature adjustment factor for the decoding subnetwork in the auxiliary bitstream. The decoding end may decode the feature adjustment factor for the decoding subnetwork from the auxiliary bitstream, and perform feature adjustment on a feature in the decoding subnetwork based on the feature adjustment factor. For example, after the decoding subnetwork obtains feature A (feature A is any feature in the decoding subnetwork), the feature adjustment factor may be used to perform feature adjustment on feature A to obtain the adjusted feature B. The decoding subnetwork continues processing based on feature B to finally obtain the first feature.

[0251] In a possible embodiment, see Example 8 Fig. 8A As shown, the feature adjustment factor for the decoding subnetwork can be used in the feature recovery process in the decoding subnetwork, that is, in the feature recovery process, the feature adjustment factor is used to adjust a certain feature. The feature adjustment process can refer to the feature adjustment process of the second feature and will not be repeated here.

[0252] In a possible embodiment, see Example 9 Figure 8B As shown, the feature adjustment factor for the decoding subnetwork can be used in the residual recovery process in the decoding subnetwork, that is, in the residual recovery process, the feature adjustment factor is used to adjust a certain feature. The feature adjustment process can refer to the feature adjustment process of the second feature, which will not be repeated here.

[0253] Illustratively, the above-mentioned embodiments can be implemented individually or in combination. For example, each embodiment in Embodiment 1 to Embodiment 17 can be implemented individually, and at least two embodiments in Embodiment 1 to Embodiment 17 can be implemented in combination.

[0254] Illustratively, in the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.

[0255] Based on the same application concept as the above method, a decoding device is also proposed in the embodiment of the present application. The device is applied to the decoding end, and the device includes: a memory, which is configured to store video data; a decoder, which is configured to implement the decoding method in the above embodiments 1 to 17, that is, the processing flow of the decoding end.

[0256] For example, in one possible implementation, the decoder is configured to implement:

[0257] Acquire a first feature corresponding to the current image block through a first neural network, and acquire a feature adjustment factor corresponding to the current image block, wherein the first neural network includes at least one convolutional layer;

[0258] determining a target feature based on the first feature and the feature adjustment factor;

[0259] Based on the target feature, a reconstructed image block corresponding to the current image block is obtained through a second neural network, where the second neural network includes at least one convolutional layer.

[0260] Based on the same application concept as the above method, an encoding device is also proposed in the embodiment of the present application. The device is applied to the encoding end, and the device includes: a memory, which is configured to store video data; an encoder, which is configured to implement the encoding method in the above embodiments 1 to 17, that is, the processing flow of the encoding end.

[0261] For example, in one possible implementation, the encoder is configured to implement:

[0262] Acquire a first feature corresponding to the current image block through a first neural network, where the first neural network includes at least one convolutional layer; and acquire a feature adjustment factor corresponding to the current image block based on the first feature;

[0263] The feature adjustment factor is encoded in an auxiliary code stream corresponding to the current image block.

[0264] Based on the same application concept as the above method, the decoding end device (also referred to as a video decoder) provided in the embodiment of the present application, from the hardware level, its hardware architecture diagram can be specifically referred to as Fig.11A As shown. It includes: a processor 1101 and a machine-readable storage medium 1102, the machine-readable storage medium 1102 stores machine-executable instructions that can be executed by the processor 1101; the processor 1101 is used to execute the machine-executable instructions to implement the decoding method of the above-mentioned embodiments 1-17 of the present application. For example, in a possible implementation, the decoding end device is used to implement:

[0265] Acquire a first feature corresponding to the current image block through a first neural network, and acquire a feature adjustment factor corresponding to the current image block, wherein the first neural network includes at least one convolutional layer;

[0266] determining a target feature based on the first feature and the feature adjustment factor;

[0267] Based on the target feature, a reconstructed image block corresponding to the current image block is obtained through a second neural network, where the second neural network includes at least one convolutional layer.

[0268] Based on the same application concept as the above method, the encoding end device (also referred to as a video encoder) provided in the embodiment of the present application, from the hardware level, its hardware architecture diagram can be specifically referred to as Fig. 11B As shown. It includes: a processor 1111 and a machine-readable storage medium 1112, the machine-readable storage medium 1112 stores machine-executable instructions that can be executed by the processor 1111; the processor 1111 is used to execute the machine-executable instructions to implement the encoding method of the above-mentioned embodiments 1-17 of the present application. For example, in a possible implementation, the encoding end device is used to implement:

[0269] Acquire a first feature corresponding to the current image block through a first neural network, where the first neural network includes at least one convolutional layer; and acquire a feature adjustment factor corresponding to the current image block based on the first feature;

[0270] The feature adjustment factor is encoded in an auxiliary code stream corresponding to the current image block.

[0271] Based on the same application concept as the above method, an embodiment of the present application provides an electronic device, including: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the decoding method or encoding method of the above embodiments 1-17 of the present application.

[0272] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented, such as the decoding method or encoding method in the above embodiments.

[0273] Based on the same application concept as the above method, an embodiment of the present application also provides a computer application, which, when executed by a processor, can implement the decoding method or encoding method disclosed in the above example of the present application.

[0274] Based on the same application concept as the above method, a decoding device is also proposed in an embodiment of the present application. The decoding device can be applied to a decoding end. The decoding device may include: an acquisition module, used to acquire a first feature corresponding to a current image block through a first neural network, and acquire a feature adjustment factor corresponding to the current image block, wherein the first neural network includes at least one convolutional layer; a determination module, used to determine a target feature based on the first feature and the feature adjustment factor; the acquisition module is also used to acquire a reconstructed image block corresponding to the current image block through a second neural network based on the target feature, wherein the second neural network includes at least one convolutional layer.

[0275] Exemplarily, when the acquisition module acquires the first feature corresponding to the current image block through the first neural network, it is specifically used to: acquire probability distribution parameters based on the first code stream corresponding to the current image block; determine a probability distribution model based on the probability distribution parameters, and decode the second code stream corresponding to the current image block based on the probability distribution model to obtain decoded image features; and determine the first feature corresponding to the current image block based on the decoded image features.

[0276] Exemplarily, when the acquisition module acquires the first feature corresponding to the current image block through the first neural network, it is specifically used to: acquire probability distribution parameters and prediction values ​​based on the first code stream corresponding to the current image block; determine a probability distribution model based on the probability distribution parameters, and decode the second code stream corresponding to the current image block based on the probability distribution model to obtain decoded image features; perform residual recovery on the decoded image features to obtain residual features; and determine the first feature corresponding to the current image block based on the residual features and the prediction value.

[0277] Exemplarily, when the acquisition module acquires the feature adjustment factor corresponding to the current image block, it is specifically used to: decode the auxiliary code stream corresponding to the current image block to obtain the feature adjustment factor corresponding to the current image block; or determine a fixed parameter value as the feature adjustment factor corresponding to the current image block.

[0278] Exemplarily, when the determination module determines the target feature based on the first feature and the feature adjustment factor, it is specifically used to: perform feature enhancement on the first feature to obtain a second feature; perform feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature; and determine the target feature based on the third feature.

[0279] Exemplarily, when the determination module performs feature enhancement on the first feature to obtain the second feature, it is specifically used to: determine an initial feature corresponding to the attention subnetwork based on the first feature; perform feature enhancement on the initial feature through the attention subnetwork to obtain the second feature; Exemplarily, when the determination module determines the target feature based on the third feature, it is specifically used to: determine the third feature as the target feature.

[0280] The attention subnetwork includes a residual enhancement subnetwork and a weight generation subnetwork. The determination module performs feature enhancement on the initial feature through the attention subnetwork to obtain the second feature by: using the residual enhancement subnetwork to perform feature enhancement on the initial feature to obtain an enhanced feature; using the weight generation subnetwork to generate a weight feature corresponding to the initial feature; and generating the second feature based on the initial feature, the enhanced feature and the weight feature.

[0281] Exemplarily, the weight generation subnetwork may include a residual block subnetwork, a convolution subnetwork and a feature mapping subnetwork. When the determination module uses the weight generation subnetwork to generate the weight feature corresponding to the initial feature, it is specifically used to: use the residual block subnetwork to perform convolution activation on the initial feature to obtain a post-convolution feature; use the convolution subnetwork to convolve the post-convolution activation feature to obtain a post-convolution feature; use the feature mapping subnetwork to perform feature mapping on the post-convolution feature to obtain the weight feature corresponding to the initial feature.

[0282] Exemplarily, when the determination module performs feature enhancement on the first feature to obtain the second feature, it is specifically used to: determine the initial feature corresponding to the attention subnetwork based on the first feature; perform feature enhancement on the initial feature through the first subnetwork in the attention subnetwork to obtain the second feature; when the determination module determines the target feature based on the third feature, it is specifically used to: process the third feature through the second subnetwork in the attention subnetwork to obtain a fourth feature, and determine the fourth feature as the target feature.

[0283] Exemplarily, the determination module performs feature enhancement on the initial feature through the first subnetwork in the attention subnetwork, and when obtaining the second feature, is specifically used for: using a residual enhancement subnetwork to perform feature enhancement on the initial feature to obtain an enhanced feature; using a weight generation subnetwork to generate a weight feature corresponding to the initial feature; generating the second feature based on the enhanced feature and the weight feature; the determination module processes the third feature through the second subnetwork in the attention subnetwork, and when obtaining the fourth feature, is specifically used for: generating the fourth feature based on the initial feature and the third feature.

[0284] Exemplarily, the determination module performs feature enhancement on the initial feature through the first subnetwork in the attention subnetwork, and when obtaining the second feature, is specifically used for: using a residual block subnetwork to perform feature enhancement on the initial feature to obtain an enhanced feature; using a convolution subnetwork to convolve the enhanced feature to obtain a convolution feature, and determining the second feature based on the convolution feature; the determination module processes the third feature through the second subnetwork in the attention subnetwork, and when obtaining the fourth feature, is specifically used for: using a feature mapping subnetwork to perform feature mapping on the third feature to obtain a weight feature; using a residual enhancement subnetwork to perform feature enhancement on the initial feature to obtain an enhanced feature; and generating the fourth feature based on the initial feature, the enhanced feature and the weight feature.

[0285] Exemplarily, when the determination module determines the initial feature corresponding to the attention subnetwork based on the first feature, it is specifically used to: use an initial enhancement subnetwork to perform feature enhancement on the first feature to obtain an enhanced feature; and use an upsampling convolution subnetwork to upsample and convolve the enhanced feature to obtain an initial feature.

[0286] Exemplarily, the determination module performs feature adjustment on the second feature based on the feature adjustment factor to obtain the third feature, which is specifically used for: if the second feature includes C*H*W feature values, C represents the number of channels, H represents the feature height, and W represents the feature width, then the feature adjustment value corresponding to each feature value is determined based on the feature adjustment factor, and the feature value is feature adjusted based on the feature adjustment value corresponding to the feature value to obtain the adjusted feature value corresponding to the feature value; the third feature is generated based on the adjusted feature value corresponding to each feature value.

[0287] Exemplarily, when the determination module determines the feature adjustment value corresponding to each feature value based on the feature adjustment factor, it is specifically used to: if the feature adjustment factor includes C*H*W feature adjustment values, then determine the feature adjustment values ​​corresponding to the C*H*W feature values ​​respectively based on the C*H*W feature adjustment values; or, if the feature adjustment factor includes H*W feature adjustment values, then determine the feature adjustment values ​​corresponding to the C*H*W feature values ​​respectively based on the H*W feature adjustment values; or, if the feature adjustment factor includes C feature adjustment values, then determine the feature adjustment values ​​corresponding to the C*H*W feature values ​​respectively based on the C feature adjustment values.

[0288] Exemplarily, the determination module performs feature adjustment on the eigenvalue based on the feature adjustment value corresponding to the eigenvalue, and obtains the adjusted eigenvalue corresponding to the eigenvalue by: if the eigenvalue corresponds to only one feature adjustment value, the adjusted eigenvalue is determined by the following formula: yr=rf*y; wherein yr represents the adjusted eigenvalue, rf represents the feature adjustment value, and y represents the eigenvalue; or, if the eigenvalue corresponds to N+1 feature adjustment values, and N is a positive integer, the adjusted eigenvalue is determined by the following formula: yr=rf_0*y 0 +rf_1*y 1 +rf_2*y 2 +…+rf_N*y N ; Among them, yr represents the adjusted eigenvalue, rf_0, rf_1, rf_2, ..., rf_N represent N+1 feature adjustment values, and y represents the eigenvalue.

[0289] Exemplarily, the second neural network includes a reconstruction decoding subnetwork, and the acquisition module is specifically used to obtain the reconstructed image block corresponding to the current image block through the second neural network based on the target feature: processing the target feature through the reconstruction decoding subnetwork to obtain the reconstructed image block; wherein the reconstruction decoding subnetwork includes an upsampling convolution subnetwork, and upsampling convolution is performed on the target feature through the upsampling convolution subnetwork to obtain the reconstructed image block; or, the reconstruction decoding subnetwork includes an upsampling convolution subnetwork and a color space transformation subnetwork, and upsampling convolution is performed on the target feature through the upsampling convolution subnetwork, and color space transformation is performed on the upsampled convolution feature through the color space transformation subnetwork to obtain the reconstructed image block.

[0290] Based on the same application concept as the above method, an encoding device is also proposed in an embodiment of the present application, and the device is applied to the encoding end, and the device includes: an acquisition module, used to obtain a first feature corresponding to the current image block through a first neural network, and the first neural network includes at least one convolutional layer; based on the first feature, a feature adjustment factor corresponding to the current image block is obtained; an encoding module, used to encode the feature adjustment factor in an auxiliary code stream corresponding to the current image block.

[0291] Exemplarily, when the acquisition module acquires the feature adjustment factor corresponding to the current image block based on the first feature, it is specifically used to: acquire at least one candidate feature adjustment factor; determine the rate-distortion cost corresponding to each candidate feature adjustment factor based on the first feature; and select a candidate feature adjustment factor from all candidate feature adjustment factors as the feature adjustment factor corresponding to the current image block based on the rate-distortion cost corresponding to each candidate feature adjustment factor.

[0292] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. The present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. The embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The above is only an embodiment of the present application and is not intended to limit the present application.

[0293] For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image decoding method, characterized in that: The method comprises: Acquire a first feature corresponding to the current image block through a first neural network, and acquire a feature adjustment factor corresponding to the current image block; Performing feature enhancement on the first feature to obtain a second feature; Based on the feature adjustment factor, feature adjustment is performed on the second feature to obtain a third feature; wherein, if the second feature includes C*H*W feature values, C represents the number of channels, H represents the feature height, and W represents the feature width, feature adjustment values ​​corresponding to the C*H*W feature values ​​are determined based on the feature adjustment factor, and feature adjustment is performed on the C*H*W feature values ​​based on the feature adjustment value to obtain adjusted feature values ​​corresponding to the C*H*W feature values; and the third feature is generated based on the adjusted feature values ​​corresponding to the C*H*W feature values; determining a target feature based on the third feature; Based on the target feature, a reconstructed image block corresponding to the current image block is obtained through a second neural network.

2. The method according to claim 1, characterized in that: The determining, based on the feature adjustment factor, the feature adjustment values ​​corresponding to the C*H*W feature values ​​comprises: If the feature adjustment factor includes C*H*W feature adjustment values, then determining the feature adjustment values ​​respectively corresponding to the C*H*W feature values ​​based on the C*H*W feature adjustment values; or, If the feature adjustment factor includes H*W feature adjustment values, then determining the feature adjustment values ​​corresponding to the C*H*W feature values ​​respectively based on the H*W feature adjustment values; or, If the feature adjustment factor includes C feature adjustment values, the feature adjustment values ​​respectively corresponding to the C*H*W feature values ​​are determined based on the C feature adjustment values.

3. The method according to claim 1, characterized in that: The first neural network includes a decoding subnetwork, and obtaining a first feature corresponding to the current image block through the first neural network includes: Obtaining probability distribution parameters based on the bitstream corresponding to the current image block; determining a probability distribution model based on the probability distribution parameters, and decoding the bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features; determining a first feature corresponding to the current image block based on the decoded image features; Alternatively, probability distribution parameters and prediction values ​​are obtained based on the code stream corresponding to the current image block; a probability distribution model is determined based on the probability distribution parameters, and the code stream corresponding to the current image block is decoded based on the probability distribution model to obtain decoded image features; residual recovery is performed on the decoded image features to obtain residual features; and based on the residual features and the prediction value, a first feature corresponding to the current image block is determined.

4. The method according to claim 1, characterized in that: The obtaining the feature adjustment factor corresponding to the current image block includes: Decoding the code stream corresponding to the current image block to obtain a feature adjustment factor corresponding to the current image block; or, The fixed parameter value is determined as the feature adjustment factor corresponding to the current image block.

5. The method according to claim 1, characterized in that The second neural network includes a reconstruction decoding subnetwork, and the step of obtaining a reconstructed image block corresponding to the current image block through the second neural network based on the target feature includes: Processing the target feature through a reconstruction decoding subnetwork to obtain the reconstructed image block; Among them, the reconstruction and decoding subnetwork includes an upsampling convolution subnetwork, and the target features are upsampled and convolved through the upsampling convolution subnetwork to obtain the reconstructed image block; or, the reconstruction and decoding subnetwork includes an upsampling convolution subnetwork and a color space transformation subnetwork, and the target features are upsampled and convolved through the upsampling convolution subnetwork, and the color space transformation is performed on the features after the upsampling convolution through the color space transformation subnetwork to obtain the reconstructed image block.

6. The method according to claim 1, characterized in that The method further comprises: If the current image block enables the feature adjustment mode, then the feature adjustment factor corresponding to the current image block is obtained; wherein the process of determining whether the current image block enables the feature adjustment mode includes: Parsing a feature adjustment flag from the bitstream, and if the feature adjustment flag allows the current image block to enable the feature adjustment mode, determining that the current image block enables the feature adjustment mode; or, A first value range of a characteristic value is parsed from the code stream, and if the characteristic value of the second characteristic is within the first value range, it is determined that a characteristic adjustment mode is enabled for the current image block.

7. An image encoding method, characterized in that: The method comprises: Acquire a first feature corresponding to the current image block through a first neural network, and acquire a feature adjustment factor corresponding to the current image block based on the first feature; and encode the feature adjustment factor in a bitstream corresponding to the current image block; Performing feature enhancement on the first feature to obtain a second feature; Based on the feature adjustment factor, feature adjustment is performed on the second feature to obtain a third feature; wherein, if the second feature includes C*H*W feature values, C represents the number of channels, H represents the feature height, and W represents the feature width, feature adjustment values ​​corresponding to the C*H*W feature values ​​are determined based on the feature adjustment factor, and feature adjustment is performed on the C*H*W feature values ​​based on the feature adjustment value to obtain adjusted feature values ​​corresponding to the C*H*W feature values; and the third feature is generated based on the adjusted feature values ​​corresponding to the C*H*W feature values; determining a target feature based on the third feature; Based on the target feature, a reconstructed image block corresponding to the current image block is obtained through a second neural network.

8. An image decoding device, characterized in that: The device comprises: An acquisition module, configured to acquire a first feature corresponding to a current image block through a first neural network, and acquire a feature adjustment factor corresponding to the current image block; A determination module, configured to perform feature enhancement on the first feature to obtain a second feature; perform feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature; wherein, if the second feature includes C*H*W feature values, C represents the number of channels, H represents the feature height, and W represents the feature width, then feature adjustment values ​​corresponding to the C*H*W feature values ​​are determined based on the feature adjustment factor, and feature adjustment is performed on the C*H*W feature values ​​based on the feature adjustment value to obtain adjusted feature values ​​corresponding to the C*H*W feature values; generate the third feature based on the adjusted feature values ​​corresponding to the C*H*W feature values; and determine the target feature based on the third feature; The acquisition module is further used to acquire the reconstructed image block corresponding to the current image block through a second neural network based on the target feature.

9. An image encoding device, characterized in that: The device comprises: an acquisition module, configured to acquire a first feature corresponding to a current image block through a first neural network, and acquire a feature adjustment factor corresponding to the current image block based on the first feature; An encoding module, used for encoding the feature adjustment factor in a bit stream corresponding to the current image block; A determination module, configured to perform feature enhancement on the first feature to obtain a second feature; perform feature adjustment on the second feature based on the feature adjustment factor to obtain a third feature; wherein, if the second feature includes C*H*W feature values, C represents the number of channels, H represents the feature height, and W represents the feature width, then feature adjustment values ​​corresponding to the C*H*W feature values ​​are determined based on the feature adjustment factor, and feature adjustment is performed on the C*H*W feature values ​​based on the feature adjustment value to obtain adjusted feature values ​​corresponding to the C*H*W feature values; generate the third feature based on the adjusted feature values ​​corresponding to the C*H*W feature values; and determine the target feature based on the third feature; The acquisition module is further used to acquire the reconstructed image block corresponding to the current image block through a second neural network based on the target feature.

10. An image decoding device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method according to any one of claims 1 to 6.

11. An image encoding device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is configured to execute machine executable instructions to implement the method of claim 7.

12. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores a plurality of computer instructions, and when the computer instructions are executed by a processor, the method described in any one of claims 1 to 6 is implemented; or, when the computer instructions are executed by a processor, the method described in claim 7 is implemented.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented; or, when the computer program is executed by a processor, the method according to claim 7 is implemented.

Images