Image decoding and encoding method, apparatus, and device based on a neural network

Abstract

The present invention provides an image decoding and encoding method, apparatus, and device based on a neural network. The method includes decoding a control parameter and image information corresponding to a current block from a bitstream, obtaining neural network information corresponding to a decoding processing unit from the control parameter, generating a decoding neural network corresponding to the decoding processing unit based on the neural network information, determining input features corresponding to the decoding processing unit based on the image information, and processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit. According to the present invention, the encoding performance and decoding performance can be improved.

Description

Technical Field

[0001] The present invention relates to the field of encoding and decoding technologies, and particularly relates to an image decoding, encoding method, device, and its device based on a neural network.

Background Art

[0002] In order to save space, all video images are encoded and then transmitted. Complete video encoding may include processes such as prediction, transformation, quantization, entropy encoding, filtering, etc. For the prediction process, the prediction process may include intra-frame prediction and inter-frame prediction. Inter-frame prediction utilizes the temporal correlation of the video to predict the current pixel using the pixels of adjacent encoded images, thereby achieving the purpose of effectively removing the temporal redundancy of the video. Intra-frame prediction utilizes the spatial correlation of the video to predict the current pixel using the pixels of the encoded blocks of the image in the current frame, thereby achieving the purpose of removing the spatial redundancy of the video.

[0003] With the rapid development of deep learning, deep learning has achieved good results in many high-level computer vision problems such as image classification and target detection. Deep learning has also gradually begun to be applied in the field of encoding and decoding, that is, it has become possible to encode and decode images using a neural network. Although the encoding and decoding methods based on neural networks show great performance potential, the encoding and decoding methods based on neural networks still have problems such as low stability, low generalization ability, and high complexity.

Summary of the Invention

[0004] In view of this, the present invention provides an image decoding, encoding method, device, and its device based on a neural network, which improve the encoding performance and decoding performance and solve problems such as low stability, low generalization ability, and high complexity.

[0005] The present invention relates to an image decoding method based on a neural network applied to the decoding side, Control parameters corresponding to the current block from the bitstream Ta The decryption step, The steps include obtaining neural network information corresponding to the decoding processing unit from the control parameters, and generating a decoding neural network corresponding to the decoding processing unit based on the neural network information, before The present invention provides a neural network-based image decoding method, which includes the steps of determining input features corresponding to a decoding processing unit, and processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit.

[0006] In some embodiments, when the neural network information includes basic layer information, the step of generating a decoding neural network corresponding to the decoding processing unit based on the neural network information is: The steps include determining the basic layer corresponding to the decoding processing unit based on the basic layer information, The process includes the step of generating a decoding neural network corresponding to the decoding processing unit based on the basic layer.

[0007] In some embodiments, the step of determining the base layer corresponding to the decoding processing unit based on the base layer information is: The basic layer information includes a basic layer default network use flag bit, and if the basic layer default network use flag bit indicates that the basic layer uses the default network, the step includes obtaining the basic layer of the default network structure.

[0008] In some embodiments, the step of determining the base layer corresponding to the decoding processing unit based on the base layer information is: The basic layer information includes a basic layer predesign network use flag bit and a basic layer predesign network index number, and if the basic layer predesign network use flag bit indicates that the basic layer uses a predesign network, the step includes selecting a basic layer of the predesign network structure corresponding to the basic layer predesign network index number from the predesign neural network pool. The aforementioned pre-designed neural network pool includes at least one network layer of a pre-designed network structure.

[0009] In some embodiments, when the basic layer is a first feature decoding network, the step of processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit is: The process includes the steps of decoding a first bitstream of the current block that is input based on the first feature decoding network and obtaining coefficient hyperparameter feature information corresponding to the current block that is output.

[0010] In some embodiments, when the basic layer is a coefficient hyperparameter feature generation network, the step of processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit is: The process includes processing input coefficient hyperparameter feature coefficient reconstruction values ​​based on the coefficient hyperparameter feature generation network and obtaining output coefficient hyperparameter feature values ​​corresponding to the current block, wherein the coefficient hyperparameter feature coefficient reconstruction values ​​input to the coefficient hyperparameter feature generation network are determined based on coefficient hyperparameter feature information output by the first feature decoding network.

[0011] In some embodiments, the step of determining the coefficient hyperparameter feature coefficient reconstruction value based on the coefficient hyperparameter feature information is: If the control parameter includes first enable information, and the first enable information indicates that a first inverse quantization process is enabled, the process includes the step of performing inverse quantization on the coefficient hyperparameter feature information to obtain a coefficient hyperparameter feature coefficient reconstruction value.

[0012] In some embodiments, when the basic layer is a second feature decoding network, the step of processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit is: The process includes decoding a second bitstream of the current block that is input based on the second feature decoding network and obtaining image feature information corresponding to the current block that is output.

[0013] In some embodiments, the step of decoding the second bitstream of the current block that is input based on the second feature decoding network and obtaining image feature information corresponding to the current block that is output is: The process includes the step of decoding the second bitstream based on the coefficient hyperparameter feature values ​​input to the second feature decoding network to obtain the image feature information, wherein the coefficient hyperparameter feature values ​​are output features of the coefficient hyperparameter feature generation network.

[0014] In some embodiments, when the basic layer is a transformation network, the step of processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit is: The step includes processing the input image feature reconstruction values ​​based on the transformation network and obtaining the output low-level image feature values ​​corresponding to the current block, The image feature reconstruction value input to the transformation network is determined based on the image feature information output by the second feature decoding network. The aforementioned low-level image feature values ​​are used to obtain the reconstructed image block corresponding to the current block.

[0015] In some embodiments, the step of determining the image feature reconstruction value based on the image feature information is: If the control parameter includes a second enable information, and the second enable information indicates that a second inverse quantization process is enabled, the process includes the step of performing inverse quantization on the image feature information to obtain an image feature reconstruction value.

[0016] In some embodiments, when the neural network information includes reinforcement layer information, the step of generating a decoding neural network corresponding to the decoding processing unit based on the neural network information is: The steps include determining the reinforcement layer corresponding to the decoding processing unit based on the reinforcement layer information, The process includes the step of generating a decoding neural network corresponding to the decoding processing unit based on the reinforcement layer.

[0017] In some embodiments, the step of determining the reinforcement layer corresponding to the decoding processing unit based on the reinforcement layer information is: The enhancement layer information includes an enhancement layer default network use flag bit, and if the enhancement layer default network use flag bit indicates that the enhancement layer uses the default network, the step includes obtaining the enhancement layer of the default network structure.

[0018] In some embodiments, the step of determining the reinforcement layer corresponding to the decoding processing unit based on the reinforcement layer information is: The enhancement layer information includes an enhancement layer predesign network use flag bit and an enhancement layer predesign network index number, and if the enhancement layer predesign network use flag bit indicates that the enhancement layer uses a predesign network, the step includes selecting an enhancement layer with a predesign network structure corresponding to the enhancement layer predesign network index number from the predesign neural network pool. The aforementioned pre-designed neural network pool includes at least one network layer of a pre-designed network structure.

[0019] In some embodiments, when the enhancement layer is a transformation network, the step of determining the input features corresponding to the decoding processing unit is: The steps include decoding the bitstream of the current block to obtain image feature information corresponding to the current block, The process includes the step of determining an image feature reconstruction value corresponding to the current block based on the image feature information.

[0020] In some embodiments, the step of processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit is: The process includes processing the input image feature reconstruction values ​​based on the transformation network to obtain output image low-level feature values ​​corresponding to the current block, the image low-level feature values ​​being used to obtain a reconstructed image block corresponding to the current block.

[0021] In some embodiments, the step of determining the image feature reconstruction value corresponding to the current block based on the image feature information is: If the control parameter includes a second enable information, and the second enable information indicates that a second inverse quantization process is enabled, the process includes the step of performing inverse quantization on the image feature information to obtain the image feature reconstruction value.

[0022] In some embodiments, when the enhancement layer is a quality improvement network, the step of processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit is: The step includes performing enhancement processing on the low-level image feature values ​​corresponding to the input current block based on the high-quality network, and obtaining a reconstructed image block corresponding to the output current block.

[0023] In some embodiments, the step of performing enhancement processing on low-level image feature values ​​corresponding to the current block based on the quality enhancement network to obtain a reconstructed image block corresponding to the current block includes the step of performing enhancement processing on low-level image feature values ​​corresponding to the current block based on the quality enhancement network to obtain a reconstructed image block corresponding to the current block, if the control parameter includes third enable information and the third enable information indicates that the quality enhancement processing should be enabled.

[0024] In some embodiments, the step of decoding the control parameter corresponding to the current block from the bitstream is: The process includes the step of decoding the bitstream corresponding to the current block via a control parameter decoding network to obtain control parameters corresponding to the current block, wherein the control parameters include control parameters of the transformation network and control parameters of the quality enhancement network.

[0025] In some embodiments, the step of determining the reinforcement layer corresponding to the decoding processing unit based on the reinforcement layer information is: If the enhancement layer information includes network parameters for generating the enhancement layer, the step includes generating the enhancement layer corresponding to the decoding processing unit based on the network parameters. The aforementioned network parameters are: The neural network includes at least one of the following: number of layers, number of filters, filtering size index, zero filtering coefficient flag bit, and filtering coefficient.

[0026] The present invention relates to an image coding method based on a neural network applied to the coding side, current A step of obtaining control parameters corresponding to a current block, wherein the control parameters include neural network information corresponding to a decoding processing unit, and the neural network information is used to determine the decoding neural network corresponding to the decoding processing unit. The bitstream corresponds to the current block. The system The present invention provides a neural network-based image coding method, which includes the step of coding the parameters.

[0027] The present invention A memory configured to store video data, Control parameters corresponding to the current block from the bitstream Ta The decryption step, The steps include obtaining neural network information corresponding to the decoding processing unit from the control parameters, and generating a decoding neural network corresponding to the decoding processing unit based on the neural network information, before The present invention provides a neural network-based image decoding device, which includes a decoder configured to perform the steps of: determining input features corresponding to a decoding processing unit, and processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit.

[0028] The present invention A memory configured to store video data, currentA step of obtaining control parameters corresponding to a current block, wherein the control parameters include neural network information corresponding to a decoding processing unit, and the neural network information is used to determine the decoding neural network corresponding to the decoding processing unit. The bitstream corresponds to the current block. The system The present invention provides a neural network-based image coding device, which includes an encoder configured to perform the steps of encoding parameters.

[0029] The present invention relates to a decoding device comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor. The processor provides a decoding device used to execute machine-executable instructions and implement the neural network-based image decoding method described above.

[0030] The present invention relates to an encoding device comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor. The processor provides an encoding device used to execute machine-executable instructions and implement the image encoding method based on the neural network described above.

[0031] As can be seen from the above technical proposals, in the embodiments of the present invention, control parameters corresponding to the current block are decoded from the bitstream, neural network information corresponding to the decoding processing unit is obtained from the control parameters, a decoding neural network corresponding to the decoding processing unit is generated based on the neural network information, and image decoding can be realized based on the decoding neural network, thereby improving decoding performance. Image coding can be realized based on the encoding neural network corresponding to the encoding processing unit, thereby improving encoding performance. By encoding and decoding images using neural networks (e.g., decoding neural network, encoding neural network, etc.), transmitting neural network information in a bitstream, and generating a decoding neural network corresponding to the decoding processing unit based on the neural network information, problems such as low stability, low generalizability, and high complexity can be solved, i.e., high stability, high generalizability, and low complexity can be achieved. A solution that dynamically adjusts the complexity of encoding and decoding can be provided, and it has better encoding and decoding performance compared to a single neural network framework. Since each current block corresponds to a control parameter, the neural network information obtained from the control parameter is the neural network information for the current block. A decoding neural network is generated for each current block, meaning that the decoding neural networks for different current blocks may be the same or different. Thus, the block-level decoding neural network, i.e., the decoding neural network, is variable and adjustable. [Brief explanation of the drawing]

[0032] [Figure 1] This is a schematic diagram of a video coding framework. [Figure 2A] This is a schematic diagram of a video coding framework. [Figure 2B] This is a schematic diagram of a video coding framework. [Figure 2C]This is a schematic diagram of a video coding framework. [Figure 3] This is a flowchart of a neural network-based image decoding method in one embodiment of the present invention. [Figure 4] This is a flowchart of a neural network-based image coding method in one embodiment of the present invention. [Figure 5A] This is a schematic diagram of an image encoding method and an image decoding method according to one embodiment of the present invention. [Figure 5B] This is a schematic diagram of boundary padding in one embodiment of the present invention. [Figure 5C] This is a schematic diagram of an image encoding method and an image decoding method according to one embodiment of the present invention. [Figure 5D] This is a schematic diagram of boundary padding in one embodiment of the present invention. [Figure 5E] This is a schematic diagram of the image region transformation of the original image in one embodiment of the present invention. [Figure 6A] This is a schematic diagram of the decoding side in one embodiment of the present invention. [Figure 6B] This is a schematic diagram of the decoding side in one embodiment of the present invention. [Figure 6C] This is a schematic diagram of the coefficient hyperparameter feature generation unit in one embodiment of the present invention. [Figure 6D] This is a schematic diagram of the image feature inverse transformation unit in one embodiment of the present invention. [Figure 7A] This is a schematic diagram of the encoding side in one embodiment of the present invention. [Figure 7B] This is a schematic diagram of the encoding side in one embodiment of the present invention. [Figure 7C] This is a schematic diagram of the encoding side in one embodiment of the present invention. [Figure 8A] This is a hardware structure diagram of the decoding device in one embodiment of the present invention. [Figure 8B] This is a hardware structure diagram of an encoding device in one embodiment of the present invention. [Modes for carrying out the invention]

[0033] The terms used in the embodiments of this invention are merely for the purpose of describing specific embodiments and are not intended to limit the invention. The singular forms “one kind,” “the said,” and “the” used in the embodiments and claims of this invention are also intended to include the plural form unless the context clearly indicates otherwise. Furthermore, it should be understood that the term “and / or” used in this invention means including any or all possible combinations of one or more related enumerated items. The embodiments of this invention may use terms such as first, second, third, etc. to describe various types of information, but it should be understood that this information is not limited to these terms. These terms are used only to distinguish the same type of information. For example, as long as it does not deviate from the scope of the embodiments of this invention, depending on the context, first information may be called second information, and similarly, second information may be called first information. Furthermore, the word “…case” used herein may be interpreted as “…and,” “…when,” or “in response to a decision.”

[0034] Embodiments of the present invention provide a neural network-based image decoding and encoding method, which may relate to the following concepts.

[0035] Neural Network (NN): This refers to an artificial neural network, not a biological one. A neural network is a computational model composed of a large number of interconnected nodes (called neurons). In a neural network, neuron processing units can represent different objects, such as features, alphabets, concepts, or several meaningful abstract modes. There are three types of processing units in a neural network: input units, output units, and hidden units. Input units receive external signals and data, output units produce the output of the processing results, and hidden units are units that are between the input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the strength of the connections between units, and the representation and processing of information are reflected in the connection relationships of the processing units. A neural network is an unprogrammed, brain-like information processing method, and its essence is to acquire parallel and distributed information processing capabilities through the transformation and dynamic actions of the neural network, mimicking the information processing capabilities of the human brain and nervous system to different degrees and levels. In the field of video processing, commonly used neural networks may include, but are not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and fully connected networks.

[0036] Convolutional Neural Networks (CNNs): Convolutional neural networks are feedforward neural networks and one of the representative network structures in deep learning techniques. The artificial neurons in a convolutional neural network can respond to peripheral units within a certain coverage area, exhibiting excellent performance in large-scale image processing. The basic structure of a convolutional neural network consists of two layers: one is a feature extraction layer (also called a convolutional layer), where the input of each neuron is connected to the local receptive field of the previous layer, and the local features are extracted. Once the local features are extracted, their positional relationship with other features is also determined. The other is a feature mapping layer (also called an activation layer), where each computational layer of the neural network consists of multiple feature mappings, each feature mapping is a plane, and the weights of all neurons in the plane are equal. The feature mapping structure may use functions such as the Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, and GDN function as activation functions for the convolutional network. Furthermore, because neurons on a single mapping plane share weights, the number of free parameters in the network decreases.

[0037] One advantage of convolutional neural networks compared to image processing algorithms is that they can avoid complex pre-processing processes for images (such as extracting artificial features), directly inputting original images and performing end-to-end learning. Another advantage of convolutional neural networks compared to general neural networks is that while general neural networks employ a fully connected architecture, meaning all neurons from the input layer to the hidden layer are connected, resulting in a huge number of parameters and making network training time-consuming and difficult, convolutional neural networks avoid this difficulty through methods such as local connections and weight sharing.

[0038] Deconvolution: Also known as transposed convolution, deconvolution and convolution layers operate similarly. The main difference is that deconvolution uses padding to make the output larger than the input (though they can also be the same). A stride of 1 indicates that the output size is equal to the input size. A stride of N indicates that the width of the output features is N times the width of the input features, and the height of the output features is N times the height of the input features.

[0039] Generalization Ability: Generalization ability may also refer to a machine learning algorithm's ability to adapt to fresh samples. The goal of learning is to learn the underlying rules of the data, and the trained network can produce appropriate outputs even for data outside the training set that has the same rules. This ability may be called generalization ability.

[0040] The Rate-Distortion Optimization principle: Encoding efficiency is evaluated using two metrics: bitrate and PSNR (Peak Signal to Noise Ratio). A smaller bitstream results in greater compression, and a higher PSNR results in better reconstructed image quality. When selecting a mode, the discriminant is essentially a comprehensive evaluation of both. For example, the cost corresponding to a mode is J(mode) = D + λ*R, where D represents distortion, which can usually be evaluated using the SSE (Sum of the Squared Errors) metric, where SSE is the mean square sum of the differences between the reconstructed image block and the source image. To consider the cost, the SAD metric may also be used, where SAD is the sum of the absolute differences between the reconstructed image block and the source image, λ is the Lagrangian multiplier, and R is the actual number of bits required to encode the image block in that mode, including the total number of bits required to encode mode information, motion information, residuals, etc. When selecting a mode, comparing and determining the encoding mode using the rate distortion principle usually guarantees optimal encoding performance.

[0041] Video encoding framework: Figure 1 is a schematic diagram of the encoding-side video encoding framework. The encoding-side processing process in the embodiment of the present invention may be implemented by this video encoding framework. The schematic diagram of the video decoding framework may be similar to that in Figure 1, so its explanation is omitted here. However, the decoding-side processing process in the embodiment of the present invention may be implemented by a video decoding framework.

[0042] Exemplary, as shown in Figure 1, a video coding framework may include modules such as prediction, transformation, quantization, entropy encoder, inverse quantization, inverse transformation, reconstruction, and filtering. On the coding side, the coding process can be realized through the cooperation of these modules. Similarly, a video decoding framework may include modules such as prediction, transformation, quantization, entropy decoder, inverse quantization, inverse transformation, reconstruction, and filtering, and on the decoding side, the decoding process can be realized through the cooperation of these modules.

[0043] For each module on the encoding side, a great many encoding tools have been proposed, and each tool usually has many modes. The encoding tool that yields optimal encoding performance often differs for different video sequences. Therefore, in the encoding process, Rate-Distortion Optimize (RDO) is typically used to compare the encoding performance of different tools or modes and select the optimal mode. After determining the optimal tool or mode, the decision information is transmitted by encoding mark information into the bitstream. While this method introduces encoding complexity, it allows for the adaptive selection of the optimal mode combination for different content, thereby achieving optimal encoding performance. On the decoding side, the relevant mode information can be obtained by directly analyzing the mark information, minimizing the impact of complexity.

[0044] The structures of the encoding and decoding sides are briefly described below. Figure 2A is a schematic block diagram of an example encoding side for realizing an embodiment of the present invention. In Figure 2A, the encoding side includes a prediction processing unit, a residual calculation unit, a transformation processing unit, a quantization unit, an encoding unit, an inverse quantization unit, an inverse transformation processing unit, a reconstruction unit, and a filter unit. In one example, the encoding side may further include a buffer and a decoded picture buffer (DPB), the buffer being used to buffer the reconstructed image blocks output by the reconstruction unit, and the decoded picture buffer being used to buffer the filtered image blocks output by the filter unit.

[0045] The input to the encoding side (also called the encoder) is an image block of an image (which may also be called the image to be encoded), and the image block is also called the current block or the block to be encoded. The encoding side may further include a splitting unit (not shown) for dividing the image to be encoded into multiple image blocks. The encoding side is used to encode the image to be encoded block by block, for example, by performing an encoding process for each image block. The prediction processing unit is used to receive or acquire the image block (which may also be called the current image block to be encoded of the current image to be encoded, or the current block, and which may be understood as the true value of the image block) and the reconstructed image data, and to predict the current block based on relevant data in the reconstructed image data to obtain the predicted block for the current block. In one example, the prediction processing unit may include an inter-frame prediction unit, an intra-frame prediction unit, and a mode selection unit, the mode selection unit is used to select an intra-frame prediction mode or an inter-frame prediction mode, and if the intra-frame prediction mode is selected, the prediction process may be performed by the intra-frame prediction unit, and if the inter-frame prediction mode is selected, the prediction process may be performed by the inter-frame prediction unit.

[0046] The residual calculation unit is used to obtain a residual block by calculating the residual between the true value of an image block and the predicted block of that image block. For example, the residual calculation unit may subtract the pixel value of the predicted block from the pixel value of the image block for each pixel.

[0047] The transformation processing unit is used to obtain transformation coefficients in the transformation domain by performing transformations on the residual block, such as a discrete cosine transform (DCT) or a discrete sine transform (DST). These transformation coefficients may also be called transformation residual coefficients, and these transformation residual coefficients can represent the residual block in the transformation domain.

[0048] A quantization unit is used to quantize transformation coefficients by applying scalar or vector quantization to obtain quantized transformation coefficients, which may also be called quantized residual coefficients. The quantization process can reduce the bit depth for some or all of the transformation coefficients. For example, n-bit transformation coefficients may be truncated to m-bit transformation coefficients during quantization, where n is greater than m. The degree of quantization can be changed by adjusting the quantization parameter (QP). For example, in the case of scalar quantization, finer or coarser quantization can be achieved by applying different scales. A small quantization stride corresponds to finer quantization, and a large quantization stride corresponds to coarser quantization. The quantization parameter (QP) may specify an appropriate quantization stride.

[0049] The coding unit encodes the quantized residual coefficients, outputs the encoded image data (i.e., the encoded result of the current image block to be encoded) in the form of an encoded bitstream, and transmits the encoded bitstream to the decoder, or may store it for later transmission to the decoder or for use in retrieval. The coding unit may further be used to encode other syntax elements of the current image block, such as encoding the prediction mode into a bitstream. The coding algorithms include, but are not limited to, variable length coding (VLC) algorithms, context adaptive VLC (CAVLC) algorithms, arithmetic coding algorithms, context adaptive binary arithmetic coding (CABAC) algorithms, syntax-based context-adaptive binary arithmetic coding (SBAC) algorithms, and probability interval partitioning entropy (PIPE) algorithms.

[0050] The inverse quantization unit is used to obtain inverse quantized coefficients by performing inverse quantization on the quantized coefficients, and this inverse quantization is the reverse application of the quantization unit, and may, for example, apply an inverse quantization scheme of the quantization scheme applied by the quantization unit, based on or using the same quantization stride as the quantization unit. The inverse quantized coefficients may also be called inverse quantized residual coefficients.

[0051] The inverse transformation processing unit is used to perform an inverse transformation on the inverse quantized coefficients, and this inverse transformation is the inverse application of the transformation processing unit. For example, the inverse transformation may include an inverse discrete cosine transform (IDCT) or an inverse discrete sine transform (IDST) to obtain an inverse transformation block in the pixel region (or sample region). The inverse transformation block may also be called an inverse transformation inverse quantized block or an inverse transformation residual block.

[0052] The reconstruction unit is used to obtain a reconstructed block in the sample region by adding an inverse transform block (i.e., an inverse transform residual block) to the prediction block. The reconstruction unit may be an adder, for example, adding the sample values ​​(i.e., pixel values) of the residual block to the sample values ​​of the prediction block. The reconstructed block output by the reconstruction unit may be used later to predict other image blocks, such as in an in-frame prediction mode.

[0053] A filter unit (or abbreviated as "filter") is used to filter a reconstructed block to obtain a filtered block, either to smooth pixel transformation or to improve image quality. A filter unit may be a loop filter unit intended to represent one or more loop filters, and for example, a filter unit may be a deblocking filter, a sample-adaptive offset (SAO) filter, or other filters such as a bilateral filter, an adaptive loop filter (ALF), a sharpening or smoothing filter, or a co-filter. In one example, the filtered block output by the filtering unit may be used later to predict other image blocks, such as in inter-frame prediction mode, and is not limited thereto.

[0054] Figure 2B is a schematic block diagram of an example of a decoding side (also called a decoder) for realizing an embodiment of the present invention. The decoder is used, for example, to receive encoded image data (i.e., encoded bitstream, including, for example, the encoded bitstream of an image block and associated syntax elements) encoded by an encoder and to obtain a decoded image. The decoder includes a decoding unit, an inverse quantization unit, an inverse transformation unit, a prediction unit, a reconstruction unit, and a filter unit. In some embodiments, the decoder may perform a decoding process that is substantially the reverse of the encoding process described for the encoder in Figure 2A. In one example, the decoder may further include a buffer and a decoded image buffer, the buffer being used to buffer the reconstructed image blocks output by the reconstruction unit, and the decoded image buffer being used to buffer the filtered image blocks output by the filter unit.

[0055] The decoding unit is used to perform decoding on the encoded image data to obtain quantized coefficients and / or decoded encoded parameters (for example, the encoded parameters may include one or all of the inter-frame prediction parameters, intra-frame prediction parameters, filter parameters, and / or other syntax elements). The decoding unit is further used to transfer the decoded encoded parameters to the prediction processing unit so that the prediction processing unit can perform a prediction process based on the encoded parameters. The function of the inverse quantization unit may be the same as that of the encoder's inverse quantization unit and is used to inverse quantize the quantized coefficients decoded by the decoding unit.

[0056] The function of the inverse transform processing unit may be the same as that of the encoder's inverse transform processing unit, and the function of the reconstruction unit (e.g., adder) may be the same as that of the encoder's reconstruction unit. These units are used to perform an inverse transform (e.g., inverse DCT, inverse integer transform, or a conceptually similar inverse transform process) on the quantized coefficients to obtain an inverse transform block (also called an inverse transform residual block), which is the residual block of the current image block in the pixel region.

[0057] A prediction processing unit is used to receive or acquire encoded image data (e.g., encoded bitstream of the current image block) and reconstructed image data. The prediction processing unit may further receive or acquire, for example, prediction-related parameters and / or information about a selected prediction mode (i.e., decoded encoded parameters) from a decoding unit, and predict the current image block based on the relevant data in the reconstructed image data and the decoded encoded parameters to obtain a predicted block of the current image block.

[0058] In one example, the prediction processing unit may include an inter-frame prediction unit, an intra-frame prediction unit, and a mode selection unit, the mode selection unit being used to select either an intra-frame prediction mode or an inter-frame prediction mode. If the intra-frame prediction mode is selected, the prediction process is executed by the intra-frame prediction unit; if the inter-frame prediction mode is selected, the prediction process is executed by the inter-frame prediction unit.

[0059] The reconstruction unit is used to obtain a reconstructed block in the sample region by adding an inverse transform block (i.e., an inverse transform residual block) to the prediction block, for example, by adding the sample values ​​of the inverse transform residual block to the sample values ​​of the prediction block.

[0060] The filter unit is used to filter the reconstructed blocks to obtain filtered blocks, which are decoded image blocks.

[0061] In the encoder and decoder of the embodiment of the present invention, the processing result for a certain process may be further processed, and the further processed result may be output to the next process. For example, after processes such as interpolation filtering, motion vector derivation, or filtering, further processing such as clipping or shifting may be performed on the processing result of the corresponding process.

[0062] Based on the encoder and decoder, embodiments of the present invention provide a possible implementation of encoding / decoding, as shown in Figure 2C, which is a schematic flowchart of encoding and decoding provided by embodiments of the present invention, and the encoding and decoding implementation includes processes 1 to 5, which may be performed by the decoder and encoder described above. Process 1: Divide the image of one frame into one or more non-overlapping parallel encoding units. The one or more parallel encoding units can encode and decode completely in parallel and independently of each other, such as parallel encoding unit 1 and parallel encoding unit 2 shown in Figure 2C.

[0063] Process 2: Each parallel coding unit may be further divided into one or more independent coding units that do not overlap with each other, and each independent coding unit does not have to be dependent on one another, but may share some parallel coding unit header information. For example, the width of an independent coding unit is w_lcu and the height is h_lcu. If a parallel coding unit is divided into one independent coding unit, the size of the independent coding unit is exactly the same as the parallel coding unit; otherwise, the width of the independent coding unit must be greater than its height (except in edge regions).

[0064] Typically, an independent coding unit may be a fixed w_lcu × h_lcu, where both w_lcu and h_lcu are 2 to the power of N (N≧0). For example, the sizes of independent coding units can be 128×4, 64×4, 32×4, 16×4, 8×4, 32×2, 16×2, or 8×2.

[0065] As one possible example, an independent coding unit may be a fixed 128×4. If the size of a parallel coding unit is 256×8, the parallel coding unit may be equally divided into four independent coding units. If the size of a parallel coding unit is 288×10, the parallel coding unit may be divided such that the first and second rows each consist of two 128×4 and one 32×4 independent coding units, and the third row consists of two 128×2 and one 32×2 independent coding units. An independent coding unit may contain three components: luminance Y, chromaticity Cb, and chromaticity Cr; three components: red (R), green (G), and blue (B); or three components: luminance Y, chromaticity Co, and chromaticity Cg; or it may contain only one of these components. If an independent coding unit contains three components, the sizes of these three components may be exactly the same or different, specifically related to the image input format.

[0066] Process 3: Each independent coding unit may be further divided into one or more non-overlapping subcoding units, and each subcoding unit within an independent coding unit may be dependent on one another. For example, multiple subcoding units may cross-reference to perform pre-coding and decoding.

[0067] When the sizes of the sub-coding units and the independent coding units are the same (i.e., when the independent coding unit is divided into only one sub-coding unit), the size may be any of the sizes described in Process 2. When an independent coding unit is divided into multiple sub-coding units that do not overlap, possible divisions include horizontal equal division (where the height of the sub-coding units is the same as that of the independent coding unit, but the width is different, and may be 1 / 2, 1 / 4, 1 / 8, 1 / 16, etc.), vertical equal division (where the width of the sub-coding units is the same as that of the independent coding unit, but the height is different, and may be 1 / 2, 1 / 4, 1 / 8, 1 / 16, etc.), and horizontal and vertical equal division (quadtree division), with horizontal equal division being preferred.

[0068] The width of a sub-encoding unit is w_cu, and its height is h_cu, with the width being greater than the height (except in edge regions). Typically, a sub-encoding unit is a fixed w_cu × h_cu, where both w_cu and h_cu are 2 to the power of N (where N is greater than or equal to 0), such as 16×4, 8×4, 16×2, 8×2, 8×1, 4×1, etc. For example, the sub-encoding unit is a fixed 16×4. If the size of the independent encoding unit is 64×4, it is divided equally into four sub-encoding units; if the size of the independent encoding unit is 72×4, it is divided into four 16×4 + one 8×4 sub-encoding units. A sub-encoding unit may contain three components: luminance Y, chromaticity Cb, and chromaticity Cr (or three components: red R, green G, and blue B, or luminance Y, chromaticity Co, and chromaticity Cg), or it may contain only one of these components. If it contains three components, the sizes of these components may be exactly the same or different, specifically related to the image input format.

[0069] Process 3 may be an optional step in the encoding and decoding method, and the encoder / decoder may perform encoding and decoding on the residual coefficients (or residual values) of the independent encoding units obtained in Process 2.

[0070] Process 4: For each sub-coding unit, it may be further divided into one or more non-overlapping prediction groups (PGs), and a PG may be abbreviated as a Group. Each PG is coded and decoded according to the selected prediction mode, the predicted values ​​of the PGs are obtained to constitute the predicted values ​​of the entire sub-coding unit, and the residual values ​​of the sub-coding unit are obtained based on the predicted values ​​and original values ​​of the sub-coding unit.

[0071] Process 5: Based on the residual values ​​of the sub-encoded units, the sub-encoded units are grouped, one or more non-overlapping residual blocks (RBs) are obtained, and the residual coefficients of each RB are encoded and decoded according to the selected mode to form a residual coefficient stream. Specifically, the residual coefficients can be divided into those that undergo transformation and those that do not.

[0072] Here, the selected mode for encoding and decoding the residual coefficients in process 5 may include, but is not limited to, semi-fixed-length coding mode, exponential Golomb coding mode, Golomb-Rice coding mode, truncated unary coding mode, run-length coding mode, or direct encoding of the original residual values. For example, the encoder may directly encode the coefficients in the RB. In another example, the encoder may perform a transformation (e.g., DCT, DST, Hadamard transform) on the residual block and then encode the transformed coefficients. As one possible example, if the RB is relatively small, the encoder may directly perform a unified quantization on each coefficient in the RB and then perform binary encoding. If the RB is relatively large, it may be further divided into multiple coefficient groups (CGs), and a unified quantization may be performed on each CG before binary encoding. In some embodiments of the present invention, the coefficient group (CG) and the quantization group (QG) may be the same, and of course, the coefficient group and the quantization group may be different.

[0073] The following provides an illustrative explanation of the encoding of residual coefficients using a semi-fixed-length coding scheme. First, the maximum absolute value of residuals within a single RB block is defined as the modified maximum (MM). Next, the number of bits required to encode the residual coefficients within that RB block is determined (the number of bits required to encode residual coefficients within the same RB block is consistent). For example, if the critical limit (CL) of the current RB block is 2 and the current residual coefficient is 1, then 2 bits are needed to encode the residual coefficient of 1, which is represented as 01. If the CL of the current RB block is 7, this means encoding an 8-bit residual coefficient and a 1-bit sign bit. Determining the CL involves finding the smallest M value that satisfies the condition that all residuals in the current subblock are within the range [-2^(M-1), 2^(M-1)]. If both boundary values ​​-2^(M-1) and 2^(M-1) exist simultaneously, M is incremented by 1, meaning M+1 bits are needed to encode all residuals in the current RB block. If only one of the two boundary values ​​-2^(M-1) and 2^(M-1) exists, one trailing bit is encoded to determine whether the boundary value is -2^(M-1) or 2^(M-1). If neither -2^(M-1) nor 2^(M-1) exists in any of the residuals, the trailing bit does not need to be encoded. In some special cases, the encoder may encode the original image value directly instead of the residual value.

[0074] With the rapid development of deep learning, neural networks can adaptively construct feature descriptions based on training data, possessing greater flexibility and versatility. Therefore, deep learning has achieved success in many high-level computer vision problems, such as image classification and target detection. It is also gradually being applied to the coding and decoding fields, specifically, using neural networks to encode and decode images. For example, by using a convolutional neural network (VRCNN, Variable Filter Size Convolutional Neural Network) instead of deblocking filtering and sample-adaptive offset techniques, post-processing filtering can be performed on images after in-frame coding, significantly improving the primary and objective quality of reconstructed images. Furthermore, applying neural networks to in-frame prediction, we propose an in-frame prediction mode based on block-up and downsampling. For blocks of in-frame prediction, downsampling coding is performed first, followed by upsampling of the reconstructed pixels by the neural network, achieving a performance improvement of up to 9.0% in ultra-high-resolution sequences. Clearly, neural networks can effectively circumvent the limitations of artificially set modes, obtain data-driven neural networks that meet real-world needs, and significantly improve coding performance.

[0075] While neural network-based encoding and decoding methods demonstrate significant performance potential, they still suffer from problems such as low stability, poor generalizability, and high complexity. Firstly, neural networks are still evolving rapidly, with new network structures constantly emerging. It remains unclear which network structure is optimal for a general problem, let alone a specific problem specific to a particular module of an encoder. Therefore, using a single, fixed neural network for a particular module of an encoder carries a high risk. Secondly, the formation of a neural network heavily depends on the training data; if the training data does not contain the features of the actual problem, performance tends to degrade when processing that problem. In related technologies, one mode typically uses only one neural network, and if that neural network lacks generalizability, this mode results in reduced encoding performance. Furthermore, because video coding requires high data density and low latency, standard video coding techniques are highly complex, especially in terms of decoding complexity. To achieve good coding performance, the number of parameters in the neural network for coding is generally very large (e.g., 1M or more), and the number of multiplications and additions generated by a single application of the neural network averages more than 100K per pixel. On the other hand, simplified neural networks with fewer network layers or parameters cannot achieve optimal coding performance, but the coding and decoding complexity resulting from the use of such neural networks can be significantly reduced. Also, many image coding schemes take the entire frame image as input (resulting in very high cache overhead), making it difficult to effectively control the output bitrate. However, in practical applications, it is usually necessary to obtain images at arbitrary bitrates.

[0076] In response to the above findings, embodiments of the present invention provide a neural network-based image decoding method and an image coding method, which can encode and decode images using a neural network (e.g., a decoding neural network, an encoding neural network, etc.). In these embodiments, the optimization approach should focus not only on coding and decoding performance, but also on complexity (particularly the degree of parallelism in coefficient coding and decoding) and applicable functionality (support for bitrate variability and fine-tuning, i.e., bitrate controllability). Based on the above optimization approach, embodiments of the present invention provide several possible methods: a) The structure of the neural network is fixed, but the network parameters (i.e., weight parameters) of the neural network are not limited, and the network parameters may be indexed using an ID scheme. b) The structure of the neural network is flexible, and related structural parameters may be transmitted by syntax coding, and network parameters may be indexed using an ID scheme (some parameters may be transmitted by syntax coding, which increases a certain bitrate cost). c) The neural network structure is flexible (high-level syntax setting), with some network parameters (e.g., shallow networks) fixed (saving bitrate), and the network parameters of the remaining networks transmitted by syntax coding (preserving space for performance optimization).

[0077] The decoding method and encoding method in the embodiments of the present invention will be described in detail below in relation to several specific embodiments.

[0078] Example 1: An embodiment of the present invention provides an image decoding method based on a neural network, Figure 3 being a schematic flowchart of the method, the method may be applied to the decoding side (also called a video decoder), and the method may include steps 301 to 303.

[0079] In step 301, the control parameters and image information corresponding to the current block are decoded from the bitstream.

[0080] In step 302, neural network information corresponding to the decoding processing unit is obtained from the control parameters, and a decoding neural network corresponding to the decoding processing unit is generated based on the neural network information.

[0081] In step 303, input features corresponding to the decoding processing unit are determined based on the image information, and output features corresponding to the decoding processing unit are obtained by processing these input features based on the decoding neural network.

[0082] In one possible embodiment, if the neural network information includes basic layer information and enhancement layer information, the basic layer corresponding to the decoding unit is determined based on the basic layer information, the enhancement layer corresponding to the decoding unit is determined based on the enhancement layer information, and then the decoding neural network corresponding to the decoding unit is generated based on the basic layer and the enhancement layer.

[0083] For example, regarding the process of determining the base layer, if the base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network, then the base layer of the default network structure is obtained.

[0084] For example, in the process of determining the base layer, if the base layer information includes a base layer predesign network use flag bit and a base layer predesign network index number, and the base layer predesign network use flag bit indicates that the base layer uses a predesign network, then the base layer of the predesign network structure corresponding to the base layer predesign network index number may be selected from the predesign neural network pool.

[0085] Here, the pre-designed neural network pool may include at least one network layer of pre-designed network structures.

[0086] For example, regarding the process of determining the reinforcement layer, if the reinforcement layer information includes a reinforcement layer default network use flag bit, and the reinforcement layer default network use flag bit indicates that the reinforcement layer uses the default network, then the reinforcement layer of the default network structure is obtained.

[0087] For example, in the process of determining an augmentation layer, if the augmentation layer information includes an augmentation layer predesign network use flag bit and an augmentation layer predesign network index number, and the augmentation layer predesign network use flag bit indicates that the augmentation layer uses a predesign network, then an augmentation layer of the predesign network structure corresponding to the augmentation layer predesign network index number may be selected from the predesign neural network pool.

[0088] Here, the pre-designed neural network pool includes at least one network layer of pre-designed network structures.

[0089] For example, regarding the process of determining the reinforcement layer, if the reinforcement layer information includes network parameters for generating the reinforcement layer, the reinforcement layer corresponding to the decoding unit is generated based on the network parameters. Here, the network parameters may include, but are not limited to, at least one of the following: the number of layers in the neural network, the deconvolutional layer flag bit, the number of deconvolutional layers, the quantization stride of each deconvolutional layer, the number of channels of each deconvolutional layer, the size of the convolutional kernel, the number of filters, the filtering size index, the filtering coefficient zero flag bit, the filtering coefficient, the activation layer flag bit, and the activation layer type. Of course, the above are just some examples and are not limiting.

[0090] In one possible embodiment, the image information may include coefficient hyperparameter feature information and image feature information, and the steps of determining input features corresponding to a decoding unit based on the image information and processing the input features based on a decoding neural network to obtain output features corresponding to the decoding unit may include, when performing a decoding process of coefficient hyperparameter feature generation, determining coefficient hyperparameter feature coefficient reconstruction values ​​based on the coefficient hyperparameter feature information and performing an inverse transform operation on the coefficient hyperparameter feature coefficient reconstruction values ​​based on a decoding neural network to obtain coefficient hyperparameter feature values, wherein the coefficient hyperparameter feature values ​​are used to decode the image feature information from the bitstream; and when performing a decoding process of inverse image feature transformation, determining image feature reconstruction values ​​based on the image feature information and performing an inverse transform operation on the image feature reconstruction values ​​based on a decoding neural network to obtain low-level image feature values, wherein the low-level image feature values ​​are used to obtain a reconstructed image block corresponding to the current block.

[0091] For example, the step of determining a coefficient hyperparameter feature coefficient reconstruction value based on coefficient hyperparameter feature information may include, but is not limited to, a step of performing inverse quantization on the coefficient hyperparameter feature information to obtain a coefficient hyperparameter feature coefficient reconstruction value, if the control parameter includes first enable information and the first enable information indicates that a first inverse quantization process is enabled. The step of determining an image feature reconstruction value based on image feature information may include, but is not limited to, a step of performing inverse quantization on the image feature information to obtain an image feature reconstruction value, if the control parameter includes second enable information and the second enable information indicates that a second inverse quantization process is enabled.

[0092] For example, if the control parameter includes a third enable information, and the third enable information indicates that the quality enhancement process is enabled, then when performing the quality enhancement decoding process, low-level image features may be obtained, and based on the decoding neural network, enhancement processing may be performed on these low-level image features to obtain a reconstructed image block corresponding to the current block.

[0093] In one possible embodiment, the decoding device may include a control parameter decoding unit, a first feature decoding unit, a second feature decoding unit, a coefficient hyperparameter feature generation unit, and an image feature inverse transform unit, and the image information may include coefficient hyperparameter feature information and image feature information. Here, the control parameter decoding unit can decode control parameters from a bitstream, the first feature decoding unit can decode coefficient hyperparameter feature information from a bitstream, and the second feature decoding unit can decode image feature information from a bitstream. If the coefficient hyperparameter feature generation unit is a decoding processing unit, it can determine coefficient hyperparameter feature coefficient reconstruction values ​​based on the coefficient hyperparameter feature information, and the coefficient hyperparameter feature generation unit can perform an inverse transform on the coefficient hyperparameter feature coefficient reconstruction values ​​based on the decoding neural network to obtain coefficient hyperparameter feature values, which are used by the second feature decoding unit to decode image feature information from a bitstream. If the image feature inverse transformation unit is a decoding unit, it determines the image feature reconstruction value based on the image feature information, and the image feature inverse transformation unit can perform an inverse transformation on the image feature reconstruction value based on the decoding neural network to obtain low-level image feature values, which are used to obtain the reconstructed image block corresponding to the current block.

[0094] Exemplary, the decoding device may further include a first inverse quantization unit and a second inverse quantization unit, and the control parameter may include first enable information for the first inverse quantization unit, and if the first enable information indicates that the first inverse quantization unit should be enabled, the first inverse quantization unit may acquire coefficient hyperparameter feature information from the first feature decoding unit, perform inverse quantization on the coefficient hyperparameter feature information to obtain a coefficient hyperparameter feature coefficient reconstruction value, and provide the coefficient hyperparameter feature coefficient reconstruction value to the coefficient hyperparameter feature generation unit, and the control parameter may also include second enable information for the second inverse quantization unit, and if the second enable information indicates that the second inverse quantization unit should be enabled, the second inverse quantization unit may acquire image feature information from the second feature decoding unit, perform inverse quantization on the image feature information to obtain an image feature reconstruction value, and provide the image feature reconstruction value to the image feature inverse transform unit.

[0095] Exemplary, the decoding device may further include a quality enhancement unit, where the control parameters may include a third enable information for the quality enhancement unit, and the third enable information indicates that the quality enhancement unit is enabled. If the quality enhancement unit is a decoding processing unit, the quality enhancement unit may obtain low-level image feature values ​​from an inverse image feature transformation unit and perform enhancement processing on the low-level image feature values ​​based on the decoding neural network to obtain a reconstructed image block corresponding to the current block.

[0096] For illustrative purposes, the above execution order is merely illustrative for the sake of clarity, and in actual applications, the order of execution between steps may be changed and is not limiting. Furthermore, in other embodiments, the steps of the corresponding method may not necessarily be performed in the order shown and described herein, and the method may include more or fewer steps than those described herein. Also, a single step described herein may be broken down into multiple steps in other embodiments, and multiple steps described herein may be combined into a single step in other embodiments.

[0097] As can be seen from the above technical proposals, in the embodiments of the present invention, control parameters corresponding to the current block are decoded from the bitstream, neural network information corresponding to the decoding processing unit is obtained from the control parameters, a decoding neural network corresponding to the decoding processing unit is generated based on the neural network information, and image decoding can be realized based on the decoding neural network, thereby improving decoding performance. Image coding can be realized based on the encoding neural network corresponding to the encoding processing unit, thereby improving encoding performance. By encoding and decoding images using deep learning networks (e.g., decoding neural networks, encoding neural networks, etc.), transmitting neural network information in a bitstream, and generating a decoding neural network corresponding to the decoding processing unit based on the neural network information, problems such as low stability, low generalizability, and high complexity can be solved, i.e., high stability, high generalizability, and low complexity can be achieved. A solution that dynamically adjusts the complexity of encoding and decoding can be provided, and it has better encoding performance compared to a single deep learning network framework. Since each current block corresponds to a control parameter, the neural network information obtained from the control parameter is the neural network information for the current block. A decoding neural network is generated for each current block, meaning that the decoding neural networks for different current blocks may be the same or different. Thus, the block-level decoding neural network, i.e., the decoding neural network, is variable and adjustable.

[0098] Example 2: An embodiment of the present invention provides an image encoding method based on a neural network, Figure 4 being a schematic flowchart of the method, the method may be applied to the encoding side (also called a video encoder), and the method may include steps 401 to 403.

[0099] In step 401, input features corresponding to the coding processing unit are determined based on the current block, the input features are processed based on the coding neural network corresponding to the coding processing unit to obtain output features corresponding to the coding processing unit, and image information corresponding to the current block, such as coefficient hyperparameter feature information and image feature information, is determined based on the output features.

[0100] In step 402, control parameters corresponding to the current block are obtained, and these control parameters may include neural network information corresponding to the decoding processing unit, and this neural network information is used to determine the decoding neural network corresponding to the decoding processing unit.

[0101] In step 403, the image information and control parameters corresponding to the current block are encoded into the bitstream.

[0102] In one possible embodiment, the neural network information includes base layer information and reinforcement layer information, and the decoding neural network includes a base layer determined based on the base layer information and a reinforcement layer determined based on the reinforcement layer information.

[0103] For example, if the base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network, then the decoding neural network uses the base layer of the default network structure.

[0104] For example, if the base layer information includes a base layer predesign network use flag bit and a base layer predesign network index number, and the base layer predesign network use flag bit indicates that the base layer uses a predesign network, then the decoding neural network may use the base layer of a predesign network structure corresponding to the base layer predesign network index number, selected from a pool of predesign neural networks.

[0105] Here, the pre-designed neural network pool may include at least one network layer of pre-designed network structures.

[0106] For example, if the reinforcement layer information includes a reinforcement layer default network use flag bit, and the reinforcement layer default network use flag bit indicates that the reinforcement layer uses the default network, then the decoding neural network uses the reinforcement layer of the default network structure.

[0107] For example, if the reinforcement layer information includes a reinforcement layer predesign network use flag bit and a reinforcement layer predesign network index number, and the reinforcement layer predesign network use flag bit indicates that the reinforcement layer uses a predesign network, then the decoding neural network may use a reinforcement layer of a predesign network structure corresponding to the reinforcement layer predesign network index number, selected from a pool of predesign neural networks.

[0108] Here, the pre-designed neural network pool includes at least one network layer of pre-designed network structures.

[0109] For example, if the reinforcement layer information includes network parameters for generating the reinforcement layer, the decoding neural network may use reinforcement layers generated based on the network parameters. Here, the network parameters may include, but are not limited to, at least one of the following: the number of layers in the neural network, the deconvolutional layer flag bit, the number of deconvolutional layers, the quantization stride of each deconvolutional layer, the number of channels of each deconvolutional layer, the size of the convolutional kernel, the number of filters, the filtering size index, the filtering coefficient zero flag bit, the filtering coefficient, the activation layer flag bit, and the activation layer type. Of course, the above are just some examples of network parameters and are not limited thereto.

[0110] In one possible embodiment, the image information may include coefficient hyperparameter feature information and image feature information, and the steps of determining input features corresponding to an encoding processing unit based on the current block, processing the input features based on an encoding neural network corresponding to the encoding processing unit to obtain output features corresponding to the encoding processing unit, and determining image information corresponding to the current block based on the output features may include, but are not limited to, the steps of, performing a feature transformation on the current block based on an encoding neural network in an encoding process that performs a feature transformation to obtain image feature values ​​corresponding to the current block, wherein the image feature values ​​are used to determine the image feature information, and performing a coefficient hyperparameter feature transformation on the image feature values ​​based on an encoding neural network in an encoding process that performs a coefficient hyperparameter feature transformation to obtain coefficient hyperparameter feature coefficient values, wherein the coefficient hyperparameter feature coefficient values ​​are used to determine the coefficient hyperparameter feature information.

[0111] For example, in a process of determining coefficient hyperparameter feature information based on coefficient hyperparameter feature coefficient values, quantization may be performed on the coefficient hyperparameter feature coefficient values ​​to obtain coefficient hyperparameter feature coefficient quantized values, and coefficient hyperparameter feature information may be determined based on the coefficient hyperparameter feature coefficient quantized values. Here, the control parameters may further include first enable information, which is used to indicate that the first quantization process has been enabled.

[0112] For example, in a process of determining image feature information based on image feature values, quantization may be performed on the image feature values ​​to obtain quantized image feature values, and the image feature information may be determined based on the quantized image feature values. Here, the control parameters may further include second enable information, which is used to indicate that the second quantization process has been enabled.

[0113] For example, the step of obtaining control parameters corresponding to the current block may include, but is not limited to, the step of determining neural network information used for decoding the inverse image feature transformation process of a decoding device, based on the network structure of the encoding neural network used for encoding the feature transformation process, wherein the neural network information is used to determine the decoding neural network corresponding to the decoding process of the inverse image feature transformation process of the decoding device, and / or the step of determining neural network information used for decoding the coefficient hyperparameter feature generation process of a decoding device, based on the network structure of the encoding neural network used for encoding the coefficient hyperparameter feature transformation process, wherein the neural network information is used to determine the decoding neural network corresponding to the decoding process of the coefficient hyperparameter feature generation process of the decoding device.

[0114] In one possible embodiment, the encoding device may include a control parameter encoding unit, a first feature encoding unit, a second feature encoding unit, a feature transformation unit, and a coefficient hyperparameter feature transformation unit, and the image information may include coefficient hyperparameter feature information and image feature information. Here, the control parameter encoding unit encodes control parameters into a bitstream, the first feature encoding unit encodes coefficient hyperparameter feature information into a bitstream, the second feature encoding unit encodes image feature information into a bitstream, and if the feature transformation unit is an encoding processing unit, the feature transformation unit can perform a feature transformation on the current block based on an encoding neural network to obtain an image feature value corresponding to the current block, and the image feature value is used to determine the image feature information. If the coefficient hyperparameter feature transformation unit is an encoding processing unit, the coefficient hyperparameter feature transformation unit can perform a coefficient hyperparameter feature transformation on the image feature value based on an encoding neural network to obtain a coefficient hyperparameter feature coefficient value, and the coefficient hyperparameter feature coefficient value is used to determine the coefficient hyperparameter feature information.

[0115] Exemplary, the encoding device may further include a first quantization unit and a second quantization unit, the first quantization unit may obtain coefficient hyperparameter feature coefficient values ​​from a coefficient hyperparameter feature transformation unit, perform quantization on the coefficient hyperparameter feature coefficient values ​​to obtain coefficient hyperparameter feature coefficient quantized values, and determine coefficient hyperparameter feature information based on the coefficient hyperparameter feature coefficient quantized values. Here, the control parameter may further include first enable information for the first quantization unit, the first enable information is used to indicate that the first quantization unit has been enabled. The second quantization unit may also obtain image feature values ​​from a feature transformation unit, perform quantization on the image feature values ​​to obtain image feature quantized values, and determine image feature information based on the image feature quantized values. Here, the control parameter may further include second enable information for the second quantization unit, the second enable information is used to indicate that the second quantization unit has been enabled.

[0116] Exemplary, the step of obtaining control parameters corresponding to the current block may include, but is not limited to, the step of determining neural network information corresponding to the inverse image feature transform unit of the decoding device based on the network structure of the encoding neural network of the feature transform unit, wherein the neural network information is used to determine the decoding neural network corresponding to the inverse image feature transform unit of the decoding device, and / or the step of determining neural network information corresponding to the coefficient hyperparameter feature generation unit of the decoding device based on the network structure of the encoding neural network of the coefficient hyperparameter feature transform unit, wherein the neural network information is used to determine the decoding neural network corresponding to the coefficient hyperparameter feature generation unit of the decoding device.

[0117] In one possible embodiment, the image information may include coefficient hyperparameter feature information and image feature information. When performing a decoding process for coefficient hyperparameter feature generation, a coefficient hyperparameter feature coefficient reconstruction value may be determined based on the coefficient hyperparameter feature information, and an inverse transform operation may be performed on the coefficient hyperparameter feature coefficient reconstruction value based on the decoding neural network to obtain a coefficient hyperparameter feature value. This coefficient hyperparameter feature value may be used to decode the image feature information from the bitstream. When performing a decoding process for inverse image feature transformation, an image feature reconstruction value may be determined based on the image feature information, and an inverse transform operation may be performed on the image feature reconstruction value based on the decoding neural network to obtain an image low-level feature value. This image low-level feature value may be used to obtain a reconstructed image block corresponding to the current block.

[0118] For example, the step of determining the reconstructed coefficient values ​​of the coefficient hyperparameter features based on the coefficient hyperparameter feature information may include, but is not limited to, the step of performing inverse quantization on the coefficient hyperparameter feature information to obtain the reconstructed coefficient values ​​of the coefficient hyperparameter features. The step of determining the reconstructed image feature values ​​based on the image feature information may include, but is not limited to, the step of performing inverse quantization on the image feature information to obtain the reconstructed image feature values.

[0119] For example, when performing a high-quality decoding process, low-level image features may be obtained, and based on the decoding neural network, these low-level image features may be enhanced to obtain a reconstructed image block corresponding to the current block.

[0120] Illustratively, the step of obtaining control parameters corresponding to the current block may include, but is not limited to, at least one of the following: determining neural network information used in the decoding process of coefficient hyperparameter feature generation of a decoding device, based on the network structure of the decoding neural network used in the encoding process of coefficient hyperparameter feature generation of the encoding device, wherein the neural network information is used to determine the decoding neural network used in the decoding process of coefficient hyperparameter feature generation of the decoding device; determining neural network information used in the decoding process of inverse image feature transformation of a decoding device, based on the network structure of the decoding neural network used in the encoding process of inverse image feature transformation of the encoding device, wherein the neural network information is used to determine the decoding neural network used in the decoding process of inverse image feature transformation of the decoding device; and determining neural network information used in the decoding process of quality improvement of a decoding device, based on the network structure of the decoding neural network used in the encoding process of quality improvement of the encoding device, wherein the neural network information is used to determine the decoding neural network used in the decoding process of quality improvement of the decoding device.

[0121] In one possible embodiment, the encoding device may include a first feature decoding unit, a second feature decoding unit, a coefficient hyperparameter feature generation unit, and an image feature inverse transform unit, and the image information may include coefficient hyperparameter feature information and image feature information. Here, the first feature decoding unit can decode coefficient hyperparameter feature information from a bitstream, the second feature decoding unit can decode image feature information from a bitstream, and after determining coefficient hyperparameter feature coefficient reconstruction values ​​based on the coefficient hyperparameter feature information, the coefficient hyperparameter feature generation unit can perform an inverse transform operation on the coefficient hyperparameter feature coefficient reconstruction values ​​based on the decoding neural network to obtain coefficient hyperparameter feature values, which are used by the second feature decoding unit to decode image feature information from a bitstream, and by the second feature encoding unit to encode image feature information into a bitstream, and after determining image feature reconstruction values ​​based on the image feature information, the image feature inverse transform unit can perform an inverse transform operation on the image feature reconstruction values ​​based on the decoding neural network to obtain image low-level feature values, which are used to obtain a reconstructed image block corresponding to the current block.

[0122] Exemplary, the encoding device may further include a first inverse quantization unit and a second inverse quantization unit, the first inverse quantization unit may acquire coefficient hyperparameter feature information from a first feature decoding unit and perform inverse quantization on the coefficient hyperparameter feature information to obtain a coefficient hyperparameter feature coefficient reconstruction value, and the second inverse quantization unit may acquire image feature information from a second feature decoding unit and perform inverse quantization on the image feature information to obtain an image feature reconstruction value.

[0123] Exemplary, the encoding device may further include a quality enhancement unit, which may obtain low-level image feature values ​​from an inverse image feature transformation unit and perform enhancement processing on the low-level image feature values ​​based on a decoding neural network to obtain a reconstructed image block.

[0124] Exemplary, the step of obtaining control parameters corresponding to the current block may include, but is not limited to, at least one of the following: determining neural network information corresponding to the coefficient hyperparameter feature generation unit of the decoding device based on the network structure of the decoding neural network of the coefficient hyperparameter feature generation unit of the decoding device, wherein the neural network information is used to determine the decoding neural network corresponding to the coefficient hyperparameter feature generation unit of the decoding device; determining neural network information corresponding to the inverse image feature transformation unit of the decoding device based on the network structure of the decoding neural network of the inverse image feature transformation unit of the decoding device, wherein the neural network information is used to determine the decoding neural network corresponding to the inverse image feature transformation unit of the decoding device; and determining neural network information corresponding to the quality enhancement unit of the decoding device based on the network structure of the decoding neural network of the quality enhancement unit of the decoding device, wherein the neural network information is used to determine the decoding neural network corresponding to the quality enhancement unit of the decoding device.

[0125] In one possible embodiment, the current image may be divided into N non-overlapping image blocks, where N is a positive integer, before determining the input features corresponding to the encoding unit based on the current block, and boundary padding may be performed on each image block to obtain a boundary-padding image block, where the padding value does not depend on the reconstructed pixel values ​​of adjacent image blocks, and the N current blocks may be generated based on the boundary-padding image blocks.

[0126] In one possible embodiment, the current image may be divided into a plurality of basic blocks before determining the input features corresponding to the encoding processing unit based on the current block, each basic block containing at least one image block, and boundary padding may be performed on each image block to obtain a boundary-padding image block, wherein when boundary padding is performed on each image block, the padding value of the image block may depend on the reconstructed pixel values ​​of image blocks in different basic blocks, and not on the reconstructed pixel values ​​of other image blocks in the same basic block, and a plurality of current blocks may be generated based on the boundary-padding image blocks.

[0127] For illustrative purposes, the above execution order is merely illustrative for the sake of clarity, and in actual applications, the order of execution between steps may be changed and is not limiting. Furthermore, in other embodiments, the steps of the corresponding method may not necessarily be performed in the order shown and described herein, and the method may include more or fewer steps than those described herein. Also, a single step described herein may be broken down into multiple steps in other embodiments, and multiple steps described herein may be combined into a single step in other embodiments.

[0128] As can be seen from the above technical proposals, in the embodiments of the present invention, control parameters corresponding to the current block are decoded from the bitstream, neural network information corresponding to the decoding processing unit is obtained from the control parameters, a decoding neural network corresponding to the decoding processing unit is generated based on the neural network information, and image decoding can be realized based on the decoding neural network, thereby improving decoding performance. Image coding can be realized based on the encoding neural network corresponding to the encoding processing unit, thereby improving encoding performance. By encoding and decoding images using deep learning networks (e.g., decoding neural networks, encoding neural networks, etc.), transmitting neural network information in a bitstream, and generating a decoding neural network corresponding to the decoding processing unit based on the neural network information, problems such as low stability, low generalizability, and high complexity can be solved, i.e., high stability, high generalizability, and low complexity can be achieved. A solution that dynamically adjusts the complexity of encoding and decoding can be provided, and it has better encoding performance compared to a single deep learning network framework. Since each current block corresponds to a control parameter, the neural network information obtained from the control parameter is the neural network information for the current block. A decoding neural network is generated for each current block, meaning that the decoding neural networks for different current blocks may be the same or different. Thus, the block-level decoding neural network, i.e., the decoding neural network, is variable and adjustable.

[0129] Example 3: An embodiment of the present invention provides an image coding method and an image decoding method based on a neural network with a variable and adjustable bitrate, which can achieve high parallelism of image blocks and has a controllable and adjustable bitrate. Figure 5A is a schematic diagram of the image coding method and the image decoding method, showing the coding process of the image coding method and the decoding process of the image decoding method.

[0130] For example, regarding an image coding method based on a neural network, the coding process may include the following steps S11 to S15.

[0131] In step S11, the block division unit divides the current image (i.e., the original image) into N non-overlapping image blocks (i.e., original image blocks, which may be written as original image block 1, original image block 2, ..., original image block N), where N is a positive integer.

[0132] In step S12, boundary padding is performed on each original image block to obtain an image block after boundary padding, and N current blocks are generated based on the image blocks after boundary padding; that is, boundary padding is performed on each of the N original image blocks to obtain N image blocks after boundary padding, and these N image blocks after boundary padding are designated as the N current blocks.

[0133] As an example, as shown in Figure 5B, when performing boundary padding for each original image block, the padding value does not need to depend on the reconstructed pixel values ​​of adjacent image blocks. This ensures that each original image block can be encoded independently and in parallel, thereby improving encoding performance.

[0134] In step S13, the encoding unit determines the encoding parameters for the current block based on the information of the encoded block, and these encoding parameters are used to control the magnitude of the encoding bitrate for the current block (for example, parameters such as the quantization stride), and are not limited to these encoding parameters.

[0135] In step S14, the control unit writes control parameters required by the decoding side but which cannot be derived into the bitstream.

[0136] In step S15, the padded image block (i.e., the current block) is input to a neural network-based encoding unit, which encodes the current block based on encoding parameters and outputs the bitstream of the current block. Exemplarily, when the encoding unit encodes the current block based on encoding parameters, the encoding unit may use a neural network to encode the current block.

[0137] For example, regarding an image decoding method based on a neural network, the decoding process may include the following steps S21 to S24.

[0138] In step S21, the decoder decodes the control parameters required by the current block from the bitstream that cannot be derived otherwise.

[0139] In step S22, based on the control parameters and the bitstream of the current block, a neural network-based decoding unit obtains a reconstructed image block corresponding to the current block, that is, it decodes the current block to obtain reconstructed image blocks, for example, reconstructed image block 1 corresponding to original image block 1, reconstructed image block 2 corresponding to original image block 2, ..., and reconstructed image block N corresponding to original image block N.

[0140] For example, when a decoding unit decodes the current block, it may use a neural network to decode the current block.

[0141] In step S23, based on the control parameters, it is determined whether or not to perform filtering on a given current block. If filtering is performed, the filtering merging unit performs filtering based on the information of the current block and at least one adjacent reconstructed image block to obtain a filtered image block.

[0142] In step S24, the filtered image blocks are merged to obtain a reconstructed image.

[0143] In one possible embodiment, when performing boundary padding on an original image block, the padding value may be a padding preset value, which may be a default value agreed upon by encoding and decoding (e.g., 0, or 1<<(1-depth), where depth is the bit depth, e.g., 8, 10, 12), which may be a value transmitted to the decoding side by high-level syntax encoding, which may be obtained by performing processes such as mirroring or nearest neighbor copying based on the pixels of the current block, and the method of obtaining this padding value is not limited.

[0144] In one possible embodiment, when performing boundary padding on an original image block, the size of the padding extension to the surrounding block may be a default value agreed upon by encoding and decoding (e.g., 1, 2, 4, etc.), a value related to the current block size, or a value transmitted to the decoding side by high-level syntax coding, and is not limited to the size of this surrounding block padding extension.

[0145] Example 4: An embodiment of the present invention provides an image coding method and an image decoding method based on a neural network with a variable and adjustable bitrate, which can achieve high parallelism of image blocks, has a controllable and adjustable bitrate, and may also utilize information between adjacent blocks (e.g., reconstructed pixels of adjacent blocks). Figure 5C is a schematic diagram of the image coding method and the image decoding method, showing the coding process of the image coding method and the decoding process of the image decoding method.

[0146] For example, regarding an image encoding method based on a neural network, the encoding process may include the following steps S31 to S35.

[0147] In step S31, the block division unit divides the current image (i.e., the original image) into multiple basic blocks, each basic block containing at least one image block, with M basic blocks as an example, where M is a positive integer, and the M basic blocks contain a total of N non-overlapping image blocks (i.e., original image blocks, denoted as original image block 1, original image block 2, ..., original image block N), where N is a positive integer.

[0148] In step S32, boundary padding is performed on each original image block to obtain an image block after boundary padding, and N current blocks are generated based on the image blocks after boundary padding; that is, boundary padding is performed on each of the N original image blocks to obtain N image blocks after boundary padding, and these N image blocks after boundary padding are designated as the N current blocks.

[0149] For example, when performing boundary padding for each original image block, the padding value of that original image block is permitted to depend on the reconstructed pixel values ​​of original image blocks in different basic blocks, rather than depending on the reconstructed pixel values ​​of other original image blocks within the same basic block.

[0150] In this embodiment, the concept of a basic block is introduced, and as shown in Figure 5D, each basic block contains at least one image block (i.e., the original image block), and the image blocks within a basic block do not refer to each other, but they can refer to the reconstruction information of other image blocks located in different basic blocks. For example, as shown in Figure 5D, for image block 1, the image block to its left (i.e., the adjacent block to the left of image block 1) is located in the same basic block as image block 1, so the reconstruction information of that image block cannot be used as the padding value for image block 1, and the image block is padded using a padding preset value. For image block 1, the image block above it (i.e., the adjacent block above image block 1) is located in a different basic block from image block 1, so the reconstruction information of that image block can be used as the padding value for image block 1. When using the reconstruction information of that image block as the padding value for image block 1, the reconstruction value of that image block may be used, or the reconstruction value of that image block before filtering may be used.

[0151] Clearly, introducing basic blocks guarantees parallelism (each image block within a basic block can be encoded and decoded in parallel) and also helps improve performance (reconstruction information from adjacent image blocks within basic blocks can be utilized).

[0152] In step S33, the encoding unit determines the encoding parameters for the current block based on the information of the encoded block, and these encoding parameters are used to control the magnitude of the encoding bitrate for the current block (for example, parameters such as the quantization stride), and are not limited to these encoding parameters.

[0153] In step S34, the control unit writes control parameters required by the decoding side but which cannot be derived to the bitstream.

[0154] In step S35, the padded image block (i.e., the current block) is input to a neural network-based encoding unit, which encodes the current block based on encoding parameters and outputs the bitstream of the current block. Exemplarily, when the encoding unit encodes the current block based on encoding parameters, the encoding unit may use a neural network to encode the current block.

[0155] For example, regarding an image decoding method based on a neural network, the decoding process may include the following steps S41 to S44.

[0156] In step S41, the decoder decodes the control parameters required by the current block from the bitstream that cannot be derived otherwise.

[0157] In step S42, based on the control parameters and the bitstream of the current block, a neural network-based decoding unit obtains a reconstructed image block corresponding to the current block, that is, it decodes the current block to obtain reconstructed image blocks, for example, reconstructed image block 1 corresponding to original image block 1, reconstructed image block 2 corresponding to original image block 2, ..., and reconstructed image block N corresponding to original image block N.

[0158] In step S43, based on the control parameters, it is determined whether or not to perform filtering on a given current block. If filtering is performed, the filtering merging unit performs filtering based on the information of the current block and at least one adjacent reconstructed image block to obtain a filtered image block.

[0159] In step S44, the filtered image blocks are merged to obtain a reconstructed image.

[0160] In one possible embodiment, when performing boundary padding on an original image block, the padding value may be a padding preset value, which may be a default value agreed upon by encoding and decoding (e.g., 0, or 1<<(1-depth), where depth is the bit depth, e.g., 8, 10, 12), which may be a value transmitted to the decoding side by high-level syntax encoding, which may be obtained by performing processes such as mirroring or nearest neighbor copying based on the pixels of the current block, and the method of obtaining this padding value is not limited.

[0161] In one possible embodiment, when performing boundary padding on an original image block, the size of the padding extension to the surrounding block may be a default value agreed upon by encoding and decoding (e.g., 1, 2, 4, etc.), a value related to the current block size, or a value transmitted to the decoding side by high-level syntax coding, and is not limited to the size of this surrounding block padding extension.

[0162] In one possible embodiment, the number of image blocks included in the base block may be a default value agreed upon by encoding and decoding (e.g., 1, 4, 16, etc.), a value related to the current image size, or a value transmitted to the decoding side by high-level syntax coding, and the number of image blocks in this base block is not limited and may be selected according to the actual needs.

[0163] In one possible embodiment, in Examples 3 and 4, the block division process of the original image (steps S11 and S31) may be performed by applying an image domain transformation to the original image, as shown in Figure 5E, and then performing block division on the image after the image domain transformation. Accordingly, the filtered image blocks may be merged into a single image, an inverse image domain transformation may be performed on this image, and a reconstructed image may be obtained based on the image after the inverse image domain transformation. Exemplarily, applying an image domain transformation to the original image may be a transformation process from an RGB domain image to a YUV domain image (the corresponding inverse image domain transformation is the inverse transformation process from a YUV domain image to an RGB domain image), or it may be a process such as a wavelet transform or Fourier transform introduced to generate an image of a new domain (the corresponding inverse image domain transformation is the inverse wavelet transform or inverse Fourier transform process). The image domain transformation process can be implemented using a neural network or a non-neural network, and is not limited to this image domain transformation process.

[0164] Example 5: An embodiment of the present invention provides an image decoding method based on a neural network, which may be applied to a decoding side (also called a video decoder), Figure 6A being a schematic diagram of the decoding side, which may include a control parameter decoding unit, a first feature decoding unit, a second feature decoding unit, a coefficient hyperparameter feature generation unit, an image feature inverse transform unit, a first inverse quantization unit, a second inverse quantization unit, and a quality enhancement unit. The first inverse quantization unit, the second inverse quantization unit, and the quality enhancement unit are optional units, and in certain scenarios, the processes of these optional units may be turned off or skipped.

[0165] In this embodiment, for each current block (i.e., an image block), the bitstream corresponding to the current block includes three parts: bitstream 0 (a bitstream containing control parameters), bitstream 1 (a bitstream containing coefficient hyperparameter feature information), and bitstream 2 (a bitstream containing image feature information). The coefficient hyperparameter feature information and image feature information may be collectively referred to as image information.

[0166] For example, the neural network-based image decoding method in this embodiment may include the following steps S51 to S58.

[0167] In step S51, bitstream 0 corresponding to the current block is decoded to obtain the control parameters corresponding to the current block. For example, the control parameter decoding unit may decode bitstream 0 corresponding to the current block to obtain the control parameters corresponding to the current block, that is, the control parameter decoding unit may decode the control parameters from bitstream 0, and these control parameters may include the control parameters of the first inverse quantization unit, the control parameters of the second inverse quantization unit, the control parameters of the coefficient hyperparameter feature generation unit, the control parameters of the image feature inverse transformation unit, and the control parameters of the quality enhancement unit, and the contents of the control parameters may be described in subsequent embodiments, and will not be described here.

[0168] In step S52, the bitstream 1 corresponding to the current block is decoded to obtain coefficient hyperparameter feature information corresponding to the current block. For example, the first feature decoding unit may decode the bitstream 1 corresponding to the current block to obtain coefficient hyperparameter feature information corresponding to the current block; that is, the first feature decoding unit may decode coefficient hyperparameter feature information from bitstream 1.

[0169] In step S53, the coefficient hyperparameter feature coefficient reconstruction value is determined based on the coefficient hyperparameter feature information.

[0170] For example, if the control parameter includes first enable information corresponding to a first inverse quantization unit, the first enable information may indicate that the first inverse quantization unit is enabled (i.e., that the first inverse quantization unit is enabled to perform the first inverse quantization process), or it may indicate that the first inverse quantization unit is not enabled. For example, if the first enable information is a first value, it indicates that the first inverse quantization unit is enabled, and if the first enable information is a second value, it indicates that the first inverse quantization unit is not enabled.

[0171] For example, if the first enable information indicates that the first inverse quantization unit should be enabled, the coefficient hyperparameter feature information may be the coefficient hyperparameter feature coefficient quantization value C_q, and the first inverse quantization unit may perform inverse quantization on the coefficient hyperparameter feature coefficient quantization value C_q to obtain the coefficient hyperparameter feature coefficient reconstruction value C'. If the first enable information indicates that the first inverse quantization unit should not be enabled, the coefficient hyperparameter feature information may be the coefficient hyperparameter feature coefficient reconstruction value C', that is, the coefficient hyperparameter feature coefficient reconstruction value C' is directly decoded from bitstream 1.

[0172] In step S54, an inverse transform is performed on the coefficient hyperparameter feature coefficient reconstruction value C' to obtain the coefficient hyperparameter feature value P. For example, the coefficient hyperparameter feature generation unit obtains the coefficient hyperparameter feature value P by performing an inverse transform on the coefficient hyperparameter feature coefficient reconstruction value C', and for example, the coefficient hyperparameter feature generation unit obtains the coefficient hyperparameter feature value P by performing an inverse transform on the coefficient hyperparameter feature coefficient reconstruction value C' based on the decoding neural network.

[0173] In one possible embodiment, the coefficient hyperparameter feature generation unit may obtain neural network information 1 corresponding to the coefficient hyperparameter feature generation unit from control parameters and generate a decoded neural network 1 corresponding to the coefficient hyperparameter feature generation unit based on the neural network information 1.

[0174] Furthermore, the coefficient hyperparameter feature generation unit may determine an input feature corresponding to the coefficient hyperparameter feature generation unit (e.g., a coefficient hyperparameter feature coefficient reconstruction value C'), process the input feature based on the decoding neural network 1 (e.g., perform an inverse transform on the coefficient hyperparameter feature coefficient reconstruction value C'), and obtain an output feature corresponding to the coefficient hyperparameter feature generation unit (e.g., a coefficient hyperparameter feature value P).

[0175] For example, the neural network information 1 may include basic layer information and reinforcement layer information, and the coefficient hyperparameter feature generation unit may determine the basic layer corresponding to the coefficient hyperparameter feature generation unit based on the basic layer information and determine the reinforcement layer corresponding to the coefficient hyperparameter feature generation unit based on the reinforcement layer information. The coefficient hyperparameter feature generation unit may also generate a decoded neural network 1 corresponding to the coefficient hyperparameter feature generation unit based on the basic layer and reinforcement layer, for example, by combining the basic layer and the reinforcement layer to obtain the decoded neural network 1.

[0176] After obtaining the decoded neural network 1, the coefficient hyperparameter feature generation unit may perform an inverse transformation process on the coefficient hyperparameter feature reconstruction value C' via the decoded neural network 1 to obtain the coefficient hyperparameter feature value P, and this inverse transformation process is not limited.

[0177] In step S55, the bitstream 2 corresponding to the current block is decoded to obtain image feature information corresponding to the current block. For example, the second feature decoding unit may decode the bitstream 2 corresponding to the current block to obtain image feature information corresponding to the current block; that is, the second feature decoding unit may decode image feature information from bitstream 2. When decoding the bitstream 2 corresponding to the current block, the second feature decoding unit may decode the bitstream 2 corresponding to the current block using the coefficient hyperparameter feature value P, and is not limited to this decoding process.

[0178] In step S56, the image feature reconstruction value is determined based on the image feature information.

[0179] For example, if the control parameter includes second enable information corresponding to a second inverse quantization unit, the second enable information may indicate that the second inverse quantization unit is enabled (i.e., that the second inverse quantization unit is enabled to perform the second inverse quantization process), or it may indicate that the second inverse quantization unit is not enabled. For example, if the second enable information is a first value, it indicates that the second inverse quantization unit is enabled, and if the second enable information is a second value, it indicates that the second inverse quantization unit is not enabled.

[0180] For example, if the second enable information indicates that the second inverse quantization unit should be enabled, the image feature information may be the image feature quantization value F_q, and the second inverse quantization unit may obtain the image feature quantization value F_q and perform inverse quantization on the image feature quantization value F_q to obtain the image feature reconstruction value F'. If the second enable information indicates that the second inverse quantization unit should not be enabled, the image feature information may be the image feature reconstruction value F', that is, the image feature reconstruction value F' is directly decoded from bitstream 2.

[0181] In step S57, an inverse transformation is performed on the image feature reconstruction value F' to obtain the image low-level feature value LF. For example, the image feature inverse transformation unit performs an inverse transformation on the image feature reconstruction value F' to obtain the image low-level feature value LF, and for example, based on a decoding neural network, an inverse transformation is performed on the image feature reconstruction value F' to obtain the image low-level feature value LF.

[0182] In one possible embodiment, the image feature inverse transform unit may obtain neural network information 2 corresponding to the image feature inverse transform unit from control parameters and generate a decoding neural network 2 corresponding to the image feature inverse transform unit based on the neural network information 2. The image feature inverse transform unit may determine an input feature (e.g., an image feature reconstruction value F') corresponding to the image feature inverse transform unit, process the input feature based on the decoding neural network 2 (e.g., perform an inverse transform operation on the image feature reconstruction value F'), and obtain an output feature (e.g., an image low-level feature value LF) corresponding to the image feature inverse transform unit.

[0183] For example, the neural network information 2 may include basic layer information and reinforcement layer information, and the image feature inverse transform unit may determine the basic layer corresponding to the image feature inverse transform unit based on the basic layer information and determine the reinforcement layer corresponding to the image feature inverse transform unit based on the reinforcement layer information. The image feature inverse transform unit may generate the decoding neural network 2 corresponding to the image feature inverse transform unit based on the basic layer and the reinforcement layer, for example, by combining the basic layer and the reinforcement layer to obtain the decoding neural network 2.

[0184] After obtaining the decoding neural network 2, the image feature inverse transformation unit may perform an inverse transformation on the image feature reconstruction value F' via the decoding neural network 2 to obtain the image low-level feature value LF, and this inverse transformation process is not limited.

[0185] In step S58, the reconstructed image block I corresponding to the current block is determined based on the image low-level feature value LF.

[0186] For example, if the control parameter includes third enable information corresponding to a quality improvement unit, the third enable information may indicate that the quality improvement unit is enabled (i.e., that the quality improvement unit is enabled to perform quality improvement processing), or it may indicate that the quality improvement unit is not enabled. For example, if the third enable information is a first value, it may indicate that the quality improvement unit is enabled, and if the third enable information is a second value, it may indicate that the quality improvement unit is not enabled.

[0187] For example, if the third enable information indicates that the quality enhancement unit should be enabled, the quality enhancement unit obtains the image low-level feature value LF, performs enhancement processing on the image low-level feature value LF, and obtains the reconstructed image block I corresponding to the current block. If the third enable information indicates that the quality enhancement unit should not be enabled, the image low-level feature value LF is used as the reconstructed image block I corresponding to the current block.

[0188] For example, when the quality enhancement unit performs enhancement on low-level image feature values ​​LF, the quality enhancement unit may perform enhancement on the low-level image feature values ​​LF based on the decoding neural network to obtain a reconstructed image block I corresponding to the current block.

[0189] In one possible embodiment, the quality enhancement unit may obtain neural network information 3 corresponding to the quality enhancement unit from control parameters and generate a decoding neural network 3 corresponding to the quality enhancement unit based on the neural network information 3.

[0190] The quality enhancement unit may determine an input feature corresponding to the quality enhancement unit (e.g., an image low-level feature value LF), process the input feature based on the decoding neural network 3 (e.g., perform enhancement processing on the image low-level feature value LF), and obtain an output feature corresponding to the quality enhancement unit (e.g., a reconstructed image block I corresponding to the current block).

[0191] For example, the neural network information 3 may include basic layer information and reinforcement layer information, and the quality enhancement unit may determine the basic layer corresponding to the quality enhancement unit based on the basic layer information and determine the reinforcement layer corresponding to the quality enhancement unit based on the reinforcement layer information. The quality enhancement unit may generate the decoding neural network 3 corresponding to the quality enhancement unit based on the basic layer and the reinforcement layer, for example, by combining the basic layer and the reinforcement layer to obtain the decoding neural network 3.

[0192] After obtaining the decoding neural network 3, the quality enhancement unit may perform an enhancement process on the low-level image feature values ​​LF via the decoding neural network 3 to obtain a reconstructed image block I corresponding to the current block, and this enhancement process is not limited to this method.

[0193] Example 6: An embodiment of the present invention provides an image decoding method based on a neural network, which may be applied to the decoding side. Figure 6B is a schematic diagram of the decoding side, which may include a control parameter decoding unit, a first feature decoding unit, a second feature decoding unit, a coefficient hyperparameter feature generation unit, and an image feature inverse transform unit. In this embodiment, for each current block (i.e., an image block), the bitstream corresponding to the current block includes three parts: bitstream 0 (a bitstream containing control parameters), bitstream 1 (a bitstream containing coefficient hyperparameter feature information), and bitstream 2 (a bitstream containing image feature information).

[0194] For example, the neural network-based image decoding method in this embodiment may include the following steps S61 to S66.

[0195] In step S61, the control parameter decoding unit decodes bitstream 0 corresponding to the current block and obtains the control parameters corresponding to the current block.

[0196] In step S62, the first feature decoding unit decodes the bitstream 1 corresponding to the current block to obtain coefficient hyperparameter feature information corresponding to the current block, and this coefficient hyperparameter feature information may be the coefficient hyperparameter feature coefficient reconstruction value C'.

[0197] In step S63, the coefficient hyperparameter feature generation unit performs an inverse transform operation on the coefficient hyperparameter feature coefficient reconstruction value C' to obtain the coefficient hyperparameter feature value P, for example, by performing an inverse transform operation on the coefficient hyperparameter feature coefficient reconstruction value C' based on the decoding neural network to obtain the coefficient hyperparameter feature value P.

[0198] In step S64, the second feature decoding unit decodes the bitstream 2 corresponding to the current block to obtain image feature information corresponding to the current block, which may be an image feature reconstruction value F'. Exemplarily, the second feature decoding unit may obtain a coefficient hyperparameter feature value P, and use the coefficient hyperparameter feature value P to decode the bitstream 2 corresponding to the current block to obtain an image feature reconstruction value F'.

[0199] In step S65, the image feature inverse transform unit performs an inverse transform operation on the image feature reconstruction value F' to obtain the image low-level feature value LF, for example, by performing an inverse transform operation on the image feature reconstruction value F' based on a decoding neural network to obtain the image low-level feature value LF.

[0200] In step S66, the reconstructed image block I corresponding to the current block is determined based on the image low-level feature value LF.

[0201] For example, the image low-level feature value LF may be directly used as the reconstructed image block I. Alternatively, if the decoding device includes a further quality enhancement unit, the quality enhancement unit may perform enhancement processing on the image low-level feature value LF to obtain the reconstructed image block I corresponding to the current block, for example, based on a decoding neural network, to perform enhancement processing on the image low-level feature value LF to obtain the reconstructed image block I.

[0202] Example 7: In Examples 5 and 6, the first feature decoding unit may decode bitstream 1 corresponding to the current block to obtain coefficient hyperparameter feature information corresponding to the current block. For example, the first feature decoding unit includes at least one coefficient decoding module, and in one possible embodiment, the coefficient decoding module may perform coefficient decoding using an entropy decoding method, that is, decode bitstream 1 corresponding to the current block using an entropy decoding method to obtain coefficient hyperparameter feature information corresponding to the current block.

[0203] For example, the entropy decoding method may include, but is not limited to, entropy decoding methods such as CAVLC (Context-Adaptive Varialbe Length Coding) or CABAC (Context-based Adaptive Binary Arithmetic Coding).

[0204] For example, when performing coefficient decoding using an entropy decoding method, a pre-defined probability model may be used as the probabilistic model for entropy decoding, and this pre-defined probability model may be set according to the actual needs; the system is not limited to this pre-defined probability model. For example, based on a pre-defined probability model, the coefficient decoding module may perform coefficient decoding using an entropy decoding method.

[0205] Example 8: In the case of Example 5, the first inverse quantization unit may perform inverse quantization on the coefficient hyperparameter feature coefficient quantization value C_q (i.e., coefficient hyperparameter feature information) to obtain the coefficient hyperparameter feature coefficient reconstruction value C'. For example, the first inverse quantization unit may not exist, or if the first inverse quantization unit exists, the first inverse quantization unit may be selectively skipped based on control parameters (e.g., high-level syntax, e.g., first enable information), or it may be decided to enable the first inverse quantization unit based on control parameters.

[0206] For example, if the first inverse quantization unit does not exist, the reconstructed coefficient hyperparameter feature coefficient value C' is the same as the quantized coefficient hyperparameter feature coefficient value C_q, i.e., there is no need to perform inverse quantization on the quantized coefficient hyperparameter feature coefficient value C_q. If the first inverse quantization unit is selectively skipped based on the control parameter, the reconstructed coefficient hyperparameter feature coefficient value C' is the same as the quantized coefficient hyperparameter feature coefficient value C_q, i.e., there is no need to perform inverse quantization on the quantized coefficient hyperparameter feature coefficient value C_q. If it is decided to enable the first inverse quantization unit based on the control parameter, but the step parameter qstep corresponding to the quantized coefficient hyperparameter feature coefficient value C_q is 1, the reconstructed coefficient hyperparameter feature coefficient value C' is the same as the quantized coefficient hyperparameter feature coefficient value C_q, i.e., there is no need to perform inverse quantization on the quantized coefficient hyperparameter feature coefficient value C_q.

[0207] Exemplarily, it is determined to enable the first inverse quantization unit based on a control parameter, and when the step parameter qstep corresponding to the coefficient hyperparameter feature coefficient quantization value C_q is not 1, the first inverse quantization unit may perform inverse quantization on the coefficient hyperparameter feature coefficient quantization value C_q based on the control parameter (e.g., a quantization-related parameter) to obtain a coefficient hyperparameter feature coefficient reconstruction value C'. For example, the first inverse quantization unit performs the following processing. It obtains a quantization-related parameter corresponding to the coefficient hyperparameter feature coefficient quantization value C_q from the control parameter (the control parameter may be included in the bit stream and the control parameter may include the quantization-related parameter), for example, the step parameter qstep or the quantization parameter qp. Based on the step parameter qstep or the quantization parameter qp, it determines a multiplication factor mult and a shift factor shift corresponding to the coefficient hyperparameter feature coefficient quantization value C_q. Assuming that the coefficient hyperparameter feature coefficient quantization value C_q is Coff_hyper and the coefficient hyperparameter feature coefficient reconstruction value C' is Coff_hyper_rec, then Coff_hyper_rec = (Coff_hyper * mult) << shift. As described above, when performing inverse quantization on the coefficient hyperparameter feature coefficient quantization value C_q, the coefficient hyperparameter feature coefficient reconstruction value C' may be obtained using the above formula.

[0208] Furthermore, with respect to the quantization-related parameters (e.g., step parameter qstep) corresponding to the coefficient hyperparameter feature coefficient quantization value C_q, it is also possible to include: 1) using the same step parameter qstep for each coefficient hyperparameter feature coefficient quantization value in each feature channel; 2) using different step parameters qstep for each coefficient hyperparameter feature coefficient quantization value in each feature channel, but using the same step parameter qstep for each coefficient hyperparameter feature coefficient quantization value within the feature channel; or 3) using different step parameters qstep for each coefficient hyperparameter feature coefficient quantization value in each feature channel. In the above process, the step parameter qstep may also be called the quantization stride.

[0209] Example 9: In Examples 5 and 6, the coefficient hyperparameter feature generation unit may obtain the coefficient hyperparameter feature value P by performing an inverse transform operation on the coefficient hyperparameter feature coefficient reconstruction value C' based on the decoding neural network. In one possible embodiment, as shown in Figure 6C, the coefficient hyperparameter feature generation unit may include a decoding neural network 1, the decoding neural network 1 may include a base layer and an enhancement layer, the coefficient hyperparameter feature coefficient reconstruction value C' is an input feature of the decoding neural network 1, the coefficient hyperparameter feature value P is an output feature of the decoding neural network 1, and the decoding neural network 1 is used to perform an inverse transform operation on the coefficient hyperparameter feature coefficient reconstruction value C'.

[0210] In this embodiment, the decoding neural network 1 is divided into a basic layer and an enhancement layer. The basic layer may contain at least one network layer, or it may not contain any network layers; that is, the basic layer may be empty. The enhancement layer may contain at least one network layer, or it may not contain any network layers; that is, the enhancement layer may be empty. Furthermore, for multiple network layers in the decoding neural network 1, multiple network layers may be divided into basic layers and enhancement layers according to actual needs. For example, the first M1 network layers may be used as the basic layer and the remaining network layers as the enhancement layer, or the first M2 network layers may be used as the enhancement layer and the remaining network layers as the basic layer, or the next M3 network layers may be used as the basic layer and the remaining network layers as the enhancement layer, or the next M4 network layers may be used as the enhancement layer and the remaining network layers as the basic layer, or the odd-numbered network layers may be used as the basic layer and the remaining network layers as the enhancement layer, or the even-numbered network layers may be used as the basic layer and the remaining network layers as the enhancement layer. Of course, the above are just a few examples and do not limit the scope to this particular partitioning method.

[0211] For example, a network layer with a fixed network structure may be designated as the base layer, and a network layer with an unfixed network structure may be designated as the reinforcement layer. For example, in a decoding neural network 1, if a certain network layer uses the same network structure when decoding multiple image blocks, that network layer is designated as a network layer with a fixed network structure, and is designated as the base layer. Alternatively, if a certain network layer in a decoding neural network 1 uses different network structures when decoding multiple image blocks, that network layer is designated as a network layer with an unfixed network structure, and is designated as the reinforcement layer.

[0212] For example, in decoding neural network 1, the size of the output features may be larger than the size of the input features, or the size of the output features may be equal to the size of the input features, or the size of the output features may be smaller than the size of the input features.

[0213] For decoding neural network 1, the base layer and reinforcement layer each include at least one deconvolutional layer. For example, the base layer may include at least one deconvolutional layer, and the reinforcement layer may include at least one deconvolutional layer, or it may not include any deconvolutional layers. Alternatively, the reinforcement layer may include at least one deconvolutional layer, and the base layer may include at least one deconvolutional layer, or it may not include any deconvolutional layers.

[0214] For example, decoding neural network 1 may include, but is not limited to, deconvolutional layers and activation layers. For instance, decoding neural network 1 includes, in order, a deconvolutional layer with a stride of 2, one activation layer, a deconvolutional layer with a stride of 1, and one activation layer. All of the above network layers may be considered basic layers, that is, the basic layers include a deconvolutional layer with a stride of 2, one activation layer, a deconvolutional layer with a stride of 1, and one activation layer. In this case, the reinforcement layer is empty, and of course, some network layers may be reinforcement layers, and is not limited to this. Also, for example, decoding neural network 1 includes, in order, a deconvolutional layer with a stride of 2, and a deconvolutional layer with a stride of 1. All of the above network layers may be used as base layers, that is, the base layers include a deconvolutional layer with a stride of 2 and a deconvolutional layer with a stride of 1. In this case, the reinforcement layer is empty, and of course, some network layers may be used as reinforcement layers, and this is not limited to that. Also, for example, decoding neural network 1 includes, in order, a deconvolutional layer with a stride of 2, one activation layer, a deconvolutional layer with a stride of 2, one activation layer, a deconvolutional layer with a stride of 1, and one activation layer. All of the above network layers may be used as base layers, that is, the base layers include a deconvolutional layer with a stride of 2, one activation layer, a deconvolutional layer with a stride of 2, one activation layer, a deconvolutional layer with a stride of 1, and one activation layer. In this case, the reinforcement layer is empty, and of course, some network layers may be used as reinforcement layers, and this is not limited to that. Of course, the above are just some examples and are not limited to that.

[0215] In one possible embodiment, a network layer of a default network structure (which may consist of at least one network layer) may be set for the coefficient hyperparameter feature generation unit, and all network parameters relating to the network layer of the default network structure are fixed. For example, the network parameters may include, but are not limited to, the number of layers in the neural network, the deconvolutional layer flag bit, the number of deconvolutional layers, the quantization stride of each deconvolutional layer, the number of channels of each deconvolutional layer, the size of the convolutional kernel, the number of filters, the filtering size index, the filtering coefficient zero flag bit, the filtering coefficient, the number of activation layers, the activation layer flag bit, and the activation layer type; i.e., all of the above network parameters are fixed. For example, in the network layer of the default network structure, the number of deconvolutional layers is fixed, the number of activation layers is fixed, the number of channels in each deconvolutional layer is fixed, the size of the convolutional kernel is fixed, and the filtering coefficients are fixed. For example, the number of channels in the deconvolutional layer may be 4, 8, 16, 32, 64, 128, or 256, and the size of the convolutional kernel may be 1*1, 3*3, or 5*5. Clearly, all network parameters in the network layer of the default network structure are fixed, and since these network parameters are known, the network layer of the default network structure can be obtained directly.

[0216] In one possible embodiment, a pre-designed neural network pool may be set up for the coefficient hyperparameter feature generation unit, the pre-designed neural network pool may include at least one network layer of a pre-designed network structure (the network layer of the pre-designed network structure may consist of at least one network layer), and the network parameters relating to the network layer of the pre-designed network structure may all be set according to the actual needs, for example, the network parameters may include, but are not limited to, at least one of the following: the number of layers in the neural network, the deconvolutional layer flag bit, the number of deconvolutional layers, the quantization stride of each deconvolutional layer, the number of channels of each deconvolutional layer, the size of the convolutional kernel, the number of filters, the filtering size index, the filtering coefficient zero flag bit, the filtering coefficient, the number of activation layers, the activation layer flag bit, and the activation layer type, i.e., any of the above network parameters may all be set according to the actual needs.

[0217] For example, a pre-designed neural network pool may include the network layers of pre-designed network structure s1, pre-designed network structure s2, and pre-designed network structure s3. Here, for the network layer of pre-designed network structure s1, network parameters such as the number of deconvolutional layers, the number of activation layers, the number of channels in each deconvolutional layer, the size of the convolution kernel, and the filtering coefficients may be set in advance. After all network parameters have been set, the network layer of pre-designed network structure s1 can be obtained. Similarly, the network layers of pre-designed network structure s2 and pre-designed network structure s3 can be obtained, and this is not repeated here.

[0218] In one possible embodiment, based on network parameters, a network layer of a variable network structure (which may consist of at least one network layer) may be dynamically generated for a coefficient hyperparameter feature generation unit, where the network parameters for the network layer of the variable network structure are not pre-set but are dynamically generated by the encoding side, for example, the encoding side may send network parameters corresponding to the coefficient hyperparameter feature generation unit to the decoding side, where these network parameters may include, but are not limited to, at least one of the following: the number of layers in the neural network, the deconvolutional layer flag bit, the number of deconvolutional layers, the quantization stride of each deconvolutional layer, the number of channels of each deconvolutional layer, the size of the convolutional kernel, the number of filters, the filtering size index, the filtering coefficient zero flag bit, the filtering coefficient, the number of activation layers, the activation layer flag bit, and the activation layer type. The decoding side dynamically generates the network layer of the variable network structure based on the above network parameters. For example, the encoding side may encode network parameters such as the number of deconvolutional layers, the number of activation layers, the number of channels in each deconvolutional layer, the size of the convolution kernel, and filtering coefficients into the bitstream, and the decoding side may analyze the above network parameters from the bitstream and generate a network layer with a variable network structure based on these network parameters, and this generation process is not limited.

[0219] In one possible embodiment, the decoding neural network 1 may be divided into a basic layer and an enhancement layer, and the combination method of the basic layer and the enhancement layer may include, but is not limited to, the following methods: Method 1: The basic layer uses a network layer with a default network structure, and the enhancement layer uses a network layer with a default network structure, and the basic layer and the enhancement layer are combined to form the decoding neural network 1. Method 2: The basic layer uses a network layer with a default network structure, and the enhancement layer uses a network layer with a pre-designed network structure, and the basic layer and the enhancement layer are combined to form the decoding neural network 1. Method 3: The basic layer uses a network layer with a default network structure, and the enhancement layer uses a network layer with a variable network structure, and the basic layer and the enhancement layer are combined to form the decoding neural network 1. Method 4: The basic layer uses a network layer with a pre-designed network structure, and the enhancement layer uses a network layer with a default network structure, and the basic layer and the enhancement layer are combined to form the decoding neural network 1. Method 5: The basic layer uses a network layer with a pre-designed network structure, and the reinforcement layer uses a network layer with a pre-designed network structure. The basic layer and the reinforcement layer are combined to form the decoding neural network 1. Method 6: The basic layer uses a network layer with a pre-designed network structure, and the reinforcement layer uses a network layer with a variable network structure. The basic layer and the reinforcement layer are combined to form the decoding neural network 1.

[0220] In one possible embodiment, the control parameters may include neural network information 1 corresponding to a coefficient hyperparameter feature generation unit, and the coefficient hyperparameter feature generation unit may analyze the neural network information 1 from the control parameters and generate a decoded neural network 1 based on the neural network information 1. For example, the neural network information 1 may include basic layer information and reinforcement layer information, and the basic layer may be determined based on the basic layer information, the reinforcement layer may be determined based on the reinforcement layer information, and the decoded neural network 1 may be obtained by combining the basic layer and the reinforcement layer. For example, the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit may be obtained using the following case.

[0221] Case 1: The base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network. In this case, the coefficient hyperparameter feature generation unit knows, based on the base layer information, that the base layer uses the network layer of the default network structure, and therefore obtains the base layer of the default network structure (i.e., the network layer of the default network structure). The reinforcement layer information includes a reinforcement layer default network use flag bit, and the reinforcement layer default network use flag bit indicates that the reinforcement layer uses the default network. In this case, the coefficient hyperparameter feature generation unit knows, based on the reinforcement layer information, that the reinforcement layer uses the network layer of the default network structure, and therefore obtains the reinforcement layer of the default network structure (i.e., the network layer of the default network structure). Based on this, the base layer of the default network structure and the reinforcement layer of the default network structure may be combined to obtain the decoded neural network 1.

[0222] Case 2: The base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network. In this case, the coefficient hyperparameter feature generation unit knows, based on the base layer information, that the base layer uses the network layer of the default network structure, and therefore obtains the base layer of the default network structure (i.e., the network layer of the default network structure). The enhancement layer information includes an enhancement layer predesign network use flag bit and an enhancement layer predesign network index number. The enhancement layer predesign network use flag bit indicates that the enhancement layer uses a predesign network. In this case, the coefficient hyperparameter feature generation unit knows, based on the enhancement layer information, that the enhancement layer uses the network layer of the predesign network structure, and therefore selects the enhancement layer of the predesign network structure corresponding to the enhancement layer predesign network index number from the predesign neural network pool (for example, if the enhancement layer predesign network index number is 0, the network layer of predesign network structure s1 is used as the enhancement layer; if the enhancement layer predesign network index number is 1, the network layer of predesign network structure s2 is used as the enhancement layer). Based on this, a decoding neural network 1 may be obtained by combining the basic layer of the default network structure and the reinforcement layer of the pre-designed network structure.

[0223] Case 3: The base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network. In this case, the coefficient hyperparameter feature generation unit knows, based on the base layer information, that the base layer uses the network layer of the default network structure, and therefore obtains the base layer of the default network structure (i.e., the network layer of the default network structure). The enhancement layer information includes network parameters for generating the enhancement layer. In this case, the coefficient hyperparameter feature generation unit may analyze the network parameters from the control parameters and generate the enhancement layer of the variable network structure (i.e., the network layer of the variable network structure) based on the network parameters. Here, the network parameters may include, but are not limited to, at least one of the following: the number of layers in the neural network, the deconvolutional layer flag bit, the number of deconvolutional layers, the quantization stride of each deconvolutional layer, the number of channels in each deconvolutional layer, the size of the convolutional kernel, the number of filters, the filtering size index, the filtering coefficient zero flag bit, the filtering coefficient, the number of activation layers, the activation layer flag bit, and the activation layer type. The coefficient hyperparameter feature generation unit may generate enhancement layers for a variable network structure based on the above network parameters. For example, the coefficient hyperparameter feature generation unit analyzes network parameters such as the number of deconvolutional layers, the quantization stride of each deconvolutional layer, the number of channels in each deconvolutional layer, the size of the convolutional kernel, and the activation layer type from the control parameters, and generates enhancement layers for a variable network structure based on these network parameters. Based on this, the coefficient hyperparameter feature generation unit may combine the base layer of the default network structure and the enhancement layers of the variable network structure to obtain the decoded neural network 1.

[0224] Case 4: The base layer information includes a base layer predesign network use flag bit and a base layer predesign network index number, and the base layer predesign network use flag bit indicates that the base layer uses a predesign network. In this case, the coefficient hyperparameter feature generation unit knows, based on the base layer information, that the base layer uses a network layer of a predesign network structure, and therefore selects the base layer of the predesign network structure corresponding to the base layer predesign network index number from the predesign neural network pool (for example, if the base layer predesign network index number is 0, the network layer of predesign network structure s1 is used as the base layer; if the base layer predesign network index number is 1, the network layer of predesign network structure s2 is used as the base layer). The enhancement layer information includes an enhancement layer default network use flag bit, and the enhancement layer default network use flag bit indicates that the enhancement layer uses a default network. In this case, the coefficient hyperparameter feature generation unit knows, based on the enhancement layer information, that the enhancement layer uses a network layer of a default network structure, and therefore obtains the enhancement layer of the default network structure (i.e., the network layer of the default network structure). Based on this, a decoding neural network 1 may be obtained by combining the basic layer of the pre-designed network structure with the reinforcement layer of the default network structure.

[0225] Case 5: The base layer information includes a base layer predesign network use flag bit and a base layer predesign network index number, and the base layer predesign network use flag bit indicates that the base layer uses a predesign network. In this case, the coefficient hyperparameter feature generation unit knows, based on the base layer information, that the base layer uses a network layer of a predesign network structure, and therefore may select a base layer of a predesign network structure corresponding to the base layer predesign network index number (e.g., a network layer of predesign network structure s1) from the predesign neural network pool. The enhancement layer information includes an enhancement layer predesign network use flag bit and an enhancement layer predesign network index number, and the enhancement layer predesign network use flag bit indicates that the enhancement layer uses a predesign network. In this case, the coefficient hyperparameter feature generation unit knows, based on the enhancement layer information, that the enhancement layer uses a network layer of a predesign network structure, and therefore may select an enhancement layer of a predesign network structure corresponding to the enhancement layer predesign network index number (e.g., a network layer of predesign network structure s1) from the predesign neural network pool. Based on this, a decoding neural network 1 may be obtained by combining the basic layer of the pre-designed network structure with the reinforcement layer of the pre-designed network structure.

[0226] Case 6: The base layer information includes a base layer predesign network use flag bit and a base layer predesign network index number, and the base layer predesign network use flag bit indicates that the base layer uses a predesign network. In this case, the coefficient hyperparameter feature generation unit knows, based on the base layer information, that the base layer uses a network layer of a predesign network structure, and therefore may select a base layer of a predesign network structure corresponding to the base layer predesign network index number from the predesign neural network pool (e.g., a network layer of predesign network structure s1). The enhancement layer information includes network parameters for generating an enhancement layer. In this case, the coefficient hyperparameter feature generation unit may analyze network parameters from control parameters and generate an enhancement layer of a variable network structure (i.e., a network layer of a variable network structure) based on these network parameters. For example, the coefficient hyperparameter feature generation unit may analyze network parameters such as the number of deconvolutional layers, the quantization stride of each deconvolutional layer, the number of channels in each deconvolutional layer, the size of the convolutional kernel, and the activation layer type from the control parameters, and the coefficient hyperparameter feature generation unit generates reinforcement layers of a variable network structure based on these network parameters. Based on this, the coefficient hyperparameter feature generation unit may combine the basic layer of the predesigned network structure and the reinforcement layer of the variable network structure to obtain the decoded neural network 1.

[0227] In one possible embodiment, the network structure of the base layer and the enhancement layer may both be determined by decoded control parameters, the control parameters may include neural network information 1, and the neural network information 1 may include base layer information and enhancement layer information. An example of neural network information 1 for a coefficient hyperparameter feature generation unit is shown in Table 1, where u(n) represents an n-bit fixed-length coding method and ae(v) represents a variable-length coding method. [Table 1]

[0228] In Table 1, hyper_basic_layer_use_default_para_flag is the flag bit for using the default network in the basic layer for the coefficient hyperparameter feature generation unit. hyper_basic_layer_use_default_para_flag is a binary variable where a value of 1 indicates that the basic layer of the coefficient hyperparameter feature generation unit uses the default network, and a value of 0 indicates that the basic layer of the coefficient hyperparameter feature generation unit does not use the default network. The value of HyperBasicLayerUseDefaultParaFlag is equal to the value of hyper_basic_layer_use_default_para_flag.

[0229] In Table 1, hyper_basic_layer_use_predesigned_para_flag is a flag bit for using a predesigned network in the basic layer for the coefficient hyperparameter feature generation unit, and hyper_basic_layer_use_predesigned_para_flag is a binary variable. A value of 1 for this binary variable indicates that the basic layer of the coefficient hyperparameter feature generation unit uses a predesigned network, and a value of 0 for this binary variable indicates that the basic layer of the coefficient hyperparameter feature generation unit does not use a predesigned network. The value of HyperBasicLayerUsePredesignedParaFlag may be equal to the value of hyper_basic_layer_use_predesigned_para_flag.

[0230] In Table 1, hyper_basic_id is the index number of the basic layer predesign network for the coefficient hyperparameter feature generation unit, and may be a 32-bit unsigned integer, indicating the index number in the predesign neural network pool for the neural network used in the basic layer.

[0231] In Table 1, hyper_enhance_layer_use_default_para_flag is the default network use flag bit for the coefficient hyperparameter feature generation unit, and hyper_enhance_layer_use_default_para_flag is a binary variable. A value of 1 for the binary variable indicates that the enhancement layer of the coefficient hyperparameter feature generation unit uses the default network, and a value of 0 for the binary variable indicates that the enhancement layer of the coefficient hyperparameter feature generation unit does not use the default network. The value of HyperEnhanceLayerUseDefaultParaFlag may be equal to the value of hyper_enhance_layer_use_default_para_flag.

[0232] In Table 1, hyper_enhance_layer_use_predesigned_para_flag is a flag bit for using a predesigned network in the enhancement layer for a coefficient hyperparameter feature generation unit, and hyper_enhance_layer_use_predesigned_para_flag is a binary variable. A value of 1 in the binary variable indicates that the enhancement layer of the coefficient hyperparameter feature generation unit uses a predesigned network, and a value of 0 in the binary variable indicates that the enhancement layer of the coefficient hyperparameter feature generation unit does not use a predesigned network. The value of HyperEnhanceLayerUsePredesignedParaFlag may be equal to the value of hyper_enhance_layer_use_predesigned_para_flag.

[0233] In Table 1, hyper_enhance_id is the enhancement layer predesign network index number for the coefficient hyperparameter feature generation unit, and may be a 32-bit unsigned integer, indicating the index number in the predesign neural network pool for the neural network used in the enhancement layer.

[0234] In the above process, the range of hyper_basic_id is [id_min, id_max], where id_min is preferably 0 and id_max is preferably 2^32-1, and segment [a, b] is a reserved segment for later expansion of the predesigned neural network pool. The predesigned neural network pool of the basic layer may include, for example, two, three, or four basic layer predesigned networks, or dozens of basic layer predesigned networks, or even more basic layer predesigned networks, and is not limited thereto. Clearly, the preference of id_max being 2^32-1 is merely illustrative, and the value of id_max may be dynamically adjusted depending on the circumstances.

[0235] In the above process, the range of hyper_enhance_id is [id_min, id_max], where id_min is preferably 0 and id_max is preferably 2^32-1, and segment [a, b] is a reserved segment for later expansion of the predesigned neural network pool. The predesigned neural network pool for the enhancement layer may include, for example, two, three, or four enhancement layer predesign networks, or dozens of enhancement layer predesign networks, or even more enhancement layer predesign networks, and is not limited thereto. Clearly, the preference of id_max being 2^32-1 is merely illustrative, and the value of id_max may be dynamically adjusted as needed.

[0236] In one possible embodiment, the network structure of the reinforcement layer may be determined by decoded control parameters, the control parameters may include neural network information 1, and the neural network information 1 may include reinforcement layer information. Examples of neural network information 1 for coefficient hyperparameter feature generation units are shown in Tables 2 and 3. [Table 2] [Table 3]

[0237] In Table 2, `layer_num` indicates the number of layers in the neural network. It is used to indicate the number of network layers in a neural network, and if an activation layer is included in the network structure, the layer count is not extra, and the value of `LayerNum` is equal to `layer_num`.

[0238] In Table 3, deconv_layer_flag represents the deconvolutional layer flag bit, and deconv_layer_flag is a binary variable. If the value of this binary variable is 1, it indicates that the current layer is a deconvolutional layer network; if the value of this binary variable is 0, it indicates that the current layer is not a deconvolutional layer network, and the value of DeconvLayerFlag is equal to the value of deconv_layer_flag.

[0239] In Table 3, stride_num represents the quantized stride of the deconvolutional layer.

[0240] In Table 3, filter_num represents the number of filters, i.e., the number of filters in the current layer.

[0241] In Table 3, `filter_size_index` represents the filtering size index, i.e., the current filtering size index value.

[0242] In Table 3, filter_coeff_zero_flag[i][j] indicates the filtering coefficient zero flag bit and is a binary variable. When the value of the binary variable is 1, it indicates that the current filtering coefficient is 0. When the value of the binary variable is 0, it indicates that the current filtering coefficient is not 0. The value of FilterCoeffZeroFlag[i][j] is equal to the value of filter_coeff_zero_flag[i][j].

[0243] In Table 3, filter_coeff[i][j] indicates the filtering coefficient, that is, the current filtering coefficient value.

[0244] In Table 3, activation_layer_flag indicates the activation layer flag bit and activation_layer_flag is a binary variable. When the value of the binary variable is 1, it indicates that the current layer is an activation layer. When the value of the binary variable is 0, it indicates that the current layer is not an activation layer. The value of ActivationLayerFlag is equal to the value of activation_layer_flag.

[0245] In Table 3, activation_layer_type indicates the activation layer type, that is, the specific type of the activation layer of the current layer.

[0246] Example 10: For Example 5 and Example 6, the second feature decoding unit may decode the bitstream 2 corresponding to the current block to obtain the image feature information corresponding to the current block. For example, the second feature decoding unit may include at least one coefficient decoding module and one probability model acquisition module. In one possible embodiment, the coefficient decoding module performs coefficient decoding using an entropy decoding method, that is, it may decode the bitstream 2 corresponding to the current block using the entropy decoding method to obtain the image feature information corresponding to the current block.

[0247] Exemplarily, the entropy decoding method may include, but is not limited to, CAVLC or CABAC, etc., and is not limited thereto.

[0248] Exemplarily, the method of using the features generated by the coefficient hyperparameter feature generation unit in the second feature decoding unit may include the following.

[0249] Method 1: The probability model acquisition module is used to acquire the probability model of entropy decoding. For example, the probability model acquisition module acquires the coefficient hyperparameter feature value P from the coefficient hyperparameter feature generation unit. Based on this, the coefficient decoding module can acquire the coefficient hyperparameter feature value P from the probability model acquisition module, and based on the coefficient hyperparameter feature value P, the coefficient decoding module can perform coefficient decoding using the entropy decoding method.

[0250] Method 2: The coefficient analysis process (for example, the CABAC or CAVCL decoding process) does not depend on the features generated by the coefficient hyperparameter feature generation unit and can directly analyze to obtain the coefficient value (in this way, the analysis throughput or rate can be guaranteed). Based on the features generated by the coefficient hyperparameter feature generation unit, the analyzed coefficient value is converted to obtain the image feature quantization value F_q. As an example, when the coefficient value obtained in the coefficient analysis process is 0 and the corresponding feature value generated by the coefficient hyperparameter feature generation unit is u, F_q = u; when the coefficient value obtained in the coefficient analysis process is 1 and the corresponding feature value generated by the coefficient hyperparameter feature generation unit is u, F_q = u + x, where x is the corresponding coefficient variance.

[0251] Example 11: In the case of Example 5, the second inverse quantization unit may perform inverse quantization on the image feature quantization value F_q (i.e., image feature information) to obtain the image feature reconstruction value F'. For example, the second inverse quantization unit may not exist, or if the second inverse quantization unit exists, the second inverse quantization unit may be selectively skipped based on control parameters (e.g., high-level syntax, e.g., second enable information), or it may be decided to enable the second inverse quantization unit based on control parameters. Exemplarily, if the second inverse quantization unit does not exist, the image feature reconstruction value F' is the same as the image feature quantization value F_q, i.e., there is no need to perform inverse quantization on the image feature quantization value F_q. If the second inverse quantization unit is selectively skipped based on control parameters, the image feature reconstruction value F' is the same as the image feature quantization value F_q, i.e., there is no need to perform inverse quantization on the image feature quantization value F_q. We decided to enable the second inverse quantization unit based on the control parameters, but when the step parameter qstep, which corresponds to the image feature quantization value F_q, is 1, the image feature reconstruction value F' is the same as the image feature quantization value F_q, meaning that inverse quantization does not need to be performed on the image feature quantization value F_q.

[0252] Exemplarily, it is determined to enable the second inverse quantization unit based on a control parameter, and if the step parameter qstep corresponding to the image feature quantization value Fq is not 1, the second inverse quantization unit may perform inverse quantization on the image feature quantization value Fq based on the control parameter (e.g., quantization-related parameter) to obtain an image feature reconstruction value F'. For example, the second inverse quantization unit performs the following processing. It obtains a quantization-related parameter corresponding to the image feature quantization value Fq from the control parameter (the control parameter may be included in the bitstream and the control parameter may include the quantization-related parameter), for example, the step parameter qstep or the quantization parameter qp. Based on the step parameter qstep or the quantization parameter qp, it determines a multiplication factor mult and a shift factor shift corresponding to the image feature quantization value Fq. Assuming the image feature quantization value Fq is Coff_hyper and the image feature reconstruction value F' is Coff_hyper_rec, then Coff_hyper_rec = (Coff_hyper * mult) << shift, that is, when performing inverse quantization, the above formula may be used to obtain the image feature reconstruction value F'.

[0253] Regarding the quantization-related parameter (e.g., step parameter qstep) corresponding to the image feature quantization value Fq, it includes: 1) all image feature quantization values of each feature channel use the same step parameter qstep; 2) the image feature quantization values of each feature channel use different step parameters qstep, but each image feature quantization value within the feature channel uses the same step parameter qstep; 3) all image feature quantization values of each feature channel use different step parameters qstep.

[0254] Example 12: In Examples 5 and 6, the image feature inverse transform unit may perform an inverse transform operation on the image feature reconstruction value F' based on the decoding neural network to obtain the image low-level feature value LF. In one possible embodiment, as shown in Figure 6D, the image feature inverse transform unit may include a decoding neural network 2, which may include a base layer and an enhancement layer, the image feature reconstruction value F' is an input feature of the decoding neural network 2, the image low-level feature value LF is an output feature of the decoding neural network 2, and the decoding neural network 2 is used to perform an inverse transform operation on the image feature reconstruction value F'.

[0255] In this embodiment, the decoding neural network 2 may be divided into a basic layer and an enhancement layer. The basic layer may contain at least one network layer, or it may not contain any network layers; that is, the basic layer may be empty. The enhancement layer may contain at least one network layer, or it may not contain any network layers; that is, the enhancement layer may be empty. Furthermore, for multiple network layers in the decoding neural network 2, multiple network layers may be divided into a basic layer and an enhancement layer according to the actual needs. For example, a network layer with a fixed network structure may be used as the basic layer, and a network layer with an unfixed network structure may be used as the enhancement layer.

[0256] For example, in decoding neural network 2, the size of the output features may be larger than the size of the input features, or the size of the output features may be equal to the size of the input features, or the size of the output features may be smaller than the size of the input features.

[0257] For decoding neural network 2, the base layer and reinforcement layer each include at least one deconvolutional layer. For example, the base layer may include at least one deconvolutional layer, and the reinforcement layer may include at least one deconvolutional layer, or it may not include any deconvolutional layers. Alternatively, the reinforcement layer may include at least one deconvolutional layer, and the base layer may include at least one deconvolutional layer, or it may not include any deconvolutional layers.

[0258] In decoding neural network 2, the base layer and reinforcement layer each include at least one residual structure layer. For example, the base layer includes at least one residual structure layer, and the reinforcement layer may or may not include a residual structure layer. Alternatively, the reinforcement layer includes at least one residual structure layer, and the base layer may or may not include a residual structure layer.

[0259] For example, decoding neural network 2 may include, but is not limited to, deconvolutional layers and activation layers. For instance, decoding neural network 2 may include, in order, a deconvolutional layer with a stride of 2, one activation layer, a deconvolutional layer with a stride of 1, and one activation layer. All of the above network layers may be considered basic layers, and the reinforcement layer may be empty. Alternatively, for example, decoding neural network 2 may include, in order, a deconvolutional layer with a stride of 2, and one deconvolutional layer with a stride of 1. All of the above network layers may be considered basic layers, and the reinforcement layer may be empty. Alternatively, for example, decoding neural network 2 may include, in order, a deconvolutional layer with a stride of 2, one activation layer, a deconvolutional layer with a stride of 2, one activation layer, a deconvolutional layer with a stride of 1, and one activation layer. All of the above network layers may be considered basic layers, and the reinforcement layer may be empty.

[0260] For example, if no quality enhancement unit is present, the number of output features (filters) in the final network layer of the inverse image feature transform unit is either 1 or 3. Specifically, if the output is only a single channel value (e.g., a grayscale image), the number of output features in the final network layer of the inverse image feature transform unit is 1, and if the output is only a three-channel value (e.g., RGB or YUV format), the number of output features in the final network layer of the inverse image feature transform unit is 3.

[0261] For example, if a quality enhancement unit is present, the number of output features (filters) in the last network layer of the image feature inverse transform unit may be 1 or 3, or any other value, and is not limited thereto.

[0262] In one possible embodiment, a network layer of a default network structure may be set up for the image feature inverse transform unit, and all network parameters relating to the network layer of the default network structure are fixed. For example, in the network layer of the default network structure, the number of deconvolutional layers is fixed, the number of activation layers is fixed, the number of channels in each deconvolutional layer is fixed, the size of the convolution kernel is fixed, and the filtering coefficients are fixed, and so on. Clearly, all network parameters in the network layer of the default network structure are fixed and these network parameters are known, so the network layer of the default network structure can be obtained directly.

[0263] In one possible embodiment, a pre-designed neural network pool may be set up for the image feature inverse transform unit, the pre-designed neural network pool includes network layers of at least one pre-designed network structure, and the network parameters for the network layers of the pre-designed network structure may all be set according to the actual needs. For example, the pre-designed neural network pool includes network layers of pre-designed network structure t1, network layers of pre-designed network structure t2, and network layers of pre-designed network structure t3. Here, for the network layers of pre-designed network structure t1, network parameters such as the number of deconvolutional layers, the number of activation layers, the number of channels in each deconvolutional layer, the size of the convolution kernel, and the filtering coefficients may be set in advance, and after all network parameters have been set, the network layers of pre-designed network structure t1 can be obtained, and so on.

[0264] In one possible embodiment, a network layer with a variable network structure may be dynamically generated for the image feature inverse transform unit based on network parameters, where the network parameters for the network layer with a variable network structure are not pre-set but are dynamically generated by the encoding side. For example, the encoding side may encode network parameters such as the number of deconvolutional layers, the number of activation layers, the number of channels in each deconvolutional layer, the size of the convolution kernel, and filtering coefficients into the bitstream, and the decoding side may analyze the above network parameters from the bitstream and generate a network layer with a variable network structure based on these network parameters.

[0265] In one possible embodiment, the control parameters may include neural network information 2 corresponding to an image feature inverse transform unit, and the image feature inverse transform unit may analyze the neural network information 2 from the control parameters and generate a decoding neural network 2 based on the neural network information 2. For example, the neural network information 2 may include basic layer information and reinforcement layer information, and the basic layer may be determined based on the basic layer information, the reinforcement layer may be determined based on the reinforcement layer information, and the decoding neural network 2 may be obtained by combining the basic layer and the reinforcement layer. For example, the neural network information 2 may be obtained using the following case.

[0266] Case 1: The base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network. In this case, the image feature inverse transform unit knows, based on the base layer information, that the base layer uses the network layer of the default network structure, and therefore obtains the base layer of the default network structure. The enhancement layer information includes an enhancement layer default network use flag bit, and the enhancement layer default network use flag bit indicates that the enhancement layer uses the default network. In this case, based on the enhancement layer information, the unit knows that the enhancement layer uses the network layer of the default network structure, and therefore obtains the enhancement layer of the default network structure. The base layer of the default network structure and the enhancement layer of the default network structure are combined to obtain decoding neural network 2.

[0267] Case 2: The base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network. In this case, the image feature inverse transform unit knows, based on the base layer information, that the base layer uses the network layer of the default network structure, and therefore obtains the base layer of the default network structure. The enhancement layer information includes an enhancement layer predesign network use flag bit and an enhancement layer predesign network index number, and the enhancement layer predesign network use flag bit indicates that the enhancement layer uses a predesign network. In this case, based on the enhancement layer information, the unit knows that the enhancement layer uses the network layer of the predesign network structure, and therefore selects the enhancement layer of the predesign network structure corresponding to the enhancement layer predesign network index number from the predesign neural network pool. The base layer of the default network structure and the enhancement layer of the predesign network structure may be combined to obtain decoding neural network 2.

[0268] Case 3: The base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network. Based on the base layer information, the image feature inverse transform unit knows that the base layer uses a network layer with the default network structure, and therefore obtains a base layer with the default network structure. The reinforcement layer information includes network parameters for generating the reinforcement layer. In this case, the network parameters are analyzed from the control parameters, and a reinforcement layer with a variable network structure is generated based on these network parameters. The base layer with the default network structure and the reinforcement layer with the variable network structure are combined to obtain the decoded neural network 2.

[0269] Case 4: The base layer information includes a base layer pre-designed network usage flag bit and a base layer pre-designed network index number, and the base layer pre-designed network usage flag bit indicates that the base layer uses the pre-designed network. In this case, based on the base layer information, the image feature inverse conversion unit knows that the base layer uses the network layer of the pre-designed network structure. Therefore, it selects the base layer of the pre-designed network structure corresponding to the base layer pre-designed network index number from the pre-designed neural network pool. The enhancement layer information includes an enhancement layer default network usage flag bit, and the enhancement layer default network usage flag bit indicates that the enhancement layer uses the default network. In this case, based on the enhancement layer information, it knows that the enhancement layer uses the network layer of the default network structure. Therefore, it obtains the enhancement layer of the default network structure. Combining the base layer of the pre-designed network structure and the enhancement layer of the default network structure, a decoding neural network 2 may be obtained.

[0270] Case 5: The basic layer information includes a basic layer predesigned network use flag bit and a basic layer predesigned network index number, and the basic layer predesigned network use flag bit indicates that the basic layer uses a predesigned network. In this case, the image feature inverse transform unit knows, based on the basic layer information, that the basic layer uses a network layer of a predesigned network structure, and therefore may select a basic layer of a predesigned network structure corresponding to the basic layer predesigned network index number from the predesigned neural network pool. The enhancement layer information includes an enhancement layer predesigned network use flag bit and an enhancement layer predesigned network index number, and the enhancement layer predesigned network use flag bit indicates that the enhancement layer uses a predesigned network. In this case, based on the enhancement layer information, the enhancement layer knows that it uses a network layer of a predesigned network structure, and therefore may select an enhancement layer of a predesigned network structure corresponding to the enhancement layer predesigned network index number from the predesigned neural network pool. The basic layer of the predesigned network structure and the enhancement layer of the predesigned network structure may be combined to obtain the decoding neural network 2.

[0271] Case 6: The base layer information includes a base layer predesign network use flag bit and a base layer predesign network index number, and the base layer predesign network use flag bit indicates that the base layer uses a predesign network. In this case, the image feature inverse transform unit knows, based on the base layer information, that the base layer uses a network layer of a predesign network structure, and therefore may select a base layer of a predesign network structure corresponding to the base layer predesign network index number from the predesign neural network pool. The enhancement layer information includes network parameters for generating an enhancement layer. In this case, the network parameters are analyzed from the control parameters, and an enhancement layer of a variable network structure is generated based on these network parameters. The base layer of the predesign network structure and the enhancement layer of the variable network structure are combined to obtain the decoding neural network 2.

[0272] In one possible embodiment, the above network structure of the base layer and the reinforcement layer may both be determined by decoded control parameters, the control parameters may include neural network information 2, and the neural network information 2 may include base layer information and reinforcement layer information, and the neural network information 2 for the image feature inverse transform unit is similar to that in Tables 1, 2, and 3, but the relevant information is for the image feature inverse transform unit and not for the coefficient hyperparameter feature generation unit, and is not repeated here.

[0273] Example 13: In Examples 5 and 6, the quality enhancement unit may acquire low-level image feature values ​​LF and perform enhancement processing on the low-level image feature values ​​LF based on the decoding neural network to obtain a reconstructed image block I corresponding to the current block.

[0274] For example, a quality enhancement unit may not exist, or if a quality enhancement unit exists, it may be selectively skipped based on control parameters (e.g., high-level syntax, e.g., third enable information), or it may be decided to enable the quality enhancement unit based on the control parameters. Exemplarily, if it is decided to enable the quality enhancement unit based on control parameters, the quality enhancement unit may be used to eliminate image quality degradation problems such as blocking artifacts between blocks and quantization distortion. For example, the quality enhancement unit may perform enhancement processing on low-level image feature values ​​LF based on a decoding neural network to obtain a reconstructed image block I corresponding to the current block.

[0275] In one possible embodiment, the quality enhancement unit may include a decoding neural network 3, which may include a base layer and an enhancement layer, where image low-level feature values ​​LF are input features of the decoding neural network 3, and reconstructed image block I is an output feature of the decoding neural network 3, and the decoding neural network 3 is used to perform enhancement processing on the image low-level feature values ​​LF.

[0276] In this embodiment, the decoding neural network 3 may be divided into a basic layer and an enhancement layer. The basic layer may include at least one network layer, or it may not include any network layers; that is, the basic layer may be empty. The enhancement layer may include at least one network layer, or it may not include any network layers; that is, the enhancement layer may be empty. Furthermore, for multiple network layers in the decoding neural network 3, multiple network layers may be divided into a basic layer and an enhancement layer according to the actual needs. For example, a network layer with a fixed network structure may be used as the basic layer, and a network layer with an unfixed network structure may be used as the enhancement layer.

[0277] For example, in decoding neural network 3, the size of the output features may be larger than the size of the input features, or the size of the output features may be equal to the size of the input features, or the size of the output features may be smaller than the size of the input features.

[0278] For decoding neural network 3, the base layer and reinforcement layer each include at least one deconvolutional layer. For example, the base layer may include at least one deconvolutional layer, and the reinforcement layer may include at least one deconvolutional layer, or it may not include any deconvolutional layers. Alternatively, the reinforcement layer may include at least one deconvolutional layer, and the base layer may include at least one deconvolutional layer, or it may not include any deconvolutional layers.

[0279] In the decoding neural network 3, the base layer and the reinforcement layer each include at least one residual structure layer. For example, the base layer includes at least one residual structure layer, and the reinforcement layer may or may not include a residual structure layer. Alternatively, the reinforcement layer includes at least one residual structure layer, and the base layer may or may not include a residual structure layer.

[0280] For example, the decoding neural network 3 may include, but is not limited to, deconvolutional layers and activation layers. For instance, the decoding neural network 3 includes, in order, a deconvolutional layer with a stride of 2, one activation layer, a deconvolutional layer with a stride of 1, and one activation layer. All of the above network layers are considered basic layers, and the reinforcement layer may be empty. Alternatively, for example, the decoding neural network 3 includes, in order, a deconvolutional layer with a stride of 2, and one deconvolutional layer with a stride of 1. All of the above network layers are considered basic layers, and the reinforcement layer may be empty. Alternatively, for example, the decoding neural network 3 includes, in order, a deconvolutional layer with a stride of 2, one activation layer, a deconvolutional layer with a stride of 2, one activation layer, a deconvolutional layer with a stride of 1, and one activation layer. All of the above network layers are considered basic layers, and the reinforcement layer may be empty.

[0281] For example, the number of output features (filters) in the final network layer of the quality enhancement unit is either 1 or 3. Specifically, if the output is only a single channel value (e.g., a grayscale image), the number of output features in the final network layer is 1, and if the output is only a three-channel value (e.g., RGB or YUV format), the number of output features in the final network layer is 3.

[0282] In one possible embodiment, a network layer of a default network structure may be configured for the high-quality unit, and all network parameters for the network layer of the default network structure are fixed. A pre-designed neural network pool may be configured for the high-quality unit, and the pre-designed neural network pool includes at least one network layer of a pre-designed network structure, and all network parameters for the network layer of the pre-designed network structure may be configured according to actual needs. A network layer of a variable network structure may be dynamically generated for the high-quality unit based on network parameters, and the network parameters for the network layer of the variable network structure are not pre-configured but are dynamically generated by the encoding side.

[0283] In one possible embodiment, the control parameters may include neural network information 3 corresponding to the quality enhancement unit, and the quality enhancement unit may analyze the neural network information 3 from the control parameters and generate a decoding neural network 3 based on the neural network information 3. For example, the neural network information 3 may include basic layer information and reinforcement layer information, and the basic layer may be determined based on the basic layer information, the reinforcement layer may be determined based on the reinforcement layer information, and the decoding neural network 3 may be obtained by combining the basic layer and the reinforcement layer. For example, the quality enhancement unit may obtain the neural network information 3 using the following case.

[0284] Case 1: The base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network. In this case, the enhancement unit knows, based on the base layer information, that the base layer uses the network layer of the default network structure, and therefore obtains the base layer of the default network structure. The enhancement layer information includes an enhancement layer default network use flag bit, and the enhancement layer default network use flag bit indicates that the enhancement layer uses the default network. In this case, based on the enhancement layer information, the enhancement layer knows that the enhancement layer uses the network layer of the default network structure, and therefore obtains the enhancement layer of the default network structure. The base layer of the default network structure and the enhancement layer of the default network structure are combined to obtain the decoding neural network 3.

[0285] Case 2: The base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network. In this case, the enhancement unit knows, based on the base layer information, that the base layer uses the network layer of the default network structure, and therefore obtains the base layer of the default network structure. The enhancement layer information includes an enhancement layer predesign network use flag bit and an enhancement layer predesign network index number, and the enhancement layer predesign network use flag bit indicates that the enhancement layer uses a predesign network. In this case, based on the enhancement layer information, the enhancement layer knows that the enhancement layer uses the network layer of the predesign network structure, and therefore selects the enhancement layer of the predesign network structure corresponding to the enhancement layer predesign network index number from the predesign neural network pool. The base layer of the default network structure and the enhancement layer of the predesign network structure may be combined to obtain the decoding neural network 3.

[0286] Case 3: The base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network. Based on the base layer information, the enhancement unit knows that the base layer uses a network layer with the default network structure, and therefore obtains a base layer with the default network structure. The enhancement layer information includes network parameters for generating the enhancement layer. In this case, the network parameters are analyzed from the control parameters, and an enhancement layer with a variable network structure is generated based on these network parameters. The base layer with the default network structure and the enhancement layer with the variable network structure are combined to obtain the decoding neural network 3.

[0287] Case 4: The base layer information includes a base layer predesign network use flag bit and a base layer predesign network index number, and the base layer predesign network use flag bit indicates that the base layer uses a predesign network. In this case, the enhancement unit knows, based on the base layer information, that the base layer uses a network layer of the predesign network structure, and therefore selects the base layer of the predesign network structure corresponding to the base layer predesign network index number from the predesign neural network pool. The enhancement layer information includes an enhancement layer default network use flag bit, and the enhancement layer default network use flag bit indicates that the enhancement layer uses a default network. In this case, based on the enhancement layer information, the enhancement layer knows that the enhancement layer uses a network layer of the default network structure, and therefore obtains the enhancement layer of the default network structure. The base layer of the predesign network structure and the enhancement layer of the default network structure may be combined to obtain the decoding neural network 3.

[0288] Case 5: The basic layer information includes a basic layer predesign network use flag bit and a basic layer predesign network index number, and the basic layer predesign network use flag bit indicates that the basic layer uses a predesign network. In this case, the quality enhancement unit knows, based on the basic layer information, that the basic layer uses a network layer of a predesign network structure, and therefore may select the basic layer of the predesign network structure corresponding to the basic layer predesign network index number from the predesign neural network pool. The enhancement layer information includes an enhancement layer predesign network use flag bit and an enhancement layer predesign network index number, and the enhancement layer predesign network use flag bit indicates that the enhancement layer uses a predesign network. In this case, the enhancement layer information knows, based on the enhancement layer information, that the enhancement layer uses a network layer of a predesign network structure, and therefore may select the enhancement layer of the predesign network structure corresponding to the enhancement layer predesign network index number from the predesign neural network pool. The basic layer of the predesign network structure and the enhancement layer of the predesign network structure may be combined to obtain the decoding neural network 3.

[0289] Case 6: The base layer information includes a base layer predesign network use flag bit and a base layer predesign network index number, and the base layer predesign network use flag bit indicates that the base layer uses a predesign network. In this case, the quality enhancement unit knows, based on the base layer information, that the base layer uses a network layer of a predesign network structure, and therefore may select a base layer of a predesign network structure corresponding to the base layer predesign network index number from the predesign neural network pool. The enhancement layer information includes network parameters for generating an enhancement layer. In this case, the network parameters are analyzed from the control parameters, and an enhancement layer of a variable network structure is generated based on these network parameters. The base layer of the predesign network structure and the enhancement layer of the variable network structure are combined to obtain the decoding neural network 3.

[0290] In one possible embodiment, the above network structure of the base layer and the enhancement layer may both be determined by decoded control parameters, the control parameters may include neural network information 3, and the neural network information 3 may include base layer information and enhancement layer information. The neural network information 3 for the quality enhancement unit is similar to that in Tables 1, 2, and 3, but the relevant information is for the quality enhancement unit and not for the coefficient hyperparameter feature generation unit, and is not repeated here.

[0291] Example 14: An embodiment of the present invention provides an image coding method based on a neural network, which may be applied to the coding side (also called a video encoder), and Figure 7A is a schematic diagram of the coding side, which may include a control parameter coding unit, a feature transformation unit, a coefficient hyperparameter feature transformation unit, a first quantization unit, a second quantization unit, a first feature coding unit, a second feature coding unit, a first feature decoding unit, a second feature decoding unit, a coefficient hyperparameter feature generation unit, an image feature inverse transformation unit, a first inverse quantization unit, a second inverse quantization unit, and a quality enhancement unit.

[0292] For example, the first quantization unit, the second quantization unit, the first inverse quantization unit, the second inverse quantization unit, and the quality enhancement unit are optional units, and in certain scenarios, it is possible to turn off or skip the processes of these optional units.

[0293] In this embodiment, for each current block (i.e., an image block), the bitstream corresponding to the current block includes three parts: bitstream 0 (a bitstream containing control parameters), bitstream 1 (a bitstream containing coefficient hyperparameter feature information), and bitstream 2 (a bitstream containing image feature information). The coefficient hyperparameter feature information and image feature information may be collectively referred to as image information.

[0294] For example, the neural network-based image coding method in this embodiment may include the following steps S71 to S79.

[0295] In step S71, a feature transformation is performed on the current block I to obtain the image feature value F corresponding to the current block I. For example, the feature transformation unit may perform a feature transformation on the current block I to obtain the image feature value F corresponding to the current block I. Exemplarily, the feature transformation unit may perform a feature transformation on the current block I based on an encoded neural network to obtain the image feature value F corresponding to the current block I. Here, the current block I is the input feature of the encoded neural network, and the image feature value F is the output feature of the encoded neural network.

[0296] In step S72, image feature information is determined based on the image feature value F.

[0297] For example, when enabling a second quantization unit, the second quantization unit may obtain an image feature value F from the feature transformation unit, perform quantization on the image feature value F to obtain an image feature quantization value F_q, and determine image feature information based on the image feature quantization value F_q; that is, the image feature information may be the image feature quantization value F_q. In this case, the control parameter coding unit may encode the second enable information of the second quantization unit in bitstream 0; that is, the control parameter includes the second enable information of the second quantization unit, and the second enable information is used to indicate that the second quantization unit has been enabled. The control parameter coding unit may also encode quantization-related parameters corresponding to the second quantization unit in bitstream 0, such as the step parameter qstep or the quantization parameter qp.

[0298] For example, if the second quantization unit is not enabled, the image feature information may be determined based on the image feature value F, i.e., the image feature information may be the image feature value F. In this case, the control parameter coding unit may encode the second enable information for the second quantization unit into bitstream 0, and the second enable information is used to indicate that the second quantization unit is not enabled.

[0299] In step S73, a coefficient hyperparameter feature transformation is performed on the image feature value F to obtain the coefficient hyperparameter feature coefficient value C. For example, the coefficient hyperparameter feature transformation unit performs a coefficient hyperparameter feature transformation on the image feature value F to obtain the coefficient hyperparameter feature coefficient value C, and for example, based on the coding neural network, a coefficient hyperparameter feature transformation is performed on the image feature value F to obtain the coefficient hyperparameter feature coefficient value C. Here, the image feature value F is the input feature of the coding neural network, and the coefficient hyperparameter feature coefficient value C is the output feature of the coding neural network.

[0300] In step S74, the coefficient hyperparameter feature information is determined based on the coefficient hyperparameter feature coefficient value C.

[0301] For example, when enabling the first quantization unit, the first quantization unit may obtain a coefficient hyperparameter feature coefficient value C from the coefficient hyperparameter feature transformation unit, perform quantization on the coefficient hyperparameter feature coefficient value C to obtain a coefficient hyperparameter feature coefficient quantization value C_q, and determine the coefficient hyperparameter feature information based on the coefficient hyperparameter feature coefficient quantization value C_q; that is, the coefficient hyperparameter feature information may be the coefficient hyperparameter feature coefficient quantization value C_q. In this case, the control parameter coding unit may encode the first enable information of the first quantization unit in bitstream 0; that is, the control parameter includes the first enable information of the first quantization unit, and the first enable information is used to indicate that the first quantization unit has been enabled. The control parameter coding unit may encode quantization-related parameters corresponding to the first quantization unit in bitstream 0.

[0302] For example, if the first quantization unit is not enabled, the coefficient hyperparameter feature information is determined based on the coefficient hyperparameter feature coefficient value C, i.e., the coefficient hyperparameter feature information may be the coefficient hyperparameter feature coefficient value C. In this case, the control parameter coding unit may encode the first enable information of the first quantization unit into bitstream 0, and the first enable information is used to indicate that the first quantization unit is not enabled.

[0303] In step S75, the coefficient hyperparameter feature information (for example, the coefficient hyperparameter feature coefficient quantization value C_q or the coefficient hyperparameter feature coefficient value C) is encoded to obtain bitstream 1. For example, the first feature encoding unit may encode the coefficient hyperparameter feature information into the bitstream corresponding to the current block, and for ease of distinction, the bitstream containing the coefficient hyperparameter feature information is denoted as bitstream 1.

[0304] In step S76, bitstream 1 corresponding to the current block is decoded to obtain coefficient hyperparameter feature information (for example, coefficient hyperparameter feature coefficient quantization value C_q or coefficient hyperparameter feature coefficient value C), and the coefficient hyperparameter feature coefficient reconstruction value is determined based on the coefficient hyperparameter feature information.

[0305] For example, the first feature decoding unit may decode bitstream 1 corresponding to the current block to obtain coefficient hyperparameter feature information.

[0306] For example, when the first quantization unit is enabled, the coefficient hyperparameter feature information is the coefficient hyperparameter feature coefficient quantization value C_q. In this case, the first inverse quantization unit may perform inverse quantization on the coefficient hyperparameter feature coefficient quantization value C_q to obtain the coefficient hyperparameter feature coefficient reconstruction value C'. Clearly, the coefficient hyperparameter feature coefficient reconstruction value C' may be the same as the coefficient hyperparameter feature coefficient value C. When the first quantization unit is not enabled, the coefficient hyperparameter feature information is the coefficient hyperparameter feature coefficient value C. In this case, the coefficient hyperparameter feature coefficient value C may be used as the coefficient hyperparameter feature coefficient reconstruction value C'. As described above, the coefficient hyperparameter feature coefficient reconstruction value C' can be obtained.

[0307] As can be seen from the above, the coefficient hyperparameter feature coefficient reconstruction value C' is the same as the coefficient hyperparameter feature coefficient value C, so the process in step S76 can be omitted, and the coefficient hyperparameter feature coefficient value C can be directly used as the coefficient hyperparameter feature coefficient reconstruction value C'. In this case, the encoding side structure shown in Figure 7A can be improved to obtain the encoding side structure shown in Figure 7B.

[0308] In step S77, an inverse transform is performed on the coefficient hyperparameter feature coefficient reconstruction value C' to obtain the coefficient hyperparameter feature value P. For example, the coefficient hyperparameter feature generation unit obtains the coefficient hyperparameter feature value P by performing an inverse transform on the coefficient hyperparameter feature coefficient reconstruction value C'.

[0309] In step S78, the image feature information (e.g., image feature quantization value F_q or image feature value F) is encoded based on the coefficient hyperparameter feature value P to obtain bitstream 2. For example, the second feature encoding unit may encode the image feature information into the bitstream corresponding to the current block, and for ease of distinction, the bitstream containing the image feature information is denoted as bitstream 2.

[0310] In step S79, the control parameter coding unit obtains the control parameter corresponding to the current block, which may include neural network information, encodes the control parameter corresponding to the current block into a bitstream, and denotes the bitstream containing the control parameter as bitstream 0.

[0311] In one possible embodiment, a feature transformation unit may perform a feature transformation on the current block I using an encoding neural network, and a control parameter encoding unit may determine neural network information 2 corresponding to the image feature inverse transformation unit of the decoding device based on the network structure of the encoding neural network, the neural network information 2 being used to determine the decoding neural network 2 corresponding to the image feature inverse transformation unit, and encoding the neural network information 2 corresponding to the image feature inverse transformation unit into bitstream 0.

[0312] In one possible embodiment, a coefficient hyperparameter feature transformation unit may perform a coefficient hyperparameter feature transformation on image feature values ​​F using an encoding neural network, and a control parameter encoding unit may determine neural network information 1 corresponding to a coefficient hyperparameter feature generation unit of a decoding device based on the network structure of the encoding neural network, the neural network information 1 being used to determine a decoding neural network 1 corresponding to a coefficient hyperparameter feature generation unit, and encoding the neural network information 1 corresponding to a coefficient hyperparameter feature generation unit into bitstream 0.

[0313] In one possible embodiment, the following steps S80 to S86 may be further included, as shown in Figures 7A and 7B.

[0314] In step S80, bitstream 1 corresponding to the current block is decoded to obtain coefficient hyperparameter feature information corresponding to the current block. For example, the first feature decoding unit may decode bitstream 1 corresponding to the current block to obtain coefficient hyperparameter feature information corresponding to the current block.

[0315] In step S81, the coefficient hyperparameter feature coefficient reconstruction value is determined based on the coefficient hyperparameter feature information.

[0316] For example, when the first quantization unit is enabled, the coefficient hyperparameter feature information is the coefficient hyperparameter feature coefficient quantization value C_q, and in this case, the first inverse quantization unit may perform inverse quantization on the coefficient hyperparameter feature coefficient quantization value C_q to obtain the coefficient hyperparameter feature coefficient reconstruction value C'. Alternatively, when the first quantization unit is not enabled, the coefficient hyperparameter feature information is the coefficient hyperparameter feature coefficient value C, and in this case, the coefficient hyperparameter feature coefficient value C may be used as the coefficient hyperparameter feature coefficient reconstruction value C'.

[0317] In step S82, an inverse transform is performed on the coefficient hyperparameter feature coefficient reconstruction value C' to obtain the coefficient hyperparameter feature value P. For example, the coefficient hyperparameter feature generation unit obtains the coefficient hyperparameter feature value P by performing an inverse transform on the coefficient hyperparameter feature coefficient reconstruction value C', and for example, the coefficient hyperparameter feature generation unit obtains the coefficient hyperparameter feature value P by performing an inverse transform on the coefficient hyperparameter feature coefficient reconstruction value C' based on the decoding neural network.

[0318] In step S83, the bitstream 2 corresponding to the current block is decoded to obtain image feature information corresponding to the current block. For example, the second feature decoding unit may decode the bitstream 2 corresponding to the current block to obtain image feature information corresponding to the current block. When decoding the bitstream 2 corresponding to the current block, the second feature decoding unit may decode the bitstream 2 corresponding to the current block using the coefficient hyperparameter feature value P.

[0319] In step S84, the image feature reconstruction value is determined based on the image feature information.

[0320] For example, if the second quantization unit is enabled, the image feature information is the image feature quantization value F_q, and the second inverse quantization unit may perform inverse quantization on the image feature quantization value F_q to obtain the image feature reconstruction value F'. If the second quantization unit is not enabled, the image feature information is the image feature value F, and the image feature value F may be used as the image feature reconstruction value F'.

[0321] In step S85, an inverse transformation is performed on the image feature reconstruction value F' to obtain the image low-level feature value LF. For example, the image feature inverse transformation unit performs an inverse transformation on the image feature reconstruction value F' to obtain the image low-level feature value LF, and for example, based on a decoding neural network, an inverse transformation is performed on the image feature reconstruction value F' to obtain the image low-level feature value LF.

[0322] In step S86, the reconstructed image block I corresponding to the current block is determined based on the image low-level feature value LF. For example, if the quality enhancement unit is enabled, the quality enhancement unit performs enhancement on the image low-level feature value LF to obtain the reconstructed image block I corresponding to the current block, for instance, based on a decoding neural network. If the quality enhancement unit is not enabled, the image low-level feature value LF is used as the reconstructed image block I.

[0323] For example, steps S80 to S86 may be described by referring to Example 5, which will not be repeated here.

[0324] In one possible embodiment, the coefficient hyperparameter feature generation unit may perform an inverse transform operation on the coefficient hyperparameter feature coefficient reconstruction value C' using a decoding neural network to obtain the coefficient hyperparameter feature value P, and the control parameter coding unit may determine neural network information 1 corresponding to the coefficient hyperparameter feature generation unit of the decoding device based on the network structure of the decoding neural network, and the neural network information 1 is used to determine the decoding neural network 1 corresponding to the coefficient hyperparameter feature generation unit and encodes the neural network information 1 into bitstream 0.

[0325] In one possible embodiment, the image feature inverse transform unit may perform an inverse transform operation on the image feature reconstruction value F' using a decoding neural network to obtain the image low-level feature value LF, and the control parameter coding unit may determine neural network information 2 corresponding to the image feature inverse transform unit of the decoding device based on the network structure of the decoding neural network, the neural network information 2 is used to determine the decoding neural network 2 corresponding to the image feature inverse transform unit, and the neural network information 2 is encoded into bitstream 0.

[0326] In one possible embodiment, the enhancement unit may use a decoding neural network to enhance the low-level image feature values ​​LF to obtain a reconstructed image block I corresponding to the current block, and the control parameter coding unit may determine neural network information 3 corresponding to the enhancement unit of the decoding device based on the network structure of the decoding neural network, the neural network information 3 is used to determine the decoding neural network 3 corresponding to the enhancement unit and encodes the neural network information 3 into bitstream 0.

[0327] Example 15: An embodiment of the present invention provides an image coding method based on a neural network, which may also be applied to the coding side. Figure 7C is a schematic diagram of the coding side, which may include a control parameter coding unit, a feature transformation unit, a coefficient hyperparameter feature transformation unit, a first feature coding unit, a second feature coding unit, a first feature decoding unit, a second feature decoding unit, a coefficient hyperparameter feature generation unit, an image feature inverse transformation unit, and a quality enhancement unit.

[0328] For example, the neural network-based image coding method in this embodiment may include the following steps S91 to S99.

[0329] In step S91, a feature transformation is performed on the current block I to obtain the image feature value F corresponding to the current block I. For example, the feature transformation unit may perform a feature transformation on the current block I based on an encoded neural network to obtain the image feature value F corresponding to the current block I.

[0330] In step S92, a coefficient hyperparameter feature transformation is performed on the image feature value F to obtain the coefficient hyperparameter feature coefficient value C. For example, the coefficient hyperparameter feature transformation unit performs a coefficient hyperparameter feature transformation on the image feature value F based on the coding neural network to obtain the coefficient hyperparameter feature coefficient value C.

[0331] In step S93, the coefficient hyperparameter feature coefficient value C is encoded to obtain bitstream 1. For example, the first feature encoding unit may encode the coefficient hyperparameter feature coefficient value C into the bitstream corresponding to the current block to obtain bitstream 1.

[0332] In step S94, the coefficient hyperparameter feature coefficient reconstruction value C' is determined based on the coefficient hyperparameter feature coefficient value C. For example, the coefficient hyperparameter feature coefficient reconstruction value C' is the same as the coefficient hyperparameter feature coefficient value C, and an inverse transformation is performed on the coefficient hyperparameter feature coefficient reconstruction value C' to obtain the coefficient hyperparameter feature value P. For example, the coefficient hyperparameter feature generation unit obtains the coefficient hyperparameter feature value P by performing an inverse transformation on the coefficient hyperparameter feature coefficient reconstruction value C'.

[0333] In step S95, the image feature value F is encoded based on the coefficient hyperparameter feature value P to obtain bitstream 2. For example, the second feature encoding unit encodes the image feature value F into the bitstream corresponding to the current block to obtain bitstream 2.

[0334] In step S96, the control parameter coding unit obtains the control parameter corresponding to the current block, which may include neural network information, encodes the control parameter corresponding to the current block into a bitstream, and denotes the bitstream containing the control parameter as bitstream 0.

[0335] In step S97, the bitstream 1 corresponding to the current block is decoded to obtain the coefficient hyperparameter feature coefficient value C corresponding to the current block. For example, the first feature decoding unit may decode the bitstream 1 corresponding to the current block to obtain the coefficient hyperparameter feature coefficient value C corresponding to the current block.

[0336] Furthermore, the coefficient hyperparameter feature coefficient reconstruction value C' is determined based on the coefficient hyperparameter feature coefficient value C.

[0337] In step S98, an inverse transform is performed on the coefficient hyperparameter feature coefficient reconstruction value C' to obtain the coefficient hyperparameter feature value P. For example, the coefficient hyperparameter feature generation unit obtains the coefficient hyperparameter feature value P by performing an inverse transform on the coefficient hyperparameter feature coefficient reconstruction value C' based on the decoding neural network. The bitstream 2 corresponding to the current block is decoded to obtain the image feature value F corresponding to the current block, and the image feature value F is set as the image feature reconstruction value F'. For example, the second feature decoding unit decodes the bitstream 2 corresponding to the current block using the coefficient hyperparameter feature value P.

[0338] In step S99, an inverse transformation is performed on the image feature reconstruction value F' to obtain the image low-level feature value LF. For example, the image feature inverse transformation unit performs an inverse transformation on the image feature reconstruction value F' based on the decoding neural network to obtain the image low-level feature value LF. Based on the image low-level feature value LF, the reconstructed image block I corresponding to the current block is determined. For example, the quality enhancement unit performs an enhancement on the image low-level feature value LF based on the decoding neural network to obtain the reconstructed image block I corresponding to the current block.

[0339] Example 16: In Examples 14 and 15, the first feature encoding unit may encode coefficient hyperparameter feature information into bitstream 1 corresponding to the current block, and the encoding process of the first feature encoding unit corresponds to the decoding process of the first feature decoding unit, referring to Example 7, for example, the first feature encoding unit may encode coefficient hyperparameter feature information using an entropy encoding method (e.g., an entropy encoding method such as CAVLC or CABAC), which will not be repeated here. The first feature decoding unit may decode bitstream 1 corresponding to the current block to obtain coefficient hyperparameter feature information corresponding to the current block, and for this process, refer to Example 7.

[0340] In Examples 14 and 15, the second feature encoding unit may encode image feature information based on the coefficient hyperparameter feature value P to obtain bitstream 2. The encoding process of the second feature encoding unit corresponds to the decoding process of the second feature decoding unit, and referring to Example 10, for example, the second feature encoding unit may encode image feature information using an entropy encoding method (e.g., an entropy encoding method such as CAVLC or CABAC), which will not be repeated here. The second feature decoding unit may decode bitstream 2 corresponding to the current block to obtain image feature information corresponding to the current block, and for this process, referring to Example 10 may be used.

[0341] In Examples 14 and 15, the first quantization unit may perform quantization on the coefficient hyperparameter feature coefficient value C to obtain the coefficient hyperparameter feature coefficient quantization value C_q. The quantization process of the first quantization unit corresponds to the inverse quantization process of the first inverse quantization unit, referring to Example 8. For example, the first quantization unit performs quantization on the coefficient hyperparameter feature coefficient value C based on quantization-related parameters, and this is not repeated here. Regarding the quantization-related parameters (for example, the step parameter qstep, also called the quantization stride), 1) each feature value in each feature channel uses the same quantization stride, 2) each feature channel uses a different quantization stride, but each feature value within the feature channel uses the same quantization stride, and 3) each feature value in each feature channel uses a different quantization stride. The first inverse quantization unit on the encoding side may perform inverse quantization on the coefficient hyperparameter feature coefficient quantization value C_q to obtain the coefficient hyperparameter feature coefficient reconstruction value C', and the processing process of the first inverse quantization unit on the encoding side may refer to Example 10.

[0342] The second quantization unit may perform quantization on the image feature value F to obtain the image feature quantized value F_q. The quantization process of the second quantization unit corresponds to the inverse quantization process of the second inverse quantization unit, referring to Example 11. For example, the second quantization unit performs quantization on the image feature value F based on quantization-related parameters, which is not repeated here. Regarding the quantization stride, 1) each feature value in each feature channel uses the same quantization stride, 2) each feature channel uses a different quantization stride, but each feature value within the feature channel uses the same quantization stride, and 3) each feature value in each feature channel uses a different quantization stride. The second inverse quantization unit on the encoding side may perform inverse quantization on the image feature quantized value F_q to obtain the image feature reconstruction value F'. The processing process of the second inverse quantization unit on the encoding side may refer to Example 11.

[0343] Example 17: In Examples 14 and 15, the feature transformation unit may perform a feature transformation on the current block I based on the encoding neural network to obtain an image feature value F corresponding to the current block I. Based on this, the neural network information 2 corresponding to the image feature inverse transformation unit of the decoding device may be determined based on the network structure of the encoding neural network, and the neural network information 2 corresponding to the image feature inverse transformation unit may be encoded in bitstream 0. For a method by which the decoding device generates the decoding neural network 2 corresponding to the image feature inverse transformation unit based on the neural network information 2, refer to Example 12, which will not be repeated here.

[0344] In one possible embodiment, the feature transformation unit may include an encoding neural network 2, the encoding neural network 2 may include a base layer and an enhancement layer, the encoding neural network 2 may be divided into a base layer and an enhancement layer, the base layer may include at least one network layer, or the base layer may not include a network layer, i.e., the base layer may be empty. The enhancement layer may include at least one network layer, or the enhancement layer may not include a network layer, i.e., the enhancement layer may be empty. Furthermore, with respect to multiple network layers in the encoding neural network 2, multiple network layers may be divided into a base layer and an enhancement layer according to the actual needs. For example, a network layer with a fixed network structure may be used as the base layer, and a network layer with an unfixed network structure may be used as the enhancement layer.

[0345] For example, in coding neural network 2, the size of the output features may be smaller than the size of the input features, or the size of the output features may be equal to the size of the input features, or the size of the output features may be larger than the size of the input features.

[0346] For example, in coding neural network 2, the base layer and the reinforcement layer each include at least one convolutional layer. For instance, the base layer may include at least one convolutional layer, and the reinforcement layer may include at least one convolutional layer, or it may not include any convolutional layers. Alternatively, the reinforcement layer may include at least one convolutional layer, and the base layer may include at least one convolutional layer, or it may not include any convolutional layers.

[0347] In the coded neural network 2, the base layer and the reinforcement layer each include at least one residual structure layer. For example, the base layer includes at least one residual structure layer, and the reinforcement layer may or may not include a residual structure layer. Alternatively, the reinforcement layer includes at least one residual structure layer, and the base layer may or may not include a residual structure layer.

[0348] The coded neural network 2 may include, but is not limited to, convolutional layers and activation layers. For example, coded neural network 2 may include, in order, a convolutional layer with a stride of 2, one activation layer, a convolutional layer with a stride of 1, and one activation layer, with all of the above network layers forming the basic layers. Alternatively, coded neural network 2 may include, in order, a convolutional layer with a stride of 2, one convolutional layer with a stride of 1, and all of the above network layers forming the basic layers. Alternatively, coded neural network 2 may include, in order, a convolutional layer with a stride of 2, one activation layer, a convolutional layer with a stride of 2, one activation layer, a convolutional layer with a stride of 1, and one activation layer, with all of the above network layers forming the basic layers.

[0349] For example, the number of input channels in the first network layer of the feature transformation unit is either 1 or 3. If the input image block contains only one channel (e.g., a grayscale image), the number of input channels in the first network layer of the feature transformation unit is 1; if the input image block contains three channels (e.g., RGB or YUV format), the number of input channels in the first network layer of the feature transformation unit is 3.

[0350] Furthermore, the network structure of the encoding neural network 2 of the feature transformation unit and the network structure of the decoding neural network 2 of the image feature inverse transformation unit (see Example 12) may be symmetrical, and the network parameters of the encoding neural network 2 of the feature transformation unit and the network parameters of the decoding neural network 2 of the image feature inverse transformation unit may be the same or different.

[0351] In one possible embodiment, a network layer of a default network structure may be set for the feature transformation unit, and all network parameters relating to the network layer of the default network structure are fixed. For example, in the network layer of the default network structure, the number of deconvolutional layers is fixed, the number of activation layers is fixed, the number of channels in each deconvolutional layer is fixed, the size of the convolution kernel is fixed, and the filtering coefficients are fixed. Note that the network layer of the default network structure set for the feature transformation unit and the network layer of the default network structure set for the image feature inverse transformation unit (see Example 12) may be symmetrical.

[0352] In one possible embodiment, a pre-designed neural network pool (corresponding to the pre-designed neural network pool of the decoding device) may be set up for the feature transformation unit, and the pre-designed neural network pool includes at least one network layer of a pre-designed network structure, the network parameters of the network layers of the pre-designed network structure may all be set according to the actual needs. For example, the pre-designed neural network pool may include the network layer of pre-designed network structure t1', the network layer of pre-designed network structure t2', the network layer of pre-designed network structure t3', and so on. The network layer of pre-designed network structure t1' and the network layer of pre-designed network structure t1 (see Example 12) may be symmetrical. The network layer of pre-designed network structure t2' and the network layer of pre-designed network structure t2 may be symmetrical. The network layer of pre-designed network structure t3' and the network layer of pre-designed network structure t3 may be symmetrical.

[0353] In one possible embodiment, a network layer of a variable network structure may be dynamically generated for the feature transformation unit based on network parameters, where the network parameters for the network layer of the variable network structure are not pre-set but are dynamically generated by the encoding side.

[0354] In one possible embodiment, the encoding side may determine neural network information 2 corresponding to an image feature inverse transform unit based on the network structure of the encoding neural network 2, and encode the neural network information 2 corresponding to the image feature inverse transform unit into bitstream 0. For example, the encoding side may determine the neural network information 2 using the following case.

[0355] Case 1: When the basic layer of the encoded neural network 2 uses a network layer with a default network structure, and the enhancement layer also uses a network layer with a default network structure, the encoding side encodes the basic layer information and the enhancement layer information into bitstream 0, and the basic layer information and the enhancement layer information constitute the neural network information 2 corresponding to the image feature inverse transform unit. The basic layer information includes a basic layer default network use flag bit, and the basic layer default network use flag bit indicates that the basic layer uses the default network. The enhancement layer information includes an enhancement layer default network use flag bit, and the enhancement layer default network use flag bit indicates that the enhancement layer uses the default network.

[0356] Case 2: When the basic layer of the coding neural network 2 uses a network layer with a default network structure, and the enhancement layer uses a network layer with a pre-designed network structure, the coding side encodes the basic layer information and the enhancement layer information into bitstream 0, and the basic layer information and the enhancement layer information constitute the neural network information 2 corresponding to the image feature inverse transform unit. The basic layer information includes a basic layer default network use flag bit, and the basic layer default network use flag bit indicates that the basic layer uses the default network. The enhancement layer information includes an enhancement layer pre-designed network use flag bit and an enhancement layer pre-designed network index number, and the enhancement layer pre-designed network use flag bit indicates that the enhancement layer uses a pre-designed network, and the enhancement layer pre-designed network index number indicates the corresponding index in the pre-designed neural network pool for the network layer of the pre-designed network structure. For example, when the enhancement layer of the coding neural network 2 uses a network layer with a pre-designed network structure t1', the enhancement layer pre-designed network index number indicates the network layer of the first pre-designed network structure t1' in the pre-designed neural network pool.

[0357] Case 3: When the basic layer of the encoded neural network 2 uses a network layer with a default network structure, and the enhancement layer uses an enhancement layer with a variable network structure, the encoding side encodes the basic layer information and the enhancement layer information into bitstream 0, and the basic layer information and the enhancement layer information constitute the neural network information 2 corresponding to the image feature inverse transform unit. The basic layer information includes a basic layer default network use flag bit, and the basic layer default network use flag bit indicates that the basic layer uses the default network. The enhancement layer information includes network parameters for generating the enhancement layer, and these network parameters may include, but are not limited to, at least one of the following: the number of layers in the neural network, the deconvolutional layer flag bit, the number of deconvolutional layers, the quantization stride of each deconvolutional layer, the number of channels of each deconvolutional layer, the size of the convolution kernel, the number of filters, the filtering size index, the filtering coefficient zero flag bit, the filtering coefficient, the activation layer flag bit, and the activation layer type. Furthermore, the network parameters in the reinforcement layer information may differ from the network parameters used in the reinforcement layer of the coded neural network 2. That is, for each network parameter used in the reinforcement layer, a network parameter symmetric to that network parameter may be generated. The generation process is not limited, and these symmetric network parameters are transmitted to the decoding side as reinforcement layer information.

[0358] Case 4: When the basic layer of the encoded neural network 2 uses a network layer with a pre-designed network structure and the enhancement layer uses a network layer with a default network structure, the basic layer information and enhancement layer information are encoded into bitstream 0, and the basic layer information and enhancement layer information constitute neural network information 2 corresponding to the image feature inverse transform unit. The basic layer information includes a basic layer pre-designed network use flag bit and a basic layer pre-designed network index number, the basic layer pre-designed network use flag bit indicating that the basic layer uses a pre-designed network, and the basic layer pre-designed network index number indicating the corresponding index in the pre-designed neural network pool of the network layer with the pre-designed network structure. The enhancement layer information includes an enhancement layer default network use flag bit, the enhancement layer default network use flag bit indicating that the enhancement layer uses a default network.

[0359] Case 5: When the basic layer of the encoded neural network 2 uses a network layer with a pre-designed network structure, and the enhancement layer also uses a network layer with a pre-designed network structure, the encoding side encodes the basic layer information and the enhancement layer information into bitstream 0, and the basic layer information and the enhancement layer information constitute the neural network information 2 corresponding to the image feature inverse transform unit. The basic layer information may include a basic layer pre-designed network use flag bit and a basic layer pre-designed network index number, the basic layer pre-designed network use flag bit is used to indicate that the basic layer uses a pre-designed network, and the basic layer pre-designed network index number is used to indicate the corresponding index in the pre-designed neural network pool of the network layer with the pre-designed network structure. The enhancement layer information may also include an enhancement layer pre-designed network use flag bit and an enhancement layer pre-designed network index number, the enhancement layer pre-designed network use flag bit is used to indicate that the enhancement layer uses a pre-designed network, and the enhancement layer pre-designed network index number is used to indicate the corresponding index in the pre-designed neural network pool of the network layer with the pre-designed network structure.

[0360] Case 6: When the basic layer of the encoded neural network 2 uses a network layer with a pre-designed network structure and the enhancement layer uses an enhancement layer with a variable network structure, the basic layer information and enhancement layer information are encoded into bitstream 0, and the basic layer information and enhancement layer information constitute neural network information 2 corresponding to the image feature inverse transform unit. The basic layer information includes a basic layer pre-designed network use flag bit and a basic layer pre-designed network index number, the basic layer pre-designed network use flag bit indicating that the basic layer uses a pre-designed network, and the basic layer pre-designed network index number indicating the corresponding index in the pre-designed neural network pool for the network layer of the pre-designed network structure. The enhancement layer information includes network parameters for generating the enhancement layer, which may include, but are not limited to, the number of layers in the neural network, the deconvolutional layer flag bit, the number of deconvolutional layers, the quantization stride of each deconvolutional layer, the number of channels of each deconvolutional layer, the size of the convolutional kernel, the number of filters, the filtering size index, the filtering coefficient zero flag bit, the filtering coefficient, the activation layer flag bit, and the activation layer type. The network parameters in the reinforcement layer information may differ from the network parameters used in the reinforcement layer of the coded neural network 2. That is, for each network parameter used in the reinforcement layer, a network parameter symmetric to that network parameter may be generated. The generation process is not limited, and these symmetric network parameters are transmitted to the decoding side as reinforcement layer information.

[0361] Example 18: In Examples 14 and 15, the encoding-side image feature inverse transform unit may perform an inverse transform operation on the image feature reconstruction value F' based on the decoding neural network to obtain the image low-level feature value LF. Based on this, neural network information 2 corresponding to the image feature inverse transform unit of the decoding device may be determined based on the network structure of the decoding neural network, and the neural network information 2 corresponding to the image feature inverse transform unit may be encoded into bitstream 0.

[0362] In one possible embodiment, the encoding-side image feature inverse transform unit may include a decoding neural network 2, the decoding neural network 2 may include a base layer and an enhancement layer, and the network structure of the encoding-side decoding neural network 2 is the same as the network structure of the decoding-side decoding neural network 2, and can be referred to Example 12, which is omitted here.

[0363] In one possible embodiment, the encoding side may configure a network layer of a default network structure for the inverse image feature transform unit, which is the same as the network layer of the default network structure on the decoding side. The encoding side may configure a pre-designed neural network pool for the inverse image feature transform unit, which may include at least one network layer of a pre-designed network structure, which is the same as the pre-designed neural network pool on the decoding side. The encoding side may dynamically generate a network layer of a variable network structure for the inverse image feature transform unit based on network parameters. The encoding side may determine neural network information 2 corresponding to the inverse image feature transform unit based on the network structure of the decoding neural network 2 and encode the neural network information 2 into bitstream 0.

[0364] For example, if the base layer of decoding neural network 2 uses a network layer with a default network structure and the reinforcement layer uses a network layer with a default network structure, the base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network. The reinforcement layer information includes a reinforcement layer default network use flag bit, and the reinforcement layer default network use flag bit indicates that the reinforcement layer uses the default network. Also, for example, if the base layer of decoding neural network 2 uses a network layer with a default network structure and the reinforcement layer uses a network layer with a pre-designed network structure, the base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network. The reinforcement layer information includes a reinforcement layer pre-designed network use flag bit and a reinforcement layer pre-designed network index number, and the reinforcement layer pre-designed network use flag bit indicates that the reinforcement layer uses a pre-designed network, and the reinforcement layer pre-designed network index number indicates the corresponding index in the pre-designed neural network pool of the network layer with the pre-designed network structure. Furthermore, for example, if the base layer of decoding neural network 2 uses a network layer with a default network structure, and the reinforcement layer uses a reinforcement layer with a variable network structure, the base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network. The reinforcement layer information includes network parameters for generating the reinforcement layer, and the network parameters in the reinforcement layer information may be the same as the network parameters used in the reinforcement layer of decoding neural network 2.

[0365] For example, if the base layer of decoding neural network 2 uses a network layer with a pre-designed network structure and the reinforcement layer uses a network layer with a default network structure, the base layer information includes a base layer pre-designed network use flag bit and a base layer pre-designed network index number. The base layer pre-designed network use flag bit indicates that the base layer uses a pre-designed network, and the base layer pre-designed network index number indicates the corresponding index in the pre-designed neural network pool for the network layer with the pre-designed network structure. The reinforcement layer information includes an reinforcement layer default network use flag bit. The reinforcement layer default network use flag bit indicates that the reinforcement layer uses the default network. Also, for example, if the base layer of decoding neural network 2 uses a network layer with a pre-designed network structure and the reinforcement layer uses a network layer with a pre-designed network structure, the base layer information includes a base layer pre-designed network use flag bit and a base layer pre-designed network index number. The base layer pre-designed network use flag bit indicates that the base layer uses a pre-designed network, and the base layer pre-designed network index number indicates the corresponding index in the pre-designed neural network pool for the network layer with the pre-designed network structure. The reinforcement layer information includes a reinforcement layer predesign network use flag bit and a reinforcement layer predesign network index number, the reinforcement layer predesign network use flag bit indicating that the reinforcement layer uses a predesign network, and the reinforcement layer predesign network index number indicating the corresponding index in the predesign neural network pool of the network layer of the predesign network structure.Furthermore, for example, if the base layer of decoding neural network 2 uses a network layer with a pre-designed network structure and the reinforcement layer uses a reinforcement layer with a variable network structure, the base layer information includes a base layer pre-designed network use flag bit and a base layer pre-designed network index number. The base layer pre-designed network use flag bit indicates that the base layer uses a pre-designed network, and the base layer pre-designed network index number indicates the corresponding index in the pre-designed neural network pool for the network layer with the pre-designed network structure. The reinforcement layer information includes network parameters, and the network parameters in the reinforcement layer information are the same as the network parameters used in the reinforcement layer of decoding neural network 2.

[0366] Example 19: In Examples 14 and 15, the coefficient hyperparameter feature transformation unit may perform a coefficient hyperparameter feature transformation on the image feature value F based on the encoded neural network to obtain a coefficient hyperparameter feature coefficient value C. Based on this, the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit of the decoding device may be determined based on the network structure of the encoded neural network, and the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit may be encoded in bitstream 0. For a method by which the decoding device generates the decoding neural network 1 corresponding to the coefficient hyperparameter feature generation unit based on the neural network information 1, refer to Example 9, which will not be repeated here.

[0367] In one possible embodiment, the coefficient hyperparameter feature transformation unit may include an encoding neural network 1, the encoding neural network 1 may include a base layer and an enhancement layer, the encoding neural network 1 may be divided into a base layer and an enhancement layer, the base layer may include at least one network layer, or the base layer may not include a network layer, i.e., the base layer may be empty. The enhancement layer may include at least one network layer, or the enhancement layer may not include a network layer, i.e., the enhancement layer may be empty. The network structure of the encoding neural network 1 of the coefficient hyperparameter feature transformation unit and the network structure of the decoding neural network 1 of the coefficient hyperparameter feature generation unit (see Example 9) may be symmetrical, and the network parameters of the encoding neural network 1 of the coefficient hyperparameter feature transformation unit and the network parameters of the decoding neural network 1 of the coefficient hyperparameter feature generation unit may be the same or different, and the network structure of this encoding neural network 1 will not be described further.

[0368] In one possible embodiment, a network layer of a default network structure may be set for the coefficient hyperparameter feature transformation unit, and all network parameters for the network layer of the default network structure are fixed. The network layer of the default network structure set for the coefficient hyperparameter feature transformation unit and the network layer of the default network structure set for the coefficient hyperparameter feature generation unit may be symmetrical. A pre-designed neural network pool (corresponding to the pre-designed neural network pool of the decoding device) may be set for the coefficient hyperparameter feature transformation unit, and the pre-designed neural network pool includes at least one network layer of a pre-designed network structure, and all network parameters for the network layer of the pre-designed network structure may be set according to actual needs. Based on the network parameters, a network layer of a variable network structure may be dynamically generated for the coefficient hyperparameter feature transformation unit, and the network parameters for the network layer of the variable network structure are dynamically generated by the encoding side.

[0369] In one possible embodiment, the encoding side may determine neural network information 1 corresponding to a coefficient hyperparameter feature generation unit based on the network structure of the encoded neural network 1, and encode the neural network information 1 into bitstream 0. Here, the neural network information 1 may include basic layer information and reinforcement layer information, and the method for encoding the neural network information 1 is similar to the method for encoding the neural network information 2, and can be found in Example 17, which will not be repeated here. For example, if the basic layer of the encoded neural network 1 uses a network layer with a default network structure, and the reinforcement layer uses a network layer with a default network structure, the basic layer information includes a basic layer default network use flag bit, and the basic layer default network use flag bit indicates that the basic layer uses the default network. The reinforcement layer information includes a reinforcement layer default network use flag bit, and the reinforcement layer default network use flag bit indicates that the reinforcement layer uses the default network.

[0370] Example 20: In Examples 14 and 15, the encoding-side coefficient hyperparameter feature generation unit may perform an inverse transform on the coefficient hyperparameter feature coefficient reconstruction value C' based on the decoding neural network to obtain the coefficient hyperparameter feature value P. Based on this, the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit of the decoding device may be determined based on the network structure of the decoding neural network, and the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit may be encoded into bitstream 0.

[0371] In one possible embodiment, the encoding-side coefficient hyperparameter feature generation unit may include a decoding neural network 1, the decoding neural network 1 may include a base layer and an enhancement layer, and the network structure of the encoding-side decoding neural network 1 is the same as the network structure of the decoding-side decoding neural network 1, and refer to Example 9, which will be omitted from this description.

[0372] In one possible embodiment, the encoding side may configure a network layer of a default network structure for the coefficient hyperparameter feature generation unit, and this network layer of the default network structure is the same as the network layer of the default network structure on the decoding side. The encoding side may configure a pre-designed neural network pool for the coefficient hyperparameter feature generation unit, and this pre-designed neural network pool may include at least one network layer of a pre-designed network structure, and this pre-designed neural network pool is the same as the pre-designed neural network pool on the decoding side. The encoding side may dynamically generate a network layer of a variable network structure for the coefficient hyperparameter feature generation unit based on network parameters. The encoding side may determine neural network information 1 corresponding to the coefficient hyperparameter feature generation unit based on the network structure of the decoding neural network 1 and encode the neural network information 1 into bitstream 0. Here, the neural network information 1 may include basic layer information and reinforcement layer information, and the method for encoding the neural network information 1 is similar to the method for encoding the neural network information 2, and can be found in Example 18, which will not be repeated here. For example, if the base layer of decoding neural network 1 (used on the encoding side) uses a network layer with a default network structure, and the reinforcement layer uses a reinforcement layer with a variable network structure, then the base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network. The reinforcement layer information includes network parameters for generating the reinforcement layer, and the network parameters in the reinforcement layer information may be the same as the network parameters used in the reinforcement layer of decoding neural network 1 (i.e., the reinforcement layer used on the encoding side).

[0373] Example 21: In Examples 14 and 15, the encoding-side quality enhancement unit may perform enhancement processing on the low-level image feature value LF based on the decoding neural network to obtain a reconstructed image block I corresponding to the current block. Based on this, neural network information 3 corresponding to the quality enhancement unit of the decoding device may be determined based on the network structure of the decoding neural network, and the neural network information 3 corresponding to the quality enhancement unit may be encoded into bitstream 0. For example, the encoding-side quality enhancement unit may include a decoding neural network 3, and the decoding neural network 3 may include a basic layer and an enhancement layer, and the network structure of the encoding-side decoding neural network 3 is the same as the network structure of the decoding-side decoding neural network 3, and Example 13 may be referred to, but the explanation is omitted here.

[0374] Exemplary, the encoding side may set up a network layer of a default network structure for the quality enhancement unit, and this network layer of the default network structure is the same as the network layer of the default network structure on the decoding side. The encoding side may set up a pre-designed neural network pool for the quality enhancement unit, and this pre-designed neural network pool may include at least one network layer of a pre-designed network structure, and this pre-designed neural network pool is the same as the pre-designed neural network pool on the decoding side. The encoding side may dynamically generate a network layer of a variable network structure for the quality enhancement unit based on network parameters. The encoding side determines the neural network information 3 corresponding to the quality enhancement unit based on the network structure of the decoding neural network 3 and encodes the neural network information 3 into bitstream 0. Here, the neural network information 3 may include basic layer information and enhancement layer information, and the method for encoding the neural network information 3 is similar to the method for encoding the neural network information 2, and can be found in Example 18, which will not be repeated here. For example, if the base layer of decoding neural network 3 uses a network layer with a default network structure, and the reinforcement layer uses a reinforcement layer with a variable network structure, the base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network. The reinforcement layer information includes network parameters for generating the reinforcement layer, and the network parameters in the reinforcement layer information may be the same as the network parameters used in the reinforcement layer of decoding neural network 3.

[0375] In one possible embodiment, the network parameters in Examples 1 to 21 may all be fixed-point network parameters, for example, the filtering weights in the network parameters may be represented with widths of 4, 8, 16, 32, and 64 bits, and the feature values ​​output from the network may be limited by bit width, for example, to be represented with widths of 4, 8, 16, 32, and 64 bits.

[0376] For example, the bit width of the network parameters may be limited to 8 bits, and the magnitude of their values ​​may be limited to [-127, 127], or the bit width of the feature values ​​output from the network may be limited to 8 bits, and the magnitude of their values ​​may be limited to [-127, 127].

[0377] Example 22: Regarding the related syntax table for the image header, Table 4 provides syntax information for the image header, i.e., image-level syntax. In Table 4, u(n) is used to represent an n-bit fixed-length encoding method. [Table 4]

[0378] In Table 4, pic_width represents the width of the image, pic_height represents the height of the image, and pic_format represents the image format, such as RGB444, YUV444, YUV420, or YUV422. bu_width represents the width of the base block, bu_height represents the height of the base block, block_width represents the width of the image block, block_height represents the height of the image block, bit_depth represents the image bit depth, pic_qp represents the quantization parameter in the current image, lossless_flag represents a flag indicating whether or not to apply lossless coding marks to the current image, feature_map_max_bit_depth represents the maximum bit depth of the feature map, which is used to limit the maximum and minimum values ​​of the network's input or output feature maps.

[0379] For example, each of the above embodiments may be implemented individually or in combination. For instance, each of the embodiments from Embodiments 1 to 22 may be implemented individually, or at least two of the embodiments from Embodiments 1 to 22 may be implemented in combination.

[0380] For example, in each of the above embodiments, the contents of the encoding side may be applied to the decoding side, that is, processed by the decoding side in the same manner, and the contents of the decoding side may be applied to the encoding side, that is, processed by the encoding side in the same manner.

[0381] Based on the same concept as described above, embodiments of the present invention further provide a neural network-based image decoding device, which is applied to the decoding side and includes a memory configured to store video data, and a decoder configured to perform the decoding method in embodiments 1 to 22, i.e., the decoding side processing process.

[0382] For example, in one possible embodiment, the decoder is The steps include decoding control parameters and image information corresponding to the current block from the bitstream, The steps include obtaining neural network information corresponding to the decoding processing unit from the control parameters, and generating a decoding neural network corresponding to the decoding processing unit based on the neural network information, The system is configured to perform the steps of determining input features corresponding to the decoding processing unit based on the image information, and processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit.

[0383] Based on the same concept as described above, embodiments of the present invention further provide a neural network-based image encoding device, which is applied to the encoding side and includes a memory configured to store video data, and an encoder configured to perform the encoding method in embodiments 1 to 22, i.e., the encoding side processing process.

[0384] For example, in one possible embodiment, the encoder is The steps include: determining input features corresponding to an encoding processing unit based on the current block; processing the input features based on an encoding neural network corresponding to the encoding processing unit to obtain output features corresponding to the encoding processing unit; and determining image information corresponding to the current block based on the output features. A step of obtaining control parameters corresponding to the current block, wherein the control parameters include neural network information corresponding to a decoding processing unit, and the neural network information is used to determine the decoding neural network corresponding to the decoding processing unit. The system is configured to perform the steps of encoding image information and control parameters corresponding to the current block into a bitstream.

[0385] Based on the same concept as described above, a schematic hardware architecture diagram of a decoding device (also called a video decoder) provided by an embodiment of the present invention may be shown in Figure 8A, specifically. The device includes a processor 811 and a machine-readable storage medium 812, the machine-readable storage medium 812 which stores machine-executable instructions that can be executed by the processor 811, and the processor 811 is used to execute the machine-executable instructions and carry out the decoding methods of embodiments 1 to 22 of the present invention.

[0386] Based on the same concept as described above, a schematic hardware architecture diagram of an encoding device (also called a video encoder) provided by an embodiment of the present invention may be shown in Figure 8B from a hardware perspective. The device includes a processor 821 and a machine-readable storage medium 822, the machine-readable storage medium 822 which stores machine-executable instructions that can be executed by the processor 821, and the processor 821 is used to execute the machine-executable instructions and carry out the encoding methods of embodiments 1 to 22 of the present invention.

[0387] Based on the same concept as described above, embodiments of the present invention further provide a machine-readable storage medium in which several computer instructions are stored, and when the computer instructions are executed by a processor, the methods disclosed in the above examples of the present invention, such as the decoding method or encoding method in each of the above embodiments, can be performed.

[0388] Based on the same concept as described above, embodiments of the present invention further provide a computer application program, and when the computer application program is executed by a processor, the decoding or encoding method disclosed in the above examples of the present invention can be performed.

[0389] Based on the same concept as described above, an embodiment of the present invention further provides a neural network-based image decoding device, which is applied to the decoding side, and includes: a decoding module for decoding control parameters and image information corresponding to the current block from a bitstream; an acquisition module for acquiring neural network information corresponding to a decoding processing unit from the control parameters and generating a decoding neural network corresponding to the decoding processing unit based on the neural network information; and a processing module for determining input features corresponding to the decoding processing unit based on the image information and processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit.

[0390] For example, if the neural network information includes basic layer information and reinforcement layer information, the acquisition module is used to generate a decoding neural network corresponding to the decoding processing unit based on the neural network information. Specifically, it is used to determine the basic layer corresponding to the decoding processing unit based on the basic layer information, determine the reinforcement layer corresponding to the decoding processing unit based on the reinforcement layer information, and generate a decoding neural network corresponding to the decoding processing unit based on the basic layer and the reinforcement layer.

[0391] Exemplary, when the acquisition module determines the base layer corresponding to the decoding unit based on the base layer information, specifically, if the base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network, the acquisition module is used to acquire the base layer of the default network structure.

[0392] Exemplary, when the acquisition module determines the base layer corresponding to the decoding unit based on the base layer information, specifically, the base layer information includes a base layer predesign network use flag bit and a base layer predesign network index number, and the base layer predesign network use flag bit indicates that the base layer uses a predesign network, it is used to select the base layer of the predesign network structure corresponding to the base layer predesign network index number from the predesign neural network pool, the predesign neural network pool includes a network layer of at least one predesign network structure.

[0393] Exemplary, when the acquisition module determines the enhancement layer corresponding to the decoding unit based on the enhancement layer information, specifically, if the enhancement layer information includes an enhancement layer default network use flag bit, and the enhancement layer default network use flag bit indicates that the enhancement layer uses the default network, the acquisition module is used to acquire the enhancement layer of the default network structure.

[0394] Exemplary, when the acquisition module determines the enhancement layer corresponding to the decoding unit based on the enhancement layer information, specifically, the enhancement layer information includes an enhancement layer predesign network use flag bit and an enhancement layer predesign network index number, and the enhancement layer predesign network use flag bit indicates that the enhancement layer uses a predesign network, it is used to select the enhancement layer of the predesign network structure corresponding to the enhancement layer predesign network index number from the predesign neural network pool, the predesign neural network pool includes a network layer of at least one predesign network structure.

[0395] Exemplary, when the acquisition module determines an enhancement layer corresponding to the decoding unit based on the enhancement layer information, specifically, if the enhancement layer information includes network parameters for generating the enhancement layer, it is used to generate the enhancement layer corresponding to the decoding unit based on the network parameters, the network parameters including at least one of the following: the number of layers in the neural network, the deconvolutional layer flag bit, the number of deconvolutional layers, the quantization stride of each deconvolutional layer, the number of channels of each deconvolutional layer, the size of the convolutional kernel, the number of filters, the filtering size index, the filtering coefficient zero flag bit, the filtering coefficient, the activation layer flag bit, and the activation layer type.

[0396] Exemplary, the image information includes coefficient hyperparameter feature information and image feature information. The processing module determines input features corresponding to the decoding processing unit based on the image information and processes the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit. Specifically, when executing a decoding process for coefficient hyperparameter feature generation, it determines coefficient hyperparameter feature coefficient reconstruction values ​​based on the coefficient hyperparameter feature information and performs an inverse transform operation on the coefficient hyperparameter feature coefficient reconstruction values ​​based on the decoding neural network to obtain coefficient hyperparameter feature values. When executing a decoding process for image feature inverse transform, it determines image feature reconstruction values ​​based on the image feature information and performs an inverse transform operation on the image feature reconstruction values ​​based on the decoding neural network to obtain image low-level feature values. The coefficient hyperparameter feature values ​​are used to decode the image feature information from the bitstream, and the image low-level feature values ​​are used to obtain the reconstructed image block corresponding to the current block.

[0397] For example, when the processing module determines a coefficient hyperparameter feature coefficient reconstruction value based on the coefficient hyperparameter feature information, specifically when the control parameter includes first enable information and the first enable information indicates that a first inverse quantization process is enabled, it is used to perform inverse quantization on the coefficient hyperparameter feature information to obtain a coefficient hyperparameter feature coefficient reconstruction value.

[0398] For example, when the processing module determines an image feature reconstruction value based on the image feature information, specifically when the control parameter includes a second enable information, and the second enable information indicates that a second inverse quantization process is enabled, it is used to perform inverse quantization on the image feature information to obtain an image feature reconstruction value.

[0399] For example, the processing module is further used to perform a high-quality decoding process if the control parameter includes a third enable information, and the third enable information indicates that the high-quality processing should be enabled, by acquiring the low-level image feature values ​​and performing an enhancement process on the low-level image feature values ​​based on the decoding neural network to obtain a reconstructed image block corresponding to the current block.

[0400] Based on the same concept as described above, an embodiment of the present invention further provides a neural network-based image coding apparatus, which is applied to the coding side, and includes a processing module for determining input features corresponding to a coding processing unit based on the current block, processing the input features based on a coding neural network corresponding to the coding processing unit to obtain output features corresponding to the coding processing unit, and determining image information corresponding to the current block based on the output features; an acquisition module for obtaining control parameters corresponding to the current block, wherein the control parameters include neural network information corresponding to a decoding processing unit, and the neural network information is used to determine the decoding neural network corresponding to the decoding processing unit; and an coding module for encoding the image information and control parameters corresponding to the current block into a bitstream.

[0401] For example, the neural network information includes basic layer information and reinforcement layer information, and the decoding neural network includes a basic layer determined based on the basic layer information and a reinforcement layer determined based on the reinforcement layer information.

[0402] For example, if the base layer information includes a base layer default network use flag bit, and the base layer default network use flag bit indicates that the base layer uses the default network, then the decoding neural network uses the base layer of the default network structure.

[0403] Exemplary, if the base layer information includes a base layer predesign network use flag bit and a base layer predesign network index number, and the base layer predesign network use flag bit indicates that the base layer uses a predesign network, then the decoding neural network uses a base layer of a predesign network structure corresponding to the base layer predesign network index number, selected from a predesign neural network pool, and the predesign neural network pool includes a network layer of at least one predesign network structure.

[0404] For example, if the reinforcement layer information includes a reinforcement layer default network use flag bit, and the reinforcement layer default network use flag bit indicates that the reinforcement layer uses the default network, then the decoding neural network uses the reinforcement layer of the default network structure.

[0405] Exemplary, if the enhancement layer information includes an enhancement layer predesign network use flag bit and an enhancement layer predesign network index number, and the enhancement layer predesign network use flag bit indicates that the enhancement layer uses a predesign network, then the decoding neural network uses an enhancement layer of a predesign network structure corresponding to the enhancement layer predesign network index number, selected from a predesign neural network pool, and the predesign neural network pool includes a network layer of at least one predesign network structure.

[0406] For example, if the reinforcement layer information includes network parameters for generating the reinforcement layer, the decoding neural network uses the reinforcement layer generated based on the network parameters, the network parameters include at least one of the following: the number of layers in the neural network, the deconvolutional layer flag bit, the number of deconvolutional layers, the quantization stride of each deconvolutional layer, the number of channels of each deconvolutional layer, the size of the convolutional kernel, the number of filters, the filtering size index, the filtering coefficient zero flag bit, the filtering coefficient, the activation layer flag bit, and the activation layer type.

[0407] For example, the processing module further divides the current image into N non-overlapping image blocks, where N is a positive integer, and performs boundary padding on each image block to obtain a boundary-padding image block. When performing boundary padding on each image block, the padding value does not depend on the reconstructed pixel values ​​of adjacent image blocks and is used to generate N current blocks based on the boundary-padding image block.

[0408] Exemplary, the processing module further divides the current image into a plurality of basic blocks, each basic block containing at least one image block, and performs boundary padding on each image block to obtain a boundary-padding image block, wherein when performing boundary padding on each image block, the padding value of the image block is permitted to depend on the reconstructed pixel values ​​of image blocks in different basic blocks, and not on the reconstructed pixel values ​​of other image blocks within the same basic block, and is used to generate a plurality of current blocks based on the boundary-padding image blocks.

[0409] Those skilled in the art will understand that embodiments of the present invention may be provided as methods, systems, or computer program products. The present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Embodiments of the present invention may take the form of computer program products implemented on one or more computer-compatible storage media (including, but not limited to, magnetic disk memory, CD-ROM, optical memory, etc.) containing computer-compatible program code.

[0410] The above are merely embodiments of the present invention and do not limit the present invention. Those skilled in the art will know that the present invention can be modified and altered in various ways. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention shall be included within the scope of the claims of the present invention. [Explanation of Symbols]

[0411] 811 Processor 812 Machine-readable storage medium 821 Processor 822 Machine-readable storage medium

Claims

1. The steps include decoding the control parameters corresponding to the current block from the bitstream, The steps include obtaining neural network information corresponding to the decoding processing unit from the control parameters, and generating a decoding neural network corresponding to the decoding processing unit based on the neural network information, The steps include determining the input features corresponding to the decoding processing unit, and processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit, If the neural network information includes basic layer information, the step of generating a decoding neural network corresponding to the decoding processing unit based on the neural network information is: The steps include determining the basic layer corresponding to the decoding processing unit based on the basic layer information, The step of generating a decoding neural network corresponding to the decoding processing unit based on the basic layer, The step of determining the basic layer corresponding to the decoding processing unit based on the basic layer information is: The basic layer information includes a basic layer default network use flag bit, and if the basic layer default network use flag bit indicates that the basic layer uses the default network, the step includes obtaining the basic layer of the default network structure. A neural network-based image decoding method characterized by the following.

2. The step of determining the basic layer corresponding to the decoding processing unit based on the basic layer information further includes: The basic layer information includes a basic layer predesign network use flag bit and a basic layer predesign network index number, and if the basic layer predesign network use flag bit indicates that the basic layer uses a predesign network, the step includes selecting a basic layer of the predesign network structure corresponding to the basic layer predesign network index number from the predesign neural network pool. The aforementioned pre-designed neural network pool includes a network layer of at least one pre-designed network structure. The method according to feature 1.

3. If the basic layer is a first feature decoding network, the step of processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit is: The process includes the steps of decoding a first bitstream of the current block that is input based on the first feature decoding network and obtaining coefficient hyperparameter feature information corresponding to the current block that is output, The method according to 1 or 2, characterized by the above.

4. If the basic layer is a coefficient hyperparameter feature generation network, the step of processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit is: The step of processing input coefficient hyperparameter feature coefficient reconstruction values ​​based on the coefficient hyperparameter feature generation network and obtaining output coefficient hyperparameter feature values ​​corresponding to the current block, wherein the coefficient hyperparameter feature coefficient reconstruction values ​​input to the coefficient hyperparameter feature generation network are determined based on coefficient hyperparameter feature information output by the first feature decoding network. The method according to feature 3.

5. The step of determining the coefficient hyperparameter feature coefficient reconstruction value based on the coefficient hyperparameter feature information is: If the control parameter includes first enable information, and the first enable information indicates that a first inverse quantization process is enabled, the process includes the step of performing inverse quantization on the coefficient hyperparameter feature information to obtain a coefficient hyperparameter feature coefficient reconstruction value. The method according to feature 4.

6. If the basic layer is a second feature decoding network, the step of processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit is: The process includes the steps of decoding the second bitstream of the current block that is input based on the second feature decoding network and obtaining image feature information corresponding to the current block that is output, The method according to 1 or 2, characterized by the above.

7. The step of decoding the second bitstream of the current block that is input based on the second feature decoding network and obtaining image feature information corresponding to the current block that is output is: The process includes the step of decoding the second bitstream based on the coefficient hyperparameter feature values ​​input to the second feature decoding network to obtain the image feature information, wherein the coefficient hyperparameter feature values ​​are output features of the coefficient hyperparameter feature generation network. The method according to feature 6.

8. If the basic layer is a transformation network, the step of processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit is: The step includes processing the input image feature reconstruction values ​​based on the transformation network and obtaining the output low-level image feature values ​​corresponding to the current block, The image feature reconstruction value input to the transformation network is determined based on the image feature information output by the second feature decoding network. The aforementioned low-level image feature values ​​are used to obtain the reconstructed image block corresponding to the current block. The method according to feature 6.

9. The step of determining the image feature reconstruction value based on the aforementioned image feature information is: If the control parameter includes a second enable information, and the second enable information indicates that a second inverse quantization process is enabled, the process includes the step of performing inverse quantization on the image feature information to obtain an image feature reconstruction value. The method according to feature 8.

10. If the neural network information includes reinforcement layer information, the step of generating a decoding neural network corresponding to the decoding processing unit based on the neural network information is: The steps include determining the reinforcement layer corresponding to the decoding processing unit based on the reinforcement layer information, The step of generating a decoding neural network corresponding to the decoding processing unit based on the reinforcement layer, The method according to feature 1.

11. The step of determining the reinforcement layer corresponding to the decoding processing unit based on the reinforcement layer information is: The reinforcement layer information includes a reinforcement layer default network use flag bit, and if the reinforcement layer default network use flag bit indicates that the reinforcement layer uses the default network, the step includes obtaining the reinforcement layer of the default network structure. The method according to the present invention, characterized by the present invention.

12. The step of determining the reinforcement layer corresponding to the decoding processing unit based on the reinforcement layer information is: The enhancement layer information includes an enhancement layer predesign network use flag bit and an enhancement layer predesign network index number, and if the enhancement layer predesign network use flag bit indicates that the enhancement layer uses a predesign network, the step includes selecting an enhancement layer with a predesign network structure corresponding to the enhancement layer predesign network index number from the predesign neural network pool. The aforementioned pre-designed neural network pool includes a network layer of at least one pre-designed network structure. The method according to the present invention, characterized by the present invention.

13. If the enhancement layer is a transformation network, the step of determining the input features corresponding to the decoding processing unit is: The steps include decoding the bitstream of the current block to obtain image feature information corresponding to the current block, The step of determining an image feature reconstruction value corresponding to the current block based on the image feature information is included. The method according to 11 or 12, characterized by the features described above.

14. The step of processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit is: The process includes processing the input image feature reconstruction values ​​based on the transformation network and obtaining output image low-level feature values ​​corresponding to the current block, wherein the image low-level feature values ​​are used to obtain the reconstructed image block corresponding to the current block. The method according to the present invention, characterized by the present invention.

15. The step of determining the image feature reconstruction value corresponding to the current block based on the aforementioned image feature information is: If the control parameter includes a second enable information, and the second enable information indicates that a second inverse quantization process is enabled, the process includes the step of performing inverse quantization on the image feature information to obtain the image feature reconstruction value. The method according to the present invention, characterized by the present invention.

16. If the enhancement layer is a high-quality network, the step of processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit is: The process includes performing enhancement processing on the low-level image feature values ​​corresponding to the input current block based on the aforementioned high-quality network, and obtaining a reconstructed image block corresponding to the output current block. The method according to feature 14.

17. The step of performing enhancement processing on the low-level image feature values ​​corresponding to the current block based on the quality enhancement network to obtain a reconstructed image block corresponding to the current block includes, if the control parameter includes third enable information and the third enable information indicates that the quality enhancement processing should be enabled, the step of performing enhancement processing on the low-level image feature values ​​corresponding to the current block based on the quality enhancement network to obtain a reconstructed image block corresponding to the current block. The method according to 16, characterized by...

18. The step of decoding the control parameters corresponding to the current block from the bitstream is: The process includes the step of decoding the bitstream corresponding to the current block via a control parameter decoding network to obtain control parameters corresponding to the current block, wherein the control parameters include control parameters of the transformation network and control parameters of the quality enhancement network. The method according to any one of claims 1 to 2 or 10 to 12, characterized by...

19. The step of determining the reinforcement layer corresponding to the decoding processing unit based on the reinforcement layer information is: If the enhancement layer information includes network parameters for generating the enhancement layer, the step includes generating the enhancement layer corresponding to the decoding processing unit based on the network parameters. The aforementioned network parameters are: The neural network includes at least one of the following: number of layers, number of filters, filtering size index, filtering coefficient zero flag bit, and filtering coefficient. The method according to any one of claims 10 to 12, characterized by...

20. A step of obtaining control parameters corresponding to the current block, wherein the control parameters include neural network information corresponding to a decoding processing unit, and the neural network information is used to determine the decoding neural network corresponding to the decoding processing unit. The step of encoding control parameters corresponding to the current block into a bitstream, If the neural network information includes basic layer information, the decoding neural network corresponding to the decoding processing unit is: The steps include determining the basic layer corresponding to the decoding processing unit based on the basic layer information, The steps include generating a decoding neural network corresponding to the decoding processing unit based on the aforementioned basic layer, and generating by the decoding side, The step of determining the basic layer corresponding to the decoding processing unit based on the basic layer information is: The basic layer information includes a basic layer default network use flag bit, and if the basic layer default network use flag bit indicates that the basic layer uses the default network, the step includes obtaining the basic layer of the default network structure. A neural network-based image encoding method characterized by the following.

21. A memory configured to store video data, The steps include decoding the control parameters corresponding to the current block from the bitstream, The steps include obtaining neural network information corresponding to the decoding processing unit from the control parameters, and generating a decoding neural network corresponding to the decoding processing unit based on the neural network information, A decoder configured to perform the steps of determining input features corresponding to the decoding processing unit and processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit, If the neural network information includes basic layer information, the step of generating a decoding neural network corresponding to the decoding processing unit based on the neural network information is: The steps include determining the basic layer corresponding to the decoding processing unit based on the basic layer information, The step of generating a decoding neural network corresponding to the decoding processing unit based on the basic layer, The step of determining the basic layer corresponding to the decoding processing unit based on the basic layer information is: The basic layer information includes a basic layer default network use flag bit, and if the basic layer default network use flag bit indicates that the basic layer uses the default network, the step includes obtaining the basic layer of the default network structure. A neural network-based image decoding device characterized by the following:

22. A memory configured to store video data, A step of obtaining control parameters corresponding to the current block, wherein the control parameters include neural network information corresponding to a decoding processing unit, and the neural network information is used to determine the decoding neural network corresponding to the decoding processing unit. The encoder is configured to perform the step of encoding control parameters corresponding to the current block into a bitstream, If the neural network information includes basic layer information, the decoding neural network corresponding to the decoding processing unit is: The steps include determining the basic layer corresponding to the decoding processing unit based on the basic layer information, The steps include generating a decoding neural network corresponding to the decoding processing unit based on the aforementioned basic layer, and generating by the decoding side, The step of determining the basic layer corresponding to the decoding processing unit based on the basic layer information is: The basic layer information includes a basic layer default network use flag bit, and if the basic layer default network use flag bit indicates that the basic layer uses the default network, the step includes obtaining the basic layer of the default network structure. An image encoding device based on a neural network, characterized by the following:

Images