Image decoding and coding method and device based on neural network and equipment thereof

Abstract

The invention provides an image decoding and coding method and device based on a neural network and equipment thereof. The method comprises the following steps: decoding a control parameter and image information corresponding to a current block from a code stream; obtaining neural network information corresponding to a decoding processing unit from the control parameters, and generating a decoding neural network corresponding to the decoding processing unit based on the neural network information; and determining an input feature corresponding to the decoding processing unit based on the image information, and processing the input feature based on the decoding neural network to obtain an output feature corresponding to the decoding processing unit. According to the invention, the coding performance and the decoding performance can be improved.

Description

Technical Field

[0001] The present application relates to the field of coding and decoding technology, and in particular to a method, device and equipment for image decoding and coding based on a neural network. Background Art

[0002] In order to save space, video images are transmitted after being encoded. Complete video encoding can include prediction, transformation, quantization, entropy coding, filtering and other processes. For the prediction process, the prediction process can include intra-frame prediction and inter-frame prediction. Inter-frame prediction refers to the use of the correlation in the video time domain to use the pixels of the adjacent encoded images to predict the current pixel, so as to effectively remove the redundancy in the video time domain. Intra-frame prediction refers to the use of the correlation in the video space domain to use the pixels of the encoded blocks of the current frame image to predict the current pixel, so as to remove the redundancy in the video space domain.

[0003] With the rapid development of deep learning, deep learning has achieved success in many high-level computer vision problems, such as image classification and target detection. Deep learning has also gradually begun to be applied in the field of encoding and decoding, that is, neural networks can be used to encode and decode images. Although the encoding and decoding methods based on neural networks have shown great performance potential, they still have problems such as poor stability, poor generalization, and high complexity. Summary of the invention

[0004] In view of this, the present application provides a neural network-based image decoding, encoding method, device and equipment, which improves the encoding performance and decoding performance, and solves problems such as poor stability, poor generalization and high complexity.

[0005] The present application provides an image decoding method based on a neural network, which is applied to a decoding end, and the method includes:

[0006] Decoding the control parameters and image information corresponding to the current block from the bitstream;

[0007] Acquire neural network information corresponding to a decoding processing unit from the control parameter, and generate a decoding neural network corresponding to the decoding processing unit based on the neural network information;

[0008] An input feature corresponding to the decoding processing unit is determined based on the image information, and the input feature is processed based on the decoding neural network to obtain an output feature corresponding to the decoding processing unit.

[0009] The present application provides an image encoding method based on a neural network, which is applied to an encoding end, and the method includes:

[0010] Determine an input feature corresponding to a coding processing unit based on the current block, process the input feature based on a coding neural network corresponding to the coding processing unit to obtain an output feature corresponding to the coding processing unit, and determine image information corresponding to the current block based on the output feature;

[0011] Acquire control parameters corresponding to the current block, wherein the control parameters include neural network information corresponding to a decoding processing unit, and the neural network information is used to determine a decoding neural network corresponding to the decoding processing unit;

[0012] The image information and control parameters corresponding to the current block are encoded in the bitstream.

[0013] The present application provides an image decoding device based on a neural network, the device comprising:

[0014] a memory configured to store video data;

[0015] A decoder configured to implement:

[0016] Decoding the control parameters and image information corresponding to the current block from the bitstream;

[0017] Acquire neural network information corresponding to a decoding processing unit from the control parameter, and generate a decoding neural network corresponding to the decoding processing unit based on the neural network information;

[0018] An input feature corresponding to the decoding processing unit is determined based on the image information, and the input feature is processed based on the decoding neural network to obtain an output feature corresponding to the decoding processing unit.

[0019] The present application provides an image encoding device based on a neural network, the device comprising:

[0020] a memory configured to store video data;

[0021] An encoder configured to implement:

[0022] Determine an input feature corresponding to a coding processing unit based on the current block, process the input feature based on a coding neural network corresponding to the coding processing unit to obtain an output feature corresponding to the coding processing unit, and determine image information corresponding to the current block based on the output feature;

[0023] Acquire control parameters corresponding to the current block, wherein the control parameters include neural network information corresponding to a decoding processing unit, and the neural network information is used to determine a decoding neural network corresponding to the decoding processing unit;

[0024] The image information and control parameters corresponding to the current block are encoded in the bitstream.

[0025] The present application provides a decoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0026] The processor is used to execute machine executable instructions to implement the above-mentioned neural network-based image decoding method.

[0027] The present application provides an encoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0028] The processor is used to execute machine executable instructions to implement the above-mentioned neural network-based image encoding method.

[0029] As can be seen from the above technical solutions, in the embodiment of the present application, the control parameters corresponding to the current block can be decoded from the code stream, the neural network information corresponding to the decoding processing unit can be obtained from the control parameters, and the decoding neural network corresponding to the decoding processing unit can be generated based on the neural network information, and then the image decoding can be realized based on the decoding neural network to improve the decoding performance. Image encoding can be realized based on the encoding neural network corresponding to the encoding processing unit to improve the encoding performance. Neural networks (such as decoding neural networks and encoding neural networks, etc.) can be used to encode and decode images, and neural network information can be transmitted through the code stream, and then the decoding neural network corresponding to the decoding processing unit can be generated based on the neural network information, thereby solving the problems of poor stability, poor generalization and high complexity, that is, good stability, good generalization and low complexity. A solution for dynamic adjustment of encoding and decoding complexity can be provided, which has better encoding and decoding performance than the framework of a single neural network. Since each current block corresponds to a control parameter, the neural network information obtained from the control parameter is the neural network information for the current block, so that a decoding neural network is generated for each current block, that is, the decoding neural network of different current blocks may be the same or different, so that the decoding neural network at the block level, that is, the decoding neural network can be changed and adjusted. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the video coding framework;

[0031] Figure 2A-2C It is a schematic diagram of the video coding framework;

[0032] Figure 3 is a flowchart of a neural network-based image decoding method in one embodiment of the present application;

[0033] Figure 4is a flowchart of a neural network-based image encoding method in one embodiment of the present application;

[0034] Figure 5A and Figure 5C is a schematic diagram of an image encoding method and an image decoding method in one embodiment of the present application;

[0035] Figure 5B and Figure 5D is a schematic diagram of border filling in one embodiment of the present application;

[0036] Figure 5E is a schematic diagram of image domain transformation of an original image in one embodiment of the present application;

[0037] Fig. 6A and Figure 6B is a schematic diagram of the structure of a decoding end in one implementation of the present application;

[0038] Figure 6C It is a structural schematic diagram of a coefficient hyperparameter feature generation unit in one implementation of the present application;

[0039] Fig.6D is a schematic diagram of the structure of an image feature inverse transformation unit in one embodiment of the present application;

[0040] Fig. 7A , Figure 7B and Figure 7C is a schematic diagram of the structure of an encoding end in one embodiment of the present application;

[0041] Fig. 8A is a hardware structure diagram of a decoding end device in one embodiment of the present application;

[0042] Figure 8B It is a hardware structure diagram of an encoding end device in one implementation of the present application. DETAILED DESCRIPTION

[0043] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than for limiting the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items. It should be understood that, although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, depending on the context. In addition, the word "if" used can be interpreted as "at ... ", or "when ... ", or "in response to determination".

[0044] In the embodiment of the present application, a neural network-based image decoding and encoding method is proposed, which may involve the following concepts:

[0045] Neural Network (NN): Neural network refers to artificial neural network, not biological neural network. Neural network is a computing model composed of a large number of nodes (or neurons) connected to each other. In a neural network, neuron processing units can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. The types of processing units in a neural network are divided into three categories: input unit, output unit and hidden unit. The input unit receives signals and data from the outside world; the output unit realizes the output of the processing results; the hidden unit is located between the input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units, and the representation and processing of information are reflected in the connection relationship of the processing units. Neural network is a non-programmed, brain-like information processing method. Its essence is to obtain a parallel distributed information processing function through the transformation and dynamic behavior of neural networks, and imitate the information processing function of the human brain nervous system to varying degrees and levels. In the field of video processing, commonly used neural networks may include but are not limited to: convolutional neural network (CNN), recurrent neural network (RNN), fully connected network, etc.

[0046] Convolutional Neural Network (CNN): Convolutional neural network is a feedforward neural network and one of the most representative network structures in deep learning technology. The artificial neurons of convolutional neural network can respond to surrounding units within a certain coverage range and have excellent performance in large-scale image processing. The basic structure of convolutional neural network includes two layers. One is the feature extraction layer (also called convolution layer). The input of each neuron is connected to the local receptive field of the previous layer and extracts the local features. Once the local feature is extracted, its positional relationship with other features is also determined. The second is the feature mapping layer (also called activation layer). Each computing layer of the neural network consists of multiple feature maps. Each feature map is a plane, and the weights of all neurons on the plane are equal. The feature mapping structure can use Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, GDN function, etc. as the activation function of the convolutional network. In addition, since the neurons on a mapping surface share weights, the number of free parameters of the network is reduced.

[0047] One of the advantages of convolutional neural networks over image processing algorithms is that they avoid the complex pre-processing of images (such as extracting artificial features) and can directly input the original image for end-to-end learning. One of the advantages of convolutional neural networks over ordinary neural networks is that ordinary neural networks are fully connected, that is, neurons from the input layer to the hidden layer are all connected, which will result in a huge number of parameters, making network training time-consuming or even difficult to train, while convolutional neural networks avoid this difficulty through methods such as local connection and weight sharing.

[0048] Deconvolution layer: Deconvolution layer is also called transposed convolution layer. The working process of deconvolution layer and convolution layer is very similar. The main difference is that deconvolution layer will make the output larger than the input through padding (of course, it can also remain the same). If stride is 1, it means that the output size is equal to the input size; if stride is N, it means that the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0049] Generalization Ability: Generalization ability refers to the ability of a machine learning algorithm to adapt to fresh samples. The purpose of learning is to learn the rules hidden behind the data pairs. For data outside the learning set with the same rules, the trained network can also give appropriate outputs. This ability can be called generalization ability.

[0050] Rate-Distortion Optimized: There are two major indicators for evaluating coding efficiency: bit rate and PSNR (Peak Signal to Noise Ratio). The smaller the bit stream, the greater the compression rate, and the greater the PSNR, the better the quality of the reconstructed image. When selecting a mode, the discriminant formula is essentially a comprehensive evaluation of the two. For example, the cost corresponding to the mode: J(mode) = D + λ*R, where D represents Distortion, which can usually be measured using the SSE indicator. SSE refers to the mean square sum of the difference between the reconstructed image block and the source image. In order to achieve cost considerations, the SAD indicator can also be used. SAD refers to the sum of the absolute values ​​of the difference between the reconstructed image block and the source image; λ is the Lagrange multiplier, and R is the actual number of bits required for encoding the image block in this mode, including the sum of bits required for encoding mode information, motion information, residuals, etc. When selecting a mode, if the rate-distortion principle is used to make a comparison decision on the encoding mode, the best encoding performance can usually be guaranteed.

[0051] Video Coding Framework: See Figure 1 FIG. 1 is a schematic diagram of a video encoding framework of an encoding end, and the video encoding framework can be used to implement the encoding end processing flow of the embodiment of the present application. The schematic diagram of the video decoding framework can be similar to FIG. Figure 1 Similarly, without repeating the details here, the video decoding framework can be used to implement the decoding end processing flow of the embodiment of the present application.

[0052] For example, see Figure 1 As shown, the video encoding framework may include modules such as prediction, transformation, quantization, entropy encoder, inverse quantization, inverse transformation, reconstruction, and filtering. At the encoding end, the processing flow of the encoding end can be realized through the cooperation between these modules. In addition, the video decoding framework may include modules such as prediction, transformation, quantization, entropy decoder, inverse quantization, inverse transformation, reconstruction, and filtering. At the decoding end, the processing flow of the decoding end can be realized through the cooperation between these modules.

[0053] A large number of coding tools have been proposed for each module of the encoding end, and each tool often has multiple modes. For different video sequences, the coding tools that can achieve the best coding performance are often different. Therefore, in the encoding process, RDO (Rate-Distortion Optimize) is usually used to compare the coding performance of different tools or modes to select the best mode. After determining the optimal tool or mode, the decision information of the tool or mode is transmitted by encoding the marker information in the bitstream. Although this method brings higher coding complexity, it can adaptively select the optimal mode combination for different content to obtain the best coding performance. The decoding end can obtain relevant mode information by directly parsing the flag information, and the complexity impact is small.

[0054] The following is a brief introduction to the structure of the encoding and decoding ends. Figure 2A , a schematic block diagram of an example of an encoding end for implementing an embodiment of the present application is shown. Figure 2A In the embodiment, the encoding end includes a prediction processing unit, a residual calculation unit, a transform processing unit, a quantization unit, an encoding unit, an inverse quantization unit (also referred to as an inverse quantization unit), an inverse transform processing unit (also referred to as an inverse transform processing unit), a reconstruction unit (or referred to as a reconstruction unit), and a filter unit. In one example, the encoding end may further include a buffer and a decoded image buffer, wherein the buffer is used to cache the reconstructed image blocks output by the reconstruction unit, and the decoded image buffer is used to cache the filtered image blocks output by the filter unit.

[0055] The input of the encoding end (also called an encoder) is an image block of an image (which can be called an image to be encoded), and the image block can also be called a current block or a block to be encoded. The encoding end can also include a segmentation unit (not shown in the figure), which is used to segment the image to be encoded into multiple image blocks. The encoding end is used to encode block by block to complete the encoding of the image to be encoded, for example, to perform the encoding process on each image block. The prediction processing unit is used to receive or obtain an image block (the current image block to be encoded of the current image to be encoded, which can also be called a current block, and the image block can be understood as the true value of the image block) and reconstructed image data, and predict the current block based on the relevant data in the reconstructed image data to obtain a prediction block of the current block. In an example, the prediction processing unit can include an inter-frame prediction unit, an intra-frame prediction unit and a mode selection unit, and the mode selection unit is used to select an intra-frame prediction mode or an inter-frame prediction mode. If the intra-frame prediction mode is selected, the prediction process is performed by the intra-frame prediction unit, and if the inter-frame prediction mode is selected, the prediction process can be performed by the inter-frame prediction unit.

[0056] The residual calculation unit is used to calculate the residual between the real value of the image block and the prediction block of the image block to obtain the residual block. For example, the residual calculation unit can subtract the pixel value of the prediction block from the pixel value of the image block pixel by pixel.

[0057] The transform processing unit is used to perform a transform, such as discrete cosine transform (DCT) or discrete sine transform (DST), on the residual block to obtain a transform coefficient in the transform domain. The transform coefficient may also be referred to as a transform residual coefficient, which may represent the residual block in the transform domain.

[0058] The quantization unit is used to quantize the transform coefficients by applying scalar quantization or vector quantization to obtain quantized transform coefficients, which may also be referred to as quantized residual coefficients. The quantization process may reduce the bit depth associated with some or all of the transform coefficients. For example, an n-bit transform coefficient may be rounded down to an m-bit transform coefficient during quantization, where n is greater than m. The degree of quantization may be modified by adjusting a quantization parameter (QP). For example, for scalar quantization, different scales may be applied to achieve finer or coarser quantization. A smaller quantization step size corresponds to finer quantization, while a larger quantization step size corresponds to coarser quantization. A suitable quantization step size may be indicated by a quantization parameter (QP).

[0059] The coding unit is used to encode the above-mentioned quantized residual coefficients, output the encoded image data (i.e., the encoding result of the current image block to be encoded) in the form of an encoded bit stream, and then transmit the encoded bit stream to the decoder, or store it, and then transmit it to the decoder or use it for retrieval. The coding unit can also be used to encode other syntax elements of the current image block, such as encoding the prediction mode into the bit stream, etc. The coding algorithm includes but is not limited to a variable length coding (VLC) algorithm, a context adaptive VLC (CAVLC) algorithm, an arithmetic coding algorithm, a context adaptive binary arithmetic coding (CABAC) algorithm, a syntax-based context-adaptive binary arithmetic coding (SBAC) algorithm, and a probability interval partitioning entropy (PIPE) algorithm.

[0060] The inverse quantization unit is used to inversely quantize the quantized coefficients to obtain inversely quantized coefficients, where the inverse quantization is a reverse application of the quantization unit, for example, based on or using the same quantization step size as the quantization unit, applying an inverse quantization scheme of the quantization scheme applied by the quantization unit. The inversely quantized coefficients may also be referred to as inversely quantized residual coefficients.

[0061] The inverse transform processing unit is used to perform an inverse transform on the inverse quantized coefficients. It should be understood that the inverse transform is the reverse application of the transform processing unit. For example, the inverse transform may include an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST) to obtain an inverse transform block in a pixel domain (or sample domain). The inverse transform block may also be referred to as an inverse transform dequantized block or an inverse transform residual block.

[0062] The reconstruction unit is used to add the inverse transform block (i.e., the inverse transform residual block) to the prediction block to obtain a reconstructed block in the sample domain. The reconstruction unit can be a summer, for example, adding the sample value (i.e., pixel value) of the residual block to the sample value of the prediction block. The reconstructed block output by the reconstruction unit can be subsequently used to predict other image blocks, for example, in an intra-frame prediction mode.

[0063] The filter unit (or simply "filter") is used to filter the reconstructed block to obtain a filtered block, so as to smoothly perform pixel conversion or improve image quality. The filter unit can be a loop filter unit, which is intended to represent one or more loop filters. For example, the filter unit can be a deblocking filter, a sample adaptive offset (sample-adaptive offset, SAO) filter or other filters, such as a bilateral filter, an adaptive loop filter (adaptive loop filter, ALF), or a sharpening or smoothing filter, or a collaborative filter. In an example, the filtered block output by the filter unit can be subsequently used to predict other image blocks, for example, in an inter-frame prediction mode, without limitation.

[0064] See also Figure 2B , a schematic block diagram of an example of an encoding end (also referred to as a decoder) for implementing an embodiment of the present application is shown. The decoder is used to receive, for example, encoded image data (i.e., an encoded bit stream, for example, an encoded bit stream including an image block and associated syntax elements) encoded by an encoder to obtain a decoded image. The decoder includes a decoding unit, an inverse quantization unit, an inverse transform processing unit, a prediction processing unit, a reconstruction unit, and a filter unit. In some examples, the decoder may perform operations substantially similar to Figure 2AThe decoding pass is the inverse of the encoding pass described by the encoder. In one example, the decoder may further include a buffer and a decoded image buffer, wherein the buffer is used to cache the reconstructed image blocks output by the reconstruction unit, and the decoded image buffer is used to cache the filtered image blocks output by the filter unit.

[0065] The decoding unit is used to decode the encoded image data to obtain quantized coefficients and / or decoded encoding parameters (for example, the decoded parameters may include any one or all of inter-frame prediction parameters, intra-frame prediction parameters, filter parameters and / or other syntax elements). The decoding unit is also used to forward the above-mentioned decoded encoding parameters to the prediction processing unit so that the prediction processing unit performs a prediction process according to the encoding parameters. The function of the inverse quantization unit may be the same as that of the inverse quantization unit of the encoder, and is used to inverse quantize (i.e., inverse quantize) the quantized coefficients decoded by the decoding unit.

[0066] The function of the inverse transform processing unit may be the same as that of the inverse transform processing unit of the encoder, and the function of the reconstruction unit (e.g., the summer) may be the same as that of the reconstruction unit of the encoder, and is used to perform an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) on the above-mentioned quantized coefficients to obtain an inverse transform block (also referred to as an inverse transform residual block), which is the residual block of the current image block in the pixel domain.

[0067] A prediction processing unit is used to receive or obtain encoded image data (e.g., an encoded bit stream of a current image block) and reconstructed image data. The prediction processing unit can also receive or obtain prediction-related parameters and / or information about a selected prediction mode (i.e., decoded encoding parameters) from, for example, a decoding unit, and predict the current image block based on the relevant data in the reconstructed image data and the decoded encoding parameters to obtain a prediction block of the current image block.

[0068] In one example, the prediction processing unit may include an inter-frame prediction unit, an intra-frame prediction unit and a mode selection unit. The mode selection unit is used to select an intra-frame prediction mode or an inter-frame prediction mode. If the intra-frame prediction mode is selected, the prediction process is performed by the intra-frame prediction unit. If the inter-frame prediction mode is selected, the prediction process is performed by the inter-frame prediction unit.

[0069] The reconstruction unit is used to add the inverse transform block (ie, the inverse transform residual block) to the prediction block to obtain a reconstructed block in the sample domain. For example, the sample values ​​of the inverse transform residual block may be added to the sample values ​​of the prediction block.

[0070] The filter unit is used for filtering the reconstructed block to obtain a filtered block, where the filtered block is a decoded image block.

[0071] It should be understood that in the encoder and decoder of the embodiments of the present application, the processing results of a certain link can also be output to the next link after further processing. For example, after interpolation filtering, motion vector derivation or filtering, the processing results of the corresponding link can be further subjected to operations such as Clip or shift.

[0072] Based on the encoder and decoder, the embodiment of the present application provides a possible encoding / decoding implementation method, such as Figure 2C As shown, Figure 2C A schematic diagram of a coding and decoding process provided in an embodiment of the present application, wherein the coding and decoding implementation method includes process ① to process ⑤, and process ① to process ⑤ can be performed by the above-mentioned decoder and encoder. Process ①: Divide a frame of image into one or more non-overlapping parallel coding units. The one or more parallel coding units have no dependency relationship and can be completely parallel / independently coded and decoded, such as Figure 2C A parallel encoding unit 1 and a parallel encoding unit 2 are shown.

[0073] Process ②: For each parallel coding unit, it can be divided into one or more non-overlapping independent coding units. Each independent coding unit may not depend on each other, but may share some parallel coding unit header information. For example, the width of an independent coding unit is w_lcu and the height is h_lcu. If a parallel coding unit is divided into an independent coding unit, the size of the independent coding unit is exactly the same as the parallel coding unit; otherwise, the width of the independent coding unit should be greater than the height (unless it is an edge area).

[0074] Typically, the independent coding unit may be a fixed w_lcu×h_lcu, where w_lcu and h_lcu are both 2 to the power of N (N≥0), such as the size of the independent coding unit is 128×4, 64×4, 32×4, 16×4, 8×4, 32×2, 16×2 or 8×2.

[0075] As a possible example, the independent coding unit can be a fixed 128×4. If the size of the parallel coding unit is 256×8, the parallel coding unit can be equally divided into 4 independent coding units; if the size of the parallel coding unit is 288×10, the parallel coding unit is divided into: the first / second line is 2 128×4 + 1 32×4 independent coding units; the third line is 2 128×2 + 1 32×2 independent coding units. It is worth noting that the independent coding unit can include three components of brightness Y, chrominance Cb, and chrominance Cr, or three components of red (red, R), green (green, G), and blue (blue, B), or three components of brightness Y, chrominance Co, and chrominance Cg, or only one of them. If the independent coding unit contains three components, the sizes of the three components can be exactly the same or different, which is related to the input format of the image.

[0076] Process ③: Each independent coding unit can be further divided into one or more non-overlapping coding units. The coding units within the independent coding unit can be interdependent, such as multiple coding units can perform mutual reference pre-coding and decoding.

[0077] If the coding unit is the same size as the independent coding unit (i.e., the independent coding unit is divided into only one coding unit), its size can be all the sizes described in process ②. If the independent coding unit is divided into multiple non-overlapping coding units, the feasible division examples include: horizontal equal division (the height of the coding unit is the same as that of the independent coding unit, but the width is different, which can be 1 / 2, 1 / 4, 1 / 8, 1 / 16, etc.), vertical equal division (the width of the coding unit is the same as that of the independent coding unit, but the height is different, which can be 1 / 2, 1 / 4, 1 / 8, 1 / 16, etc.), horizontal and vertical equal division (quadtree division), etc., preferably horizontal equal division.

[0078] The width of the coding unit is w_cu, and the height is h_cu. The width should be greater than the height (unless it is an edge area). Usually, the coding unit is fixed w_cu x h_cu, and w_cu and h_cu are both 2N powers (N is greater than or equal to 0), such as 16x4, 8x4, 16x2, 8x2, 8x1, 4x1, etc. For example, the coding unit is fixed to 16x4. If the size of the independent coding unit is 64x4, the independent coding unit is divided into 4 coding units; if the size of the independent coding unit is 72x4, the coding unit is divided into: 4 16x4 + 1 8x4. It is worth noting that the coding unit can include three components of brightness Y, chroma Cb, and chroma Cr (or three components of red R, green G, and blue B, or brightness Y, chroma Co, and chroma Cg), or it can only contain one of the components. If it contains three components, the sizes of the components can be exactly the same or different, which is related to the image input format.

[0079] It is worth noting that process ③ may be an optional step in the encoding and decoding method, and the encoder / decoder may encode and decode the residual coefficients (or residual values) of the independent coding units obtained in process ②.

[0080] Process ④: For the coding unit, it can be further divided into one or more non-overlapping prediction groups (Prediction Group, PG), PG is also referred to as Group, each PG is encoded and decoded according to the selected prediction mode to obtain the prediction value of PG, which constitutes the prediction value of the entire coding unit, and the residual value of the coding unit is obtained based on the prediction value and the original value of the coding unit.

[0081] Process ⑤: Based on the residual values ​​of the coding units, the coding units are grouped to obtain one or more non-overlapping residual blocks (RBs). The residual coefficients of each RB are encoded and decoded according to the selected mode to form a residual coefficient stream. Specifically, it can be divided into two categories: transforming the residual coefficients and not transforming them.

[0082] Among them, the selected mode of the residual coefficient encoding and decoding method in process ⑤ may include, but is not limited to any of the following: semi-fixed length encoding method, exponential Golomb encoding method, Golomb-Rice encoding method, truncated unary code encoding method, run-length encoding method, direct encoding of original residual values, etc. For example, the encoder may directly encode the coefficients in the RB. For another example, the encoder may also transform the residual block, such as DCT, DST, Hadamard transform, etc., and then encode the transformed coefficients. As a possible example, when the RB is small, the encoder may directly quantize each coefficient in the RB and then perform binary encoding. If the RB is large, it can be further divided into multiple coefficient groups (coefficient groups, CG), and then each CG is uniformly quantized and then binarized. In some embodiments of the present application, the coefficient group (CG) and the quantization group (QG) may be the same, and of course, the coefficient group and the quantization group may also be different.

[0083] The following is an exemplary explanation of the residual coefficient encoding part in a semi-fixed length coding method. First, the maximum value of the absolute value of the residual in an RB block is defined as the modified maximum value (modified maximum, mm). Secondly, the number of coding bits for the residual coefficient in the RB block is determined (the number of coding bits for the residual coefficient in the same RB block is consistent). For example, if the critical limit (CL) of the current RB block is 2 and the current residual coefficient is 1, then 2 bits are required to encode the residual coefficient 1, which is expressed as 01. If the CL of the current RB block is 7, it means encoding an 8-bit residual coefficient and a 1-bit sign bit. The determination of CL is to find the minimum M value that satisfies that all residuals of the current sub-block are within the range of [-2^(M-1), 2^(M-1)]. If both -2^(M-1) and 2^(M-1) exist, M increases by 1, that is, M+1 bits are needed to encode all the residuals of the current RB block; if only one of the two boundary values ​​-2^(M-1) and 2^(M-1) exists, a trailing bit is encoded to determine whether the boundary value is -2^(M-1) or 2^(M-1); if all residuals do not exist in either -2^(M-1) or 2^(M-1), there is no need to encode the trailing bit. For some special cases, the encoder can directly encode the original value of the image instead of the residual value.

[0084] With the rapid development of deep learning, neural networks can adaptively construct feature descriptions driven by training data, with higher flexibility and universality, making deep learning successful in many high-level computer vision problems, such as image classification, target detection, etc., and deep learning has gradually begun to be applied in the field of coding and decoding, that is, using neural networks to encode and decode images. For example, the convolutional neural network VRCNN (Variable Filter Size Convolutional Neural Network) is used to replace the deblocking filter technology and adaptive sample point compensation technology, and the intra-coded image is post-processed and filtered, so that the subjective and objective quality of the reconstructed image is greatly improved. For another example, neural networks can be applied to intra-frame prediction, and an intra-frame prediction mode based on block up-sampling and down-sampling is proposed. For the intra-frame predicted blocks, down-sampling encoding is performed first, and then the reconstructed pixels are up-sampled through the neural network, which can obtain a performance gain of up to 9.0% for ultra-high-definition sequences. Obviously, neural networks can effectively get rid of the limitations of artificially set modes, obtain neural networks that meet actual needs through data-driven, and significantly improve encoding performance.

[0085] For example, although the neural network-based encoding and decoding method shows great performance potential, the neural network-based encoding and decoding method still has problems such as poor stability, poor generalization and high complexity. First of all, the neural network is still in the process of rapid iteration, and new network structures emerge in an endless stream. Even for some common problems, there is still no definite number for which network structure is optimal, not to mention the specific problems of certain modules of the encoder. Therefore, for a certain module of the encoder, using a single fixed neural network is more risky. Secondly, since the formation of the neural network is highly dependent on the training data, if the training data does not contain a certain feature of the actual problem, poor performance is likely to occur when dealing with the problem. In the related art, a mode usually only uses one neural network. When the generalization ability of the neural network is insufficient, the mode will lead to a reduction in coding performance. Thirdly, due to the high data density and low latency requirements of video coding, video coding standard technology has extremely strict requirements on complexity, especially decoding complexity. In order to obtain better coding performance, the number of parameters of the neural network used for coding is often very large (such as more than 1M). The average number of multiplications and additions generated by a neural network can reach more than 100K times per pixel. Although the simplified network layers or neural networks with fewer parameters cannot achieve the best coding performance, the encoding and decoding complexity brought by such neural networks can be greatly reduced. In addition, for image coding schemes, most schemes use the entire frame of image as input (with huge cache overhead), and cannot effectively control the output bit rate. In practical applications, it is often necessary to obtain images with any bit rate.

[0086] In view of the above findings, an image decoding method and an image encoding method based on a neural network are proposed in the embodiments of the present application. A neural network (such as a decoding neural network and an encoding neural network, etc.) can be used to encode and decode images. In this embodiment, the optimization idea is: in addition to paying attention to encoding performance and decoding performance, it is also necessary to pay attention to complexity (especially the parallelism of coefficient encoding and decoding) and application functionality (supporting variable and fine-tunable bit rates, that is, controllable bit rates). Based on the above optimization ideas, several feasible methods are given in the embodiments of the present application: a) The structure of the neural network is fixed, but the network parameters of the neural network (that is, weight parameters) are not limited, and the network parameters can be indexed by ID. b) The structural flexibility of the neural network, the relevant structural parameters can be transmitted through syntax encoding, and the network parameters can be indexed by ID (some parameters can be transmitted through syntax encoding, increasing a certain bit rate cost). c) The structural flexibility of the neural network (high-level syntax configuration), the network parameters of some networks (such as shallow networks) are fixed (saving bit rate), and the network parameters of the remaining networks are transmitted through coding syntax (maintaining performance optimization space).

[0087] The decoding method and encoding method in the embodiments of the present application are described in detail below in combination with several specific embodiments.

[0088] Example 1: In the present application, a neural network-based image decoding method is proposed, see Figure 3 As shown, it is a flow chart of the method, which can be applied to a decoding end (also called a video decoder), and the method may include:

[0089] Step 301: Decode control parameters and image information corresponding to the current block from the bitstream.

[0090] Step 302: Obtain the neural network information corresponding to the decoding processing unit from the control parameter, and generate a decoding neural network corresponding to the decoding processing unit based on the neural network information.

[0091] Step 303: Determine the input features corresponding to the decoding processing unit based on the image information, process the input features based on the decoding neural network, and obtain the output features corresponding to the decoding processing unit.

[0092] In one possible implementation, if the neural network information includes basic layer information and enhanced layer information, the basic layer corresponding to the decoding processing unit can be determined based on the basic layer information, and the enhanced layer corresponding to the decoding processing unit can be determined based on the enhanced layer information; then, a decoding neural network corresponding to the decoding processing unit is generated based on the basic layer and the enhanced layer.

[0093] Exemplarily, with respect to the process of determining the basic layer, if the basic layer information includes a flag bit indicating that the basic layer uses a default network, and the flag bit indicating that the basic layer uses a default network indicates that the basic layer uses a default network, then the basic layer of the default network structure is obtained.

[0094] Exemplarily, with respect to the process of determining the basic layer, if the basic layer information includes a basic layer using a prefabricated network flag and a basic layer prefabricated network index number, and the basic layer using a prefabricated network flag indicates that the basic layer uses a prefabricated network, then the basic layer of the prefabricated network structure corresponding to the basic layer prefabricated network index number can be selected from the prefabricated neural network pool.

[0095] The prefabricated neural network pool may include at least one network layer of a prefabricated network structure.

[0096] Exemplarily, for the process of determining the enhancement layer, if the enhancement layer information includes a flag bit indicating that the enhancement layer uses a default network, and the flag bit indicating that the enhancement layer uses a default network indicates that the enhancement layer uses a default network, then the enhancement layer of the default network structure is obtained.

[0097] Exemplarily, with respect to the process of determining the enhancement layer, if the enhancement layer information includes an enhancement layer using a prefabricated network flag and an enhancement layer prefabricated network index number, and the enhancement layer using a prefabricated network flag indicates that the enhancement layer uses a prefabricated network, then an enhancement layer of a prefabricated network structure corresponding to the enhancement layer prefabricated network index number can be selected from the prefabricated neural network pool.

[0098] The prefabricated neural network pool includes at least one network layer of a prefabricated network structure.

[0099] Exemplarily, for the process of determining the enhancement layer, if the enhancement layer information includes network parameters for generating the enhancement layer, the enhancement layer corresponding to the decoding processing unit is generated based on the network parameters; wherein the network parameters may include but are not limited to at least one of the following: number of neural network layers, deconvolution layer flag, number of deconvolution layers, quantization step size of each deconvolution layer, number of channels of each deconvolution layer, size of convolution kernel, number of filters, filter size index, filter coefficient zero flag, filter coefficient, activation layer flag, activation layer type. Of course, the above are just a few examples and are not limited to this.

[0100] In a possible implementation, the image information may include coefficient hyperparameter feature information and image feature information, and the input feature corresponding to the decoding processing unit is determined based on the image information, and the input feature is processed based on the decoding neural network to obtain the output feature corresponding to the decoding processing unit, which may include but is not limited to: when performing the decoding process of coefficient hyperparameter feature generation, the coefficient hyperparameter feature coefficient reconstruction value may be determined based on the coefficient hyperparameter feature information; the coefficient hyperparameter feature coefficient reconstruction value is inversely transformed based on the decoding neural network to obtain the coefficient hyperparameter feature value; wherein the coefficient hyperparameter feature value can be used to decode the image feature information from the bitstream. When performing the decoding process of image feature inverse transformation, the image feature reconstruction value may be determined based on the image feature information; the image feature reconstruction value is inversely transformed based on the decoding neural network to obtain the image low-order feature value; wherein the image low-order feature value can be used to obtain the reconstructed image block corresponding to the current block.

[0101] Exemplarily, determining the coefficient hyperparameter feature coefficient reconstruction value based on the coefficient hyperparameter feature information may include but is not limited to: if the control parameter includes first enabling information, and the first enabling information indicates enabling a first inverse quantization operation, then the coefficient hyperparameter feature information may be inversely quantized to obtain the coefficient hyperparameter feature coefficient reconstruction value. Determining the image feature reconstruction value based on the image feature information may include but is not limited to: if the control parameter includes second enabling information, and the second enabling information indicates enabling a second inverse quantization operation, then the image feature information may be inversely quantized to obtain the image feature reconstruction value.

[0102] Exemplarily, if the control parameters include third enabling information, and the third enabling information indicates enabling the quality enhancement operation, when performing the quality enhancement decoding process, the low-order eigenvalues ​​of the image can be obtained, and the low-order eigenvalues ​​of the image can be enhanced based on the decoding neural network to obtain the reconstructed image block corresponding to the current block.

[0103] In a possible implementation, the decoding end device may include a control parameter decoding unit, a first feature decoding unit, a second feature decoding unit, a coefficient super-parameter feature generation unit, and an image feature inverse transformation unit; the image information may include coefficient super-parameter feature information and image feature information. Wherein: the control parameter decoding unit may decode the control parameter from the bitstream, the first feature decoding unit may decode the coefficient super-parameter feature information from the bitstream, and the second feature decoding unit may decode the image feature information from the bitstream. When the coefficient super-parameter feature generation unit is a decoding processing unit, the coefficient super-parameter feature coefficient reconstruction value may be determined based on the coefficient super-parameter feature information; the coefficient super-parameter feature generation unit may perform an inverse transformation operation on the coefficient super-parameter feature coefficient reconstruction value based on the decoding neural network to obtain the coefficient super-parameter feature value; wherein the coefficient super-parameter feature value is used to enable the second feature decoding unit to decode the image feature information from the bitstream. When the image feature inverse transformation unit is a decoding processing unit, the image feature reconstruction value is determined based on the image feature information; the image feature inverse transformation unit may perform an inverse transformation operation on the image feature reconstruction value based on the decoding neural network to obtain the image low-order feature value; wherein the image low-order feature value is used to obtain the reconstructed image block corresponding to the current block.

[0104] Exemplarily, the decoding end device also includes a first dequantization unit and a second dequantization unit; wherein: the control parameter may include first enable information of the first dequantization unit. If the first enable information indicates enabling the first dequantization unit, the first dequantization unit may obtain coefficient hyperparameter feature information from the first feature decoding unit, and dequantize the coefficient hyperparameter feature information to obtain coefficient hyperparameter feature coefficient reconstruction value, and provide the coefficient hyperparameter feature coefficient reconstruction value to the coefficient hyperparameter feature generation unit; the control parameter may include second enable information of the second dequantization unit. If the second enable information indicates enabling the second dequantization unit, the second dequantization unit may obtain image feature information from the second feature decoding unit, dequantize the image feature information to obtain image feature reconstruction value, and provide the image feature reconstruction value to the image feature inverse transformation unit.

[0105] Exemplarily, the decoding end device may also include a quality enhancement unit; wherein: the control parameter may include third enabling information of the quality enhancement unit. If the third enabling information indicates enabling the quality enhancement unit, then, when the quality enhancement unit is a decoding processing unit, the quality enhancement unit may obtain low-order eigenvalues ​​of the image from the image feature inverse transformation unit, and enhance the low-order eigenvalues ​​of the image based on the decoding neural network to obtain a reconstructed image block corresponding to the current block.

[0106] Exemplarily, the above execution order is only for the convenience of describing the examples given. In practical applications, the execution order between the steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0107] As can be seen from the above technical solutions, in the embodiment of the present application, the control parameters corresponding to the current block can be decoded from the code stream, the neural network information corresponding to the decoding processing unit can be obtained from the control parameters, and the decoding neural network corresponding to the decoding processing unit can be generated based on the neural network information, and then the image decoding can be realized based on the decoding neural network to improve the decoding performance. Image encoding can be realized based on the encoding neural network corresponding to the encoding processing unit to improve the encoding performance. A deep learning network (such as a decoding neural network and an encoding neural network, etc.) can be used to encode and decode the image, and the neural network information is transmitted through the code stream, and then the decoding neural network corresponding to the decoding processing unit is generated based on the neural network information, thereby solving the problems of poor stability, poor generalization and high complexity, that is, good stability, good generalization and low complexity. A solution for dynamic adjustment of encoding and decoding complexity can be provided, which has better encoding performance than the framework of a single deep learning network. Since each current block corresponds to a control parameter, the neural network information obtained from the control parameter is the neural network information for the current block, so that a decoding neural network is generated for each current block, that is, the decoding neural network of different current blocks may be the same or different, so that the decoding neural network at the block level, that is, the decoding neural network can be changed and adjusted.

[0108] Example 2: In the present application, a neural network-based image encoding method is proposed, see Figure 4 As shown, it is a flowchart of the method, which can be applied to an encoding end (also called a video encoder), and the method may include:

[0109] Step 401: determine the input features corresponding to the encoding processing unit based on the current block, process the input features based on the encoding neural network corresponding to the encoding processing unit to obtain the output features corresponding to the encoding processing unit, and determine the image information corresponding to the current block based on the output features, such as coefficient hyperparameter feature information and image feature information.

[0110] Step 402: Obtain control parameters corresponding to the current block. The control parameters may include neural network information corresponding to the decoding processing unit. The neural network information is used to determine a decoding neural network corresponding to the decoding processing unit.

[0111] Step 403: Encode the image information and control parameters corresponding to the current block in the bitstream.

[0112] In a possible implementation, the neural network information may include basic layer information and enhanced layer information, and the decoding neural network includes a basic layer determined based on the basic layer information and an enhanced layer determined based on the enhanced layer information.

[0113] Exemplarily, if the basic layer information includes a flag bit that the basic layer uses a default network, and the flag bit that the basic layer uses a default network indicates that the basic layer uses a default network, then the decoding neural network adopts the basic layer of the default network structure.

[0114] Exemplarily, if the basic layer information includes a basic layer using a prefabricated network flag and a basic layer prefabricated network index number, and the basic layer using a prefabricated network flag indicates that the basic layer uses a prefabricated network, then the decoding neural network can adopt the basic layer of the prefabricated network structure corresponding to the basic layer prefabricated network index number selected from the prefabricated neural network pool.

[0115] The prefabricated neural network pool may include at least one network layer of a prefabricated network structure.

[0116] Exemplarily, if the enhancement layer information includes a flag bit that the enhancement layer uses a default network, and the flag bit that the enhancement layer uses a default network indicates that the enhancement layer uses a default network, the decoding neural network adopts the enhancement layer of the default network structure.

[0117] Exemplarily, if the enhancement layer information includes an enhancement layer using a prefabricated network flag and an enhancement layer prefabricated network index number, and the enhancement layer using a prefabricated network flag indicates that the enhancement layer uses a prefabricated network, the decoding neural network can adopt an enhancement layer of a prefabricated network structure corresponding to the enhancement layer prefabricated network index number selected from a prefabricated neural network pool.

[0118] The prefabricated neural network pool includes at least one network layer of a prefabricated network structure.

[0119] Exemplarily, if the enhancement layer information includes network parameters for generating the enhancement layer, the decoding neural network may adopt the enhancement layer generated based on the network parameters; wherein the network parameters may include but are not limited to at least one of the following: number of neural network layers, deconvolution layer flag, number of deconvolution layers, quantization step size of each deconvolution layer, number of channels of each deconvolution layer, size of convolution kernel, number of filters, filter size index, filter coefficient zero flag, filter coefficient, activation layer flag, activation layer type. Of course, the above are just a few examples of network parameters and are not limited to them.

[0120] In a possible implementation, the image information may include coefficient hyperparameter feature information and image feature information, and the input feature corresponding to the encoding processing unit is determined based on the current block, and the input feature is processed based on the encoding neural network corresponding to the encoding processing unit to obtain the output feature corresponding to the encoding processing unit, and the image information corresponding to the current block is determined based on the output feature, which may include but is not limited to: in the encoding process of performing feature transformation, the current block may be subjected to feature transformation based on the encoding neural network to obtain the image feature value corresponding to the current block; wherein the image feature value is used to determine the image feature information. In the encoding process of performing coefficient hyperparameter feature transformation, the image feature value may be subjected to coefficient hyperparameter feature transformation based on the encoding neural network to obtain the coefficient hyperparameter feature coefficient value, and the coefficient hyperparameter feature coefficient value is used to determine the coefficient hyperparameter feature information.

[0121] Exemplarily, in the process of determining the coefficient hyperparameter feature information based on the coefficient hyperparameter feature coefficient value, the coefficient hyperparameter feature coefficient value can be quantized to obtain the coefficient hyperparameter feature coefficient quantization value, and the coefficient hyperparameter feature information is determined based on the coefficient hyperparameter feature coefficient quantization value. Wherein, the control parameter can also include first enabling information, and the first enabling information is used to indicate that the first quantization operation has been enabled.

[0122] Exemplarily, in the process of determining the image feature information based on the image feature value, the image feature value can be quantized to obtain the image feature quantization value, and the image feature information can be determined based on the image feature quantization value. Wherein, the control parameter can also include second enabling information, and the second enabling information is used to indicate that the second quantization operation has been enabled.

[0123] Exemplarily, obtaining the control parameters corresponding to the current block may include, but are not limited to: determining neural network information used in the decoding process of inverse transformation of image features of a decoding end device based on the network structure of the encoding neural network used in the encoding process of feature transformation, the neural network information being used to determine the decoding neural network corresponding to the decoding process of inverse transformation of image features of the decoding end device; and / or determining neural network information used in the decoding process of coefficient hyperparameter feature generation of a decoding end device based on the network structure of the encoding neural network used in the encoding process of coefficient hyperparameter feature transformation, the neural network information being used to determine the decoding neural network corresponding to the decoding process of coefficient hyperparameter feature generation of the decoding end device.

[0124] In a possible implementation, the encoding end device may include a control parameter encoding unit, a first feature encoding unit, a second feature encoding unit, a feature conversion unit, and a coefficient super-parameter feature conversion unit; the image information may include coefficient super-parameter feature information and image feature information. Wherein: the control parameter encoding unit encodes the control parameter in the bitstream, the first feature encoding unit encodes the coefficient super-parameter feature information in the bitstream, and the second feature encoding unit encodes the image feature information in the bitstream; when the feature conversion unit is a coding processing unit, the feature conversion unit may perform feature conversion on the current block based on the coding neural network to obtain the image feature value corresponding to the current block; wherein the image feature value is used to determine the image feature information; when the coefficient super-parameter feature conversion unit is a coding processing unit, the coefficient super-parameter feature conversion unit may perform coefficient super-parameter feature conversion on the image feature value based on the coding neural network to obtain the coefficient super-parameter feature coefficient value, and the coefficient super-parameter feature coefficient value is used to determine the coefficient super-parameter feature information.

[0125] Exemplarily, the encoding end device may further include a first quantization unit and a second quantization unit; wherein: the first quantization unit may obtain a coefficient superparameter feature coefficient value from a coefficient superparameter feature transformation unit, quantize the coefficient superparameter feature coefficient value, obtain a coefficient superparameter feature coefficient quantization value, and determine the coefficient superparameter feature information based on the coefficient superparameter feature coefficient quantization value. Wherein, the control parameter may further include first enabling information of the first quantization unit, and the first enabling information is used to indicate that the first quantization unit has been enabled. And, the second quantization unit may obtain an image feature value from a feature transformation unit, quantize the image feature value, obtain an image feature quantization value, and determine the image feature information based on the image feature quantization value; wherein, the control parameter may further include second enabling information of the second quantization unit, and the second enabling information is used to indicate that the second quantization unit has been enabled.

[0126] Exemplarily, obtaining the control parameters corresponding to the current block may include but is not limited to: determining the neural network information corresponding to the image feature inverse transformation unit of the decoding end device based on the network structure of the encoding neural network of the feature transformation unit, and the neural network information is used to determine the decoding neural network corresponding to the image feature inverse transformation unit of the decoding end device; and / or determining the neural network information corresponding to the coefficient super-parameter feature generation unit of the decoding end device based on the network structure of the encoding neural network of the coefficient super-parameter feature transformation unit, and the neural network information is used to determine the decoding neural network corresponding to the coefficient super-parameter feature generation unit of the decoding end device.

[0127] In a possible implementation, the image information may include coefficient hyperparameter feature information and image feature information. When performing the decoding process of coefficient hyperparameter feature generation, the coefficient hyperparameter feature coefficient reconstruction value may be determined based on the coefficient hyperparameter feature information; the coefficient hyperparameter feature coefficient reconstruction value may be inversely transformed based on the decoding neural network to obtain the coefficient hyperparameter feature value; the coefficient hyperparameter feature value may be used to decode the image feature information from the bitstream. When performing the decoding process of image feature inverse transformation, the image feature reconstruction value may be determined based on the image feature information; the image feature reconstruction value may be inversely transformed based on the decoding neural network to obtain the image low-order feature value; the image low-order feature value may be used to obtain the reconstructed image block corresponding to the current block.

[0128] Exemplarily, determining the coefficient hyperparameter feature coefficient reconstruction value based on the coefficient hyperparameter feature information may include but is not limited to: the coefficient hyperparameter feature information may be dequantized to obtain the coefficient hyperparameter feature coefficient reconstruction value. Determining the image feature reconstruction value based on the image feature information may include but is not limited to: the image feature information may be dequantized to obtain the image feature reconstruction value.

[0129] Exemplarily, when performing a quality enhancement decoding process, low-order eigenvalues ​​of the image may be obtained, and the low-order eigenvalues ​​of the image may be enhanced based on a decoding neural network to obtain a reconstructed image block corresponding to the current block.

[0130] Exemplarily, obtaining the control parameters corresponding to the current block may include but is not limited to at least one of the following: determining the neural network information used in the decoding process of the coefficient hyperparameter feature generation of the decoding end device based on the network structure of the decoding neural network used in the encoding process of the coefficient hyperparameter feature generation of the encoding end device, and the neural network information is used to determine the decoding neural network used in the decoding process of the coefficient hyperparameter feature generation of the decoding end device. Based on the network structure of the decoding neural network used in the encoding process of the image feature inverse transformation of the encoding end device, determine the neural network information used in the decoding process of the image feature inverse transformation of the decoding end device, and the neural network information is used to determine the decoding neural network used in the decoding process of the image feature inverse transformation of the decoding end device. Based on the network structure of the decoding neural network used in the encoding process of quality enhancement of the encoding end device, determine the neural network information used in the decoding process of quality enhancement of the decoding end device, and the neural network information is used to determine the decoding neural network used in the decoding process of quality enhancement of the decoding end device.

[0131] In a possible implementation, the encoding end device may include a first feature decoding unit, a second feature decoding unit, a coefficient super-parameter feature generation unit, and an image feature inverse transformation unit; the image information may include coefficient super-parameter feature information and image feature information. Wherein: the first feature decoding unit may decode the coefficient super-parameter feature information from the bitstream, and the second feature decoding unit may decode the image feature information from the bitstream; after determining the coefficient super-parameter feature coefficient reconstruction value based on the coefficient super-parameter feature information, the coefficient super-parameter feature generation unit may perform an inverse transformation operation on the coefficient super-parameter feature coefficient reconstruction value based on the decoding neural network to obtain the coefficient super-parameter feature value; wherein the coefficient super-parameter feature value is used to enable the second feature decoding unit to decode the image feature information from the bitstream; and, to enable the second feature encoding unit to encode the image feature information in the bitstream; after determining the image feature reconstruction value based on the image feature information, the image feature inverse transformation unit may perform an inverse transformation operation on the image feature reconstruction value based on the decoding neural network to obtain the image low-order feature value; wherein the image low-order feature value is used to obtain the reconstructed image block corresponding to the current block.

[0132] Exemplarily, the encoding end device may also include a first dequantization unit and a second dequantization unit; wherein: the first dequantization unit can obtain coefficient hyperparameter feature information from the first feature decoding unit, and dequantize the coefficient hyperparameter feature information to obtain the coefficient hyperparameter feature coefficient reconstruction value; the second dequantization unit can obtain image feature information from the second feature decoding unit, and dequantize the image feature information to obtain the image feature reconstruction value.

[0133] Exemplarily, the encoding end device also includes a quality enhancement unit; the quality enhancement unit obtains low-order eigenvalues ​​of the image from the image feature inverse transformation unit, enhances the low-order eigenvalues ​​of the image based on a decoding neural network, and obtains a reconstructed image block.

[0134] Exemplarily, obtaining the control parameters corresponding to the current block may include but is not limited to at least one of the following: based on the network structure of the decoding neural network of the coefficient super-parameter feature generation unit of the encoding end device, neural network information corresponding to the coefficient super-parameter feature generation unit of the decoding end device may be determined, and the neural network information is used to determine the decoding neural network corresponding to the coefficient super-parameter feature generation unit of the decoding end device; based on the network structure of the decoding neural network of the image feature inverse transformation unit of the encoding end device, neural network information corresponding to the image feature inverse transformation unit of the decoding end device may be determined, and the neural network information is used to determine the decoding neural network corresponding to the image feature inverse transformation unit of the decoding end device; based on the network structure of the decoding neural network of the quality enhancement unit of the encoding end device, neural network information corresponding to the quality enhancement unit of the decoding end device may be determined, and the neural network information is used to determine the decoding neural network corresponding to the quality enhancement unit of the decoding end device.

[0135] In a possible implementation, before determining the input features corresponding to the encoding processing unit based on the current block, the current image may be divided into N non-overlapping image blocks, where N is a positive integer; the borders of each image block may be filled to obtain a border-filled image block; wherein, when the borders of each image block are filled, the filling value may not depend on the reconstructed pixel values ​​of adjacent image blocks; and N current blocks may be generated based on the border-filled image block.

[0136] In a possible implementation, before determining the input features corresponding to the encoding processing unit based on the current block, the current image can also be divided into multiple basic blocks, and each basic block includes at least one image block; the boundary of each image block can be filled to obtain an image block after the boundary is filled; wherein, when the boundary of each image block is filled, the filling value of the image block does not depend on the reconstructed pixel values ​​of other image blocks in the same basic block, and is allowed to depend on the reconstructed pixel values ​​of image blocks in different basic blocks; multiple current blocks can be generated based on the image block after the boundary is filled.

[0137] Exemplarily, the above execution order is only for the convenience of describing the examples given. In practical applications, the execution order between the steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0138] As can be seen from the above technical solutions, in the embodiment of the present application, the control parameters corresponding to the current block can be decoded from the code stream, the neural network information corresponding to the decoding processing unit can be obtained from the control parameters, and the decoding neural network corresponding to the decoding processing unit can be generated based on the neural network information, and then the image decoding can be realized based on the decoding neural network to improve the decoding performance. Image encoding can be realized based on the encoding neural network corresponding to the encoding processing unit to improve the encoding performance. A deep learning network (such as a decoding neural network and an encoding neural network, etc.) can be used to encode and decode the image, and the neural network information is transmitted through the code stream, and then the decoding neural network corresponding to the decoding processing unit is generated based on the neural network information, thereby solving the problems of poor stability, poor generalization and high complexity, that is, good stability, good generalization and low complexity. A solution for dynamic adjustment of encoding and decoding complexity can be provided, which has better encoding performance than the framework of a single deep learning network. Since each current block corresponds to a control parameter, the neural network information obtained from the control parameter is the neural network information for the current block, so that a decoding neural network is generated for each current block, that is, the decoding neural network of different current blocks may be the same or different, so that the decoding neural network at the block level, that is, the decoding neural network can be changed and adjusted.

[0139] Embodiment 3: In the embodiment of the present application, a neural network-based image encoding method and image decoding method with variable and adjustable bit rate are proposed, which can achieve high parallelism of image blocks and controllable and adjustable bit rate, see Figure 5A , which is a schematic diagram of an image encoding method and an image decoding method, illustrates the encoding process of the image encoding method and the decoding process of the image decoding method.

[0140] Exemplarily, for the image encoding method based on a neural network, the encoding process may include the following steps:

[0141] Step S11, the blocking unit divides the current image (ie, the original image) into N non-overlapping image blocks (ie, original image blocks, which can be recorded as original image block 1, original image block 2, ..., original image block N), where N is a positive integer.

[0142] Step S12: performing border filling on each original image block to obtain a border-filled image block, and generating N current blocks based on the border-filled image block, that is, performing border filling on the N original image blocks respectively to obtain N border-filled image blocks, and using the N border-filled image blocks as the N current blocks.

[0143] For example, see Figure 5B As shown, when filling the border of each original image block, the filling value may not depend on the reconstructed pixel value of the adjacent image block, thereby ensuring that each original image block can be independently and parallelly encoded, thereby improving the encoding performance.

[0144] Step S13: The encoding unit determines the encoding parameters of the current block based on the information of the encoded block. The encoding parameters are used to control the encoding rate of the current block (such as parameters such as quantization step size), and there is no restriction on the encoding parameters.

[0145] Step S14: The control unit writes the control parameters required by the decoding end and which cannot be derived into the bitstream.

[0146] Step S15: input the filled image block (i.e., the current block) into a coding unit based on a neural network, and the coding unit encodes the current block based on the coding parameters and outputs a code stream of the current block. Exemplarily, when the coding unit encodes the current block based on the coding parameters, the coding unit may encode the current block using a neural network.

[0147] Exemplarily, for the image decoding method based on a neural network, the decoding process may include the following steps:

[0148] Step S21: The decoding end decodes the control parameters that are required for the current block and cannot be derived from the bitstream.

[0149] Step S22, based on the control parameters and the code stream of the current block, a reconstructed image block corresponding to the current block is obtained through a decoding unit based on a neural network, that is, the current block is decoded to obtain a reconstructed image block, such as reconstructed image block 1 corresponding to original image block 1, reconstructed image block 2 corresponding to original image block 2, ..., reconstructed image block N corresponding to original image block N.

[0150] Exemplarily, when the decoding unit decodes the current block, a neural network may be used to decode the current block.

[0151] Step S23, based on the control parameters, determine whether to perform a filtering process on a current block. If yes, the filtering merging unit performs a filtering process based on the information of the current block and at least one adjacent reconstructed image block to obtain a filtered image block.

[0152] Step S24: merging the filtered image blocks to obtain a reconstructed image.

[0153] In a possible implementation, when filling the boundaries of the original image block, the filling value may be a preset filling value, which may be a default value agreed upon by the codec (such as 0, or 1<<(1-depth), etc., where depth is the bit depth, such as 8, 10, 12, etc.), or a value passed to the decoding end through high-level syntax encoding, or may be obtained by performing operations such as mirroring and nearest neighbor copying based on the pixels of the current block, and there is no restriction on the method for obtaining this filling value.

[0154] In one possible implementation, when filling the borders of the original image block, the size of the surrounding block filling extension may be a default value agreed upon by the codec (such as 1, 2, 4, etc.), or a value related to the size of the current block, or a value passed to the decoding end via high-level syntax encoding, and there is no restriction on the size of the surrounding block filling extension.

[0155] Embodiment 4: In the embodiment of the present application, a neural network-based image encoding method and an image decoding method with variable and adjustable bit rate are proposed, which can achieve high parallelism of image blocks and controllable and adjustable bit rate. On this basis, information between adjacent blocks (such as reconstructed pixels of adjacent blocks, etc.) can also be used. See Figure 5C , which is a schematic diagram of an image encoding method and an image decoding method, illustrates the encoding process of the image encoding method and the decoding process of the image decoding method.

[0156] Exemplarily, for the image encoding method based on a neural network, the encoding process may include the following steps:

[0157] Step S31, the blocking unit divides the current image (i.e., the original image) into multiple basic blocks, each basic block includes at least one image block, taking M basic blocks as an example, M is a positive integer, and the M basic blocks include N non-overlapping image blocks (i.e., original image blocks, denoted as original image block 1, original image block 2, ..., original image block N), where N is a positive integer.

[0158] Step S32, performing border filling on each original image block to obtain a border-filled image block, and generating N current blocks based on the border-filled image block, that is, performing border filling on the N original image blocks respectively to obtain N border-filled image blocks, and using the N border-filled image blocks as the N current blocks.

[0159] Exemplarily, when filling the boundary of each original image block, the filling value of the original image block does not depend on the reconstructed pixel values ​​of other original image blocks in the same basic block, and is allowed to depend on the reconstructed pixel values ​​of original image blocks in different basic blocks.

[0160] The concept of basic blocks is introduced in this embodiment. Figure 5D As shown, each basic block includes at least one image block (i.e., the original image block). The image blocks in the basic block do not refer to each other, but can refer to the reconstruction information of other image blocks in different basic blocks. Figure 5D As shown, for image block 1, the image block on its left side (i.e., the adjacent block on the left side of image block 1) is in the same basic block as image block 1, therefore, the reconstruction information of the image block cannot be used as the filling value of image block 1, but the image block is filled with a preset filling value. For image block 1, the image block on its upper side (i.e., the adjacent block on the upper side of image block 1) is in a different basic block from image block 1, therefore, the reconstruction information of the image block can be used as the filling value of image block 1. When the reconstruction information of the image block is used as the filling value of image block 1, the reconstruction value of the image block can be used, and the reconstruction value of the image block before filtering can be used.

[0161] Obviously, by introducing basic blocks, it is possible to ensure parallelism (the image blocks within the basic blocks can be encoded and decoded in parallel) and also help improve performance (the reconstruction information of the image blocks within adjacent basic blocks can be utilized).

[0162] Step S33: The encoding unit determines the encoding parameters of the current block based on the information of the encoded block. The encoding parameters are used to control the encoding rate of the current block (such as parameters such as quantization step size), and there is no restriction on the encoding parameters.

[0163] Step S34: The control unit writes the control parameters required by the decoding end and which cannot be derived into the bitstream.

[0164] Step S35: input the filled image block (i.e., the current block) into a coding unit based on a neural network, and the coding unit encodes the current block based on the coding parameters and outputs a code stream of the current block. Exemplarily, when the coding unit encodes the current block based on the coding parameters, the coding unit may encode the current block using a neural network.

[0165] Exemplarily, for the image decoding method based on a neural network, the decoding process may include the following steps:

[0166] Step S41: The decoding end decodes the control parameters that are required for the current block and cannot be derived from the bitstream.

[0167] Step S42, based on the control parameters and the code stream of the current block, a reconstructed image block corresponding to the current block is obtained through a decoding unit based on a neural network, that is, the current block is decoded to obtain a reconstructed image block, such as reconstructed image block 1 corresponding to original image block 1, reconstructed image block 2 corresponding to original image block 2, ..., reconstructed image block N corresponding to original image block N.

[0168] Step S43, based on the control parameters, determine whether to perform a filtering process on a current block. If yes, the filtering merging unit performs a filtering process based on the information of the current block and at least one adjacent reconstructed image block to obtain a filtered image block.

[0169] Step S44: merging the filtered image blocks to obtain a reconstructed image.

[0170] In a possible implementation, when filling the boundaries of the original image block, the filling value may be a preset filling value, which may be a default value agreed upon by the codec (such as 0, or 1<<(1-depth), etc., where depth is the bit depth, such as 8, 10, 12, etc.), or a value passed to the decoding end through high-level syntax encoding, or may be obtained by performing operations such as mirroring and nearest neighbor copying based on the pixels of the current block, and there is no restriction on the method for obtaining this filling value.

[0171] In one possible implementation, when filling the borders of the original image block, the size of the surrounding block filling extension may be a default value agreed upon by the codec (such as 1, 2, 4, etc.), or a value related to the size of the current block, or a value passed to the decoding end via high-level syntax encoding, and there is no restriction on the size of the surrounding block filling extension.

[0172] In a possible implementation, the number of image blocks included in the basic block can be a default value agreed upon by the codec (such as 1, 4, 16, etc.), or a value related to the size of the current image, or a value passed to the decoding end through high-level syntax encoding. There is no limit on the number of image blocks in this basic block, and it can be selected according to actual needs.

[0173] In a possible implementation manner, in Example 3 and Example 4, for the block division process of the original image (step S11 and step S31), the original image may be subjected to image domain transformation, and the image after image domain transformation may be divided into blocks. Accordingly, the filtered image blocks may be merged into one image, and the image may be subjected to image domain inverse transformation, and a reconstructed image may be obtained based on the image after image domain inverse transformation, see Figure 5E As shown. Exemplarily, the image domain transformation of the original image can be a transformation process from an RGB domain image to a YUV domain image (the corresponding image domain inverse transformation is the inverse transformation process from a YUV domain image to an RGB domain image), or it can be an introduction of a wavelet transform or Fourier transform process to generate an image in a new domain (the corresponding image domain inverse transformation is the inverse wavelet transform or inverse Fourier transform process). The image domain transformation process can be implemented using a neural network or a non-neural network, and there is no limitation on this image domain transformation process.

[0174] Embodiment 5: In the embodiment of the present application, a neural network-based image decoding method is proposed, which can be applied to a decoding end (also called a video decoder), see Fig. 6A As shown, it is a schematic diagram of the structure of the decoding end, and the decoding end may include a control parameter decoding unit, a first feature decoding unit, a second feature decoding unit, a coefficient super parameter feature generation unit, an image feature inverse transformation unit, a first inverse quantization unit, a second inverse quantization unit, and a quality enhancement unit. The first inverse quantization unit, the second inverse quantization unit, and the quality enhancement unit are optional units, and in some scenarios, the process of these optional units can be turned off or skipped.

[0175] In this embodiment, for each current block (i.e., image block), the bitstream corresponding to the current block includes three parts: bitstream 0 (bitstream containing control parameters), bitstream 1 (bitstream containing coefficient super-parameter feature information), and bitstream 2 (bitstream containing image feature information). The coefficient super-parameter feature information and the image feature information can be collectively referred to as image information.

[0176] Exemplarily, the neural network-based image decoding method in this embodiment may include the following steps:

[0177] Step S51, decode the code stream 0 corresponding to the current block to obtain the control parameters corresponding to the current block. For example, the control parameter decoding unit can decode the code stream 0 corresponding to the current block to obtain the control parameters corresponding to the current block, that is, the control parameter decoding unit can decode the control parameters from the code stream 0, and the control parameters may include the control parameters of the first dequantization unit, the control parameters of the second dequantization unit, the control parameters of the coefficient hyperparameter feature generation unit, the control parameters of the image feature inverse transformation unit, and the control parameters of the quality enhancement unit. For the content of the control parameters, please refer to the subsequent embodiments, which will not be repeated here.

[0178] Step S52: Decode the bitstream 1 corresponding to the current block to obtain the coefficient super-parameter feature information corresponding to the current block. For example, the first feature decoding unit can decode the bitstream 1 corresponding to the current block to obtain the coefficient super-parameter feature information corresponding to the current block, that is, the first feature decoding unit can decode the coefficient super-parameter feature information from the bitstream 1.

[0179] Step S53: Determine the coefficient hyper-parameter characteristic coefficient reconstruction value based on the coefficient hyper-parameter characteristic information.

[0180] Exemplarily, if the control parameter includes first enabling information corresponding to the first dequantization unit, the first enabling information may indicate enabling the first dequantization unit (i.e., enabling the first dequantization unit to perform the first dequantization operation), and the first enabling information may also indicate disabling the first dequantization unit. For example, if the first enabling information is a first value, it indicates enabling the first dequantization unit, and if the first enabling information is a second value, it indicates disabling the first dequantization unit.

[0181] Exemplarily, if the first enabling information indicates that the first dequantization unit is enabled, the coefficient hyperparameter feature information may be the coefficient hyperparameter feature coefficient quantization value C_q, and the first dequantization unit may dequantize the coefficient hyperparameter feature coefficient quantization value C_q to obtain the coefficient hyperparameter feature coefficient reconstruction value C'. If the first enabling information indicates that the first dequantization unit is not enabled, the coefficient hyperparameter feature information may be the coefficient hyperparameter feature coefficient reconstruction value C', that is, the coefficient hyperparameter feature coefficient reconstruction value C' is directly decoded from bitstream 1.

[0182] Step S54, perform an inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value C' to obtain the coefficient hyperparameter feature value P. For example, the coefficient hyperparameter feature generation unit performs an inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value C' to obtain the coefficient hyperparameter feature value P, such as the coefficient hyperparameter feature generation unit performs an inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value C' based on the decoding neural network to obtain the coefficient hyperparameter feature value P.

[0183] In one possible implementation, the coefficient hyperparameter feature generation unit can obtain the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit from the control parameters, and generate a decoding neural network 1 corresponding to the coefficient hyperparameter feature generation unit based on the neural network information 1.

[0184] In addition, the coefficient hyperparameter feature generation unit can determine the input features corresponding to the coefficient hyperparameter feature generation unit (such as the coefficient hyperparameter feature coefficient reconstruction value C'), and process the input features based on the decoding neural network 1 (such as performing an inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value C') to obtain the output features corresponding to the coefficient hyperparameter feature generation unit (such as the coefficient hyperparameter feature value P).

[0185] Exemplarily, the neural network information 1 may include basic layer information and enhanced layer information, and the coefficient super-parameter feature generation unit may determine the basic layer corresponding to the coefficient super-parameter feature generation unit based on the basic layer information, and determine the enhanced layer corresponding to the coefficient super-parameter feature generation unit based on the enhanced layer information. The coefficient super-parameter feature generation unit may generate a decoding neural network 1 corresponding to the coefficient super-parameter feature generation unit based on the basic layer and the enhanced layer. For example, the basic layer and the enhanced layer may be combined to obtain a decoding neural network 1.

[0186] After obtaining the decoding neural network 1, the coefficient super-parameter feature generation unit can perform an inverse transformation operation on the coefficient super-parameter feature coefficient reconstruction value C' through the decoding neural network 1 to obtain the coefficient super-parameter feature value P. There is no restriction on this inverse transformation operation process.

[0187] Step S55, decode the code stream 2 corresponding to the current block to obtain the image feature information corresponding to the current block. For example, the second feature decoding unit can decode the code stream 2 corresponding to the current block to obtain the image feature information corresponding to the current block, that is, the second feature decoding unit can decode the image feature information from the code stream 2. When decoding the code stream 2 corresponding to the current block, the second feature decoding unit can use the coefficient hyperparameter feature value P to decode the code stream 2 corresponding to the current block, and there is no restriction on this decoding process.

[0188] Step S56: Determine an image feature reconstruction value based on the image feature information.

[0189] Exemplarily, if the control parameter includes second enabling information corresponding to the second inverse quantization unit, the second enabling information may indicate enabling the second inverse quantization unit (i.e., enabling the second inverse quantization unit to perform the second inverse quantization operation), and the second enabling information may also indicate disabling the second inverse quantization unit. For example, if the second enabling information is a first value, it indicates enabling the second inverse quantization unit, and if the second enabling information is a second value, it indicates disabling the second inverse quantization unit.

[0190] Exemplarily, if the second enabling information indicates that the second inverse quantization unit is enabled, the image feature information may be the image feature quantization value F_q, and the second inverse quantization unit may obtain the image feature quantization value F_q, and inversely quantize the image feature quantization value F_q to obtain the image feature reconstruction value F'. If the second enabling information indicates that the second inverse quantization unit is not enabled, the image feature information may be the image feature reconstruction value F', that is, the image feature reconstruction value F' is directly decoded from the bitstream 2.

[0191] Step S57, perform an inverse transformation operation on the image feature reconstruction value F' to obtain the image low-order feature value LF. For example, the image feature inverse transformation unit performs an inverse transformation operation on the image feature reconstruction value F' to obtain the image low-order feature value LF, such as performing an inverse transformation operation on the image feature reconstruction value F' based on a decoding neural network to obtain the image low-order feature value LF.

[0192] In a possible implementation, the image feature inverse transformation unit may obtain the neural network information 2 corresponding to the image feature inverse transformation unit from the control parameters, and generate a decoding neural network 2 corresponding to the image feature inverse transformation unit based on the neural network information 2. The image feature inverse transformation unit may determine the input feature corresponding to the image feature inverse transformation unit (such as the image feature reconstruction value F'), and process the input feature based on the decoding neural network 2 (such as performing an inverse transformation operation on the image feature reconstruction value F'), to obtain the output feature corresponding to the image feature inverse transformation unit (such as the image low-order feature value LF).

[0193] Exemplarily, the neural network information 2 may include basic layer information and enhanced layer information. The image feature inverse transformation unit may determine the basic layer corresponding to the image feature inverse transformation unit based on the basic layer information, and determine the enhanced layer corresponding to the image feature inverse transformation unit based on the enhanced layer information. The image feature inverse transformation unit may generate a decoding neural network 2 corresponding to the image feature inverse transformation unit based on the basic layer and the enhanced layer. For example, the basic layer and the enhanced layer may be combined to obtain a decoding neural network 2.

[0194] After obtaining the decoding neural network 2, the image feature inverse transformation unit can perform an inverse transformation operation on the image feature reconstruction value F' through the decoding neural network 2 to obtain the image low-order feature value LF. There is no restriction on this inverse transformation operation process.

[0195] Step S58: Determine the reconstructed image block I corresponding to the current block based on the image low-order eigenvalue LF.

[0196] Exemplarily, if the control parameter includes third enabling information corresponding to the quality enhancement unit, the third enabling information may indicate enabling the quality enhancement unit (ie, enabling the quality enhancement unit to perform a quality enhancement operation), and the third enabling information may also indicate disabling the quality enhancement unit. For example, if the third enabling information is a first value, it may indicate enabling the quality enhancement unit, and if the third enabling information is a second value, it may indicate disabling the quality enhancement unit.

[0197] Exemplarily, if the third enabling information indicates that the quality enhancement unit is enabled, the quality enhancement unit obtains the image low-order eigenvalue LF, performs enhancement processing on the image low-order eigenvalue LF, and obtains the reconstructed image block I corresponding to the current block. If the third enabling information indicates that the quality enhancement unit is not enabled, the image low-order eigenvalue LF is used as the reconstructed image block I corresponding to the current block.

[0198] Exemplarily, when the quality enhancement unit performs enhancement processing on the low-order eigenvalue LF of the image, the quality enhancement unit may perform enhancement processing on the low-order eigenvalue LF of the image based on a decoding neural network to obtain a reconstructed image block I corresponding to the current block.

[0199] In a possible implementation, the quality enhancement unit may obtain the neural network information 3 corresponding to the quality enhancement unit from the control parameters, and generate a decoding neural network 3 corresponding to the quality enhancement unit based on the neural network information 3.

[0200] The quality enhancement unit can determine the input features corresponding to the quality enhancement unit (such as the low-order eigenvalues ​​LF of the image), and process the input features based on the decoding neural network 3 (such as enhancing the low-order eigenvalues ​​LF of the image) to obtain the output features corresponding to the quality enhancement unit (such as the reconstructed image block I corresponding to the current block).

[0201] Exemplarily, the neural network information 3 may include basic layer information and enhanced layer information. The quality enhancement unit may determine the basic layer corresponding to the quality enhancement unit based on the basic layer information, and determine the enhanced layer corresponding to the quality enhancement unit based on the enhanced layer information. The quality enhancement unit may generate a decoding neural network 3 corresponding to the quality enhancement unit based on the basic layer and the enhanced layer. For example, the basic layer and the enhanced layer may be combined to obtain a decoding neural network 3.

[0202] After obtaining the decoding neural network 3, the quality enhancement unit can enhance the low-order eigenvalue LF of the image through the decoding neural network 3 to obtain the reconstructed image block I corresponding to the current block, and there is no restriction on this enhancement process.

[0203] Embodiment 6: In the embodiment of the present application, a neural network-based image decoding method is proposed, which can be applied to the decoding end, see Figure 6B As shown, it is a schematic diagram of the structure of the decoding end, and the decoding end may include a control parameter decoding unit, a first feature decoding unit, a second feature decoding unit, a coefficient super parameter feature generation unit, and an image feature inverse transformation unit. In this embodiment, for each current block (i.e., image block), the bitstream corresponding to the current block includes three parts: bitstream 0 (bitstream containing control parameters), bitstream 1 (bitstream containing coefficient super parameter feature information), and bitstream 2 (bitstream containing image feature information).

[0204] Exemplarily, the neural network-based image decoding method in this embodiment may include the following steps:

[0205] Step S61: The control parameter decoding unit decodes the code stream 0 corresponding to the current block to obtain the control parameters corresponding to the current block.

[0206] Step S62: The first feature decoding unit decodes the code stream 1 corresponding to the current block to obtain coefficient super-parameter feature information corresponding to the current block. The coefficient super-parameter feature information may be a coefficient super-parameter feature coefficient reconstruction value C'.

[0207] Step S63, the coefficient super-parameter feature generation unit performs an inverse transformation operation on the coefficient super-parameter feature coefficient reconstruction value C' to obtain the coefficient super-parameter feature value P, such as performing an inverse transformation operation on the coefficient super-parameter feature coefficient reconstruction value C' based on a decoding neural network to obtain the coefficient super-parameter feature value P.

[0208] Step S64: The second feature decoding unit decodes the bitstream 2 corresponding to the current block to obtain image feature information corresponding to the current block, which may be an image feature reconstruction value F'. Exemplarily, the second feature decoding unit may obtain a coefficient hyperparameter feature value P, and use the coefficient hyperparameter feature value P to decode the bitstream 2 corresponding to the current block to obtain an image feature reconstruction value F'.

[0209] Step S65, the image feature inverse transformation unit performs an inverse transformation operation on the image feature reconstruction value F' to obtain the image low-order feature value LF, such as performing an inverse transformation operation on the image feature reconstruction value F' based on a decoding neural network to obtain the image low-order feature value LF.

[0210] Step S66: Determine the reconstructed image block I corresponding to the current block based on the image low-order eigenvalue LF.

[0211] For example, the low-order eigenvalue LF of the image can be directly used as the reconstructed image block I. Alternatively, if the decoding end device also includes a quality enhancement unit, the quality enhancement unit performs enhancement processing on the low-order eigenvalue LF of the image to obtain the reconstructed image block I corresponding to the current block, such as performing enhancement processing on the low-order eigenvalue LF of the image based on a decoding neural network to obtain the reconstructed image block I.

[0212] Embodiment 7: For Embodiment 5 and Embodiment 6, the first feature decoding unit may decode the bitstream 1 corresponding to the current block to obtain coefficient hyperparameter feature information corresponding to the current block. For example, the first feature decoding unit includes at least one coefficient decoding module. In one possible implementation, the coefficient decoding module may use an entropy decoding method to perform coefficient decoding, that is, use an entropy decoding method to decode the bitstream 1 corresponding to the current block to obtain coefficient hyperparameter feature information corresponding to the current block.

[0213] Exemplarily, the entropy decoding method may include, but is not limited to, entropy decoding methods such as CAVLC (Context-Adaptive VariablebeLength Coding) or CABAC (Context-based Adaptive BinaryArithmetic Coding), and there is no limitation on this.

[0214] Exemplarily, when the entropy decoding method is used to decode the coefficients, the probability model of the entropy decoding can adopt a preset probability model, and the preset probability model can be configured according to actual needs, and there is no restriction on the preset probability model. For example, based on the preset probability model, the coefficient decoding module can adopt the entropy decoding method to decode the coefficients.

[0215] Embodiment 8: With respect to embodiment 5, the first dequantization unit may dequantize the coefficient super-parameter characteristic coefficient quantization value C_q (i.e., the coefficient super-parameter characteristic information) to obtain the coefficient super-parameter characteristic coefficient reconstruction value C'. For example, the first dequantization unit may not exist, or, if the first dequantization unit exists, the first dequantization unit may be selectively skipped based on control parameters (such as high-level syntax, such as first enabling information, etc.), or the first dequantization unit may be enabled based on the control parameters.

[0216] Exemplarily, if the first dequantization unit does not exist, the coefficient hyperparameter feature coefficient reconstruction value C' is the same as the coefficient hyperparameter feature coefficient quantization value C_q, that is, it is not necessary to dequantize the coefficient hyperparameter feature coefficient quantization value C_q. If the first dequantization unit is selectively skipped based on the control parameter, the coefficient hyperparameter feature coefficient reconstruction value C' is the same as the coefficient hyperparameter feature coefficient quantization value C_q, that is, it is not necessary to dequantize the coefficient hyperparameter feature coefficient quantization value C_q. If it is determined based on the control parameter that the first dequantization unit is enabled, however, the step parameter qstep corresponding to the coefficient hyperparameter feature coefficient quantization value C_q is 1, the coefficient hyperparameter feature coefficient reconstruction value C' is the same as the coefficient hyperparameter feature coefficient quantization value C_q, that is, it is not necessary to dequantize the coefficient hyperparameter feature coefficient quantization value C_q.

[0217] Exemplarily, if it is determined based on the control parameter that the first dequantization unit is enabled and the step parameter qstep corresponding to the coefficient hyperparameter feature coefficient quantization value C_q is not 1, the first dequantization unit can dequantize the coefficient hyperparameter feature coefficient quantization value C_q based on the control parameter (such as quantization-related parameters) to obtain the coefficient hyperparameter feature coefficient reconstruction value C'. For example, the first dequantization unit performs the following operations: obtain the quantization-related parameters corresponding to the coefficient hyperparameter feature coefficient quantization value C_q from the control parameter (the control parameter is included in the bitstream and the control parameter can include quantization-related parameters), such as the step parameter qstep or the quantization parameter qp. Determine the multiplication factor mult and the shift factor shift corresponding to the coefficient hyperparameter feature coefficient quantization value C_q based on the step parameter qstep or the quantization parameter qp; assuming that the coefficient hyperparameter feature coefficient quantization value C_q is Coff_hyper and the coefficient hyperparameter feature coefficient reconstruction value C' is Coff_hyper_rec, then Coff_hyper_rec = (Coff_hyper * mult) << shift. In summary, when dequantizing the coefficient hyperparameter feature coefficient quantization value C_q, the above formula can be used to obtain the coefficient hyperparameter feature coefficient reconstruction value C'.

[0218] It should be noted that for the quantization-related parameters (such as the step parameter qstep) corresponding to the coefficient hyperparameter feature coefficient quantization value C_q, it can include: 1) The same step parameter qstep is used for each coefficient hyperparameter feature quantization value of each feature channel; 2) Different step parameters qstep are used for the coefficient hyperparameter feature quantization values of each feature channel, however, the same step parameter qstep is used for each coefficient hyperparameter feature quantization value within the feature channel; 3) Different step parameters qstep are used for each coefficient hyperparameter feature quantization value of each feature channel. In the above process, the step parameter qstep can also be referred to as the quantization step.

[0219] Embodiment 9: With respect to Embodiment 5 and Embodiment 6, the coefficient super-parameter feature generation unit may perform an inverse transformation operation on the coefficient super-parameter feature coefficient reconstruction value C' based on the decoding neural network to obtain the coefficient super-parameter feature value P. In a possible implementation manner, see Figure 6C As shown, the coefficient hyperparameter feature generation unit may include a decoding neural network 1, which may include a basic layer and an enhancement layer. The coefficient hyperparameter feature coefficient reconstruction value C' serves as an input feature of the decoding neural network 1, and the coefficient hyperparameter feature value P serves as an output feature of the decoding neural network 1. The decoding neural network 1 is used to perform an inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value C'.

[0220] In this embodiment, the decoding neural network 1 is divided into a basic layer and an enhancement layer. The basic layer may include at least one network layer, or the basic layer may not include a network layer, that is, the basic layer is empty. The enhancement layer may include at least one network layer, or the enhancement layer may not include a network layer, that is, the enhancement layer is empty. It should be noted that for multiple network layers in the decoding neural network 1, multiple network layers can be divided into basic layers and enhancement layers according to actual needs. For example, the first M1 network layers are used as basic layers and the remaining network layers are used as enhancement layers, or the first M2 network layers are used as enhancement layers and the remaining network layers are used as basic layers, or the following M3 network layers are used as basic layers and the remaining network layers are used as enhancement layers, or the following M4 network layers are used as enhancement layers and the remaining network layers are used as basic layers, or the odd-numbered network layers are used as basic layers and the remaining network layers are used as enhancement layers, or the even-numbered network layers are used as basic layers and the remaining network layers are used as enhancement layers. Of course, the above are just a few examples, and there is no limitation on this division method.

[0221] For example, a network layer with a fixed network structure can be used as a basic layer, and a network layer with a non-fixed network structure can be used as an enhanced layer. For example, for a network layer in the decoding neural network 1, if the network layer uses the same network structure when decoding multiple image blocks, the network layer is used as a network layer with a fixed network structure, and the network layer is used as a basic layer. For another example, for a network layer in the decoding neural network 1, if the network layer uses different network structures when decoding multiple image blocks, the network layer is used as a network layer with a non-fixed network structure, and the network layer is used as an enhanced layer.

[0222] Exemplarily, for the decoding neural network 1, the size of the output feature may be larger than the size of the input feature, or the size of the output feature may be equal to the size of the input feature, or the size of the output feature may be smaller than the size of the input feature.

[0223] For the decoding neural network 1, the base layer and the enhancement layer include at least one deconvolution layer. For example, the base layer includes at least one deconvolution layer, and the enhancement layer may include at least one deconvolution layer, or may not include a deconvolution layer. Alternatively, the enhancement layer includes at least one deconvolution layer, and the base layer may include at least one deconvolution layer, or may not include a deconvolution layer.

[0224] Exemplarily, for the decoding neural network 1, it may include but is not limited to deconvolution layers and activation layers, etc., and there is no limitation on this. For example, the decoding neural network 1 includes 1 deconvolution layer with a stride of 2, 1 activation layer, 1 deconvolution layer with a stride of 1, and 1 activation layer in sequence. All of the above network layers can be used as basic layers, that is, the basic layer includes 1 deconvolution layer with a stride of 2, 1 activation layer, 1 deconvolution layer with a stride of 1, and 1 activation layer. In this case, the enhancement layer is empty. Of course, some network layers can also be used as enhancement layers, and there is no limitation on this. For another example, the decoding neural network 1 includes 1 deconvolution layer with a stride of 2 and 1 deconvolution layer with a stride of 1 in sequence. All of the above network layers can be used as basic layers, that is, the basic layer includes 1 deconvolution layer with a stride of 2 and 1 deconvolution layer with a stride of 1. In this case, the enhancement layer is empty. Of course, some network layers can also be used as enhancement layers, and there is no limitation on this. For another example, the decoding neural network 1 includes, in sequence, 1 deconvolution layer with a stride of 2, 1 activation layer, 1 deconvolution layer with a stride of 2, 1 activation layer, 1 deconvolution layer with a stride of 1, and 1 activation layer. All of the above network layers can be used as basic layers, that is, the basic layer includes 1 deconvolution layer with a stride of 2, 1 activation layer, 1 deconvolution layer with a stride of 2, 1 activation layer, 1 deconvolution layer with a stride of 1, and 1 activation layer. In this case, the enhancement layer is empty. Of course, some network layers can also be used as enhancement layers, and there is no limitation on this. Of course, the above are just a few examples and there is no limitation on this.

[0225] In a possible implementation, a network layer of a default network structure can be configured for the coefficient hyperparameter feature generation unit (the network layer of the default network structure can be composed of at least one network layer), and the network parameters related to the network layer of the default network structure are all fixed. For example, the network parameters may include but are not limited to at least one of the following: the number of neural network layers, the deconvolution layer flag, the number of deconvolution layers, the quantization step size of each deconvolution layer, the number of channels of each deconvolution layer, the size of the convolution kernel, the number of filters, the filter size index, the filter coefficient zero flag, the filter coefficient, the number of activation layers, the activation layer flag, the activation layer type, that is, the above network parameters are all fixed. For example, in the network layer of the default network structure, the number of deconvolution layers is fixed, the number of activation layers is fixed, the number of channels of each deconvolution layer is fixed, the size of the convolution kernel is fixed, the filter coefficient is fixed, etc., such as the number of channels of the deconvolution layer is 4, 8, 16, 32, 64, 128, or 256, etc., and the size of the convolution kernel is 1*1, 3*3 or 5x5, etc. Obviously, since the network parameters in the network layer of the default network structure are fixed and known, the network layer of the default network structure can be directly obtained.

[0226] In one possible implementation, a prefabricated neural network pool may be configured for the coefficient hyperparameter feature generation unit, and the prefabricated neural network pool may include at least one network layer of a prefabricated network structure (the network layer of the prefabricated network structure may be composed of at least one network layer), and the network parameters related to the network layer of the prefabricated network structure may be configured according to actual needs. For example, the network parameters may include but are not limited to at least one of the following: the number of neural network layers, the deconvolution layer flag, the number of deconvolution layers, the quantization step of each deconvolution layer, the number of channels of each deconvolution layer, the size of the convolution kernel, the number of filters, the filter size index, the filter coefficient zero flag, the filter coefficient, the number of activation layers, the activation layer flag, and the activation layer type. In other words, the above network parameters may be configured according to actual needs.

[0227] For example, the prefabricated neural network pool may include the network layers of the prefabricated network structure s1, the network layers of the prefabricated network structure s2, and the network layers of the prefabricated network structure s3. Among them, for the network layers of the prefabricated network structure s1, the number of deconvolution layers, the number of activation layers, the number of channels of each deconvolution layer, the size of the convolution kernel, the filter coefficient and other network parameters can be preconfigured. After all network parameters are configured, the network layers of the prefabricated network structure s1 can be obtained. Similarly, the network layers of the prefabricated network structure s2 and the network layers of the prefabricated network structure s3 can be obtained, which will not be repeated here.

[0228] In a possible implementation, a network layer of a variable network structure can be dynamically generated for a coefficient super-parameter feature generation unit based on network parameters (a network layer of a variable network structure can be composed of at least one network layer). The network parameters related to the network layer of the variable network structure are dynamically generated by the encoding end, rather than pre-configured. For example, the encoding end can send the network parameters corresponding to the coefficient super-parameter feature generation unit to the decoding end. The network parameters may include but are not limited to at least one of the following: the number of neural network layers, the deconvolution layer flag, the number of deconvolution layers, the quantization step size of each deconvolution layer, the number of channels of each deconvolution layer, the size of the convolution kernel, the number of filters, the filter size index, the filter coefficient zero flag, the filter coefficient, the number of activation layers, the activation layer flag, and the activation layer type. The decoding end dynamically generates a network layer of a variable network structure based on the above network parameters. For example, the encoding end encodes the number of deconvolution layers, the number of activation layers, the number of channels of each deconvolution layer, the size of the convolution kernel, the filter coefficient and other network parameters in the bitstream. The decoding end can parse the above network parameters from the bitstream and generate a network layer of a variable network structure based on these network parameters. There is no restriction on this generation process.

[0229] In a possible implementation, the decoding neural network 1 can be divided into a basic layer and an enhancement layer. The combination of the basic layer and the enhancement layer may include but is not limited to the following methods: Method 1: The basic layer adopts a network layer with a default network structure, and the enhancement layer adopts a network layer with a default network structure, and the basic layer and the enhancement layer form a decoding neural network 1. Method 2: The basic layer adopts a network layer with a default network structure, and the enhancement layer adopts a network layer with a prefabricated network structure, and the basic layer and the enhancement layer form a decoding neural network 1. Method 3: The basic layer adopts a network layer with a default network structure, and the enhancement layer adopts a network layer with a variable network structure, and the basic layer and the enhancement layer form a decoding neural network 1. Method 4: The basic layer adopts a network layer with a prefabricated network structure, and the enhancement layer adopts a network layer with a default network structure, and the basic layer and the enhancement layer form a decoding neural network 1. Method 5: The basic layer adopts a network layer with a prefabricated network structure, and the enhancement layer adopts a network layer with a prefabricated network structure, and the basic layer and the enhancement layer form a decoding neural network 1. Method 6: The basic layer adopts a network layer with a prefabricated network structure, and the enhancement layer adopts a network layer with a variable network structure, and the basic layer and the enhancement layer form a decoding neural network 1.

[0230] In a possible implementation, the control parameters may include neural network information 1 corresponding to the coefficient hyperparameter feature generation unit, and the coefficient hyperparameter feature generation unit may parse the neural network information 1 from the control parameters, and generate a decoding neural network 1 based on the neural network information 1. For example, the neural network information 1 may include basic layer information and enhanced layer information, and the basic layer may be determined based on the basic layer information, and the enhanced layer may be determined based on the enhanced layer information, and the basic layer and the enhanced layer may be combined to obtain the decoding neural network 1. For example, the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit may be obtained by the following situation:

[0231] Case 1: The basic layer information includes a flag that the basic layer uses the default network, and the flag that the basic layer uses the default network indicates that the basic layer uses the default network. In this case, the coefficient hyperparameter feature generation unit obtains the network layer that the basic layer uses the default network structure based on the basic layer information, and thus obtains the basic layer of the default network structure (i.e., the network layer of the default network structure). The enhanced layer information includes a flag that the enhanced layer uses the default network, and the flag that the enhanced layer uses the default network indicates that the enhanced layer uses the default network. In this case, the coefficient hyperparameter feature generation unit obtains the network layer that the enhanced layer uses the default network structure based on the enhanced layer information, and thus obtains the enhanced layer of the default network structure (i.e., the network layer of the default network structure). On this basis, the basic layer of the default network structure and the enhanced layer of the default network structure can be combined to obtain decoding neural network 1.

[0232] Case 2: The basic layer information includes a flag that the basic layer uses a default network, and the flag that the basic layer uses a default network indicates that the basic layer uses a default network. In this case, the coefficient hyperparameter feature generation unit obtains the network layer of the default network structure used by the basic layer based on the basic layer information, and thus obtains the basic layer of the default network structure (i.e., the network layer of the default network structure). The enhanced layer information includes a flag that the enhanced layer uses a prefabricated network and an enhanced layer prefabricated network index number, and the flag that the enhanced layer uses a prefabricated network indicates that the enhanced layer uses a prefabricated network. In this case, the coefficient hyperparameter feature generation unit obtains the network layer of the enhanced layer using a prefabricated network structure based on the enhanced layer information, and thus selects the enhanced layer of the prefabricated network structure corresponding to the enhanced layer prefabricated network index number from the prefabricated neural network pool (e.g., when the enhanced layer prefabricated network index number is 0, the network layer of the prefabricated network structure s1 is used as the enhanced layer, and when the enhanced layer prefabricated network index number is 1, the network layer of the prefabricated network structure s2 is used as the enhanced layer). On this basis, the basic layer of the default network structure and the enhanced layer of the prefabricated network structure can be combined to obtain a decoding neural network 1.

[0233] Case 3: The basic layer information includes a flag that the basic layer uses a default network, and the flag that the basic layer uses a default network indicates that the basic layer uses a default network. In this case, the coefficient hyperparameter feature generation unit learns that the basic layer uses a network layer with a default network structure based on the basic layer information, and thus obtains a basic layer with a default network structure (i.e., a network layer with a default network structure). The enhancement layer information includes network parameters for generating the enhancement layer. In this case, the coefficient hyperparameter feature generation unit can parse the network parameters from the control parameters and generate an enhancement layer with a variable network structure (i.e., a network layer with a variable network structure) based on the network parameters. The network parameters may include but are not limited to at least one of the following: the number of neural network layers, the deconvolution layer flag, the number of deconvolution layers, the quantization step size of each deconvolution layer, the number of channels of each deconvolution layer, the size of the convolution kernel, the number of filters, the filter size index, the filter coefficient zero flag, the filter coefficient, the number of activation layers, the activation layer flag, and the activation layer type. The coefficient hyperparameter feature generation unit may generate an enhancement layer with a variable network structure based on the above network parameters. For example, the coefficient hyperparameter feature generation unit parses the number of deconvolution layers, the quantization step size (stride) of each deconvolution layer, the number of channels of each deconvolution layer, the size of the convolution kernel, the activation layer type and other network parameters from the control parameters, and the coefficient hyperparameter feature generation unit generates an enhancement layer of a variable network structure based on these network parameters. On this basis, the coefficient hyperparameter feature generation unit can combine the basic layer of the default network structure and the enhancement layer of the variable network structure to obtain a decoding neural network 1.

[0234] Case 4: The basic layer information includes a basic layer using a prefabricated network flag and a basic layer prefabricated network index number, and the basic layer using a prefabricated network flag indicates that the basic layer uses a prefabricated network. In this case, the coefficient hyperparameter feature generation unit obtains the network layer of the basic layer using a prefabricated network structure based on the basic layer information, and therefore selects the basic layer of the prefabricated network structure corresponding to the basic layer prefabricated network index number from the prefabricated neural network pool (such as when the basic layer prefabricated network index number is 0, the network layer of the prefabricated network structure s1 is used as the basic layer, and when the basic layer prefabricated network index number is 1, the network layer of the prefabricated network structure s2 is used as the basic layer). The enhanced layer information includes an enhanced layer using a default network flag, and the enhanced layer using a default network flag indicates that the enhanced layer uses a default network. In this case, the coefficient hyperparameter feature generation unit obtains the network layer of the enhanced layer using a default network structure based on the enhanced layer information, and therefore obtains the enhanced layer of the default network structure (i.e., the network layer of the default network structure). On this basis, the basic layer of the prefabricated network structure and the enhanced layer of the default network structure can be combined to obtain a decoding neural network 1.

[0235] Case 5: The basic layer information includes a prefabricated network flag bit and a prefabricated network index number for the basic layer, and the prefabricated network flag bit for the basic layer indicates that the basic layer uses a prefabricated network. In this case, the coefficient hyperparameter feature generation unit obtains the network layer of the prefabricated network structure used by the basic layer based on the basic layer information. Therefore, the basic layer of the prefabricated network structure corresponding to the prefabricated network index number of the basic layer (such as the network layer of the prefabricated network structure s1, etc.) can be selected from the prefabricated neural network pool. The enhanced layer information includes a prefabricated network flag bit and a prefabricated network index number for the enhanced layer, and the prefabricated network flag bit for the enhanced layer indicates that the enhanced layer uses a prefabricated network. In this case, the coefficient hyperparameter feature generation unit obtains the network layer of the prefabricated network structure used by the enhanced layer based on the enhanced layer information. Therefore, the enhanced layer of the prefabricated network structure corresponding to the prefabricated network index number of the enhanced layer (such as the network layer of the prefabricated network structure s1, etc.) can be selected from the prefabricated neural network pool. On this basis, the basic layer of the prefabricated network structure and the enhanced layer of the prefabricated network structure can be combined to obtain a decoding neural network 1.

[0236] Case 6: The basic layer information includes a prefabricated network flag and a prefabricated network index number for the basic layer, and the prefabricated network flag for the basic layer indicates that the basic layer uses a prefabricated network. In this case, the coefficient hyperparameter feature generation unit obtains the network layer of the prefabricated network structure used by the basic layer based on the basic layer information. Therefore, the basic layer of the prefabricated network structure corresponding to the prefabricated network index number of the basic layer (such as the network layer of the prefabricated network structure s1, etc.) can be selected from the prefabricated neural network pool. The enhanced layer information includes network parameters for generating the enhanced layer. In this case, the coefficient hyperparameter feature generation unit can parse the network parameters from the control parameters and generate an enhanced layer of a variable network structure (i.e., a network layer of a variable network structure) based on the network parameters. For example, the coefficient hyperparameter feature generation unit can parse the number of deconvolution layers, the quantization step size (stride) of each deconvolution layer, the number of channels of each deconvolution layer, the size of the convolution kernel, the activation layer type and other network parameters from the control parameters. The coefficient hyperparameter feature generation unit generates an enhanced layer of a variable network structure based on these network parameters. On this basis, the coefficient hyperparameter feature generation unit can combine the basic layer of the prefabricated network structure and the enhanced layer of the variable network structure to obtain the decoding neural network 1.

[0237] In a possible implementation, the above-mentioned network structures for the base layer and the enhancement layer can be determined by decoded control parameters, the control parameters may include neural network information 1, and the neural network information 1 may include base layer information and enhancement layer information. An example of the neural network information 1 for the coefficient hyperparameter feature generation unit can be seen in Table 1. In Table 1, u(n) represents an n-bit fixed-length code encoding method, and ae(v) represents a variable-length coding method.

[0238] Table 1

[0239]

[0240] In Table 1, hyper_basic_layer_use_default_para_flag is a flag for using the default network for the basic layer of the coefficient hyperparameter feature generation unit. hyper_basic_layer_use_default_para_flag is a binary variable. A value of 1 indicates that the basic layer of the coefficient hyperparameter feature generation unit uses the default network. A value of 0 indicates that the basic layer of the coefficient hyperparameter feature generation unit does not use the default network. The value of HyperBasicLayerUseDefaultParaFlag is equal to the value of hyper_basic_layer_use_default_para_flag.

[0241] In Table 1, hyper_basic_layer_use_predesigned_para_flag is a flag for using a prefabricated network for the basic layer of the coefficient hyperparameter feature generation unit, and hyper_basic_layer_use_predesigned_para_flag is a binary variable. When the value of the binary variable is 1, it indicates that the basic layer of the coefficient hyperparameter feature generation unit uses a prefabricated network, and when the value of the binary variable is 0, it indicates that the basic layer of the coefficient hyperparameter feature generation unit does not use a prefabricated network, and the value of HyperBasicLayerUsePredesignedParaFlag may be equal to the value of hyper_basic_layer_use_predesigned_para_flag.

[0242] In Table 1, hyper_basic_id is the basic layer prefabricated network index number for the coefficient hyperparameter feature generation unit, which can be a 32-bit unsigned integer, indicating the index number of the neural network used in the basic layer in the prefabricated neural network pool.

[0243] In Table 1, hyper_enhance_layer_use_default_para_flag is a flag for using the default network for the enhancement layer of the coefficient hyperparameter feature generation unit, and hyper_enhance_layer_use_default_para_flag is a binary variable. When the value of the binary variable is 1, it indicates that the enhancement layer of the coefficient hyperparameter feature generation unit uses the default network, and when the value of the binary variable is 0, it indicates that the enhancement layer of the coefficient hyperparameter feature generation unit does not use the default network, and the value of HyperEnhanceLayerUseDefaultParaFlag may be equal to the value of hyper_enhance_layer_use_default_para_flag.

[0244] In Table 1, hyper_enhance_layer_use_predesigned_para_flag is a flag for using a prefabricated network for the enhancement layer of the coefficient hyperparameter feature generation unit, and hyper_enhance_layer_use_predesigned_para_flag is a binary variable. When the value of the binary variable is 1, it indicates that the enhancement layer of the coefficient hyperparameter feature generation unit uses a prefabricated network, and when the value of the binary variable is 0, it indicates that the enhancement layer of the coefficient hyperparameter feature generation unit does not use a prefabricated network, and the value of HyperEnhanceLayerUsePredesignedParaFlag may be equal to the value of hyper_enhance_layer_use_predesigned_para_flag.

[0245] In Table 1, hyper_enhance_id is the enhancement layer prefabricated network index number for the coefficient hyperparameter feature generation unit, which can be a 32-bit unsigned integer, indicating the index number of the neural network used in the enhancement layer in the prefabricated neural network pool.

[0246] In the above process, the range of hyper_basic_id is [id_min, id_max], id_min is preferably 0, id_max is preferably 2^32-1, and the [a, b] segment is a reserved segment for the later expansion of the prefabricated neural network pool. It should be noted that for the prefabricated neural network pool of the basic layer, the prefabricated neural network pool may include several basic layer prefabricated networks, such as 2, 3, 4, etc., and may include dozens of basic layer prefabricated networks, and may also include more basic layer prefabricated networks, without limitation. Obviously, id_max is preferably 2^32-1 is just an example, and the value of id_max can be dynamically adjusted in different situations.

[0247] In the above process, the range of hyper_enhance_id is [id_min, id_max], id_min is preferably 0, id_max is preferably 2^32-1, and the [a, b] segment is a reserved segment for the later expansion of the prefabricated neural network pool. It should be noted that for the prefabricated neural network pool of the enhancement layer, the prefabricated neural network pool may include several enhancement layer prefabricated networks, such as 2, 3, 4, etc., and may include dozens of enhancement layer prefabricated networks, and may also include more enhancement layer prefabricated networks, without limitation. Obviously, id_max is preferably 2^32-1 is just an example, and the value of id_max can be dynamically adjusted in different situations.

[0248] In a possible implementation, the above-mentioned network structure for the enhancement layer can be determined by decoded control parameters, the control parameters may include neural network information 1, and the neural network information 1 may include enhancement layer information. An example of the neural network information 1 for the coefficient hyperparameter feature generation unit can be seen in Tables 2 and 3.

[0249] Table 2

[0250]

[0251] Table 3

[0252]

[0253]

[0254] In Table 2, layer_num represents the number of neural network layers, which is used to indicate the number of network layers of the neural network. If the activation layer is included in a certain network structure, the number of layers is not calculated additionally, and the value of LayerNum is equal to layer_num.

[0255] In Table 3, deconv_layer_flag indicates the deconvolution layer flag, and deconv_layer_flag is a binary variable. When the value of the binary variable is 1, it indicates that the current layer is a deconvolution layer network, and when the value of the binary variable is 0, it indicates that the current layer is not a deconvolution layer network, and the value of DeconvLayerFlag is equal to the value of deconv_layer_flag.

[0256] In Table 3, stride_num represents the quantization step size of the deconvolution layer.

[0257] In Table 3, filter_num represents the number of filters, that is, the number of filters in the current layer.

[0258] In Table 3, filter_size_index represents the filter size index, that is, the current filter size index value.

[0259] In Table 3, filter_coeff_zero_flag[i][j] indicates that the filter coefficient is zero flag, which is a binary variable. When the value of the binary variable is 1, it indicates that the current filter coefficient is 0. When the value of the binary variable is 0, it indicates that the current filter coefficient is not 0. The value of FilterCoeffZeroFlag[i][j] is equal to the value of filter_coeff_zero_flag[i][j].

[0260] In Table 3, filter_coeff[i][j] represents the filter coefficient, that is, the current filter coefficient value.

[0261] In Table 3, activation_layer_flag indicates the activation layer flag, and activation_layer_flag is a binary variable. When the value of the binary variable is 1, it indicates that the current layer is an activation layer, and when the value of the binary variable is 0, it indicates that the current layer is not an activation layer, and the value of ActivationLayerFlag is equal to the value of activation_layer_flag.

[0262] In Table 3, activation_layer_type indicates the activation layer type, that is, the specific type of the activation layer of the current layer.

[0263] Example 10: For Example 5 and Example 6, the second feature decoding unit can decode the code stream 2 corresponding to the current block to obtain image feature information corresponding to the current block. For example, the second feature decoding unit includes at least one coefficient decoding module and a probability model acquisition module. In one possible implementation, the coefficient decoding module can use an entropy decoding method to perform coefficient decoding, that is, use the entropy decoding method to decode the code stream 2 corresponding to the current block to obtain image feature information corresponding to the current block.

[0264] Exemplarily, the entropy decoding method may include but is not limited to CAVLC or CABAC, etc., which is not limited thereto.

[0265] Exemplarily, the manner in which the features generated by the coefficient hyperparameter feature generation unit are used in the second feature decoding unit may include:

[0266] Method 1: The probability model acquisition module is used to obtain the probability model of entropy decoding. For example, the probability model acquisition module obtains the coefficient hyperparameter feature value P from the coefficient hyperparameter feature generation unit. On this basis, the coefficient decoding module can obtain the coefficient hyperparameter feature value P from the probability model acquisition module. Based on the coefficient hyperparameter feature value P, the coefficient decoding module can use the entropy decoding method to perform coefficient decoding.

[0267] Method 2: The coefficient parsing process (such as the CABAC or CAVCL decoding process) does not rely on the features generated by the coefficient super-parameter feature generation unit, and the coefficient value can be directly parsed (this can ensure the parsing throughput or rate). The parsed coefficient value is then converted based on the features generated by the coefficient super-parameter feature generation unit to obtain the image feature quantization value F_q. An example is that if the coefficient value obtained by the coefficient parsing process is 0, and the corresponding feature value generated by the coefficient super-parameter feature generation unit is u, then F_q=u; if the coefficient value obtained by the coefficient parsing process is 1, and the corresponding feature value generated by the coefficient super-parameter feature generation unit is u, then F_q=u+x, where x is the corresponding coefficient variance.

[0268] Embodiment 11: With respect to Embodiment 5, the second dequantization unit may dequantize the image feature quantization value F_q (i.e., the image feature information) to obtain the image feature reconstruction value F'. For example, the second dequantization unit may not exist, or, if the second dequantization unit exists, the second dequantization unit may be selectively skipped based on the control parameter (such as high-level syntax, such as the second enable information, etc.), or the second dequantization unit may be enabled based on the control parameter. Exemplarily, if the second dequantization unit does not exist, the image feature reconstruction value F' is the same as the image feature quantization value F_q, that is, the image feature quantization value F_q does not need to be dequantized. If the second dequantization unit is selectively skipped based on the control parameter, the image feature reconstruction value F' is the same as the image feature quantization value F_q, that is, the image feature quantization value F_q does not need to be dequantized. If the second dequantization unit is enabled based on the control parameter, but the step size parameter qstep corresponding to the image feature quantization value F_q is 1, the image feature reconstruction value F' is the same as the image feature quantization value F_q, that is, the image feature quantization value F_q does not need to be dequantized.

[0269] Exemplarily, if it is determined to enable the second inverse quantization unit based on the control parameter, and the step parameter qstep corresponding to the image feature quantization value F_q is not 1, the second inverse quantization unit may perform inverse quantization on the image feature quantization value F_q based on the control parameter (such as quantization-related parameters) to obtain the image feature reconstruction value F'. For example, the second inverse quantization unit performs the following operations: obtain the quantization-related parameters corresponding to the image feature quantization value F_q from the control parameter (the control parameter is included in the bitstream, and the control parameter may include quantization-related parameters), such as the step parameter qstep or the quantization parameter qp. Determine the multiplication factor mult and the shift factor shift corresponding to the image feature quantization value F_q based on the step parameter qstep or the quantization parameter qp; assume that the image feature quantization value F_q is Coff_hyper and the image feature reconstruction value F' is Coff_hyper_rec, then Coff_hyper_rec = (Coff_hyper * mult) << shift, that is to say, when performing inverse quantization, the above formula can be used to obtain the image feature reconstruction value F'.

[0270] It should be noted that for the quantization-related parameters (such as the step parameter qstep) corresponding to the image feature quantization value F_q, it includes: 1) Each image feature quantization value of each feature channel uses the same step parameter qstep; 2) The image feature quantization values of each feature channel use different step parameter qsteps, however, each image feature quantization value within the feature channel uses the same step parameter qstep; 3) Each image feature quantization value of each feature channel uses different step parameter qsteps.

[0271] Embodiment 12: For Embodiments 5 and 6, the image feature inverse transformation unit may perform an inverse transformation operation on the image feature reconstruction value F' based on the decoding neural network to obtain the image low-order feature value LF. In a possible implementation manner, as shown in Fig.6D shown, the image feature inverse transformation unit may include a decoding neural network 2. The decoding neural network 2 may include a basic layer and an enhancement layer. The image feature reconstruction value F' is used as the input feature of the decoding neural network 2, and the image low-order feature value LF is used as the output feature of the decoding neural network 2. The decoding neural network 2 is used to perform an inverse transformation operation on the image feature reconstruction value F'.

[0272] In this embodiment, the decoding neural network 2 can be divided into a basic layer and an enhanced layer. The basic layer may include at least one network layer, or the basic layer may not include a network layer, that is, the basic layer is empty. The enhanced layer may include at least one network layer, or the enhanced layer may not include a network layer, that is, the enhanced layer is empty. It should be noted that for the multiple network layers in the decoding neural network 2, the multiple network layers can be divided into basic layers and enhanced layers according to actual needs. For example, a network layer with a fixed network structure can be used as a basic layer, and a network layer with a non-fixed network structure can be used as an enhanced layer.

[0273] Exemplarily, for the decoding neural network 2, the size of the output feature may be larger than the size of the input feature, or the size of the output feature may be equal to the size of the input feature, or the size of the output feature may be smaller than the size of the input feature.

[0274] For the decoding neural network 2, the base layer and the enhancement layer include at least one deconvolution layer. For example, the base layer includes at least one deconvolution layer, and the enhancement layer may include at least one deconvolution layer, or may not include a deconvolution layer. Alternatively, the enhancement layer includes at least one deconvolution layer, and the base layer may include at least one deconvolution layer, or may not include a deconvolution layer.

[0275] For the decoding neural network 2, the base layer and the enhancement layer include at least one residual structure layer. For example, the base layer includes at least one residual structure layer, and the enhancement layer may include the residual structure layer or not. Alternatively, the enhancement layer includes at least one residual structure layer, and the base layer may include the residual structure layer or not.

[0276] Exemplarily, for the decoding neural network 2, it may include but is not limited to deconvolution layers and activation layers, etc., and there is no limitation on this. For example, the decoding neural network 2 includes 1 deconvolution layer with a stride of 2, 1 activation layer, 1 deconvolution layer with a stride of 1, and 1 activation layer in sequence. All of the above network layers can be used as basic layers, and the enhancement layers are empty. For another example, the decoding neural network 2 includes 1 deconvolution layer with a stride of 2, 1 activation layer, 1 deconvolution layer with a stride of 2, 1 activation layer, 1 deconvolution layer with a stride of 1, and 1 activation layer in sequence. All of the above network layers can be used as basic layers, and the enhancement layers are empty. For another example, the decoding neural network 2 includes 1 deconvolution layer with a stride of 2, 1 activation layer, 1 deconvolution layer with a stride of 2, 1 activation layer, 1 deconvolution layer with a stride of 1, and 1 activation layer in sequence. All of the above network layers can be used as basic layers, and the enhancement layers are empty.

[0277] Exemplarily, if there is no quality enhancement unit, the number of output features (number of filters) of the last network layer of the image feature inverse transformation unit is 1 or 3. Specifically, if the output is only the value of one channel (such as a grayscale image), the number of output features of the last network layer of the image feature inverse transformation unit is 1; if the output is only the value of three channels (such as RGB or YUV format), the number of output features of the last network layer of the image feature inverse transformation unit is 3.

[0278] Exemplarily, if there is a quality enhancement unit, the output feature number (number of filters) of the last network layer of the image feature inverse transformation unit can be 1 or 3, or other values, without limitation.

[0279] In a possible implementation, a network layer of a default network structure may be configured for the image feature inverse transformation unit, and the network parameters related to the network layer of the default network structure are all fixed. For example, in the network layer of the default network structure, the number of deconvolution layers is fixed, the number of activation layers is fixed, the number of channels of each deconvolution layer is fixed, the size of the convolution kernel is fixed, the filter coefficient is fixed, etc. Obviously, since the network parameters in the network layer of the default network structure are all fixed and these network parameters are known, the network layer of the default network structure can be directly obtained.

[0280] In a possible implementation, a prefabricated neural network pool can be configured for the image feature inverse transformation unit, and the prefabricated neural network pool includes at least one network layer of a prefabricated network structure, and the network parameters related to the network layer of the prefabricated network structure can be configured according to actual needs. For example, the prefabricated neural network pool includes a network layer of a prefabricated network structure t1, a network layer of a prefabricated network structure t2, and a network layer of a prefabricated network structure t3. Among them, for the network layer of the prefabricated network structure t1, the number of deconvolution layers, the number of activation layers, the number of channels of each deconvolution layer, the size of the convolution kernel, the filter coefficient and other network parameters can be preconfigured. After all network parameters are configured, the network layer of the prefabricated network structure t1 can be obtained, and so on.

[0281] In a possible implementation, a network layer of a variable network structure can be dynamically generated for the image feature inverse transformation unit based on network parameters, and the network parameters related to the network layer of the variable network structure are dynamically generated by the encoding end, rather than pre-configured. For example, the encoding end can encode network parameters such as the number of deconvolution layers, the number of activation layers, the number of channels of each deconvolution layer, the size of the convolution kernel, and the filter coefficient in the bitstream, so that the decoding end can parse the above network parameters from the bitstream and generate a network layer of a variable network structure based on these network parameters.

[0282] In a possible implementation, the control parameter may include the neural network information 2 corresponding to the image feature inverse transformation unit, and the image feature inverse transformation unit may parse the neural network information 2 from the control parameter, and generate the decoding neural network 2 based on the neural network information 2. For example, the neural network information 2 may include basic layer information and enhanced layer information, and the basic layer may be determined based on the basic layer information, and the enhanced layer may be determined based on the enhanced layer information, and the basic layer and the enhanced layer may be combined to obtain the decoding neural network 2. For example, the neural network information 2 may be obtained by the following situation:

[0283] Case 1: The basic layer information includes a flag that the basic layer uses a default network, and the flag that the basic layer uses a default network indicates that the basic layer uses a default network. In this case, the image feature inverse transformation unit learns that the basic layer uses a network layer with a default network structure based on the basic layer information, and thus obtains the basic layer with a default network structure. The enhanced layer information includes a flag that the enhanced layer uses a default network, and the flag that the enhanced layer uses a default network indicates that the enhanced layer uses a default network. In this case, the enhanced layer learns that the enhanced layer uses a network layer with a default network structure based on the enhanced layer information, and thus obtains the enhanced layer with a default network structure. The basic layer with a default network structure and the enhanced layer with a default network structure are combined to obtain a decoding neural network 2.

[0284] Case 2: The basic layer information includes a flag that the basic layer uses a default network, and the flag that the basic layer uses a default network indicates that the basic layer uses a default network. In this case, the image feature inverse transformation unit obtains the network layer that the basic layer uses a default network structure based on the basic layer information, and thus obtains the basic layer with a default network structure. The enhanced layer information includes a flag that the enhanced layer uses a prefabricated network and an enhanced layer prefabricated network index number, and the flag that the enhanced layer uses a prefabricated network indicates that the enhanced layer uses a prefabricated network. In this case, the network layer that the enhanced layer uses a prefabricated network structure is obtained based on the enhanced layer information, and thus the enhanced layer with a prefabricated network structure corresponding to the enhanced layer prefabricated network index number is selected from the prefabricated neural network pool. The basic layer with a default network structure and the enhanced layer with a prefabricated network structure can be combined to obtain a decoding neural network 2.

[0285] Case 3: The basic layer information includes a flag that the basic layer uses a default network, and the flag that the basic layer uses a default network indicates that the basic layer uses a default network. The image feature inverse transformation unit learns that the basic layer uses a network layer with a default network structure based on the basic layer information, and thus obtains the basic layer with a default network structure. The enhancement layer information includes network parameters for generating the enhancement layer. In this case, the network parameters are parsed from the control parameters, and an enhancement layer with a variable network structure is generated based on the network parameters. The basic layer with a default network structure and the enhancement layer with a variable network structure are combined to obtain a decoding neural network 2.

[0286] Case 4: The basic layer information includes a prefabricated network flag and a prefabricated network index number for the basic layer, and the prefabricated network flag for the basic layer indicates that the basic layer uses a prefabricated network. In this case, the image feature inverse transformation unit obtains the network layer of the prefabricated network structure used by the basic layer based on the basic layer information, and therefore selects the basic layer of the prefabricated network structure corresponding to the prefabricated network index number of the basic layer from the prefabricated neural network pool. The enhanced layer information includes a default network flag for the enhanced layer, and the default network flag for the enhanced layer indicates that the enhanced layer uses the default network. In this case, the network layer of the default network structure used by the enhanced layer is obtained based on the enhanced layer information, and therefore, the enhanced layer of the default network structure is obtained. The basic layer of the prefabricated network structure and the enhanced layer of the default network structure can be combined to obtain a decoding neural network 2.

[0287] Case 5: The basic layer information includes a prefabricated network flag bit and a prefabricated network index number for the basic layer, and the prefabricated network flag bit for the basic layer indicates that the basic layer uses a prefabricated network. In this case, the image feature inverse transformation unit obtains the network layer of the prefabricated network structure used by the basic layer based on the basic layer information, and therefore, the basic layer of the prefabricated network structure corresponding to the prefabricated network index number of the basic layer can be selected from the prefabricated neural network pool. The enhanced layer information includes a prefabricated network flag bit and a prefabricated network index number for the enhanced layer, and the prefabricated network flag bit for the enhanced layer indicates that the enhanced layer uses a prefabricated network. In this case, the network layer of the prefabricated network structure used by the enhanced layer is obtained based on the enhanced layer information, and therefore, the enhanced layer of the prefabricated network structure corresponding to the prefabricated network index number of the enhanced layer can be selected from the prefabricated neural network pool. The basic layer of the prefabricated network structure and the enhanced layer of the prefabricated network structure can be combined to obtain a decoding neural network 2.

[0288] Case 6: The basic layer information includes a prefabricated network flag and a prefabricated network index number for the basic layer, and the prefabricated network flag for the basic layer indicates that the basic layer uses a prefabricated network. In this case, the image feature inverse transformation unit obtains the network layer of the prefabricated network structure based on the basic layer information. Therefore, the basic layer of the prefabricated network structure corresponding to the prefabricated network index number of the basic layer can be selected from the prefabricated neural network pool. The enhanced layer information includes network parameters for generating the enhanced layer. In this case, the network parameters are parsed from the control parameters, and an enhanced layer of a variable network structure is generated based on the network parameters. The basic layer of the prefabricated network structure and the enhanced layer of the variable network structure are combined to obtain a decoding neural network 2.

[0289] In a possible implementation, the above-mentioned network structures for the base layer and the enhancement layer can be determined by decoded control parameters, and the control parameters may include neural network information 2, and the neural network information 2 may include base layer information and enhancement layer information. The neural network information 2 for the image feature inverse transformation unit is similar to Table 1, Table 2, and Table 3, except that the relevant information is for the image feature inverse transformation unit, rather than the coefficient hyperparameter feature generation unit, and will not be repeated here.

[0290] Embodiment 13: With respect to Embodiment 5 and Embodiment 6, the quality enhancement unit may obtain the low-order eigenvalue LF of the image, and enhance the low-order eigenvalue LF of the image based on the decoding neural network to obtain the reconstructed image block I corresponding to the current block.

[0291] For example, the quality enhancement unit may not exist, or, if the quality enhancement unit exists, the quality enhancement unit may be selectively skipped based on control parameters (such as high-level syntax, such as the third enabling information, etc.), or the quality enhancement unit may be enabled based on the control parameters. Exemplarily, if the quality enhancement unit is enabled based on the control parameters, the quality enhancement unit may be used to remove image quality degradation problems such as block effects and quantization distortion between blocks. For example, the quality enhancement unit may enhance the low-order eigenvalues ​​LF of the image based on the decoding neural network to obtain a reconstructed image block I corresponding to the current block.

[0292] In one possible implementation, the quality enhancement unit may include a decoding neural network 3, which may include a base layer and an enhancement layer, wherein the image low-order eigenvalue LF serves as an input feature of the decoding neural network 3, and the reconstructed image block I serves as an output feature of the decoding neural network 3, and the decoding neural network 3 is used to perform enhancement processing on the image low-order eigenvalue LF.

[0293] In this embodiment, the decoding neural network 3 can be divided into a basic layer and an enhanced layer. The basic layer may include at least one network layer, or the basic layer may not include a network layer, that is, the basic layer is empty. The enhanced layer may include at least one network layer, or the enhanced layer may not include a network layer, that is, the enhanced layer is empty. It should be noted that for the multiple network layers in the decoding neural network 3, the multiple network layers can be divided into basic layers and enhanced layers according to actual needs. For example, a network layer with a fixed network structure can be used as a basic layer, and a network layer with a non-fixed network structure can be used as an enhanced layer.

[0294] Exemplarily, for the decoding neural network 3, the size of the output feature may be larger than the size of the input feature, or the size of the output feature may be equal to the size of the input feature, or the size of the output feature may be smaller than the size of the input feature.

[0295] For the decoding neural network 3, the base layer and the enhancement layer include at least one deconvolution layer. For example, the base layer includes at least one deconvolution layer, and the enhancement layer may include at least one deconvolution layer, or may not include a deconvolution layer. Alternatively, the enhancement layer includes at least one deconvolution layer, and the base layer may include at least one deconvolution layer, or may not include a deconvolution layer.

[0296] For the decoding neural network 3, the base layer and the enhancement layer include at least one residual structure layer. For example, the base layer includes at least one residual structure layer, and the enhancement layer may include the residual structure layer or not. Alternatively, the enhancement layer includes at least one residual structure layer, and the base layer may include the residual structure layer or not.

[0297] Exemplarily, for the decoding neural network 3, it may include but is not limited to deconvolution layers and activation layers, etc., and there is no limitation on this. For example, the decoding neural network 3 includes 1 deconvolution layer with a stride of 2, 1 activation layer, 1 deconvolution layer with a stride of 1, and 1 activation layer in sequence. All of the above network layers can be used as basic layers, and the enhancement layers are empty. For another example, the decoding neural network 3 includes 1 deconvolution layer with a stride of 2, 1 activation layer, 1 deconvolution layer with a stride of 2, 1 activation layer, 1 deconvolution layer with a stride of 1, and 1 activation layer in sequence. All of the above network layers can be used as basic layers, and the enhancement layers are empty. For another example, the decoding neural network 3 includes 1 deconvolution layer with a stride of 2, 1 activation layer, 1 deconvolution layer with a stride of 2, 1 activation layer, 1 deconvolution layer with a stride of 1, and 1 activation layer in sequence. All of the above network layers can be used as basic layers, and the enhancement layers are empty.

[0298] Exemplarily, the output feature number (number of filters) of the last network layer of the quality enhancement unit is 1 or 3. Specifically, if the output is only the value of one channel (such as a grayscale image), the output feature number of the last network layer is 1; if the output is only the value of three channels (such as RGB or YUV format), the output feature number of the last network layer is 3.

[0299] In a possible implementation, a network layer of a default network structure may be configured for the quality enhancement unit, and network parameters related to the network layer of the default network structure are all fixed. A prefabricated neural network pool may be configured for the quality enhancement unit, and the prefabricated neural network pool includes at least one network layer of a prefabricated network structure, and network parameters related to the network layer of the prefabricated network structure may be configured according to actual needs. A network layer of a variable network structure may be dynamically generated for the quality enhancement unit based on the network parameters, and network parameters related to the network layer of the variable network structure are dynamically generated by the encoding end, rather than pre-configured.

[0300] In a possible implementation, the control parameter may include neural network information 3 corresponding to the quality enhancement unit, and the quality enhancement unit may parse the neural network information 3 from the control parameter, and generate a decoding neural network 3 based on the neural network information 3. For example, the neural network information 3 may include basic layer information and enhanced layer information, and the basic layer may be determined based on the basic layer information, and the enhanced layer may be determined based on the enhanced layer information, and the basic layer and the enhanced layer may be combined to obtain the decoding neural network 3. For example, the quality enhancement unit may obtain the neural network information 3 by:

[0301] Case 1: The basic layer information includes a flag bit that the basic layer uses a default network, and the flag bit that the basic layer uses a default network indicates that the basic layer uses a default network. In this case, the quality enhancement unit learns that the basic layer uses a network layer with a default network structure based on the basic layer information, and thus obtains the basic layer with a default network structure. The enhanced layer information includes a flag bit that the enhanced layer uses a default network, and the flag bit that the enhanced layer uses a default network indicates that the enhanced layer uses a default network. In this case, the network layer with a default network structure is learned based on the enhanced layer information, and thus the enhanced layer with a default network structure is obtained. The basic layer with a default network structure and the enhanced layer with a default network structure are combined to obtain a decoding neural network 3.

[0302] Case 2: The basic layer information includes a flag that the basic layer uses a default network, and the flag that the basic layer uses a default network indicates that the basic layer uses a default network. In this case, the quality enhancement unit obtains the network layer of the default network structure based on the basic layer information, and thus obtains the basic layer of the default network structure. The enhanced layer information includes a flag that the enhanced layer uses a prefabricated network and an enhanced layer prefabricated network index number, and the flag that the enhanced layer uses a prefabricated network indicates that the enhanced layer uses a prefabricated network. In this case, the network layer of the enhanced layer using a prefabricated network structure is obtained based on the enhanced layer information, and thus the enhanced layer of the prefabricated network structure corresponding to the enhanced layer prefabricated network index number is selected from the prefabricated neural network pool. The basic layer of the default network structure and the enhanced layer of the prefabricated network structure can be combined to obtain a decoding neural network 3.

[0303] Case 3: The basic layer information includes a flag bit that the basic layer uses a default network, and the flag bit that the basic layer uses a default network indicates that the basic layer uses a default network. The quality enhancement unit learns that the basic layer uses a network layer with a default network structure based on the basic layer information, and thus obtains the basic layer with a default network structure. The enhancement layer information includes network parameters for generating the enhancement layer. In this case, the network parameters are parsed from the control parameters, and an enhancement layer with a variable network structure is generated based on the network parameters. The basic layer with a default network structure and the enhancement layer with a variable network structure are combined to obtain a decoding neural network 3.

[0304] Case 4: The basic layer information includes a prefabricated network flag bit and a prefabricated network index number of the basic layer, and the prefabricated network flag bit of the basic layer indicates that the basic layer uses a prefabricated network. In this case, the quality enhancement unit obtains the network layer of the prefabricated network structure of the basic layer based on the basic layer information, and therefore selects the basic layer of the prefabricated network structure corresponding to the prefabricated network index number of the basic layer from the prefabricated neural network pool. The enhanced layer information includes a default network flag bit of the enhanced layer, and the default network flag bit of the enhanced layer indicates that the enhanced layer uses the default network. In this case, the network layer of the default network structure of the enhanced layer is obtained based on the enhanced layer information, and therefore, the enhanced layer of the default network structure is obtained. The basic layer of the prefabricated network structure and the enhanced layer of the default network structure can be combined to obtain a decoding neural network 3.

[0305] Case 5: The basic layer information includes a prefabricated network flag bit and a prefabricated network index number for the basic layer, and the prefabricated network flag bit for the basic layer indicates that the basic layer uses a prefabricated network. In this case, the quality enhancement unit obtains the network layer of the prefabricated network structure used by the basic layer based on the basic layer information, and therefore, the basic layer of the prefabricated network structure corresponding to the prefabricated network index number of the basic layer can be selected from the prefabricated neural network pool. The enhanced layer information includes a prefabricated network flag bit and a prefabricated network index number for the enhanced layer, and the prefabricated network flag bit for the enhanced layer indicates that the enhanced layer uses a prefabricated network. In this case, the network layer of the prefabricated network structure used by the enhanced layer is obtained based on the enhanced layer information, and therefore, the enhanced layer of the prefabricated network structure corresponding to the prefabricated network index number of the enhanced layer can be selected from the prefabricated neural network pool. The basic layer of the prefabricated network structure and the enhanced layer of the prefabricated network structure can be combined to obtain a decoding neural network 3.

[0306] Case 6: The basic layer information includes a prefabricated network flag and a prefabricated network index number for the basic layer, and the prefabricated network flag for the basic layer indicates that the basic layer uses a prefabricated network. In this case, the quality enhancement unit learns that the basic layer uses a network layer of a prefabricated network structure based on the basic layer information. Therefore, the basic layer of the prefabricated network structure corresponding to the prefabricated network index number of the basic layer can be selected from the prefabricated neural network pool. The enhanced layer information includes network parameters for generating the enhanced layer. In this case, the network parameters are parsed from the control parameters, and an enhanced layer of a variable network structure is generated based on the network parameters. The basic layer of the prefabricated network structure and the enhanced layer of the variable network structure are combined to obtain a decoding neural network 3.

[0307] In a possible implementation, the above-mentioned network structures for the basic layer and the enhancement layer can be determined by decoded control parameters. The control parameters may include neural network information 3, and the neural network information 3 may include basic layer information and enhancement layer information. The neural network information 3 for the quality enhancement unit is similar to Table 1, Table 2, and Table 3, except that the relevant information is for the quality enhancement unit, rather than the coefficient hyperparameter feature generation unit, and will not be repeated here.

[0308] Embodiment 14: In the embodiment of the present application, a neural network-based image encoding method is proposed, which can be applied to an encoding end (also called a video encoder), see Fig. 7A As shown, it is a structural diagram of the encoding end, and the encoding end may include a control parameter encoding unit, a feature transformation unit, a coefficient super-parameter feature transformation unit, a first quantization unit, a second quantization unit, a first feature encoding unit, a second feature encoding unit, a first feature decoding unit, a second feature decoding unit, a coefficient super-parameter feature generation unit, an image feature inverse transformation unit, a first inverse quantization unit, a second inverse quantization unit, and a quality enhancement unit.

[0309] Exemplarily, the first quantization unit, the second quantization unit, the first inverse quantization unit, the second inverse quantization unit, and the quality enhancement unit are optional units, and in some scenarios, the processes of these optional units may be closed or skipped.

[0310] In this embodiment, for each current block (i.e., image block), the bitstream corresponding to the current block includes three parts: bitstream 0 (bitstream containing control parameters), bitstream 1 (bitstream containing coefficient super-parameter feature information), and bitstream 2 (bitstream containing image feature information). The coefficient super-parameter feature information and the image feature information can be collectively referred to as image information.

[0311] Exemplarily, the neural network-based image encoding method in this embodiment may include the following steps:

[0312] Step S71, perform feature transformation on the current block I to obtain an image feature value F corresponding to the current block I. For example, the feature transformation unit may perform feature transformation on the current block I to obtain an image feature value F corresponding to the current block I. Exemplarily, the feature transformation unit may perform feature transformation on the current block I based on an encoding neural network to obtain an image feature value F corresponding to the current block I. The current block I is used as an input feature of the encoding neural network, and the image feature value F is used as an output feature of the encoding neural network.

[0313] Step S72: Determine image feature information based on the image feature value F.

[0314] Exemplarily, if the second quantization unit is enabled, the second quantization unit can obtain the image feature value F from the feature transformation unit, and quantize the image feature value F to obtain the image feature quantization value F_q, and determine the image feature information based on the image feature quantization value F_q, that is, the image feature information can be the image feature quantization value F_q. In this case, the control parameter encoding unit can encode the second enabling information of the second quantization unit in the bitstream 0, that is, the control parameter includes the second enabling information of the second quantization unit, and the second enabling information is used to indicate that the second quantization unit has been enabled. The control parameter encoding unit can also encode the quantization-related parameters corresponding to the second quantization unit in the bitstream 0, such as the step size parameter qstep or the quantization parameter qp.

[0315] Exemplarily, if the second quantization unit is not enabled, the image feature information is determined based on the image feature value F, that is, the image feature information may be the image feature value F. In this case, the control parameter encoding unit may encode the second enabling information of the second quantization unit in the bitstream 0, and the second enabling information is used to indicate that the second quantization unit is not enabled.

[0316] Step S73, perform coefficient hyperparameter feature transformation on the image feature value F to obtain coefficient hyperparameter feature coefficient value C. For example, the coefficient hyperparameter feature transformation unit performs coefficient hyperparameter feature transformation on the image feature value F to obtain coefficient hyperparameter feature coefficient value C, such as performing coefficient hyperparameter feature transformation on the image feature value F based on the encoding neural network to obtain coefficient hyperparameter feature coefficient value C. The image feature value F is used as the input feature of the encoding neural network, and the coefficient hyperparameter feature coefficient value C is used as the output feature of the encoding neural network.

[0317] Step S74: determine the coefficient hyper-parameter feature information based on the coefficient hyper-parameter feature coefficient value C.

[0318] Exemplarily, if the first quantization unit is enabled, the first quantization unit can obtain the coefficient superparameter feature coefficient value C from the coefficient superparameter feature transformation unit, and quantize the coefficient superparameter feature coefficient value C to obtain the coefficient superparameter feature coefficient quantization value C_q, and determine the coefficient superparameter feature information based on the coefficient superparameter feature coefficient quantization value C_q, that is, the coefficient superparameter feature information can be the coefficient superparameter feature coefficient quantization value C_q. In this case, the control parameter encoding unit can encode the first enabling information of the first quantization unit in bitstream 0, that is, the control parameter includes the first enabling information of the first quantization unit, and the first enabling information is used to indicate that the first quantization unit has been enabled. The control parameter encoding unit can also encode the quantization-related parameters corresponding to the first quantization unit in bitstream 0.

[0319] Exemplarily, if the first quantization unit is not enabled, the coefficient hyperparameter feature information is determined based on the coefficient hyperparameter feature coefficient value C, that is, the coefficient hyperparameter feature information may be the coefficient hyperparameter feature coefficient value C. In this case, the control parameter encoding unit may encode the first enabling information of the first quantization unit in the bitstream 0, and the first enabling information is used to indicate that the first quantization unit is not enabled.

[0320] Step S75: Encode the coefficient super-parameter feature information (such as the coefficient super-parameter feature coefficient quantization value C_q or the coefficient super-parameter feature coefficient value C) to obtain a bitstream 1. For example, the first feature encoding unit can encode the coefficient super-parameter feature information in the bitstream corresponding to the current block. For the convenience of distinction, the bitstream including the coefficient super-parameter feature information is recorded as bitstream 1.

[0321] Step S76, decode the code stream 1 corresponding to the current block, obtain the coefficient super-parameter feature information (such as the coefficient super-parameter feature coefficient quantization value C_q or the coefficient super-parameter feature coefficient value C), and determine the coefficient super-parameter feature coefficient reconstruction value based on the coefficient super-parameter feature information.

[0322] Exemplarily, the first feature decoding unit may decode the code stream 1 corresponding to the current block to obtain coefficient hyper-parameter feature information.

[0323] Exemplarily, if the first quantization unit is enabled, the coefficient hyperparameter feature information is the coefficient hyperparameter feature coefficient quantization value C_q. In this case, the first dequantization unit can dequantize the coefficient hyperparameter feature coefficient quantization value C_q to obtain the coefficient hyperparameter feature coefficient reconstruction value C'. Obviously, the coefficient hyperparameter feature coefficient reconstruction value C' and the coefficient hyperparameter feature coefficient value C can be the same. If the first quantization unit is not enabled, the coefficient hyperparameter feature information is the coefficient hyperparameter feature coefficient value C. In this case, the coefficient hyperparameter feature coefficient value C can be used as the coefficient hyperparameter feature coefficient reconstruction value C'. In summary, the coefficient hyperparameter feature coefficient reconstruction value C' can be obtained.

[0324] In summary, it can be seen that the coefficient super-parameter feature coefficient reconstruction value C' is the same as the coefficient super-parameter feature coefficient value C. Therefore, the process of step S76 can be omitted, and the coefficient super-parameter feature coefficient value C is directly used as the coefficient super-parameter feature coefficient reconstruction value C'. In this case, Fig. 7A The coding end structure shown in the figure is improved to obtain Figure 7B The encoding end structure shown.

[0325] Step S77: perform an inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value C' to obtain the coefficient hyperparameter feature value P. For example, the coefficient hyperparameter feature generation unit performs an inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value C' to obtain the coefficient hyperparameter feature value P.

[0326] Step S78: Based on the coefficient hyperparameter feature value P, encode the image feature information (such as the image feature quantization value F_q or the image feature value F) to obtain a code stream 2. For example, the second feature encoding unit can encode the image feature information in the code stream corresponding to the current block. For the convenience of distinction, the code stream including the image feature information is recorded as code stream 2.

[0327] Step S79: The control parameter encoding unit obtains the control parameter corresponding to the current block, which may include neural network information, and encodes the control parameter corresponding to the current block in the bitstream, and records the bitstream including the control parameter as bitstream 0.

[0328] In one possible implementation, the feature transformation unit may use an encoding neural network to perform feature transformation on the current block I, and the control parameter encoding unit may determine the neural network information 2 corresponding to the image feature inverse transformation unit of the decoding end device based on the network structure of the encoding neural network. The neural network information 2 is used to determine the decoding neural network 2 corresponding to the image feature inverse transformation unit, and encode the neural network information 2 corresponding to the image feature inverse transformation unit in the bitstream 0.

[0329] In one possible implementation, the coefficient hyperparameter feature transformation unit can use an encoding neural network to perform coefficient hyperparameter feature transformation on the image eigenvalue F, and the control parameter encoding unit can determine the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit of the decoding end device based on the network structure of the encoding neural network. The neural network information 1 is used to determine the decoding neural network 1 corresponding to the coefficient hyperparameter feature generation unit, and encode the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit in the bitstream 0.

[0330] In one possible implementation, see Fig. 7A and Figure 7B As shown, the following steps may also be included:

[0331] Step S80: Decode the bitstream 1 corresponding to the current block to obtain coefficient super-parameter feature information corresponding to the current block. For example, the first feature decoding unit can decode the bitstream 1 corresponding to the current block to obtain coefficient super-parameter feature information corresponding to the current block.

[0332] Step S81, determining the coefficient hyper-parameter characteristic coefficient reconstruction value based on the coefficient hyper-parameter characteristic information.

[0333] For example, if the first quantization unit is enabled, the coefficient hyperparameter feature information is the coefficient hyperparameter feature coefficient quantization value C_q, in which case the first dequantization unit can dequantize the coefficient hyperparameter feature coefficient quantization value C_q to obtain the coefficient hyperparameter feature coefficient reconstruction value C'. Alternatively, if the first quantization unit is not enabled, the coefficient hyperparameter feature information is the coefficient hyperparameter feature coefficient value C, in which case the coefficient hyperparameter feature coefficient value C can be used as the coefficient hyperparameter feature coefficient reconstruction value C'.

[0334] Step S82, perform an inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value C' to obtain the coefficient hyperparameter feature value P. For example, the coefficient hyperparameter feature generation unit performs an inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value C' to obtain the coefficient hyperparameter feature value P, such as the coefficient hyperparameter feature generation unit performs an inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value C' based on the decoding neural network to obtain the coefficient hyperparameter feature value P.

[0335] Step S83, decode the code stream 2 corresponding to the current block to obtain the image feature information corresponding to the current block. For example, the second feature decoding unit can decode the code stream 2 corresponding to the current block to obtain the image feature information corresponding to the current block. When decoding the code stream 2 corresponding to the current block, the second feature decoding unit can use the coefficient hyperparameter feature value P to decode the code stream 2 corresponding to the current block.

[0336] Step S84: Determine an image feature reconstruction value based on the image feature information.

[0337] For example, if the second quantization unit is enabled, the image feature information is the image feature quantization value F_q, and the second dequantization unit can dequantize the image feature quantization value F_q to obtain the image feature reconstruction value F'. If the second quantization unit is not enabled, the image feature information is the image feature value F, and the image feature value F can be used as the image feature reconstruction value F'.

[0338] Step S85, perform an inverse transformation operation on the image feature reconstruction value F' to obtain the image low-order feature value LF. For example, the image feature inverse transformation unit performs an inverse transformation operation on the image feature reconstruction value F' to obtain the image low-order feature value LF, such as performing an inverse transformation operation on the image feature reconstruction value F' based on a decoding neural network to obtain the image low-order feature value LF.

[0339] Step S86, determining the reconstructed image block I corresponding to the current block based on the image low-order eigenvalue LF. Exemplarily, if the quality enhancement unit is enabled, the quality enhancement unit performs enhancement processing on the image low-order eigenvalue LF to obtain the reconstructed image block I corresponding to the current block, such as performing enhancement processing on the image low-order eigenvalue LF based on a decoding neural network to obtain the reconstructed image block I corresponding to the current block. If the quality enhancement unit is not enabled, the image low-order eigenvalue LF is used as the reconstructed image block I.

[0340] For example, regarding steps S80 to S86, please refer to Example 5, which will not be repeated here.

[0341] In one possible implementation, the coefficient hyperparameter feature generation unit may use a decoding neural network to perform an inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value C' to obtain a coefficient hyperparameter feature value P. The control parameter encoding unit may determine the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit of the decoding end device based on the network structure of the decoding neural network. The neural network information 1 is used to determine the decoding neural network 1 corresponding to the coefficient hyperparameter feature generation unit, and encode the neural network information 1 in the bitstream 0.

[0342] In one possible implementation, the image feature inverse transformation unit may use a decoding neural network to perform an inverse transformation operation on the image feature reconstruction value F' to obtain the image low-order feature value LF, and the control parameter encoding unit may determine the neural network information 2 corresponding to the image feature inverse transformation unit of the decoding end device based on the network structure of the decoding neural network. The neural network information 2 is used to determine the decoding neural network 2 corresponding to the image feature inverse transformation unit, and encode the neural network information 2 in the bitstream 0.

[0343] In one possible implementation, the quality enhancement unit may use a decoding neural network to enhance the low-order eigenvalues ​​LF of the image to obtain a reconstructed image block I corresponding to the current block. The control parameter encoding unit may determine the neural network information 3 corresponding to the quality enhancement unit of the decoding end device based on the network structure of the decoding neural network. The neural network information 3 is used to determine the decoding neural network 3 corresponding to the quality enhancement unit, and encode the neural network information 3 in the bitstream 0.

[0344] Embodiment 15: In the embodiment of the present application, a neural network-based image encoding method is proposed, which can be applied to the encoding end, see Figure 7C As shown, it is a structural diagram of the encoding end, and the encoding end may include a control parameter encoding unit, a feature transformation unit, a coefficient super-parameter feature transformation unit, a first feature encoding unit, a second feature encoding unit, a first feature decoding unit, a second feature decoding unit, a coefficient super-parameter feature generation unit, an image feature inverse transformation unit, and a quality enhancement unit.

[0345] Exemplarily, the neural network-based image encoding method in this embodiment may include the following steps:

[0346] Step S91: Perform feature transformation on the current block I to obtain an image feature value F corresponding to the current block I. For example, the feature transformation unit may perform feature transformation on the current block I based on a coding neural network to obtain an image feature value F corresponding to the current block I.

[0347] Step S92: Perform coefficient super-parameter feature transformation on the image feature value F to obtain coefficient super-parameter feature coefficient value C. For example, the coefficient super-parameter feature transformation unit performs coefficient super-parameter feature transformation on the image feature value F based on the encoding neural network to obtain coefficient super-parameter feature coefficient value C.

[0348] Step S93: Encode the coefficient super-parameter characteristic coefficient value C to obtain a bitstream 1. For example, the first feature encoding unit may encode the coefficient super-parameter characteristic coefficient value C in the bitstream corresponding to the current block to obtain a bitstream 1.

[0349] Step S94: Determine the coefficient super-parameter feature coefficient reconstruction value C' based on the coefficient super-parameter feature coefficient value C. If the coefficient super-parameter feature coefficient reconstruction value C' is the same as the coefficient super-parameter feature coefficient value C, perform an inverse transformation operation on the coefficient super-parameter feature coefficient reconstruction value C' to obtain the coefficient super-parameter feature value P. For example, the coefficient super-parameter feature generation unit performs an inverse transformation operation on the coefficient super-parameter feature coefficient reconstruction value C' to obtain the coefficient super-parameter feature value P.

[0350] Step S95: Encode the image feature value F based on the coefficient hyperparameter feature value P to obtain a code stream 2. For example, the second feature encoding unit encodes the image feature value F in the code stream corresponding to the current block to obtain a code stream 2.

[0351] Step S96: The control parameter encoding unit obtains the control parameter corresponding to the current block, which may include neural network information, and encodes the control parameter corresponding to the current block in the bitstream, and records the bitstream including the control parameter as bitstream 0.

[0352] Step S97, decode the bitstream 1 corresponding to the current block to obtain the coefficient super-parameter characteristic coefficient value C corresponding to the current block. For example, the first characteristic decoding unit can decode the bitstream 1 corresponding to the current block to obtain the coefficient super-parameter characteristic coefficient value C corresponding to the current block.

[0353] And, based on the coefficient hyper-parameter characteristic coefficient value C, the coefficient hyper-parameter characteristic coefficient reconstruction value C' is determined.

[0354] Step S98, perform an inverse transformation operation on the coefficient super-parameter feature coefficient reconstruction value C' to obtain the coefficient super-parameter feature value P. For example, the coefficient super-parameter feature generation unit performs an inverse transformation operation on the coefficient super-parameter feature coefficient reconstruction value C' based on the decoding neural network to obtain the coefficient super-parameter feature value P. Decode the bitstream 2 corresponding to the current block to obtain the image feature value F corresponding to the current block, and use the image feature value F as the image feature reconstruction value F', such as the second feature decoding unit uses the coefficient super-parameter feature value P to decode the bitstream 2 corresponding to the current block.

[0355] Step S99, perform an inverse transformation operation on the image feature reconstruction value F' to obtain the image low-order feature value LF. For example, the image feature inverse transformation unit performs an inverse transformation operation on the image feature reconstruction value F' based on the decoding neural network to obtain the image low-order feature value LF. Determine the reconstructed image block I corresponding to the current block based on the image low-order feature value LF. For example, the quality enhancement unit performs enhancement processing on the image low-order feature value LF based on the decoding neural network to obtain the reconstructed image block I corresponding to the current block.

[0356] Embodiment 16: For Embodiment 14 and Embodiment 15, the first feature encoding unit may encode coefficient super-parameter feature information in the bitstream 1 corresponding to the current block. The encoding process of the first feature encoding unit corresponds to the decoding process of the first feature decoding unit, see Embodiment 7. For example, the first feature encoding unit may use an entropy encoding method (such as an entropy encoding method such as CAVLC or CABAC) to encode the coefficient super-parameter feature information, which will not be repeated here. The first feature decoding unit may decode the bitstream 1 corresponding to the current block to obtain the coefficient super-parameter feature information corresponding to the current block. The process may refer to Embodiment 7.

[0357] For Example 14 and Example 15, the second feature encoding unit can encode the image feature information based on the coefficient hyperparameter feature value P to obtain the code stream 2. The encoding process of the second feature encoding unit corresponds to the decoding process of the second feature decoding unit. See Example 10. For example, the second feature encoding unit can encode the image feature information using an entropy encoding method (such as CAVLC or CABAC entropy encoding method), which will not be repeated here. The second feature decoding unit can decode the code stream 2 corresponding to the current block to obtain the image feature information corresponding to the current block. The process can be referred to in Example 10.

[0358] For Example 14 and Example 15, the first quantization unit can quantize the coefficient super-parameter characteristic coefficient value C to obtain the coefficient super-parameter characteristic coefficient quantization value C_q. The quantization process of the first quantization unit corresponds to the dequantization process of the first dequantization unit. See Example 8. For example, the first quantization unit quantizes the coefficient super-parameter characteristic coefficient value C based on the quantization-related parameters, which will not be repeated here. It should be noted that for quantization-related parameters (such as step size parameter qstep, also called quantization step size), 1) each eigenvalue of each feature channel uses the same quantization step size; 2) each feature channel uses a different quantization step size, but each eigenvalue in the feature channel uses the same quantization step size; 3) each eigenvalue of each feature channel uses a different quantization step size. The first dequantization unit at the encoding end can dequantize the coefficient super-parameter characteristic coefficient quantization value C_q to obtain the coefficient super-parameter characteristic coefficient reconstruction value C'. The processing process of the first dequantization unit at the encoding end can refer to Example 10.

[0359] The second quantization unit can quantize the image feature value F to obtain the image feature quantization value F_q. The quantization process of the second quantization unit corresponds to the inverse quantization process of the second inverse quantization unit. See Example 11. For example, the second quantization unit quantizes the image feature value F based on the quantization-related parameters, which will not be repeated here. It should be noted that for the quantization step size, 1) each feature value of each feature channel uses the same quantization step size; 2) each feature channel uses a different quantization step size, but each feature value in the feature channel uses the same quantization step size; 3) each feature value of each feature channel uses a different quantization step size. The second inverse quantization unit at the encoding end can inverse quantize the image feature quantization value F_q to obtain the image feature reconstruction value F'. The processing process of the second inverse quantization unit at the encoding end can refer to Example 11.

[0360] Embodiment 17: With respect to Embodiment 14 and Embodiment 15, the feature transformation unit may perform feature transformation on the current block I based on the coding neural network to obtain the image feature value F corresponding to the current block I. On this basis, the neural network information 2 corresponding to the image feature inverse transformation unit of the decoding end device may be determined based on the network structure of the coding neural network, and the neural network information 2 corresponding to the image feature inverse transformation unit may be encoded in the bitstream 0. For how the decoding end device generates the decoding neural network 2 corresponding to the image feature inverse transformation unit based on the neural network information 2, please refer to Embodiment 12, which will not be repeated here.

[0361] In a possible implementation, the feature transformation unit may include a coding neural network 2, and the coding neural network 2 may include a basic layer and an enhancement layer. The coding neural network 2 may be divided into a basic layer and an enhancement layer. The basic layer may include at least one network layer, and the basic layer may not include a network layer, that is, the basic layer is empty. The enhancement layer may include at least one network layer, and the enhancement layer may not include a network layer, that is, the enhancement layer is empty. It should be noted that for multiple network layers in the coding neural network 2, the multiple network layers can be divided into basic layers and enhancement layers according to actual needs. For example, a network layer with a fixed network structure can be used as a basic layer, and a network layer with a non-fixed network structure can be used as an enhancement layer.

[0362] Exemplarily, for the encoding neural network 2, the size of the output feature may be smaller than the size of the input feature, or the size of the output feature may be equal to the size of the input feature, or the size of the output feature may be larger than the size of the input feature.

[0363] Exemplarily, for the coding neural network 2, the base layer and the enhancement layer include at least one convolution layer. For example, the base layer includes at least one convolution layer, and the enhancement layer may include at least one convolution layer, or may not include a convolution layer. Alternatively, the enhancement layer includes at least one convolution layer, and the base layer may include at least one convolution layer, or may not include a convolution layer.

[0364] For the coding neural network 2, the base layer and the enhancement layer include at least one residual structure layer. For example, the base layer includes at least one residual structure layer, and the enhancement layer may include the residual structure layer or not. Alternatively, the enhancement layer includes at least one residual structure layer, and the base layer may include the residual structure layer or not.

[0365] For the coding neural network 2, it may include but is not limited to convolutional layers and activation layers, etc., and there is no limitation on this. For example, the coding neural network 2 includes 1 convolutional layer with a stride of 2, 1 activation layer, 1 convolutional layer with a stride of 1, and 1 activation layer in sequence, and all of the above network layers are used as basic layers. For another example, the coding neural network 2 includes 1 convolutional layer with a stride of 2, 1 activation layer, 1 convolutional layer with a stride of 1, and 1 activation layer in sequence, and all of the above network layers are used as basic layers. For another example, the coding neural network 2 includes 1 convolutional layer with a stride of 2, 1 activation layer, 1 convolutional layer with a stride of 2, 1 activation layer, 1 convolutional layer with a stride of 1, and 1 activation layer in sequence, and all of the above network layers are used as basic layers.

[0366] Exemplarily, the number of input channels of the first network layer of the feature conversion unit is 1 or 3. If the input image block contains only one channel (such as a grayscale image), the number of input channels of the first network layer of the feature conversion unit is 1; if the input image block contains three channels (such as RGB or YUV format), the number of input channels of the first network layer of the feature conversion unit is 3.

[0367] It should be noted that the network structure of the encoding neural network 2 of the feature transformation unit can be symmetrical with the network structure of the decoding neural network 2 of the image feature inverse transformation unit (see Example 12), and the network parameters of the encoding neural network 2 of the feature transformation unit can be the same or different with the network parameters of the decoding neural network 2 of the image feature inverse transformation unit.

[0368] In one possible implementation, a network layer of a default network structure may be configured for the feature transformation unit, and the network parameters related to the network layer of the default network structure are fixed. For example, in the network layer of the default network structure, the number of deconvolution layers is fixed, the number of activation layers is fixed, the number of channels of each deconvolution layer is fixed, the size of the convolution kernel is fixed, the filter coefficient is fixed, etc. It should be noted that the network layer of the default network structure configured for the feature transformation unit and the network layer of the default network structure configured for the image feature inverse transformation unit (see Example 12) can be a symmetrical structure.

[0369] In one possible implementation, a prefabricated neural network pool (corresponding to the prefabricated neural network pool of the decoding end device) may be configured for the feature transformation unit, and the prefabricated neural network pool includes at least one network layer of a prefabricated network structure, and the network parameters related to the network layer of the prefabricated network structure may be configured according to actual needs. For example, the prefabricated neural network pool may include a network layer of a prefabricated network structure t1', a network layer of a prefabricated network structure t2', a network layer of a prefabricated network structure t3', and the like. It should be noted that the network layer of the prefabricated network structure t1' and the network layer of the prefabricated network structure t1 (see Example 12) may be symmetrical structures. The network layer of the prefabricated network structure t2' and the network layer of the prefabricated network structure t2 may be symmetrical structures. The network layer of the prefabricated network structure t3' and the network layer of the prefabricated network structure t3 may be symmetrical structures.

[0370] In a possible implementation, a network layer of a variable network structure may be dynamically generated for a feature transformation unit based on network parameters, and network parameters related to the network layer of the variable network structure are dynamically generated by the encoding end rather than being pre-configured.

[0371] In a possible implementation, the encoding end may determine the neural network information 2 corresponding to the image feature inverse transformation unit based on the network structure of the encoding neural network 2, and encode the neural network information 2 corresponding to the image feature inverse transformation unit in the bitstream 0. For example, the encoding end may determine the neural network information 2 using the following situation:

[0372] Case 1: If the basic layer of the coding neural network 2 uses a network layer of a default network structure, and the enhancement layer uses a network layer of a default network structure, the encoder encodes the basic layer information and the enhancement layer information in the bitstream 0, and the basic layer information and the enhancement layer information constitute the neural network information 2 corresponding to the image feature inverse transformation unit. Among them, the basic layer information includes a flag bit that the basic layer uses the default network, and the flag bit that the basic layer uses the default network indicates that the basic layer uses the default network. The enhancement layer information includes a flag bit that the enhancement layer uses the default network, and the flag bit that the enhancement layer uses the default network indicates that the enhancement layer uses the default network.

[0373] Case 2: If the basic layer of the coding neural network 2 adopts a network layer of a default network structure, and the enhancement layer adopts a network layer of a prefabricated network structure, the encoder encodes the basic layer information and the enhancement layer information in the bitstream 0, and the basic layer information and the enhancement layer information constitute the neural network information 2 corresponding to the image feature inverse transformation unit. Among them, the basic layer information includes a flag bit that the basic layer uses the default network, and the flag bit that the basic layer uses the default network indicates that the basic layer uses the default network. The enhancement layer information includes a flag bit that the enhancement layer uses a prefabricated network and an enhancement layer prefabricated network index number, and the flag bit that the enhancement layer uses a prefabricated network indicates that the enhancement layer uses a prefabricated network, and the enhancement layer prefabricated network index number indicates the corresponding index of the network layer of the prefabricated network structure in the prefabricated neural network pool. For example, if the enhancement layer of the coding neural network 2 adopts a network layer of a prefabricated network structure t1', the enhancement layer prefabricated network index number indicates the first network layer of the prefabricated network structure t1' in the prefabricated neural network pool.

[0374] Case 3: If the basic layer of the coding neural network 2 uses a network layer with a default network structure, and the enhancement layer uses an enhancement layer with a variable network structure, the encoder encodes the basic layer information and the enhancement layer information in the bitstream 0, and the basic layer information and the enhancement layer information constitute the neural network information 2 corresponding to the image feature inverse transformation unit. Among them, the basic layer information includes a flag bit that the basic layer uses the default network, and the flag bit that the basic layer uses the default network indicates that the basic layer uses the default network. The enhancement layer information includes network parameters for generating the enhancement layer, and the network parameters may include but are not limited to at least one of the following: the number of neural network layers, the deconvolution layer flag bit, the number of deconvolution layers, the quantization step size of each deconvolution layer, the number of channels of each deconvolution layer, the size of the convolution kernel, the number of filters, the filter size index, the filter coefficient zero flag bit, the filter coefficient, the activation layer flag bit, and the activation layer type. It should be noted that the network parameters in the enhancement layer information may be different from the network parameters used in the enhancement layer of the coding neural network 2, that is, for each network parameter used in the enhancement layer, a network parameter symmetrical to the network parameter can be generated, and there is no restriction on this generation process, and this symmetrical network parameter is transmitted to the decoding end as the enhancement layer information.

[0375] Case 4: If the basic layer of the coding neural network 2 uses a network layer of a prefabricated network structure, and the enhancement layer uses a network layer of a default network structure, the basic layer information and the enhancement layer information are encoded in the bitstream 0, and the basic layer information and the enhancement layer information constitute the neural network information 2 corresponding to the image feature inverse transformation unit. Among them, the basic layer information includes a basic layer using a prefabricated network flag and a basic layer prefabricated network index number, the basic layer using a prefabricated network flag indicates that the basic layer uses a prefabricated network, and the basic layer prefabricated network index number indicates the corresponding index of the network layer of the prefabricated network structure in the prefabricated neural network pool. The enhancement layer information includes an enhancement layer using a default network flag, and the enhancement layer using a default network flag indicates that the enhancement layer uses a default network.

[0376] Case 5: If the basic layer of the coding neural network 2 uses a network layer of a prefabricated network structure, and the enhancement layer uses a network layer of a prefabricated network structure, the encoding end encodes the basic layer information and the enhancement layer information in the bitstream 0, and the basic layer information and the enhancement layer information constitute the neural network information 2 corresponding to the image feature inverse transformation unit. Among them, the basic layer information may include a basic layer using a prefabricated network flag and a basic layer prefabricated network index number, the basic layer using a prefabricated network flag is used to indicate that the basic layer uses a prefabricated network, and the basic layer prefabricated network index number is used to indicate the corresponding index of the network layer of the prefabricated network structure in the prefabricated neural network pool. The enhancement layer information may include an enhancement layer using a prefabricated network flag and an enhancement layer prefabricated network index number, and the enhancement layer using a prefabricated network flag is used to indicate that the enhancement layer uses a prefabricated network, and the enhancement layer prefabricated network index number is used to indicate the corresponding index of the network layer of the prefabricated network structure in the prefabricated neural network pool.

[0377] Case 6: If the basic layer of the coding neural network 2 uses a network layer of a prefabricated network structure, and the enhancement layer uses an enhancement layer of a variable network structure, the basic layer information and the enhancement layer information are encoded in the bitstream 0, and the basic layer information and the enhancement layer information constitute the neural network information 2 corresponding to the image feature inverse transformation unit. Among them, the basic layer information includes a prefabricated network flag for the basic layer and a prefabricated network index number for the basic layer, the prefabricated network flag for the basic layer indicates that the basic layer uses a prefabricated network, and the prefabricated network index number for the basic layer indicates the corresponding index of the network layer of the prefabricated network structure in the prefabricated neural network pool. The enhancement layer information includes network parameters for generating the enhancement layer, and the network parameters may include but are not limited to at least one of the following: the number of neural network layers, the deconvolution layer flag, the number of deconvolution layers, the quantization step size of each deconvolution layer, the number of channels of each deconvolution layer, the size of the convolution kernel, the number of filters, the filter size index, the filter coefficient zero flag, the filter coefficient, the activation layer flag, and the activation layer type. The network parameters in the enhancement layer information may be different from the network parameters used in the enhancement layer of the coding neural network 2. That is to say, for each network parameter used in the enhancement layer, a network parameter symmetrical to the network parameter can be generated. There is no restriction on this generation process, and this symmetrical network parameter is transmitted to the decoding end as the enhancement layer information.

[0378] Embodiment 18: With respect to Embodiment 14 and Embodiment 15, the image feature inverse transformation unit at the encoding end can perform an inverse transformation operation on the image feature reconstruction value F' based on the decoding neural network to obtain the image low-order feature value LF. On this basis, the neural network information 2 corresponding to the image feature inverse transformation unit of the decoding end device can also be determined based on the network structure of the decoding neural network, and the neural network information 2 corresponding to the image feature inverse transformation unit can be encoded in the bitstream 0.

[0379] In one possible implementation, the image feature inverse transformation unit at the encoding end may include a decoding neural network 2, and the decoding neural network 2 may include a basic layer and an enhancement layer. The network structure of the decoding neural network 2 at the encoding end is the same as the network structure of the decoding neural network 2 at the decoding end. Please refer to Example 12 and will not be repeated here.

[0380] In one possible implementation, the encoder can configure a network layer of a default network structure for the image feature inverse transformation unit, and the network layer of the default network structure is the same as the network layer of the default network structure of the decoder. The encoder can configure a prefabricated neural network pool for the image feature inverse transformation unit, and the prefabricated neural network pool can include at least one network layer of a prefabricated network structure, and the prefabricated neural network pool is the same as the prefabricated neural network pool of the decoder. The encoder can dynamically generate a network layer of a variable network structure for the image feature inverse transformation unit based on network parameters. The encoder can determine the neural network information 2 corresponding to the image feature inverse transformation unit based on the network structure of the decoding neural network 2, and encode the neural network information 2 in the bitstream 0.

[0381] For example, if the basic layer of the decoding neural network 2 adopts a network layer of a default network structure, and the enhancement layer adopts a network layer of a default network structure, the basic layer information includes a basic layer using a default network flag, and the basic layer using a default network flag indicates that the basic layer uses the default network. The enhancement layer information includes an enhancement layer using a default network flag, and the enhancement layer using a default network flag indicates that the enhancement layer uses the default network. For another example, if the basic layer of the decoding neural network 2 adopts a network layer of a default network structure, and the enhancement layer adopts a network layer of a prefabricated network structure, the basic layer information includes a basic layer using a default network flag, and the basic layer using a default network flag indicates that the basic layer uses the default network. The enhancement layer information includes an enhancement layer using a prefabricated network flag and an enhancement layer prefabricated network index number, and the enhancement layer using a prefabricated network flag indicates that the enhancement layer uses a prefabricated network, and the enhancement layer prefabricated network index number indicates the corresponding index of the network layer of the prefabricated network structure in the prefabricated neural network pool. For another example, if the base layer of the decoding neural network 2 uses a network layer of a default network structure, and the enhancement layer uses an enhancement layer of a variable network structure, the base layer information includes a base layer using a default network flag, and the base layer using a default network flag indicates that the base layer uses a default network. The enhancement layer information includes network parameters for generating the enhancement layer, and the network parameters in the enhancement layer information may be the same as the network parameters used by the enhancement layer of the decoding neural network 2.

[0382] For example, if the basic layer of the decoding neural network 2 adopts a network layer of a prefabricated network structure, and the enhanced layer adopts a network layer of a default network structure, then the basic layer information includes a basic layer using a prefabricated network flag and a basic layer prefabricated network index number, the basic layer using a prefabricated network flag indicates that the basic layer uses a prefabricated network, and the basic layer prefabricated network index number indicates the corresponding index of the network layer of the prefabricated network structure in the prefabricated neural network pool. The enhanced layer information includes an enhanced layer using a default network flag, and the enhanced layer using a default network flag indicates that the enhanced layer uses the default network. For another example, if the basic layer of the decoding neural network 2 adopts a network layer of a prefabricated network structure, and the enhanced layer adopts a network layer of a prefabricated network structure, then the basic layer information includes a basic layer using a prefabricated network flag and a basic layer prefabricated network index number, the basic layer using a prefabricated network flag indicates that the basic layer uses a prefabricated network, and the basic layer prefabricated network index number indicates the corresponding index of the network layer of the prefabricated network structure in the prefabricated neural network pool. The enhancement layer information includes a flag that the enhancement layer uses a prefabricated network and an index number of the prefabricated network of the enhancement layer. The flag that the enhancement layer uses a prefabricated network indicates that the enhancement layer uses a prefabricated network, and the index number of the prefabricated network of the enhancement layer indicates the corresponding index of the network layer of the prefabricated network structure in the prefabricated neural network pool. For another example, if the basic layer of the decoding neural network 2 uses a network layer of a prefabricated network structure, and the enhancement layer uses an enhancement layer of a variable network structure, then the basic layer information includes a flag that the basic layer uses a prefabricated network and an index number of the prefabricated network of the basic layer. The flag that the basic layer uses a prefabricated network indicates that the basic layer uses a prefabricated network, and the index number of the prefabricated network of the basic layer indicates the corresponding index of the network layer of the prefabricated network structure in the prefabricated neural network pool. The enhancement layer information includes network parameters, and the network parameters in the enhancement layer information are the same as the network parameters used by the enhancement layer of the decoding neural network 2.

[0383] Example 19: With respect to Example 14 and Example 15, the coefficient super-parameter feature transformation unit can perform coefficient super-parameter feature transformation on the image feature value F based on the encoding neural network to obtain the coefficient super-parameter feature coefficient value C. On this basis, the neural network information 1 corresponding to the coefficient super-parameter feature generation unit of the decoding end device can be determined based on the network structure of the encoding neural network, and the neural network information 1 corresponding to the coefficient super-parameter feature generation unit can be encoded in the bitstream 0. For how the decoding end device generates the decoding neural network 1 corresponding to the coefficient super-parameter feature generation unit based on the neural network information 1, please refer to Example 9, which will not be repeated here.

[0384] In a possible implementation, the coefficient super-parameter feature conversion unit may include a coding neural network 1, and the coding neural network 1 may include a basic layer and an enhancement layer. The coding neural network 1 may be divided into a basic layer and an enhancement layer, and the basic layer may include at least one network layer, or the basic layer may not include a network layer, that is, the basic layer is empty. The enhancement layer may include at least one network layer, or the enhancement layer may not include a network layer, that is, the enhancement layer is empty. It should be noted that the network structure of the coding neural network 1 of the coefficient super-parameter feature conversion unit may be symmetrical with the network structure of the decoding neural network 1 of the coefficient super-parameter feature generation unit (see Example 9), and the network parameters of the coding neural network 1 of the coefficient super-parameter feature conversion unit may be the same as or different from the network parameters of the decoding neural network 1 of the coefficient super-parameter feature generation unit, and the network structure of the coding neural network 1 will not be described in detail.

[0385] In a possible implementation, a network layer of a default network structure may be configured for the coefficient hyperparameter feature conversion unit, and the network parameters related to the network layer of the default network structure are all fixed. It should be noted that the network layer of the default network structure configured for the coefficient hyperparameter feature conversion unit and the network layer of the default network structure configured for the coefficient hyperparameter feature generation unit may be symmetrical structures. A prefabricated neural network pool (corresponding to the prefabricated neural network pool of the decoding end device) may be configured for the coefficient hyperparameter feature conversion unit, the prefabricated neural network pool including at least one network layer of a prefabricated network structure, and the network parameters related to the network layer of the prefabricated network structure may be configured according to actual needs. A network layer of a variable network structure may be dynamically generated for the coefficient hyperparameter feature conversion unit based on the network parameters, and the network parameters related to the network layer of the variable network structure are dynamically generated by the encoding end.

[0386] In a possible implementation, the encoding end may determine the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit based on the network structure of the encoding neural network 1, and encode the neural network information 1 in the bitstream 0. The neural network information 1 may include basic layer information and enhanced layer information. The method for encoding the neural network information 1 is similar to the method for encoding the neural network information 2. Please refer to Example 17, which will not be repeated here. For example, if the basic layer of the encoding neural network 1 adopts a network layer with a default network structure, and the enhanced layer adopts a network layer with a default network structure, the basic layer information includes a basic layer using a default network flag, and the basic layer using a default network flag indicates that the basic layer uses the default network. The enhanced layer information includes an enhanced layer using a default network flag, and the enhanced layer using a default network flag indicates that the enhanced layer uses the default network.

[0387] Embodiment 20: With respect to Embodiment 14 and Embodiment 15, the coefficient super-parameter feature generation unit at the encoding end may perform an inverse transformation operation on the coefficient super-parameter feature coefficient reconstruction value C' based on the decoding neural network to obtain the coefficient super-parameter feature value P. On this basis, the neural network information 1 corresponding to the coefficient super-parameter feature generation unit of the decoding end device may also be determined based on the network structure of the decoding neural network, and the neural network information 1 corresponding to the coefficient super-parameter feature generation unit may be encoded in the bitstream 0.

[0388] In one possible implementation, the coefficient hyperparameter feature generation unit at the encoding end may include a decoding neural network 1, which may include a basic layer and an enhancement layer. The network structure of the decoding neural network 1 at the encoding end is the same as the network structure of the decoding neural network 1 at the decoding end. Please refer to Example 9 and will not be repeated here.

[0389] In a possible implementation, the encoding end may configure a network layer of a default network structure for the coefficient super-parameter feature generation unit, and the network layer of the default network structure is the same as the network layer of the default network structure of the decoding end. The encoding end may configure a prefabricated neural network pool for the coefficient super-parameter feature generation unit, and the prefabricated neural network pool may include at least one network layer of a prefabricated network structure, and the prefabricated neural network pool is the same as the prefabricated neural network pool of the decoding end. The encoding end may dynamically generate a network layer of a variable network structure for the coefficient super-parameter feature generation unit based on network parameters. The encoding end may determine the neural network information 1 corresponding to the coefficient super-parameter feature generation unit based on the network structure of the decoding neural network 1, and encode the neural network information 1 in the bitstream 0. Among them, the neural network information 1 may include basic layer information and enhanced layer information. The method for encoding the neural network information 1 is similar to the method for encoding the neural network information 2, and can be referred to Example 18, which will not be repeated here. For example, if the basic layer of the decoding neural network 1 (adopted by the encoding end) adopts a network layer of a default network structure, and the enhanced layer adopts an enhanced layer of a variable network structure, the basic layer information includes a basic layer using a default network flag, and the basic layer using a default network flag indicates that the basic layer uses a default network. The enhancement layer information includes network parameters used to generate the enhancement layer. The network parameters in the enhancement layer information may be the same as the network parameters used by the enhancement layer of the decoding neural network 1 (ie, the enhancement layer used by the encoding end).

[0390] Embodiment 21: With respect to Embodiment 14 and Embodiment 15, the quality enhancement unit at the encoding end may enhance the low-order feature value LF of the image based on the decoding neural network to obtain the reconstructed image block I corresponding to the current block. On this basis, the neural network information 3 corresponding to the quality enhancement unit of the decoding end device may also be determined based on the network structure of the decoding neural network, and the neural network information 3 corresponding to the quality enhancement unit may be encoded in the bitstream 0. For example, the quality enhancement unit at the encoding end may include a decoding neural network 3, and the decoding neural network 3 may include a basic layer and an enhancement layer. The network structure of the decoding neural network 3 at the encoding end is the same as the network structure of the decoding neural network 3 at the decoding end, and reference may be made to Embodiment 13, which will not be described in detail here.

[0391] Exemplarily, the encoding end may configure a network layer of a default network structure for the quality enhancement unit, and the network layer of the default network structure is the same as the network layer of the default network structure of the decoding end. The encoding end may configure a prefabricated neural network pool for the quality enhancement unit, and the prefabricated neural network pool may include at least one network layer of a prefabricated network structure, and the prefabricated neural network pool is the same as the prefabricated neural network pool of the decoding end. The encoding end may dynamically generate a network layer of a variable network structure for the quality enhancement unit based on network parameters. The encoding end determines the neural network information 3 corresponding to the quality enhancement unit based on the network structure of the decoding neural network 3, and encodes the neural network information 3 in the bitstream 0. Among them, the neural network information 3 may include basic layer information and enhanced layer information. Regarding the method of encoding the neural network information 3, similar to the method of encoding the neural network information 2, please refer to Example 18, which will not be repeated here. For example, if the basic layer of the decoding neural network 3 adopts a network layer of a default network structure, and the enhanced layer adopts an enhanced layer of a variable network structure, the basic layer information includes a basic layer using a default network flag, and the basic layer using a default network flag indicates that the basic layer uses the default network. The enhancement layer information includes network parameters used to generate the enhancement layer. The network parameters in the enhancement layer information may be the same as the network parameters used in the enhancement layer of the decoding neural network 3.

[0392] In a possible implementation, the network parameters in Examples 1 to 21 may all be fixed-point network parameters, such as filter weights in the network parameters represented by 4, 8, 16, 32, or 64 bits in width. The characteristic values ​​output by the network may also be restricted in bit width, such as being restricted to 4, 8, 16, 32, or 64 bits in width.

[0393] Exemplarily, the bit width of the network parameter can be limited to 8 bits, and its value size can be limited to between [-127, 127]; the bit width of the eigenvalue output by the network can be limited to 8 bits, and its value size can be limited to between [-127, 127].

[0394] Embodiment 22: For the syntax table related to the picture header, see Table 4, which provides syntax information related to the picture header, namely, picture-level syntax. In Table 4, u(n) is used to represent an n-bit fixed-length code encoding method.

[0395] Table 4

[0396]

[0397]

[0398] In Table 4, pic_width indicates the image width, pic_height indicates the image height, and pic_format indicates the image format, such as RGB 444, YUV444, YUV420, and YUV422. bu_width indicates the width of the basic block, bu_height indicates the height of the basic block, block_width indicates the width of the image block, block_height indicates the height of the image block, bit_depth indicates the image bit depth, pic_qp indicates the quantization parameter in the current image, lossless_flag indicates whether the current image is marked for lossless coding, and feature_map_max_bit_depth indicates the maximum bit depth of the feature map, which is used to limit the maximum and minimum values ​​of the network input or output feature map.

[0399] Illustratively, the above-mentioned embodiments can be implemented individually or in combination. For example, each embodiment in Embodiment 1 to Embodiment 22 can be implemented individually, and at least two embodiments in Embodiment 1 to Embodiment 22 can be implemented in combination.

[0400] Illustratively, in the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.

[0401] Based on the same application concept as the above method, an image decoding device based on a neural network is also proposed in an embodiment of the present application. The device is applied to a decoding end, and the device includes: a memory, which is configured to store video data; a decoder, which is configured to implement the decoding method in the above embodiments 1 to 22, that is, the processing flow of the decoding end.

[0402] For example, in one possible implementation, the decoder is configured to implement:

[0403] Decoding the control parameters and image information corresponding to the current block from the bitstream;

[0404] Acquire neural network information corresponding to a decoding processing unit from the control parameter, and generate a decoding neural network corresponding to the decoding processing unit based on the neural network information;

[0405] An input feature corresponding to the decoding processing unit is determined based on the image information, and the input feature is processed based on the decoding neural network to obtain an output feature corresponding to the decoding processing unit.

[0406] Based on the same application concept as the above method, a neural network-based image encoding device is also proposed in the embodiment of the present application. The device is applied to the encoding end, and the device includes: a memory, which is configured to store video data; an encoder, which is configured to implement the encoding method in the above-mentioned embodiments 1 to 22, that is, the processing flow of the encoding end.

[0407] For example, in one possible implementation, the encoder is configured to implement:

[0408] Determine an input feature corresponding to a coding processing unit based on the current block, process the input feature based on a coding neural network corresponding to the coding processing unit to obtain an output feature corresponding to the coding processing unit, and determine image information corresponding to the current block based on the output feature;

[0409] Acquire control parameters corresponding to the current block, wherein the control parameters include neural network information corresponding to a decoding processing unit, and the neural network information is used to determine a decoding neural network corresponding to the decoding processing unit;

[0410] The image information and control parameters corresponding to the current block are encoded in the bitstream.

[0411] Based on the same application concept as the above method, the decoding end device (also referred to as a video decoder) provided in the embodiment of the present application, from the hardware level, its hardware architecture diagram can be specifically referred to as Fig. 8A As shown. It includes: a processor 811 and a machine-readable storage medium 812, the machine-readable storage medium 812 stores machine-executable instructions that can be executed by the processor 811; the processor 811 is used to execute the machine-executable instructions to implement the decoding method of the above embodiments 1-22 of the present application.

[0412] Based on the same application concept as the above method, the encoding end device (also referred to as a video encoder) provided in the embodiment of the present application, from the hardware level, its hardware architecture diagram can be specifically referred to as Figure 8BThe system comprises: a processor 821 and a machine-readable storage medium 822, wherein the machine-readable storage medium 822 stores machine-executable instructions that can be executed by the processor 821; the processor 821 is used to execute the machine-executable instructions to implement the encoding method of the above-mentioned embodiments 1-22 of the present application.

[0413] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented, such as the decoding method or encoding method in the above embodiments.

[0414] Based on the same application concept as the above method, an embodiment of the present application also provides a computer application, which, when executed by a processor, can implement the decoding method or encoding method disclosed in the above example of the present application.

[0415] Based on the same application concept as the above method, an image decoding device based on a neural network is also proposed in an embodiment of the present application. The device is applied to a decoding end, and the device includes: a decoding module, used to decode control parameters and image information corresponding to a current block from a bit stream; an acquisition module, used to acquire neural network information corresponding to a decoding processing unit from the control parameters, and generate a decoding neural network corresponding to the decoding processing unit based on the neural network information; a processing module, used to determine input features corresponding to the decoding processing unit based on the image information, and process the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit.

[0416] Exemplarily, if the neural network information includes basic layer information and enhanced layer information, when the acquisition module generates a decoding neural network corresponding to the decoding processing unit based on the neural network information, it is specifically used to: determine the basic layer corresponding to the decoding processing unit based on the basic layer information; determine the enhanced layer corresponding to the decoding processing unit based on the enhanced layer information; and generate a decoding neural network corresponding to the decoding processing unit based on the basic layer and the enhanced layer.

[0417] Exemplarily, when the acquisition module determines the basic layer corresponding to the decoding processing unit based on the basic layer information, it is specifically used to: if the basic layer information includes a basic layer using a default network flag, and the basic layer using a default network flag indicates that the basic layer uses a default network, then acquire the basic layer of the default network structure.

[0418] Exemplarily, when the acquisition module determines the basic layer corresponding to the decoding processing unit based on the basic layer information, it is specifically used: if the basic layer information includes a basic layer using a prefabricated network flag and a basic layer prefabricated network index number, and the basic layer using a prefabricated network flag indicates that the basic layer uses a prefabricated network, then the basic layer of the prefabricated network structure corresponding to the basic layer prefabricated network index number is selected from the prefabricated neural network pool; wherein the prefabricated neural network pool includes at least one network layer of a prefabricated network structure.

[0419] Exemplarily, when the acquisition module determines the enhancement layer corresponding to the decoding processing unit based on the enhancement layer information, it is specifically used to: if the enhancement layer information includes an enhancement layer using a default network flag, and the enhancement layer using a default network flag indicates that the enhancement layer uses a default network, then acquire the enhancement layer of the default network structure.

[0420] Exemplarily, when the acquisition module determines the enhancement layer corresponding to the decoding processing unit based on the enhancement layer information, it is specifically used: if the enhancement layer information includes an enhancement layer using a prefabricated network flag and an enhancement layer prefabricated network index number, and the enhancement layer using a prefabricated network flag indicates that the enhancement layer uses a prefabricated network, then the enhancement layer of the prefabricated network structure corresponding to the enhancement layer prefabricated network index number is selected from the prefabricated neural network pool; wherein the prefabricated neural network pool includes at least one network layer of a prefabricated network structure.

[0421] Exemplarily, when the acquisition module determines the enhancement layer corresponding to the decoding processing unit based on the enhancement layer information, it is specifically used to: if the enhancement layer information includes network parameters for generating the enhancement layer, then the enhancement layer corresponding to the decoding processing unit is generated based on the network parameters; wherein the network parameters include at least one of the following: the number of neural network layers, the deconvolution layer flag, the number of deconvolution layers, the quantization step of each deconvolution layer, the number of channels of each deconvolution layer, the size of the convolution kernel, the number of filters, the filter size index, the filter coefficient zero flag, the filter coefficient, the activation layer flag, and the activation layer type.

[0422] Exemplarily, the image information includes coefficient hyperparameter feature information and image feature information, and the processing module determines the input feature corresponding to the decoding processing unit based on the image information, processes the input feature based on the decoding neural network, and obtains the output feature corresponding to the decoding processing unit, which is specifically used for: when executing the decoding process of coefficient hyperparameter feature generation, determining the coefficient superparameter feature coefficient reconstruction value based on the coefficient superparameter feature information; performing an inverse transformation operation on the coefficient superparameter feature coefficient reconstruction value based on the decoding neural network to obtain the coefficient superparameter feature value; wherein the coefficient hyperparameter feature value is used to decode the image feature information from the bitstream; when executing the decoding process of image feature inverse transformation, determining the image feature reconstruction value based on the image feature information; performing an inverse transformation operation on the image feature reconstruction value based on the decoding neural network to obtain the image low-order feature value; wherein the image low-order feature value is used to obtain the reconstructed image block corresponding to the current block.

[0423] Exemplarily, when the processing module determines the coefficient hyperparameter characteristic coefficient reconstruction value based on the coefficient hyperparameter characteristic information, it is specifically used for: if the control parameter includes first enable information, and the first enable information indicates enabling the first dequantization operation, then the coefficient hyperparameter characteristic information is dequantized to obtain the coefficient hyperparameter characteristic coefficient reconstruction value.

[0424] Exemplarily, when the processing module determines the image feature reconstruction value based on the image feature information, it is specifically used to: if the control parameter includes second enable information, and the second enable information indicates enabling a second dequantization operation, then dequantize the image feature information to obtain the image feature reconstruction value.

[0425] Exemplarily, the processing module is also used for: if the control parameters include third enabling information, the third enabling information indicates enabling the quality enhancement operation, when performing the quality enhancement decoding process, obtaining the low-order eigenvalues ​​of the image, and performing enhancement processing on the low-order eigenvalues ​​of the image based on the decoding neural network to obtain the reconstructed image block corresponding to the current block.

[0426] Based on the same application concept as the above method, an image encoding device based on a neural network is also proposed in an embodiment of the present application. The device is applied to the encoding end, and the device includes: a processing module, which is used to determine the input features corresponding to the encoding processing unit based on the current block, and process the input features based on the encoding neural network corresponding to the encoding processing unit to obtain the output features corresponding to the encoding processing unit, and determine the image information corresponding to the current block based on the output features; an acquisition module, which is used to obtain the control parameters corresponding to the current block, and the control parameters include the neural network information corresponding to the decoding processing unit, and the neural network information is used to determine the decoding neural network corresponding to the decoding processing unit; an encoding module, which is used to encode the image information and control parameters corresponding to the current block in the bitstream.

[0427] Exemplarily, the neural network information includes basic layer information and enhanced layer information, and the decoding neural network includes a basic layer determined based on the basic layer information, and an enhanced layer determined based on the enhanced layer information.

[0428] Exemplarily, if the basic layer information includes a basic layer using a default network flag, and the basic layer using a default network flag indicates that the basic layer uses a default network, the decoding neural network adopts the basic layer of the default network structure.

[0429] Exemplarily, if the basic layer information includes a basic layer using a prefabricated network flag and a basic layer prefabricated network index number, and the basic layer using a prefabricated network flag indicates that the basic layer uses a prefabricated network, then the decoding neural network adopts the basic layer of the prefabricated network structure corresponding to the basic layer prefabricated network index number selected from the prefabricated neural network pool; wherein the prefabricated neural network pool includes at least one network layer of a prefabricated network structure.

[0430] Exemplarily, if the enhancement layer information includes a flag bit that the enhancement layer uses a default network, and the flag bit that the enhancement layer uses a default network indicates that the enhancement layer uses a default network, the decoding neural network adopts the enhancement layer of the default network structure.

[0431] Exemplarily, if the enhancement layer information includes an enhancement layer using a prefabricated network flag and an enhancement layer prefabricated network index number, and the enhancement layer using a prefabricated network flag indicates that the enhancement layer uses a prefabricated network, then the decoding neural network adopts an enhancement layer of a prefabricated network structure corresponding to the enhancement layer prefabricated network index number selected from a prefabricated neural network pool; wherein the prefabricated neural network pool includes at least one network layer of a prefabricated network structure.

[0432] Exemplarily, if the enhancement layer information includes network parameters for generating an enhancement layer, the decoding neural network adopts an enhancement layer generated based on the network parameters; wherein the network parameters include at least one of the following: the number of neural network layers, the deconvolution layer flag, the number of deconvolution layers, the quantization step size of each deconvolution layer, the number of channels of each deconvolution layer, the size of the convolution kernel, the number of filters, the filter size index, the filter coefficient zero flag, the filter coefficient, the activation layer flag, and the activation layer type.

[0433] Exemplarily, the processing module is also used to: divide the current image into N non-overlapping image blocks, where N is a positive integer; perform boundary filling on each image block to obtain a boundary-filled image block; wherein, when performing boundary filling on each image block, the filling value does not depend on the reconstructed pixel values ​​of adjacent image blocks; and generate N current blocks based on the boundary-filled image blocks.

[0434] Exemplarily, the processing module is also used to: divide the current image into multiple basic blocks, and each basic block includes at least one image block; fill the boundaries of each image block to obtain an image block after the boundaries are filled; wherein, when filling the boundaries of each image block, the filling value of the image block does not depend on the reconstructed pixel values ​​of other image blocks in the same basic block, and is allowed to depend on the reconstructed pixel values ​​of image blocks in different basic blocks; generate multiple current blocks based on the image blocks after the boundaries are filled.

[0435] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. The present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. The embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0436] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image decoding method, characterized in that: The method comprises: Decode the control parameters corresponding to the current block from the bitstream; Acquire enhancement layer information from the control parameter, the enhancement layer information including network parameters for configuring an enhancement layer network, and configure the enhancement layer network based on the network parameters; Determine input features corresponding to the enhancement layer network, process the input features based on the enhancement layer network, and obtain output features corresponding to the enhancement layer.

2. The method according to claim 1, characterized in that The network parameters include at least one of the following: Number of neural network layers, number of filters, filter size index, filter coefficient zero flag, filter coefficient.

3. The method according to claim 1, characterized in that If the enhancement layer network is a quality enhancement network, determining the input features corresponding to the enhancement layer network includes: Decoding the code stream of the current block to obtain image feature information corresponding to the current block; Determine an image feature reconstruction value corresponding to the current block based on the image feature information; The image feature reconstruction value is processed to obtain a low-order image feature value corresponding to the current block.

4. The method according to claim 3, characterized in that The processing of the input features based on the enhancement layer network to obtain output features corresponding to the enhancement layer includes: If the input features include low-order feature values ​​of the image corresponding to the current block, the low-order feature values ​​of the image corresponding to the current block are enhanced based on the quality enhancement network to obtain a reconstructed image block corresponding to the current block.

5. The method according to claim 4, characterized in that The enhancing process is performed on the low-order feature values ​​of the image corresponding to the current block based on the quality enhancement network to obtain the reconstructed image block corresponding to the current block, including: if the control parameter also includes third enabling information, and the third enabling information indicates enabling the quality enhancement operation, then the enhancing process is performed on the low-order feature values ​​of the image corresponding to the current block based on the quality enhancement network to obtain the reconstructed image block corresponding to the current block.

6. The method according to any one of claims 1 to 5, characterized in that: The decoding of the control parameters corresponding to the current block from the bitstream includes: decoding the bitstream corresponding to the current block through a control parameter decoding network to obtain the control parameters corresponding to the current block, wherein the control parameters include control parameters of a quality enhancement network.

7. An image encoding method, characterized in that: The method comprises: Acquire control parameters corresponding to the current block, wherein the control parameters include enhancement layer information; wherein the enhancement layer information includes network parameters for configuring an enhancement layer network; The control parameters corresponding to the current block are encoded in the bitstream.

8. An image decoding device, characterized in that: The device comprises: A decoding module, used for decoding control parameters corresponding to the current block from the bitstream; An acquisition module, configured to acquire enhancement layer information from the control parameters, the enhancement layer information including network parameters for configuring an enhancement layer network, and configure the enhancement layer network based on the network parameters; A processing module is used to determine the input features corresponding to the enhancement layer network, and process the input features based on the enhancement layer network to obtain the output features corresponding to the enhancement layer.

9. An image encoding device, characterized in that: The device comprises: An acquisition module, used to acquire control parameters corresponding to the current block, wherein the control parameters include enhancement layer information; wherein the enhancement layer information includes network parameters for configuring an enhancement layer network; The encoding module is used to encode the control parameters corresponding to the current block in the code stream.

10. An image decoding device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method according to any one of claims 1 to 6.

11. An image encoding device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is configured to execute machine executable instructions to implement the method of claim 7.

12. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions that can be executed by a processor; wherein the processor is configured to execute the machine-executable instructions to implement the method of any one of claims 1 to 6, or to implement the method of claim 7.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented; or, when the computer program is executed by a processor, the method according to claim 7 is implemented.

Images