A neural network-based image decoding and encoding method, device, and apparatus

Abstract

The application provides a neural network-based image decoding and encoding method, device and equipment. The method comprises: decoding control parameters and image information corresponding to a current block from a code stream; obtaining neural network information corresponding to a decoding processing unit from the control parameters, and generating a decoding neural network corresponding to the decoding processing unit based on the neural network information; determining input features corresponding to the decoding processing unit based on the image information, and processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit. The application can improve the encoding and decoding performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of coding and decoding, in particular to an image decoding and encoding method based on a neural network, an image decoding and encoding device and equipment thereof. BACKGROUND

[0002] In order to save space, video images are transmitted after being encoded. Complete video encoding can include prediction, transformation, quantization, entropy encoding, filtering and the like. For the prediction process, the prediction process can include intra prediction and inter prediction. Inter prediction refers to using the correlation in the time domain of a video to predict the current pixel using the pixels of the adjacent coded image, so as to effectively remove the temporal redundancy of the video. Intra prediction refers to using the correlation in the spatial domain of a video to predict the current pixel using the pixels of the coded block of the current frame image, so as to remove the spatial redundancy of the video.

[0003] With the rapid development of deep learning, deep learning has achieved success in many high-level computer vision problems such as image classification and target detection. Deep learning has also gradually begun to be applied in the field of coding and decoding, that is, a neural network can be used to encode and decode images. Although the neural network-based coding and decoding method has great performance potential, the neural network-based coding and decoding method still has problems such as poor stability, poor generalization and high complexity. SUMMARY

[0004] Therefore, the present application provides an image decoding and encoding method based on a neural network, an image decoding and encoding device and equipment thereof, which improves the encoding and decoding performance and solves the problems of poor stability, poor generalization and high complexity.

[0005] The present application provides an image decoding method based on a neural network, applied to a decoding end, the method comprising:

[0006] Decoding control parameters and image information corresponding to a current block from a bitstream;

[0007] Obtaining neural network information corresponding to a decoding processing unit from the control parameters, and generating a decoding neural network corresponding to the decoding processing unit based on the neural network information;

[0008] Determining an input feature corresponding to the decoding processing unit based on the image information, processing the input feature based on the decoding neural network, and obtaining an output feature corresponding to the decoding processing unit.

[0009] The present application provides an image encoding method based on a neural network, applied to an encoding end, the method comprising:

[0010] determine an input feature corresponding to the encoding processing unit based on the current block, process the input feature based on an encoding neural network corresponding to the encoding processing unit to obtain an output feature corresponding to the encoding processing unit, and determine the image information corresponding to the current block based on the output feature;

[0011] obtain control parameters corresponding to the current block, the control parameters comprising neural network information corresponding to a decoding processing unit, the neural network information being used to determine a decoding neural network corresponding to the decoding processing unit;

[0012] encode the image information and the control parameters corresponding to the current block in a bitstream.

[0013] The present application provides a neural network-based image decoding apparatus, which comprises:

[0014] a memory configured to store video data;

[0015] a decoder configured to implement:

[0016] decode control parameters and image information corresponding to a current block from a bitstream;

[0017] obtain neural network information corresponding to a decoding processing unit from the control parameters, and generate a decoding neural network corresponding to the decoding processing unit based on the neural network information;

[0018] determine an input feature corresponding to the decoding processing unit based on the image information, and process the input feature based on the decoding neural network to obtain an output feature corresponding to the decoding processing unit.

[0019] The present application provides a neural network-based image encoding apparatus, which comprises:

[0020] a memory configured to store video data;

[0021] an encoder configured to implement:

[0022] determine an input feature corresponding to the encoding processing unit based on the current block, process the input feature based on an encoding neural network corresponding to the encoding processing unit to obtain an output feature corresponding to the encoding processing unit, and determine the image information corresponding to the current block based on the output feature;

[0023] obtain control parameters corresponding to the current block, the control parameters comprising neural network information corresponding to a decoding processing unit, the neural network information being used to determine a decoding neural network corresponding to the decoding processing unit;

[0024] encode the image information and the control parameters corresponding to the current block in a bitstream.

[0025] The application provides a decoding end device, comprising a processor and a machine readable storage medium, wherein the machine readable storage medium stores machine executable instructions capable of being executed by the processor;

[0026] The processor is configured to execute the machine executable instructions to implement the above-mentioned neural network based image decoding method.

[0027] The application provides an encoding end device, comprising a processor and a machine readable storage medium, wherein the machine readable storage medium stores machine executable instructions capable of being executed by the processor;

[0028] The processor is configured to execute the machine executable instructions to implement the above-mentioned neural network based image encoding method.

[0029] As can be seen from the above technical solutions, in the embodiments of the application, the control parameters corresponding to the current block can be decoded from the code stream, the neural network information corresponding to the decoding processing unit can be obtained from the control parameters, and the decoding neural network corresponding to the decoding processing unit can be generated based on the neural network information, and then the image decoding can be implemented based on the decoding neural network, thereby improving the decoding performance. The image encoding can be implemented based on the encoding neural network corresponding to the encoding processing unit, thereby improving the encoding performance. The neural network (such as the decoding neural network and the encoding neural network) can be used to encode and decode the image, and the neural network information can be transmitted through the code stream, and then the decoding neural network corresponding to the decoding processing unit can be generated based on the neural network information, thereby solving the problems of poor stability, poor generalization and high complexity, i.e. good stability, good generalization and low complexity. The scheme of dynamically adjusting the encoding and decoding complexity can be provided, and the encoding performance and the decoding performance are better than those of the single neural network framework. Since the control parameters correspond to each current block, the neural network information obtained from the control parameters is the neural network information for the current block, and the decoding neural network is generated for each current block, i.e. the decoding neural networks of different current blocks can be the same or different, and thus the block level decoding neural network, i.e. the decoding neural network, can be changed and adjusted. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a schematic diagram of a video encoding framework;

[0031] Figures 2A-2C is a schematic diagram of a video encoding framework;

[0032] Figure 3 is a flowchart of a neural network based image decoding method in an embodiment of the application;

[0033] Figure 4is a flowchart of a neural network-based image encoding method in an embodiment of the present application;

[0034] Figure 5A and Figure 5C is a schematic diagram of an image encoding method and an image decoding method in an embodiment of the present application;

[0035] Figure 5B and Figure 5D is a schematic diagram of boundary padding in an embodiment of the present application;

[0036] Figure 5E is a schematic diagram of image domain transformation of an original image in an embodiment of the present application;

[0037] Figure 6A and Figure 6B is a structural schematic diagram of a decoding end in an embodiment of the present application;

[0038] Figure 6C is a structural schematic diagram of a coefficient hyperparameter feature generation unit in an embodiment of the present application;

[0039] Figure 6D is a structural schematic diagram of an image feature inverse transformation unit in an embodiment of the present application;

[0040] Figure 7A 、 Figure 7B and Figure 7C is a structural schematic diagram of an encoding end in an embodiment of the present application;

[0041] Figure 8A is a hardware structural diagram of a decoding end device in an embodiment of the present application;

[0042] Figure 8B is a hardware structural diagram of an encoding end device in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the description of the embodiments of the present application and the claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It also will be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that, although the terms first, second, third, etc. can be used herein to describe various information, but those terms are not intended to denote a temporal or chronological order. Rather, those terms are used only to distinguish one from another. For example, a first information can be termed a second information, and similarly, a second information can also be termed a first information, depending on the context. In addition, the word "if' can be interpreted to mean "when" or "upon" or "in response to determining." The embodiments of the present application can be used with any type of video processing system.

[0044] The embodiments of the present application propose a neural network-based image decoding and encoding method, which can involve the following concepts:

[0045] Neural Network (NN): Neural network refers to artificial neural network, rather than biological neural network. Neural network is an operation model composed of a large number of nodes (or neurons) connected with each other. In neural network, neuron processing units can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. The types of processing units in neural network are divided into three categories: input units, output units and hidden units. Input units accept signals and data from the outside world; output units output the processing results; hidden units are units between input and output units, which cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units, and the representation and processing of information are embodied in the connection relationship of processing units. Neural network is a non-programmed, brain-like information processing method. Its essence is to obtain a parallel distributed information processing function through the transformation and dynamics of neural network, and to imitate the information processing function of the human brain neural system at different levels. In the field of video processing, commonly used neural networks can include but are not limited to convolutional neural network (CNN), recurrent neural network (RNN), fully connected network, etc.

[0046] Convolutional Neural Network (CNN): Convolutional Neural Network is a kind of feedforward neural network, which is one of the most representative network structures in deep learning technology. The artificial neurons of the convolutional neural network can respond to a part of the surrounding units within the coverage range, and have excellent performance for large image processing. The basic structure of the convolutional neural network includes two layers. One is the feature extraction layer (also known as the convolution layer), and the input of each neuron is connected to the local receptive field of the previous layer, and the local features are extracted. Once the local features are extracted, the positional relationship between them is also determined. The second is the feature mapping layer (also known as the activation layer), and each calculation layer of the neural network is composed of multiple feature mappings. Each feature mapping is a plane, and all the weights of the neurons on the plane are equal. The feature mapping structure can use Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, GDN function, etc. as the activation function of the convolutional network. In addition, since the neurons on a mapping plane share weights, the number of free parameters of the network is reduced.

[0047] One of the advantages of the convolutional neural network compared with the image processing algorithm is that it avoids the complex pre-processing process of the image (extracting artificial features, etc.), and can directly input the original image for end-to-end learning. One of the advantages of the convolutional neural network compared with the ordinary neural network is that the ordinary neural network uses full connection, that is, all the neurons from the input layer to the hidden layer are connected, which will result in a large number of parameters, making the network training time-consuming and even difficult to train. The convolutional neural network avoids this difficulty through local connection and weight sharing.

[0048] Deconvolution: Deconvolution is also known as transposed convolution layer. The working process of deconvolution layer is similar to that of convolution layer, and the main difference is that deconvolution layer will pad to make the output larger than the input (of course, it can also remain the same). If the stride is 1, the output size is equal to the input size; if the stride is N, the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0049] Generalization Ability: Generalization ability can refer to the adaptability of machine learning algorithm to new samples. The purpose of learning is to learn the rules hidden in the data pairs. The trained network can also give appropriate output for data outside the learning set with the same rules, and this ability can be called generalization ability.

[0050] Rate-Distortion Optimized: There are two indicators to evaluate the coding efficiency: code rate and PSNR (Peak Signal to Noise Ratio). The smaller the bitstream is, the greater the compression rate is, and the greater the PSNR is, the better the reconstructed image quality is. In mode selection, the decision formula is essentially a comprehensive evaluation of the two. For example, the cost of a mode: J(mode) = D + λ * R, where D represents Distortion, which can usually be measured using the SSE indicator, which is the sum of the squares of the differences between the reconstructed image block and the source image. In order to achieve cost consideration, the SAD indicator can also be used, which is the sum of the absolute values of the differences between the reconstructed image block and the source image. λ is the Lagrange multiplier, and R is the actual number of bits required for encoding the image block under the mode, including the total number of bits required for encoding mode information, motion information, residual error, etc. When selecting a mode, using the rate-distortion principle to compare and decide the encoding mode can usually guarantee the best encoding performance.

[0051] Video encoding framework: see Figure 1 Fig. 1 shows a schematic diagram of a video encoding framework at the encoding end, which can be used to implement the encoding end processing flow of the embodiments of the present application. The schematic diagram of the video decoding framework can be similar to that of the video encoding framework, which will not be repeated here. The video decoding framework can be used to implement the decoding end processing flow of the embodiments of the present application. Figure 1

[0052] Exemplarily, see Figure 1 Fig. 1 shows a video encoding framework that can include prediction, transformation, quantization, entropy encoder, inverse quantization, inverse transformation, reconstruction, filtering, and other modules. At the encoding end, through the cooperation between these modules, the processing flow at the encoding end can be implemented. In addition, the video decoding framework can include prediction, transformation, quantization, entropy decoder, inverse quantization, inverse transformation, reconstruction, filtering, and other modules. At the decoding end, through the cooperation between these modules, the processing flow at the decoding end can be implemented.

[0053] ​Numerous encoding tools have been proposed for various modules at the encoding end, and each tool often has multiple modes. The optimal encoding tool for different video sequences often differs. Therefore, during encoding, Rate-Distortion Optimization (RDO) is typically used to compare the encoding performance of different tools or modes to select the best mode. After determining the optimal tool or mode, the decision information is transmitted by encoding marker information in the bitstream. Although this method introduces higher encoding complexity, it can adaptively select the optimal mode combination for different content to achieve the best encoding performance. The decoding end can obtain the relevant mode information by directly parsing the marker information, with minimal impact from complexity.

[0054] The following is a brief introduction to the structure of the encoder and decoder. (See also...) Figure 2A The diagram illustrates a schematic block diagram of an example of an encoding end used to implement embodiments of this application. Figure 2A In this architecture, the encoding end includes a prediction processing unit, a residual calculation unit, a transform processing unit, a quantization unit, an encoding unit, an inverse quantization unit (also called an inverse quantization unit), an inverse transform processing unit (also called an inverse transform processing unit), a reconstruction unit (or rebuilding unit), and a filter unit. In one example, the encoding end may also include a buffer and a decoded image buffer, wherein the buffer is used to buffer the reconstructed image block output by the reconstruction unit, and the decoded image buffer is used to buffer the filtered image block output by the filter unit.

[0055] The input to the encoding end (also called the encoder) is image blocks of an image (which can be called the image to be encoded). These image blocks can also be called the current block or the block to be encoded. The encoding end may also include a segmentation unit (not shown in the figure), which is used to segment the image to be encoded into multiple image blocks. The encoding end is used to encode block by block to complete the encoding of the image to be encoded; for example, it performs the encoding process for each image block. The prediction processing unit is used to receive or acquire image blocks (the current image block to be encoded in the current image to be encoded, which can also be called the current block, and this image block can be understood as the true value of the image block) and reconstructed image data. Based on the relevant data in the reconstructed image data, it predicts the current block to obtain the predicted block of the current block. In one example, the prediction processing unit may include an inter-frame prediction unit, an intra-frame prediction unit, and a mode selection unit. The mode selection unit is used to select either the intra-frame prediction mode or the inter-frame prediction mode. If the intra-frame prediction mode is selected, the intra-frame prediction unit performs the prediction process; if the inter-frame prediction mode is selected, the inter-frame prediction unit performs the prediction process.

[0056] The residual calculation unit is configured to calculate a residual between the true value of the image block and the prediction block of the image block to obtain a residual block. For example, the residual calculation unit can subtract the pixel value of the prediction block from the pixel value of the image block pixel by pixel.

[0057] The transform processing unit is configured to perform a transform, such as a discrete cosine transform (DCT) or a discrete sine transform (DST), on the residual block to obtain transform coefficients in a transform domain. The transform coefficients can also be referred to as transform residual coefficients, which can represent the residual block in the transform domain.

[0058] The quantization unit is configured to quantize the transform coefficients by applying scalar quantization or vector quantization to obtain quantized transform coefficients, which can also be referred to as quantized residual coefficients. The quantization process can reduce the bit depth associated with some or all of the transform coefficients. For example, an n-bit transform coefficient can be rounded down to an m-bit transform coefficient during quantization, where n is greater than m. The degree of quantization can be modified by adjusting a quantization parameter (QP). For example, for scalar quantization, different scales can be applied to achieve finer or coarser quantization. A smaller quantization step corresponds to finer quantization, while a larger quantization step corresponds to coarser quantization. The appropriate quantization step can be indicated by a quantization parameter (QP).

[0059] The encoding unit is configured to encode the quantized residual coefficients described above to output encoded image data in the form of an encoded bitstream (i.e., the encoding result of the current image block to be encoded), and then transmit the encoded bitstream to a decoder or store it for subsequent transmission to the decoder or retrieval. The encoding unit can also be configured to encode other syntax elements of the current image block, such as encoding the prediction mode into the bitstream, etc. The encoding algorithm includes but is not limited to a variable length coding (VLC) algorithm, a context adaptive VLC (CAVLC) algorithm, an arithmetic coding algorithm, a context adaptive binary arithmetic coding (CABAC) algorithm, a syntax-based context-adaptive binary arithmetic coding (SBAC) algorithm, a probability interval partitioning entropy (PIPE) algorithm.

[0060] The inverse quantization unit is configured to inverse quantize the quantized coefficients to obtain inverse quantized coefficients, which is inverse application of the quantization unit, e.g., inverse quantization scheme of the quantization scheme applied by the quantization unit is applied based on or using the same quantization step size. The inverse quantized coefficients can also be referred to as inverse quantized residual coefficients.

[0061] The inverse transform processing unit is configured to inverse transform the inverse quantized coefficients, which should be understood as inverse application of the transform processing unit, e.g., inverse discrete cosine transform (DCT) or inverse discrete sine transform (DST) can be included in the inverse transform to obtain an inverse transformed block in the pixel domain (or also referred to as sample domain). The inverse transformed block can also be referred to as inverse transformed inverse quantized block or inverse transformed residual block.

[0062] The reconstruction unit is configured to add the inverse transformed block (i.e., inverse transformed residual block) to the prediction block to obtain a reconstructed block in the sample domain, which can be a summer, e.g., adding sample values (i.e., pixel values) of the residual block to sample values of the prediction block. The reconstructed block output by the reconstruction unit can be subsequently used for predicting other image blocks, e.g., in intra prediction mode.

[0063] The filter unit (or simply referred to as filter) is configured to filter the reconstructed block to obtain a filtered block for smooth pixel transitions or improved image quality. The filter unit can be a loop filter unit, which is intended to represent one or more loop filters, e.g., the filter unit can be a deblocking filter, a sample adaptive offset (SAO) filter or other filters, e.g., bilateral filter, adaptive loop filter (ALF), or sharpening or smoothing filter, or collaborative filter. In one example, the filtered block output by the filter unit can be subsequently used for predicting other image blocks, e.g., in inter prediction mode, without limitation.

[0064] Referring to Figure 2B As shown, a schematic block diagram of an example of an encoding end (which can also be referred to as decoder) for implementing embodiments of the present application is shown. The decoder is configured to receive encoded image data (i.e., encoded bitstream, e.g., including encoded bitstream of image blocks and associated syntax elements) encoded by an encoder, e.g., to obtain decoded images. The decoder includes a decoding unit, an inverse quantization unit, an inverse transform processing unit, a prediction processing unit, a reconstruction unit, a filter unit. In some examples, the decoder can perform substantially the same as the encoder, e.g., the decoder can be configured to perform substantially the same as the encoder. Figure 2AThe decoder can further comprise a buffer for buffering the reconstructed image blocks output by the reconstruction unit, and a decoded image buffer for buffering the filtered image blocks output by the filter unit.

[0065] The decoding unit is configured to decode the encoded image data to obtain quantized coefficients and / or decoded coding parameters (e.g., the decoded coding parameters can comprise any one or all of inter prediction parameters, intra prediction parameters, filter parameters and / or other syntax elements). The decoding unit is further configured to forward the decoded coding parameters to the prediction processing unit for the prediction processing unit to perform a prediction process according to the coding parameters. The inverse quantization unit can have the same function as the inverse quantization unit of the encoder, and is configured to inverse quantize (i.e., de-quantize) the quantized coefficients decoded by the decoding unit.

[0066] The inverse transform processing unit can have the same function as the inverse transform processing unit of the encoder, and the reconstruction unit (e.g., a summer) can have the same function as the reconstruction unit of the encoder, and is configured to inverse transform (e.g., inverse DCT, inverse integer transform or conceptually similar inverse transform process) the quantized coefficients to obtain an inverse transform block (also referred to as an inverse transform residual block), which is a residual block of the current image block in the pixel domain.

[0067] The prediction processing unit is configured to receive or obtain the encoded image data (e.g., the encoded bitstream of the current image block) and the reconstructed image data, and can further receive or obtain prediction-related parameters and / or information about the selected prediction mode (i.e., the decoded coding parameters) from, for example, the decoding unit, and predict the current image block based on the relevant data in the reconstructed image data and the decoded coding parameters to obtain a prediction block of the current image block.

[0068] In one example, the prediction processing unit can comprise an inter prediction unit, an intra prediction unit and a mode selection unit, the mode selection unit is configured to select an intra prediction mode or an inter prediction mode, and if the intra prediction mode is selected, the prediction process is performed by the intra prediction unit, and if the inter prediction mode is selected, the prediction process is performed by the inter prediction unit.

[0069] The reconstruction unit is configured to add the inverse transform block (i.e., the inverse transform residual block) to the prediction block to obtain a reconstructed block in the sample domain, for example, by adding the sample values of the inverse transform residual block to the sample values of the prediction block.

[0070] The filter unit is configured to filter the reconstructed block to obtain a filtered block, which is the decoded image block.

[0071] It should be understood that in the encoder and decoder of the embodiments of the present application, the processing result of a certain link can also be output to the next link after further processing, for example, after the links of interpolation filtering, motion vector derivation or filtering, the processing result of the corresponding link is further subjected to operations such as clipping or shifting.

[0072] On the basis of the encoder and the decoder, the embodiments of the present application provide a possible encoding / decoding implementation manner, as shown in Figure 2C Figure 2C A flowchart of encoding and decoding provided by the embodiments of the present application is shown in FIG. 4. The encoding and decoding implementation manner includes processes ① to ⑤, which can be executed by the above-mentioned decoder and encoder. Process ①: divide a frame of image into one or more mutually non-overlapping parallel coding units. The one or more parallel coding units are independent of each other, and can be completely parallel / independent encoded and decoded, as shown in the parallel coding unit 1 and the parallel coding unit 2. Figure 2C

[0073] Process ②: for each parallel coding unit, it can be further divided into one or more mutually non-overlapping independent coding units, each independent coding unit can be independent of each other, but can share some parallel coding unit header information. For example, the width of the independent coding unit is w_lcu, and the height is h_lcu. If the parallel coding unit is divided into one independent coding unit, the size of the independent coding unit is completely the same as that of the parallel coding unit; otherwise, the width of the independent coding unit should be greater than the height (unless it is an edge area).

[0074] Generally, the independent coding unit can be fixed w_lcu×h_lcu, and both w_lcu and h_lcu are 2 raised to the power of N (N≥0), such as the size of the independent coding unit is 128×4, 64×4, 32×4, 16×4, 8×4, 32×2, 16×2 or 8×2, etc.

[0075] ​​As a possible example, the independent coding unit can be fixed at 128x4. If the size of the parallel coding unit is 256x8, the parallel coding unit can be equally divided into 4 independent coding units; if the size of the parallel coding unit is 288x10, the parallel coding unit is divided as follows: the first / second line is divided into 2 128x4+1 32x4 independent coding units; the third line is divided into 2 128x2+1 32x2 independent coding units. It is worth noting that the independent coding unit can include three components of luminance Y, chrominance Cb, and chrominance Cr, or three components of red (R), green (G), and blue (B), or three components of luminance Y, chrominance Co, and chrominance Cg, or only one of the components. If the independent coding unit includes three components, the sizes of the three components can be completely the same or different, which is related to the input format of the image.

[0076] Process ③: For each independent coding unit, it can be further divided into one or more non-overlapping coding units, and each coding unit in the independent coding unit can be dependent on each other, for example, multiple coding units can be pre-decoded by mutual reference.

[0077] If the coding unit has the same size as the independent coding unit (i.e., the independent coding unit is only divided into one coding unit), the size can be all the sizes described in process ②. If the independent coding unit is divided into multiple non-overlapping coding units, possible division examples include: horizontal equal division (the height of the coding unit is the same as that of the independent coding unit, but the width is different, which can be 1 / 2, 1 / 4, 1 / 8, 1 / 16, etc.), vertical equal division (the width of the coding unit is the same as that of the independent coding unit, but the height is different, which can be 1 / 2, 1 / 4, 1 / 8, 1 / 16, etc.), horizontal and vertical equal division (quad-tree division), etc., and the horizontal equal division is preferred.

[0078] The width of the coding unit is w cu and the height is h cu, and the width should be greater than the height (unless it is an edge area). Generally, the coding unit is fixed at w cu x h cu, and both w cu and h cu are 2 raised to the power of N (N is greater than or equal to 0), such as 16x4, 8x4, 16x2, 8x2, 8x1, 4x1, etc. For example, the coding unit is fixed at 16x4. If the size of the independent coding unit is 64x4, the independent coding unit is equally divided into 4 coding units; if the size of the independent coding unit is 72x4, the coding unit is divided as follows: 4 16x4+1 8x4. It is worth noting that the coding unit can include three components of luminance Y, chrominance Cb, and chrominance Cr (or three components of red R, green G, and blue B, or three components of luminance Y, chrominance Co, and chrominance Cg), or only one of the components. If the coding unit includes three components, the sizes of the three components can be completely the same or different, which is related to the input format of the image.

[0079] It is worth noting that process ③ can be an optional step in the coding method, and the encoder / decoder can encode and decode the residual coefficients (or residual values) of the independent coding unit obtained in process ②.

[0080] Process ④: For the coding unit, it can be further divided into one or more non-overlapping prediction groups (PGs), which are also referred to as Groups. Each PG is coded according to a selected prediction mode to obtain a prediction value of the PG, which constitutes a prediction value of the entire coding unit. Based on the prediction value and the original value of the coding unit, a residual value of the coding unit is obtained.

[0081] Process ⑤: Based on the residual value of the coding unit, the coding unit is grouped to obtain one or more non-overlapping residual blocks (RBs). The residual coefficients of each RB are coded according to a selected mode to form a residual coefficient stream. Specifically, there are two types: one is to transform the residual coefficients, and the other is not to transform the residual coefficients.

[0082] The selected mode of the residual coefficient coding method in process ⑤ can include, but is not limited to, any one of the following: semi-definite length coding, exponential Golomb coding method, Golomb-Rice coding method, truncated unary coding method, run-length coding method, and direct coding of original residual values. For example, the encoder can directly code the coefficients in the RB. For another example, the encoder can also transform the residual block, such as DCT, DST, Hadamard transform, and then code the transformed coefficients. As a possible example, when the RB is small, the encoder can directly uniformly quantize each coefficient in the RB and then binarize and code. If the RB is large, it can be further divided into multiple coefficient groups (CGs), and then each CG is uniformly quantized and binarized and coded. In some embodiments of the present application, the coefficient group (CG) and the quantization group (QG) can be the same, and of course, the coefficient group and the quantization group can also be different.

[0083] The following illustrates the part of encoding the residual coefficients in the semi-fixed length coding mode. First, the maximum value of the residual absolute values in one RB block is defined as the modified maximum (mm). Second, the number of coding bits of the residual coefficients in the RB block is determined (the number of coding bits of the residual coefficients in the same RB block is consistent). For example, if the critical limit (CL) of the current RB block is 2 and the current residual coefficient is 1, 2 bits are needed to encode the residual coefficient 1, which is expressed as 01. If the CL of the current RB block is 7, it means that an 8-bit residual coefficient and a 1-bit sign bit are encoded. The determination of the CL is to find the minimum M value that satisfies that all the residuals of the current sub-block are within the range of [-2^(M-1), 2^(M-1)]. If there are two boundary values of -2^(M-1) and 2^(M-1) at the same time, M is increased by 1, that is, M+1 bits are needed to encode all the residuals of the current RB block; if there is only one of the two boundary values of -2^(M-1) and 2^(M-1), a Trailing bit is encoded to determine whether the boundary value is -2^(M-1) or 2^(M-1); if all the residuals do not have any of -2^(M-1) and 2^(M-1), the Trailing bit does not need to be encoded. For some special cases, the encoder can directly encode the original value of the image instead of the residual value.

[0084] With the rapid development of deep learning, neural networks can adaptively construct feature descriptions driven by training data, with higher flexibility and universality, making deep learning successful in many high-level computer vision problems such as image classification and object detection. Deep learning has also gradually begun to be applied in the field of coding and decoding, i.e. using neural networks to encode and decode images. For example, a convolutional neural network VRCNN (Variable Filter Size Convolutional Neural Network) is used to replace the block filtering technology and adaptive sample compensation technology to perform post-processing filtering on the intra-coded image, which greatly improves the subjective and objective quality of the reconstructed image. For another example, neural networks can be applied to intra prediction, and a block-based and down-sampling-based intra prediction mode is proposed. For the intra prediction block, down-sampling coding is performed first, and then the reconstructed pixels are up-sampled through a neural network. For ultra-high-definition sequences, a performance gain of up to 9.0% can be obtained. Obviously, neural networks can effectively get rid of the limitations of artificial mode setting, obtain a neural network that meets the actual needs through data-driven, and significantly improve the coding performance.

[0085] Exemplarily, although the neural network-based coding method exhibits great performance potential, the neural network-based coding method still has problems of poor stability, poor generalization, and high complexity. First, the neural network is still in a process of rapid iteration, and new network structures emerge in an endless stream. Even for some common problems, it is still uncertain which network structure is optimal, not to mention specific problems of some modules of the encoder. Therefore, for a certain module of the encoder, the risk is high to use a single fixed neural network. Second, since the formation of the neural network is highly dependent on the training data, if the training data does not contain a certain feature of the actual problem, the performance will be poor when processing the problem. In the related art, a mode usually uses only one neural network, and when the neural network has insufficient generalization ability, the mode will bring about a reduction in coding performance. Third, since video coding needs to process data with high density and has a low delay requirement, the video coding standard technology has particularly strict requirements on complexity, especially decoding complexity. In order to obtain better coding performance, the parameter quantity of the neural network used for coding is often very large (such as more than 1M), and the multiplication and addition times generated by applying a neural network can reach more than 100K times on average per pixel. Although a neural network with fewer layers or parameters can greatly reduce the coding and decoding complexity, it cannot obtain optimal coding performance. In addition, for image coding schemes, most schemes use an entire frame of image as input (with great cache overhead), and cannot effectively control the output code rate. In actual applications, an image with an arbitrary code rate is often required.

[0086] In view of the above findings, the embodiments of the present application propose a neural network-based image decoding method and an image coding method, which can use a neural network (such as a decoding neural network and a coding neural network) to code and decode an image. In the embodiments, the optimization idea is that, in addition to focusing on coding performance and decoding performance, complexity (especially parallelism of coefficient coding and decoding) and application functionality (supporting variable and fine-tunable code rate, i.e., code rate controllable) also need to be focused on. Based on the above optimization idea, several feasible ways are given in the embodiments of the present application: a) the structure of the neural network is fixed, but the network parameters (i.e., weight parameters) of the neural network are not limited, and the network parameters can be indexed by ID. b) The structure of the neural network is flexible, and relevant structure parameters can be transmitted through syntax coding. The network parameters can be indexed by ID (part of the parameters can be transmitted through syntax coding, which increases a certain code rate cost). c) The structure of the neural network is flexible (high-level syntax configuration), part of the network parameters (such as shallow network) are fixed (to save code rate), and the remaining network parameters are transmitted through coding syntax (to maintain performance optimization space).

[0087] The decoding method and the encoding method in the embodiments of the present application are described in detail below in combination with several specific embodiments.

[0088] Embodiment 1: An image decoding method based on a neural network is proposed in the embodiments of the present application, as shown in Figure 3 The method can be applied to a decoding end (also referred to as a video decoder), and the method can include the following steps:

[0089] Step 301: Decoding control parameters and image information corresponding to a current block from a bitstream.

[0090] Step 302: Obtaining neural network information corresponding to a decoding processing unit from the control parameters, and generating a decoding neural network corresponding to the decoding processing unit based on the neural network information.

[0091] Step 303: Determining input features corresponding to the decoding processing unit based on the image information, and processing the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit.

[0092] In a possible implementation, if the neural network information includes basic layer information and enhancement layer information, the basic layer corresponding to the decoding processing unit can be determined based on the basic layer information, and the enhancement layer corresponding to the decoding processing unit can be determined based on the enhancement layer information; then, the decoding neural network corresponding to the decoding processing unit is generated based on the basic layer and the enhancement layer.

[0093] For example, for the process of determining the basic layer, if the basic layer information includes a basic layer uses default network flag, and the basic layer uses default network flag indicates that the basic layer uses a default network, the basic layer with a default network structure is obtained.

[0094] For example, for the process of determining the basic layer, if the basic layer information includes a basic layer uses pre-made network flag and a basic layer pre-made network index number, and the basic layer uses pre-made network flag indicates that the basic layer uses a pre-made network, the basic layer with a pre-made network structure corresponding to the basic layer pre-made network index number can be selected from a pre-made neural network pool.

[0095] The pre-made neural network pool can include at least one network layer with a pre-made network structure.

[0096] For example, for the process of determining the enhancement layer, if the enhancement layer information includes an enhancement layer uses default network flag, and the enhancement layer uses default network flag indicates that the enhancement layer uses a default network, the enhancement layer with a default network structure is obtained.

[0097] For example, for the process of determining the enhancement layer, if the enhancement layer information includes the enhancement layer using pre-made network flag and the enhancement layer pre-made network index number, and the enhancement layer using pre-made network flag indicates that the enhancement layer uses the pre-made network, the enhancement layer of the pre-made network structure corresponding to the enhancement layer pre-made network index number can be selected from the pre-made neural network pool.

[0098] The pre-made neural network pool includes at least one network layer of a pre-made network structure.

[0099] For example, for the process of determining the enhancement layer, if the enhancement layer information includes network parameters for generating the enhancement layer, the enhancement layer corresponding to the decoding processing unit can be generated based on the network parameters; wherein the network parameters can include but are not limited to at least one of the following: the number of neural network layers, the deconvolution layer flag, the number of deconvolution layers, the quantization step of each deconvolution layer, the number of channels of each deconvolution layer, the size of the convolution kernel, the number of filters, the filter size index, the filter coefficient zero flag, the filter coefficient, the activation layer flag, and the activation layer type. Of course, the above are only a few examples, and this is not limited.

[0100] In a possible implementation, the image information can include coefficient hyperparameter feature information and image feature information, the input feature corresponding to the decoding processing unit is determined based on the image information, and the output feature corresponding to the decoding processing unit is obtained by processing the input feature based on the decoding neural network, which can include but is not limited to: in the decoding process of performing the coefficient hyperparameter feature generation, the coefficient hyperparameter feature coefficient reconstruction value can be determined based on the coefficient hyperparameter feature information; the decoding neural network is used to perform the inverse transform operation on the coefficient hyperparameter feature coefficient reconstruction value to obtain the coefficient hyperparameter feature value; wherein the coefficient hyperparameter feature value can be used to decode the image feature information from the bitstream. In the decoding process of performing the image feature inverse transform, the image feature reconstruction value can be determined based on the image feature information; the decoding neural network is used to perform the inverse transform operation on the image feature reconstruction value to obtain the image low-order feature value; wherein the image low-order feature value can be used to obtain the reconstruction image block corresponding to the current block.

[0101] For example, the coefficient hyperparameter feature coefficient reconstruction value can be determined based on the coefficient hyperparameter feature information, which can include but is not limited to: if the control parameter includes the first enable information, and the first enable information indicates to enable the first inverse quantization operation, the coefficient hyperparameter feature information can be inverse quantized to obtain the coefficient hyperparameter feature coefficient reconstruction value. The image feature reconstruction value can be determined based on the image feature information, which can include but is not limited to: if the control parameter includes the second enable information, and the second enable information indicates to enable the second inverse quantization operation, the image feature information can be inverse quantized to obtain the image feature reconstruction value.

[0102] For example, if the control parameter includes third enabling information, and the third enabling information indicates to enable the quality enhancement operation, in the decoding process of performing quality enhancement, the image low-order feature value can be obtained, and the decoding neural network is used to enhance the image low-order feature value to obtain the reconstructed image block corresponding to the current block.

[0103] In a possible implementation, the decoding end device can include a control parameter decoding unit, a first feature decoding unit, a second feature decoding unit, a coefficient hyperparameter feature generation unit, and an image feature inverse transformation unit. The image information can include coefficient hyperparameter feature information and image feature information. The control parameter decoding unit can decode the control parameter from the code stream, the first feature decoding unit can decode the coefficient hyperparameter feature information from the code stream, and the second feature decoding unit can decode the image feature information from the code stream. When the coefficient hyperparameter feature generation unit is a decoding processing unit, the coefficient hyperparameter feature value can be determined based on the coefficient hyperparameter feature information. The coefficient hyperparameter feature generation unit can perform inverse transformation on the coefficient hyperparameter feature value based on the decoding neural network to obtain the coefficient hyperparameter feature value. The coefficient hyperparameter feature value is used to enable the second feature decoding unit to decode the image feature information from the code stream. When the image feature inverse transformation unit is a decoding processing unit, the image feature reconstruction value is determined based on the image feature information. The image feature inverse transformation unit can perform inverse transformation on the image feature reconstruction value based on the decoding neural network to obtain the image low-order feature value. The image low-order feature value is used to obtain the reconstructed image block corresponding to the current block.

[0104] For example, the decoding end device further includes a first inverse quantization unit and a second inverse quantization unit. The control parameter can include first enabling information of the first inverse quantization unit. If the first enabling information indicates to enable the first inverse quantization unit, the first inverse quantization unit can obtain the coefficient hyperparameter feature information from the first feature decoding unit, and perform inverse quantization on the coefficient hyperparameter feature information to obtain the coefficient hyperparameter feature coefficient reconstruction value, and provide the coefficient hyperparameter feature coefficient reconstruction value to the coefficient hyperparameter feature generation unit. The control parameter can include second enabling information of the second inverse quantization unit. If the second enabling information indicates to enable the second inverse quantization unit, the second inverse quantization unit can obtain the image feature information from the second feature decoding unit, perform inverse quantization on the image feature information to obtain the image feature reconstruction value, and provide the image feature reconstruction value to the image feature inverse transformation unit.

[0105] Exemplarily, the decoding-side device can further include a quality enhancement unit; wherein: the control parameter can include third enabling information of the quality enhancement unit, if the third enabling information indicates to enable the quality enhancement unit, then when the quality enhancement unit is the decoding processing unit, the quality enhancement unit can obtain the image low-order feature value from the image feature inverse transformation unit, and perform enhancement processing on the image low-order feature value based on the decoding neural network to obtain the reconstructed image block corresponding to the current block.

[0106] Exemplarily, the above execution sequence is only an example given for the convenience of description, and in actual application, the execution sequence between steps can also be changed, and the execution sequence is not limited. Moreover, in other embodiments, the steps of the corresponding method can not necessarily be executed in the order shown and described in the specification, and the steps included in the method can be more or less than those described in the specification. In addition, a single step described in the specification can be divided into multiple steps for description in other embodiments, and multiple steps described in the specification can also be combined into a single step for description.

[0107] From the above technical solutions, in the embodiments of the present application, the control parameter corresponding to the current block can be decoded from the code stream, the neural network information corresponding to the decoding processing unit can be obtained from the control parameter, and the decoding neural network corresponding to the decoding processing unit can be generated based on the neural network information, and then the image decoding is realized based on the decoding neural network, thereby improving the decoding performance. The image encoding can be realized based on the encoding neural network corresponding to the encoding processing unit, thereby improving the encoding performance. The deep learning network (such as the decoding neural network and the encoding neural network) can be used to encode and decode the image, and the neural network information is transmitted through the code stream, and then the decoding neural network corresponding to the decoding processing unit is generated based on the neural network information, thereby solving the problems of poor stability, poor generalization and high complexity, i.e. good stability, good generalization and low complexity. The scheme of dynamically adjusting the encoding and decoding complexity can be provided, and the encoding performance is better than that of the single deep learning network framework. Since the control parameter corresponds to each current block, the neural network information obtained from the control parameter is the neural network information for the current block, and the decoding neural network is generated for each current block, i.e. the decoding neural networks of different current blocks can be the same or different, and thus the block-level decoding neural network, i.e. the decoding neural network, can be changed and adjusted.

[0108] Embodiment 2: In the embodiments of the present application, a neural network-based image encoding method is proposed, as shown in FIG. 2, which is a flowchart of the method. The method can be applied to an encoding side (also referred to as a video encoder), and the method can include the following steps. Figure 4

[0109] ​Step 401, determining input features corresponding to the encoding processing unit based on the current block, processing the input features based on the encoding neural network corresponding to the encoding processing unit to obtain output features corresponding to the encoding processing unit, and determining image information corresponding to the current block based on the output features, such as coefficient hyperparameter feature information and image feature information.

[0110] Step 402, obtaining control parameters corresponding to the current block, which can include neural network information corresponding to the decoding processing unit, the neural network information being used to determine the decoding neural network corresponding to the decoding processing unit.

[0111] Step 403, encoding the image information corresponding to the current block and the control parameters in the code stream.

[0112] In a possible implementation, the neural network information can include basic layer information and enhancement layer information, and the decoding neural network includes a basic layer determined based on the basic layer information and an enhancement layer determined based on the enhancement layer information.

[0113] For example, if the basic layer information includes a basic layer uses default network flag, and the basic layer uses default network flag indicates that the basic layer uses a default network, the decoding neural network adopts a basic layer with a default network structure.

[0114] For example, if the basic layer information includes a basic layer uses pre-made network flag and a basic layer pre-made network index number, and the basic layer uses pre-made network flag indicates that the basic layer uses a pre-made network, the decoding neural network can adopt a basic layer with a pre-made network structure corresponding to the basic layer pre-made network index number selected from a pre-made neural network pool.

[0115] The pre-made neural network pool can include at least one network layer with a pre-made network structure.

[0116] For example, if the enhancement layer information includes an enhancement layer uses default network flag, and the enhancement layer uses default network flag indicates that the enhancement layer uses a default network, the decoding neural network adopts an enhancement layer with a default network structure.

[0117] For example, if the enhancement layer information includes an enhancement layer uses pre-made network flag and an enhancement layer pre-made network index number, and the enhancement layer uses pre-made network flag indicates that the enhancement layer uses a pre-made network, the decoding neural network can adopt an enhancement layer with a pre-made network structure corresponding to the enhancement layer pre-made network index number selected from a pre-made neural network pool.

[0118] The pre-made neural network pool includes at least one network layer with a pre-made network structure.

[0119] For example, if the enhancement layer information includes network parameters used to generate the enhancement layer, the decoding neural network can generate the enhancement layer based on the network parameters; wherein the network parameters can include, but are not limited to, at least one of the following: the number of neural network layers, the deconvolution layer flag, the number of deconvolution layers, the quantization step of each deconvolution layer, the number of channels of each deconvolution layer, the size of the convolution kernel, the number of filters, the filter size index, the filter coefficient zero flag, the filter coefficient, the activation layer flag, and the activation layer type. Of course, the above are only a few examples of network parameters, and this is not limited thereto.

[0120] In a possible implementation, the image information can include coefficient hyperparameter feature information and image feature information, the input feature corresponding to the encoding processing unit is determined based on the current block, the input feature is processed based on the encoding neural network corresponding to the encoding processing unit to obtain the output feature corresponding to the encoding processing unit, and the image information corresponding to the current block is determined based on the output feature, which can include but is not limited to: in the encoding process of performing feature transformation, the current block can be subjected to feature transformation based on the encoding neural network to obtain image feature values corresponding to the current block; wherein the image feature values are used to determine the image feature information. In the encoding process of performing coefficient hyperparameter feature transformation, the image feature values can be subjected to coefficient hyperparameter feature transformation based on the encoding neural network to obtain coefficient hyperparameter feature coefficient values, which are used to determine the coefficient hyperparameter feature information.

[0121] For example, in the process of determining the coefficient hyperparameter feature information based on the coefficient hyperparameter feature coefficient values, the coefficient hyperparameter feature coefficient values can be quantized to obtain coefficient hyperparameter feature quantized values, and the coefficient hyperparameter feature information is determined based on the coefficient hyperparameter feature quantized values. Wherein the control parameter can further include first enabling information, and the first enabling information is used to indicate that the first quantization operation has been enabled.

[0122] For example, in the process of determining the image feature information based on the image feature values, the image feature values can be quantized to obtain image feature quantized values, and the image feature information is determined based on the image feature quantized values. Wherein the control parameter can further include second enabling information, and the second enabling information is used to indicate that the second quantization operation has been enabled.

[0123] For example, obtaining the control parameter corresponding to the current block can include, but is not limited to: determining, based on a network structure of an encoding neural network used in the feature transformation-based encoding process, neural network information used in a decoding process of the image feature inverse transformation of the decoding end device, the neural network information being used to determine a decoding neural network corresponding to the decoding process of the image feature inverse transformation of the decoding end device; and / or determining, based on a network structure of an encoding neural network used in the coefficient hyperparameter feature transformation-based encoding process, neural network information used in a decoding process of the coefficient hyperparameter feature generation of the decoding end device, the neural network information being used to determine a decoding neural network corresponding to the decoding process of the coefficient hyperparameter feature generation of the decoding end device.

[0124] In a possible implementation, the encoding end device can include a control parameter encoding unit, a first feature encoding unit, a second feature encoding unit, a feature transformation unit, and a coefficient hyperparameter feature transformation unit; and the image information can include coefficient hyperparameter feature information and image feature information. The control parameter encoding unit encodes the control parameter in the code stream, the first feature encoding unit encodes the coefficient hyperparameter feature information in the code stream, and the second feature encoding unit encodes the image feature information in the code stream. When the feature transformation unit is an encoding processing unit, the feature transformation unit can perform feature transformation on the current block based on an encoding neural network to obtain image feature values corresponding to the current block; the image feature values are used to determine the image feature information. When the coefficient hyperparameter feature transformation unit is an encoding processing unit, the coefficient hyperparameter feature transformation unit can perform coefficient hyperparameter feature transformation on the image feature values based on an encoding neural network to obtain coefficient hyperparameter feature coefficient values, and the coefficient hyperparameter feature coefficient values are used to determine the coefficient hyperparameter feature information.

[0125] For example, the encoding end device can further include a first quantization unit and a second quantization unit. The first quantization unit can obtain the coefficient hyperparameter feature coefficient values from the coefficient hyperparameter feature transformation unit, quantize the coefficient hyperparameter feature coefficient values to obtain coefficient hyperparameter feature quantized values, and determine the coefficient hyperparameter feature information based on the coefficient hyperparameter feature quantized values. The control parameter can further include first enabling information of the first quantization unit, and the first enabling information is used to indicate that the first quantization unit has been enabled. The second quantization unit can obtain the image feature values from the feature transformation unit, quantize the image feature values to obtain image feature quantized values, and determine the image feature information based on the image feature quantized values. The control parameter can further include second enabling information of the second quantization unit, and the second enabling information is used to indicate that the second quantization unit has been enabled.

[0126] For example, obtaining the control parameter corresponding to the current block can include, but is not limited to: determining, based on a network structure of an encoding neural network of a feature transform unit, neural network information corresponding to an image feature inverse transform unit of the decoding device, the neural network information being used to determine a decoding neural network corresponding to the image feature inverse transform unit of the decoding device; and / or determining, based on a network structure of an encoding neural network of a coefficient hyper-parameter feature transform unit, neural network information corresponding to a coefficient hyper-parameter feature generation unit of the decoding device, the neural network information being used to determine a decoding neural network corresponding to the coefficient hyper-parameter feature generation unit of the decoding device.

[0127] In a possible implementation, the image information can include coefficient hyper-parameter feature information and image feature information, and in the decoding process of the coefficient hyper-parameter feature generation, a coefficient hyper-parameter feature coefficient reconstruction value can be determined based on the coefficient hyper-parameter feature information; an inverse transform operation is performed on the coefficient hyper-parameter feature coefficient reconstruction value based on a decoding neural network to obtain a coefficient hyper-parameter feature value; and the coefficient hyper-parameter feature value can be used to decode the image feature information from a bitstream. In the decoding process of the image feature inverse transform, an image feature reconstruction value can be determined based on the image feature information; an inverse transform operation is performed on the image feature reconstruction value based on a decoding neural network to obtain an image low-order feature value; and the image low-order feature value can be used to obtain a reconstructed image block corresponding to the current block.

[0128] For example, determining the coefficient hyper-parameter feature coefficient reconstruction value based on the coefficient hyper-parameter feature information can include, but is not limited to: performing inverse quantization on the coefficient hyper-parameter feature information to obtain the coefficient hyper-parameter feature coefficient reconstruction value. Determining the image feature reconstruction value based on the image feature information can include, but is not limited to: performing inverse quantization on the image feature information to obtain the image feature reconstruction value.

[0129] For example, in the decoding process of the quality enhancement, an image low-order feature value can be obtained, and an enhancement processing is performed on the image low-order feature value based on a decoding neural network to obtain a reconstructed image block corresponding to the current block.

[0130] For example, obtaining the control parameter corresponding to the current block can include, but is not limited to, at least one of the following: determining a network structure of a decoding neural network used in a decoding process of the coefficient hyperparameter feature of the decoding terminal device based on the network structure of the decoding neural network used in the encoding process of the coefficient hyperparameter feature of the encoding terminal device, the network structure being used to determine the decoding neural network used in the decoding process of the coefficient hyperparameter feature of the decoding terminal device; determining a network structure of a decoding neural network used in a decoding process of the image feature inverse transformation of the decoding terminal device based on the network structure of the decoding neural network used in the encoding process of the image feature inverse transformation of the encoding terminal device, the network structure being used to determine the decoding neural network used in the decoding process of the image feature inverse transformation of the decoding terminal device; and determining a network structure of a decoding neural network used in a decoding process of the quality enhancement of the decoding terminal device based on the network structure of the decoding neural network used in the encoding process of the quality enhancement of the encoding terminal device, the network structure being used to determine the decoding neural network used in the decoding process of the quality enhancement of the decoding terminal device.

[0131] In a possible implementation, the encoding terminal device can include a first feature decoding unit, a second feature decoding unit, a coefficient hyperparameter feature generating unit, and an image feature inverse transformation unit; and the image information can include coefficient hyperparameter feature information and image feature information. The first feature decoding unit can decode the coefficient hyperparameter feature information from the bitstream, and the second feature decoding unit can decode the image feature information from the bitstream. After determining the coefficient hyperparameter feature coefficient reconstruction value based on the coefficient hyperparameter feature information, the coefficient hyperparameter feature generating unit can perform an inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value based on the decoding neural network to obtain a coefficient hyperparameter feature value, which is used to enable the second feature decoding unit to decode the image feature information from the bitstream and to enable the second feature encoding unit to encode the image feature information in the bitstream. After determining the image feature reconstruction value based on the image feature information, the image feature inverse transformation unit can perform an inverse transformation operation on the image feature reconstruction value based on the decoding neural network to obtain an image low-order feature value, which is used to obtain the reconstruction image block corresponding to the current block.

[0132] For example, the encoding terminal device can further include a first inverse quantization unit and a second inverse quantization unit. The first inverse quantization unit can obtain the coefficient hyperparameter feature information from the first feature decoding unit and perform inverse quantization on the coefficient hyperparameter feature information to obtain the coefficient hyperparameter feature coefficient reconstruction value. The second inverse quantization unit can obtain the image feature information from the second feature decoding unit and perform inverse quantization on the image feature information to obtain the image feature reconstruction value.

[0133] The encoding end device further includes a quality enhancement unit. The quality enhancement unit obtains the low-order feature value of the image from the image feature inverse transformation unit, and performs enhancement processing on the low-order feature value of the image based on the decoding neural network to obtain the reconstructed image block.

[0134] The control parameter corresponding to the current block can be obtained in the following manner, for example, but not limited to: based on the network structure of the decoding neural network of the coefficient hyperparameter feature generation unit of the encoding end device, the neural network information corresponding to the coefficient hyperparameter feature generation unit of the decoding end device is determined, and the neural network information is used to determine the decoding neural network corresponding to the coefficient hyperparameter feature generation unit of the decoding end device; based on the network structure of the decoding neural network of the image feature inverse transformation unit of the encoding end device, the neural network information corresponding to the image feature inverse transformation unit of the decoding end device is determined, and the neural network information is used to determine the decoding neural network corresponding to the image feature inverse transformation unit of the decoding end device; based on the network structure of the decoding neural network of the quality enhancement unit of the encoding end device, the neural network information corresponding to the quality enhancement unit of the decoding end device is determined, and the neural network information is used to determine the decoding neural network corresponding to the quality enhancement unit of the decoding end device.

[0135] In a possible implementation, before the input feature corresponding to the encoding processing unit is determined based on the current block, the current image can be divided into N non-overlapping image blocks, N being a positive integer; each image block can be subjected to boundary padding to obtain a boundary-padded image block; wherein when each image block is subjected to boundary padding, the padding value can not depend on the reconstructed pixel values of adjacent image blocks; and the N current blocks can be generated based on the boundary-padded image blocks.

[0136] In a possible implementation, before the input feature corresponding to the encoding processing unit is determined based on the current block, the current image can be divided into a plurality of basic blocks, and each basic block includes at least one image block; each image block can be subjected to boundary padding to obtain a boundary-padded image block; wherein when each image block is subjected to boundary padding, the padding value of the image block does not depend on the reconstructed pixel values of other image blocks in the same basic block, and is allowed to depend on the reconstructed pixel values of image blocks in different basic blocks; and the plurality of current blocks can be generated based on the boundary-padded image blocks.

[0137] For example, the execution sequence described above is only an example given for the convenience of description, and in actual application, the execution sequence between steps can also be changed, and the execution sequence is not limited. Moreover, in other embodiments, the steps of the corresponding method can not necessarily be executed in the order shown and described in the specification, and the steps included in the method can be more or less than those described in the specification. In addition, a single step described in the specification can be divided into multiple steps for description in other embodiments; multiple steps described in the specification can also be combined into a single step for description in other embodiments.

[0138] From the above technical solutions, in the embodiments of the present application, the control parameters corresponding to the current block can be decoded from the code stream, the neural network information corresponding to the decoding processing unit can be obtained from the control parameters, and the decoding neural network corresponding to the decoding processing unit can be generated based on the neural network information, and then the image decoding is realized based on the decoding neural network, thereby improving the decoding performance. The image encoding can be realized based on the encoding neural network corresponding to the encoding processing unit, thereby improving the encoding performance. The deep learning network (such as the decoding neural network and the encoding neural network) can be used to encode and decode the image, and the neural network information is transmitted through the code stream, and then the decoding neural network corresponding to the decoding processing unit is generated based on the neural network information, thereby solving the problems of poor stability, poor generalization, and high complexity, i.e., good stability, good generalization, and low complexity. The scheme of dynamically adjusting the encoding and decoding complexity can be provided, and the encoding performance is better than that of the single deep learning network framework. Since the control parameters correspond to each current block, the neural network information obtained from the control parameters is the neural network information for the current block, and the decoding neural network is generated for each current block, i.e., the decoding neural networks of different current blocks can be the same or different, and thus the block-level decoding neural network, i.e., the decoding neural network, can be changed and adjusted.

[0139] Embodiment 3: In the embodiments of the present application, a neural network-based image encoding method and an image decoding method with variable and adjustable code rate are proposed, which can realize high parallelism of image blocks and controllable and adjustable code rate. Referring to FIG. 3, an image encoding method and an image decoding method are shown, and the encoding process of the image encoding method and the decoding process of the image decoding method are shown. Figure 5A

[0140] For example, for the neural network-based image encoding method, the encoding process can include the following steps:

[0141] Step S11, the block division unit divides the current image (i.e., the original image) into N mutually non-overlapping image blocks (i.e., the original image blocks, which can be denoted as original image block 1, original image block 2, …, original image block N), and N is a positive integer.

[0142] ​Step S12, boundary padding is performed on each original image block to obtain a boundary padded image block, and N current blocks are generated based on the boundary padded image block, that is, boundary padding is performed on the N original image blocks respectively to obtain N boundary padded image blocks, and the N boundary padded image blocks are taken as the N current blocks.

[0143] For example, referring to FIG. 1, when boundary padding is performed on each original image block, the padding value can not depend on the reconstructed pixel value of the adjacent image block, so that each original image block can be independently and parallelly encoded, and the encoding performance is improved. Figure 5B

[0144] Step S13, the encoding unit determines the encoding parameter of the current block based on the information of the encoded block, and the encoding parameter is used to control the encoding code rate size (such as the quantization step length and the like) of the current block, and the encoding parameter is not limited.

[0145] Step S14, the control unit writes the control parameter required by the decoding end and unable to be derived into the code stream.

[0146] Step S15, the padded image block (i.e. the current block) is input into the neural network-based encoding unit, the current block is encoded by the encoding unit based on the encoding parameter, and the code stream of the current block is output. For example, when the encoding unit encodes the current block based on the encoding parameter, the encoding unit can encode the current block by using the neural network.

[0147] For example, for the neural network-based image decoding method, the decoding process can include the following steps:

[0148] Step S21, the decoding end decodes the control parameter required by the current block and unable to be derived from the code stream.

[0149] Step S22, based on the control parameter and the code stream of the current block, the current block corresponding to the reconstructed image block is obtained through the neural network-based decoding unit, that is, the current block is decoded to obtain the reconstructed image block, such as the reconstructed image block 1 corresponding to the original image block 1, the reconstructed image block 2 corresponding to the original image block 2, …, and the reconstructed image block N corresponding to the original image block N.

[0150] For example, when the decoding unit decodes the current block, the neural network can be used to decode the current block.

[0151] Step S23, based on the control parameter, it is determined whether to perform a filtering process on a certain current block, if yes, the filtering and merging unit performs a filtering process based on the information of the current block and at least one adjacent reconstructed image block to obtain a filtered image block.

[0152] Step S24, the reconstructed image is obtained by merging the filtered image block.​

[0153] In a possible implementation, when performing the boundary padding on the original image block, the padding value can be a padding preset value, which can be a default value agreed by coding (such as 0 or 1 << (1-depth), where depth is a bit depth, such as 8, 10, 12, etc.), can be a value delivered to the decoding end through high-level syntax coding, or can be obtained through mirroring, nearest neighbor copying or other operations based on the pixels of the current block. The manner of obtaining the padding value is not limited.

[0154] In a possible implementation, when performing the boundary padding on the original image block, the size of the padding extension of the surrounding block can be a default value agreed by coding (such as 1, 2, 4, etc.), can be a value related to the size of the current block, or can be a value delivered to the decoding end through high-level syntax coding. The size of the padding extension of the surrounding block is not limited.

[0155] Embodiment 4: In the embodiment of the present application, a code rate variable adjustable neural network-based image encoding method and image decoding method are proposed, which can realize high parallelism of image blocks and code rate controllable adjustment. On this basis, information (such as reconstructed pixels of adjacent blocks) between adjacent blocks can also be used, as shown in Figure 5C The figure shows the encoding process of the image encoding method and the decoding process of the image decoding method.

[0156] Exemplarily, the encoding process of the neural network-based image encoding method can include the following steps:

[0157] Step S31: The blocking unit divides a current image (that is, an original image) into a plurality of basic blocks, each basic block including at least one image block. Taking M basic blocks as an example, M is a positive integer, and the M basic blocks together include N mutually non-overlapping image blocks (that is, original image blocks, denoted as original image block 1, original image block 2, …, original image block N), N is a positive integer.

[0158] Step S32: Each original image block is subjected to boundary padding to obtain a boundary-padded image block, and N current blocks are generated based on the boundary-padded image block. That is, N original image blocks are subjected to boundary padding respectively to obtain N boundary-padded image blocks, and the N boundary-padded image blocks are taken as the N current blocks.

[0159] Exemplarily, when each original image block is subjected to boundary padding, the padding value of the original image block does not depend on the reconstructed pixel values of other original image blocks in the same basic block, and is allowed to depend on the reconstructed pixel values of original image blocks in different basic blocks.

[0160] In the embodiment, the concept of basic block is introduced, as shown inFigure 5D As shown, each basic block includes at least one image block (i.e. original image block), and each image block in a basic block does not reference each other, but can reference the reconstructed information of other image blocks in different basic blocks. For example, referring to FIG. 2, for image block 1, the image block on the left side of image block 1 (i.e. the neighboring block on the left side of image block 1) is in the same basic block as image block 1, and thus the reconstructed information of the image block on the left side of image block 1 cannot be used as the padding value of image block 1, but the image block is padded with a preset padding value. For image block 1, the image block on the upper side of image block 1 (i.e. the neighboring block on the upper side of image block 1) is in a different basic block from image block 1, and thus the reconstructed information of the image block on the upper side of image block 1 can be used as the padding value of image block 1. When the reconstructed information of the image block on the upper side of image block 1 is used as the padding value of image block 1, the reconstructed value of the image block on the upper side of image block 1 can be used, and the reconstructed value of the image block on the upper side of image block 1 before filtering can be used. Figure 5D As shown, each basic block includes at least one image block (i.e. original image block), and each image block in a basic block does not reference each other, but can reference the reconstructed information of other image blocks in different basic blocks. For example, referring to FIG. 2, for image block 1, the image block on the left side of image block 1 (i.e. the neighboring block on the left side of image block 1) is in the same basic block as image block 1, and thus the reconstructed information of the image block on the left side of image block 1 cannot be used as the padding value of image block 1, but the image block is padded with a preset padding value. For image block 1, the image block on the upper side of image block 1 (i.e. the neighboring block on the upper side of image block 1) is in a different basic block from image block 1, and thus the reconstructed information of the image block on the upper side of image block 1 can be used as the padding value of image block 1. When the reconstructed information of the image block on the upper side of image block 1 is used as the padding value of image block 1, the reconstructed value of the image block on the upper side of image block 1 can be used, and the reconstructed value of the image block on the upper side of image block 1 before filtering can be used.

[0161] Obviously, by introducing the basic block, the parallelism (each image block in a basic block can be coded and decoded in parallel) can be ensured, and the performance can be improved (the reconstructed information of the image block in the neighboring basic block can be used).

[0162] In step S33, the encoding unit determines the encoding parameter of the current block based on the information of the encoded block, and the encoding parameter is used to control the size of the encoding code rate of the current block (such as the quantization step length and the like), and the encoding parameter is not limited.

[0163] In step S34, the control unit writes the control parameter required by the decoding end and unable to be derived into the code stream.

[0164] In step S35, the padded image block (i.e. the current block) is input into the neural network-based encoding unit, and the encoding unit encodes the current block based on the encoding parameter, and outputs the code stream of the current block. For example, when the encoding unit encodes the current block based on the encoding parameter, the encoding unit can encode the current block by using the neural network.

[0165] For example, the decoding process of the neural network-based image decoding method can include the following steps:

[0166] In step S41, the decoding end decodes the control parameter required by the current block and unable to be derived from the code stream.

[0167] In step S42, based on the control parameter and the code stream of the current block, the reconstructed image block corresponding to the current block is obtained by using the neural network-based decoding unit, i.e. the current block is decoded to obtain the reconstructed image block, such as the reconstructed image block 1 corresponding to the original image block 1, the reconstructed image block 2 corresponding to the original image block 2, and the reconstructed image block N corresponding to the original image block N.

[0168] Step S43, based on the control parameter, determine whether to perform filtering process on the current block, if yes, the filtering and merging unit performs filtering process based on the information of the current block and at least one neighboring reconstructed image block, to obtain a filtered image block.

[0169] Step S44, merge the filtered image block to obtain a reconstructed image.

[0170] In a possible implementation, when performing boundary padding on the original image block, the padding value can be a padding preset value, which can be a default value agreed by the codec (such as 0, or 1 << (1-depth), where depth is bit depth, such as 8, 10, 12, etc.), can also be a value transmitted to the decoding end through high-level syntax coding, and can also be obtained by mirror, nearest neighbor copy, etc. based on the pixels of the current block. The manner of obtaining the padding value is not limited.

[0171] In a possible implementation, when performing boundary padding on the original image block, the size of the surrounding block padding extension can be a default value agreed by the codec (such as 1, 2, 4, etc.), can also be a value related to the size of the current block, and can also be a value transmitted to the decoding end through high-level syntax coding. The size of the surrounding block padding extension is not limited.

[0172] In a possible implementation, the number of image blocks included in the basic block can be a default value agreed by the codec (such as 1, 4, 16, etc.), can also be a value related to the size of the current image, and can also be a value transmitted to the decoding end through high-level syntax coding. The number of image blocks in the basic block is not limited and can be selected according to actual needs.

[0173] In a possible implementation, in Embodiment 3 and Embodiment 4, the blocking process (step S11 and step S31) of the original image can also be image domain transformation on the original image, and the filtered image blocks can be merged into an image, and the image can be image domain inverse transformation, and the reconstructed image can be obtained based on the image after the image domain inverse transformation. See Figure 5E For example, the image domain transformation on the original image can be a transformation process from an RGB domain image to a YUV domain image (the corresponding image domain inverse transformation is a transformation process from a YUV domain image to an RGB domain image), or can be a process of introducing wavelet transformation or Fourier transformation to generate a new domain image (the corresponding image domain inverse transformation is an inverse wavelet transformation or inverse Fourier transformation process). The image domain transformation process can be implemented by a neural network or a non-neural network. The image domain transformation process is not limited.

[0174] Example 5: This application proposes a neural network-based image decoding method, which can be applied to the decoding end (also known as a video decoder). See [link to example]. Figure 6A The diagram shows the structure of the decoding end. The decoding end may include a control parameter decoding unit, a first feature decoding unit, a second feature decoding unit, a coefficient hyperparameter feature generation unit, an image feature inverse transform unit, a first inverse quantization unit, a second inverse quantization unit, and a quality enhancement unit. The first inverse quantization unit, the second inverse quantization unit, and the quality enhancement unit are optional units; in certain scenarios, the process of these optional units can be turned off or skipped.

[0175] In this embodiment, for each current block (i.e., image block), the corresponding bitstream includes three parts: bitstream 0 (bitstream containing control parameters), bitstream 1 (bitstream containing coefficient hyperparameter feature information), and bitstream 2 (bitstream containing image feature information). The coefficient hyperparameter feature information and the image feature information can be collectively referred to as image information.

[0176] For example, the neural network-based image decoding method in this embodiment may include the following steps:

[0177] Step S51: Decode the bitstream 0 corresponding to the current block to obtain the control parameters corresponding to the current block. For example, the control parameter decoding unit can decode the bitstream 0 corresponding to the current block to obtain the control parameters corresponding to the current block. That is, the control parameter decoding unit can decode the control parameters from the bitstream 0. The control parameters may include the control parameters of the first inverse quantization unit, the control parameters of the second inverse quantization unit, the control parameters of the coefficient hyperparameter feature generation unit, the control parameters of the image feature inverse transform unit, and the control parameters of the quality enhancement unit. For details on the content of the control parameters, please refer to the subsequent embodiments, which will not be repeated here.

[0178] Step S52: Decode the bitstream 1 corresponding to the current block to obtain the coefficient hyperparameter feature information corresponding to the current block. For example, the first feature decoding unit can decode the bitstream 1 corresponding to the current block to obtain the coefficient hyperparameter feature information corresponding to the current block, that is, the first feature decoding unit can decode the coefficient hyperparameter feature information from the bitstream 1.

[0179] Step S53: Determine the reconstructed value of the coefficient hyperparameter feature coefficient based on the coefficient hyperparameter feature information.

[0180] For example, if the control parameter includes first enable information corresponding to the first dequantization unit, the first enable information can indicate that the first dequantization unit is enabled (i.e., that the first dequantization unit is enabled to perform the first dequantization operation), or the first enable information can indicate that the first dequantization unit is disabled. For instance, if the first enable information is a first value, it indicates that the first dequantization unit is enabled; if the first enable information is a second value, it indicates that the first dequantization unit is disabled.

[0181] For example, if the first enabling information indicates enabling the first inverse quantization unit, the coefficient hyper-parameter characteristic information can be a coefficient hyper-parameter characteristic quantization value C_q, and the first inverse quantization unit can perform inverse quantization on the coefficient hyper-parameter characteristic quantization value C_q to obtain a coefficient hyper-parameter characteristic coefficient reconstructed value C'. If the first enabling information indicates not enabling the first inverse quantization unit, the coefficient hyper-parameter characteristic information can be the coefficient hyper-parameter characteristic coefficient reconstructed value C', that is, the coefficient hyper-parameter characteristic coefficient reconstructed value C' is directly decoded from the code stream 1.

[0182] In step S54, inverse transform operation is performed on the coefficient hyper-parameter characteristic coefficient reconstructed value C' to obtain a coefficient hyper-parameter characteristic value P. For example, the coefficient hyper-parameter characteristic generation unit performs inverse transform operation on the coefficient hyper-parameter characteristic coefficient reconstructed value C' to obtain the coefficient hyper-parameter characteristic value P, such as performing inverse transform operation on the coefficient hyper-parameter characteristic coefficient reconstructed value C' based on the decoding neural network of the coefficient hyper-parameter characteristic generation unit to obtain the coefficient hyper-parameter characteristic value P.

[0183] In a possible implementation, the coefficient hyper-parameter characteristic generation unit can obtain, from the control parameter, neural network information 1 corresponding to the coefficient hyper-parameter characteristic generation unit, and generate a decoding neural network 1 corresponding to the coefficient hyper-parameter characteristic generation unit based on the neural network information 1.

[0184] In addition, the coefficient hyper-parameter characteristic generation unit can determine an input feature (such as the coefficient hyper-parameter characteristic coefficient reconstructed value C') corresponding to the coefficient hyper-parameter characteristic generation unit, and process the input feature (such as performing inverse transform operation on the coefficient hyper-parameter characteristic coefficient reconstructed value C') based on the decoding neural network 1 to obtain an output feature (such as the coefficient hyper-parameter characteristic value P) corresponding to the coefficient hyper-parameter characteristic generation unit.

[0185] For example, the neural network information 1 can include basic layer information and enhancement layer information, and the coefficient hyper-parameter characteristic generation unit can determine a basic layer corresponding to the coefficient hyper-parameter characteristic generation unit based on the basic layer information, and determine an enhancement layer corresponding to the coefficient hyper-parameter characteristic generation unit based on the enhancement layer information. The coefficient hyper-parameter characteristic generation unit can generate the decoding neural network 1 corresponding to the coefficient hyper-parameter characteristic generation unit based on the basic layer and the enhancement layer, such as combining the basic layer and the enhancement layer to obtain the decoding neural network 1.

[0186] After obtaining the decoding neural network 1, the coefficient hyper-parameter characteristic generation unit can perform inverse transform operation on the coefficient hyper-parameter characteristic coefficient reconstructed value C' through the decoding neural network 1 to obtain the coefficient hyper-parameter characteristic value P, and the inverse transform operation process is not limited.

[0187] Step S55, decoding the code stream 2 corresponding to the current block to obtain the image feature information corresponding to the current block. For example, the second feature decoding unit can decode the code stream 2 corresponding to the current block to obtain the image feature information corresponding to the current block, that is, the second feature decoding unit can decode the image feature information from the code stream 2. When decoding the code stream 2 corresponding to the current block, the second feature decoding unit can decode the code stream 2 corresponding to the current block by using the coefficient hyperparameter feature value P, and the decoding process is not limited.

[0188] Step S56, determining the image feature reconstruction value based on the image feature information.

[0189] For example, if the control parameter includes the second enabling information corresponding to the second dequantization unit, the second enabling information can indicate enabling the second dequantization unit (that is, enabling the second dequantization unit to perform the second dequantization operation), and the second enabling information can also indicate not enabling the second dequantization unit. For example, if the second enabling information is a first value, it indicates enabling the second dequantization unit, and if the second enabling information is a second value, it indicates not enabling the second dequantization unit.

[0190] For example, if the second enabling information indicates enabling the second dequantization unit, the image feature information can be the image feature quantization value F_q, the second dequantization unit can obtain the image feature quantization value F_q and dequantize the image feature quantization value F_q to obtain the image feature reconstruction value F'. If the second enabling information indicates not enabling the second dequantization unit, the image feature information can be the image feature reconstruction value F', that is, the image feature reconstruction value F' is directly decoded from the code stream 2.

[0191] Step S57, performing inverse transform operation on the image feature reconstruction value F' to obtain the image low-order feature value LF. For example, the image feature inverse transform unit performs inverse transform operation on the image feature reconstruction value F' to obtain the image low-order feature value LF, such as performing inverse transform operation on the image feature reconstruction value F' based on the decoding neural network to obtain the image low-order feature value LF.

[0192] In a possible implementation, the image feature inverse transform unit can obtain the neural network information 2 corresponding to the image feature inverse transform unit from the control parameter, and generate the decoding neural network 2 corresponding to the image feature inverse transform unit based on the neural network information 2. The image feature inverse transform unit can determine the input feature (such as the image feature reconstruction value F') corresponding to the image feature inverse transform unit, and process the input feature (such as performing inverse transform operation on the image feature reconstruction value F') based on the decoding neural network 2 to obtain the output feature (such as the image low-order feature value LF) corresponding to the image feature inverse transform unit.

[0193] Exemplarily, the neural network information 2 can include base layer information and enhancement layer information, the image feature inverse transformation unit can determine the base layer corresponding to the image feature inverse transformation unit based on the base layer information, and determine the enhancement layer corresponding to the image feature inverse transformation unit based on the enhancement layer information. The image feature inverse transformation unit can generate the decoding neural network 2 corresponding to the image feature inverse transformation unit based on the base layer and the enhancement layer, for example, the base layer and the enhancement layer can be combined to obtain the decoding neural network 2.

[0194] After obtaining the decoding neural network 2, the image feature inverse transformation unit can perform inverse transformation operation on the image feature reconstruction value F’ through the decoding neural network 2 to obtain the image low-order feature value LF, and the inverse transformation operation process is not limited.

[0195] In step S58, the reconstructed image block I corresponding to the current block is determined based on the image low-order feature value LF.

[0196] Exemplarily, if the control parameter includes third enabling information corresponding to the quality enhancement unit, the third enabling information can indicate enabling the quality enhancement unit (i.e., enabling the quality enhancement unit to perform quality enhancement operation), and the third enabling information can also indicate not enabling the quality enhancement unit. For example, if the third enabling information is a first value, it can indicate enabling the quality enhancement unit, and if the third enabling information is a second value, it can indicate not enabling the quality enhancement unit.

[0197] Exemplarily, if the third enabling information indicates enabling the quality enhancement unit, the quality enhancement unit obtains the image low-order feature value LF, and performs enhancement processing on the image low-order feature value LF to obtain the reconstructed image block I corresponding to the current block. If the third enabling information indicates not enabling the quality enhancement unit, the image low-order feature value LF is taken as the reconstructed image block I corresponding to the current block.

[0198] Exemplarily, when the quality enhancement unit performs enhancement processing on the image low-order feature value LF, the quality enhancement unit can perform enhancement processing on the image low-order feature value LF based on the decoding neural network to obtain the reconstructed image block I corresponding to the current block.

[0199] In a possible implementation, the quality enhancement unit can obtain neural network information 3 corresponding to the quality enhancement unit from the control parameter, and generate a decoding neural network 3 corresponding to the quality enhancement unit based on the neural network information 3.

[0200] The quality enhancement unit can determine the input feature (such as the image low-order feature value LF) corresponding to the quality enhancement unit, and perform processing (such as enhancement processing on the image low-order feature value LF) on the input feature based on the decoding neural network 3 to obtain the output feature (such as the reconstructed image block I corresponding to the current block) corresponding to the quality enhancement unit.

[0201] For example, the neural network information 3 can include base layer information and enhancement layer information, and the quality enhancement unit can determine the base layer corresponding to the quality enhancement unit based on the base layer information, and determine the enhancement layer corresponding to the quality enhancement unit based on the enhancement layer information. The quality enhancement unit can generate the decoding neural network 3 corresponding to the quality enhancement unit based on the base layer and the enhancement layer, for example, the base layer and the enhancement layer can be combined to obtain the decoding neural network 3.

[0202] After obtaining the decoding neural network 3, the quality enhancement unit can perform enhancement processing on the image low-order feature value LF through the decoding neural network 3 to obtain the reconstructed image block I corresponding to the current block, and the enhancement processing process is not limited.

[0203] Embodiment 6: In the embodiments of the present application, a neural network-based image decoding method is proposed, which can be applied to a decoding end. Referring to FIG. 6, it is a structural schematic diagram of the decoding end, which can include a control parameter decoding unit, a first feature decoding unit, a second feature decoding unit, a coefficient hyperparameter feature generation unit, and an image feature inverse transformation unit. In the embodiments, for each current block (i.e. image block), the code stream corresponding to the current block includes three parts: code stream 0 (code stream containing control parameters), code stream 1 (code stream containing coefficient hyperparameter feature information), and code stream 2 (code stream containing image feature information). Figure 6B

[0204] For example, the neural network-based image decoding method in the embodiments can include the following steps:

[0205] Step S61: The control parameter decoding unit decodes the code stream 0 corresponding to the current block to obtain the control parameter corresponding to the current block.

[0206] Step S62: The first feature decoding unit decodes the code stream 1 corresponding to the current block to obtain the coefficient hyperparameter feature information corresponding to the current block, which can be the coefficient hyperparameter feature coefficient reconstruction value C’.

[0207] Step S63: The coefficient hyperparameter feature generation unit performs inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value C’ to obtain the coefficient hyperparameter feature value P, for example, performs inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value C’ based on the decoding neural network to obtain the coefficient hyperparameter feature value P.

[0208] Step S64: The second feature decoding unit decodes the code stream 2 corresponding to the current block to obtain the image feature information corresponding to the current block, which can be the image feature reconstruction value F’. For example, the second feature decoding unit can obtain the coefficient hyperparameter feature value P, and decode the code stream 2 corresponding to the current block using the coefficient hyperparameter feature value P to obtain the image feature reconstruction value F’. ​

[0209] Step S65, the image feature inverse transformation unit performs an inverse transformation operation on the image feature reconstruction value F’ to obtain the image low-order feature value LF, such as performing an inverse transformation operation on the image feature reconstruction value F’ based on the decoding neural network to obtain the image low-order feature value LF.

[0210] Step S66, the image low-order feature value LF is used to determine the reconstruction image block I corresponding to the current block.

[0211] For example, the image low-order feature value LF can be directly used as the reconstruction image block I. Alternatively, if the decoding end device further includes a quality enhancement unit, the image low-order feature value LF is enhanced by the quality enhancement unit to obtain the reconstruction image block I corresponding to the current block, such as enhancing the image low-order feature value LF based on the decoding neural network to obtain the reconstruction image block I.

[0212] Embodiment 7: For embodiments 5 and 6, the first feature decoding unit can decode the code stream 1 corresponding to the current block to obtain the coefficient hyperparameter feature information corresponding to the current block. For example, the first feature decoding unit includes at least one coefficient decoding module. In a possible implementation, the coefficient decoding module can perform coefficient decoding by using an entropy decoding method, that is, the code stream 1 corresponding to the current block is decoded by using an entropy decoding method to obtain the coefficient hyperparameter feature information corresponding to the current block.

[0213] For example, the entropy decoding method can include, but is not limited to, CAVLC (Context-Adaptive Variable Length Coding) or CABAC (Context-based Adaptive Binary Arithmetic Coding), and the like, without limitation.

[0214] For example, when the coefficient decoding is performed by using the entropy decoding method, the probability model of the entropy decoding can use a preset probability model, and the preset probability model can be configured according to actual needs, without limitation. For example, based on the preset probability model, the coefficient decoding module can perform coefficient decoding by using the entropy decoding method.

[0215] Embodiment 8: For embodiment 5, the first inverse quantization unit can perform inverse quantization on the coefficient hyperparameter feature quantization value C_q (i.e., the coefficient hyperparameter feature information) to obtain the coefficient hyperparameter feature coefficient reconstruction value C’. For example, the first inverse quantization unit can not exist, or if the first inverse quantization unit exists, the first inverse quantization unit can be selectively skipped based on a control parameter (such as a high-level syntax, such as first enabling information, etc.), or the first inverse quantization unit can be enabled based on a control parameter.

[0216] For example, if the first dequantization unit does not exist, the coefficient hyper-parameter characteristic coefficient reconstructed value C' is the same as the coefficient hyper-parameter characteristic coefficient quantized value C_q, i.e., the coefficient hyper-parameter characteristic coefficient quantized value C_q does not need to be dequantized. If the first dequantization unit is selectively skipped based on the control parameter, the coefficient hyper-parameter characteristic coefficient reconstructed value C' is the same as the coefficient hyper-parameter characteristic coefficient quantized value C_q, i.e., the coefficient hyper-parameter characteristic coefficient quantized value C_q does not need to be dequantized. If the first dequantization unit is enabled based on the control parameter, but the step parameter qstep corresponding to the coefficient hyper-parameter characteristic coefficient quantized value C_q is 1, the coefficient hyper-parameter characteristic coefficient reconstructed value C' is the same as the coefficient hyper-parameter characteristic coefficient quantized value C_q, i.e., the coefficient hyper-parameter characteristic coefficient quantized value C_q does not need to be dequantized.

[0217] For example, if the first dequantization unit is enabled based on the control parameter, and the step parameter qstep corresponding to the coefficient hyper-parameter characteristic coefficient quantized value C_q is not 1, the first dequantization unit can dequantize the coefficient hyper-parameter characteristic coefficient quantized value C_q based on the control parameter (such as quantization-related parameters) to obtain the coefficient hyper-parameter characteristic coefficient reconstructed value C'. For example, the first dequantization unit performs the following operations: obtains the quantization-related parameters (including the control parameter in the code stream, and the control parameter can include the quantization-related parameters) corresponding to the coefficient hyper-parameter characteristic coefficient quantized value C_q, such as the step parameter qstep or the quantization parameter qp, from the control parameter. The multiplication factor mult and the shift factor shift corresponding to the coefficient hyper-parameter characteristic coefficient quantized value C_q are determined based on the step parameter qstep or the quantization parameter qp; assuming that the coefficient hyper-parameter characteristic coefficient quantized value C_q is Coff_hyper, and the coefficient hyper-parameter characteristic coefficient reconstructed value C' is Coff_hyper_rec, then Coff_hyper_rec = (Coff_hyper * mult) << shift. In summary, when dequantizing the coefficient hyper-parameter characteristic coefficient quantized value C_q, the coefficient hyper-parameter characteristic coefficient reconstructed value C' can be obtained by using the above formula.

[0218] It should be noted that the quantization-related parameters (such as the step parameter qstep) corresponding to the coefficient hyper-parameter characteristic coefficient quantized value C_q can include: 1) the same step parameter qstep is used for each coefficient hyper-parameter characteristic coefficient quantized value of each characteristic channel; 2) different step parameters qstep are used for the coefficient hyper-parameter characteristic coefficient quantized values of each characteristic channel, but the same step parameter qstep is used for each coefficient hyper-parameter characteristic coefficient quantized value in the characteristic channel; 3) different step parameters qstep are used for each coefficient hyper-parameter characteristic coefficient quantized value of each characteristic channel. In the above process, the step parameter qstep can also be referred to as the quantization step.

[0219] Embodiment 9: For Embodiment 5 and Embodiment 6, the coefficient hyperparameter feature generation unit can perform an inverse transform operation on the coefficient hyperparameter feature coefficient reconstruction value C’ based on the decoding neural network to obtain the coefficient hyperparameter feature value P. In a possible implementation, as shown in FIG. 9, the coefficient hyperparameter feature generation unit can include a decoding neural network 1, the decoding neural network 1 can include a basic layer and an enhancement layer, the coefficient hyperparameter feature coefficient reconstruction value C’ is taken as an input feature of the decoding neural network 1, and the coefficient hyperparameter feature value P is taken as an output feature of the decoding neural network 1. The decoding neural network 1 is configured to perform an inverse transform operation on the coefficient hyperparameter feature coefficient reconstruction value C’. Figure 6C As shown in FIG. 9, the coefficient hyperparameter feature generation unit can include a decoding neural network 1, the decoding neural network 1 can include a basic layer and an enhancement layer, the coefficient hyperparameter feature coefficient reconstruction value C’ is taken as an input feature of the decoding neural network 1, and the coefficient hyperparameter feature value P is taken as an output feature of the decoding neural network 1. The decoding neural network 1 is configured to perform an inverse transform operation on the coefficient hyperparameter feature coefficient reconstruction value C’.

[0220] In this embodiment, the decoding neural network 1 is divided into a basic layer and an enhancement layer. The basic layer can include at least one network layer, or the basic layer can not include a network layer, that is, the basic layer is empty. The enhancement layer can include at least one network layer, or the enhancement layer can not include a network layer, that is, the enhancement layer is empty. It should be noted that for multiple network layers in the decoding neural network 1, the multiple network layers can be divided into a basic layer and an enhancement layer according to actual needs. For example, the first M1 network layers are taken as the basic layer, and the remaining network layers are taken as the enhancement layer. Or, the first M2 network layers are taken as the enhancement layer, and the remaining network layers are taken as the basic layer. Or, the last M3 network layers are taken as the basic layer, and the remaining network layers are taken as the enhancement layer. Or, the last M4 network layers are taken as the enhancement layer, and the remaining network layers are taken as the basic layer. Or, the odd-numbered network layers are taken as the basic layer, and the remaining network layers are taken as the enhancement layer. Or, the even-numbered network layers are taken as the basic layer, and the remaining network layers are taken as the enhancement layer. Of course, the above is only an example, and the division manner is not limited.

[0221] For example, a network layer with a fixed network structure can be taken as the basic layer, and a network layer with a non-fixed network structure can be taken as the enhancement layer. For example, for a network layer in the decoding neural network 1, if the same network structure is used when decoding multiple image blocks, the network layer is taken as a network layer with a fixed network structure, and the network layer is taken as the basic layer. For another example, for a network layer in the decoding neural network 1, if different network structures are used when decoding multiple image blocks, the network layer is taken as a network layer with a non-fixed network structure, and the network layer is taken as the enhancement layer.

[0222] For example, for the decoding neural network 1, the size of the output feature can be greater than the size of the input feature, or the size of the output feature can be equal to the size of the input feature, or the size of the output feature can be less than the size of the input feature.

[0223] For the decoding neural network 1, at least one deconvolutional layer is included in the base layer and the enhancement layer. For example, the base layer includes at least one deconvolutional layer, and the enhancement layer can include at least one deconvolutional layer or no deconvolutional layer. Alternatively, the enhancement layer includes at least one deconvolutional layer, and the base layer can include at least one deconvolutional layer or no deconvolutional layer.

[0224] For example, the decoding neural network 1 can include, but is not limited to, deconvolutional layers, activation layers, and the like, without limitation. For example, the decoding neural network 1 includes, in sequence, one deconvolutional layer with a stride of 2, one activation layer, one deconvolutional layer with a stride of 1, and one activation layer. All of the network layers described above can be included in the base layer, i.e., the base layer includes one deconvolutional layer with a stride of 2, one activation layer, one deconvolutional layer with a stride of 1, and one activation layer, in which case the enhancement layer is empty. Alternatively, some of the network layers can be included in the enhancement layer, without limitation. For another example, the decoding neural network 1 includes, in sequence, one deconvolutional layer with a stride of 2 and one deconvolutional layer with a stride of 1. All of the network layers described above can be included in the base layer, i.e., the base layer includes one deconvolutional layer with a stride of 2 and one deconvolutional layer with a stride of 1, in which case the enhancement layer is empty. Alternatively, some of the network layers can be included in the enhancement layer, without limitation. For another example, the decoding neural network 1 includes, in sequence, one deconvolutional layer with a stride of 2, one activation layer, one deconvolutional layer with a stride of 2, one activation layer, one deconvolutional layer with a stride of 1, and one activation layer. All of the network layers described above can be included in the base layer, i.e., the base layer includes one deconvolutional layer with a stride of 2, one activation layer, one deconvolutional layer with a stride of 2, one activation layer, one deconvolutional layer with a stride of 1, and one activation layer, in which case the enhancement layer is empty. Alternatively, some of the network layers can be included in the enhancement layer, without limitation. Of course, the above examples are merely illustrative, without limitation.

[0225] In a possible implementation, the coefficient hyperparameter feature generation unit can be configured with network layers of a default network structure (the network layers of the default network structure can be composed of at least one network layer), and network parameters related to the network layers of the default network structure are fixed. For example, the network parameters can include, but are not limited to, at least one of the following: the number of neural network layers, the deconvolution layer flag, the number of deconvolution layers, the quantization step of each deconvolution layer, the number of channels of each deconvolution layer, the size of the convolution kernel, the number of filters, the filter size index, the filter coefficient zero flag, the filter coefficient, the number of activation layers, the activation layer flag, and the activation layer type. That is, the above network parameters are fixed. For example, in the network layers of the default network structure, the number of deconvolution layers is fixed, the number of activation layers is fixed, the number of channels of each deconvolution layer is fixed, the size of the convolution kernel is fixed, the filter coefficient is fixed, and the like. For example, the number of channels of the deconvolution layer is 4, 8, 16, 32, 64, 128, or 256, and the size of the convolution kernel is 1*1, 3*3, or 5x5. Obviously, since the network parameters in the network layers of the default network structure are fixed and known, the network layers of the default network structure can be directly obtained.

[0226] In a possible implementation, the coefficient hyperparameter feature generation unit can be configured with a prefabricated neural network pool, which can include at least one network layer of a prefabricated network structure (the network layers of the prefabricated network structure can be composed of at least one network layer), and network parameters related to the network layers of the prefabricated network structure can be configured according to actual needs. For example, the network parameters can include, but are not limited to, at least one of the following: the number of neural network layers, the deconvolution layer flag, the number of deconvolution layers, the quantization step of each deconvolution layer, the number of channels of each deconvolution layer, the size of the convolution kernel, the number of filters, the filter size index, the filter coefficient zero flag, the filter coefficient, the number of activation layers, the activation layer flag, and the activation layer type. That is, the above network parameters can be configured according to actual needs.

[0227] For example, the prefabricated neural network pool can include network layers of a prefabricated network structure s1, network layers of a prefabricated network structure s2, and network layers of a prefabricated network structure s3. For the network layers of the prefabricated network structure s1, the number of deconvolution layers, the number of activation layers, the number of channels of each deconvolution layer, the size of the convolution kernel, and the filter coefficient can be pre-configured as network parameters. After all network parameters are configured, the network layers of the prefabricated network structure s1 can be obtained. Similarly, the network layers of the prefabricated network structure s2 and the network layers of the prefabricated network structure s3 can be obtained, and details are not repeated here.

[0228] In a possible implementation, the network layer of the variable network structure can be dynamically generated by the coefficient hyperparameter feature generation unit based on network parameters. The network parameters related to the network layer of the variable network structure are dynamically generated at the encoding end rather than preconfigured, for example, the encoding end can send the network parameters corresponding to the coefficient hyperparameter feature generation unit to the decoding end, which can include but is not limited to at least one of the following: the number of neural network layers, the deconvolution layer flag, the number of deconvolution layers, the quantization step of each deconvolution layer, the number of channels of each deconvolution layer, the size of the convolution kernel, the number of filters, the filter size index, the filter coefficient zero flag, the filter coefficient, the number of activation layers, the activation layer flag, and the activation layer type. The decoding end dynamically generates the network layer of the variable network structure based on the above network parameters. For example, the encoding end encodes the number of deconvolution layers, the number of activation layers, the number of channels of each deconvolution layer, the size of the convolution kernel, and the filter coefficient in the code stream. The decoding end can parse the above network parameters from the code stream and generate the network layer of the variable network structure based on these network parameters. The generation process is not limited.

[0229] In a possible implementation, the decoding neural network 1 can be divided into a basic layer and an enhancement layer. Regarding the combination of the basic layer and the enhancement layer, the following methods can be included but are not limited to: method 1, the basic layer adopts the network layer of the default network structure, the enhancement layer adopts the network layer of the default network structure, and the basic layer and the enhancement layer constitute the decoding neural network 1. Method 2, the basic layer adopts the network layer of the default network structure, the enhancement layer adopts the network layer of the prefabricated network structure, and the basic layer and the enhancement layer constitute the decoding neural network 1. Method 3, the basic layer adopts the network layer of the default network structure, the enhancement layer adopts the network layer of the variable network structure, and the basic layer and the enhancement layer constitute the decoding neural network 1. Method 4, the basic layer adopts the network layer of the prefabricated network structure, the enhancement layer adopts the network layer of the default network structure, and the basic layer and the enhancement layer constitute the decoding neural network 1. Method 5, the basic layer adopts the network layer of the prefabricated network structure, the enhancement layer adopts the network layer of the prefabricated network structure, and the basic layer and the enhancement layer constitute the decoding neural network 1. Method 6, the basic layer adopts the network layer of the prefabricated network structure, the enhancement layer adopts the network layer of the variable network structure, and the basic layer and the enhancement layer constitute the decoding neural network 1.

[0230] In a possible implementation, the control parameter can include neural network information 1 corresponding to the coefficient hyperparameter feature generation unit. The coefficient hyperparameter feature generation unit can parse the neural network information 1 from the control parameter, and generate the decoding neural network 1 based on the neural network information 1. For example, the neural network information 1 can include basic layer information and enhancement layer information. The basic layer can be determined based on the basic layer information, and the enhancement layer can be determined based on the enhancement layer information. The basic layer and the enhancement layer are combined to obtain the decoding neural network 1. For example, the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit can be obtained as follows:

[0231] Case 1: The basic layer information includes a basic layer uses default network flag, and the basic layer uses default network flag indicates that the basic layer uses a default network. In this case, the coefficient hyperparameter feature generation unit learns, based on the basic layer information, that the basic layer adopts a network layer of a default network structure. Therefore, the basic layer of the default network structure (i.e., the network layer of the default network structure) is obtained. The enhancement layer information includes an enhancement layer uses default network flag, and the enhancement layer uses default network flag indicates that the enhancement layer uses a default network. In this case, the coefficient hyperparameter feature generation unit learns, based on the enhancement layer information, that the enhancement layer adopts a network layer of a default network structure. Therefore, the enhancement layer of the default network structure (i.e., the network layer of the default network structure) is obtained. On this basis, the basic layer of the default network structure and the enhancement layer of the default network structure can be combined to obtain the decoding neural network 1.

[0232] Case 2: The basic layer information includes a basic layer uses default network flag, and the basic layer uses default network flag indicates that the basic layer uses a default network. In this case, the coefficient hyperparameter feature generation unit learns, based on the basic layer information, that the basic layer adopts a network layer of a default network structure. Therefore, the basic layer of the default network structure (i.e., the network layer of the default network structure) is obtained. The enhancement layer information includes an enhancement layer uses pre-prepared network flag and an enhancement layer pre-prepared network index number, and the enhancement layer uses pre-prepared network flag indicates that the enhancement layer uses a pre-prepared network. In this case, the coefficient hyperparameter feature generation unit learns, based on the enhancement layer information, that the enhancement layer adopts a network layer of a pre-prepared network structure. Therefore, the enhancement layer of the pre-prepared network structure corresponding to the enhancement layer pre-prepared network index number is selected from a pre-prepared neural network pool (for example, when the enhancement layer pre-prepared network index number is 0, the network layer of the pre-prepared network structure s1 is used as the enhancement layer, and when the enhancement layer pre-prepared network index number is 1, the network layer of the pre-prepared network structure s2 is used as the enhancement layer). On this basis, the basic layer of the default network structure and the enhancement layer of the pre-prepared network structure can be combined to obtain the decoding neural network 1.

[0233] In case 3, the base layer information includes a base layer using default network flag, and the base layer using default network flag indicates that the base layer uses a default network. In this case, the coefficient hyperparameter feature generation unit learns, based on the base layer information, that the base layer adopts a network layer of a default network structure, and thus, acquires the base layer of the default network structure (i.e., the network layer of the default network structure). The enhancement layer information includes network parameters used for generating the enhancement layer. In this case, the coefficient hyperparameter feature generation unit can parse the network parameters from the control parameters, and generate the enhancement layer of a variable network structure based on the network parameters. The network parameters can include, but are not limited to, at least one of the following: the number of neural network layers, a deconvolution layer flag, the number of deconvolution layers, the quantization step of each deconvolution layer, the number of channels of each deconvolution layer, the size of a convolution kernel, the number of filters, a filter size index, a filter coefficient zero flag, a filter coefficient, the number of activation layers, an activation layer flag, and an activation layer type. The coefficient hyperparameter feature generation unit can generate the enhancement layer of the variable network structure based on the above network parameters. For example, the coefficient hyperparameter feature generation unit parses, from the control parameters, the number of deconvolution layers, the quantization step of each deconvolution layer, the number of channels of each deconvolution layer, the size of a convolution kernel, and an activation layer type, and generates the enhancement layer of the variable network structure based on these network parameters. On this basis, the coefficient hyperparameter feature generation unit can combine the base layer of the default network structure and the enhancement layer of the variable network structure to obtain the decoding neural network 1.

[0234] In case 4, the base layer information includes a base layer using pre-prepared network flag and a base layer pre-prepared network index number, and the base layer using pre-prepared network flag indicates that the base layer uses a pre-prepared network. In this case, the coefficient hyperparameter feature generation unit learns, based on the base layer information, that the base layer adopts a network layer of a pre-prepared network structure, and thus, selects, from a pre-prepared neural network pool, the base layer of the pre-prepared network structure corresponding to the base layer pre-prepared network index number (for example, when the base layer pre-prepared network index number is 0, the network layer of the pre-prepared network structure s1 is used as the base layer, and when the base layer pre-prepared network index number is 1, the network layer of the pre-prepared network structure s2 is used as the base layer). The enhancement layer information includes an enhancement layer using default network flag, and the enhancement layer using default network flag indicates that the enhancement layer uses a default network. In this case, the coefficient hyperparameter feature generation unit learns, based on the enhancement layer information, that the enhancement layer adopts a network layer of a default network structure, and thus, acquires the enhancement layer of the default network structure (i.e., the network layer of the default network structure). On this basis, the base layer of the pre-prepared network structure and the enhancement layer of the default network structure can be combined to obtain the decoding neural network 1.

[0235] In case 5, the base layer information includes a base layer using pre-configured network flag and a base layer pre-configured network index number, and the base layer using pre-configured network flag indicates that the base layer uses a pre-configured network. In this case, the coefficient hyperparameter feature generation unit learns, based on the base layer information, that the network layer of the base layer adopts a pre-configured network structure, and thus can select, from the pool of pre-configured neural network structures, the base layer of the pre-configured network structure corresponding to the base layer pre-configured network index number (e.g., the network layer of the pre-configured network structure s1, etc.). The enhancement layer information includes an enhancement layer using pre-configured network flag and an enhancement layer pre-configured network index number, and the enhancement layer using pre-configured network flag indicates that the enhancement layer uses a pre-configured network. In this case, the coefficient hyperparameter feature generation unit learns, based on the enhancement layer information, that the network layer of the enhancement layer adopts a pre-configured network structure, and thus can select, from the pool of pre-configured neural network structures, the enhancement layer of the pre-configured network structure corresponding to the enhancement layer pre-configured network index number (e.g., the network layer of the pre-configured network structure s1, etc.). On this basis, the coefficient hyperparameter feature generation unit can combine the base layer of the pre-configured network structure and the enhancement layer of the pre-configured network structure to obtain the decoding neural network 1.

[0236] In case 6, the base layer information includes a base layer using pre-configured network flag and a base layer pre-configured network index number, and the base layer using pre-configured network flag indicates that the base layer uses a pre-configured network. In this case, the coefficient hyperparameter feature generation unit learns, based on the base layer information, that the network layer of the base layer adopts a pre-configured network structure, and thus can select, from the pool of pre-configured neural network structures, the base layer of the pre-configured network structure corresponding to the base layer pre-configured network index number (e.g., the network layer of the pre-configured network structure s1, etc.). The enhancement layer information includes network parameters used to generate the enhancement layer. In this case, the coefficient hyperparameter feature generation unit can parse the network parameters from the control parameters and generate the enhancement layer of the variable network structure based on the network parameters (i.e., the network layer of the variable network structure). For example, the coefficient hyperparameter feature generation unit can parse, from the control parameters, network parameters such as the number of deconvolution layers, the quantization stride of each deconvolution layer, the number of channels of each deconvolution layer, the size of the convolution kernel, the type of activation layer, etc., and generate the enhancement layer of the variable network structure based on these network parameters. On this basis, the coefficient hyperparameter feature generation unit can combine the base layer of the pre-configured network structure and the enhancement layer of the variable network structure to obtain the decoding neural network 1.

[0237] In a possible implementation, the above network structures for the base layer and the enhancement layer can be determined by the decoded control parameters. The control parameters can include the neural network information 1, and the neural network information 1 can include the base layer information and the enhancement layer information. For an example of the neural network information 1 of the coefficient hyperparameter feature generation unit, refer to Table 1, in which u(n) represents an n-bit fixed-length code encoding method, and ae(v) represents a variable-length encoding method.

[0238] Table 1

[0239]

[0240] In Table 1, hyper_basic_layer_use_default_para_flag is a flag indicating whether the basic layer of the coefficient hyper-parameter feature generation unit uses a default network, and hyper_basic_layer_use_default_para_flag is a binary variable. When the value of the binary variable is 1, it indicates that the basic layer of the coefficient hyper-parameter feature generation unit uses a default network, and when the value of the binary variable is 0, it indicates that the basic layer of the coefficient hyper-parameter feature generation unit does not use a default network. The value of HyperBasicLayerUseDefaultParaFlag can be equal to the value of hyper_basic_layer_use_default_para_flag.

[0241] In Table 1, hyper_basic_layer_use_predesigned_para_flag is a flag indicating whether the basic layer of the coefficient hyper-parameter feature generation unit uses a pre-designed network, and hyper_basic_layer_use_predesigned_para_flag is a binary variable. When the value of the binary variable is 1, it indicates that the basic layer of the coefficient hyper-parameter feature generation unit uses a pre-designed network, and when the value of the binary variable is 0, it indicates that the basic layer of the coefficient hyper-parameter feature generation unit does not use a pre-designed network. The value of HyperBasicLayerUsePredesignedParaFlag can be equal to the value of hyper_basic_layer_use_predesigned_para_flag.

[0242] In Table 1, hyper_basic_id is a pre-designed network index number for the basic layer of the coefficient hyper-parameter feature generation unit, which can be a 32-bit unsigned integer, indicating the index number of the neural network used by the basic layer in the pre-designed neural network pool.

[0243] In Table 1, hyper_enhance_layer_use_default_para_flag is a flag indicating whether the enhancement layer of the coefficient hyperparameter feature generation unit uses a default network, and hyper_enhance_layer_use_default_para_flag is a binary variable. When the value of the binary variable is 1, it indicates that the enhancement layer of the coefficient hyperparameter feature generation unit uses a default network, and when the value of the binary variable is 0, it indicates that the enhancement layer of the coefficient hyperparameter feature generation unit does not use a default network. The value of HyperEnhanceLayerUseDefaultParaFlag can be equal to the value of hyper_enhance_layer_use_default_para_flag.

[0244] In Table 1, hyper_enhance_layer_use_predesigned_para_flag is a flag indicating whether the enhancement layer of the coefficient hyperparameter feature generation unit uses a pre-designed network, and hyper_enhance_layer_use_predesigned_para_flag is a binary variable. When the value of the binary variable is 1, it indicates that the enhancement layer of the coefficient hyperparameter feature generation unit uses a pre-designed network, and when the value of the binary variable is 0, it indicates that the enhancement layer of the coefficient hyperparameter feature generation unit does not use a pre-designed network. The value of HyperEnhanceLayerUsePredesignedParaFlag can be equal to the value of hyper_enhance_layer_use_predesigned_para_flag.

[0245] In Table 1, hyper_enhance_id is an index number of the pre-designed network of the enhancement layer of the coefficient hyperparameter feature generation unit, which can be a 32-bit unsigned integer, indicating the index number of the neural network used by the enhancement layer in the pre-designed neural network pool.

[0246] In the above process, the range of hyper_basic_id is [id_min, id_max], id_min is preferably 0, and id_max is preferably 2^32-1. The segment [a, b] is a reserved segment for later expansion of the pre-designed neural network pool. It should be noted that for the pre-designed neural network pool of the basic layer, the pre-designed neural network pool can include several basic layer pre-designed networks, such as 2, 3, 4, etc. It can include dozens of basic layer pre-designed networks, and it can also include more basic layer pre-designed networks, and there is no limitation on this. Obviously, id_max is preferably 2^32-1 is only an example, and the value of id_max can be dynamically adjusted in different cases.

[0247] In the above process, the range of hyper_enhance_id is [id_min, id_max], id_min is preferably 0, id_max is preferably 2^32-1, and the [a, b] segment is a reserved segment for expansion of the prefabricated neural network pool in the future. It should be noted that, for the prefabricated neural network pool of the enhancement layer, the prefabricated neural network pool can include several enhancement layer prefabricated networks, such as 2, 3, 4, etc., can include several tens of enhancement layer prefabricated networks, and can also include more enhancement layer prefabricated networks, and no limitation is made to this. Obviously, id_max is preferably 2^32-1 is only an example, and the value of id_max can be dynamically adjusted in different cases.

[0248] In a possible implementation, for the above network structure of the enhancement layer, the control parameter determined by decoding can be used, the control parameter can include neural network information 1, and the neural network information 1 can include enhancement layer information. For an example of the neural network information 1 of the coefficient hyperparameter feature generation unit, refer to Tables 2 and 3.

[0249] Table 2

[0250]

[0251] Table 3

[0252]

[0253]

[0254] In Table 2, layer_num represents the number of neural network layers, which is used to represent the number of network layers of the neural network. If the activation layer is included in a certain network structure, the number of layers is not calculated additionally, and the value of LayerNum is equal to layer_num.

[0255] In Table 3, deconv_layer_flag represents a deconvolution layer flag bit. deconv_layer_flag is a binary variable. When the value of the binary variable is 1, it indicates that the current layer is a deconvolution layer network. When the value of the binary variable is 0, it indicates that the current layer is not a deconvolution layer network. The value of DeconvLayerFlag is equal to the value of deconv_layer_flag.

[0256] In Table 3, stride_num represents the quantization step length of the deconvolution layer.

[0257] In Table 3, filter_num represents the number of filters, i.e., the number of filters of the current layer.

[0258] In Table 3, filter_size_index represents the filter size index, i.e., the current filter size index value.

[0259] In Table 3, filter_coeff_zero_flag[i][j] represents the filter coefficient zero flag, which is a binary variable. When the value of the binary variable is 1, it indicates that the current filter coefficient is 0, and when the value of the binary variable is 0, it indicates that the current filter coefficient is not 0. The value of FilterCoeffZeroFlag[i][j] is equal to the value of filter_coeff_zero_flag[i][j].

[0260] In Table 3, filter_coeff[i][j] represents the filter coefficient, i.e., the current filter coefficient value.

[0261] In Table 3, activation_layer_flag represents the activation layer flag, which is a binary variable. When the value of the binary variable is 1, it indicates that the current layer is an activation layer, and when the value of the binary variable is 0, it indicates that the current layer is not an activation layer. The value of ActivationLayerFlag is equal to the value of activation_layer_flag.

[0262] In Table 3, activation_layer_type represents the activation layer type, i.e., the specific type of the activation layer of the current layer.

[0263] Embodiment 10: For Embodiment 5 and Embodiment 6, the second feature decoding unit can decode the code stream 2 corresponding to the current block to obtain the image feature information corresponding to the current block. For example, the second feature decoding unit includes at least one coefficient decoding module and one probability model obtaining module. In one possible implementation, the coefficient decoding module can use an entropy decoding method to decode the code stream 2 corresponding to the current block to obtain the image feature information corresponding to the current block.

[0264] For example, the entropy decoding method can include, but is not limited to, CAVLC or CABAC, etc., without limitation.

[0265] For example, the way in which the coefficient hyperparameter feature generating unit generates the features for the second feature decoding unit can include:

[0266] Way 1, the probability model obtaining module is configured to obtain the probability model for entropy decoding. For example, the probability model obtaining module obtains the coefficient hyper-parameter feature value P from the coefficient hyper-parameter feature generating unit. On this basis, the coefficient decoding module can obtain the coefficient hyper-parameter feature value P from the probability model obtaining module, and based on the coefficient hyper-parameter feature value P, the coefficient decoding module can perform coefficient decoding using an entropy decoding method.

[0267] Way 2, the coefficient analysis process (such as the CABAC or CAVCL decoding process) does not depend on the features generated by the coefficient hyper-parameter feature generating unit, and can directly analyze the coefficient values (which can ensure the analysis throughput or rate). Based on the features generated by the coefficient hyper-parameter feature generating unit, the analyzed coefficient values are converted to obtain the image feature quantization value F_q. For example, if the coefficient value obtained by the coefficient analysis process is 0, and the corresponding feature value generated by the coefficient hyper-parameter feature generating unit is u, then F_q = u; if the coefficient value obtained by the coefficient analysis process is 1, and the corresponding feature value generated by the coefficient hyper-parameter feature generating unit is u, then F_q = u + x, where x is the corresponding coefficient variance.

[0268] Embodiment 11: For Embodiment 5, the second dequantization unit can dequantize the image feature quantization value F_q (i.e., the image feature information) to obtain the image feature reconstruction value F'. For example, the second dequantization unit can not exist, or if the second dequantization unit exists, it can be selectively skipped based on a control parameter (such as a high-level syntax, such as second enabling information, etc.), or it can be determined to be enabled based on a control parameter. For example, if the second dequantization unit does not exist, the image feature reconstruction value F' is the same as the image feature quantization value F_q, i.e., the image feature quantization value F_q does not need to be dequantized. If the second dequantization unit is selectively skipped based on a control parameter, the image feature reconstruction value F' is the same as the image feature quantization value F_q, i.e., the image feature quantization value F_q does not need to be dequantized. If the second dequantization unit is determined to be enabled based on a control parameter, but the step size parameter qstep corresponding to the image feature quantization value F_q is 1, then the image feature reconstruction value F' is the same as the image feature quantization value F_q, i.e., the image feature quantization value F_q does not need to be dequantized.

[0269] Exemplarily, if it is determined to enable the second inverse quantization unit based on the control parameter, and the step parameter qstep corresponding to the image feature quantization value F_q is not 1, the second inverse quantization unit can perform inverse quantization on the image feature quantization value F_q based on the control parameter (such as quantization-related parameters) to obtain the image feature reconstruction value F'. For example, the second inverse quantization unit performs the following operations: obtain the quantization-related parameters corresponding to the image feature quantization value F_q from the control parameter (the control parameter is included in the bitstream, and the control parameter can include quantization-related parameters), such as the step parameter qstep or the quantization parameter qp. Determine the multiplication factor mult and the shift factor shift corresponding to the image feature quantization value F_q based on the step parameter qstep or the quantization parameter qp; assume that the image feature quantization value F_q is Coff_hyper and the image feature reconstruction value F' is Coff_hyper_rec, then Coff_hyper_rec = (Coff_hyper * mult) << shift, that is, when performing inverse quantization, the above formula can be used to obtain the image feature reconstruction value F'.

[0270] It should be noted that for the quantization-related parameters (such as the step parameter qstep) corresponding to the image feature quantization value F_q, it includes: 1) Each image feature quantization value of each feature channel uses the same step parameter qstep; 2) The image feature quantization values of each feature channel use different step parameter qsteps, however, each image feature quantization value within the feature channel uses the same step parameter qstep; 3) Each image feature quantization value of each feature channel uses different step parameter qsteps.

[0271] Embodiment 12: For Embodiment 5 and Embodiment 6, the image feature inverse transformation unit can perform an inverse transformation operation on the image feature reconstruction value F' based on the decoding neural network to obtain the image low-order feature value LF. In a possible implementation manner, as shown in Figure 6D shown, the image feature inverse transformation unit can include a decoding neural network 2, and the decoding neural network 2 can include a basic layer and an enhancement layer. The image feature reconstruction value F' is used as the input feature of the decoding neural network 2, and the image low-order feature value LF is used as the output feature of the decoding neural network 2. The decoding neural network 2 is used to perform an inverse transformation operation on the image feature reconstruction value F'.

[0272] In this embodiment, the decoding neural network 2 can be divided into a basic layer and an enhancement layer. The basic layer can include at least one network layer, or the basic layer can not include a network layer, i.e., the basic layer is empty. The enhancement layer can include at least one network layer, or the enhancement layer can not include a network layer, i.e., the enhancement layer is empty. It should be noted that, for multiple network layers in the decoding neural network 2, the multiple network layers can be divided into the basic layer and the enhancement layer according to actual needs. For example, a network layer with a fixed network structure can be used as the basic layer, and a network layer with a non-fixed network structure can be used as the enhancement layer.

[0273] For the decoding neural network 2, the size of the output feature can be greater than the size of the input feature, or the size of the output feature can be equal to the size of the input feature, or the size of the output feature can be less than the size of the input feature.

[0274] For the decoding neural network 2, at least one deconvolution layer is included in the basic layer and the enhancement layer. For example, the basic layer includes at least one deconvolution layer, and the enhancement layer can include at least one deconvolution layer or no deconvolution layer. Alternatively, the enhancement layer includes at least one deconvolution layer, and the basic layer can include at least one deconvolution layer or no deconvolution layer.

[0275] For the decoding neural network 2, at least one residual structure layer is included in the basic layer and the enhancement layer. For example, the basic layer includes at least one residual structure layer, and the enhancement layer can include a residual structure layer or no residual structure layer. Alternatively, the enhancement layer includes at least one residual structure layer, and the basic layer can include a residual structure layer or no residual structure layer.

[0276] For the decoding neural network 2, the following layers can be included but are not limited to, such as a deconvolution layer and an activation layer, and the like, without limitation. For example, the decoding neural network 2 sequentially includes a deconvolution layer with a stride of 2, an activation layer, a deconvolution layer with a stride of 1, and an activation layer. All the network layers described above can be used as the basic layer, and the enhancement layer is empty. For another example, the decoding neural network 2 sequentially includes a deconvolution layer with a stride of 2 and a deconvolution layer with a stride of 1. All the network layers described above can be used as the basic layer, and the enhancement layer is empty. For another example, the decoding neural network 2 sequentially includes a deconvolution layer with a stride of 2, an activation layer, a deconvolution layer with a stride of 2, an activation layer, a deconvolution layer with a stride of 1, and an activation layer. All the network layers described above can be used as the basic layer, and the enhancement layer is empty.

[0277] For example, if the quality enhancement unit is not present, the output feature number (filter number) of the last network layer of the image feature deconvolution unit is 1 or 3. Specifically, if the output is only one channel of values (such as a grayscale image), the output feature number of the last network layer of the image feature deconvolution unit is 1; if the output is only three channels of values (such as an RGB or YUV format), the output feature number of the last network layer of the image feature deconvolution unit is 3.

[0278] For example, if the quality enhancement unit is present, the output feature number (filter number) of the last network layer of the image feature deconvolution unit can be 1 or 3, or other values, which are not limited.

[0279] In a possible implementation, the image feature deconvolution unit can be configured with a network layer of a default network structure, and network parameters related to the network layer of the default network structure are fixed. For example, in the network layer of the default network structure, the number of deconvolution layers is fixed, the number of activation layers is fixed, the number of channels of each deconvolution layer is fixed, the size of the convolution kernel is fixed, the filter coefficient is fixed, and the like. Obviously, since the network parameters in the network layer of the default network structure are fixed and known, the network layer of the default network structure can be directly obtained.

[0280] In a possible implementation, the image feature deconvolution unit can be configured with a pre-prepared neural network pool, and the pre-prepared neural network pool includes at least one network layer of a pre-prepared network structure, and network parameters related to the network layer of the pre-prepared network structure can be configured according to actual needs. For example, the pre-prepared neural network pool includes a network layer of a pre-prepared network structure t1, a network layer of a pre-prepared network structure t2, and a network layer of a pre-prepared network structure t3. For the network layer of the pre-prepared network structure t1, the number of deconvolution layers, the number of activation layers, the number of channels of each deconvolution layer, the size of the convolution kernel, the filter coefficient, and the like can be pre-configured, and after all the network parameters are configured, the network layer of the pre-prepared network structure t1 can be obtained, and the like.

[0281] In a possible implementation, the image feature deconvolution unit can be dynamically generated with a network layer of a variable network structure based on network parameters, and network parameters related to the network layer of the variable network structure are dynamically generated at the encoding end rather than pre-configured. For example, the encoding end can encode the number of deconvolution layers, the number of activation layers, the number of channels of each deconvolution layer, the size of the convolution kernel, the filter coefficient, and the like in the code stream, and thus the decoding end can parse the above network parameters from the code stream and generate the network layer of the variable network structure based on the network parameters.

[0282] In a possible implementation, the control parameter can include neural network information 2 corresponding to the image feature inverse transformation unit, the image feature inverse transformation unit can parse the neural network information 2 from the control parameter, and generate the decoding neural network 2 based on the neural network information 2. For example, the neural network information 2 can include basic layer information and enhancement layer information, the basic layer can be determined based on the basic layer information, the enhancement layer can be determined based on the enhancement layer information, and the basic layer and the enhancement layer are combined to obtain the decoding neural network 2. For example, the neural network information 2 can be obtained as follows:

[0283] Case 1, the basic layer information includes a basic layer using default network flag, and the basic layer using default network flag indicates that the basic layer uses a default network, in this case, the image feature inverse transformation unit learns, based on the basic layer information, that the basic layer uses a network layer of a default network structure, and thus, the basic layer of the default network structure is obtained. The enhancement layer information includes an enhancement layer using default network flag, and the enhancement layer using default network flag indicates that the enhancement layer uses a default network, in this case, the enhancement layer uses a network layer of a default network structure is learned based on the enhancement layer information, and thus, the enhancement layer of the default network structure is obtained. The basic layer of the default network structure and the enhancement layer of the default network structure are combined to obtain the decoding neural network 2.

[0284] Case 2, the basic layer information includes a basic layer using default network flag, and the basic layer using default network flag indicates that the basic layer uses a default network, in this case, the image feature inverse transformation unit learns, based on the basic layer information, that the basic layer uses a network layer of a default network structure, and thus, the basic layer of the default network structure is obtained. The enhancement layer information includes an enhancement layer using pre-prepared network flag and an enhancement layer pre-prepared network index number, and the enhancement layer using pre-prepared network flag indicates that the enhancement layer uses a pre-prepared network, in this case, the enhancement layer uses a network layer of a pre-prepared network structure is learned based on the enhancement layer information, and thus, the enhancement layer of the pre-prepared network structure corresponding to the pre-prepared network index number is selected from the pre-prepared neural network pool. The basic layer of the default network structure and the enhancement layer of the pre-prepared network structure can be combined to obtain the decoding neural network 2.

[0285] Case 3, the basic layer information includes a basic layer using default network flag, and the basic layer using default network flag indicates that the basic layer uses a default network, the image feature inverse transformation unit learns, based on the basic layer information, that the basic layer uses a network layer of a default network structure, and thus, the basic layer of the default network structure is obtained. The enhancement layer information includes network parameters used to generate the enhancement layer, in this case, the network parameters are parsed from the control parameter, and the enhancement layer of a variable network structure is generated based on the network parameters. The basic layer of the default network structure and the enhancement layer of the variable network structure are combined to obtain the decoding neural network 2.

[0286] Case 4, the base layer information includes a base layer using pre-made network flag bit and a base layer pre-made network index number, and the base layer using pre-made network flag bit indicates that the base layer uses a pre-made network, in this case, the image feature inverse transformation unit learns from the base layer information that the base layer uses a network layer of a pre-made network structure, and therefore, the base layer of the pre-made network structure corresponding to the base layer pre-made network index number is selected from the pre-made neural network pool. The enhancement layer information includes an enhancement layer using default network flag bit, and the enhancement layer using default network flag bit indicates that the enhancement layer uses a default network, in this case, the enhancement layer uses a network layer of a default network structure is learned based on the enhancement layer information, and therefore, the enhancement layer of the default network structure is obtained. The base layer of the pre-made network structure and the enhancement layer of the default network structure can be combined to obtain the decoding neural network 2.

[0287] Case 5, the base layer information includes a base layer using pre-made network flag bit and a base layer pre-made network index number, and the base layer using pre-made network flag bit indicates that the base layer uses a pre-made network, in this case, the image feature inverse transformation unit learns from the base layer information that the base layer uses a network layer of a pre-made network structure, and therefore, the base layer of the pre-made network structure corresponding to the base layer pre-made network index number can be selected from the pre-made neural network pool. The enhancement layer information includes an enhancement layer using pre-made network flag bit and an enhancement layer pre-made network index number, and the enhancement layer using pre-made network flag bit indicates that the enhancement layer uses a pre-made network, in this case, the enhancement layer uses a network layer of a pre-made network structure is learned based on the enhancement layer information, and therefore, the enhancement layer of the pre-made network structure corresponding to the enhancement layer pre-made network index number can be selected from the pre-made neural network pool. The base layer of the pre-made network structure and the enhancement layer of the pre-made network structure can be combined to obtain the decoding neural network 2.

[0288] Case 6, the base layer information includes a base layer using pre-made network flag bit and a base layer pre-made network index number, and the base layer using pre-made network flag bit indicates that the base layer uses a pre-made network, in this case, the image feature inverse transformation unit learns from the base layer information that the base layer uses a network layer of a pre-made network structure, and therefore, the base layer of the pre-made network structure corresponding to the base layer pre-made network index number can be selected from the pre-made neural network pool. The enhancement layer information includes network parameters for generating the enhancement layer, in this case, the network parameters are parsed from the control parameters, and the enhancement layer of a variable network structure is generated based on the network parameters. The base layer of the pre-made network structure and the enhancement layer of the variable network structure are combined to obtain the decoding neural network 2.

[0289] In a possible implementation, the network structure for the base layer and the enhancement layer can be determined by the decoded control parameters, and the control parameters can include the neural network information 2, and the neural network information 2 can include the base layer information and the enhancement layer information. The neural network information 2 for the image feature inverse transformation unit is similar to Table 1, Table 2, and Table 3, except that the related information is for the image feature inverse transformation unit, instead of the coefficient hyperparameter feature generation unit, which will not be repeated here.

[0290] In Embodiment 13, for Embodiments 5 and 6, the quality enhancement unit can obtain the image low-order feature value LF, perform enhancement processing on the image low-order feature value LF based on the decoded neural network, and obtain the reconstructed image block I corresponding to the current block.

[0291] For example, the quality enhancement unit can not exist, or if the quality enhancement unit exists, the quality enhancement unit can be selectively skipped based on the control parameters (such as high-level syntax, such as third enabling information, etc.), or the quality enhancement unit can be enabled based on the control parameters. For example, if the quality enhancement unit is enabled based on the control parameters, the quality enhancement unit can be used to remove blockiness, quantization distortion, and other image quality degradation problems. For example, the quality enhancement unit can perform enhancement processing on the image low-order feature value LF based on the decoded neural network, and obtain the reconstructed image block I corresponding to the current block.

[0292] In a possible implementation, the quality enhancement unit can include the decoded neural network 3, the decoded neural network 3 can include a base layer and an enhancement layer, the image low-order feature value LF is used as the input feature of the decoded neural network 3, the reconstructed image block I is used as the output feature of the decoded neural network 3, and the decoded neural network 3 is used to perform enhancement processing on the image low-order feature value LF.

[0293] In this embodiment, the decoded neural network 3 can be divided into a base layer and an enhancement layer. The base layer can include at least one network layer, or the base layer can not include a network layer, i.e., the base layer is empty. The enhancement layer can include at least one network layer, or the enhancement layer can not include a network layer, i.e., the enhancement layer is empty. It should be noted that for multiple network layers in the decoded neural network 3, the multiple network layers can be divided into the base layer and the enhancement layer according to actual needs. For example, the network layer with fixed network structure can be used as the base layer, and the network layer with non-fixed network structure can be used as the enhancement layer.

[0294] For example, for the decoded neural network 3, the size of the output feature can be greater than the size of the input feature, or the size of the output feature can be equal to the size of the input feature, or the size of the output feature can be less than the size of the input feature.

[0295] For the decoding neural network 3, at least one deconvolutional layer is included in the base layer and the enhancement layer. For example, the base layer includes at least one deconvolutional layer, and the enhancement layer can include at least one deconvolutional layer or no deconvolutional layer. Alternatively, the enhancement layer includes at least one deconvolutional layer, and the base layer can include at least one deconvolutional layer or no deconvolutional layer.

[0296] For the decoding neural network 3, at least one residual structure layer is included in the base layer and the enhancement layer. For example, the base layer includes at least one residual structure layer, and the enhancement layer can include a residual structure layer or no residual structure layer. Alternatively, the enhancement layer includes at least one residual structure layer, and the base layer can include a residual structure layer or no residual structure layer.

[0297] For example, the decoding neural network 3 can include, but is not limited to, a deconvolutional layer, an activation layer, etc., without limitation. For example, the decoding neural network 3 includes, in sequence, one deconvolutional layer with a stride of 2, one activation layer, one deconvolutional layer with a stride of 1, and one activation layer. All the network layers described above can be the base layer, and the enhancement layer is empty. For another example, the decoding neural network 3 includes, in sequence, one deconvolutional layer with a stride of 2 and one deconvolutional layer with a stride of 1. All the network layers described above can be the base layer, and the enhancement layer is empty. For another example, the decoding neural network 3 includes, in sequence, one deconvolutional layer with a stride of 2, one activation layer, one deconvolutional layer with a stride of 2, one activation layer, one deconvolutional layer with a stride of 1, and one activation layer. All the network layers described above can be the base layer, and the enhancement layer is empty.

[0298] For example, the output feature number (filter number) of the last network layer of the quality enhancement unit is 1 or 3. Specifically, if the output is only one channel of values (such as a grayscale image), the output feature number of the last network layer is 1; if the output is only three channels of values (such as an RGB or YUV format), the output feature number of the last network layer is 3.

[0299] In a possible implementation, the quality enhancement unit can be configured with a network layer of a default network structure, and network parameters related to the network layer of the default network structure are fixed. The quality enhancement unit can be configured with a pre-prepared neural network pool, and the pre-prepared neural network pool includes at least one network layer of a pre-prepared network structure, and network parameters related to the network layer of the pre-prepared network structure can be configured according to actual needs. The quality enhancement unit can be dynamically generated with a network layer of a variable network structure based on network parameters, and network parameters related to the network layer of the variable network structure are dynamically generated at the encoding end, rather than being pre-configured.

[0300] In a possible implementation, the control parameter can include neural network information 3 corresponding to the quality enhancement unit, the quality enhancement unit can parse the neural network information 3 from the control parameter, and generate the decoding neural network 3 based on the neural network information 3. For example, the neural network information 3 can include base layer information and enhancement layer information, the base layer can be determined based on the base layer information, the enhancement layer can be determined based on the enhancement layer information, and the base layer and the enhancement layer are combined to obtain the decoding neural network 3. For example, the quality enhancement unit can obtain the neural network information 3 in the following cases:

[0301] Case 1: The base layer information includes a base layer uses default network flag, and the base layer uses default network flag indicates that the base layer uses a default network, in this case, the quality enhancement unit learns from the base layer information that the base layer uses a network layer of a default network structure, and thus obtains the base layer of the default network structure. The enhancement layer information includes an enhancement layer uses default network flag, and the enhancement layer uses default network flag indicates that the enhancement layer uses a default network, in this case, the quality enhancement unit learns from the enhancement layer information that the enhancement layer uses a network layer of a default network structure, and thus obtains the enhancement layer of the default network structure. The base layer of the default network structure and the enhancement layer of the default network structure are combined to obtain the decoding neural network 3.

[0302] Case 2: The base layer information includes a base layer uses default network flag, and the base layer uses default network flag indicates that the base layer uses a default network, in this case, the quality enhancement unit learns from the base layer information that the base layer uses a network layer of a default network structure, and thus obtains the base layer of the default network structure. The enhancement layer information includes an enhancement layer uses pre-made network flag and an enhancement layer pre-made network index number, and the enhancement layer uses pre-made network flag indicates that the enhancement layer uses a pre-made network, in this case, the quality enhancement unit learns from the enhancement layer information that the enhancement layer uses a network layer of a pre-made network structure, and thus selects the enhancement layer of the pre-made network structure corresponding to the enhancement layer pre-made network index number from a pre-made neural network pool. The base layer of the default network structure and the enhancement layer of the pre-made network structure can be combined to obtain the decoding neural network 3.

[0303] Case 3: The base layer information includes a base layer uses default network flag, and the base layer uses default network flag indicates that the base layer uses a default network, the quality enhancement unit learns from the base layer information that the base layer uses a network layer of a default network structure, and thus obtains the base layer of the default network structure. The enhancement layer information includes network parameters used to generate the enhancement layer, in this case, the network parameters are parsed from the control parameter, and the enhancement layer of a variable network structure is generated based on the network parameters. The base layer of the default network structure and the enhancement layer of the variable network structure are combined to obtain the decoding neural network 3.

[0304] Case 4, the base layer information includes a base layer using pre-made network flag bit and a base layer pre-made network index number, and the base layer using pre-made network flag bit indicates that the base layer uses a pre-made network, in this case, the quality enhancement unit learns from the base layer information that the base layer adopts a network layer of a pre-made network structure, therefore, the base layer of the pre-made network structure corresponding to the base layer pre-made network index number is selected from the pre-made neural network pool. The enhancement layer information includes an enhancement layer using default network flag bit, and the enhancement layer using default network flag bit indicates that the enhancement layer uses a default network, in this case, based on the enhancement layer information, it is learned that the enhancement layer adopts a network layer of a default network structure, therefore, the enhancement layer of the default network structure is obtained. The base layer of the pre-made network structure and the enhancement layer of the default network structure can be combined to obtain the decoding neural network 3.

[0305] Case 5, the base layer information includes a base layer using pre-made network flag bit and a base layer pre-made network index number, and the base layer using pre-made network flag bit indicates that the base layer uses a pre-made network, in this case, the quality enhancement unit learns from the base layer information that the base layer adopts a network layer of a pre-made network structure, therefore, the base layer of the pre-made network structure corresponding to the base layer pre-made network index number can be selected from the pre-made neural network pool. The enhancement layer information includes an enhancement layer using pre-made network flag bit and an enhancement layer pre-made network index number, and the enhancement layer using pre-made network flag bit indicates that the enhancement layer uses a pre-made network, in this case, based on the enhancement layer information, it is learned that the enhancement layer adopts a network layer of a pre-made network structure, therefore, the enhancement layer of the pre-made network structure corresponding to the enhancement layer pre-made network index number can be selected from the pre-made neural network pool. The base layer of the pre-made network structure and the enhancement layer of the pre-made network structure can be combined to obtain the decoding neural network 3.

[0306] Case 6, the base layer information includes a base layer using pre-made network flag bit and a base layer pre-made network index number, and the base layer using pre-made network flag bit indicates that the base layer uses a pre-made network, in this case, the quality enhancement unit learns from the base layer information that the base layer adopts a network layer of a pre-made network structure, therefore, the base layer of the pre-made network structure corresponding to the base layer pre-made network index number can be selected from the pre-made neural network pool. The enhancement layer information includes network parameters for generating the enhancement layer, in this case, the network parameters are parsed from the control parameters, and the enhancement layer of a variable network structure is generated based on the network parameters. The base layer of the pre-made network structure and the enhancement layer of the variable network structure are combined to obtain the decoding neural network 3.

[0307] In a possible implementation, the network structure described above for the base layer and the enhancement layer can be determined by the decoded control parameters, and the control parameters can include the neural network information 3, and the neural network information 3 can include the base layer information and the enhancement layer information. The neural network information 3 for the quality enhancement unit is similar to Table 1, Table 2, and Table 3, except that the related information is for the quality enhancement unit instead of the coefficient hyperparameter feature generation unit, which will not be repeated here.

[0308] Embodiment 14: In the embodiments of the present application, a neural network-based image encoding method is proposed, which can be applied to an encoding end (also referred to as a video encoder). As shown in Figure 7A The encoding end can include a control parameter encoding unit, a feature transformation unit, a coefficient hyperparameter feature transformation unit, a first quantization unit, a second quantization unit, a first feature encoding unit, a second feature encoding unit, a first feature decoding unit, a second feature decoding unit, a coefficient hyperparameter feature generation unit, an image feature inverse transformation unit, a first inverse quantization unit, a second inverse quantization unit, and a quality enhancement unit.

[0309] For example, the first quantization unit, the second quantization unit, the first inverse quantization unit, the second inverse quantization unit, and the quality enhancement unit are optional units, and in some scenarios, the processes of these optional units can be selected to be closed or skipped.

[0310] In the embodiments, for each current block (i.e., image block), the code stream corresponding to the current block includes three parts: code stream 0 (code stream containing control parameters), code stream 1 (code stream containing coefficient hyperparameter feature information), and code stream 2 (code stream containing image feature information). The coefficient hyperparameter feature information and the image feature information can be collectively referred to as image information.

[0311] For example, the neural network-based image encoding method in the embodiments can include the following steps:

[0312] In step S71, the feature transformation unit performs feature transformation on the current block I to obtain the image feature value F corresponding to the current block I. For example, the feature transformation unit can perform feature transformation on the current block I to obtain the image feature value F corresponding to the current block I. For example, the feature transformation unit can perform feature transformation on the current block I based on the encoding neural network to obtain the image feature value F corresponding to the current block I. The current block I is used as the input feature of the encoding neural network, and the image feature value F is used as the output feature of the encoding neural network.

[0313] In step S72, the image feature information is determined based on the image feature value F.

[0314] For example, if the second quantization unit is enabled, the second quantization unit can obtain the image feature value F from the feature transformation unit, quantize the image feature value F to obtain an image feature quantized value F_q, and determine the image feature information based on the image feature quantized value F_q, i.e., the image feature information can be the image feature quantized value F_q. In this case, the control parameter encoding unit can encode the second enable information of the second quantization unit in the code stream 0, i.e., the control parameter includes the second enable information of the second quantization unit, and the second enable information is used to indicate that the second quantization unit is enabled. The control parameter encoding unit can also encode the quantization related parameters corresponding to the second quantization unit in the code stream 0, such as the step parameter qstep or the quantization parameter qp, etc.

[0315] For example, if the second quantization unit is not enabled, the image feature information is determined based on the image feature value F, i.e., the image feature information can be the image feature value F. In this case, the control parameter encoding unit can encode the second enable information of the second quantization unit in the code stream 0, and the second enable information is used to indicate that the second quantization unit is not enabled.

[0316] Step S73, performing coefficient hyperparameter feature transformation on the image feature value F to obtain a coefficient hyperparameter feature coefficient value C. For example, the coefficient hyperparameter feature transformation unit performs coefficient hyperparameter feature transformation on the image feature value F to obtain the coefficient hyperparameter feature coefficient value C, such as performing coefficient hyperparameter feature transformation on the image feature value F based on an encoding neural network to obtain the coefficient hyperparameter feature coefficient value C. Wherein, the image feature value F is the input feature of the encoding neural network, and the coefficient hyperparameter feature coefficient value C is the output feature of the encoding neural network.

[0317] Step S74, determining the coefficient hyperparameter feature information based on the coefficient hyperparameter feature coefficient value C.

[0318] For example, if the first quantization unit is enabled, the first quantization unit can obtain the coefficient hyperparameter feature coefficient value C from the coefficient hyperparameter feature transformation unit, quantize the coefficient hyperparameter feature coefficient value C to obtain a coefficient hyperparameter feature quantized value C_q, and determine the coefficient hyperparameter feature information based on the coefficient hyperparameter feature quantized value C_q, i.e., the coefficient hyperparameter feature information can be the coefficient hyperparameter feature quantized value C_q. In this case, the control parameter encoding unit can encode the first enable information of the first quantization unit in the code stream 0, i.e., the control parameter includes the first enable information of the first quantization unit, and the first enable information is used to indicate that the first quantization unit is enabled. The control parameter encoding unit can also encode the quantization related parameters corresponding to the first quantization unit in the code stream 0.

[0319] For example, if the first quantization unit is not enabled, the coefficient hyper-parameter feature information is determined based on the coefficient hyper-parameter feature coefficient value C, i.e., the coefficient hyper-parameter feature information can be the coefficient hyper-parameter feature coefficient value C. In this case, the control parameter encoding unit can encode the first enabling information of the first quantization unit in the code stream 0, and the first enabling information is used to indicate that the first quantization unit is not enabled.

[0320] Step S75, encode the coefficient hyper-parameter feature information (such as the coefficient hyper-parameter feature quantized value C_q or the coefficient hyper-parameter feature coefficient value C) to obtain a code stream 1. For example, the first feature encoding unit can encode the coefficient hyper-parameter feature information in the code stream corresponding to the current block. For convenience, the code stream including the coefficient hyper-parameter feature information is recorded as the code stream 1.

[0321] Step S76, decode the code stream 1 corresponding to the current block to obtain the coefficient hyper-parameter feature information (such as the coefficient hyper-parameter feature quantized value C_q or the coefficient hyper-parameter feature coefficient value C), and determine the coefficient hyper-parameter feature coefficient reconstructed value based on the coefficient hyper-parameter feature information.

[0322] For example, the first feature decoding unit can decode the code stream 1 corresponding to the current block to obtain the coefficient hyper-parameter feature information.

[0323] For example, if the first quantization unit is enabled, the coefficient hyper-parameter feature information is the coefficient hyper-parameter feature quantized value C_q, in which case the first inverse quantization unit can inverse quantize the coefficient hyper-parameter feature quantized value C_q to obtain the coefficient hyper-parameter feature coefficient reconstructed value C', obviously, the coefficient hyper-parameter feature coefficient reconstructed value C' can be the same as the coefficient hyper-parameter feature coefficient value C. If the first quantization unit is not enabled, the coefficient hyper-parameter feature information is the coefficient hyper-parameter feature coefficient value C, in which case the coefficient hyper-parameter feature coefficient value C can be taken as the coefficient hyper-parameter feature coefficient reconstructed value C'. In summary, the coefficient hyper-parameter feature coefficient reconstructed value C' can be obtained.

[0324] In summary, the coefficient hyper-parameter feature coefficient reconstructed value C' is the same as the coefficient hyper-parameter feature coefficient value C, therefore, the process of step S76 can be omitted, and the coefficient hyper-parameter feature coefficient value C can be directly taken as the coefficient hyper-parameter feature coefficient reconstructed value C'. In this case, the encoding end structure shown in FIG. 2 can be improved to obtain the encoding end structure shown in FIG. 3. Figure 7A Figure 7B

[0325] Step S77, perform inverse transform operation on the coefficient hyper-parameter feature coefficient reconstructed value C' to obtain the coefficient hyper-parameter feature value P. For example, the coefficient hyper-parameter feature generation unit performs inverse transform operation on the coefficient hyper-parameter feature coefficient reconstructed value C' to obtain the coefficient hyper-parameter feature value P.

[0326] ​​Step S78, encode the image feature information (e.g., the image feature quantization value F_q or the image feature value F) based on the coefficient hyperparameter feature value P, to obtain a code stream 2. For example, the second feature encoding unit can encode the image feature information in the code stream corresponding to the current block. For convenience, the code stream including the image feature information is denoted as code stream 2.

[0327] Step S79, the control parameter encoding unit acquires the control parameter corresponding to the current block, which can include the neural network information, and encodes the control parameter corresponding to the current block in the code stream. The code stream including the control parameter is denoted as code stream 0.

[0328] In a possible implementation, the feature transformation unit can perform feature transformation on the current block I by using an encoding neural network. The control parameter encoding unit can determine the neural network information 2 corresponding to the image feature inverse transformation unit of the decoding end device based on the network structure of the encoding neural network, the neural network information 2 being used to determine a decoding neural network 2 corresponding to the image feature inverse transformation unit, and encode the neural network information 2 corresponding to the image feature inverse transformation unit in the code stream 0.

[0329] In a possible implementation, the coefficient hyperparameter feature transformation unit can perform coefficient hyperparameter feature transformation on the image feature value F by using an encoding neural network. The control parameter encoding unit can determine the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit of the decoding end device based on the network structure of the encoding neural network, the neural network information 1 being used to determine a decoding neural network 1 corresponding to the coefficient hyperparameter feature generation unit, and encode the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit in the code stream 0.

[0330] In a possible implementation, continuing to refer to FIGS. 7A and 7B, the method can further include the following steps. Figure 7A Figure 7B

[0331] Step S80, decode the code stream 1 corresponding to the current block to obtain the coefficient hyperparameter feature information corresponding to the current block. For example, the first feature decoding unit can decode the code stream 1 corresponding to the current block to obtain the coefficient hyperparameter feature information corresponding to the current block.

[0332] Step S81, determine the coefficient hyperparameter feature coefficient reconstruction value based on the coefficient hyperparameter feature information.

[0333] ​​For example, if the first quantization unit is enabled, the coefficient hyper-parameter characteristic information is a coefficient hyper-parameter characteristic quantization value C_q, in which case the first dequantization unit can perform dequantization on the coefficient hyper-parameter characteristic quantization value C_q to obtain a coefficient hyper-parameter characteristic coefficient reconstruction value C'. Or, if the first quantization unit is not enabled, the coefficient hyper-parameter characteristic information is a coefficient hyper-parameter characteristic coefficient value C, in which case the coefficient hyper-parameter characteristic coefficient value C can be taken as the coefficient hyper-parameter characteristic coefficient reconstruction value C'.

[0334] Step S82, perform inverse transformation on the coefficient hyper-parameter characteristic coefficient reconstruction value C' to obtain a coefficient hyper-parameter characteristic value P. For example, the coefficient hyper-parameter characteristic generation unit performs inverse transformation on the coefficient hyper-parameter characteristic coefficient reconstruction value C' to obtain the coefficient hyper-parameter characteristic value P, such as performing inverse transformation on the coefficient hyper-parameter characteristic coefficient reconstruction value C' based on a decoding neural network to obtain the coefficient hyper-parameter characteristic value P.

[0335] Step S83, decode the code stream 2 corresponding to the current block to obtain image characteristic information corresponding to the current block. For example, the second feature decoding unit can decode the code stream 2 corresponding to the current block to obtain the image characteristic information corresponding to the current block. In decoding the code stream 2 corresponding to the current block, the second feature decoding unit can decode the code stream 2 corresponding to the current block by using the coefficient hyper-parameter characteristic value P.

[0336] Step S84, determine an image feature reconstruction value based on the image characteristic information.

[0337] For example, if the second quantization unit is enabled, the image characteristic information is an image feature quantization value F_q, and the second dequantization unit can perform dequantization on the image feature quantization value F_q to obtain the image feature reconstruction value F'. If the second quantization unit is not enabled, the image characteristic information is an image feature value F, and the image feature value F can be taken as the image feature reconstruction value F'.

[0338] Step S85, perform inverse transformation on the image feature reconstruction value F' to obtain an image low-order feature value LF. For example, the image feature inverse transformation unit performs inverse transformation on the image feature reconstruction value F' to obtain the image low-order feature value LF, such as performing inverse transformation on the image feature reconstruction value F' based on a decoding neural network to obtain the image low-order feature value LF.

[0339] Step S86, determine a reconstruction image block I corresponding to the current block based on the image low-order feature value LF. For example, if the quality enhancement unit is enabled, the quality enhancement unit performs enhancement processing on the image low-order feature value LF to obtain the reconstruction image block I corresponding to the current block, such as performing enhancement processing on the image low-order feature value LF based on a decoding neural network to obtain the reconstruction image block I corresponding to the current block. If the quality enhancement unit is not enabled, the image low-order feature value LF is taken as the reconstruction image block I.

[0340] For example, with regard to step S80-step S86, please refer to embodiment 5, which will not be repeated here.

[0341] In a possible implementation, the coefficient hyper-parameter feature generation unit can perform inverse transform operation on the coefficient hyper-parameter feature coefficient reconstruction value C' by using a decoding neural network to obtain the coefficient hyper-parameter feature value P, the control parameter encoding unit can determine the neural network information 1 corresponding to the coefficient hyper-parameter feature generation unit of the decoding end device based on the network structure of the decoding neural network, the neural network information 1 is used to determine the decoding neural network 1 corresponding to the coefficient hyper-parameter feature generation unit, and the neural network information 1 is encoded in the code stream 0.

[0342] In a possible implementation, the image feature inverse transform unit can perform inverse transform operation on the image feature reconstruction value F' by using a decoding neural network to obtain the image low-order feature value LF, the control parameter encoding unit can determine the neural network information 2 corresponding to the image feature inverse transform unit of the decoding end device based on the network structure of the decoding neural network, the neural network information 2 is used to determine the decoding neural network 2 corresponding to the image feature inverse transform unit, and the neural network information 2 is encoded in the code stream 0.

[0343] In a possible implementation, the quality enhancement unit can perform enhancement processing on the image low-order feature value LF by using a decoding neural network to obtain the reconstruction image block I corresponding to the current block, the control parameter encoding unit can determine the neural network information 3 corresponding to the quality enhancement unit of the decoding end device based on the network structure of the decoding neural network, the neural network information 3 is used to determine the decoding neural network 3 corresponding to the quality enhancement unit, and the neural network information 3 is encoded in the code stream 0.

[0344] Embodiment 15: In the embodiments of the present application, a neural network-based image encoding method is proposed, which can be applied to the encoding end. Please refer to Figure 7C As shown in the figure, the structure schematic diagram of the encoding end can include a control parameter encoding unit, a feature transform unit, a coefficient hyper-parameter feature transform unit, a first feature encoding unit, a second feature encoding unit, a first feature decoding unit, a second feature decoding unit, a coefficient hyper-parameter feature generation unit, an image feature inverse transform unit, and a quality enhancement unit.

[0345] For example, the neural network-based image encoding method in the embodiments can include the following steps:

[0346] Step S91, performing feature transformation on the current block I to obtain the image feature value F corresponding to the current block I. For example, the feature transform unit can perform feature transformation on the current block I based on the encoding neural network to obtain the image feature value F corresponding to the current block I.

[0347] Step S92, the image feature value F is transformed by the coefficient hyperparameter feature, and the coefficient hyperparameter feature coefficient value C is obtained. For example, the coefficient hyperparameter feature transformation unit transforms the image feature value F by the coefficient hyperparameter feature based on the encoding neural network, and the coefficient hyperparameter feature coefficient value C is obtained.

[0348] Step S93, the coefficient hyperparameter feature coefficient value C is encoded, and the code stream 1 is obtained. For example, the first feature encoding unit can encode the coefficient hyperparameter feature coefficient value C in the code stream corresponding to the current block, and the code stream 1 is obtained.

[0349] Step S94, the coefficient hyperparameter feature coefficient reconstruction value C' is determined based on the coefficient hyperparameter feature coefficient value C, and the coefficient hyperparameter feature value P is obtained by inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value C'. For example, the coefficient hyperparameter feature generation unit performs inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value C', and the coefficient hyperparameter feature value P is obtained.

[0350] Step S95, the image feature value F is encoded based on the coefficient hyperparameter feature value P, and the code stream 2 is obtained. For example, the second feature encoding unit encodes the image feature value F in the code stream corresponding to the current block, and the code stream 2 is obtained.

[0351] Step S96, the control parameter encoding unit acquires the control parameter corresponding to the current block, which can include neural network information, and encodes the control parameter corresponding to the current block in the code stream, and the code stream including the control parameter is recorded as code stream 0.

[0352] Step S97, the code stream 1 corresponding to the current block is decoded, and the coefficient hyperparameter feature coefficient value C corresponding to the current block is obtained. For example, the first feature decoding unit can decode the code stream 1 corresponding to the current block, and the coefficient hyperparameter feature coefficient value C corresponding to the current block is obtained.

[0353] And the coefficient hyperparameter feature coefficient reconstruction value C' is determined based on the coefficient hyperparameter feature coefficient value C.

[0354] Step S98, inverse transformation operation is performed on the coefficient hyperparameter feature coefficient reconstruction value C', and the coefficient hyperparameter feature value P is obtained. For example, the coefficient hyperparameter feature generation unit performs inverse transformation operation on the coefficient hyperparameter feature coefficient reconstruction value C' based on the decoding neural network, and the coefficient hyperparameter feature value P is obtained. The code stream 2 corresponding to the current block is decoded, and the image feature value F corresponding to the current block is obtained, which is taken as the image feature reconstruction value F'. For example, the second feature decoding unit decodes the code stream 2 corresponding to the current block by using the coefficient hyperparameter feature value P.

[0355] Step S99, inverse transform operation is performed on the image feature reconstruction value F' to obtain the image low-order feature value LF. For example, the image feature inverse transform unit performs inverse transform operation on the image feature reconstruction value F' based on the decoding neural network to obtain the image low-order feature value LF. The image low-order feature value LF is used to determine the reconstruction image block I corresponding to the current block. For example, the quality enhancement unit performs enhancement processing on the image low-order feature value LF based on the decoding neural network to obtain the reconstruction image block I corresponding to the current block.

[0356] In the embodiments 14 and 15, the first feature encoding unit can encode the coefficient hyperparameter feature information in the code stream 1 corresponding to the current block. The encoding process of the first feature encoding unit corresponds to the decoding process of the first feature decoding unit, which can be referred to in the embodiment 7. For example, the first feature encoding unit can encode the coefficient hyperparameter feature information by using an entropy encoding method (such as CAVLC or CABAC), which will not be repeated here. The first feature decoding unit can decode the code stream 1 corresponding to the current block to obtain the coefficient hyperparameter feature information corresponding to the current block, which can be referred to in the embodiment 7.

[0357] In the embodiments 14 and 15, the second feature encoding unit can encode the image feature information based on the coefficient hyperparameter feature value P to obtain the code stream 2. The encoding process of the second feature encoding unit corresponds to the decoding process of the second feature decoding unit, which can be referred to in the embodiment 10. For example, the second feature encoding unit can encode the image feature information by using an entropy encoding method (such as CAVLC or CABAC), which will not be repeated here. The second feature decoding unit can decode the code stream 2 corresponding to the current block to obtain the image feature information corresponding to the current block, which can be referred to in the embodiment 10.

[0358] In the embodiments 14 and 15, the first quantization unit can quantize the coefficient hyperparameter feature coefficient value C to obtain the coefficient hyperparameter feature quantized value C_q. The quantization process of the first quantization unit corresponds to the inverse quantization process of the first inverse quantization unit, which can be referred to in the embodiment 8. For example, the first quantization unit quantizes the coefficient hyperparameter feature coefficient value C based on the quantization related parameters, which will not be repeated here. It should be noted that for the quantization related parameters (such as the step parameter qstep, also known as the quantization step), 1) the same quantization step is used for each feature value in each feature channel; 2) different quantization steps are used for each feature channel, but the same quantization step is used for each feature value in the feature channel; 3) different quantization steps are used for each feature value in each feature channel. The first inverse quantization unit at the encoding end can inverse quantize the coefficient hyperparameter feature quantized value C_q to obtain the coefficient hyperparameter feature coefficient reconstruction value C', and the processing process of the first inverse quantization unit at the encoding end can be referred to in the embodiment 10.

[0359] The second quantization unit can quantize the image feature value F to obtain an image feature quantized value F_q. The quantization process of the second quantization unit corresponds to the dequantization process of the second dequantization unit, which is described in Embodiment 11. For example, the second quantization unit quantizes the image feature value F based on quantization-related parameters, which are not repeated here. It should be noted that, for the quantization step, 1) the same quantization step is used for each feature value in each feature channel; 2) different quantization steps are used for each feature channel, but the same quantization step is used for each feature value in the feature channel; 3) different quantization steps are used for each feature value in each feature channel. The second dequantization unit at the encoding end can dequantize the image feature quantized value F_q to obtain the image feature reconstructed value F'. The processing process of the second dequantization unit at the encoding end can be referred to Embodiment 11.

[0360] In Embodiments 14 and 15, the feature transformation unit can transform the current block I based on the encoding neural network to obtain the image feature value F corresponding to the current block I. On this basis, the neural network information 2 corresponding to the image feature inverse transformation unit of the decoding device can be determined based on the network structure of the encoding neural network, and the neural network information 2 corresponding to the image feature inverse transformation unit is encoded in the code stream 0. How the decoding device generates the decoding neural network 2 corresponding to the image feature inverse transformation unit according to the neural network information 2 can be referred to Embodiment 12, which is not repeated here.

[0361] In a possible implementation, the feature transformation unit can include the encoding neural network 2, which can include a basic layer and an enhancement layer. The encoding neural network 2 can be divided into the basic layer and the enhancement layer. The basic layer can include at least one network layer, or the basic layer can be empty. The enhancement layer can include at least one network layer, or the enhancement layer can be empty. It should be noted that, for multiple network layers in the encoding neural network 2, the multiple network layers can be divided into the basic layer and the enhancement layer according to actual needs. For example, the network layer with a fixed network structure can be regarded as the basic layer, and the network layer with a non-fixed network structure can be regarded as the enhancement layer.

[0362] For example, the size of the output feature of the encoding neural network 2 can be smaller than, equal to, or larger than the size of the input feature.

[0363] For example, the basic layer of the encoding neural network 2 includes at least one convolutional layer, and the enhancement layer includes at least one convolutional layer or does not include a convolutional layer. Alternatively, the enhancement layer of the encoding neural network 2 includes at least one convolutional layer, and the basic layer includes at least one convolutional layer or does not include a convolutional layer.

[0364] For example, the basic layer of the encoding neural network 2 includes at least one residual structure layer, and the enhancement layer includes a residual structure layer or does not include a residual structure layer. Alternatively, the enhancement layer of the encoding neural network 2 includes at least one residual structure layer, and the basic layer includes a residual structure layer or does not include a residual structure layer.

[0365] For example, the encoding neural network 2 includes, but is not limited to, a convolutional layer and an activation layer, and the like, and is not limited in this regard. For example, the encoding neural network 2 sequentially includes a convolutional layer with a stride of 2, an activation layer, a convolutional layer with a stride of 1, and an activation layer, and all the network layers are basic layers. For another example, the encoding neural network 2 sequentially includes a convolutional layer with a stride of 2 and a convolutional layer with a stride of 1, and all the network layers are basic layers. For another example, the encoding neural network 2 sequentially includes a convolutional layer with a stride of 2, an activation layer, a convolutional layer with a stride of 2, an activation layer, a convolutional layer with a stride of 1, and an activation layer, and all the network layers are basic layers.

[0366] For example, the first network layer of the feature transformation unit has an input channel number of 1 or 3. If the input image block only has one channel (such as a grayscale image), the first network layer of the feature transformation unit has an input channel number of 1; if the input image block has three channels (such as an RGB or YUV format), the first network layer of the feature transformation unit has an input channel number of 3.

[0367] It should be noted that the network structure of the encoding neural network 2 of the feature transformation unit can be symmetrical to the network structure of the decoding neural network 2 of the image feature inverse transformation unit (see embodiment 12), and the network parameters of the encoding neural network 2 of the feature transformation unit can be the same as or different from the network parameters of the decoding neural network 2 of the image feature inverse transformation unit.

[0368] In a possible implementation, the feature transformation unit can be configured with a network layer of a default network structure, and the network parameters related to the network layer of the default network structure are fixed. For example, in the network layer of the default network structure, the number of deconvolution layers is fixed, the number of activation layers is fixed, the number of channels of each deconvolution layer is fixed, the size of the convolution kernel is fixed, the filter coefficients are fixed, and the like. It should be noted that the network layer of the default network structure configured for the feature transformation unit and the network layer of the default network structure configured for the image feature inverse transformation unit (see embodiment 12) can be a symmetric structure.

[0369] In a possible implementation, the feature transformation unit can be configured with a prefabricated neural network pool (corresponding to the prefabricated neural network pool of the decoding device), and the prefabricated neural network pool includes at least one network layer of a prefabricated network structure, and the network parameters related to the network layer of the prefabricated network structure can be configured according to actual needs. For example, the prefabricated neural network pool can include a network layer of a prefabricated network structure t1', a network layer of a prefabricated network structure t2', a network layer of a prefabricated network structure t3', and the like. It should be noted that the network layer of the prefabricated network structure t1' and the network layer of the prefabricated network structure t1 (see embodiment 12) can be a symmetric structure. The network layer of the prefabricated network structure t2' and the network layer of the prefabricated network structure t2 can be a symmetric structure. The network layer of the prefabricated network structure t3' and the network layer of the prefabricated network structure t3 can be a symmetric structure.

[0370] In a possible implementation, the feature transformation unit can be dynamically generated with a network layer of a variable network structure based on network parameters, and the network parameters related to the network layer of the variable network structure are dynamically generated by the encoding end rather than being preconfigured.

[0371] In a possible implementation, the encoding end can determine the neural network information 2 corresponding to the image feature inverse transformation unit based on the network structure of the encoding neural network 2, and encode the neural network information 2 corresponding to the image feature inverse transformation unit in the code stream 0. For example, the encoding end can determine the neural network information 2 as follows:

[0372] Case 1, if the basic layer of the encoding neural network 2 adopts the network layer of the default network structure, and the enhancement layer adopts the network layer of the default network structure, the encoding end encodes the basic layer information and the enhancement layer information in the code stream 0, and the basic layer information and the enhancement layer information constitute the neural network information 2 corresponding to the image feature inverse transformation unit. The basic layer information includes a default network flag for the basic layer, and the default network flag for the basic layer indicates that the basic layer uses the default network. The enhancement layer information includes a default network flag for the enhancement layer, and the default network flag for the enhancement layer indicates that the enhancement layer uses the default network.

[0373] Case 2, if the basic layer of the encoding neural network 2 adopts the network layer of the default network structure, and the enhancement layer adopts the network layer of the pre-prepared network structure, the encoding end encodes the basic layer information and the enhancement layer information in the code stream 0, and the basic layer information and the enhancement layer information constitute the neural network information 2 corresponding to the image feature inverse transformation unit. Among them, the basic layer information includes a basic layer using a default network flag, and the basic layer using a default network flag indicates that the basic layer uses a default network. The enhancement layer information includes an enhancement layer using a pre-prepared network flag and an enhancement layer pre-prepared network index number, and the enhancement layer using a pre-prepared network flag indicates that the enhancement layer uses a pre-prepared network, and the enhancement layer pre-prepared network index number indicates the index corresponding to the network layer of the pre-prepared network structure in the pre-prepared neural network pool. For example, if the enhancement layer of the encoding neural network 2 adopts the network layer of the pre-prepared network structure t1', the enhancement layer pre-prepared network index number indicates the network layer of the first pre-prepared network structure t1' in the pre-prepared neural network pool.

[0374] Case 3, if the basic layer of the encoding neural network 2 adopts the network layer of the default network structure, and the enhancement layer adopts the enhancement layer of the variable network structure, the encoding end encodes the basic layer information and the enhancement layer information in the code stream 0, and the basic layer information and the enhancement layer information constitute the neural network information 2 corresponding to the image feature inverse transformation unit. Among them, the basic layer information includes a basic layer using a default network flag, and the basic layer using a default network flag indicates that the basic layer uses a default network. The enhancement layer information includes network parameters used to generate the enhancement layer, which can include but not limited to at least one of the following: the number of neural network layers, the inverse convolution layer flag, the number of inverse convolution layers, the quantization step of each inverse convolution layer, the number of channels of each inverse convolution layer, the size of the convolution kernel, the number of filters, the filter size index, the filter coefficient zero flag, the filter coefficient, the activation layer flag, and the activation layer type. It should be noted that the network parameters in the enhancement layer information and the network parameters used by the enhancement layer of the encoding neural network 2 can be different, that is, for each network parameter used by the enhancement layer, a network parameter symmetrical to the network parameter can be generated, and the generation process is not limited, and the symmetrical network parameter is transmitted to the decoding end as the enhancement layer information.

[0375] Case 4, if the basic layer of the coded neural network 2 uses the network layer of the pre-made network structure, and the enhancement layer uses the network layer of the default network structure, the basic layer information and the enhancement layer information are coded in the code stream 0, and the basic layer information and the enhancement layer information constitute the neural network information 2 corresponding to the image feature inverse transformation unit. Among them, the basic layer information includes a basic layer using a pre-made network flag and a basic layer pre-made network index number, the basic layer using a pre-made network flag indicates that the basic layer uses a pre-made network, and the basic layer pre-made network index number indicates the index corresponding to the network layer of the pre-made network structure in the pre-made neural network pool. The enhancement layer information includes an enhancement layer using a default network flag, and the enhancement layer using a default network flag indicates that the enhancement layer uses a default network.

[0376] Case 5, if the basic layer of the coded neural network 2 uses the network layer of the pre-made network structure, and the enhancement layer uses the network layer of the pre-made network structure, the basic layer information and the enhancement layer information are coded in the code stream 0, and the basic layer information and the enhancement layer information constitute the neural network information 2 corresponding to the image feature inverse transformation unit. Among them, the basic layer information can include a basic layer using a pre-made network flag and a basic layer pre-made network index number, the basic layer using a pre-made network flag is used to indicate that the basic layer uses a pre-made network, and the basic layer pre-made network index number is used to indicate the index corresponding to the network layer of the pre-made network structure in the pre-made neural network pool. The enhancement layer information can include an enhancement layer using a pre-made network flag and an enhancement layer pre-made network index number, and the enhancement layer using a pre-made network flag is used to indicate that the enhancement layer uses a pre-made network, and the enhancement layer pre-made network index number is used to indicate the index corresponding to the network layer of the pre-made network structure in the pre-made neural network pool.

[0377] Case 6, if the basic layer of the encoding neural network 2 adopts the network layer of the pre-made network structure, and the enhancement layer adopts the enhancement layer of the variable network structure, the basic layer information and the enhancement layer information are encoded in the code stream 0, and the basic layer information and the enhancement layer information constitute the neural network information 2 corresponding to the image feature inverse transformation unit. Among them, the basic layer information includes a basic layer pre-made network index number and a basic layer pre-made network index number. The basic layer pre-made network index number indicates that the basic layer uses the pre-made network, and the basic layer pre-made network index number indicates that the network layer of the pre-made network structure corresponds to the index in the pre-made neural network pool. The enhancement layer information includes the network parameters used to generate the enhancement layer, which can include but is not limited to at least one of the following: the number of neural network layers, the inverse convolution layer flag, the number of inverse convolution layers, the quantization step of each inverse convolution layer, the number of channels of each inverse convolution layer, the size of the convolution kernel, the number of filters, the filter size index, the filter coefficient zero flag, the filter coefficient, the activation layer flag, and the activation layer type. The network parameters in the enhancement layer information can be different from the network parameters used by the enhancement layer of the encoding neural network 2, that is, for each network parameter used by the enhancement layer, a network parameter symmetrical to the network parameter can be generated, and the generation process is not limited, and the symmetrical network parameter is transmitted to the decoding end as the enhancement layer information.

[0378] Embodiment 18: For embodiments 14 and 15, the image feature inverse transformation unit of the encoding end can perform inverse transformation operation on the image feature reconstruction value F' based on the decoding neural network to obtain the image low-order feature value LF. On this basis, the neural network information 2 corresponding to the image feature inverse transformation unit of the decoding end device can also be determined based on the network structure of the decoding neural network, and the neural network information 2 corresponding to the image feature inverse transformation unit is encoded in the code stream 0.

[0379] In a possible implementation, the image feature inverse transformation unit of the encoding end can include a decoding neural network 2, and the decoding neural network 2 can include a basic layer and an enhancement layer. The network structure of the decoding neural network 2 of the encoding end is the same as the network structure of the decoding neural network 2 of the decoding end, which can be referred to in embodiment 12 and will not be repeated here.

[0380] In a possible implementation, the encoding end can configure the image feature inverse transformation unit with a network layer of a default network structure, which is the same as the network layer of the default network structure of the decoding end. The encoding end can configure the image feature inverse transformation unit with a pre-made neural network pool, which can include at least one network layer of a pre-made network structure, and the pre-made neural network pool is the same as the pre-made neural network pool of the decoding end. The encoding end can dynamically generate a network layer of a variable network structure for the image feature inverse transformation unit based on network parameters. The encoding end can determine the neural network information 2 corresponding to the image feature inverse transformation unit based on the network structure of the decoding neural network 2, and encode the neural network information 2 in the code stream 0.

[0381] For example, if the basic layer of the decoding neural network 2 adopts a network layer of a default network structure, and the enhancement layer adopts a network layer of a default network structure, the basic layer information includes a basic layer uses default network flag, and the basic layer uses default network flag indicates that the basic layer uses a default network. The enhancement layer information includes an enhancement layer uses default network flag, and the enhancement layer uses default network flag indicates that the enhancement layer uses a default network. For another example, if the basic layer of the decoding neural network 2 adopts a network layer of a default network structure, and the enhancement layer adopts a network layer of a pre-made network structure, the basic layer information includes a basic layer uses default network flag, and the basic layer uses default network flag indicates that the basic layer uses a default network. The enhancement layer information includes an enhancement layer uses pre-made network flag and an enhancement layer pre-made network index number, and the enhancement layer uses pre-made network flag indicates that the enhancement layer uses a pre-made network, and the enhancement layer pre-made network index number indicates the index corresponding to the network layer of the pre-made network structure in the pre-made neural network pool. For another example, if the basic layer of the decoding neural network 2 adopts a network layer of a default network structure, and the enhancement layer adopts a network layer of a variable network structure, the basic layer information includes a basic layer uses default network flag, and the basic layer uses default network flag indicates that the basic layer uses a default network. The enhancement layer information includes network parameters used to generate the enhancement layer, and the network parameters in the enhancement layer information can be the same as the network parameters used by the enhancement layer of the decoding neural network 2.

[0382] For example, if the base layer of the decoding neural network 2 adopts the network layer of the pre-prepared network structure, and the enhancement layer adopts the network layer of the default network structure, the base layer information includes a base layer pre-prepared network flag bit and a base layer pre-prepared network index number. The base layer pre-prepared network flag bit indicates that the base layer uses a pre-prepared network, and the base layer pre-prepared network index number indicates the index corresponding to the network layer of the pre-prepared network structure in the pre-prepared neural network pool. The enhancement layer information includes an enhancement layer default network flag bit, which indicates that the enhancement layer uses a default network. For another example, if the base layer of the decoding neural network 2 adopts the network layer of the pre-prepared network structure, and the enhancement layer adopts the network layer of the pre-prepared network structure, the base layer information includes a base layer pre-prepared network flag bit and a base layer pre-prepared network index number. The base layer pre-prepared network flag bit indicates that the base layer uses a pre-prepared network, and the base layer pre-prepared network index number indicates the index corresponding to the network layer of the pre-prepared network structure in the pre-prepared neural network pool. The enhancement layer information includes an enhancement layer pre-prepared network flag bit and an enhancement layer pre-prepared network index number. The enhancement layer pre-prepared network flag bit indicates that the enhancement layer uses a pre-prepared network, and the enhancement layer pre-prepared network index number indicates the index corresponding to the network layer of the pre-prepared network structure in the pre-prepared neural network pool. For another example, if the base layer of the decoding neural network 2 adopts the network layer of the pre-prepared network structure, and the enhancement layer adopts the enhancement layer of the variable network structure, the base layer information includes a base layer pre-prepared network flag bit and a base layer pre-prepared network index number. The base layer pre-prepared network flag bit indicates that the base layer uses a pre-prepared network, and the base layer pre-prepared network index number indicates the index corresponding to the network layer of the pre-prepared network structure in the pre-prepared neural network pool. The enhancement layer information includes network parameters, and the network parameters in the enhancement layer information are the same as the network parameters adopted by the enhancement layer of the decoding neural network 2.

[0383] For embodiments 14 and 15, the coefficient hyperparameter feature transformation unit can perform coefficient hyperparameter feature transformation on the image feature value F based on the encoding neural network to obtain a coefficient hyperparameter feature coefficient value C. On this basis, the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit of the decoding end device can be determined based on the network structure of the encoding neural network, and the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit is encoded in the code stream 0. How the decoding end device generates the decoding neural network 1 corresponding to the coefficient hyperparameter feature generation unit according to the neural network information 1 can be referred to in embodiment 9, which will not be repeated here.

[0384] In a possible implementation, the coefficient hyper-parameter feature transformation unit can include an encoding neural network 1, the encoding neural network 1 can include a basic layer and an enhancement layer, the encoding neural network 1 can be divided into the basic layer and the enhancement layer, the basic layer can include at least one network layer, or the basic layer can not include a network layer, that is, the basic layer is empty. The enhancement layer can include at least one network layer, or the enhancement layer can not include a network layer, that is, the enhancement layer is empty. It should be noted that the network structure of the encoding neural network 1 of the coefficient hyper-parameter feature transformation unit can be symmetrical to the network structure of the decoding neural network 1 of the coefficient hyper-parameter feature generation unit (see embodiment 9), and the network parameters of the encoding neural network 1 of the coefficient hyper-parameter feature transformation unit can be the same as or different from the network parameters of the decoding neural network 1 of the coefficient hyper-parameter feature generation unit. The network structure of the encoding neural network 1 will not be repeated here.

[0385] In a possible implementation, the coefficient hyper-parameter feature transformation unit can be configured with a network layer of a default network structure, and the network parameters related to the network layer of the default network structure are fixed. It should be noted that the network layer of the default network structure configured for the coefficient hyper-parameter feature transformation unit can be a symmetrical structure to the network layer of the default network structure configured for the coefficient hyper-parameter feature generation unit. The coefficient hyper-parameter feature transformation unit can be configured with a prefabricated neural network pool (corresponding to the prefabricated neural network pool of the decoding device), and the prefabricated neural network pool includes at least one network layer of a prefabricated network structure. The network parameters related to the network layer of the prefabricated network structure can be configured according to actual needs. The coefficient hyper-parameter feature transformation unit can be dynamically generated with a network layer of a variable network structure based on the network parameters, and the network parameters related to the network layer of the variable network structure are dynamically generated by the encoding end.

[0386] In a possible implementation, the encoding end can determine the neural network information 1 corresponding to the coefficient hyper-parameter feature generation unit based on the network structure of the encoding neural network 1, and encode the neural network information 1 in the code stream 0. The neural network information 1 can include basic layer information and enhancement layer information. The method of encoding the neural network information 1 is similar to the method of encoding the neural network information 2, which can be referred to in embodiment 17 and will not be repeated here. For example, if the basic layer of the encoding neural network 1 adopts the network layer of the default network structure, and the enhancement layer adopts the network layer of the default network structure, the basic layer information includes a basic layer default network flag, and the basic layer default network flag indicates that the basic layer uses the default network. The enhancement layer information includes an enhancement layer default network flag, and the enhancement layer default network flag indicates that the enhancement layer uses the default network.

[0387] Embodiment 20: For Embodiment 14 and Embodiment 15, the coefficient hyperparameter feature generation unit at the encoding end can perform an inverse transform operation on the coefficient hyperparameter feature coefficient reconstruction value C’ based on the decoding neural network to obtain the coefficient hyperparameter feature value P. On this basis, the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit of the decoding end device can also be determined based on the network structure of the decoding neural network, and the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit is encoded in the code stream 0.

[0388] In a possible implementation, the coefficient hyperparameter feature generation unit at the encoding end can include a decoding neural network 1, which can include a basic layer and an enhancement layer. The network structure of the decoding neural network 1 at the encoding end is the same as that of the decoding neural network 1 at the decoding end. Please refer to Embodiment 9, which will not be repeated here.

[0389] In a possible implementation, the encoding end can configure the coefficient hyperparameter feature generation unit with a network layer of a default network structure, which is the same as the network layer of the default network structure at the decoding end. The encoding end can configure the coefficient hyperparameter feature generation unit with a prefabricated neural network pool, which can include at least one network layer of a prefabricated network structure, and the prefabricated neural network pool is the same as the prefabricated neural network pool at the decoding end. The encoding end can dynamically generate a network layer of a variable network structure for the coefficient hyperparameter feature generation unit based on network parameters. The encoding end can determine the neural network information 1 corresponding to the coefficient hyperparameter feature generation unit based on the network structure of the decoding neural network 1, and encode the neural network information 1 in the code stream 0. The neural network information 1 can include basic layer information and enhancement layer information. The way of encoding the neural network information 1 is similar to that of encoding the neural network information 2, which can be referred to in Embodiment 18, and will not be repeated here. For example, if the basic layer of the decoding neural network 1 (used by the encoding end) adopts a network layer of a default network structure, and the enhancement layer adopts an enhancement layer of a variable network structure, the basic layer information includes a basic layer using a default network flag, and the basic layer using a default network flag indicates that the basic layer uses a default network. The enhancement layer information includes network parameters used to generate the enhancement layer, and the network parameters in the enhancement layer information can be the same as the network parameters used by the enhancement layer of the decoding neural network 1 (i.e., the enhancement layer used by the encoding end).

[0390] For Embodiment 14 and Embodiment 15, the quality enhancement unit at the encoding end can enhance the image low-order feature value LF based on the decoding neural network to obtain a reconstructed image block I corresponding to the current block. On this basis, the neural network information 3 corresponding to the quality enhancement unit of the decoding end device can also be determined based on the network structure of the decoding neural network, and the neural network information 3 corresponding to the quality enhancement unit is encoded in the code stream 0. For example, the quality enhancement unit at the encoding end can include a decoding neural network 3, which can include a basic layer and an enhancement layer. The network structure of the decoding neural network 3 at the encoding end is the same as that of the decoding neural network 3 at the decoding end. For details, please refer to Embodiment 13, which will not be repeated here.

[0391] For example, the encoding end can configure the quality enhancement unit with a default network structure network layer, which is the same as the default network structure network layer at the decoding end. The encoding end can configure the quality enhancement unit with a prefabricated neural network pool, which can include at least one prefabricated network structure network layer, and the prefabricated neural network pool is the same as the prefabricated neural network pool at the decoding end. The encoding end can dynamically generate a variable network structure network layer for the quality enhancement unit based on network parameters. The encoding end determines the neural network information 3 corresponding to the quality enhancement unit based on the network structure of the decoding neural network 3, and encodes the neural network information 3 in the code stream 0. The neural network information 3 can include basic layer information and enhancement layer information. The way of encoding the neural network information 3 is similar to that of encoding the neural network information 2. For details, please refer to Embodiment 18, which will not be repeated here. For example, if the basic layer of the decoding neural network 3 adopts a default network structure network layer and the enhancement layer adopts a variable network structure enhancement layer, the basic layer information includes a basic layer using a default network flag, and the basic layer using a default network flag indicates that the basic layer uses a default network. The enhancement layer information includes network parameters used to generate the enhancement layer, and the network parameters in the enhancement layer information can be the same as the network parameters used by the enhancement layer of the decoding neural network 3.

[0392] In a possible implementation, for the network parameters in Embodiments 1-21, the network parameters can all be fixed-point network parameters. For example, the filter weights in the network parameters can be represented by 4, 8, 16, 32, or 64 bits wide. The feature values of the network output can also be limited to 4, 8, 16, 32, or 64 bits wide.

[0393] For example, the bit width of the network parameters can be limited to 8 bits, and the value size can be limited to [-127, 127]. The bit width of the feature values of the network output can be limited to 8 bits, and the value size can be limited to [-127, 127].

[0394] Embodiment 22: For the related syntax table of the image header, see Table 4, which gives the syntax information related to the image header, i.e., the image-level syntax. In Table 4, u(n) is used to represent an n-bit fixed-length code encoding method.

[0395] Table 4

[0396]

[0397]

[0398] In Table 4, pic_width represents the image width, pic_height represents the image height, pic_format represents the image format, such as RGB 444, YUV444, YUV420, YUV422, etc. bu_width represents the width of the basic block, bu_height represents the height of the basic block, block_width represents the width of the image block, block_height represents the height of the image block, bit_depth represents the image bit depth, pic_qp represents the quantization parameter in the current image, lossless_flag represents the flag indicating whether the current image is lossless encoded, feature_map_max_bit_depth represents the maximum feature map bit depth, which is used to limit the maximum and minimum values of the network input or output feature map.

[0399] For example, each of the above embodiments can be implemented alone, or in combination, for example, each of Embodiments 1-22 can be implemented alone, and at least two of Embodiments 1-22 can be implemented in combination.

[0400] For example, in each of the above embodiments, the content of the encoding end can also be applied to the decoding end, i.e., the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, i.e., the encoding end can be processed in the same way.

[0401] Based on the same application concept as the above method, the present embodiment also proposes a neural network-based image decoding device, which is applied to the decoding end, and includes a memory configured to store video data, and a decoder configured to implement the decoding method in Embodiments 1-22 above, i.e., the processing flow of the decoding end.

[0402] For example, in a possible implementation, the decoder is configured to implement:

[0403] decode the control parameters and image information corresponding to the current block from the code stream;

[0404] obtain the neural network information corresponding to the decoding processing unit from the control parameter, and generate a decoding neural network corresponding to the decoding processing unit based on the neural network information;

[0405] determine the input feature corresponding to the decoding processing unit based on the image information, and process the input feature based on the decoding neural network to obtain the output feature corresponding to the decoding processing unit.

[0406] Based on the same application concept as the above method, the embodiment of the present application also provides an image encoding device based on a neural network. The device is applied to an encoding end. The device comprises a memory configured to store video data; and an encoder configured to implement the encoding method in the above embodiments 1-22, that is, the processing flow of the encoding end.

[0407] For example, in a possible implementation, the encoder is configured to implement:

[0408] determine the input feature corresponding to the encoding processing unit based on the current block, process the input feature based on the encoding neural network corresponding to the encoding processing unit to obtain the output feature corresponding to the encoding processing unit, and determine the image information corresponding to the current block based on the output feature;

[0409] obtain the control parameter corresponding to the current block, the control parameter comprising neural network information corresponding to a decoding processing unit, and the neural network information is used to determine a decoding neural network corresponding to the decoding processing unit;

[0410] encode the image information and the control parameter corresponding to the current block in a bitstream.

[0411] Based on the same application concept as the above method, the decoding end device (which can also be referred to as a video decoder) provided by the embodiment of the present application can be seen from the hardware level. The hardware architecture diagram thereof is specifically shown in FIG. 8. Figure 8A The decoding end device comprises a processor 811 and a machine readable storage medium 812. The machine readable storage medium 812 stores machine executable instructions which can be executed by the processor 811. The processor 811 is used to execute the machine executable instructions to implement the decoding method in the above embodiments 1-22.

[0412] Based on the same application concept as the above method, the encoding end device (which can also be referred to as a video encoder) provided by the embodiment of the present application can be seen from the hardware level. The hardware architecture diagram thereof is specifically shown in FIG. 9. Figure 8BAs shown. Comprise: processor 821 and machine readable storage medium 822, machine readable storage medium 822 has the machine executable instructions that can be executed by processor 821; Processor 821 is used to execute machine executable instructions to realize the encoding method of the above-mentioned embodiments 1-22 of the present application.

[0413] Based on the same application concept as the above method, the embodiments of the present application also provide a machine readable storage medium, wherein the machine readable storage medium stores a plurality of computer instructions, and the computer instructions can realize the method disclosed in the above examples of the present application when executed by a processor, such as the decoding method or the encoding method in the above embodiments.

[0414] Based on the same application concept as the above method, the embodiments of the present application also provide a computer application program, which can realize the decoding method or the encoding method disclosed in the above examples of the present application when executed by a processor.

[0415] Based on the same application concept as the above method, the embodiments of the present application also provide a neural network-based image decoding device, which is applied to a decoding end, and the device comprises: a decoding module, configured to decode control parameters and image information corresponding to a current block from a code stream; an acquisition module, configured to acquire neural network information corresponding to a decoding processing unit from the control parameters, and generate a decoding neural network corresponding to the decoding processing unit based on the neural network information; and a processing module, configured to determine input features corresponding to the decoding processing unit based on the image information, and process the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit.

[0416] For example, if the neural network information includes basic layer information and enhancement layer information, the acquisition module is specifically configured to generate the decoding neural network corresponding to the decoding processing unit based on the neural network information by: determining a basic layer corresponding to the decoding processing unit based on the basic layer information; determining an enhancement layer corresponding to the decoding processing unit based on the enhancement layer information; and generating the decoding neural network corresponding to the decoding processing unit based on the basic layer and the enhancement layer.

[0417] For example, the acquisition module is specifically configured to acquire a basic layer with a default network structure if the basic layer information includes a basic layer uses default network flag, and the basic layer uses default network flag indicates that the basic layer uses a default network, when determining the basic layer corresponding to the decoding processing unit based on the basic layer information.

[0418] Exemplarily, the obtaining module determines the base layer corresponding to the decoding processing unit based on the base layer information, specifically: if the base layer information includes a base layer uses pre-made network flag and a base layer pre-made network index number, and the base layer uses pre-made network flag indicates that the base layer uses a pre-made network, then a base layer of a pre-made network structure corresponding to the base layer pre-made network index number is selected from a pre-made neural network pool.

[0419] Exemplarily, the obtaining module determines the enhancement layer corresponding to the decoding processing unit based on the enhancement layer information, specifically: if the enhancement layer information includes an enhancement layer uses default network flag, and the enhancement layer uses default network flag indicates that the enhancement layer uses a default network, then an enhancement layer of a default network structure is obtained.

[0420] Exemplarily, the obtaining module determines the enhancement layer corresponding to the decoding processing unit based on the enhancement layer information, specifically: if the enhancement layer information includes an enhancement layer uses pre-made network flag and an enhancement layer pre-made network index number, and the enhancement layer uses pre-made network flag indicates that the enhancement layer uses a pre-made network, then an enhancement layer of a pre-made network structure corresponding to the enhancement layer pre-made network index number is selected from a pre-made neural network pool; wherein the pre-made neural network pool includes at least one network layer of a pre-made network structure.

[0421] Exemplarily, the obtaining module determines the enhancement layer corresponding to the decoding processing unit based on the enhancement layer information, specifically: if the enhancement layer information includes network parameters for generating the enhancement layer, then the enhancement layer corresponding to the decoding processing unit is generated based on the network parameters; wherein the network parameters include at least one of the following: number of neural network layers, deconvolution layer flag, number of deconvolution layers, quantization step of each deconvolution layer, number of channels of each deconvolution layer, size of convolution kernel, number of filters, filter size index, filter coefficient zero flag, filter coefficient, activation layer flag, and activation layer type.

[0422] In an example, the image information includes coefficient hyper-parameter feature information and image feature information, and the processing module determines input features corresponding to the decoding processing unit based on the image information, and processes the input features based on the decoding neural network to obtain output features corresponding to the decoding processing unit, and specifically, in the decoding process of performing coefficient hyper-parameter feature generation, the processing module determines coefficient hyper-parameter feature coefficient reconstruction values based on the coefficient hyper-parameter feature information, and performs inverse transform operation on the coefficient hyper-parameter feature coefficient reconstruction values based on the decoding neural network to obtain coefficient hyper-parameter feature values, where the coefficient hyper-parameter feature values are used to decode the image feature information from a bitstream; in the decoding process of performing image feature inverse transform, the processing module determines image feature reconstruction values based on the image feature information, and performs inverse transform operation on the image feature reconstruction values based on the decoding neural network to obtain image low-order feature values, where the image low-order feature values are used to obtain a reconstructed image block corresponding to the current block.

[0423] In an example, the processing module determines coefficient hyper-parameter feature coefficient reconstruction values based on the coefficient hyper-parameter feature information, and specifically, if the control parameter includes first enabling information, and the first enabling information indicates that a first inverse quantization operation is enabled, the processing module performs inverse quantization on the coefficient hyper-parameter feature information to obtain the coefficient hyper-parameter feature coefficient reconstruction values.

[0424] In an example, the processing module determines image feature reconstruction values based on the image feature information, and specifically, if the control parameter includes second enabling information, and the second enabling information indicates that a second inverse quantization operation is enabled, the processing module performs inverse quantization on the image feature information to obtain the image feature reconstruction values.

[0425] In an example, the processing module further determines, if the control parameter includes third enabling information, the third enabling information indicating that a quality enhancement operation is enabled, a reconstructed image block corresponding to the current block by acquiring the image low-order feature values and performing enhancement processing on the image low-order feature values based on the decoding neural network in the decoding process of performing quality enhancement.

[0426] Based on the same application concept as the above method, the embodiment of the present application also proposes a neural network-based image encoding device. The device is applied to an encoding end. The device comprises: a processing module, configured to determine input features corresponding to an encoding processing unit based on a current block, process the input features based on an encoding neural network corresponding to the encoding processing unit, obtain output features corresponding to the encoding processing unit, and determine image information corresponding to the current block based on the output features; an acquisition module, configured to acquire control parameters corresponding to the current block, wherein the control parameters comprise neural network information corresponding to a decoding processing unit, and the neural network information is used to determine a decoding neural network corresponding to the decoding processing unit; and an encoding module, configured to encode the image information corresponding to the current block and the control parameters in a bitstream.

[0427] For example, the neural network information comprises base layer information and enhancement layer information, and the decoding neural network comprises a base layer determined based on the base layer information and an enhancement layer determined based on the enhancement layer information.

[0428] For example, if the base layer information comprises a base layer uses default network flag, and the base layer uses default network flag indicates that the base layer uses a default network, the decoding neural network adopts a base layer with a default network structure.

[0429] For example, if the base layer information comprises a base layer uses pre-made network flag and a base layer pre-made network index number, and the base layer uses pre-made network flag indicates that the base layer uses a pre-made network, the decoding neural network adopts a base layer with a pre-made network structure corresponding to the base layer pre-made network index number selected from a pre-made neural network pool; wherein the pre-made neural network pool comprises at least one network layer with a pre-made network structure.

[0430] For example, if the enhancement layer information comprises an enhancement layer uses default network flag, and the enhancement layer uses default network flag indicates that the enhancement layer uses a default network, the decoding neural network adopts an enhancement layer with a default network structure.

[0431] For example, if the enhancement layer information comprises an enhancement layer uses pre-made network flag and an enhancement layer pre-made network index number, and the enhancement layer uses pre-made network flag indicates that the enhancement layer uses a pre-made network, the decoding neural network adopts an enhancement layer with a pre-made network structure corresponding to the enhancement layer pre-made network index number selected from a pre-made neural network pool; wherein the pre-made neural network pool comprises at least one network layer with a pre-made network structure.

[0432] Exemplarily, if the enhancement layer information comprises network parameters used for generating the enhancement layer, the decoding neural network adopts the enhancement layer generated based on the network parameters; wherein the network parameters comprise at least one of the following: the number of neural network layers, a deconvolution layer flag, the number of deconvolution layers, the quantization step of each deconvolution layer, the number of channels of each deconvolution layer, the size of a convolution kernel, the number of filters, a filter size index, a filter coefficient zero flag, a filter coefficient, an activation layer flag, and an activation layer type.

[0433] Exemplarily, the processing module is further configured to: divide the current image into N mutually non-overlapping image blocks, N being a positive integer; perform boundary padding on each image block to obtain a boundary-padded image block; wherein, when performing boundary padding on each image block, the padding value is independent of the reconstructed pixel values of adjacent image blocks; and generate N current blocks based on the boundary-padded image blocks.

[0434] Exemplarily, the processing module is further configured to: divide the current image into a plurality of basic blocks, and each basic block comprises at least one image block; perform boundary padding on each image block to obtain a boundary-padded image block; wherein, when performing boundary padding on each image block, the padding value of the image block is independent of the reconstructed pixel values of other image blocks in the same basic block, and is allowed to be dependent on the reconstructed pixel values of image blocks in different basic blocks; and generate a plurality of current blocks based on the boundary-padded image blocks.

[0435] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. The present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. The embodiments of the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0436] The above only describes the embodiments of the present application and is not intended to limit the present application. Various modifications and changes can be made to the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of the claims of the present application.

Claims

1. An image decoding method characterized by, The method comprises: decoding control parameters corresponding to the current block from the code stream; obtaining enhancement layer information and basic layer information from the control parameters, wherein the enhancement layer information comprises network parameters used for configuring an enhancement layer network, and the enhancement layer network is configured based on the network parameters; determining input features corresponding to the enhancement layer network, processing the input features corresponding to the enhancement layer network based on the enhancement layer network to obtain output features corresponding to the enhancement layer; determining input features corresponding to a basic layer network corresponding to the basic layer information, processing the input features corresponding to the basic layer network based on the basic layer network to obtain output features corresponding to the basic layer network.

2. The method of claim 1, wherein, The network parameters comprise at least one of the following: the number of neural network layers, the number of filters, the filter size index, the filter coefficient zero flag, and the filter coefficient.

3. The method of claim 1, wherein, If the enhancement layer network is a quality enhancement network, the determination of the input features corresponding to the enhancement layer network comprises: decoding the code stream of the current block to obtain image feature information corresponding to the current block; determining an image feature reconstruction value corresponding to the current block based on the image feature information; processing the image feature reconstruction value to obtain an image low-order feature value corresponding to the current block.

4. The method of claim 3, wherein, The processing of the input features corresponding to the enhancement layer network based on the enhancement layer network to obtain the output features corresponding to the enhancement layer comprises: if the input features comprise the image low-order feature value corresponding to the current block, performing enhancement processing on the image low-order feature value corresponding to the current block based on the quality enhancement network to obtain a reconstructed image block corresponding to the current block.

5. The method of claim 4, wherein: the enhancement processing on the image low-order feature value corresponding to the current block based on the quality enhancement network to obtain the reconstructed image block corresponding to the current block comprises: if the control parameters further comprise third enabling information, and the third enabling information indicates that the quality enhancement operation is enabled, performing enhancement processing on the image low-order feature value corresponding to the current block based on the quality enhancement network to obtain the reconstructed image block corresponding to the current block.

6. The method of any one of claims 1-5, wherein: the decoding of the control parameters corresponding to the current block from the code stream comprises: decoding the code stream corresponding to the current block through a control parameter decoding network to obtain the control parameters corresponding to the current block, wherein the control parameters comprise control parameters of a quality enhancement network.

7. An image coding method characterized by, The method comprises: determining input features corresponding to an encoding processing unit based on the current block, processing the input features based on an encoding neural network corresponding to the encoding processing unit to obtain output features corresponding to the encoding processing unit, and determining image information corresponding to the current block based on the output features. obtaining control parameters corresponding to the current block, the control parameters comprising neural network information corresponding to a decoding processing unit, the neural network information comprising enhancement layer information and base layer information, the base layer information being used to determine a base layer network, and the enhancement layer information being used to determine an enhancement layer network; wherein the enhancement layer information comprises network parameters used to configure the enhancement layer network; encoding the control parameters corresponding to the current block in a bitstream.

8. An image decoding apparatus characterized by comprising: The apparatus comprises: a decoding module configured to decode control parameters corresponding to a current block from a bitstream; an obtaining module configured to obtain enhancement layer information and base layer information from the control parameters, the enhancement layer information comprising network parameters used to configure an enhancement layer network, and to configure the enhancement layer network based on the network parameters; a processing module configured to determine input features corresponding to the enhancement layer network, to process the input features corresponding to the enhancement layer network based on the enhancement layer network, and to obtain output features corresponding to the enhancement layer; and to determine input features corresponding to a base layer network corresponding to the base layer information, to process the input features corresponding to the base layer network based on the base layer network, and to obtain output features corresponding to the base layer network.

9. An image coding apparatus characterized by comprising: The apparatus comprises: a processing module configured to determine input features corresponding to a coding processing unit based on a current block, to process the input features based on a coding neural network corresponding to the coding processing unit, to obtain output features corresponding to the coding processing unit, and to determine image information corresponding to the current block based on the output features; an obtaining module configured to obtain control parameters corresponding to the current block, the control parameters comprising neural network information corresponding to a decoding processing unit, the neural network information comprising enhancement layer information and base layer information, the base layer information being used to determine a base layer network, and the enhancement layer information being used to determine an enhancement layer network; wherein the enhancement layer information comprises network parameters used to configure the enhancement layer network; an encoding module configured to encode the control parameters corresponding to the current block in a bitstream.

10. An image decoding apparatus characterized by comprising: comprise: a processor and a machine readable storage medium storing machine executable instructions executable by the processor; the processor is configured to execute the machine executable instructions to implement the method in any one of claims 1-6.

11. An image coding apparatus characterized by comprising: comprise: a processor and a machine readable storage medium storing machine executable instructions executable by the processor; the processor is configured to execute the machine executable instructions to implement the method in claim 7.

12. A machine-readable storage medium, characterized in that, The machine readable storage medium stores machine executable instructions executable by a processor; wherein the processor is configured to execute the machine executable instructions to implement the method in any one of claims 1-6, or to implement the method in claim 7.

13. A computer program product, characterised in that, The computer program product comprises a computer program, which, when executed by a processor, implements the method in any one of claims 1-6, or, when executed by a processor, implements the method in claim 7.

Images