Encoding device, program, and model generation method

Abstract

To improve the efficiency of quantization processing using a neural network model.SOLUTION: An encoder comprises a quantization unit 122 which performs quantization processing to a two-dimensional conversion coefficient obtained by performing conversion processing to a residual signal expressing a difference between a block of an original image and a block of a prediction image. The quantization unit 122 generates one-dimensional post-scale conversion coefficient and one-dimensional post-quantization conversion coefficient based on the two-dimensional conversion coefficient, and adjusts the one-dimensional post-quantization conversion coefficient according to the output obtained by inputting the one-dimensional post-scale conversion coefficient and one-dimensional post-quantization conversion coefficient to one-dimensional neural network model.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an encoding device, a program, and a model generation method. [Background technology]

[0002] In general, a coding device for coding a moving image includes a quantization unit that performs a quantization process on two-dimensional transform coefficients obtained by performing a transform process on a residual signal that represents the difference between a block of an original image and a block of a predicted image, and an entropy coding unit that converts the two-dimensional quantized transform coefficients obtained by the quantization unit into one-dimensional coefficients according to a scan order and then performs entropy coding. The quantization unit uses a quantization step (Q step ) to derive scaled transform coefficients expressed as real numbers, and then the scaled transform coefficients are converted to integers by a rounding process to derive quantized transform coefficients.

[0003] In the High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC) formats, a technique called Rate Distortion Quantized Optimization (RDOQ) is used when a quantization unit derives (i.e., quantizes) a post-quantization transform coefficient. In RDOQ, the optimal quantization process is determined by minimizing the Rate Distortion (RD) cost shown in Equation (1).

[0004]

number

[0005] Here, λ is a constant determined by a quantization parameter such as QP, x is a transform coefficient before quantization (scaled transform coefficient), and q is a transform coefficient after quantization (quantized transform coefficient). D(q,x) is the quantization distortion, which is generally expressed as the residual sum of squares of x and q. R(q) is the bit rate when q is entropy-encoded using CABAC, which is adopted in the HEVC and VVC formats.

[0006] Non-Patent Document 1 describes a technique for improving coding efficiency, in which a neural network is trained using the quantization results obtained by the greedy method as training data to generate a neural network model, and a quantization unit determines the quantization process using the neural network model.

[0007] In the technology described in Non-Patent Document 1 (hereinafter referred to as "prior research"), a quantization unit divides two-dimensional transform coefficients by a predetermined value to derive two-dimensional scaled transform coefficients, and converts the two-dimensional scaled transform coefficients to integers by rounding to derive two-dimensional quantized transform coefficients. The quantization unit then adjusts the two-dimensional quantized transform coefficients using a quantization adjustment value obtained by inputting the two-dimensional scaled transform coefficients and the two-dimensional quantized transform coefficients into a two-dimensional neural network model. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] D. Kianfar, A. Wiggers, A. Said, R. Pourreza, T. Cohen, Parallelized Rate-Distortion Optimized Quantization using Deep Learning, IEEE MMSP 2020 Summary of the Invention [Problem to be solved by the invention]

[0009] As described above, the entropy coding unit converts the two-dimensional quantized transform coefficients obtained by the quantization unit into one-dimensional ones according to the scan order, and then performs entropy coding such as CABAC. The MPS (Most Probable Symbol) occurrence probability of a binary symbol in CABAC is estimated based on the values ​​of the binary symbols that have appeared up to that point. Therefore, it is considered that the coding efficiency can be improved by considering the scan order in entropy coding in the quantization process using a neural network model.

[0010] However, in the previous study, because a two-dimensional neural network model was used, the scan order in entropy coding was not taken into consideration, and therefore, in the previous study, features of coefficients that are not adjacent in the scan order in entropy coding were also extracted, which caused a problem that erroneous quantization adjustment values ​​could be derived due to features with little correlation.

[0011] Therefore, an object of the present invention is to provide an encoding device, a program, and a model generation method that can improve the efficiency of quantization processing using a neural network model. [Means for solving the problem]

[0012] The coding device according to a first aspect includes a quantization unit that performs a quantization process on two-dimensional transform coefficients obtained by performing a transform process on a residual signal representing a difference between a block of an original image and a block of a predicted image. The quantization unit generates one-dimensional scaled transform coefficients and one-dimensional quantized transform coefficients based on the two-dimensional transform coefficients, and converts the one-dimensional scaled transform coefficients and the one-dimensional quantized transform coefficients into a quantized one-dimensional transform coefficient. Based on The one-dimensional quantized transform coefficients are adjusted.

[0013] A program according to the second aspect causes a computer to function as the encoding device according to the first aspect.

[0014] A model generation method according to a third aspect is a method for generating a trained model for encoding. The model generation method includes a step of acquiring sets of one-dimensional scaled transform coefficients, one-dimensional quantized transform coefficients, and optimal one-dimensional quantized transform coefficients derived using a full search or a greedy method or a quantization adjustment value based on the optimal one-dimensional quantized transform coefficients as training data, and a step of generating the trained model by machine learning using the training data including a plurality of the sets, the trained model having the one-dimensional scaled transform coefficients and the one-dimensional quantized transform coefficients as inputs and the optimal one-dimensional quantized transform coefficients or the quantization adjustment value as outputs. Le and generating the signal. Effect of the Invention

[0015] According to the present invention, it is possible to provide an encoding device, a program, and a model generation method that can improve the efficiency of quantization processing using a neural network model. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an encoding device according to an embodiment. [Diagram 2] FIG. 11 is a diagram illustrating an example of types of scan orders. [Diagram 3] FIG. 4 is a diagram illustrating a configuration of a quantization unit according to the embodiment. [Figure 4] 11 is a diagram for explaining the operation of a quantization unit according to the embodiment. FIG. [Diagram 5] FIG. 13 is a diagram illustrating an example of RDOQ processing. [Figure 6] FIG. 2 is a diagram illustrating an example of a one-dimensional neural network model according to the embodiment. [Figure 7] FIG. 2 illustrates a model generation method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following embodiments will be described with reference to the accompanying drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0018] (encoding device)

[0019] (1) Configuration of the encoding device First, the configuration of the encoding device according to this embodiment will be described. The encoding device according to this embodiment encodes moving images as represented by MPEG. Fig. 1 is a diagram showing the configuration of an encoding device 1 according to this embodiment. The encoding device 1 is a device that encodes an image in units of blocks obtained by dividing the image.

[0020] As shown in FIG. 1, the encoding device 1 includes a block division unit 100, a subtraction unit 110, a transformation and quantization unit 120, an entropy encoding unit 130, an inverse quantization and inverse transform unit 140, a synthesis unit 150, a loop filter 160, a memory 170, and a prediction unit 180.

[0021] The block division unit 100 divides an input image of a frame (or picture) constituting a moving image into a plurality of image blocks, and outputs the image blocks obtained by division to the subtraction unit 110. The size of the image block is, for example, 32×32 pixels, 16×16 pixels, 8×8 pixels, or 4×4 pixels. The shape of the image block is not limited to a square, but may be a rectangle (non-square). The image block is a unit (block to be coded) for coding by the coding device 1, and is a unit (block to be coded) for decoding by the decoding device. Such an image block is sometimes called a CU (Coding Unit).

[0022] The subtraction unit 110 calculates a prediction residual representing a difference (error) between a current block to be coded output by the block division unit 100 and a prediction block obtained by predicting the current block to be coded by the prediction unit 180. The subtraction unit 110 calculates the prediction residual by subtracting each pixel value of the prediction block from each pixel value of the block, and outputs the calculated prediction residual to the transformation / quantization unit 120.

[0023] The transform / quantization unit 120 performs transform processing and quantization processing on a block-by-block basis. The transform / quantization unit 120 includes a transform unit 121 and a quantization unit 122.

[0024] The transform unit 121 performs a transform process on the prediction residual output by the subtraction unit 110 to calculate a transform coefficient for each frequency component, and outputs the calculated transform coefficient to the quantization unit 122. The transform process (transformation) refers to a process of transforming a pixel domain signal into a frequency domain signal, such as a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loeb transform (KLT), or a transform obtained by converting these into integers. The transform process may also include a transform skip that adjusts the pixel domain signal by scaling or the like without transforming the pixel domain signal into a frequency domain signal.

[0025] The quantization unit 122 performs a quantization process on the transform coefficients output by the transform unit 121, and outputs the quantized transform coefficients to the entropy coding unit 130 and the inverse quantization / inverse transform unit 140. Specifically, the quantization unit 122 performs a quantization process on two-dimensional transform coefficients obtained by performing a transform process on a residual signal representing the difference between a block of an original image and a block of a predicted image, and outputs the two-dimensional quantized transform coefficients.

[0026] In the quantization process, a neural network model is used as a pre-generated trained model. The quantization unit 122 according to the present embodiment generates one-dimensional scaled transform coefficients and one-dimensional quantized transform coefficients based on two-dimensional transform coefficients, and adjusts the one-dimensional quantized transform coefficients according to an output obtained by inputting the one-dimensional scaled transform coefficients and the one-dimensional quantized transform coefficients to a one-dimensional neural network model. The quantization unit 122 will be described in detail later.

[0027] The entropy coding unit 130 performs entropy coding on the transform coefficients output by the quantization unit 122, compresses the data, generates a coded stream (bit stream), and outputs the coded stream to the outside of the coding device 1. Specifically, the entropy coding unit 130 converts the two-dimensional quantized transform coefficients obtained by the quantization unit 122 into one-dimensional ones according to the scan order, and then performs entropy coding. For the entropy coding, Huffman codes, CABAC (Context-based Adaptive Binary Arithmetic Coding), etc. can be used.

[0028] The entropy coding unit 130 may use only one type of scan order, or may select one of multiple types of scan orders according to an intra prediction mode or a transform skip. FIG. 2 is a diagram showing an example of the types of scan orders. In the following, it is assumed that the block to be coded has a size of 8×8, and the entropy coding unit 130 performs scanning in units of 4×4 coefficient groups (CGs). The scan orders include three types: diagonal shown in FIG. 2(a), horizontal shown in FIG. 2(b), and vertical shown in FIG. 2(c). The entropy coding unit 130 converts (i.e., serializes) two-dimensional quantized transform coefficients into one dimension according to the scan order selected according to the intra prediction mode or the transform skip, and then performs entropy coding such as CABAC. By such serialization, in the case of 8×8 transform coefficients, they are converted into 64×1 transform coefficients. When the entropy coding unit 130 selects one scan order from multiple types of scan orders, it may output information indicating the selected scan order to the quantization unit 122.

[0029] CABAC is an entropy coding method that uses binary arithmetic coding and context adaptive processing that estimates the probability of occurrence of a binary symbol based on the state (context) of surrounding parameters. A number line is divided into intervals based on the value of the input binary symbol and the occurrence probability of MPS (Most Probable Symbol: a symbol that has a high probability of appearing between 0 and 1), and the binary representation of the final interval is used as the codeword of the input binary symbol sequence. The MPS occurrence probability of a binary symbol is estimated based on the values ​​of binary symbols that have appeared up to that point. In other words, based on a predetermined probability estimation model, the MPS occurrence probability of a binary symbol that appears subsequently is updated by controlling to increase the MPS occurrence probability if the binary symbol is an MPS, and decrease the MPS occurrence probability if it is not an MPS.

[0030] In addition, the entropy coding unit 130 obtains control information such as the size, shape, etc. of each block to be coded from the block division unit 100, obtains control information related to the quantization process from the quantization unit 122, and obtains control information related to prediction (e.g., prediction mode and motion vector information) from the prediction unit 180, and also codes this control information.

[0031] The inverse quantization and inverse transform unit 140 performs inverse quantization processing and inverse transform processing on a block-by-block basis. The inverse quantization and inverse transform unit 140 includes an inverse quantization unit 141 and an inverse transform unit 142.

[0032] The inverse quantization unit 141 performs an inverse quantization process corresponding to the quantization process performed by the quantization unit 122. Specifically, the inverse quantization unit 141 restores the transform coefficients by performing an inverse quantization process on the quantized transform coefficients output by the quantization unit 122, and outputs the restored transform coefficients to the inverse transform unit 142.

[0033] The inverse transform unit 142 performs an inverse transform process corresponding to the transform process performed by the transform unit 121. For example, when the transform unit 121 performs a DCT, the inverse transform unit 142 performs an inverse DCT. The inverse transform unit 142 performs an inverse transform process on the transform coefficients output by the inverse quantization unit 141 to restore the prediction residual, and outputs the restored prediction residual, which is the restored prediction residual, to the synthesis unit 150.

[0034] The synthesis unit 150 synthesizes, on a pixel-by-pixel basis, the reconstructed prediction residual output by the inverse transform unit 142 with the predicted block output by the prediction unit 180. The synthesis unit 150 adds each pixel value of the reconstructed prediction residual to each pixel value of the predicted block to decode (reconstruct) the block to be coded, and outputs the decoded block to the loop filter 160.

[0035] The loop filter 160 performs filtering such as deblocking filtering and SAO (sample adaptive offset) filtering, and outputs the decoded blocks after filtering to the memory 170 .

[0036] The memory 170 accumulates the filtered decoded blocks output by the loop filter 160 as decoded images on a frame-by-frame basis. The memory 170 outputs the stored decoded images to the prediction unit 180.

[0037] The prediction unit 180 performs prediction processing on a block-by-block basis to generate a prediction block corresponding to the current block to be coded, and outputs the generated prediction block to the subtraction unit 110 and the synthesis unit 150. The prediction unit 180 includes an inter prediction unit 181, an intra prediction unit 182, and a switching unit 183.

[0038] The inter prediction unit 181 uses the decoded image stored in the memory 170 as a reference image, calculates a motion vector by a method such as block matching, predicts a block to be coded, generates an inter prediction block, and outputs the generated inter prediction block to the switching unit 183. The inter prediction unit 181 selects an optimal inter prediction method from inter prediction using multiple reference images (typically, bi-prediction) and inter prediction using one reference image (unidirectional prediction), and performs inter prediction using the selected inter prediction method.

[0039] The intra prediction unit 182 selects an optimal intra prediction mode to be applied to the encoding target block from among a plurality of intra prediction modes, and predicts the encoding target block using the selected intra prediction mode. The intra prediction unit 182 generates an intra prediction block by referring to decoded pixel values ​​adjacent to the encoding target block in the decoded image stored in the memory 170, and outputs the generated intra prediction block to the switching unit 183.

[0040] The switching unit 183 switches between the inter predicted block output by the inter prediction unit 181 and the intra predicted block output by the intra prediction unit 182, and outputs either the predicted block to the subtraction unit 110 and the synthesis unit 150.

[0041] (2) Quantization section Next, the quantization unit 122 according to this embodiment will be described. Fig. 3 is a diagram showing the configuration of the quantization unit 122 according to this embodiment. Fig. 4 is a diagram for explaining the operation of the quantization unit 122 according to this embodiment.

[0042] 3, the quantization unit 122 includes a derivation unit 122a, a dimension conversion unit 122b, a code acquisition unit 122c, a code removal unit 122d, an adjustment value acquisition unit 122e, an adjustment value addition unit 122f, and a dimension inverse conversion unit 122g. In this embodiment, the code acquisition unit 122c, the code removal unit 122d, the adjustment value acquisition unit 122e, and the adjustment value addition unit 122f configure an adjustment unit that adjusts the quantized transformation coefficients.

[0043] The derivation unit 122a derives two-dimensional scaled transform coefficients by dividing the two-dimensional transform coefficients output by the above-mentioned transform unit 121 by a predetermined value, converts the two-dimensional scaled transform coefficients to integers by rounding, derives two-dimensional quantized transform coefficients, and outputs the two-dimensional scaled transform coefficients and the two-dimensional quantized transform coefficients to the dimension transform unit 122b. Note that in Fig. 3, for convenience, the scaled transform coefficients are denoted as STC (Scaled Transform Coefficient), and the quantized transform coefficients are denoted as QTC (Quantized Transform Coefficient).

[0044] For example, the derivation unit 122a calculates a quantization step (QP) calculated from the quantization parameter QP for the transform coefficients that the transform unit 121 has performed transform processing such as DCT or DST on. step ) and quantized transform coefficients obtained by converting the scaled transform coefficients to integers by rounding are derived. The quantized transform coefficients may be subjected to conversion processing using a floor function or a ceiling function, or the results of quantization processing using a method such as HM or VTM may be used. Note that HM is a test model of the HEVC method, and VTM is a test model of the HEVC method. Details of the process of deriving the quantized transform coefficients from the scaled transform coefficients (i.e., quantization) will be described later.

[0045] The dimension conversion unit 122b converts (serializes) each of the two-dimensional scaled transform coefficients and the two-dimensional quantized transform coefficients output by the derivation unit 122a into one dimension according to the scan order, outputs the one-dimensional scaled transform coefficients to the code removal unit 122d, and outputs the one-dimensional quantized transform coefficients to the code acquisition unit 122c and the code removal unit 122d. Here, when the scan order used by the entropy coding unit 130 is fixed to one, the dimension conversion unit 122b may perform serialization according to the one scan order. On the other hand, when the entropy coding unit 130 selects one scan order from multiple scan orders and performs serialization, the dimension conversion unit 122b performs serialization according to the selected scan order.

[0046] The sign obtaining unit 122c obtains the positive / negative sign of each transform coefficient constituting the one-dimensional quantized transform coefficient output by the dimension transform unit 122b, and outputs each obtained positive / negative sign to the adjustment value adding unit 122f.

[0047] The sign removal unit 122d removes the positive and negative signs of each transform coefficient constituting the one-dimensional scaled transform coefficient and the positive and negative signs of each transform coefficient constituting the one-dimensional quantized transform coefficient, and outputs the one-dimensional scaled transform coefficient and the one-dimensional quantized transform coefficient from which the positive and negative signs have been removed to the adjustment value acquisition unit 122e. This enables processing of the neural network model that is not dependent on the positive and negative signs. Note that the ones from which the positive and negative signs have been removed are also used in learning when generating the neural network model.

[0048] The adjustment value acquisition unit 122e acquires a quantization adjustment value (specifically, an adjustment value for each transform coefficient) from the one-dimensional scaled transform coefficient and the one-dimensional quantized transform coefficient output by the code removal unit 122d using a one-dimensional neural network model, and outputs the acquired quantization adjustment value to the adjustment value addition unit 122f. The one-dimensional neural network model is a trained model generated by machine learning using a plurality of sets of one-dimensional scaled transform coefficients, one-dimensional quantized transform coefficients, and optimal one-dimensional quantized transform coefficients (or quantization adjustment values ​​based on the optimal one-dimensional quantized transform coefficients) derived using a full search or a greedy method as teacher data. Therefore, the one-dimensional neural network model receives the one-dimensional scaled transform coefficients and one-dimensional quantized transform coefficients output by the code removal unit 122d as inputs, and outputs optimal one-dimensional quantized transform coefficients (or quantization adjustment values ​​based on the optimal one-dimensional quantized transform coefficients). When the one-dimensional neural network model is configured to output optimal one-dimensional quantized transform coefficients, the adjustment value acquisition unit 122e may acquire a quantization adjustment value for each transform coefficient by calculating the difference between the optimal one-dimensional quantized transform coefficient and the one-dimensional quantized transform coefficient output by the code removal unit 122d for each transform coefficient. Note that the quantization adjustment value may be two types of adjustment values, "-1" and "±0", for each transform coefficient, but three or more types of adjustment values ​​may be used.

[0049] When there are multiple candidates for the scan order used by the entropy coding unit 130, a separate one-dimensional neural network model may be generated for each scan order. For example, a one-dimensional neural network model corresponding to the diagonal shown in FIG. 2(a), a one-dimensional neural network model corresponding to the horizontal shown in FIG. 2(b), and a one-dimensional neural network model corresponding to the vertical shown in FIG. 2(c) may be generated. In this case, the adjustment value acquisition unit 122e may specify a one-dimensional neural network model corresponding to the scan order selected by the entropy coding unit 130 from among the multiple one-dimensional neural network models corresponding to the multiple scan orders. Then, the adjustment value acquisition unit 122e may acquire a quantization adjustment value using the specified one-dimensional neural network model.

[0050] The adjustment value addition unit 122f adds the quantization adjustment value output by the adjustment value acquisition unit 122e to the one-dimensional quantized transform coefficient from which the positive and negative signs have been removed, and assigns the positive and negative signs output by the sign acquisition unit 122c to the one-dimensional quantized transform coefficient, and outputs the optimized one-dimensional quantized transform coefficient to the dimension inverse transformation unit 122g.

[0051] The dimension inverse transform unit 122g performs an inverse process of the dimension transform process performed by the dimension transform unit 122b on the optimized one-dimensional quantized transform coefficients output by the adjustment value addition unit 122f. That is, the dimension inverse transform unit 122g inversely transforms the optimized one-dimensional quantized transform coefficients into two dimensions according to the scan order, and outputs the optimized two-dimensional quantized transform coefficients.

[0052] In this way, the quantization unit 122 according to this embodiment generates one-dimensional scaled transform coefficients and one-dimensional quantized transform coefficients according to the scan order, and adjusts the one-dimensional quantized transform coefficients according to the output obtained by inputting the generated one-dimensional scaled transform coefficients and one-dimensional quantized transform coefficients into a one-dimensional neural network model.

[0053] As described above, the entropy coding unit 130 converts the two-dimensional quantized transform coefficients obtained by the quantization unit 122 into one-dimensional ones according to the scan order, and then performs entropy coding such as CABAC. The MPS occurrence probability of a binary symbol in CABAC is estimated based on the values ​​of the binary symbols that have appeared up to that point.

[0054] In previous studies using two-dimensional neural network models, the scan order in entropy coding is not taken into account. As shown in Figure 4(a), the two-dimensional neural network model extracts features of coefficients that are not adjacent in the scan order in entropy coding. This results in the problem that features with little correlation can lead to erroneous predictions (i.e., the derivation of incorrect quantization adjustment values).

[0055] In contrast, in this embodiment, as shown in Fig. 4(b), the quantization adjustment value is derived by inputting the one-dimensional scaled transform coefficients and the one-dimensional quantized transform coefficients into a one-dimensional neural network model while taking into account the scan order in entropy coding. The one-dimensional neural network model can extract the characteristics of the coefficients adjacent to each other in the scan order in entropy coding, and it becomes possible to derive an optimal quantization adjustment value. Therefore, in the quantization process using the neural network model, the coding efficiency can be improved by taking into account the scan order in entropy coding.

[0056] (3) An example of a process for derivation of quantized transform coefficients in the derivation unit Next, an example of a process (so-called RDOQ process) of deriving quantized transform coefficients from scaled transform coefficients in the derivation unit 122a according to this embodiment will be described. Fig. 5 is a diagram showing an example of the RDOQ process.

[0057] As shown in FIG. 5(a), first, the derivation unit 122a performs scalar quantization (SQ) processing on the two-dimensional scaled transform coefficients, and rounds each of the two-dimensional scaled transform coefficients.

[0058] Secondly, the derivation unit 122a performs a level estimation (LE) process to calculate the RD cost when each transform coefficient is set to "-1" or "±0," and estimates the level at which the RD cost is more optimal.

[0059] Thirdly, the derivation unit 122a performs an AZ (All-zero CG) process to determine whether or not to set all of the transform coefficients in each CG shown in FIG. 5(b) to 0, based on the RD cost.

[0060] Fourth, the derivation unit 122a performs a last significant coefficient process (LAST process) to estimate the optimal position of the last non-zero transform coefficient based on the RD cost.

[0061] Fifth, the derivation unit 122a performs SBH (Sign Bit Hiding) processing to implicitly determine whether the final transform coefficient is positive or negative depending on whether the sum of the transforms is an even number or an odd number.

[0062] (4) An example of a one-dimensional neural network model Next, an example of a one-dimensional neural network model according to this embodiment will be described below. Fig. 6 is a diagram showing an example of a one-dimensional neural network model.

[0063] The one-dimensional neural network model shown in FIG. 6 takes one-dimensional scaled transform coefficients (real numbers) with the positive and negative signs removed and one-dimensional quantized transform coefficients (integers) as input data, and outputs adjustment values ​​for the one-dimensional quantized transform coefficients.

[0064] The one-dimensional neural network model is configured to include multiple one-dimensional hidden layers. For the 8×8 transform coefficients in Figure 6, a three-layer 3×1 one-dimensional convolutional neural network model is used as an example.

[0065] First, the input data is normalized so that the transform coefficients have a mean of 0 and a variance of 1. The normalized transform coefficients are input to a 3×1 convolution layer (3×1 Conv1D).

[0066] Next, the convolution layer performs zero padding so that the output of one convolution layer is always kept at 64 × 1. Dropout and batch normalization (Dropout / BatchNorm) may be applied after the convolution. After that, the output of one layer is passed through an activation function such as the ReLU function.

[0067] The output of the final layer is passed through a 1×1 one-dimensional convolution (1×1 Conv2D), and finally a probability representing a 64×k-dimensional adjustment value for each k types of adjustment value is output.

[0068] (5) Summary of embodiments As described above, the quantization unit 122 generates one-dimensional scaled transform coefficients and one-dimensional quantized transform coefficients based on two-dimensional transform coefficients, and adjusts the one-dimensional quantized transform coefficients according to an output obtained by inputting the one-dimensional scaled transform coefficients and the one-dimensional quantized transform coefficients to a one-dimensional neural network model. By extracting features only from adjacent coefficients using the one-dimensional neural network model, it becomes possible to perform highly accurate prediction, and the coding efficiency can be improved.

[0069] (Model generation method) Next, a model generation method according to this embodiment will be described. The model generation method according to this embodiment is a method for generating the above-mentioned one-dimensional neural network model as a trained model for encoding. This model generation method may be executed by a computer different from the encoding device 1, or may be executed by the encoding device 1. Hereinafter, an apparatus that executes the model generation method is called a model generation method apparatus. FIG. 7 is a diagram showing the model generation method according to this embodiment.

[0070] As shown in FIG. 7, in step S1, the model generating device divides two-dimensional transform coefficients by a predetermined value to derive two-dimensional scaled transform coefficients, and converts the two-dimensional scaled transform coefficients into integers by rounding to derive two-dimensional quantized transform coefficients.

[0071] In step S2, the model generating device converts (serializes) the two-dimensional scaled transform coefficients and the two-dimensional quantized transform coefficients derived in step S1 into one dimension according to the scan order, thereby obtaining one-dimensional scaled transform coefficients and one-dimensional quantized transform coefficients. The model generating device also removes positive and negative signs from the one-dimensional scaled transform coefficients and the one-dimensional quantized transform coefficients.

[0072] In step S3, the model generating device acquires, as training data, a set of the one-dimensional scaled transform coefficient and the one-dimensional quantized transform coefficient acquired in step S2, and an optimal one-dimensional quantized transform coefficient (or a quantization adjustment value based on the optimal one-dimensional quantized transform coefficient) derived using a full search or a greedy method.

[0073] In step S4, the model generation device generates a one-dimensional neural network model, which is a trained model that takes one-dimensional scaled transform coefficients and one-dimensional quantized transform coefficients as input and outputs optimal one-dimensional quantized transform coefficients (or quantization adjustment values) through machine learning using training data including multiple sets acquired in step S3.

[0074] The model generating device may execute the model generating method shown in FIG. 7 for each of a plurality of predetermined scan orders to generate an individual one-dimensional neural network model for each of the plurality of scan orders.

[0075] (Other embodiments) When there are multiple candidates for the type of conversion process used by the conversion unit 121, a separate one-dimensional neural network model may be generated for each type of conversion process. For example, a one-dimensional neural network model corresponding to DCT-2 and a one-dimensional neural network model corresponding to DST-7 may be generated. In this case, the adjustment value acquisition unit 122e may specify a one-dimensional neural network model corresponding to the type of conversion process selected by the conversion unit 121 from among multiple one-dimensional neural network models corresponding to multiple types of conversion processes, and acquire (derive) a quantization adjustment value using the specified one-dimensional neural network model.

[0076] The two-dimensional transform coefficients output by the transform unit 121 may have a block size selected from a plurality of predetermined block sizes. That is, there may be a plurality of block size candidates. The block size may be selected by the block division unit 100. In this case, a one-dimensional neural network model may be generated for each block size. For example, a one-dimensional neural network model corresponding to a block size of 4×4 and a one-dimensional neural network model corresponding to a block size of 8×8 may be generated. In this case, the adjustment value acquisition unit 122e may specify a one-dimensional neural network model corresponding to the block size selected by the block division unit 100 from among a plurality of one-dimensional neural network models corresponding to a plurality of block sizes, and may acquire (derive) a quantization adjustment value using the specified one-dimensional neural network model.

[0077] A program may be provided that causes a computer to execute each process performed by the encoding device 1. A program may be provided that causes a computer to execute each step of the model generation method. The program may be recorded in a computer-readable medium. The computer-readable medium can be used to install the program in a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM. Furthermore, the circuits that execute each process performed by the encoding device 1 may be integrated, and the encoding device 1 may be configured as a semiconductor integrated circuit (chip set, SoC).

[0078] The above describes the embodiments in detail with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes, etc. are possible without departing from the spirit of the invention. [Explanation of symbols]

[0079] 1: Encoding device 100: Block division section 110: Subtraction section 120: Transformation and quantization section 121: Conversion section 122: Quantization section 122a: Derivation part 122b: Dimensional transformation section 122c: Code acquisition part 122d: Code removal section 122e: Adjustment value acquisition unit 122f: Adjustment value addition unit 122g: Dimensional inverse transformation section 130: Entropy coding unit 140: Inverse quantization and inverse transformation unit 141: Inverse quantization section 142: Inverse conversion section 150: Composite section 160: Loop filter 170: Memory 180: Prediction Department 181: Inter prediction section 182: Intra prediction unit 183: Switching section

Claims

1. A coding device including a quantization unit that performs a quantization process on two-dimensional transform coefficients obtained by performing a transform process on a residual signal representing a difference between a block of an original image and a block of a predicted image, The quantization unit generates a one-dimensional scaled transform coefficient and a one-dimensional quantized transform coefficient based on the two-dimensional transform coefficient, and adjusts the one-dimensional quantized transform coefficient based on the one-dimensional scaled transform coefficient and the one-dimensional quantized transform coefficient.

2. An entropy coding unit converts the two-dimensional quantized transform coefficients obtained by the quantization unit into one-dimensional coefficients according to a scan order and performs entropy coding on the one-dimensional coefficients, The quantization unit is a derivation unit that derives the two-dimensional scaled transform coefficients by dividing the two-dimensional transform coefficients by a predetermined value, and converts the two-dimensional scaled transform coefficients into integers by a rounding process, thereby deriving the two-dimensional quantized transform coefficients; a dimension conversion unit that converts each of the two-dimensional scaled transform coefficients and the two-dimensional quantized transform coefficients into one dimension according to the scan order; an adjustment unit that adjusts the one-dimensional quantized transform coefficient using a quantization adjustment value obtained by inputting the one-dimensional scaled transform coefficient and the one-dimensional quantized transform coefficient obtained by the dimension conversion unit into a trained model; and a dimension inverse transform unit that inversely transforms the one-dimensional quantized transform coefficients adjusted by the adjustment unit into two dimensions according to the scan order, and outputs the two-dimensional quantized transform coefficients.

3. The adjustment unit is a sign acquisition unit for acquiring a positive or negative sign of each transform coefficient constituting the one-dimensional quantized transform coefficient; a sign removal unit that removes the positive / negative signs of each of the one-dimensional scaled transform coefficients and the positive / negative signs of each of the one-dimensional quantized transform coefficients; an adjustment value acquisition unit that acquires the quantization adjustment value from the one-dimensional scaled transform coefficient from which the positive and negative signs have been removed by the sign removal unit and the one-dimensional quantized transform coefficient by using the trained model; 3. The encoding device according to claim 2, further comprising: an adjustment value addition unit that adds the quantization adjustment value obtained by the adjustment value acquisition unit to the one-dimensional quantized transform coefficient from which the positive and negative signs have been removed, and assigns the positive and negative signs obtained by the sign acquisition unit to the one-dimensional quantized transform coefficient.

4. The entropy coding unit converts the two-dimensional quantized transform coefficients into one-dimensional coefficients according to a scan order selected from a plurality of predetermined scan orders; The encoding device according to claim 2 or 3, characterized in that the adjustment unit identifies a trained model corresponding to the selected scan order from among a plurality of trained models corresponding to the plurality of scan orders, and derives the quantization adjustment value using the identified trained model.

5. a conversion unit that performs the conversion process on the residual signal, The conversion unit performs a conversion process of a type selected from a plurality of predetermined types of conversion processes, The encoding device according to claim 2 or 3, characterized in that the adjustment unit identifies a trained model corresponding to the selected type of conversion processing from among a plurality of trained models corresponding to the plurality of types of conversion processing, and derives the quantization adjustment value using the identified trained model.

6. the two-dimensional transform coefficients have a block size selected from a plurality of predetermined block sizes; The encoding device according to claim 2 or 3, characterized in that the adjustment unit identifies a trained model corresponding to the selected block size from among a plurality of trained models corresponding to the plurality of types of block sizes, and derives the quantization adjustment value using the identified trained model.

7. The encoding device according to any one of claims 2 to 6, characterized in that the trained model is a trained model generated by machine learning using multiple sets of one-dimensional scaled transform coefficients, one-dimensional quantized transform coefficients, and optimal one-dimensional quantized transform coefficients derived using a full search or a greedy method, or quantization adjustment values ​​based on the optimal one-dimensional quantized transform coefficients, as training data.

8. A program for causing a computer to function as the encoding device according to any one of claims 1 to 7.

9. A model generation method for generating a trained model for encoding, comprising: A step of acquiring a set of one-dimensional scaled transform coefficients, one-dimensional quantized transform coefficients, and optimal one-dimensional quantized transform coefficients derived using a full search or a greedy method or a quantization adjustment value based on the optimal one-dimensional quantized transform coefficients as training data; and generating the trained model by machine learning using the training data including a plurality of the sets, the trained model having the one-dimensional scaled transform coefficient and the one-dimensional quantized transform coefficient as input, and the optimal one-dimensional quantized transform coefficient or the quantization adjustment value as output.

10. The method further comprises the steps of: dividing the two-dimensional transform coefficients by a predetermined value to derive two-dimensional scaled transform coefficients; and converting the two-dimensional scaled transform coefficients to integers by a rounding process to derive two-dimensional quantized transform coefficients; 10. The model generating method of claim 9 , wherein the obtaining step includes a step of obtaining the one-dimensional scaled transform coefficients and the one-dimensional quantized transform coefficients by converting each of the two-dimensional scaled transform coefficients and the two-dimensional quantized transform coefficients into one dimension according to a scan order.

11. 11. The model generation method according to claim 9, wherein the generating step includes a step of generating the trained model for each of a plurality of predetermined scan orders.

Images