Method and device for encoding image and decoding code stream by using neural network

Abstract

The invention relates to a method for processing an image, a neural network, an encoder and a decoder. Specifically, a filling layer is added before each layer having the same function as a down-sampling layer, and a cutting layer is added after each layer having the same function as an up-sampling layer, so that the amount of data processed in a neural network is reduced, and the decoding efficiency is improved.

Description

[0001] The present invention relates to a method for processing an image using a neural network and a neural network, as well as an encoder, a decoder and a computer-readable storage medium for executing these methods. Background Art

[0002] Video coding (video encoding and decoding) is used in a wide range of digital video applications, such as broadcast digital TV, video transmission over the Internet and mobile networks, real-time conversation applications such as video chat and video conferencing, DVD and Blu-ray Discs, video content acquisition and editing systems, and camcorders for security applications.

[0003] Even if a video is relatively short, a large amount of video data is required to describe it, which can cause difficulties when the data is to be streamed or otherwise transmitted over a communication network with limited bandwidth capacity. Therefore, video data is often compressed before being transmitted over modern telecommunications networks. Since memory resources may be limited, the size of the video may also be an issue when storing the video in a storage device. Video compression devices typically use software and / or hardware on the source side to decode the video data before transmitting or storing the video data, thereby reducing the amount of data required to represent the digital video image. A video decompression device that decodes the video data then receives the compressed data on the destination side. With limited network resources and a growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase the compression ratio with little impact on image quality.

[0004] Neural networks and deep learning techniques using neural networks have been used for some time in the technical field of encoding and decoding of videos, images, etc.

[0005] In these cases, the bitstream usually represents or is data that can be reasonably represented by a two-dimensional matrix of values. For example, this applies to bitstreams that represent or are images, video sequences, or similar data. In addition to 2D data, the neural networks and frameworks mentioned in the present invention can also be applied to more source signals such as audio signals, which are usually represented as 1D signals or other signals.

[0006] For example, neural networks can be used for both image recognition using deep learning neural networks and encoding images. Correspondingly, such networks can be used to decode encoded images. Other source signals, for example, signals with less than or more than two dimensions, can also be processed by similar networks.

[0007] It is desirable to provide a neural network framework that can be efficiently applied to a variety of signals that may vary in size. Summary of the invention

[0008] The embodiments of the present invention can effectively process images, while ensuring that the original information of the images can be reconstructed with as little information loss as possible, and can also improve decoding efficiency.

[0009] One embodiment of the present invention relates to a method for processing an image using a neural network (NN). The NN includes a multi-stage context model (MCM), and the MCM includes multiple MCMs. k model, the multiple MCM k The model is applied to the reorganized data: a first lower reorganization layer is applied to the first input of the MCM process, and an upper reorganization layer is applied to the output of the MCM process. There is a first padding layer before the first lower reorganization layer, and there is a cropping layer after the upper reorganization layer; the method includes: obtaining a first tensor, wherein the first tensor is the reconstructed residual; obtaining a second tensor, wherein the second tensor is an explicit prediction, i.e., the output of the super decoder; padding the first tensor using the first padding layer; performing lower reorganization on the padded first tensor according to the first lower reorganization layer to obtain a reorganized first tensor; and obtaining a plurality of MCMs according to the plurality of MCMs. k The model processes the reorganized first tensor and the second tensor to obtain a prior space tensor; according to the upper reorganization layer, the prior space tensor is reorganized to obtain a reorganized prior space tensor; according to the cropping layer, the reorganized prior space tensor is cropped to obtain a reconstructed prior tensor. The depth and step length of the above-mentioned padding layer are equal to the depth and step length of the above-mentioned cropping layer.

[0010] The present invention adds a padding layer before a downsampling layer or any layer having the same function as the downsampling layer in an MCM structure, and adds a clipping layer after an upsampling layer or any layer having the same function as the upsampling layer, so as to avoid the size of the tensor being non-integer in any processing step and to avoid damaging device interoperability (because the fractional size of the tensor is uncertain, so different devices and processors will have different processing methods), while reducing the amount of data processed in the neural network (and memory usage), thereby improving decoding efficiency.

[0011] Understandably, MCM k The model may also be called a phase or MCM k Stages, and the number of stages can be equal to 8, but is not limited to 8, and the number of stages can also be equal to 6 or less, or equal to 10 or more.

[0012] It is also understood that the precise design of the MCM structure may change accordingly in the future. For example, the MCM structure may include more or fewer stages, more or fewer down-reorganization layers or up-reorganization layers, and the MCM structure may also include some other layers to change the size or shape of the tensor. However, no matter how the MCM structure is changed, any down-sampling layer / down-reorganization layer or any layer with the same function as the down-sampling layer must be preceded by a padding layer, and any up-sampling layer / up-reorganization layer or any layer with the same function as the up-sampling layer must be followed by a cropping layer. This can avoid the size of the tensor being non-integer in any processing step, avoid destroying device interoperability (because the fractional size of the tensor is uncertain, so different devices and processors will have different processing methods), and can also reduce the amount of data processed in the neural network (and memory usage), thereby improving decoding efficiency.

[0013] In one embodiment, the number of input tensor slices of the first lower reorganization layer is 2.

[0014] In one embodiment, the number of input tensor slices of the upper reorganization layer is 2.

[0015] In one embodiment, the first filling layer is represented by Padd(H in ,W in ), where H in and W in are the height and width of the first tensor.

[0016] In one embodiment, the step length of the first filling layer is equal to 2 and the depth is equal to 5.

[0017] In one embodiment, the step length of the second filling layer is equal to 2 and the depth is equal to 5.

[0018] In one embodiment, the step size of the clipping layer is equal to 2 and the depth is equal to 5.

[0019] In one embodiment, the crop layer is represented by Crop(H in ,W in ), where H in and W in are the height and width of the tensor output to the composite transform.

[0020] In one embodiment, the first tensor is a reconstructed residual tensor, and the second tensor is an explicit prediction tensor.

[0021] One embodiment of the present invention discloses a neural network (NN). The NN includes a multi-stage context model (MCM), wherein the MCM includes multiple MCMs.k A model, a first lower reorganization layer and an upper reorganization layer, wherein the first lower reorganization layer is preceded by a first filling layer and the upper reorganization layer is followed by a clipping layer; the first lower reorganization layer is used to change the size or shape of a tensor; and the upper reorganization layer is used to change the size or shape of a tensor.

[0022] In one embodiment, the number of input tensor slices of the first lower reorganization layer is 2.

[0023] In one embodiment, the number of input tensor slices of the upper reorganization layer is 2.

[0024] In one embodiment, the first filling layer is represented by Padd(H in ,W in ), where H in和 W in are the height and width of the first tensor.

[0025] In one embodiment, the crop layer is represented by Crop(H in ,W in ), where H in and W in are the height and width of the tensor output to the composite transform.

[0026] In one embodiment, the first padding layer receives a tensor of a first size and outputs a tensor of a second size.

[0027] In one embodiment, the first filling layer is performed by replication.

[0028] In one embodiment, the MCM k Model uses previous MCM k The model's output tensor is used as input, where k ranges from 0 to k–1.

[0029] An embodiment of the present invention discloses an encoder for encoding an image. The encoder includes a receiver for receiving an image and one or more processors for implementing a neural network (NN), wherein the NN includes a multi-stage context model (MCM), wherein the MCM includes multiple MCMs k Model, the MCM also includes a first lower reorganization layer and an upper reorganization layer, the first lower reorganization layer is preceded by a first padding layer, the upper reorganization layer is followed by a cropping layer, the encoder also includes a transmitter for outputting a code stream, and the encoder is used to execute any of the methods provided in the above embodiments.

[0030] An embodiment of the present invention discloses a decoder for decoding a code stream representing an image. The decoder includes a receiver for receiving the code stream and one or more processors for implementing a neural network (NN), wherein the NN includes a multi-stage context model (MCM), wherein the MCM includes multiple MCMs. k Model, the MCM also includes a first lower reorganization layer and an upper reorganization layer, the first lower reorganization layer is preceded by a first padding layer, the upper reorganization layer is followed by a cropping layer, the decoder also includes a transmitter for outputting a decoded image, and the decoder is used to execute any one of the methods provided in the above embodiments.

[0031] One embodiment of the present invention discloses a method for processing an image using a neural network (NN). The NN includes a very large-scale decoder, the very large-scale decoder includes a basic operation point and an advanced operation point, under the basic operation point, the very large-scale decoder includes a first quantized transposed convolution layer followed by a first cropping layer and a first rectified linear unit, a first quantized convolution layer followed by a second rectified linear unit, a second quantized transposed convolution layer followed by a second cropping layer and a third rectified linear unit, and a second quantized convolution layer;

[0032] At the advanced operating point, the ultra-large-scale decoder includes two quantized convolutional layers and three quantized convolutional layers, each of the two quantized convolutional layers is followed by a pixel reassembly layer, a cropping layer and a rectified linear unit in sequence, and two of the three quantized convolutional layers are followed by rectified linear units respectively; the method includes:

[0033] Get the input tensor;

[0034] Get the size of the input tensor;

[0035] Get the operating point indicator;

[0036] According to the operation point indicator, determine whether to use the basic operation point or the advanced operation point to process the input tensor;

[0037] Output the processed tensor.

[0038] It can be understood that the above-mentioned ultra-large-scale decoder includes two processing pipelines, one pipeline is a basic operation point, and the other pipeline is an advanced operation point. In some other embodiments, the basic operation point can also be named as a basic profile, a baseline, a basic channel, a basic sub-network or some other names, and correspondingly, the advanced operation point can also be named as an advanced profile, a high line, an advanced channel, an advanced sub-network or some other names.

[0039] It is also understandable that the precise design of the above-mentioned super-large-scale decoder may change accordingly in the future. For example, the super-large-scale decoder may include more or fewer quantized transposed convolutional layers, more or fewer quantized convolutional layers, and the super-large-scale decoder may also include some other layers to change the size or shape of the tensor, but no matter how the super-large-scale decoder is changed, any downsampling layer / down-reorganization layer or any layer with the same function as the downsampling layer must be preceded by a padding layer, and any upsampling layer / up-reorganization layer or any layer with the same function as the upsampling layer must be followed by a cropping layer.

[0040] The present invention proposes to use tensor boundary processing in a very large-scale decoder to reduce the amount of data processed in a neural network, thereby improving decoding efficiency. Tensor boundary processing refers to adding a padding layer before a downsampling layer or any layer with the same function as a downsampling layer and adding a cropping layer after an upsampling layer or any layer with the same function as an upsampling layer. This ensures that the tensor size is an integer in any processing step, thereby avoiding uncertainty (uncertain processes will lead to platform dependencies and undermine device interoperability). Since the very large-scale decoder outputs parameters for the entropy encoder / decoder, it must have bit-accurate behavior, otherwise the bits parsed from the bitstream cannot be correctly interpreted.

[0041] In one embodiment, the kernel sizes of the first quantized transposed convolution layer and the second quantized transposed convolution layer are both 4×4, and the strides are both equal to 2.

[0042] In one embodiment, the step size of the first cropping layer is equal to 2 and the depth is equal to 6.

[0043] In one embodiment, the step size of the second cropping layer is equal to 2 and the depth is equal to 5.

[0044] In one embodiment, the step size of the cropping layer after the first of the two quantized convolutional layers at the advanced operating point is equal to 2, and the depth is equal to 6; the step size of the cropping layer after the second of the two quantized convolutional layers at the advanced operating point is equal to 2, and the depth is equal to 5.

[0045] In one embodiment, the processed tensor is a very large-scale decoder standard deviation tensor.

[0046] In one embodiment, when the operation point indicator is equal to 0, the input tensor is processed using the base operation point.

[0047] In one embodiment, when the operation point indicator is equal to 1, the input tensor is processed using the high-level operation point.

[0048] In one embodiment, the kernel size of the first quantized convolutional layer is 3×3.

[0049] In one embodiment, the pixel reorganization layer is used to change the number of channels from 4C to C.

[0050] One embodiment of the present invention discloses a neural network (NN). The NN includes a very large-scale decoder, the very large-scale decoder includes a basic operation point and an advanced operation point, at the basic operation point, the very large-scale decoder includes a first quantized transposed convolutional layer followed by a first cropping layer and a first rectified linear unit, a first quantized convolutional layer followed by a second rectified linear unit, a second quantized transposed convolutional layer followed by a second cropping layer and a third rectified linear unit, and a second quantized convolutional layer;

[0051] At the advanced operating point, the ultra-large-scale decoder includes two quantized convolutional layers and three quantized convolutional layers, each of the two quantized convolutional layers is followed by a pixel reconstruction layer, a cropping layer and a rectified linear unit in sequence, and two of the three quantized convolutional layers are respectively followed by a rectified linear unit.

[0052] One embodiment of the present invention discloses an encoder for encoding an image. The encoder includes a receiver for receiving an image and one or more processors for implementing a neural network (NN), the NN includes a super-large-scale decoder, the super-large-scale decoder includes a basic operation point and an advanced operation point, under the basic operation point, the super-large-scale decoder includes a first quantized transposed convolution layer followed by a first cropping layer and a first rectified linear unit, a first quantized convolution layer followed by a second rectified linear unit, a second quantized transposed convolution layer followed by a second cropping layer and a third rectified linear unit, and a second quantized convolution layer;

[0053] At the advanced operating point, the ultra-large-scale decoder includes two quantized convolution layers and three quantized convolution layers, each of the two quantized convolution layers is followed by a pixel reconstruction layer, a cropping layer and a rectified linear unit in sequence, two of the three quantized convolution layers are followed by a rectified linear unit respectively, and the encoder also includes a transmitter for outputting a code stream, and the encoder is used to execute the method provided in any of the above embodiments.

[0054] One embodiment of the present invention discloses a decoder for decoding a code stream representing an image. The decoder includes a receiver for receiving the code stream and one or more processors for implementing a neural network (NN), the NN includes a very large-scale decoder, the very large-scale decoder includes a basic operation point and an advanced operation point, under the basic operation point, the very large-scale decoder includes a first quantized transposed convolution layer followed by a first cropping layer and a first rectified linear unit, a first quantized convolution layer followed by a second rectified linear unit, a second quantized transposed convolution layer followed by a second cropping layer and a third rectified linear unit, and a second quantized convolution layer;

[0055] At the advanced operating point, the ultra-large-scale decoder includes two quantized convolutional layers and three quantized convolutional layers, each of the two quantized convolutional layers is followed by a pixel reconstruction layer, a cropping layer and a rectified linear unit in sequence, two of the three quantized convolutional layers are followed by a rectified linear unit respectively, the decoder also includes a transmitter for outputting a decoded image, and the decoder is used to execute any of the above embodiments.

[0056] One embodiment of the present invention discloses a method for processing an image using a neural network (NN). The NN includes a synthetic transform network, the synthetic transform network includes a cascade layer for cascading a main tensor and an auxiliary tensor as an input tensor, a basic operation point and an advanced operation point, under the basic operation point, the synthetic transform network includes a lightweight residual block followed by a first transposed convolution layer combined with a first cropping layer and a first residual activation unit, a second transposed convolution layer combined with a second cropping layer and a second residual activation unit, a first convolution layer combined with a third residual activation unit, and a second convolution layer followed by a first pixel reorganization layer and a third cropping layer;

[0057] At the high-level operating point, the synthetic transform network includes two residual blocks followed by a third transposed convolutional layer combined with a fourth cropping layer and a first residual activation, a fourth transposed convolutional layer combined with a fifth cropping layer and a second residual activation, a third convolutional layer followed by a second pixel reassembly layer and a residual non-local attention block combined with a sixth cropping layer and a third residual activation, and a fifth transposed convolutional layer followed by a seventh cropping layer; the method includes:

[0058] Get the input tensor by concatenating the main tensor and the auxiliary tensor;

[0059] Get the operating point indicator;

[0060] Get the size of the input tensor;

[0061] According to the operation point indicator, determine whether to use the basic operation point or the advanced operation point to process the input tensor;

[0062] Output the processed tensor.

[0063] It can be understood that the above-mentioned synthetic transformation network includes two processing pipelines, one pipeline is a basic operation point, and the other pipeline is an advanced operation point. In some other embodiments, the basic operation point can also be named as a basic profile, a baseline, a basic pipeline, a basic channel, a basic sub-network or some other names, and correspondingly, the advanced operation point can also be named as an advanced profile, a high line, an advanced pipeline, an advanced channel, an advanced sub-network or some other names.

[0064] It is also understandable that the precise design of the above-mentioned synthetic transformation network may change accordingly in the future. For example, the synthetic transformation network may include more or fewer quantized transposed convolutional layers, more or fewer quantized convolutional layers, more or fewer pixel reorganization layers, and the synthetic transformation network may include some other layers to change the size or shape of the tensor, but no matter how the synthetic transformation network is changed, any downsampling layer / down-reorganization layer or any layer with the same function as the downsampling layer must be preceded by a padding layer, and any upsampling layer / up-reorganization layer or any layer with the same function as the upsampling layer must be followed by a cropping layer.

[0065] The present invention proposes to use tensor boundary processing in the synthetic transform network to reduce the amount of data processed in the neural network, thereby improving the decoding efficiency. Tensor boundary processing refers to adding a padding layer before the downsampling layer or any layer with the same function as the downsampling layer and adding a cropping layer after the upsampling layer or any layer with the same function as the upsampling layer. This ensures that the tensor size is an integer in any processing step without generating uncertainty.

[0066] In one embodiment, the step size of the first cropping layer is equal to 2 and the depth is equal to 4.

[0067] In one embodiment, the step size of the second cropping layer is equal to 2 and the depth is equal to 3.

[0068] In one embodiment, the step size of the third clipping layer is equal to 4 and the depth is equal to 1.

[0069] In one embodiment, the fourth clipping layer has a step size of 2 and a depth of 4.

[0070] In one embodiment, the step size of the fifth cropping layer is equal to 2 and the depth is equal to 3.

[0071] In one embodiment, the sixth clipping layer has a step size of 2 and a depth of 2.

[0072] In one embodiment, the step size of the seventh clipping layer is equal to 2 and the depth is equal to 1.

[0073] In one embodiment, when the operation point indicator is equal to 0, the input tensor is processed using the base operation point.

[0074] In one embodiment, when the operation point indicator is equal to 1, the input tensor is processed using the high-level operation point.

[0075] In one embodiment, the prior volume is a reconstructed prior space tensor.

[0076] One embodiment of the present invention discloses a neural network (NN). The NN includes a synthetic transformation network, the synthetic transformation network includes a cascade layer for cascading a main tensor and an auxiliary tensor as an input tensor, a basic operation point and an advanced operation point, under the basic operation point, the synthetic transformation network includes a lightweight residual block followed by a first transposed convolution layer combined with a first cropping layer and a first residual activation unit, a second transposed convolution layer combined with a second cropping layer and a second residual activation unit, a first convolution layer combined with a third residual activation unit, and a second convolution layer followed by a first pixel reorganization layer and a third cropping layer;

[0077] At the high-level operating point, the synthetic transform network includes two residual blocks followed by a third transposed convolutional layer combined with a fourth cropping layer and a first residual activation, a fourth transposed convolutional layer combined with a fifth cropping layer and a second residual activation, a third convolutional layer followed by a second pixel reassembly layer and a residual non-local attention block combined with a sixth cropping layer and a third residual activation, and a fifth transposed convolutional layer followed by a seventh cropping layer.

[0078] One embodiment of the present invention discloses an encoder for encoding an image. The encoder includes a receiver for receiving an image and one or more processors for implementing a neural network (NN), the NN including a synthetic transform network, the synthetic transform network including a cascade layer for cascading a main tensor and an auxiliary tensor as an input tensor, a basic operation point and an advanced operation point, under the basic operation point, the synthetic transform network includes a lightweight residual block followed by a first transposed convolution layer combined with a first cropping layer and a first residual activation unit, a second transposed convolution layer combined with a second cropping layer and a second residual activation unit, a first convolution layer combined with a third residual activation unit, and a second convolution layer followed by a first pixel reorganization layer and a third cropping layer;

[0079] At the high-level operating point, the synthetic transformer network includes two residual blocks followed by a third transposed convolutional layer combined with a fourth cropping layer and a first residual activation, a fourth transposed convolutional layer combined with a fifth cropping layer and a second residual activation, a third convolutional layer followed by a second pixel reassembly layer and a residual non-local attention block combined with a sixth cropping layer and a third residual activation, and a fifth transposed convolutional layer followed by a seventh cropping layer;

[0080] The encoder further includes a transmitter for outputting a code stream, and the encoder is used to execute the method provided in any of the above embodiments.

[0081] One embodiment of the present invention discloses a decoder for decoding a code stream representing an image. The decoder includes a receiver for receiving the code stream and one or more processors for implementing a neural network (NN), wherein the NN includes a synthetic transform network, wherein the synthetic transform network includes a cascade layer for cascading a main tensor and an auxiliary tensor as an input tensor, a basic operation point, and an advanced operation point, wherein at the basic operation point, the synthetic transform network includes a lightweight residual block followed by a first transposed convolution layer combined with a first cropping layer and a first residual activation unit, a second transposed convolution layer combined with a second cropping layer and a second residual activation unit, a first convolution layer combined with a third residual activation unit, and a second convolution layer followed by a first pixel reorganization layer and a third cropping layer;

[0082] At the high-level operating point, the synthetic transformation network includes two residual blocks followed by a third transposed convolution layer combined with a fourth cropping layer and a first residual activation, a fourth transposed convolution layer combined with a fifth cropping layer and a second residual activation, a third convolution layer followed by a second pixel reorganization layer and a residual non-local attention block combined with a sixth cropping layer and a third residual activation, and a fifth transposed convolution layer followed by a seventh cropping layer; the decoder also includes a transmitter for outputting a decoded image, and the decoder is used to perform any one of the methods provided in the above embodiments.

[0083] An embodiment of the present invention discloses a decoder for decoding a code stream representing an image. The decoder includes one or more processors for implementing a neural network (NN), and the one or more processors are used to execute the method provided by any of the above embodiments.

[0084] An embodiment of the present invention discloses an encoder for encoding an image, wherein the encoder comprises one or more processors for implementing a neural network (NN), and the one or more processors are used to execute the method provided in any of the above embodiments.

[0085] An embodiment of the present invention discloses a computer program product including computer executable instructions. When the computer executable instructions are executed on a computing system, the computing system executes any of the methods provided in the above embodiments.

[0086] One embodiment of the present invention discloses a neural network (NN). The NN includes a multi-stage context model (MCM), the MCM includes multiple MCMk models, the MCM also includes one or more lower reorganization layers and one or more upper reorganization layers, the one or more lower reorganization layers are respectively preceded by a padding layer, and the one or more upper reorganization layers are respectively followed by a clipping layer.

[0087] In one embodiment, the NN further comprises a very large-scale decoder, the very large-scale decoder comprising a baseline and a high line, the baseline comprising two or more quantized transposed convolutional layers, the two or more quantized transposed convolutional layers being followed by a cropping layer and a rectified linear unit, respectively, the high line comprising two or more quantized convolutional layers, each of the two or more quantized convolutional layers being followed by a pixel reassembly layer, a cropping layer and a rectified linear unit in sequence.

[0088] In one embodiment, the NN also includes a synthetic transformation network, which includes a baseline and a high line, the baseline includes two or more transposed convolutional layers and one or more convolutional layers, the two or more transposed convolutional layers are respectively followed by a cropping layer and a residual activation unit, and the one or more convolutional layers are respectively followed by a pixel reorganization layer and a cropping layer; the high line includes two or more transposed convolutional layers and a convolutional layer, the two or more transposed convolutional layers are respectively followed by a cropping layer and a residual activation, and the convolutional layer is followed by a pixel reorganization layer and a residual non-local attention block combined with the cropping layer.

[0089] One embodiment of the present invention discloses an encoder for encoding an image, wherein the encoder comprises a receiver for receiving an image, a transmitter for outputting a code stream, and one or more processors for implementing a neural network (NN) according to any one of claims 42 to 44.

[0090] One embodiment of the present invention discloses a decoder for decoding a code stream representing an image. The decoder comprises a receiver for receiving the code stream, a transmitter for outputting a decoded image, and one or more processors for implementing a neural network (NN) according to any one of claims 42 to 44.

[0091] An embodiment of the present invention discloses a method for processing an image using a neural network (NN). The NN includes a multi-stage context model (MCM), the MCM includes multiple MCMk models, the MCM also includes one or more lower reorganization layers and one or more upper reorganization layers, the one or more lower reorganization layers are respectively preceded by a padding layer, and the one or more upper reorganization layers are respectively followed by a cropping layer; the method includes:

[0092] Get the input tensor;

[0093] padding a first tensor using padding layers before the one or more lower reorganization layers, respectively, wherein the first tensor is the input tensor or a tensor obtained by processing the input tensor;

[0094] After the one or more upper reorganization layers respectively output second tensors, the second tensors are cropped using a cropping layer.

[0095] In one embodiment, the NN further comprises a super-large-scale decoder, the super-large-scale decoder comprises a baseline and a high line, the baseline comprises two or more quantized transposed convolutional layers, the two or more quantized transposed convolutional layers are respectively followed by a cropping layer and a rectified linear unit, the high line comprises two or more quantized convolutional layers, each of the two or more quantized convolutional layers is sequentially followed by a pixel reassembly layer, a cropping layer and a rectified linear unit; the method further comprises:

[0096] Get the operating point indicator;

[0097] Based on the operation point indicator, it is determined whether to use the baseline or the advanced operation line to process a third tensor.

[0098] In one embodiment, the NN further comprises a synthetic transformation network, the synthetic transformation network comprising a baseline and a high line, the baseline comprising two or more transposed convolutional layers and one or more convolutional layers, the two or more transposed convolutional layers are respectively followed by a cropping layer and a residual activation unit, and the one or more convolutional layers are respectively followed by a pixel reorganization layer and a cropping layer; the high line comprises two or more transposed convolutional layers and a convolutional layer, the two or more transposed convolutional layers are respectively followed by a cropping layer and a residual activation, and the convolutional layer is followed by a pixel reorganization layer and a residual non-local attention block combined with the cropping layer; the method further comprises:

[0099] Get the second input tensor by concatenating the primary tensor and the secondary tensor;

[0100] Get the operating point indicator;

[0101] Based on the operating point indicator, it is determined whether to use the baseline or the high line to process the second input tensor.

[0102] In one embodiment, when the operating point indicator is equal to 0, the second input tensor is processed using the baseline.

[0103] In one embodiment, when the operating point indicator is equal to 1, the second input tensor is processed using the high line.

[0104] Typically, an image in the context of the present invention may constitute a still image or a moving image like a video or video sequence. In addition, the term "image" may also include a portion of a larger image or a portion of a video sequence. An image (picture / image) may also be referred to as a frame (frame).

[0105] Applied to the input, changes the size of the input S in at least one dimension to size Resizing may typically involve adding or removing sample values ​​of size S from the input.

[0106] In this regard, selecting a resizing method from a plurality of resizing methods can be understood as meaning that, although a plurality of resizing methods can be used for image encoding, it is preferred not to use them arbitrarily, but to use one of them based on additional information. This may result in selecting a resizing method that is specifically suitable for the input or suitable for obtaining the expected output of the neural network (e.g., with reference to the size of the output).

[0107] The input to the neural network may be a two-dimensional input, for example, the image itself, or a matrix representing the sample values ​​of the image, or other structure representing the image. The input is not necessarily the image itself, but may be a preprocessed or otherwise processed version of the image. Preprocessing or processing the image before providing it as an input to the neural network may include preparing or modifying the image for further processing by the neural network, etc.

[0108] In the context of the present invention, a downsampling layer may be understood as a layer that reduces the size of the input by, for example, applying a convolution to the input. For example, this may include reducing the size by a factor, where the factor is also referred to as the downsampling ratio of the downsampling layer, if downsampling is applied to reduce the size S of the input to a size Then the downsampling ratio may be an integer greater than 1. The downsampling ratio may be any value, for example, 2, 4, 8, etc. The downsampling ratio may also not be a multiple of 2, for example, 5 or 13. The present invention is not limited to a specific downsampling ratio.

[0109] The output of the neural network may also be referred to as an encoded image, but the output of the neural network is not necessarily a code stream representing the encoded image. The output of encoding the image may be binarized and may also include additional information, such as information about a resizing method used to apply the resizing.

[0110] This embodiment can select a resizing method according to specific circumstances and apply the resizing method to resize. For example, in some cases, before using the neural network to process the input, the size S of the input is increased to a size greater than S during the resizing process. In other cases, reducing the size of the input S to a size smaller than S Might be more appropriate. While these are two general concepts of resizing (increasing or decreasing the size), in increasing the size S of the input to the size and reduce the size of the input to size Some of the methods may be more suitable than others, so they can be selected according to the specific situation. Alternatively or additionally, a user who wants to encode an image can preset a specific resizing method or a set of resizing methods. This can achieve user-friendly information encoding.

[0111] In another embodiment, the multiple resizing methods include one or more of the following: padding, zero padding, reflection padding, copy padding, cropping, increasing the input size S to the size Interpolation, reducing the input size S to size These methods can be used advantageously for resizing.

[0112] This evaluation can be used to determine whether increasing or decreasing the size is computationally efficient, which can determine the resizing method to use (e.g., padding or cropping).

[0113] If during the comparison it is found that C is equal to F, then no resizing method that changes the input size S can be applied. Based on these formulas, a reliable assessment can be made whether increasing or decreasing the size S is efficient.

[0114] In one embodiment, the one or more indications include an indication whose first value indicates that padding or clipping is to be applied as a resizing method and whose second value indicates that interpolation is to be applied as a resizing method. The first value and the second value of the indication in this context indicate that the indication can take the first value or the second value. Therefore, only a small amount of information is required to provide information to the encoding about which resizing method to use. To facilitate distinction from other indications, this indication may also be referred to as the "first indication" hereinafter. It may be present or absent, regardless of the presence or absence of other indications explained below.

[0115] Specifically, it may be specified that the indication is or includes a flag of 1 bit. In this way, a small amount of information may be used to indicate whether the size S of the input is increased or decreased during the resizing process.

[0116] In one embodiment, the one or more indications include an indication whose first value indicates that the size S is to be increased and whose second value indicates that the size S is to be decreased. To distinguish from other indications, this indication may also be referred to as a "second indication" hereinafter.

[0117] In another embodiment, the one or more indications include an indication whose first value indicates that padding is to be applied as a resizing method and whose second value indicates that clipping is to be applied as a resizing method. Information about whether padding or clipping is to be used when resizing can also be provided accordingly. To facilitate distinction from other indications, this indication may also be referred to as a "fourth indication" hereinafter. The presence of this indication is independent of the presence or absence of other indications. However, in some embodiments, this indication may be provided when the first indication is used to indicate that padding or clipping is to be applied as a resizing method.

[0118] Specifically, the indication may be or may include a flag of size 1 bit. This reduces the size of the indication to a minimum while ensuring that necessary information can be provided.

[0119] In another embodiment, the one or more indications include an indication whose value is used to indicate whether to apply zero fill, reflection fill, or copy fill as the resizing method. Different types of fills can be provided accordingly. For ease of distinction from other indications, this indication may also be referred to as the "fifth indication" hereinafter. The presence of this indication is independent of the presence or absence of other indications. However, in some embodiments, this indication may be provided when the fourth indication is used to indicate that filling or clipping is to be applied as the resizing method.

[0120] Specifically, the information about the resizing method used may include at least one of the following: the size of the input, the size of the image, the resizing method applied, one or more indications, and the downsampling ratio of at least one downsampling layer in the NN. These indications may be the first indication to the fifth indication mentioned above. However, other indications may also be considered. The present invention is not limited to the indications provided.

[0121] This information can be used to achieve reliable decoding of the code stream.

[0122] Specifically, it may be specified that the indication is or includes a flag of 1 bit in size, so that the size of the indication can be reduced to a minimum value.

[0123] In addition, a computer-readable storage medium including computer-executable instructions is provided. When the computer-executable instructions are executed on a computing system, the computing system is caused to execute the method provided in any of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0124] Figure 1A is a block diagram of an example of a video decoding system for implementing an embodiment of the present invention;

[0125] Figure 1B is a block diagram of another example of a video decoding system for implementing some embodiments of the present invention;

[0126] Figure 2 is a block diagram of an example of an encoding device or a decoding device;

[0127] Figure 3 is a block diagram of another example of an encoding device or a decoding device;

[0128] Figure 4 An encoder and a decoder provided by an embodiment are shown;

[0129] Figure 5 is a schematic illustration of encoding and decoding of input;

[0130] Figure 6 The encoder and decoder that conform to the VAE framework are shown;

[0131] Figure 7 An embodiment provides Figure 4 Components in the encoder shown;

[0132] Figure 8 An embodiment provides Figure 4 Components in the decoder shown;

[0133] Fig. 9 Scaling and processing of input is shown;

[0134] Fig.10 An encoder and a decoder are shown;

[0135] Fig.11 Another encoder and another decoder are shown;

[0136] Fig.12 shows the scaling and processing of input provided by one embodiment;

[0137] Fig.13 One embodiment of a signaling scaling option provided by one embodiment is shown;

[0138] Fig.14 Shows Fig.13A more specific implementation of the embodiment shown;

[0139] Fig.15 Shows Fig.14 A more specific implementation of the embodiment shown;

[0140] Fig.16 A comparison of different possibilities for filling operations is shown;

[0141] Fig.17 A further comparison of different possibilities for filling operations is shown;

[0142] Fig.18 shows an encoder and a decoder according to one embodiment and the relationship in processing the input of the encoder and the decoder;

[0143] Fig.19 is a schematic diagram of a neural network as part of an encoder provided by an embodiment;

[0144] Fig. 20 is a flowchart of a method for encoding an image provided by an embodiment;

[0145] Fig.21 An embodiment of providing one or more indications for encoding is shown;

[0146] Fig. 22 is a schematic diagram of a neural network as part of a decoder provided by an embodiment;

[0147] Fig.23 is a flowchart of a method for decoding a code stream provided by an embodiment;

[0148] Fig.24 An embodiment of providing one or more indications for decoding is shown;

[0149] Fig.25 is a schematic diagram of an encoder provided by an embodiment;

[0150] Fig.26 is a schematic diagram of a decoder provided by an embodiment;

[0151] Fig.27A is a schematic illustration of a residual activation unit;

[0152] Fig.27B is a schematic diagram of the residual activation function;

[0153] Fig.27C It is a schematic diagram of the residual non-local attention block (RBAN);

[0154] Fig.27D It is a schematic diagram of a residual block (RB);

[0155] Fig.27E It is a schematic diagram of a lightweight residual block (LRB);

[0156] Fig.28 is a flowchart of a method for processing an image provided by an embodiment;

[0157] Fig.29 An example of a multi-stage context modeling process is shown;

[0158] Fig.29A Another example of a multi-stage context modeling process is shown;

[0159] Fig.30 An example of a channel network process is shown;

[0160] Fig.31 is a flowchart of a method for processing an image provided by an embodiment;

[0161] Fig.32 An example of a very large-scale decoder is shown;

[0162] Fig.33 is a flowchart of a method for processing an image provided by an embodiment;

[0163] Fig.34 An example of a synthetic transform network is shown. DETAILED DESCRIPTION

[0164] In the following, some embodiments are described with reference to the accompanying drawings. Figure 3 Refers to video decoding systems and methods, which can be used with more specific embodiments of the present invention described in other figures. Specifically, in conjunction with Figures 1 to Figure 3 The described embodiments may be used with the encoding / decoding techniques described further below that use neural networks to encode a bitstream and / or decode a bitstream.

[0165] In the following description, reference is made to the accompanying drawings that form a part of the present invention, which show by way of illustration specific aspects of the present invention or specific aspects in which embodiments of the present invention may be used. It is understood that the embodiments may be used in other aspects and include structural or logical changes that are not depicted in the accompanying drawings. Therefore, the following specific embodiments should not be understood in a restrictive sense, and the scope of the present invention is defined by the appended claims.

[0166] For example, it is understood that the disclosure related to describing a method may also be applicable to a corresponding device or system for performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units (e.g., functional units) to perform the described one or more method steps (e.g., one unit performs one or more steps, or multiple units perform one or more steps in multiple steps respectively), even if the one or more units are not explicitly described or illustrated in the drawings. On the other hand, for example, if a specific device is described based on one or more units (e.g., functional units), the corresponding method may include a step to perform the function of one or more units (e.g., one step performs the function of one or more units, or multiple steps perform the function of one or more units in multiple units respectively), even if the one or more units are not explicitly described or illustrated in the drawings. In addition, it is understood that, unless otherwise explicitly stated, the features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0167] Video decoding (coding) generally refers to processing a sequence of images that form a video or video sequence. In the field of video decoding, the terms "frame" and "picture / image" can be used as synonyms. Video decoding (or commonly referred to as decoding) includes two parts: video encoding and video decoding. Video encoding is performed on the source side and generally includes processing (for example, by compression) the original video image to reduce the amount of data required to represent the video image (thereby more efficiently storing and / or transmitting). Video decoding is performed on the destination side and generally includes an inverse process relative to the encoder for reconstructing the video image. The "decoding" of the video images (or commonly referred to as images) involved in the embodiments should be understood as involving the "encoding" or "decoding" of the video images or respective video sequences. The encoding part and the decoding part are also collectively referred to as codecs (encoding and decoding).

[0168] In the case of lossless video decoding, the original video image can be reconstructed, that is, the reconstructed video image has the same quality as the original video image (assuming that there is no transmission loss or other data loss during storage or transmission). In the case of lossy video decoding, further compression is performed by quantization, etc. to reduce the amount of data representing the video image, and the decoder side cannot fully reconstruct the video image, that is, the quality of the reconstructed video image is lower or worse than the quality of the original video image.

[0169] Several video coding standards belong to the group of "lossy hybrid video codecs" (i.e., combining spatial and temporal prediction in the sample domain with 2D transform coding for quantization in the transform domain). Each picture in a video sequence is usually segmented into a set of non-overlapping blocks, and coding is usually performed at the block level. In other words, the encoder side usually processes the video at the block (video block) level, i.e., encodes it, for example, by using spatial (intra) prediction and / or temporal (inter) prediction to generate a prediction block, subtracting the prediction block from the current block (currently processed / to be processed block) to obtain a residual block, transforming the residual block in the transform domain and quantizing the residual block to reduce the amount of data to be transmitted (compressed); while the decoder side applies the inverse processing part relative to the encoder to the encoded or compressed block to reconstruct the current block for representation. In addition, the encoder replicates the decoder processing cycle, so that the encoder and decoder generate the same prediction (e.g., intra and inter prediction) and / or reconstruction to process, i.e., decode, the subsequent blocks. Recently, parts or the entire encoding and decoding chain have been implemented using neural networks or, in general, any machine learning or deep learning framework.

[0170] In the following embodiment of a video coding system 10 , a video encoder 20 and a video decoder 30 are described according to FIG. 1 .

[0171] Figure 1A 1 is a schematic block diagram of an exemplary decoding system 10 (e.g., video decoding system 10 (or simply decoding system 10)) that can utilize the techniques in the present application. Video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in video decoding system 10 are two examples of devices that can be used to perform various techniques according to various examples described in the present application.

[0172] like Figure 1A As shown, the decoding system 10 includes a source device 12 for providing encoded image data 21 , for example, to a destination device 14 ; the destination device decodes the encoded image data 21 .

[0173] The source device 12 includes an encoder 20 and may additionally (i.e., optionally) include an image source 16, a preprocessor (or preprocessing unit) 18 (e.g., image preprocessor 18), and a communication interface or communication unit 22. Some embodiments of the present invention (e.g., involving initial scaling or scaling between two consecutive layers) may be implemented by the encoder 20. Some embodiments (e.g., involving initial scaling) may be implemented by the image preprocessor 18.

[0174] The image source 16 may include or may be any type of image capture device such as a camera for capturing real-world images, and / or any type of image generation device such as a computer graphics processor for generating computer-animated images, or any type of other device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage that stores any of the above images.

[0175] In order to distinguish the pre-processor 18 and the processing performed by the pre-processing unit 18 , the image or image data 17 may also be referred to as a raw image or raw image data 17 .

[0176] The preprocessor 18 is used to receive (raw) image data 17 and perform preprocessing on the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. The preprocessing performed by the preprocessor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color adjustment or denoising, etc. It can be understood that the preprocessing unit 18 may be an optional component.

[0177] The video encoder 20 is configured to receive the pre-processed image data 19 and provide encoded image data 21 .

[0178] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and transmit the encoded image data 21 (or data obtained after further processing of the encoded image data 21) to another device, such as the destination device 14 or any other device, through the communication channel 13 for storage or direct reconstruction.

[0179] The destination device 14 includes a decoder 30 (eg, a video decoder 30 ), and may additionally (ie, optionally) include a communication interface or communication unit 28 , a post-processor 32 (or a post-processing unit 32 ), and a display device 34 .

[0180] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or data obtained after further processing of the encoded image data 21) directly from the source device 12 or from any other source such as a storage device (for example, an encoded image data storage device), and provide the encoded image data 21 to the decoder 30.

[0181] The communication interface 22 and the communication interface 28 can be used to send or receive the encoded image data 21 or the encoded data 21 via a direct communication link (e.g., a direct wired or wireless connection) between the source device 12 and the destination device 14 or via any type of network (e.g., a wired network or a wireless network or any combination thereof, or any type of private network and public network, or any type of combination thereof).

[0182] For example, the communication interface 22 may be used to package the encoded image data 21 into a suitable format (eg, data packets) and / or process the encoded image data by any type of transmission encoding or processing for transmission over a communication link or communication network.

[0183] For example, the communication interface 28 corresponding to the communication interface 22 may be used to receive the transmission data and process the transmission data using any type of corresponding transmission decoding or processing manner and / or decapsulation manner to obtain the encoded image data 21 .

[0184] Both the communication interface 22 and the communication interface 28 can be configured as Figure 1A The unidirectional communication interface represented by the arrow of the communication channel 13 pointing from the source device 12 to the destination device 14 may be configured as a bidirectional communication interface and may be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission (e.g., encoded image data transmission), etc.

[0185] The decoder 30 is configured to receive the encoded image data 21 and provide decoded image data 31 or decoded image 31 (eg, more details based on Figure 3 etc. are described below).

[0186] The post-processor 32 in the destination device 14 is used to post-process the decoded image data 31 (also referred to as reconstructed image data) (e.g., decoded image 31) to obtain post-processed image data 33 (e.g., post-processed image 33). The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, cropping or resampling, or any other processing to provide the decoded image data 31 for display by the display device 34, etc.

[0187] Some embodiments of the present invention may be implemented by the decoder 30 or the post-processor 32 .

[0188] The display device 34 in the destination device 14 is used to receive the post-processed image data 33 so as to display the image to a user or viewer, etc. The display device 34 can be or include any type of display for displaying the reconstructed image, such as an integrated or external display or display screen. For example, the display can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display.

[0189] although Figure 1A The source device 12 and the destination device 14 are described as separate devices, but the device embodiments may also include two devices or two functions, namely the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions. In these embodiments, the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0190] According to the description, Figure 1A It will be apparent to a skilled person that the different units or functions shown and (accurately) divided in the source device 12 and / or the destination device 14 may vary depending on the actual device and application.

[0191] The encoder 20 (eg, video encoder 20) or the decoder 30 (eg, video decoder 30), or both the encoder 20 and the decoder 30 may be configured as follows: Figure 1BThe encoder 20 may be implemented by the processing circuitry 46 to embody the various modules and / or any other encoder systems or subsystems described herein. The decoder 30 may be implemented by the processing circuitry 46 to embody the various modules and / or any other decoder systems or subsystems described herein. The processing circuitry may be used to perform the various operations discussed below. As Figure 3 As shown, if the above technology is partially implemented in software form, the device can store the instructions of the software in a suitable non-transitory computer-readable storage medium, and these instructions can be executed in hardware form by one or more processors to implement the technology in the present invention. Either of the video encoder 20 and the video decoder 30 can be integrated into a single device as part of a combined encoder / decoder (codec), for example, as Figure 1B shown.

[0192] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or a tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or a content distribution server), a broadcast receiving device, a broadcast sending device, etc., and may not use or use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.

[0193] In some cases, Figure 1AThe video decoding system 10 shown is merely exemplary, and the technology provided in the present application can be applied to video decoding settings (e.g., video encoding or video decoding), which do not necessarily include any data communication between the encoding device and the decoding device. In other examples, data is retrieved from a local memory, streamed over a network, and the like. The video encoding device can encode the data and store the data in a memory, and / or the video decoding device can retrieve the data from the memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but only encode the data to the memory and / or retrieve the data from the memory and decode the data.

[0194] For convenience of description, for example, by referring to the reference software of High-Efficiency Video Coding (HEVC) or Versatile Video coding (VVC), the next generation video coding standard developed by the ITU-T Video Coding Experts Group (VCEG) Video Coding Joint Collaboration Team on Video Coding (JCT-VC), and the ISO / IEC Motion Picture Experts Group (MPEG), some embodiments are described herein. Those of ordinary skill in the art should understand that the embodiments of the present invention are not limited to HEVC or VVC.

[0195] Figure 2 FIG. 4 is a schematic diagram of a video decoding device 400 provided by an embodiment of the present invention. The video decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 400 may be a decoder, for example, Figure 1A The video decoder 30 in the embodiment may also be an encoder, for example, Figure 1A The video encoder 20 in.

[0196] The video decoding device 400 includes an inlet port 410 (or input port 410) and a receiving unit (Rx) 420 for receiving data, a processor, a logic unit or a central processing unit (CPU) 430 for processing the data, a transmitting unit (Tx) 440 and an outlet port 450 (or output port 450) for transmitting the data, and a memory 460 for storing the data. The video decoding device 400 may also include an optical-to-electrical (OE) component and an electrical-to-optical (EO) component coupled to the inlet port 410, the receiving unit 420, the transmitting unit 440 and the outlet port 450, which are used as an outlet or an inlet of an optical signal or an electrical signal.

[0197] The processor 430 is implemented by hardware and software. The processor 430 can be implemented as one or more CPU chips, one or more cores (for example, as a multi-core processor), one or more FPGAs, one or more ASICs, and one or more DSPs. The processor 430 communicates with the input port 410, the receiving unit 420, the sending unit 440, the output port 450, and the memory 460. The processor 430 includes a decoding module 470. The decoding module 470 implements the disclosed embodiments described above. For example, the decoding module 470 performs, processes, prepares, or provides various decoding operations. Therefore, the inclusion of the decoding module 470 provides substantial improvements to the functionality of the video decoding device 400 and affects the conversion of the video decoding device 400 to different states. Optionally, the decoding module 470 is implemented with instructions stored in the memory 460 and executed by the processor 430.

[0198] The memory 460 may include one or more disks, one or more tape drives, and one or more solid-state hard disks, and may be used as an overflow data storage device to store programs when they are selected for execution, as well as to store instructions and data read during the execution of the programs. For example, the memory 460 may be a volatile and / or non-volatile memory, and may be a read-only memory (ROM), a random access memory (RAM), a ternary content-addressable memory (TCAM), and / or a static random-access memory (SRAM).

[0199] Figure 31 is a simplified block diagram of an apparatus 500 provided by an exemplary embodiment, wherein the apparatus 500 may be used as either or both of the source device 12 and the destination device 14 in FIG. 1 .

[0200] The processor 502 in the device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices that are currently available or will be developed in the future and are capable of operating or processing information. Although the disclosed implementations may be implemented using a single processor such as the processor 502 shown in the figure, using multiple processors may improve speed and efficiency.

[0201] In one implementation, the memory 504 in the apparatus 500 may be a read only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 accessed by the processor 502 via a bus 512. The memory 504 may also include an operating system 508 and an application 510, the application 510 including at least one program that causes the processor 502 to perform the methods described herein. For example, the application 510 may include application 1 to application N, and also include a video decoding application that performs the methods described herein.

[0202] The apparatus 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display that combines a display with a touch-sensitive element, wherein the touch-sensitive element can be used to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0203] Although the bus 512 in the device 500 is described herein as a single bus, the bus 512 may include multiple buses. In addition, the auxiliary memory 514 may be directly coupled to other components in the device 500 or may be accessed through a network, and may include a single integrated unit (e.g., one memory card) or multiple units (e.g., multiple memory cards). Therefore, the device 500 may be implemented in a variety of configurations.

[0204] More specific, non-limiting and exemplary embodiments of the present invention are described below. Prior to this, some explanations are provided to help understand the present invention.

[0205] An artificial neural network (ANN) or connectionist system is a computing system vaguely inspired by the biological neural networks that make up animal brains. ANNs are based on a set of connected units or nodes called artificial neurons, which loosely simulate neurons in biological brains. Each connection, like a synapse in a biological brain, can transmit signals to other neurons. The artificial neuron that receives the signal processes it and can send signals to neurons connected to it. In ANN implementations, the "signal" at the connection is a real number, and the output of each neuron can be calculated as some nonlinear function of the sum of the inputs to the neuron. These connections are called edges. Neurons and edges usually have weights that are adjusted as learning proceeds. The weight increases or decreases the strength of the signal at the connection. Neurons can have a threshold so that the signal is sent only when the aggregate signal exceeds the threshold. Usually, neurons are clustered into layers. Different layers can perform different transformations on their inputs. The signal is transmitted from the first layer (input layer) to the last layer (output layer), but may be after traversing the layers many times.

[0206] The original purpose of the ANN approach was to solve problems in the same way as the human brain. Over time, the focus shifted to performing specific tasks, leading to deviations from biology. ANNs have been applied to a wide variety of tasks, including computer vision.

[0207] The name "convolutional neural network (CNN)" indicates that the network uses a mathematical operation called convolution. Convolution is a specialized linear operation. Convolutional networks are simply neural networks that use convolution instead of general matrix multiplication in at least one of their layers. Convolutional neural networks include input and output layers and multiple hidden layers. The input layer is the layer to which the input is provided for processing. For example, Figure 6The neural network in is a CNN. The hidden layers in a CNN usually consist of a series of convolutional layers that perform convolutions via multiplication or other dot products. The result of a layer is one or more feature maps, sometimes also called channels. Subsampling may be involved in some or all of the layers. As a result, the feature maps may become smaller. The activation function in a CNN can be a Rectified Linear Unit (RELU) layer or a GDN layer, as shown in the example above, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers, which are called hidden layers because the inputs and outputs of these layers are masked by the activation function and the final convolutional layer. Although these layers are colloquially called convolutional layers, this is just a convention. Mathematically speaking, convolution is technically a sliding dot product or cross-correlation. This has important implications for the indexing in the matrix, because the sliding dot product or cross-correlation affects how the weights are determined at specific index points.

[0208] When configuring a CNN to process images (picture / image), the input is a tensor of shape (number of images) × (image width) × (image height) × (image depth). Then, after passing through the convolutional layer, the image is abstracted into a feature map with the shape of (number of images) × (feature map width) × (feature map height) × (feature map channels). The convolutional layer within a neural network should have the following properties: a convolution kernel defined by width and height (hyperparameters); the number of input channels and output channels (hyperparameters). The depth of the convolutional filter (input channels) should be equal to the number of channels (depth) of the input feature map.

[0209] In the past, traditional multilayer perceptron (MLP) models have been applied to image recognition. However, due to the full connectivity between nodes, MLP models suffer from high dimensionality and do not scale well to high-resolution images. A 1000×1000 pixel image with RGB color channels has 3 million weights, which is too high to be efficiently and feasibly processed at scale with full connectivity. This network architecture also does not take into account the spatial structure of the data, resulting in input pixels that are far apart being processed in the same way as pixels that are close together. This computationally and semantically ignores the principle of locality in image data. Therefore, full connectivity of neurons is not useful for purposes such as image recognition where spatially local input patterns are dominant. CNN models mitigate the challenges posed by MLP architectures by exploiting the strong spatially local correlations present in natural images. The convolutional layer is the core building block in CNNs. The parameters of this layer consist of a set of learnable filters (the kernels mentioned above) that have a small receptive field but extend to the full depth of the input image. During the forward propagation, each filter is convolved by the width and height of the input image, computing the dot product between the elements in the filter and the input, and producing a two-dimensional activation map for the filter. Thus, the network learns filters that activate when the network detects a certain type of feature at a certain spatial location in the input.

[0210] Stacking the activation maps of all filters along the depth dimension forms the complete output image of the convolutional layer. Therefore, each element in the output image can also be interpreted as the output of a neuron that looks at a small area in the input and shares parameters with neurons in the same activation map. A feature map or activation map is the output activation of a given filter. Feature map has the same meaning as activation. In some papers, it is called an activation map because it is a map corresponding to the activations of different parts in the image, and it is also called a feature map because it is also a map indicating where a certain feature is found in the image. High activation means that a certain feature is found.

[0211] Another important concept of CNN is pooling, which is a form of nonlinear downsampling. There are several nonlinear functions that can implement pooling, the most common of which is max pooling. Max pooling divides the input image into a set of non-overlapping rectangles and outputs the maximum value of each sub-region. Intuitively, the exact location of a feature is not as important as its rough location relative to other features. This is the reasoning behind the use of pooling in convolutional neural networks. Pooling layers are used to gradually reduce the spatial size of the representation in order to reduce the number of parameters, memory footprint, and computation in the network, and are therefore also used to control overfitting. In CNN architectures, it is common to insert pooling layers regularly between consecutive convolutional layers. The pooling operation provides another form of translation invariance.

[0212] The above ReLU is the abbreviation of rectified linear unit, which uses a non-saturated activation function. ReLU effectively removes negative values ​​from the activation map by setting them to 0. It increases the nonlinearity of the decision function and the overall network without affecting the receptive field of the convolutional layer. Other functions are also used to increase nonlinearity, for example, saturated hyperbolic tangent and sigmoid functions. ReLU is generally preferred over other functions because it trains neural networks several times faster without significantly reducing generalization accuracy.

[0213] After several convolutional and max pooling layers, high-level reasoning in the neural network is achieved through fully connected layers. Neurons in a fully connected layer have connections to all activations in the previous layer, as in regular (non-convolutional) artificial neural networks. Therefore, these activations can be calculated as an affine transformation, which consists of a matrix multiplication followed by a bias shift (a vector addition of a learned or fixed bias term).

[0214] An autoencoder is an artificial neural network used to learn efficient data decoding in an unsupervised manner. The purpose of an autoencoder is to learn a representation (encoding) of a set of data by training the network to ignore the signal "noise", usually for dimensionality reduction. When performing dimensionality reduction on the original input, it also learns how to reconstruct the original input from the reduced object, where the autoencoder tries to generate a representation as close to its original input as possible from the reduced encoding, hence the name.

[0215] Image size: refers to the width or height of an image or a width-height pair. The width and height of an image are usually measured in units of the number of brightness samples.

[0216] Downsampling: Downsampling is the process of reducing the sampling rate (sampling interval) of a discrete input signal. For example, if the input signal is an image of size height h and width w (or H and W mentioned below), and the downsampled output is height h2 and width w2, then at least one of the following is true:

[0217] h2 <h

[0218] w2 <w

[0219] In one exemplary implementation, downsampling may be implemented by keeping only every m samples, discarding the rest of the input signal (in the context of the present invention, the input signal is essentially an image).

[0220] Upsampling: Upsampling is the process of increasing the sampling rate (sampling interval) of a discrete input signal. For example, if the input image size is h and w (or H and W mentioned below), and the downsampled output is h2 and w2, then at least one of the following is true:

[0221] ·h <h2

[0222] ·w <w2

[0223] Resampling: Both downsampling and upsampling processes are examples of resampling. Resampling is the process of changing the sampling rate (sampling interval) of the input signal.

[0224] Interpolation filtering: During upsampling or downsampling, filtering can be applied to improve the accuracy of the resampled signal and reduce the effects of aliasing. Interpolation filters typically consist of a weighted combination of sample values ​​at sample positions surrounding the resampled position, where the relationship can be defined as follows:

[0225] f(x r ,y r )=∑s(x,y)C(k)

[0226] Where f() is the resampled signal, (x r ,y r ) is the resampled coordinate, C(k) is the interpolation filter coefficient, and s(x,y) is the input signal. r ,y r ) performs a sum operation on the (x,y) nearby.

[0227] Cropping: Trimming the outer edges of a digital image. Cropping can be used to make an image smaller (in terms of the number of samples) and / or to change the image's aspect ratio (ratio of length to width).

[0228] Padding: Padding refers to increasing the size of an input image (or images) by generating new samples at the image boundaries. For example, this can be achieved by using predefined sample values ​​or using sample values ​​at locations in the input image.

[0229] Resizing: Resizing is a general term for changing the size of an input image. Resizing can be achieved using one of the methods of padding or cropping. Resizing can be achieved by using a resizing operation that uses interpolation. In the following, resizing may also be referred to as rescaling.

[0230] Integer division: Integer division is division in which the fractional part (remainder) is discarded.

[0231] Convolution: Convolution is given by the following general equation. In the following, f() can be defined as the input signal and g() can be defined as the filter.

[0232]

[0233] Downsampling layer: is a processing layer, for example, a layer in a neural network that reduces at least one dimension of the input. Generally speaking, the input may have 3 or more dimensions, where these dimensions may include the number of channels, width and height. However, the present invention is not limited to such signals. Instead, signals with one or two dimensions (for example, audio signals or audio signals with multiple channels) can be processed. A downsampling layer generally refers to reducing the width and / or height dimensions. This can be achieved through operations such as convolution, averaging, and maximum pooling. Other downsampling methods can also be used, and the present invention is not limited in this regard.

[0234] Upsampling layer: A processing layer, for example, a layer in a neural network that increases one of the dimensions of the input. Generally, the input may have 3 or more dimensions, where these dimensions may include the number of channels, width, and height. An upsampling layer generally refers to increasing the width and / or height dimensions. This can be achieved by operations such as deconvolution, replication, etc. Other upsampling methods may also be used, and the present invention is not limited in this respect.

[0235] Some deep learning based image and video compression algorithms follow the variational auto-encoder (VAE) framework, for example, G-VAE: A Continuously Variable Rate Deep Image Compression Framework (Ze Cui, Jing Wang, Bo Bai, Tiansheng Guo, Yihui Feng), which can be viewed via https: / / arxiv.org / abs / 2003.02012.

[0236] The VAE framework can be viewed as a nonlinear transform decoding model.

[0237] The transformation process can be mainly divided into four parts. Figure 4 The VAE framework is illustrated in Figure 4In the example, encoder 601 maps the input image x to a priori representation (represented by y) by function y=f(x). In the following, this priori representation may also be referred to as a portion or a point in the "prior space". Function f() is a transformation function that converts the input signal x into a highly compressible representation y. Quantizer 602 Transform the prior representation y into a quantized prior representation with (discrete) values Where Q represents the quantizer function. The entropy model or super encoder / decoder (also called super prior) 603 estimates the quantized prior representation distribution to obtain the minimum rate achievable by lossless entropy source decoding.

[0238] The prior space can be understood as a compressed representation of the data, where similar data points are closer together in the prior space. The prior space is helpful for learning data features and finding simpler data representations for analysis.

[0239] The quantized prior representation T, and the side information of the super prior 3 It is included in the code stream 2 (binarized) through arithmetic coding (AE).

[0240] Furthermore, a decoder 604 is provided which transforms the quantized prior representation into a reconstructed image Signal is the estimated value of the input image x. We hope that x is as close as possible In other words, the reconstruction quality is as high as possible. However, The higher the similarity between and x, the more side information needs to be transmitted. Side information includes Figure 4 The bitstream 1 and bitstream 2 shown in FIG. 1 are generated by the encoder and transmitted to the decoder. Generally, the more side information, the higher the reconstruction quality. However, a large amount of side information means a low compression ratio. Therefore, Figure 4 One goal of the system described in is to balance the quality of reconstruction and the amount of side information represented in the codestream.

[0241] exist Figure 4 , component AE 605 is an arithmetic coding module, which converts the quantized prior representation and side information The samples are converted into binary representation code stream 1. and The samples may, for example, comprise integers or floating point numbers. One purpose of the arithmetic coding module is to convert (by a binarization process) the sample values ​​into a string of binary bits (which is then included in a codestream which may include further parts or more side information corresponding to the coded image).

[0242] Arithmetic decoding (AD) 606 is the process of restoring the binarization process, where the binary bits are converted back to sample values. Arithmetic decoding is performed by the arithmetic decoding module 606.

[0243] It should be noted that the present invention is not limited to this specific framework. In addition, the present invention is not limited to image or video compression, and can also be applied to object detection, image generation and recognition systems.

[0244] exist Figure 4 In , there are two subnetworks cascaded to each other. A subnetwork in this context is a logical division between parts of the overall network. For example, in Figure 4 In FIG. 6 , modules 601 , 602 , 604 , 605 and 606 are called “encoder / decoder” subnetworks. The “encoder / decoder” subnetwork is responsible for encoding (generating) and decoding (parsing) the first bitstream “bitstream 1”. Figure 4 The second network in includes modules 603, 608, 609, 610 and 607, which is called the "super encoder / decoder" subnetwork. The second subnetwork is responsible for generating the second bitstream "Bitstream 2". The two subnetworks have different purposes. The first subnetwork is responsible for:

[0245] · convert 601 the input image x to its prior representation y (this makes it easier to compress x),

[0246] Quantize 602 the prior representation y to a quantized prior representation

[0247] · Through the arithmetic coding module 605, the quantized prior representation is compressed using AE To obtain the code stream "code stream 1",

[0248] · Through the arithmetic decoding module 606, use AD to parse the code stream 1,

[0249] Reconstructing 604 the reconstructed image using the parsed data

[0250] The purpose of the second sub-network is to obtain the statistical characteristics of the samples in "code stream 1" (for example, the mean, variance and correlation between the samples in code stream 1), so that the first sub-network can compress code stream 1 more efficiently. The second sub-network generates a second code stream "code stream 2", which includes the information (for example, the mean, variance and correlation between the samples in code stream 1).

[0251] The second network includes an encoding part, which includes converting the quantized prior representation Transform 603 into side information z, and quantize the side information z into quantized side information And the quantized side information The encoding (e.g., binarization) 609 is performed into the bitstream 2. In this example, the binarization is performed by arithmetic encoding (AE). The decoding part of the second network includes arithmetic decoding (AD) 610, which transforms the input bitstream 2 into decoded and quantized side information. Possibly The same as arithmetic coding and decoding operations are lossless compression methods. Next, the decoded and quantized side information is Transform 607 into decoded side information express The statistical properties of The average value of the samples in , or the variance of the sample values, etc.). Then, the decoded prior representation Provided to the above-mentioned arithmetic encoder 605 and arithmetic decoder 606 to control The probability model of .

[0252] Figure 4 An example of a variational auto encoder (VAE) is described, the details of which may vary in different implementations. For example, in a specific implementation, there may be other components to more efficiently obtain statistical characteristics of samples in bitstream 1. In one such implementation, there may be a context modeler whose purpose is to extract inter-correlated information in bitstream 1. The statistical information provided by the second subnetwork can be used by the arithmetic encoder (AE) 605 and the arithmetic decoder (AD) 606 components.

[0253] Figure 4 The encoder and decoder are depicted in a single figure. It is clear to those skilled in the art that the encoder and decoder can be, and often are, embedded in mutually distinct devices.

[0254] Figure 7 describes the encoder, Figure 8 The decoder component in the VAE framework is described separately. Figure 7 and Figure 8 The explanation can also be applied to the following specific combination Fig.19 , Fig. 22 , Fig.25 and Fig.26 Further provided are the neural network and encoder and decoder.

[0255] According to some embodiments, the encoder receives an image as input. The input image may include one or more channels, such as color channels or other types of channels, such as depth channels or motion information channels. The encoder (e.g. Figure 7 The output of the encoder is bitstream 1 and bitstream 2. Bitstream 1 is the output of the first sub-network of the encoder, and bitstream 2 is the output of the second sub-network of the encoder.

[0256] Similarly, in Figure 8 In the example, two code streams, code stream 1 and code stream 2, are received as input, and two code streams, code stream 1 and code stream 2, are generated at the output. That is, the reconstructed image (decoded image).

[0257] As mentioned above, VAE can be divided into different logical units that perform different actions. Figure 7 and Figure 8 As an example, Figure 7 Components involved in encoding a signal, which may be a video, are described, wherein the signal encoding provides encoded information. Then, Figure 8 The decoder components described in receive the encoded information to perform encoding, etc. It should be noted that the components in the encoder and decoder represented by the numbers 9xx and 10xx may correspond to those described above in terms of their functions. Figure 4 Components mentioned in the documentation are denoted by numbers 6xx.

[0258] Specifically, Figure 7 As shown, the encoder includes an encoder 901, which transforms the input x into a signal y, and then provides the signal y to a quantizer 902. The quantizer 902 provides information to an arithmetic coding module 905 and a super encoder 903. The super encoder 903 provides the code stream 2 discussed above to a super decoder 907, and the super decoder 907 sends information to the arithmetic coding module 905.

[0259] The encoding can be done by convolution, as shown below: Fig.19 Decoding can be done by using deconvolution, as shown below combined with Fig.19 and Fig. 22 Describe in detail.

[0260] The output of the arithmetic coding module is code stream 1. Code stream 1 and code stream 2 are the outputs of signal encoding, both of which are then provided (transmitted) to the decoding process.

[0261] Although unit 901 is referred to as an "encoder", it is also possible to Figure 7 The complete subnetwork described in is called an "encoder". The encoding process usually refers to a unit (module) that converts an input into an encoded (e.g., compressed) output. Figure 7As can be seen in , unit 901 can actually be considered the core of the entire sub-network, because unit 901 converts input x into y, which is a compressed version of x. The compression in encoder 901 can be achieved by applying a neural network or generally by any processing network including one or more layers. In such a network, compression can be performed by a cascade process including downsampling, which reduces the size and / or number of channels of the input. Therefore, the encoder can be called a neural network (NN) based encoder, etc.

[0262] The remaining parts of the figure (quantization unit, super encoder, super decoder, arithmetic encoder / decoder) are all parts that improve the efficiency of the encoding process or are responsible for converting the compressed output y into a series of bits (bitstream). Quantization can be provided to further compress the output of the NN encoder 901 through lossy compression. The AE 905 in conjunction with the super encoder 903 and super decoder 907 used to configure the AE 905 can perform binarization, which can further compress the quantized signal through lossless compression. Therefore, it is also possible to Figure 7 The entire subnetwork in is called the "encoder".

[0263] Most image / video compression systems based on deep learning (DL) reduce the dimensionality of the signal before converting it into binary bits. For example, in the VAE framework, the encoder is a nonlinear transformation that maps the input image x to y, where the width and height of y are smaller than the width and height of x. Since y has a smaller width and height, it is smaller in size, which reduces the dimensionality (size) of the signal, so it is easier to compress the signal y. It should be noted that, in general, the encoder does not necessarily need to reduce the size in two (or usually all) dimensions. On the contrary, some exemplary implementations may provide an encoder that reduces the size in only one dimension (or usually a subset of dimensions).

[0264] The general principle of compression is as follows Figure 5 Let's take an example. The prior space is the output of the encoder and the input of the decoder, representing the compressed data. It should be noted that the size of the prior space can be much smaller than the input signal size. Here, the term "size" can refer to the resolution, for example, the number of samples in one or more feature maps output by the encoder. The resolution can be calculated as the product of the number of samples in each dimension (for example, width × height × number of channels of the input image or feature map).

[0265] The input signal size is reduced as Figure 5 For example, Figure 5 The deep learning based encoder and decoder are shown. Figure 5In , the input image x corresponds to the input data, which is the input to the encoder. The transformed signal y corresponds to a prior space that has a smaller dimension or size than the input signal in at least one dimension. Each column of circles represents a layer in the processing chain of the encoder or decoder. The number of circles in each layer indicates the size or dimension of the signal in that layer.

[0266] from Figure 5 As can be seen in , the encoding operation corresponds to a reduction in the size of the input signal, while the decoding operation corresponds to the reconstruction of the original size of the image.

[0267] One of the ways to reduce the size of a signal is by downsampling. Downsampling is the process of reducing the sampling rate of the input signal. For example, if the input image is of size h and w, and the downsampled output is h2 and w2, then at least one of the following is true:

[0268] h2 <h

[0269] w2 <w

[0270] The reduction in signal size usually occurs stepwise along a chain of processing layers, rather than all at once. For example, if the dimensions (or size of the dimensions) of the input image x are h and w (indicating height and width), and the dimensions of the prior space y are h / 16 and w / 16, then during the encoding process, the size reduction may occur over 4 layers, where each layer reduces the size of the signal by 1 / 2 in each dimension.

[0271] Some deep learning based video / image compression methods use multiple downsampling layers. For example, Figure 6 The VAE framework in uses 6 downsampling layers labeled 801 to 806. Layers that include downsampling are indicated by downward arrows in the layer description. The layer description "Conv N×5×5 / 2↓" indicates that the layer is a convolutional layer with N channels and a kernel size of 5×5. As mentioned above, 2↓ indicates that a downsampling factor of 2 is performed in this layer. Downsampling by a factor of 2 causes one of the dimensions of the input signal to be reduced by a factor of two at the output. Figure 6 In the example, 2↓ indicates that both the width and height of the input image are reduced by 1 / 2. Since there are 6 downsampling layers, if the width and height of the input image 814 (also denoted by x) are given by w and h, then the output signal The width and height of 813 are w / 64 and h / 64 respectively. The modules represented by AE and AD are the arithmetic encoder and arithmetic decoder, which have been combined above. Figure 4 , Figure 7 and Figure 8The arithmetic encoder and arithmetic decoder are specific implementations of entropy decoding. AE and AD (as part of components 813 and 815) can be replaced by other entropy decoding methods. In information theory, entropy coding is a lossless data compression scheme for converting the value of a symbol into a binary representation (a reversible process). In addition, the "Q" in the figure corresponds to the above combined Figure 4 The quantization operation mentioned above is further explained in the "Quantization" section above. In addition, the quantization operation and the corresponding quantization unit belonging to the component 813 or 815 do not necessarily exist and / or can be replaced by other units.

[0272] exist Figure 6 , a decoder including upsampling layers 807 to 812 is also shown. Another layer 820 is provided between upsampling layers 811 and 810 in the order in which the inputs are processed, this layer being implemented as a convolutional layer but providing no upsampling of the received inputs. A corresponding convolutional layer 830 is also shown for the decoder. These layers may be provided in the NN to perform operations on the inputs that do not change the size of the inputs but change certain features. However, such layers need not be provided.

[0273] When viewed in the order in which the bitstream 2 is processed by the decoder, the upsampling layers are run in the reverse order (i.e., from upsampling layer 812 to upsampling layer 807). Each upsampling layer is shown here to provide an upsampling with an upsampling ratio of 2, indicated by ↑. Of course, not all upsampling layers necessarily have the same upsampling ratio, and other upsampling ratios, such as 3, 4, 8, etc., may also be used. Layers 807 to 812 are implemented as convolutional layers (conv). Specifically, since these layers may be intended to provide an operation on the input that is opposite to the operation of the encoder, the upsampling layers may apply a deconvolution operation to the received input so that the size of the input is increased by a factor corresponding to the upsampling ratio. However, the present invention is generally not limited to deconvolution, and upsampling may be performed in any other manner, such as by bilinear interpolation between two adjacent samples, or by nearest neighbor sample replication, etc.

[0274] In the first subnetwork, some convolutional layers (801 to 803) are followed by generalized divisive normalization (GDN) at the encoding end, and by inverse GDN (IGDN) at the decoding end. In the second subnetwork, the activation function applied is ReLu. It should be noted that the present invention is not limited to this implementation, and other activation functions can generally be used instead of GDN or ReLu.

[0275] Image and video compression systems usually cannot handle arbitrary input image sizes. The reason is that some processing units in the compression system (e.g., transform units or motion compensation units) operate on minimum units and cannot process the input image if the size is not an integer multiple of the minimum processing unit.

[0276] For example, HEVC specifies four transform unit (TU) sizes of 4×4, 8×8, 16×16, and 32×32 to decode the prediction residual. Since the minimum transform unit size is 4×4, it is not possible to use the HEVC encoder and decoder to process an input image of size 3×3. Similarly, if the image size is not a multiple of 4 in one dimension, the image cannot be processed because it is impossible to split the image into sizes that can be processed by valid transform units (4×4, 8×8, 16×16, and 32×32). Therefore, the HEVC standard requires that the input image must be a multiple of the minimum decoding unit size, that is, 8×8. Otherwise, the input image cannot be compressed by HEVC. Other codecs have also made similar requirements. In order to utilize existing hardware or software, or to maintain some interoperability or even retain parts of existing codecs, it may be desirable to maintain this restriction. However, the present invention is not limited to any particular transform block size.

[0277] Some image and video compression systems based on deep neural networks (DNN) or neural networks (NN) use multiple downsampling layers. Figure 6 , 4 downsampling layers are included in the first subnetwork (layers 801 to 804), and 2 additional downsampling layers are included in the second subnetwork (layers 805 to 806). Therefore, if the size of the input image is given by w and h (indicating width and height), respectively, the output of the first subnetwork is w / 16 and h / 16, and the output of the second network is w / 64 and h / 64.

[0278] The term "depth" in deep neural networks usually refers to the number of processing layers applied to the input in sequence. When the number of layers is higher, the neural network is called a deep neural network, but there is no clear description or guidance as to which networks should be called deep networks. Therefore, for the purpose of this application, there is no significant difference between DNN and NN. DNN can refer to a NN that includes more than one layer.

[0279] During the downsampling process, for example, when convolution is applied to the input, a fractional (final) size of the encoded image may be obtained in some cases. Such fractional sizes cannot be reasonably handled by subsequent layers or decoders in the neural network.

[0280] In other words, some downsampling operations (e.g., convolution) may expect (e.g., by design) that the size of the input to a particular layer in the neural network satisfies certain conditions, such that the operation performed in the neural network performing the downsampling or within a layer after performing the downsampling is still a well-defined mathematical operation. For example, for a network with a downsampling ratio r>1 (that is, the sampling ratio is an integer value greater than 1) A downsampling layer that reduces the size of the input in at least one dimension by a downsampling ratio r. If the size of the input in that dimension is an integer multiple of the downsampling ratio r, a reasonable output is obtained. Downsampling by a downsampling ratio r means dividing the number of input samples in one dimension (e.g., width) or multiple dimensions (e.g., width and height) by the downsampling ratio (e.g., 2 in the case of r=2) to obtain the number of output samples.

[0281] To provide a numerical example, the downsampling ratio of a layer may be 4. The first input has a size of 512 in the dimension to which downsampling is applied. 512 is an integer multiple of 4, since 128×4=512. Therefore, processing of the input can be performed by the downsampling layer, thereby producing a reasonable output. The second input may have a size of 513 in the dimension to which downsampling is applied. 513 is not an integer multiple of 4, and therefore, if the downsampling layer or a subsequent upsampling layer wants a specific input size (e.g., 512) by design or the like, the input cannot be reasonably processed by the downsampling layer or the subsequent upsampling layer. In view of this, even if the size of the input is not always the same, in order to ensure that the input can be processed by each layer in the neural network in a reasonable manner (conforming to the predefined layer input size), scaling (also called resizing) may be applied before the input is processed by the neural network. This scaling includes changing or adjusting the actual size of the input of the neural network (e.g., the input layer of the neural network) so that the size satisfies the above-mentioned conditions for all downsampling layers in the neural network. This scaling is achieved by increasing or decreasing the size of the input in the dimension to which downsampling is applied, so that the size S=K∏ i r i , where r i is the downsampling ratio of the downsampling layer, and K is an integer greater than 0. In other words, the input size of the input image (signal) in the downsampling direction is adjusted to an integer multiple of the product of all downsampling ratios applied to the input image (signal) in the network processing chain in the downsampling direction (dimension).

[0282] Therefore, the size of the input to the neural network ensures that each layer can process its own input according to the predefined input size configuration of a layer.

[0283] However, by performing such scaling, the amount of reduction in the size of the image to be encoded is limited, and accordingly, the size of the encoded image that can be provided to the decoder for the purpose of reconstructing the encoded information, etc., is also limited. In addition, with the methods provided so far, a large amount of entropy may be added to the code stream (when the size of the code stream is increased by scaling), or a large amount of information may be lost (if the size of the code stream is reduced by scaling). Both of these will reduce the quality of the code stream after decoding.

[0284] Therefore, it is difficult to obtain high-quality encoded / decoded code streams and the data they represent while providing a reduced-size encoded code stream.

[0285] Since the output of a layer in the network cannot be of fractional size (it needs to have an integer number of rows and columns of samples), there is a limit on the input image size. Figure 6 In order to ensure reliable processing, the input image size is an integer multiple of 64 in both horizontal and vertical directions. Otherwise, the output of the second network will not be an integer.

[0286] To solve this problem, the input image can be padded with 0 to make it a multiple of 64 samples in each direction. According to this technical solution, the input image size can be expanded in width and height by the following amount:

[0287]

[0288]

[0289] Where, "Int" is an integer conversion. The integer conversion can calculate the quotient of the first value a and the second value b, and then can provide an output ignoring all decimal places, so it is just an integer. The newly generated sample value can be set to 0.

[0290] Another possibility to solve the above problem is to crop the input image, that is, discard the samples of each row and column from the end of the input image so that the input image size is a multiple of 64 samples. The minimum number of rows and columns of samples that need to be cropped can be calculated as follows:

[0291]

[0292]

[0293] Among them, w diff and w diff They correspond to the number of rows and columns of samples that need to be discarded from each side of the image.

[0294] According to the above content, the input image is in horizontal (h new ) and vertical (w new ) dimension is as follows:

[0295] In case of padding:

[0296] ·h new =h+h diff

[0297] ·w new =w+w diff

[0298] In case of clipping:

[0299] ·h new =hh diff

[0300] ·w new =w+w diff

[0301] This is also Fig.10 and Fig.11 In Fig.10 In the figure, it is shown that the encoder and decoder (collectively represented by 1200) may include multiple downsampling and upsampling layers. Each layer applies a downsampling factor of 2 or an upsampling factor of 2. In addition, the encoder and decoder may include more components, for example, a generalized divisive normalization (GDN) 1201 at the encoding end and an inverse GDN (IGDN) 1202 at the decoding end. In addition, both the encoder and the decoder may include one or more ReLu, specifically a leaky ReLu (leaky ReLu) 1203. A factorized entropy model 1205 may also be provided at the encoder, and a Gaussian entropy model 1206 may be provided at the decoder. In addition, multiple convolution masks 1204 may be provided. In addition, in Fig.10 and Fig.11 In the embodiment of , the encoder includes a universal quantizer (UnivQuan) 1207, and the decoder includes an attention module 1208. For ease of reference, the functionally corresponding components are Fig.11 There are corresponding numbers in .

[0302] The total number of downsampling operations and the stride size conditions the input channel size, i.e. the size of the input to the neural network.

[0303] Here, if the input channel size is an integer multiple of 64 = 2×2×2×2×2×2×2, then after all consecutive downsampling operations, the channel size remains integer. By applying the corresponding upsampling operation in the decoder during the upsampling process, and by applying the same scaling to the input after processing through the upsampling layer (e.g., using the FWD resizing module shown in this figure), the output size is again the same as the input size at the encoder.

[0304] Thus, a reliable reconstruction of the original input is achieved.

[0305] exist Fig.11 In the figure, it is shown that Fig.10 The example also shows an encoder and a decoder, collectively denoted by 1300. The m downsampling layers (and corresponding upsampling layers) have a downsampling ratio s i and the corresponding upsampling ratio. Here, if the input channel size is If the channel size is an integer multiple of , then after all m consecutive (also called consecutive or subsequent or cascaded) downsampling operations, the channel size remains an integer. Fig.11 The FWD resizing module shown in FIG. 1 scales the input accordingly before processing it, ensuring that the above equation is satisfied. In other words, the input channel size in the downsampling direction is the product of all downsampling ratios applied to the input by the corresponding m downsampling layers in the (sub)network.

[0306] The pattern of changing the input size as described above may still have some disadvantages.

[0307] exist Figure 6 In the example, the sizes of the code streams indicated by "code stream 1" and "code stream 2" are respectively equal to: and A and B are scalar parameters that describe the compression ratio. The higher the compression ratio, the smaller the numbers A and B. Therefore, the total size of the code stream is

[0308] Since the purpose of compression is to reduce the size of the bitstream while maintaining high quality reconstructed images, it is obvious that h new and w new Should be as small as possible to reduce the bitrate.

[0309] Therefore, the problem caused by "0 padding" is the increase in bit rate due to the increase in input size. In other words, the size of the input image is increased by adding redundant data to the input image, which means that more side information must be transmitted from the encoder to the decoder to reconstruct the input signal. Therefore, the size of the code stream is increased.

[0310] For example, using Figure 6 For the encoder / decoder pair in , if the input image size is 416×240, which is an image size format commonly known as Wide Quarter Video Graphics Array (WQVGA), the input image must be padded to be equal to size 448×256, which is equivalent to a 15% increase in bit rate due to the inclusion of redundant data.

[0311] The problem with the second approach (cropping the input image) is that information is lost. Since the goal of compression and decompression is to transmit the input signal while maintaining high fidelity, discarding parts of the signal defeats that purpose. Cropping is therefore disadvantageous unless it is known that some parts of the input signal are unwanted, which is often not the case.

[0312] According to one embodiment, resizing of the input image is performed before each downsampling or upsampling layer of the DNN-based image or video compression system. More specifically, if the downsampling layer has a downsampling ratio of 2 (the input size is halved at the output of the layer), if the input of the layer has an odd number of sample rows or columns, input resizing is applied to the input, and if the number of sample rows or columns is even (a multiple of 2), no padding is applied.

[0313] Furthermore, if the corresponding downsampling layer applies resizing at (its) input, the resizing operation can be applied at the end, e.g., at the output of the upsampling layer. The corresponding layer of the downsampling layer can be found by counting the number of upsampling layers starting from the reconstructed image and the number of downsampling layers starting from the input image. This is explained in Fig.18 Take an example, where upsampling layer 1 and downsampling layer 1 are corresponding layers, upsampling layer 2 and downsampling layer 2 are corresponding layers, and so on.

[0314] The resizing operation applied at the input of the downsampling layer and the resizing operation applied at the output of the upsampling layer are complementary so that the data size at both outputs remains the same.

[0315] Therefore, the increase in code stream size is minimized. Fig. 9 In contrast, it can be combined Fig.12 An exemplary embodiment is explained. Fig. 9 In , input resizing is performed before the input is provided to the DNN, and allows the resized input to be processed by the entire DNN. Fig. 9 The example shown can be used Figure 6 The encoder / decoder described in is implemented.

[0316] exist Fig.12 In the embodiment, an input image of any size is provided to the neural network. The neural network in this embodiment includes N downsampling layers, each layer i (1 <= i <= N) has a downsampling ratio r i . “<=” means less than or equal to. For different values ​​of i, the downsampling ratio r i are not necessarily the same, but in some embodiments they may all be equal, for example, they may all be r i = r = 2. Fig.12In FIG. 1 , downsampling layers 1 to M are summarized as subnetwork 1 of downsampling layers. Subnetwork 1 provides codestream 1 as output. However, in this case, the summary of downsampling layers is only for descriptive purposes. A second subnetwork 2 including layers M+1 to N provides codestream 2 as output.

[0317] In this example, before the input to a downsampling layer (e.g., downsampling layer M) is provided to the downsampling layer, but after the input has been processed by the previous downsampling layer (layer M–1 in this case), the input is resized by applying a resizing operation so that the input to downsampling layer M has size r M represents the downsampling ratio of the downsampling layer M and may be a preset value and therefore already available to the decoder. In this example, this resizing operation is performed before each downsampling layer so that the specific downsampling layer and its corresponding downsampling ratio satisfy the above conditions. In other words, the size S is adjusted or set to an integer multiple of the downsampling ratio of the next layer (after downsampling in the processing order).

[0318] exist Fig. 9 In , the input image is padded (which is a form of image resizing) to account for all the downsampling layers that will process the data one after another. Fig. 9 In , for illustration purposes, the downsampling ratio is exemplarily chosen to be 2. In this case, since there are N layers performing downsampling with a downsampling ratio of 2, the input image size is adjusted to 2 by padding (with 0) N It should be noted that, in this article, the integer "times" can still be equal to 1, that is, times has the meaning of multiplication (for example, one or more times) rather than the meaning of a plural number.

[0319] Fig.12 An embodiment is shown. Fig.12 In , input resizing is applied before each downsampling layer. The input is resized to an integer multiple of the downsampling ratio of each layer. For example, if a layer has a downsampling ratio of 3:1 (input size: output size) and the ratio is 3, then the input of this layer is resized to a multiple of 3.

[0320] Some examples can also be applied to Figure 6 .exist Figure 6 In , there are 6 downsampling layers, namely layers 801, 802, 803, 804, 805, and 806. The factor of all downsampling layers is 2. According to one example, input resizing is applied before all 6 layers. Figure 6, resizing is also applied after each of the upsampling layers (807, 808, 809, 810, 811, and 812) in a corresponding manner (this is explained in the previous paragraph). This means that the resizing applied before the downsampling layer at a specific order or position in the neural network of the encoder is applied at the corresponding position in the decoder.

[0321] In some embodiments, there may be two options for scaling input, and one of the options may be selected based on circumstances or conditions, etc., as described in detail below. Figures 13 to 15 Give a description.

[0322] The first option 1501 may include padding the input with zeros or redundant information in the input itself, etc., to increase the size of the input to a size that matches an integer multiple of the downsampling ratio. At the decoding end, for scaling, cropping may be used in this option to reduce the size of the input to a size that matches the target input size of the successive upsampling layers, etc.

[0323] This option can be implemented computationally efficiently, but only at the expense of increased size on the encoding side.

[0324] The second option 1502 can scale / adjust the input using interpolation at the encoder and interpolation at the decoder. This means that interpolation can be used to increase the size of the input to an expected size, such as an integer multiple of the downsampling ratio of all downsampling layers or a target input size for all upsampling layers, or interpolation can be used to reduce the size of the input to an expected size, such as an integer multiple of the combined downsampling ratio of all downsampling layers in the NN or a target input size for all upsampling layers in the NN. Thus, resizing can be applied at the encoder by increasing or decreasing the size of the input. In addition, in option 1502, different interpolation filters can be used, thereby providing spectral characteristic control.

[0325] Different options 1501 and 1502 may be indicated as side information in a code stream or the like. The difference between the first option (option 1) 1501 and the second option (option 2) 1502 may be indicated by an indication such as a syntax element methodIdx, which may take one of two values. For example, a first value (e.g., 0) is used to indicate padding / cropping, and a second value (e.g., 1) is used to indicate interpolation for resizing. For example, a decoder may receive a code stream for encoding an image, which may include side information with an element methodIdx. After parsing this code stream, the side information may be obtained and the value of methodIdx may be derived. Based on the value of methodIdx, the decoder may proceed with the corresponding resizing or scaling method, using padding / cropping if methodIdx has a first value, and using interpolation if methodIdx has a second value.

[0326] This is Fig.13 As shown in . Depending on whether the value of methodIdx is 0 or 1, limiting (including filling or clipping) or interpolation is selected.

[0327] It should be noted that even if Fig.13 The embodiment in refers to selecting or determining clipping (including one of padding / cropping) or interpolation as a method for implementing resizing based on methodIdx, and the present invention is not limited in this regard. Fig.13 The method of interpretation may also be implemented where the first option 1501 is an interpolation that increases the size during the resizing operation and the second option 1502 is an interpolation that decreases the size during the resizing operation. The selection may be made between any two or even more (depending on the binary size of methodIdx) different resizing methods explained above and below and may be indicated using methodIdx. In general, methodIdx need not be a separate syntax element. methodIdx may be indicated or decoded in unison with one or more other parameters.

[0328] More instructions or signs can be Fig.14 In addition to methodIdx, a size change flag (1 bit) SCIdx may also be conditionally indicated only in the case of the second option 1502. Fig.14 In the embodiment of , the second option 1502 includes using interpolation to implement the resizing. Fig.14 , the second option 1502 is selected when methodIdx=1. The size change flag SCIdx may have a third value or a fourth value, which may be a value of 0 (for example, the third value is 0) or a value of 1 (for example, the fourth value is 1). In this embodiment, "0" may represent reduction and "1" may represent enlargement. Therefore, if SCIdx is 0, the interpolation used to implement resizing is implemented in a manner that reduces the input size. If SCIdx is 1, the interpolation used to implement resizing is implemented in a manner that increases the input size. Conditional decoding of SCIdx can provide a more concise and efficient syntax. However, the present invention is not limited to such conditional syntax, and SCIdx may be indicated separately from methodIdx, or uniformly indicated (decoded) with methodIdx (for example, in a common syntax element, wherein the common syntax element can only extract a subset of values ​​from the values ​​indicating all combinations of SCIdx and methodIdx).

[0329] Like the methodIdx indicator, SCIdx can also be obtained by the decoder by parsing the bitstream, which may also decode the image to be reconstructed. After obtaining the value of SCIdx, you can choose to reduce or enlarge it.

[0330] In addition to or in lieu of the above instructions, Fig.15 As shown, other (edge) indications for resizing filter index RFIdx may be indicated (indicated in the code stream).

[0331] In some embodiments, for the second option 1502, RFIdx can be conditionally indicated, which can include: if methodIdx=1, then RFIdx is indicated, if methodIdx=0, then RFIdx is not indicated. RFIdx can have a size of more than one bit, and can indicate which interpolation filter is used in the interpolation to achieve resizing based on its value, etc. Alternatively or in addition, RFIdx can specify filter coefficients of multiple interpolation filters. For example, this can be Bilinear, Bicubic, Lanczos3, Lanczos5, Lanczos8, etc.

[0332] As described above, at least one or all or at least two of methodIdx, SCIdx and RFIdx may be provided in a code stream, which may be a code stream that also encodes the image to be reconstructed, or other code streams. Then, the decoder may parse the corresponding code stream and obtain the values ​​of methodIdx and / or SCIdx and / or RFIdx. Depending on the values, the above actions may be performed.

[0333] The filter used to implement the resized interpolation may be determined by the scaling ratio, etc.

[0334] like Fig.15 The value of RFIdx may be explicitly indicated as indicated by the mark 1701 in the lower right corner. Alternatively or additionally, RFIdx may be obtained from a lookup table such that RFIdx=LUT(SCIdx).

[0335] As another example, there may be two lookup tables, one for the case of zooming in and the other for the case of zooming out. In this case, LUT1 (SCIdx) may indicate the resizing filter when zooming out, and LUT2 (SCIdx) may indicate the resizing filter when zooming in.

[0336] Generally speaking, the present invention is not limited to any specific indication method of RFIdx. RFIdx can be indicated separately from other elements or indicated together.

[0337] The above mentioned indications methodIdx, SCIdx, RFIdx serve as a nested structure, wherein the presence or absence of SCIdx and RFIdx may depend on the value of methodIdx.However, each of methodIdx, SCIdx and RFIdx may be provided individually even if one or more other indications are not provided.

[0338] In addition, according to some embodiments, as a substitute or supplement to these indications, another indication may be provided, wherein the indication is or includes an index indicating an entry in a lookup table. The lookup table (look-up table, LUT) may include multiple entries, each of which specifies a resizing method. There may be entries in the LUT that specify padding, cropping, or interpolation to be used. Additionally or alternatively, the LUT may include entries, wherein each entry specifies a specific type of padding to be used (reflection padding, copy padding, or zero padding). Additionally or alternatively, in addition to including an entry that specifies that interpolation is to be used, it also includes an entry that specifies that interpolation is to be used to increase the size by resizing or to reduce the size by resizing and / or specifies the filter to be used.

[0339] Exemplarily, LUT may include 4 entries for filling / cropping, wherein one entry specifies cropping, one entry specifies 0 filling, one entry specifies copy filling, and one entry specifies reflection filling. In addition, the table may include entries to increase the size by resizing using interpolation. These entries may specify different interpolation filters respectively, wherein the interpolation filter may include Bilinear (bilinear), Bicubic (bicubic), Lanczos3, Lanczos5, Lanczos8 and N-tab filters. This indicates that there may be 6 entries specifying different methods (one entry for each filter) of increasing the size by interpolation. In addition, 6 entries may be provided for reducing the size by interpolation, wherein each entry specifies the corresponding filter to be used in interpolation. Therefore, an index may be provided to obtain 16 different values ​​(4 for filling method and cropping, 6 entries for interpolation to increase the size using a specific filter, and 6 entries for interpolation to reduce the size using a specific filter). LUT may be used for a decoder or encoder so that the encoder or decoder may determine the resizing method to be applied according to the indicated value.

[0340] Fig.16 and Fig.17 Some examples of resizing methods are shown. Fig.16 and Fig.17 In the figure, three different types of filling operations and their performance are described. The horizontal axis in the figure represents the sample position. The vertical axis represents the value of the corresponding sample.

[0341] It should be noted that the following explanation is merely exemplary and is not intended to limit the present invention to a specific type of padding operation. The straight vertical line indicates the boundary of the input (an image according to an embodiment), and the right side of the boundary is the sample position where the padding operation is applied to generate new samples. These parts are also referred to as "unavailable parts" below, which means that these parts do not exist in the original input, but are added by padding during the scaling operation for further processing. The left side of the input boundary line represents available samples, which are part of the input. The three padding methods described in the figure are copy padding, reflection padding, and zero padding. In the case of a downsampling operation to be performed according to some embodiments, the input of one or more downsampling layers in the NN is padding information, that is, the original input enlarged by the applied padding.

[0342] exist Fig.16 In the example above, the unavailable positions (i.e., sample positions) that can be filled by padding are positions 4 and 5. In the case of zero padding, the unavailable positions are filled with samples with value 0. In the case of reflected padding, the sample value at position 4 is set to the sample value at position 2; the value at position 5 is set to the value at position 1. In other words, reflected padding is equivalent to mirroring the available sample at position 3, which is the last available sample on the input boundary. In the case of copied padding, the sample value at position 3 is copied to positions 4 and 5. Different applications may prefer different types of padding.

[0343] Specifically, the type of padding used may depend on the task to be performed. For example,

[0344] Zero padding or filling can be reasonably used for Computer Vision (CV) tasks, such as recognition or detection tasks. Therefore, no information is added so as not to change the amount / value / importance of the information already present in the original input.

[0345] Reflective filling can be a computationally easy approach, since the added values ​​only need to be copied from the existing values ​​along defined “reflection lines” (i.e. the boundaries of the original input).

[0346] Compression tasks using convolutional layers may prefer repetition padding because most sample value and derivative continuity are preserved. The derivatives of samples (including available samples and padded samples) are Fig.16 and Fig.17 For example, in the case of reflection filling, the derivative of the signal exhibits a sudden change at position 4 (where a value of –9 is obtained for the exemplary values ​​shown in the figure). Since smooth signals (signals with small derivatives) are easier to compress, it may not be desirable to use reflection filling in video compression tasks.

[0347] In the example shown, the derivative change of the duplicate filling is the smallest. Considering the video compression task, this is very advantageous, but it will result in adding more redundant information at the boundary. For this reason, the information on the boundary may be more important than the information used for other tasks, so in some implementations, the overall performance of zero filling may exceed that of reflection filling.

[0348] Fig.18 Another embodiment is shown. Here, the encoder 2010 and the decoder 2020 are shown side by side. In the example described, the encoder includes a plurality of downsampling layers 1 to N. These downsampling layers can be grouped together or form part of sub-networks 2011 and 2012 of a neural network within the encoder 2010. For example, these sub-networks can be responsible for providing specific bitstreams 1 and 2 that can be provided to the decoder 2020. In this sense, the sub-networks of the downsampling layers of the encoder can form a logical unit that cannot be reasonably separated. Fig.18 As shown, the first sub-network 2011 in the encoder 2020 includes downsampling layers 1 to 3, each layer having its own downsampling ratio. The second sub-network 2012 includes downsampling layers M to N having their own downsampling ratios.

[0349] The decoder 2020 has a corresponding structure, including upsampling layers 1 to N. One subnetwork 2022 in the decoder 2020 includes upsampling layers N to M, and another subnetwork 2021 includes upsampling layers 3 to 1 (here, arranged in descending order to make the numbering consistent with the decoder when viewed in the processing order of the corresponding input).

[0350] As described above, the scaling applied to the input before the downsampling layer 2 in the encoder is correspondingly applied to the output of the upsampling layer 2. This means that the size of the input to the downsampling layer 2 is the same as the size of the output of the upsampling layer 2, as described above.

[0351] More generally, the scaling applied to the input of a downsampling layer n in the encoder corresponds to the scaling applied to the output of an upsampling layer n, such that the size of the scaled input is the same as the size of the scaled output.

[0352] Fig.19 Another exemplary embodiment of a neural network 2100 is described. The neural network 2100 may be a combination of Fig.25 etc., and is used to encode an image according to an embodiment of the present invention.

[0353] To this end, the neural network 2100 may include a plurality of layers 2110, 2120, 2130, and 2140. During the encoding process, it is contemplated that the size of the image inputted as input 2101, etc., is reduced by processing the input through subsequent layers in the neural network 2100. Finally, the encoded image may be provided as output 2105. Specifically, the output may be a binarized version of the encoded image, constituting a bitstream 2105, and the output may be considered as the output of the neural network 2100, or generally as the output of an encoder in which the neural network is implemented.

[0354] In the process of processing inputs through neural network 2100, input 2101, which can be an image or some processed version of an image, is sequentially input into successive layers of neural network 2100 in the processing order shown, thereby possibly generating intermediate outputs 2102, 2103, and 2104, which are output by the current layer in the neural network and provided as input to the next neural network layer. Fig.19 In the embodiment shown in FIG. 1 , one input 2101 is converted into a single output 2105 during processing using the neural network, but the neural network may also provide one or more intermediate outputs after processing the input using layer 2120. After processing the input using layer 2120, an intermediate code stream or sub-code stream that is reduced in size compared to the original input but is not processed by subsequent layers 2130 and 2140 in the neural network 2100 may be output. For example, this may be done in an encoder with Figure 4 and Figure 7 The example is provided in the case of implementation, wherein the encoder provides a first code stream (code stream 1) and a second code stream (code stream 2) as output. However, this is not necessary and can be implemented according to specific circumstances.

[0355] According to the present invention, a neural network may include one or more downsampling layers that apply downsampling to the inputs they receive, thereby reducing the size of the inputs. Fig.19 The neural network shown includes four layers 2110, 2120, 2130, and 2140. Not all of these layers must be implemented as downsampling layers. Some of these layers, such as layers 2130 and 2140, can be implemented as layers that do not apply downsampling to the input but instead process the input in another way.

[0356] The downsampling layer may be associated with a downsampling ratio r having an integer value greater than 1. When receiving an input of a given size S, the downsampling layer reduces the size of the input to a size S during processing. By applying multiple downsampling layers to process the original input image, the size of the output can be reduced by a factor of 1 divided by the product of all the downsampling ratios. This can be expressed as Where index i may list the downsampling ratios of all downsampling layers. The downsampling layers may be listed in the order in which the input is processed by the neural network, starting from i=1, up to N, where N is the last downsampling layer in the neural network. In this case, index i may take a natural value in the range from 1 to N.

[0357] For example, if the neural network includes 6 downsampling layers, each with a downsampling ratio r=2, the original size S of the input will be reduced to 1 / 64.

[0358] In general, the size of the output 2105 of the neural network can be represented as P. According to the present invention, in view of the above, the size P can generally be smaller than the size S of the input.

[0359] When the input 2101 is processed by the neural network, the input size may preferably be an integer multiple of the product of the downsampling ratios of all downsampling layers. Since the downsampling layer usually applies matrix operations or similar operations that need to process an integer number of samples, when the size S (i.e., S samples) of the input to the downsampling layer is not an integer multiple of the downsampling ratio of the layer, the input may not be properly processed.

[0360] For example, if the NN has 2 downsampling layers (e.g. Fig.19 ) (and other processing layers that do not perform downsampling), the downsampling ratio of each downsampling layer is 2, and if the size of the input image is 1024×512, no problem is noticed. After two downsampling operations, the downsampled output result is 256×128. However, if the input size is 1024×511, the NN cannot be used to process the input because after the first downsampling layer, the expected size of the intermediate output 2102 is 512×255.5, which is not an integer and may be understood as a sample fraction (sub-pixel), and the NN may not be configurable. This means that the NN in the example cannot process input images that are not multiples of 4×4, where 4 on each dimension represents the product of the downsampling ratios of the two downsampling layers in this example.

[0361] The above example illustrates the problem caused by a small number of downsampling layers (for example, 2). However, image compression is a complex task (because images are usually large) and usually requires deep neural networks to complete this task. This means that the number of downsampling layers included in the NN is usually greater than or even much greater than 2. This increases the difficulty of the problem because, for example, if the number of downsampling layers is 6 (the downsampling ratio of each downsampling layer is 2), then if the neural network applies downsampling in two dimensions, the NN can only handle 2. 6 ×2 6= Input size that is a multiple of 64 × 64. Most images acquired by different end-user devices do not meet this requirement.

[0362] To achieve downsampling, the downsampling layer can apply convolution.

[0363] This convolution involves taking the original matrix as input (in the exemplary case a matrix of 1024×512 elements, where the elements are denoted by M ij ) with a kernel K that operates on this matrix (shifted) and whose size is usually smaller than the input size. The convolution operation on 2 discrete variables can be described as:

[0364]

[0365] Therefore, computing the function (f*g)[n] for all possible values ​​of n is equivalent to running a (shifted) kernel or filter f[] on the input array g[] and performing an element-wise multiplication at each shifted position.

[0366] In the example above, kernel K is a 2×2 matrix that operates on the input with stride range 2, such that by combining kernel K with element M 11 、M 12 、M 21 、M 22 Multiply to get the first element D in the downsampled code stream D 11 Then, by computing the kernel and the element or the reduced matrix and the element M 13 、M 14 、M 23 、M 24 The inner product of , get the next element D in the horizontal direction 12 The above operation is performed in the vertical direction accordingly, and finally a matrix D is obtained, which includes the elements D obtained by calculating the corresponding inner products of M and K. ij , and there are only half as many elements in each direction or dimension.

[0367] In other words, the amount of shift used to obtain the convolution output determines the downsampling ratio. If the kernel is shifted by 2 samples between each computation step, the output is downsampled by a factor of 2. The downsampling ratio of 2 can be expressed using the above formula as follows:

[0368]

[0369] The transposed convolution operation can be mathematically represented in the same way as the convolution operation. The transposed convolution can be implemented during the decoding process of the encoded image, such as in conjunction with Figure 22 to Figure 24The term "transposed" indicates that the transposed convolution operation corresponds to the inverse operation of a particular convolution operation. However, depending on the specific implementation, the transposed convolution operation can be similarly implemented according to the above formula. The upsampling operation using the transposed convolution can be implemented by the following function:

[0370]

[0371] In the above formula, u corresponds to the upsampling ratio, and the int() function corresponds to the conversion to an integer. For example, the int() operation can be implemented as a rounding operation.

[0372] In the above formula, when the convolution kernel or filter f() and the input variable array g() are one-dimensional arrays, the values ​​m and n can be scalar indices. When the kernel and input array are multi-dimensional, the values ​​m and n can also be understood as multi-dimensional indices.

[0373] The present invention is not limited to downsampling or upsampling by convolution and deconvolution. Any possible downsampling or upsampling method can be implemented in the layers of a neural network (NN).

[0374] This process (downsampling) can be repeated if more than one downsampling layer is provided within the neural network to further reduce the size. Fig.19 , the encoded bitstream 2105 can be used as the output of the neural network. This repeated downsampling can be Figure 6 , Fig.10 and Fig.11 The encoder is implemented.

[0375] The layers in the encoder and neural network 2100 are not limited to only including downsampling layers that apply convolutions, but other downsampling layers that do not necessarily apply convolutions that achieve input size reduction can also be considered.

[0376] In addition, the layers in the neural network 2100 may include more units or may be associated with more units that perform other operations on the corresponding inputs and / or outputs of the corresponding layers in the neural network. For example, the layer 2120 in the neural network may include a downsampling layer, and in the processing order of the inputs of the layer, a rectifying linear unit (ReLu) and / or a batch normalizer may be provided before the downsampling.

[0377] It is known that the rectified linear unit applies the correction to the elements P in the matrix P ij , to obtain a modified element P' of the following form ij :

[0378]

[0379] Therefore, it is ensured that the values ​​in the modified matrix are all equal to or greater than 0. This may be necessary or advantageous for some applications.

[0380] As we all know, the batch normalizer first calculates the average value based on the elements Pij in the matrix P of size M×N, and then normalizes the values ​​in the matrix. The average value is in the following form:

[0381]

[0382] Using the average value V, the following formula is used to obtain the element P' ij The batch normalization matrix P' of

[0383] P′ ij =P ij -V

[0384] Neither the calculated values ​​obtained by the batch normalizer nor the calculated values ​​obtained by the rectified linear unit will change the number (or size) of elements, but only the values ​​in the matrix.

[0385] Depending on the situation, these units can be set before the corresponding downsampling layer or after the corresponding downsampling layer. Specifically, since the downsampling layer reduces the number of elements in the matrix, it may be more appropriate to set the batch normalizer after the corresponding downsampling layer in the processing order of the code stream. Therefore, V and P' are obtained. ij The amount of computation required is greatly reduced. Since rectified linear units can simplify multiplications to obtain a matrix of reduced size when convolution is used for downsampling layers, since some elements may be 0, it is advantageous to set up rectified linear units before applying convolution.

[0386] However, the present invention is not limited in this regard, and the batch normalizer or the rectified linear unit may be arranged in another order relative to the downsampling layer.

[0387] Furthermore, not every layer must include one of these further units, and other further units that perform other modifications or calculations may also be used. When processing inputs through a neural network, matrix operations are applied, such as the convolutions described above.

[0388] Because of the matrix calculations performed here, in order to process the input through each downsampling layer, the size of the input to the neural network 2100 is preferably an integer multiple of the product of all downsampling ratios. Based on the above example, assuming there are 6 downsampling layers, each with a downsampling ratio of 2, this means that the size of the input to the neural network should be an integer multiple of 64 so that the neural network can process it reliably. Next, consider an input with a size of 540 in at least one dimension. This input cannot be reasonably processed by the neural network because this input is not an integer multiple of the product of the downsampling ratios of all downsampling layers in the neural network.

[0389] Therefore, before processing the input with a neural network, resizing or scaling (these two terms are used interchangeably) needs to be applied to the input, thereby changing the size S of the input to a size that the neural network can reasonably handle. For example, if the input size is 540, this is not an integer multiple of 64. In this case, you can scale it to the nearest integer multiple of but less than that size (in this case 512) or the nearest integer multiple of but greater than that size (in this case 576) so that the input size S changes to a size that the neural network can reasonably handle.

[0390] As described above, resizing can be done in a variety of different ways. For example, the size of the input can be increased or decreased so that the size matches an integer multiple of the product of all downsampling ratios of the neural network 2100. The size can be reduced in different ways, such as by pruning the input (essentially involving removing sample values ​​from the input) or by applying interpolation. When interpolation is applied, instead of using two (or more) adjacent samples, a single new sample value (e.g., an average) representing the two samples is used, thereby reducing the overall size of the input by 1. The more samples that are interpolated, the more the size of the input is reduced.

[0391] Interpolation can also be used when increasing the size S of the input. In this case, an "intermediate" or new sample is generated by taking the average of two adjacent samples, then separating those adjacent samples and inserting the new sample between them. Alternatively, padding can be used, which involves inserting additional samples with specific values ​​into the input to increase the size of the input. For example, such padding can include padding with zeros or padding with information already available in the input, such as the copy padding or reflection padding described above.

[0392] The actual resizing method chosen may depend on the specific situation, for example, the expected output size P of the neural network. If size P has a specific value, it may not be appropriate to reduce the size of the input to the nearest integer multiple of the product of the neural network's downsampling ratios but less than size P, and instead it may be more appropriate to increase the size of the input.

[0393] Based on the above example where the product of the downsampling ratios is 64, assume that the input size S is 540. This is not an integer multiple of 64, but 512 and 576 are. If you want to provide an output of size P=8, then increasing the size to 576 is not appropriate. In this case, the input size S is best reduced to After processing the resized input through the neural network, the resulting output is of size 8, since 512 is equal to 8×64.

[0394] Furthermore, the user may choose to increase the size of the input to avoid information loss, or to reduce the size of the input during the encoding process, where the encoded image needs to be as small as possible. Additionally, when processing an image, an encoder performing the encoding method may try multiple resizing methods and may then select the most appropriate one to ensure high quality decoding of the bitstream including the encoded image.

[0395] To consider these options, Fig. 20 A method for encoding an image provided by an embodiment is shown.

[0396] The image or an input that is in some way related to the image (e.g., a preprocessed or otherwise modified input) has a size S (corresponding to the number of samples in the image, etc.). In step 2210, the encoder or Fig.19 The neural network 2100 in receives the image or input. In step 2220, based on additional information such as the resizing method selected by the user, the expected output size P, or other indications described in detail below, the resizing method to be used in the encoding process can be obtained. In the next step, the size S of the input can be changed to the size S by applying this resizing method. For example, the original input of size S can be cropped so that the size S is reduced to size Alternatively, zero padding can be performed on the input of size S so that the size increases to size

[0397] In the present invention, the size It is an integer multiple of the product of the downsampling ratios of all downsampling layers in the neural network.

[0398] In some embodiments, the resizing method can be obtained according to the input size S and information about the neural network. The information may include one or more downsampling ratios of the downsampling layers in the neural network, or a number indicating the product of the downsampling ratios of all downsampling layers in the neural network, etc. In addition, the information may include the expected output size P of the neural network and one or more downsampling ratios or the product of the downsampling ratios of all downsampling layers.

[0399] This information can be used to determine how to change the size S (if necessary). For example, assume that the size of the input is S=512. The information provided may indicate that the size of the output must be P=8. In addition, the product of all downsampling ratios of the downsampling layer may be 64. 8 times 64 equals 512, so it can be determined that the size of the input does not need to be changed when applying the resizing. In this case, step 2230 may include resizing that is the same resizing, i.e., not changing the size of the input.

[0400] In contrast, consider the case where the size of the input is 540, as described above. When the size of the output P is 8, this may lead to the selection of a resizing method that reduces the size of the input to 512, although in principle it is possible to increase and decrease the size of the input.

[0401] If the expected output size P is not specified, then the size S can be chosen to be increased or decreased (as a first step in choosing a resizing method) so as to modify the original input of size S as little as possible. This can include computing the difference between the nearest integer multiple of the size S that is smaller than and the nearest integer multiple of the size S that is greater than the product of all downsampling ratios of all downsampling layers in the neural network. This can be done by computing the function or Then, any of them can be compared to the input size S, for example, by subtracting the value of the corresponding function from S, or subtracting S from the value of the corresponding function.

[0402] For example, you can get the value (indicates the difference between the product of the downsampling ratios of all downsampling layers that is closest to but greater than an integer multiple of size S and the input size S) and the value (Indicating the difference between the integer multiple of size S that is closest to but smaller than the product of the downsampling ratios of all downsampling layers and the input size S.) Additionally or alternatively, the absolute values ​​of C and F may be obtained.

[0403] Depending on which of the values ​​C and F is greater or which of |C| and |F| is greater in absolute value, a resizing method may be selected that includes increasing or decreasing the size S. For example, if F is less than C, the input size S is closer to the nearest integer multiple of the product of all downsampling ratios but less than size S, and therefore, if the input size S is reduced to this nearest integer multiple of size S, the modification to the original input when reducing or increasing the size is reduced. The same applies if the value C is less than the value F. In this case, the modification to the original input size S is reduced when increasing the size to the nearest integer multiple of size S that is greater than the product of all downsampling ratios.

[0404] In addition, the expected size P of the output can be provided in the form of an index indicating an entry in a pre-stored look-up table (LUT) or the like, the table comprising a plurality of entries, each entry indicating a different output size. By providing such an indication, the size P can be selected, and as described above, a suitable resizing method can be selected.

[0405] In the selection whether to increase or decrease the size S of the input to size Thereafter, as part of obtaining the resizing method, it may be determined or obtained which resizing method is actually applied during the resizing process to increase or decrease the size S. For example, if the size S is to be decreased, cropping or interpolation may be applied. If the size is to be increased, padding or interpolation may be applied. In yet another step in step 2220, it may be determined based on additional information, etc., which resizing method is to be selected for increasing or decreasing the size.

[0406] Additionally or alternatively, one or more indications specifying a resizing method to be selected may be provided (eg, as part of the additional information), wherein a resizing method may be selected based on the indications rather than performing 2220 .

[0407] Once the resize method is obtained, in step 2230 the input is resized from size S to size Then, in step 2240, the resized input is processed by the neural network, and finally an output of size P is provided after processing by the neural network.

[0408] The output may then be binarized, providing a bitstream. Alternatively, further processing may be performed, such as inserting information about the resizing method that has been applied, such as one or more indications about the selected resizing method. After inserting or adding this information, the output of the neural network and this information may be binarized to obtain a bitstream. The bitstream may then, for example, be forwarded to a decoder, where decoding of the bitstream may be performed to reconstruct the image, possibly using information other than the encoded image in the bitstream.

[0409] For instructions on which resizing method to apply, Fig.21 Yet another example is provided.

[0410] exist Fig.21 In FIG. 2 , a plurality of ellipses 2310, 2320, 2330, 2340 and 2350 are provided. Each of these ellipses constitutes an indication, which may or may not be provided to the encoder for use in Fig. 20The resizing method is obtained in step 2220 of . The numbers within these ellipses constitute the values ​​of the indications and the corresponding reference numerals for ease of explanation. The values ​​of the indications can be understood as the values ​​that the corresponding indications may have or take. Specifically, although each indication may have multiple different values, it can be understood that each indication can actually only take one of these different values. For example, the first indication can take the value 2311 or the value 2312, but not both values ​​at the same time.

[0411] In some embodiments, all of these indications may be provided in the information provided to the encoder, regardless of their actual values. In some embodiments, it is also conceivable that one or more of these indications may only appear when the current indication takes a specific value. This is described in detail below.

[0412] exist Fig.21 , a first indication 2310 is shown. For example, the indication may take two values. A first value 2311 may indicate that a resizing method including input padding or cropping is to be applied. Another value 2312 may indicate that interpolation is to be used as a resizing method (regardless of whether the size is to be increased or decreased in the resizing). Advantageously, the first indication 2310 may be provided in the form of a flag of size 1 bit, wherein a first value 2311 (e.g., 0) indicates that padding or cropping is to be used, and a second value (e.g., 1) 2312 indicates that interpolation is to be used.

[0413] According to the actual value of the first indication 2310, it can be considered that the resizing method has been finally determined, so the encoding can be performed by applying the resizing. For example, if the value of the first indication 2310 indicates that padding or cropping is to be used (through the value 2311), according to more information such as the input size S and the expected output size P, the resizing method can be applied to the encoding. Fig. 20 In step 2220, it is determined whether padding or cropping is to be applied, without necessarily indicating it in the additional information. This is because when the input size S is known and the downsampling ratio of the downsampling layer in the neural network is fixed, the expected output size P can only be obtained in one way, that is, by increasing the input size S by applying padding, or by reducing the input size by applying cropping. In this case, the input size S can be adjusted to size Make the size It can be equal to the product of the expected output size P and the downsampling ratio of all downsampling layers.

[0414] The padding of the input can be arbitrary or determined at the encoder's discretion.

[0415] In one embodiment, if the value of the first indication 2310 indicates that interpolation is to be used, a second indication 2320 may be provided. The second indication 2320 may take a first value 2321, the first value 2321 indicating that the size S of the input is to be increased by interpolation, and a second value 2322 of the second indication may indicate that the size of the input is to be reduced. Depending on the value of the indication, the size of the input may be increased or decreased.

[0416] Like the first indication, the second indication can also be provided in the form of a flag of size 1 bit, since there are only two options, using interpolation to increase or decrease the size S of the input. These two options can be encoded using a single bit, thus reducing the amount of information.

[0417] In addition, if the first indication 2310 indicates that interpolation is to be applied as a resizing method using a value 2312, a third indication 2313 may be provided. The third indication here has multiple values ​​2323 to 2326. These values ​​may respectively refer to or indicate an interpolation filter to be applied during the interpolation process (regardless of the value of the second indication 2320, or may even depend on the value of the second indication 2320). For example, the value of the third indication 2330 may be provided as an index indicating an entry in a lookup table that may be used for an encoder or encoding method. In the lookup table, each entry may specify an interpolation filter, and by using an index, the entry in the lookup table may be identified and the interpolation filter may be derived accordingly without explicitly including the interpolation filter or its value in the third indication 2330. On the other hand, the third indication 2330 may explicitly specify an interpolation filter by one or more of the values ​​2323 to 2326.

[0418] In another case, if the first indication 2310 indicates that padding or clipping is to be used (via value 2311), a fourth indication 2314 may be provided. The fourth indication may also take different values, with one value 2313 indicating that padding is to be used for resizing, while the second value 2314 indicates that clipping is to be used. In this way, it is clearly specified whether the size of the input is to be increased (using padding) or decreased (using clipping). As with the first and second indications, the third indication may also be provided in the form of a flag of size 1 bit, where, for example, 0 indicates that padding is to be used, while 1 indicates that clipping is to be used.

[0419] In some embodiments, if the fourth indication indicates that padding is to be applied (value 2313), a fifth indication may be provided. The fifth indication 2350 may indicate whether zero padding, reflection padding, copy padding, or other padding methods are to be used in the padding according to its value 2331 to 2333. In this way, through the fourth indication and the fifth indication, the amount of padding to be used in the resizing process may be specified.

[0420] However, it is also open-ended to determine which fill mode is applied. Fig. 20 There may be no explicit indication in step 2220, so the fifth indication may not exist.

[0421] Optionally, information about the padding to be used can also be included in the fourth indication 2340 itself, rather than in the fifth indication 2350. Assuming the three exemplary padding methods described above (0-padding, reflection padding, and copy padding) are used, and clipping is also considered, the fourth indication 2340 has four values ​​that can specify which padding or clipping mode to apply. This can be encoded using an indication of size 2 bits, thereby representing four values. Therefore, this information can also be provided in an indication of relatively small size.

[0422] As above Fig.21 As described, if the value of the first indication 2310 indicates that interpolation is to be applied, the second indication and the third indication may exist. If the value of the first indication 2310 indicates that padding or cropping is to be used, the second indication and / or the third indication may not exist, thereby further reducing the amount of information. Similarly, if the first indication 2310 indicates that interpolation is to be used, the fourth indication or the fifth indication may not exist to maintain a smaller size. Conversely, it can also be considered that all of the above indications exist anyway. However, since information on whether interpolation, padding or cropping is used in resizing has been obtained by processing the first indication 2310, the values ​​of the corresponding other indications are no longer important and can be set to 0 by default or to any other reasonable value.

[0423] By processing indications and other possible information about the input size and / or the downsampling ratio of the downsampling layer in the neural network and / or the expected output size P, the encoder can determine the desired output size P. Fig. 20 The resizing method applied in step 2220.

[0424] Although the reference Fig.21 The embodiment of can be used to obtain the resizing method in step 2220 in the encoder, but Fig.21 Said indication may also be included in the bitstream comprising the output of the neural network. In this way, the decoder may obtain this information, which can then be used to apply appropriate resizing during the decoding process (as described below), thereby ensuring reliable image reconstruction.

[0425] For the above instructions, please also refer to the corresponding instructions. Fig.13 , Fig.14 and Fig.15 In this case, the first indication may be an indication represented as methodIdx. The second indication may be an indication represented as SCIdx above, and the third indication may be an indication represented as RFIdx above. Therefore, Figures 13 to 15All content in also applies to Fig.21 The first, second and third instructions mentioned in.

[0426] Fig.21 The presence of an indication shown and explained above is dependent on the value of another indication. For example, the presence of indication 2320 is dependent on the value of indication 2310 (shown as the first indication).

[0427] Alternatively, the present invention also includes that the presence of the first to fifth indications is independent of the presence of another indication.

[0428] In this case, in order to facilitate identification of different indications, the indications are named as first, second, third, etc. indications. Since they can be provided as independent indications, they can also be referred to as "indications". In addition, the numbering of first indication, second indication, etc. is not intended to restrict these indications to a specific order of appearance in a code stream, etc. Instead, it is just a way of naming different indications for easy identification.

[0429] Furthermore, according to some embodiments, as a replacement for or in addition to the first to fifth indications, one or more indications are provided according to some embodiments, wherein the indication is used to obtain the resizing method from the table.

[0430] The indication may be or may include an index indicating an entry in a lookup table. The lookup table (look-uptable, LUT) may include multiple entries, each entry specifying a resizing method. There may be entries in the LUT specifying that padding, cropping, or interpolation is to be used. Additionally or alternatively, the LUT may include entries, wherein each entry specifies a specific type of padding to be used (reflection padding, copy padding, or zero padding). Additionally or alternatively, the LUT may include entries specifying that interpolation is to be used, entries specifying that interpolation is to be used to increase the size by resizing or to reduce the size by resizing, and / or entries specifying a filter to be used in the interpolation process.

[0431] Exemplarily, the LUT may include 4 entries for padding / cropping, wherein one entry specifies cropping, one entry specifies zero padding, one entry specifies copy padding, and one entry specifies reflection padding. In addition, the table may include one or more entries for increasing the size of the input by resizing using interpolation. These entries may specify different interpolation filters, respectively, wherein the interpolation filters may include Bilinear, Bicubic, Lanczos3, Lanczos5, Lanczos8, and N-tab filters, or any other filters or any other number of different filters.

[0432] In a specific embodiment, this may mean that there are 6 entries specifying different methods of increasing the size by interpolation (one entry for each filter). In addition, 6 entries may be provided in the LUT for reducing the size by interpolation, wherein each entry specifies a corresponding filter to be used in the interpolation.

[0433] Thus, an index may be provided to obtain 16 different values ​​corresponding to 16 different entries in the LUT (4 for padding method and cropping, 6 entries for interpolation to increase size using a specific filter, and 6 entries for interpolation to decrease size using a specific filter). The LUT may be used by the encoder so that the encoder can determine the resizing method to apply based on the indicated value.

[0434] The indication including the LUT index may be provided to the encoder like the other indications mentioned above, separately from or together with the picture to be encoded, in a codestream etc. Alternatively, the indication may be derived from the input by a user specifying one or more resizing methods to be applied to the input, for example.

[0435] Fig. 22 is a schematic illustration of a neural network 2400. The neural network 2400 may be part of a decoder that receives a bitstream representing an encoded image for decoding. The input to the neural network is represented as 2401 and may be connected to Fig.19 The output 2105 of the neural network 2100 shown is related.

[0436] The general structure of neural network 2400 can be compared to Fig.19 The structure of the neural network 2100 shown is similar. Fig.19 As shown, neural network 2400 may include multiple layers, for example, layers 2410, 2420, 2430, and 2440, which process the inputs they receive. In this context, input 2401 may be processed by various layers, each layer providing an output 2402, 2403, and 2404, each output being used as an input to the next layer in the neural network until finally, after processing input 2401 using all layers in neural network 2400, output 2405 is obtained, which may be a decoded image.

[0437] To this end, the neural network 2400 includes upsampling layers that apply upsampling to the inputs they receive. This can be seen as Fig.19The inverse operation of downsampling applied in the downsampling layer shown is usually associated with an upsampling ratio u of the corresponding upsampling layer. The upsampling ratio can be a natural number greater than 1, so that when an input such as input 2401 is processed by the upsampling layer 2410 in the neural network 2400, the size of the input is increased in at least one dimension according to the upsampling ratio. For example, this can be achieved by applying a deconvolution to the input as an inverse transform of the convolution, such as Fig.19 As shown. Upsampling may be a property of a layer that generally performs a transformation on its input. For example, the layer may be a convolutional layer, or an activation layer (e.g., including a rectified linear unit) that has upsampling properties. Layers with such properties are generally referred to as upsampling layers in this application.

[0438] The output is obtained by processing the input 2401 through all the upsampling layers in the neural network 2400. Since each upsampling layer applies upsampling, the size T of the input 2401 is the same as the size of the intermediate output 2405 provided by the last upsampling layer 2440. The following relationship exists: size is proportional to T and a function of the total upsampling ratio of the neural network 2400. The total upsampling ratio applied by the NN2400 depends on the upsampling ratios applied by each layer therein. In one example, the total upsampling ratio of the NN can be obtained by the product of all upsampling ratios of the layers in the NN. The total upsampling ratio of the NN (2400) can be expressed as Π i u i , where u i The upsampling ratio of the upsampling layer i is specified, and the number of values ​​of the index i can be the same as the number of upsampling layers in the NN 2400. In another example, the total upsampling ratio of the NN can be a pre-calculated scalar value K.

[0439] The relationship between the size T of the input 2401 and the output size can be expressed as Among them, u i Specifying the upsampling ratio of upsampling layer i, the number of values ​​of index i can be the same as the number of upsampling layers in NN 2400. For example, if there are N upsampling layers (N is a natural number), the index can take all natural values ​​between 1 and N. This way of indexing or listing upsampling layers is only exemplary. For example, index i can start with a first value of 0 or -1.

[0440] To illustrate upsampling, the following is introduced.

[0441] If the input size T is 8, and the neural network 2400 includes 6 upsampling layers, each with an upsampling ratio u=2, then the size of the intermediate output (e.g., output 2405) is Because 8×2 6 =512.

[0442] As above combined Figures 19 to 21 As described above, when the encoder processes the input, resizing can be applied to reduce or increase the size S of the input received by the encoder to the size size It is usually different from the original size S that represents the image size. However, the resized input is processed using a downsampling layer during the encoding process to produce an output of size P. This output is then provided to the decoder to decode and reconstruct the image. In this case, the input size T is equal to P.

[0443] However, even if the upsampling ratio of the applied upsampling layer is the same as the downsampling ratio of the encoder's downsampling layer, the output of the neural network on the decoder side corresponds to the product of size P (equal to T) and the upsampling ratios of all upsampling layers. Therefore, the size of the output of the neural network 2400 is This is not necessarily the same size S as the original input to the encoder. This is because the upsampling applied by the decoder to the input of size T may only be to restore the downsampling applied by the encoder to the input while the encoder encodes the image to be reconstructed. However, the input to which the encoder applies downsampling to obtain an output of size P may have a different size than size S. (as described above). Downsampling is applied to a size of The resized input of produces an output of size P, which is then fed as an input to the decoder of size T. When the decoder restores the downsampling by applying upsampling (assuming the total upsampling ratio of the decoder's NN is the same as the total downsampling ratio of the encoder's NN), this may result in an intermediate output of size With size same as the decoder, because the inversion or restoration operation is applied to a The resized input is a downsample of the original input size S instead of the original input size S. Therefore, the size of the intermediate output Typically equal to the size of the resized input after the encoder applies downsampling But the size of the intermediate output It will usually not be equal to the size S of the original input (image) to the encoder.

[0444] Therefore, when processing an image with the decoder's neural network, the image is usually not reconstructed. The cascaded application of upsampling layers to the decoder's input makes it impossible for the output to reach some target size. For example, if the total upsampling ratio of the decoder is K and the input size is T, then the size of the intermediate output of the decoder can be equal to K×T. This means that the decoder neural network can only achieve an output size that is K times larger. However, if one wants the output size to be equal to the encoder's input size S, this is not possible, especially when S is not a multiple of K. This can lead to information loss (when the intermediate sizes are less than S) or information redundancy (when greater than S).

[0445] Therefore, in some embodiments of the present invention, after processing an input of size T in at least one dimension using all upsampling layers in a neural network, resizing may be applied to an intermediate output obtained by processing using all upsampling layers in the neural network, wherein the resizing reduces the size of the intermediate output Change to size

[0446] This intermediate output can be explicitly output by the neural network or the last layer in the neural network. Once the output is obtained, resizing can be applied. Alternatively, resizing can be applied while the neural network is processing the input, for example, as part of the last layer of the neural network. Resizing can be done by adding the size Resize to size For example, Can be provided as information in the bitstream (e.g., equal to the original input size S)

[0447] On the other hand, there is no explicit provision in the code stream In this case, the size can be obtained from the information obtained in the code stream. For example, the size can be obtained from the upsampling parameters (e.g., upsampling ratio) of the upsampling layer in the neural network Alternatively, the size can be obtained using an index belonging to a codestream or an extra codestream. This index can point to the output size When the value of an index is obtained from the code stream, the size associated with the index can be obtained from the lookup table. In a decoded image (e.g., having a size ) has only a limited number of allowed sizes (e.g., 512×256, 1024×512, or 2048×1024, which are commonly used for video). In this case, the decoder can use this lookup table and then use the index provided in the bitstream to get the size Make the necessary resizing.

[0448] The resizing to be applied for encoding etc. can be done in different ways including interpolation, cropping, padding, increasing or decreasing the size, etc. In the case of a fixed size, the resizing method (increasing or decreasing the size) may be fixed, but it is still necessary to determine whether to increase or decrease the size of the intermediate output to the size For example, it may be preferred to apply a resizing that corresponds to (e.g., is the opposite of) the resizing applied by the encoder. By applying a resizing that is the opposite of the resizing applied by the encoder, the quality of the reconstruction may be improved. For example, if the encoder applies padding to increase the size S of the input before processing the input using a neural network, the decoder may apply cropping instead of interpolation.

[0449] in this regard, Fig.23 A method 2500 for decoding a code stream provided by an embodiment is shown. In a first step 2510, an input of size T is received, for example, a code stream encoded with an image or some pre-processed form of the code stream. In the next step 2520 (although the time sequence may be different as described below), by using additional available information (for example, the size discussed above), or one or more instructions discussed below), etc. to obtain the resizing method to be applied.

[0450] In the next step 2530 of the method, the input of size T may be processed by the neural network. This may include successively processing the input through each upsampling layer in the neural network, thereby obtaining a size T in step 2540. Since one or more upsampling layers in a neural network apply upsampling to the corresponding inputs they receive, the size is usually larger than size T. Specifically, when considering multiple upsampling layers with associated upsampling ratios processing an input of size T, the size It can be equal to the product of the original input size T and the upsampling ratio of all upsampling layers. This can be expressed as Among them, u m is the upsampling ratio of the upsampling layer.

[0451] After obtaining the intermediate output at step 2540, the resizing method determined or obtained at step 2520 is applied to the image having a size of The intermediate output of If resizing includes increasing the size of the intermediate output Then the size Can be larger than size If resizing includes reducing the size Then the size Smaller than the size of the intermediate output

[0452] Size The output of may already constitute a decoded image, so in step 2560, the decoded image may be obtained directly after resizing. However, it is also possible that further processing is performed after applying the resizing, and then the decoded image is obtained. However, for ease of explanation, it is assumed that the resizing is applied to the intermediate output to transform it to a size of After the output of , the decoded image is immediately obtained in step 2560.

[0453] As explained above, in step 2520, the resizing method to be applied in step 2540 may be obtained. This may be efficient if the information about the resizing method to be selected is encoded or provided in the bitstream. When processing or parsing the bitstream, this information may be obtained after receiving the input, and the resizing method to be applied may be obtained from it. However, it may also be provided that only when the size is obtained, After the intermediate output is obtained, the resizing method is obtained, and then the resizing is applied in step 2550 using the obtained resizing method.

[0454] As mentioned above, the size provided as the output size The resizing method is obtained or determined by the information of the input size T and / or the upsampling ratio of the upsampling layer in the neural network. For example, the input size T can be multiplied by the upsampling ratio of all upsampling layers. This gives the size of the intermediate output Then, the result (i.e., size ) and size If the result is the same as is different, resizing is applied. For example, if Then you need to apply the size increase of the intermediate output to size If resizing is applied to reduce the size of the intermediate output. You can be sure that there is no need to resize the intermediate output to a different size.

[0455] Additionally or alternatively, information which resizing method is to be applied may be provided in the codestream or in an additional codestream in the form of one or more indications.

[0456] in this regard, Fig.24An exemplary embodiment of indications is shown, which can be provided as part of the codestream or in an additional codestream to a decoder implementing the decoding method in order to obtain the resizing method to be applied. These indications can be provided in the codestream by an encoder encoding an image, thereby ensuring that the decoder uses the appropriate information to apply the appropriate resizing method when decoding the codestream to obtain the decoded image.

[0457] In this regard, Fig.21 Most of the descriptions also apply to one or more indications provided to the decoder. Specifically, a first indication 2610 may be provided as part of the bitstream. The value of the first indication 2610 may indicate (2611) whether padding or cropping is used as a resizing method, or whether interpolation (value 2612) is used for resizing. Depending on the value of the first indication 2610, a second indication 2620 and a third indication 2630 may also be provided (as described above). Fig.21 ) or the fourth indication 2640 and the fifth indication 2650 (as above Fig.21 described above).

[0458] Fig.24 The presence of an indication shown and explained above is dependent on the value of another indication. For example, the presence of indication 2620 is dependent on the value of indication 2610 (shown as the first indication).

[0459] Alternatively, the present invention also includes that the presence of the first to fifth indications is independent of the presence of another indication. In this case, in order to facilitate identification of different indications, the indications are named as first, second, third, etc. indications. Since they can be provided as independent indications, they can also be referred to as "indications". In addition, the numbering of the first indication, the second indication, etc. is not intended to limit these indications to a specific order of appearance. Instead, this is just a naming of different indications for easy identification.

[0460] Furthermore, according to some embodiments, as a replacement for or in addition to the first to fifth indications, one (another) indication may be provided according to some embodiments, wherein the indication is used to obtain the resizing method from the table.

[0461] The indication may be or may include an index indicating an entry in a lookup table. The lookup table (look-uptable, LUT) may include multiple entries, each entry specifying a resizing method. There may be entries in the LUT specifying that padding, cropping, or interpolation is to be used. Additionally or alternatively, the LUT may include entries, wherein each entry specifies a specific type of padding to be used (reflection padding, copy padding, or zero padding). Additionally or alternatively, the LUT may include entries specifying that interpolation is to be used, entries specifying that interpolation is to be used to increase the size of an intermediate output by resizing or to reduce the size of an intermediate output by resizing, and / or entries specifying a filter to be used in the interpolation process.

[0462] Exemplarily, the LUT may include 4 entries for padding / cropping, wherein one entry specifies cropping, one entry specifies zero padding, one entry specifies copy padding, and one entry specifies reflection padding. In addition, the table may include one or more entries for increasing the size of the intermediate output by resizing using interpolation. These entries may specify different interpolation filters, respectively, wherein the interpolation filters may include Bilinear, Bicubic, Lanczos3, Lanczos5, Lanczos8, and N-tab filters, or any other filters or any other number of different filters.

[0463] In a specific embodiment, this may mean that there are 6 entries specifying different methods of increasing the size of the intermediate output by interpolation (one entry for each filter). In addition, 6 entries may be provided in the LUT for decreasing the size of the intermediate output by interpolation, wherein each entry specifies a corresponding filter to be used in the interpolation.

[0464] Thus, an index may be provided to retrieve 16 different values ​​corresponding to 16 different entries in the LUT (4 for padding method and cropping, 6 entries for interpolation to increase size using a specific filter, and 6 entries for interpolation to decrease size using a specific filter). The LUT may be made available to the decoder so that the decoder can determine the resizing method to apply based on the indicated value.

[0465] The indication including the LUT index may be provided to the decoder in the code stream or the like separately from the code stream encoded with the image or as part of the code stream encoded with the image, like the other indications mentioned above.

[0466] The decoder may use one or more of these indications and / or the expected size as described above. The decoder can be assured that the resizing method applied by the encoder during encoding is appropriately indicated to the decoder.

[0467] In this regard, it is noted that the information provided to the decoder in one or more indications may be the same as that provided in accordance with Fig.21 The information of the one or more instructions provided to the encoder is the same. Therefore, in some embodiments, the encoder can copy these one or more instructions into the bitstream. In this way, the decoder can know what operations the encoder has performed. Obviously, when the encoder applies cropping to the input before processing the downsampling layer in the neural network, it is necessary to apply padding or other resizing methods that increase the size of the intermediate output to increase the size of the intermediate output. So the decoder gets a size of The output of . This is because the process performed in the encoder and decoder is basically the opposite. If the same resizing method as the encoder is applied in the decoder, the image cannot be reconstructed.

[0468] In view of this, in one embodiment, combined with Fig.24 The instructions shown or explained are opposite (opposite / inverse) to the instructions applied by the encoder when encoding the picture. In view of this, when the encoder encodes the picture and provides instructions to the codestream, these instructions can be combined with Fig.21 The interpreted indication is obtained by inverting, for example, by inverting the value of a flag to determine whether to use an increase or decrease size operation.

[0469] Fig.25 An encoder 2700 for encoding an image is shown. The encoder includes one or more processors 2701 for implementing a neural network and a transmitter 2702 for outputting a code stream. The neural network includes a plurality of layers arranged in the order in which the image is processed by the neural network, and the layers include at least one downsampling layer for applying downsampling to the input. The encoder 2700 (specifically, one or more processors 2701 therein) can be used to encode an image in the following manner:

[0470] - Get a resize method from among multiple resize methods;

[0471] - Resize the input to size S by applying the resize method

[0472] - The neural network processes The resized input of , wherein the neural network comprises one or more downsampling layers;

[0473] -Using the neural network to process After receiving the input of , an output of the neural network is provided, wherein the size of the output is P and is smaller than S in at least one dimension.

[0474] Additionally, the encoder may further comprise a receiver 2703 for receiving the image or data associated with the image.

[0475] Fig.26 An embodiment of a decoder 2800 for decoding a code stream representing an image is shown. The decoder 2800 includes a receiver 2801 for receiving the code stream, one or more processors 2802 for implementing a neural network, and a transmitter 2803 for outputting a decoded image. The neural network includes a plurality of layers arranged in the order in which the code stream is processed by the neural network, the layers including at least one upsampling layer for applying upsampling to the input. The decoder is used to decode the image by:

[0476] - Get a resize method from among multiple resize methods;

[0477] - Processing an input of size T through the neural network, wherein the neural network includes one or more upsampling layers, thereby obtaining an input of size and the intermediate output is larger than T in at least one dimension,

[0478] - by applying the obtained resizing method, the intermediate output is resized from Adjust to size Thus, a decoded image is obtained.

[0479] Fig.25 The encoder and Fig.26 The embodiments of the decoder are used to implement all the embodiments mentioned above regarding image encoding (for the encoder) or code stream decoding (for the decoder), specifically Figures 19 to 24 Those embodiments explained in .

[0480] Fig.25 The encoder shown and Fig.26 The decoder shown can be implemented in any technically reasonable manner. The encoder and / or decoder can be implemented using hardware and software components running on the hardware, wherein the software components implement the above functions. In addition, dedicated hardware for implementing specific functions can also be provided. Similarly, the encoder and / or decoder can be implemented using virtual devices including virtual processors, etc.

[0481] The bitstreams can be parsed independently for the primary and secondary color components and reconstructed using a module consisting of the same sequence of neural network layers, with the only difference being the size of the input tensors and the number of tensor channels.

[0482] The following two different processes use the decoded hyper-prior tensor As input: SuperDecoder and SuperScaleDecoder.

[0483] The super decoder generates the explicit_prediction input of the Multi-stage Context Model (MCM). MCM includes a multi-stage neural network process, which also uses the reconstructed residual Takes as input and outputs a priori spatial tensor After Latent Scaling Before Synthesis (LSBS), the reconstructed prior space tensor Can be used for signal reconstruction. The prior tensor reconstruction of the principal component and the auxiliary component are independent of each other.

[0484] For the purposes of this document, the following terms and definitions apply.

[0485] Padding layer

[0486] The padding layer is represented by Padd(H in ,W in ,d,s d ), where H in and W in is the height and width of the tensor input to the analysis transform, s d is the stride of the current convolution, and d is the depth of the convolutional layer in the deep learnable encoder. The padding layer receives a size of [C,h d-1 ,w d-1 ] and output a tensor of size [C,s d h d ,s d w d ], where h d =ceil(h d-1 / s d ), w d =ceil(w d-1 / s d ), h0=H in , w0=W in By default, padding is performed by copying. A different padding model can be specified (e.g., 0 padding).

[0487] Cropping layer

[0488] The crop layer is represented by Crop(H in ,Win ,d,s d ), where H in and W in is the height and width of the tensor output to the composite transform, s d is the stride of the current transposed convolution, and d is the depth of the convolutional layer in the deep learnable reconstruction process. The crop layer receives the size [C, s d h d ,s d w d ] and output a tensor of size [C,h d-1 ,w d-1 ], where h d =ceil(h d-1 / s d ), w d =ceil(w d-1 / s d ), h0=H in , w0=W in . Padding is performed by discarding redundant elements.

[0489] Convolution layer

[0490] Two-dimensional convolution is represented as CONV(K ver ×K hor ,C in ,C out ,s↓). The convolutional layer receives the size of [C in ,h in ,w in ] and output a tensor of size [C out ,h out ,w out ], where h in =s·h out , w in =s·w out , the factor s is called the step size. If there is no step size parameter, the spatial resolution will not change.

[0491] Transposed convolution

[0492] Transposed convolution is represented as CONV -1 (K ver ×K hor ,C in ,C out ,s↑). The transposed convolution process receives a size of [C in ,h in ,w in ] and output a tensor of size [C out,h out ,w out ], where h out =s·h in , w out =s·w in , the factor s is called the step size.

[0493] Quantized convolution

[0494] Two-dimensional quantized convolution is represented as qCONV(K ver ×K hor ,C in ,C out ,s↓,d,p). The convolution process receives a size of [C in ,h in ,w in ] and output an integer tensor of size [C out ,h out ,w out ], where h in =s·h out , w in =s·w out , the factor s is called the step size. If there is no step size parameter (the step size parameter should be equal to 1), the spatial resolution will not be changed. The parameter d is a non-negative integer value that is used to limit the maximum amplitude of the input tensor elements after clipping. The tensor p[C out ] includes an inverse scaling shift of each channel of the output tensor.

[0495] Quantized transposed convolution

[0496] The two-dimensional quantized transposed convolution is denoted as qCONV -1 (K ver ×K hor ,C in ,C out ,s↑,d,p). The convolution process receives a size of [C in ,h in ,w in ] and output a 16-bit integer tensor of size [C out ,h out ,w out ], where h out =s·h in , w out =s·w in , the factor s is called the step size. The parameter d is a non-negative integer value used to limit the maximum amplitude of the input tensor elements after clipping. The tensor p[C out] includes an inverse scaling shift of each channel of the output tensor.

[0497] Pixel shuffle layer

[0498] The pixel reorganization layer is also called sub-pixel convolution, denoted as PixelShuffle(s), where s>1 is the dilation factor. This layer converts the shape of [C in ,h in ,w in ] to a tensor input of shape [C out ,h out ,w out ], where h out =s·h in , w out =s·w in ; C out =C in / s 2 .

[0499] Residual Activation Unit

[0500] The residual activation unit is denoted as ResAU(K ver ×K hor ). This layer receives the size of [C,h k ,w k ] and execute Fig.27A The above series of steps outputs a tensor of the same size. Here, ⊙ is an element-by-element multiplication, is the addition of tensors of the same size.

[0501] Residual Activation

[0502] The residual activation is denoted as ResA(K ver ×K hor ). This layer receives the size of [C,h k ,w k ] and execute Fig.27B The above series of steps outputs a tensor of the same size. Here, is the addition of tensors of the same size.

[0503] Residual Non-local Attention Block

[0504] The residual non-local attention block is denoted as RNAB(∝). This layer receives the size of [C, k ,w k ] and execute Fig.27C The above series of steps outputs a tensor of the same size. Here, ⊙ is an element-by-element multiplication, is the addition of tensors of the same size. The multiplier ∝ controls the strength of the modifications introduced by RNAB. When ∝ = 0, all operations introduced by RNAB are essentially ignored.

[0505] Residual Block

[0506] The residual block is denoted as RB. This layer receives the size of [C,h k ,w k ] and execute Fig.27D The described series of steps outputs a tensor of the same size.

[0507] Lightweight residual block

[0508] Lightweight residual block is denoted as LRB. This layer receives the size of [C,h k ,w k ] and execute Fig.27E The described series of steps outputs a tensor of the same size.

[0509] Rectified Linear Unit

[0510] The rectified linear unit is denoted as ReLU(). Here is the element-wise function:

[0511]

[0512] Leaky rectified linear unit

[0513] The leaky rectified linear unit is denoted as LeakyReLU(). Here is the element-wise function:

[0514] other

[0515] negative_slope=0.01.

[0516] opIdx is the identifier of the operation point, 0 represents a "basic" operation point, and 1 represents an "advanced" operation point.

[0517] abs operation

[0518] The abs operation is denoted as ABS(). The following is the element-wise function:

[0519] ABS(x)=abs(x)=|x|.

[0520] Skip the model process

[0521] Also called the SKIP mode decoder process, or the SKIP process, or the decoder-side SKIP operation. In the decoder, the input to the skip mode process is as follows:

[0522] -1D array s[num_res_elements] obtained after decoding "stream-y" through me-tANS

[0523] -mask_skip[num_skip_params,C,h4,w4].

[0524] The output of this process is:

[0525] - Residual Tensor

[0526] The output of the lossless decoding process is a 1D array {s k}, whose size is equal to the total number of "1"s in the tensor maskAggregate[C,h4,w4].

[0527] In other words, the tensor maskAggregate[C,h4,w4] determines the residual tensor Which samples in the quantized residual tensor are included in the bitstream. All other samples in the quantized residual tensor are inferred to be 0.

[0528] The residual skip mode process in the decoder is as follows:

[0529] - Set dimensions [C,h4,w4] to the number of channels, height, and width of the σ tensor.

[0530] - The tensor and maskAggregate[C,h4,w4] are initially all 0 and all 1, respectively.

[0531] - Set the counter to k=0.

[0532] - When idx = 0 ... num_skip_params – 1, perform the following steps in the following order:

[0533] - When c = 0..C-1, i = 0..h4-1, j = 0..w4-1,

[0534] maskAggregate[c,i,j]=maskAggregate[c,i,j]·mask_skip[idx,c,i,j]

[0535] - When c = 0..C-1, i = 0..h4-1, j = 0..w4-1,

[0536]

[0537] In one embodiment of the present invention, tensor boundary processing is used in multistage context modeling (MCM) to reduce the amount of data processed in the neural network and improve decoding efficiency. If the tensor size is not an even number, and it is expected to use a downsampling convolution layer or any other type of layer in the NN structure to reduce the size by a factor of 2, a padding layer must be added before this layer, and a cropping layer must be added after the deconvolution / upsampling layer. In the MCM structure, the present invention adds a padding layer before the downsampling layer or any layer with the same function as the downsampling layer, and in the MCM structure, a cropping layer is added after the upsampling layer or any layer with the same function as the upsampling layer to avoid the size of the tensor in any processing step being a non-integer, and to avoid destroying device interoperability (because the fractional size of the tensor is uncertain, so different devices and processors will have different processing methods), while reducing the amount of data processed in the neural network (and memory usage), thereby improving decoding efficiency.

[0538] One embodiment of the present invention discloses a method for processing an image using a neural network (NN), such as Fig.28 The NN includes a multi-stage context model (MCM), and the MCM includes multiple MCMs. k A model, a first lower recombinant layer, a second lower recombinant layer, and an upper recombinant layer, wherein the first lower recombinant layer is preceded by a first filling layer, the second lower recombinant layer is preceded by a second filling layer, and the upper recombinant layer is followed by a cutting layer; the method 2800 comprises:

[0539] 2810: Obtain a first tensor, wherein the first tensor is an output of a skip model process;

[0540] 2820: Obtain a second tensor, wherein the second tensor is an output of a super decoder;

[0541] 2830: Use the first filling layer to fill the first tensor;

[0542] 2840: Performing lower reorganization on the padded first tensor according to the first lower reorganization layer to obtain a reorganized first tensor;

[0543] 2850: Use the second filling layer to fill the second tensor;

[0544] 2860: Performing down reorganization on the padded second tensor according to the second down reorganization layer to obtain a reorganized second tensor;

[0545] 2870: According to the multiple MCMsk A model is provided, processing the reorganized first tensor and the reorganized second tensor to obtain a priori spatial tensor;

[0546] MCM k The model may also be called a phase or MCM k Stages, and the number of stages can be equal to 8, but is not limited to 8, and the number of stages can also be equal to 6 or less, or equal to 10 or more.

[0547] 2880: Perform up-reorganization on the prior space tensor according to the up-reorganization layer to obtain a reorganized prior space tensor;

[0548] 2890: According to the cropping layer, crop the reorganized prior space tensor to obtain a reconstructed prior tensor.

[0549] It is also understood that the precise design of the MCM structure may change accordingly in the future. For example, the MCM structure may include more or fewer stages, more or fewer down-reorganization layers or up-reorganization layers, and the MCM structure may also include some other layers to change the size or shape of the tensor. However, no matter how the MCM structure is changed, any down-sampling layer / down-reorganization layer or any layer with the same function as the down-sampling layer must be preceded by a padding layer, and any up-sampling layer / up-reorganization layer or any layer with the same function as the up-sampling layer must be followed by a cropping layer. This can avoid the size of the tensor being non-integer in any processing step, avoid destroying device interoperability (because the fractional size of the tensor is uncertain, so different devices and processors will have different processing methods), and can also reduce the amount of data processed in the neural network (and memory usage), thereby improving decoding efficiency.

[0550] In one embodiment, the number of inputs of the tensor slices of the first lower recombination layer is 2, and the number of inputs of the tensor slices of the second lower recombination layer is 1.

[0551] In one embodiment, the number of input tensor slices of the upper reorganization layer is 2.

[0552] In one embodiment, the first filling layer is represented by Padd(H in ,W in ,d,s d ), where H in and W in is the height and width of the first tensor, s d is the stride of the current convolution, and d is the depth of the convolutional layer in the deep learnable encoder.

[0553] In one embodiment, the crop layer is represented by Crop(H in ,Win ,d,s d ), where H in and W in is the height and width of the tensor output to the composite transform, s d is the stride of the current transposed convolution, and d is the depth of the convolutional layer in the deep learnable reconstruction process.

[0554] In one embodiment, the step length of the first filling layer is equal to 2 and the depth is equal to 5.

[0555] In one embodiment, the step length of the second filling layer is equal to 2 and the depth is equal to 5.

[0556] In one embodiment, the step size of the clipping layer is equal to 2 and the depth is equal to 5.

[0557] In one embodiment, the first tensor is a reconstructed residual tensor, and the second tensor is an explicit prediction tensor.

[0558] One embodiment of the present invention discloses a neural network (NN). The NN includes a multi-stage context model (MCM), wherein the MCM includes multiple MCMs. k A model, a first lower reorganization layer, a second lower reorganization layer and an upper reorganization layer, wherein the first lower reorganization layer is preceded by a first filling layer, the second lower reorganization layer is preceded by a second filling layer, and the upper reorganization layer is followed by a clipping layer; the first lower reorganization layer and the second lower reorganization layer are used to change the size or shape of a tensor; the upper reorganization layer is used to change the size or shape of a tensor.

[0559] In one embodiment, the number of inputs of the tensor slices of the first lower recombination layer is 2, and the number of inputs of the tensor slices of the second lower recombination layer is 1.

[0560] In one embodiment, the number of input tensor slices of the upper reorganization layer is 2.

[0561] In one embodiment, the first filling layer is represented by Padd(H in ,W in ,d,s d ), where H in and W in is the height and width of the first tensor, s d is the stride of the current convolution, and d is the depth of the convolutional layer in the deep learnable encoder.

[0562] In one embodiment, the crop layer is represented by Crop(H in ,W in ,d,sd ), where H in and W in is the height and width of the tensor output to the composite transform, s d is the stride of the current transposed convolution, and d is the depth of the convolutional layer in the deep learnable reconstruction process.

[0563] In one embodiment, the first padding layer receives a tensor of a first size and outputs a tensor of a second size.

[0564] In one embodiment, the first fill layer and the second fill layer are performed by replication.

[0565] In one embodiment, the MCM k Model uses previous MCM k The model's output tensor is used as input, where k ranges from 0 to k–1.

[0566] An embodiment of the present invention discloses an encoder for encoding an image. The encoder includes a receiver for receiving an image and one or more processors for implementing a neural network (NN), wherein the NN includes a multi-stage context model (MCM), wherein the MCM includes multiple MCMs k Model, the MCM also includes a first lower reorganization layer, a second lower reorganization layer and an upper reorganization layer, the first lower reorganization layer is preceded by a first padding layer, the second lower reorganization layer is preceded by a second padding layer, and the upper reorganization layer is followed by a cropping layer, the encoder also includes a transmitter for outputting a code stream, and the encoder is used to execute the method provided in any of the above embodiments.

[0567] An embodiment of the present invention discloses a decoder for decoding a code stream representing an image. The decoder includes a receiver for receiving the code stream and one or more processors for implementing a neural network (NN), wherein the NN includes a multi-stage context model (MCM), wherein the MCM includes multiple MCMs. k Model, the MCM also includes a first lower reorganization layer, a second lower reorganization layer and an upper reorganization layer, the first lower reorganization layer is preceded by a first padding layer, the second lower reorganization layer is preceded by a second padding layer, and the upper reorganization layer is followed by a cropping layer, the decoder also includes a transmitter for outputting a decoded image, and the decoder is used to execute any one of the methods provided in the above embodiments.

[0568] In one embodiment, the MCM process and some related terms are detailed as follows:

[0569] Multistage Context Modelling (MCM)

[0570] The inputs to the MCM process are as follows:

[0571] - an operating point indicator opIdx,

[0572] -The reconstructed residual tensor That is, the output of the SKIP model process,

[0573] - the explicit prediction tensor p[2C,h4,w4], which is the output of the super decoder,

[0574] -8 MCMs k Model, where k = 0...7 and parameters are defined by (modelIdx, k).

[0575] The output of the MCM process is:

[0576] - The reconstructed prior tensor

[0577] The MCM process includes the following steps:

[0578] -After the padding layer (depth = 5, stride = 2), the explicit prediction tensor p[2C,h4,w4] is reorganized by M = 2 to obtain the reshaped prediction tensor

[0579] After the padding layer (depth = 5, stride = 2), the reconstructed residual tensor Perform M=1 down-reorganization to obtain the reshaped residual tensor

[0580] -Will Divided into 4 parts l = 0...3 (each part includes 2C channels out of 8C channels),

[0581] -Will Divided into 8 parts k=0...7 (each part includes C / 2 channels out of 4C channels),

[0582] -When k = 0 ... 3,

[0583] ○MCM(k) process

[0584] Input:

[0585] ● (m=0...k-1), represents the previously reconstructed part of the reshaped prior space tensor,

[0586] ● represents the concatenated part of the reconstructed residual tensor,

[0587] ● represents the part of the reshaped explicit prediction tensor,

[0588] Output:

[0589] ●

[0590] -Pair tensor Channel net process

[0591] -When k=3...7,

[0592] ○MCM(k) process

[0593] Input:

[0594] ● (m=0...k-1), represents the previously reconstructed part of the reshaped prior space tensor,

[0595] ● represents the concatenated part of the reconstructed residual tensor,

[0596] ● represents the part of the reshaped explicit prediction tensor,

[0597] Output:

[0598] ●

[0599] -right Performing the upper reorganization with M=2 gives Then pass through the clipping layer (depth = 5, stride = 2)

[0600] -right Perform an upper recombination with M=2 to obtain μ[C,h4,w4] (further used in the LSBS process).

[0601] An example of a multi-stage context modeling process is Fig.29 The MCM process is a cyclic process: the subsequent stages use the previously obtained output tensor elements as input.

[0602] Another example of a multi-stage context modeling process is Fig.29A shown.

[0603] In this example, the multi-stage MCM includes 4 MCMs kModel,The multi-stage MCM also includes a lower reorganization layer and an upper reorganization layer, the lower reorganization layer is preceded by a padding layer, and the upper reorganization layer is followed by a cropping layer.

[0604] The multi-stage MCM can be used to implement an image processing method, the method comprising:

[0605] Obtain an input tensor, wherein the input tensor may be an output of a residual decoder, and the input tensor may be referred to as a reconstructed residual tensor;

[0606] padding a first tensor using a padding layer before a next reorganization layer, wherein the first tensor is an input tensor or a tensor obtained by processing the input tensor;

[0607] After the upper reorganization layer outputs the second tensor, the second tensor is cropped using the cropping layer.

[0608] according to Fig.29A It can be understood that the first tensor is padded and reorganized to obtain the reorganized first tensor, and then the reorganized first tensor is input into the 4 MCMs. k In the model, the upper reorganization layer is used to reconstruct four MCMs. k The output of the model is reorganized and the second tensor is output. In other words, the first tensor after reorganization is 4 MCMs k Model input, 4 MCMs k The output of the model is the input of the upper reorganization layer, and the second tensor is the output of the upper reorganization layer. k The model also includes another input, which is a tensor of superdecoder outputs.

[0609] In a specific embodiment, the inputs to the process include:

[0610] -The reconstructed residual tensor That is, the output of the inverse residual scaling process,

[0611] - The reorganized explicit prediction tensor That is, the output of the super decoder,

[0612] -4 multi-stage context modeling MCM k Model parameters, where k = 0..3.

[0613] The output of this process is:

[0614] - The reconstructed prior tensor

[0615] The process includes the following steps:

[0616] -Residual go through Filling layer, get the size [2h5,2w5], and then do Lower Reorganization , resulting in a reorganized residual tensor

[0617] -Will and Divide into 4 parts of size [C,h5,w5] (each part consists of C channels out of 4C channels):

[0618] -for l=0..3

[0619] -for c=0..C–1

[0620] -for i=0..h5–1

[0621] -for j=0..w1

[0622] -

[0623] -

[0624] For l=0..3

[0625] ○ Call the MCM(l) procedure (specified in subsections 0, 13.3.5, 13.3.6, 13.3.7)

[0626] Input:

[0627] ● (m=0,…,l–1), represents the previously reconstructed part of the reshaped prior domain tensor,

[0628] ● , represents the concatenated part of the reconstructed residual tensor,

[0629] ● , represents the concatenation part in the explicit prediction tensor,

[0630] Output

[0631] -Will Cascade

[0632] -right conduct Reorganization (Section 4.6.23), and then through the clipping layer, the size is [h4,w4], resulting in

[0633] This is a cyclic process, with later stages using previously obtained output tensor elements as input. The data flow can be traced by the arrows, with MCM stages 1 to 3 using the output of previous stages.

[0634] Understandably, Fig.29 and Fig.29A The multi-stage context modeling process in is flexible in many ways. Fig.29 and Fig.29A The specific components of the embodiments / architectures can be interchanged. For example, the number of stages can be equal to Fig.29 In addition, the number of stages can be changed to 4. In addition, the number of stages is not limited, and the number of stages can be equal to 6 or less, and can also be equal to 10 or more. The number of lower recombination layers or upper recombination layers is not limited, Fig.29 There are two lower reorganization layers and two filling layers in the MCM structure, or there may be only one lower reorganization layer and one filling layer. The MCM structure may include more or fewer lower reorganization layers or upper reorganization layers, and the MCM structure may also include other layers to change the size or shape of the tensor.

[0635] Next reorganization operation

[0636] The inputs to this process are as follows:

[0637] -M, indicates the number of tensor slices,

[0638] -a[MC,h4,w4], which indicates a 3D tensor.

[0639] The output of this process is:

[0640] - Represents a reorganized 3D tensor with identical elements.

[0641] In the next reshuffle operation, the input slices are first divided into M pieces along the channel dimension, and then the tensor elements of each slice are divided into 4 groups: 0, 1, 2, and 3 (note that the group numbers are not in raster order but in z-order), and these groups are reshuffled along the channel dimension.

[0642] Since the down-reorganization operation changes the spatial size of the tensor in a manner similar to a downsampling convolution with a stride of 2, the down-reorganization process is preceded by a padding layer (h4,w4,1,2), as shown in Fig.29 As shown. Zero padding is performed. In other words, reorganization is used to change the size or shape of a tensor.

[0643] Reorganization operation

[0644] This process is the opposite of the lower recombination operation.

[0645] The inputs to this process are as follows:

[0646] -M, indicates the number of tensor slices,

[0647] - Represents a 3D tensor.

[0648] The output of this process is:

[0649] - The reorganized 3D tensor a[MC,h4,w4] with the same elements.

[0650] In the up-reorganization operation, the input slice is first divided into M slices in the channel dimension, and then the tensor elements of each slice are reorganized in a z-shaped order. Then the number of channels is reduced by 1 / 4 and the spatial dimension is doubled.

[0651] Since the upscaling operation changes the spatial size of the tensor in a manner similar to a deconvolution with a stride of 2, this upscaling process is followed by a cropping layer ((h4,w4,1,2)). In other words, upscaling is used to change the size or shape of a tensor.

[0652] Channel Net

[0653] The inputs to this process are:

[0654] - Represents the first half of the size of the reorganized prior tensor.

[0655] The output of this process is:

[0656] - Represents a modified tensor of the same size.

[0657] The process of channel network is as follows Fig.30 As shown. The first input enters the upper reorganization process, and then a set of three convolutions with a stride of 1 3×3 is performed. The first convolution doubles the number of channels (2C→4C), the second convolution keeps the number of channels unchanged, and the third convolution reduces the number of channels to C / 2. The activation function is the rectified linear unit. Finally, the following reorganization process ends.

[0658] Stage 0 of Multi-Stage Context Modeling

[0659] The inputs to this process are as follows:

[0660] - represents the portion k=0 in the reorganized explicit prediction tensor,

[0661] - represents the part k=0 in the reorganized residual tensor,

[0662] - Prediction fusion network parameters when k=0.

[0663] The output of this process is:

[0664] - represents the part k=0 in the reshaped prior space tensor.

[0665] The process is as follows:

[0666] - After channel filling 2C→3C, a0[3C,h5,w5] is generated.

[0667] -a0[3C,h4 / 2,w4 / 2] enters the prediction fusion network k=0, generating

[0668] -

[0669] Phase 1 of Multi-Phase Context Modeling

[0670] The inputs to this process are as follows:

[0671] - represents the portion k=1 in the reorganized explicit prediction tensor,

[0672] - represents the part k=1 in the reorganized residual tensor,

[0673] - represents the portion k=0 in the reorganized and reconstructed prior tensor,

[0674] - trained parameters of CONVk=1(3×3,C / 2,C / 2),

[0675] - Prediction fusion network parameters when k=1.

[0676] The output of this process is:

[0677] - represents the portion k=1 in the reshaped prior space tensor.

[0678] The process is as follows:

[0679] - After the convolution layer CONV(3×3,C / 2,C / 2), a size of [C / 2,h5,w5] is generated

[0680] - After channel filling 2C→5C / 2, P[5C / 2,h5,w5] is generated.

[0681] - Cascaded with P1 to form a1, and enter the prediction fusion network k = 1, generating

[0682] -

[0683] Phase 2 of Multi-Phase Context Modeling

[0684] The inputs to this process are as follows:

[0685] - represents the portion k=2 in the reorganized explicit prediction tensor,

[0686] - represents the part k=2 in the reorganized residual tensor,

[0687] - represents the parts k=0 and k=1 in the reorganized and reconstructed prior tensors,

[0688] -CONV k＝2 The trained parameters of (3×3,C,C / 2) are

[0689] - Prediction fusion network parameters when k=2.

[0690] The output of this process is:

[0691] - represents the part k=2 in the reshaped prior space tensor.

[0692] The process is as follows:

[0693] - and Cascaded and enter the convolutional layer CONV(3×3,C,C / 2), generating a size of [C / 2,h5,w5]

[0694] - After channel filling 2C→5C / 2, P[5C / 2,h5,w5] is generated.

[0695] - Cascaded with P2 to form a2, and enter the prediction fusion network k = 2, generating

[0696] -

[0697] Phase 3 of Multi-Phase Context Modeling

[0698] The inputs to this process are as follows:

[0699] - represents the portion k=3 in the reorganized explicit prediction tensor,

[0700] - represents the part k=3 in the reorganized residual tensor,

[0701] - represents the parts k=0, k=1 and k=2 in the reorganized and reconstructed prior tensor,

[0702] - trained parameters of CONVk = 3 (3×3, 3C / 2, C / 2),

[0703] - Prediction fusion network parameters when k=3.

[0704] The output of this process is:

[0705] - represents the part k=3 in the reshaped prior space tensor.

[0706] The process is as follows:

[0707] - and Cascaded and passed through the convolutional layer CONV(3×3,3C / 2,C / 2), generating a size of [C / 2,h5,w5] - After channel filling 2C→5C / 2, P[5C / 2,h5,w5] is generated.

[0708] - Cascaded with P3 to form a3, and enter the prediction fusion network k=3, generating

[0709] -

[0710] Phase 4 of Multi-Phase Context Modeling

[0711] The inputs to this process are as follows:

[0712] - represents the portion k=0 in the reorganized explicit prediction tensor,

[0713] - represents the part k=4 in the reorganized residual tensor,

[0714] - The part representing the output of the channel network (E.5.4) is k=0,

[0715] - Prediction fusion network parameters when k=4.

[0716] The output of this process is:

[0717] - represents the k=4 part of the reshaped prior space tensor.

[0718] The process is as follows:

[0719] - After channel filling 2C→5C / 2, P4[5C / 2,h5,w5] is generated.

[0720] - Cascaded with P4 to form a4, and enter the prediction fusion network k=4, generating

[0721] -

[0722] Stage 5 of Multi-Stage Context Modeling

[0723] The inputs to this process are as follows:

[0724] - represents the portion k=1 in the reorganized explicit prediction tensor,

[0725] - represents the part k=5 in the reorganized residual tensor,

[0726] - The part k=1 representing the output of the channel network (E.5.4),

[0727] - represents the part k=4 in the reshaped prior space tensor,

[0728] -CONV k＝5 The trained parameters of (3×3,C / 2,C / 2) are

[0729] - Prediction fusion network parameters when k=5.

[0730] The output of this process is:

[0731] - represents the portion k=5 in the reshaped prior space tensor.

[0732] The process is as follows:

[0733] - After the convolution layer CONV(3×3,C / 2,C / 2), a size of [C / 2,h5,w5] is generated

[0734] - and The cascade forms a5 and enters the prediction fusion network k=5, generating

[0735]

[0736] Stage 6 of Multi-Stage Context Modeling

[0737] The inputs to this process are as follows:

[0738] - represents the portion k=2 in the reorganized explicit prediction tensor,

[0739] - represents the part k=6 in the reorganized residual tensor,

[0740] - The part representing the output of the channel network (E.5.4) is k=2,

[0741] - represents the part k=4,5 in the reshaped prior space tensor,

[0742] -CONV k＝6 The trained parameters of (3×3,C,C / 2) are

[0743] - Prediction fusion network parameters when k=6.

[0744] The output of this process is:

[0745] - represents the portion k=6 in the reshaped prior space tensor.

[0746] The process is as follows:

[0747] - and Cascaded and passed through the convolutional layer CONV(3×3,C,C / 2), generating a size of [C / 2,h5,w5]

[0748] - and The cascade forms a6 and enters the prediction fusion network k=6, generating

[0749] -

[0750] Stage 7 of Multi-Stage Context Modeling

[0751] The inputs to this process are as follows:

[0752] - represents the portion k=3 in the reorganized explicit prediction tensor,

[0753] - represents the part k=7 in the reorganized residual tensor,

[0754] - The part k=3 representing the output of the channel network (E.5.4),

[0755] - represents the part k=4, 5, 6 in the reshaped prior space tensor,

[0756] -CONV k＝7 The trained parameters of (3×3,3C / 2,C / 2) are

[0757] - Prediction fusion network parameters when k=7.

[0758] The output of this process is:

[0759] - represents the portion k=7 in the reshaped prior space tensor.

[0760] The process is as follows:

[0761] - and Cascaded and passed through the convolutional layer CONV(3×3,3C / 2,C / 2), generating a size of [C / 2,h5,w5]

[0762] - and The cascade forms a7 and enters the prediction fusion network k=7, generating

[0763]

[0764] MCM Design Elements

[0765] Three operations that change the size or shape of tensors: (1) reshuffling, (2) reshuffling, and (3) channel-by-channel padding. Two sub-networks: channel network and prediction fusion network.

[0766] In one embodiment of the present invention, tensor boundary processing is used in a super-large-scale decoder to reduce the amount of data processed in a neural network, thereby improving decoding efficiency. Tensor boundary processing refers to adding a padding layer before a downsampling layer / down-reorganization layer or any layer with the same function as a downsampling layer / down-reorganization layer and adding a cropping layer after an upsampling layer / up-reorganization layer or any layer with the same function as an upsampling layer / up-reorganization layer. This ensures that the tensor size is an integer in any processing step, thereby avoiding uncertainty (uncertain processes lead to platform dependencies and undermine device interoperability). Since the super-large-scale decoder outputs parameters for the entropy encoder / decoder, it must have bit-accurate behavior, otherwise the bits parsed from the bitstream cannot be correctly interpreted. Convolution has a parameter s, called "step size". If the step size is 1, the height and width of the input tensor and the output tensor are the same. However, if s>1, then h_out=h_in / s, w_out=w_in / s. Assuming h_in is 33 and s=2, the size of the output tensor is h_out=33 / 2=16.5. The tensor height is the number of elements and it must be an integer. The canonical equation h_out=h_in / s to keep the result as a decimal introduces uncertainty, rounding h from 16.5 to 16 in some implementations and to 17 in others. This is where the problem lies, different devices will operate differently and the decoder will crash when decoding a stream from another device (no device interoperability).

[0767] One embodiment of the present invention discloses a method for processing an image using a neural network (NN), such as Fig.31 The NN includes a very large-scale decoder, the very large-scale decoder includes a basic operation point and an advanced operation point, under the basic operation point, the very large-scale decoder includes a first quantized transposed convolution layer followed by a first cropping layer and a first rectified linear unit, a first quantized convolution layer followed by a second rectified linear unit, a second quantized transposed convolution layer followed by a second cropping layer and a third rectified linear unit, and a second quantized convolution layer;

[0768] At the advanced operating point, the ultra-large-scale decoder includes two quantized convolutional layers and three quantized convolutional layers, each of the two quantized convolutional layers is followed by a pixel reassembly layer, a cropping layer, and a rectified linear unit in sequence, and two of the three quantized convolutional layers are followed by rectified linear units respectively; method 3100 includes:

[0769] 3110: Get input tensor;

[0770] 3120: Get the size of the input tensor;

[0771] 3130: Get the operating point indicator;

[0772] 3140: Determine, according to the operation point indicator, whether to use the basic operation point or the advanced operation point to process the input tensor;

[0773] 3150: Output the processed tensor.

[0774] It can be understood that the above-mentioned ultra-large-scale decoder includes two processing pipelines, one pipeline is a basic operating point, and the other pipeline is a high-basic operating point. In some other embodiments, the basic operating point can also be named as a basic profile, a baseline, a basic pipeline, a basic channel, a basic sub-network or some other names, and accordingly, the advanced operating point can also be named as an advanced profile, a high line, an advanced pipeline, an advanced channel, an advanced sub-network or some other names.

[0775] It is also understandable that the precise design of the above-mentioned super-large-scale decoder may change accordingly in the future. For example, the super-large-scale decoder may include more or fewer quantized transposed convolutional layers, more or fewer quantized convolutional layers, and the super-large-scale decoder may also include some other layers to change the size or shape of the tensor, but no matter how the super-large-scale decoder is changed, any downsampling layer / down-reorganization layer or any layer with the same function as the downsampling layer must be preceded by a padding layer, and any upsampling layer / up-reorganization layer or any layer with the same function as the upsampling layer must be followed by a cropping layer.

[0776] In one embodiment, the kernel sizes of the first quantized transposed convolution layer and the second quantized transposed convolution layer are both 4×4, and the strides are both equal to 2.

[0777] In one embodiment, the step size of the first cropping layer is equal to 2 and the depth is equal to 6.

[0778] In one embodiment, the step size of the second cropping layer is equal to 2 and the depth is equal to 5.

[0779] In one embodiment, the step size of the cropping layer after the first of the two quantized convolutional layers at the advanced operating point is equal to 2, and the depth is equal to 6; the step size of the cropping layer after the second of the two quantized convolutional layers at the advanced operating point is equal to 2, and the depth is equal to 5.

[0780] In one embodiment, the processed tensor is a very large-scale decoder standard deviation tensor.

[0781] In one embodiment, when the operation point indicator is equal to 0, the input tensor is processed using the base operation point.

[0782] In one embodiment, when the operation point indicator is equal to 1, the input tensor is processed using the high-level operation point.

[0783] In one embodiment, the kernel size of the first quantized convolutional layer is 3×3.

[0784] In one embodiment, the pixel reorganization layer is used to change the number of channels from 4C to C.

[0785] One embodiment of the present invention discloses a neural network (NN). The NN includes a very large-scale decoder, the very large-scale decoder includes a basic operation point and an advanced operation point, at the basic operation point, the very large-scale decoder includes a first quantized transposed convolutional layer followed by a first cropping layer and a first rectified linear unit, a first quantized convolutional layer followed by a second rectified linear unit, a second quantized transposed convolutional layer followed by a second cropping layer and a third rectified linear unit, and a second quantized convolutional layer;

[0786] At the advanced operating point, the ultra-large-scale decoder includes two quantized convolutional layers and three quantized convolutional layers, each of the two quantized convolutional layers is followed by a pixel reconstruction layer, a cropping layer and a rectified linear unit in sequence, and two of the three quantized convolutional layers are respectively followed by a rectified linear unit.

[0787] One embodiment of the present invention discloses an encoder for encoding an image. The encoder includes a receiver for receiving an image and one or more processors for implementing a neural network (NN), the NN includes a super-large-scale decoder, the super-large-scale decoder includes a basic operation point and an advanced operation point, under the basic operation point, the super-large-scale decoder includes a first quantized transposed convolution layer followed by a first cropping layer and a first rectified linear unit, a first quantized convolution layer followed by a second rectified linear unit, a second quantized transposed convolution layer followed by a second cropping layer and a third rectified linear unit, and a second quantized convolution layer;

[0788] At the advanced operating point, the ultra-large-scale decoder includes two quantized convolution layers and three quantized convolution layers, each of the two quantized convolution layers is followed by a pixel reconstruction layer, a cropping layer and a rectified linear unit in sequence, two of the three quantized convolution layers are followed by a rectified linear unit respectively, and the encoder also includes a transmitter for outputting a code stream, and the encoder is used to execute the method provided in any of the above embodiments.

[0789] One embodiment of the present invention discloses a decoder for decoding a code stream representing an image. The decoder includes a receiver for receiving the code stream and one or more processors for implementing a neural network (NN), the NN includes a very large-scale decoder, the very large-scale decoder includes a basic operation point and an advanced operation point, under the basic operation point, the very large-scale decoder includes a first quantized transposed convolution layer followed by a first cropping layer and a first rectified linear unit, a first quantized convolution layer followed by a second rectified linear unit, a second quantized transposed convolution layer followed by a second cropping layer and a third rectified linear unit, and a second quantized convolution layer;

[0790] At the advanced operating point, the ultra-large-scale decoder includes two quantized convolutional layers and three quantized convolutional layers, each of the two quantized convolutional layers is followed by a pixel reconstruction layer, a cropping layer and a rectified linear unit in sequence, two of the three quantized convolutional layers are followed by a rectified linear unit respectively, the decoder also includes a transmitter for outputting a decoded image, and the decoder is used to execute any of the above embodiments.

[0791] The VLSI decoder is also disclosed as follows:

[0792] Hyper Scale Decoder

[0793] An example of a very large-scale decoder is Fig.32 shown.

[0794] The inputs to the VLSI decoder include:

[0795] -Reconstructed supertensor

[0796] - Size H of input / output tensors in and W in ,

[0797] - an operating point indicator opIdx,

[0798] - Model parameters of the very large decoder network defined by a pair (modelIdx, opIdx), where all multiplier parameters in these models are 8-bit integers.

[0799] The output is the ultra-large decoder standard deviation tensor σ[C,h4,w4].

[0800] All operations in the scalable super decoder are integers, all accumulators in the calculation are in the range of 32-bit integers, and the model parameters are quantized to 8-bit integers. This ensures bit-accurate behavior of the neural network module. The super-scale decoder uses special types of operations: quantized convolution and quantized transposed convolution. For each quantized convolution in the process, a set of clipping values ​​{d k} and the inverse scaling shift parameter {p k}(1≤k≤(opIdx=0)?4:5). All the limit values ​​in the quantized convolution are d k =2 15 -1. Inverse scaling shift {p k} is part of the trained model.

[0801] Note: The size of the input weights in the quantized model will not exceed 2 15 The combination of the shift and clipping values ​​ensures that the registers for the quantized convolution are within the 32-bit range.

[0802] According to the operating point indicator (opIdx), the ultra-large-scale decoder performs the following series of steps.

[0803] In the ultra-large-scale decoder, at the base operating point (opIdx=0), the number of channels in all hidden layers is C. The ultra-large-scale decoder starts with a quantized transposed convolution with a kernel size of 4×4 and a stride of 2, followed by a crop layer (stride of 2, depth of 6) and a rectified linear unit. Then there is a quantized convolution with a kernel size of 3×3 and a stride of 1, followed by a rectified linear unit. Again, there is a quantized transposed convolution with a kernel size of 4×4 and a stride of 2, followed by a crop layer (stride of 2, depth of 5) and a rectified linear unit, and finally a quantized convolution with a kernel size of 3×3 and a stride of 1.

[0804] At the high-level operating point (opIdx=1), two sets of quantized convolutions with kernel size 3×3 and stride 1 increase the number of channels to 4C, followed by pixel rebinding (stride 2) to reduce the number of channels to C, cropping layers (stride 2, depth 6 and 5), and finally rectified linear units. Then there are three quantized convolutions with kernel size 3×3 and stride 1, two of which are followed by rectified linear units.

[0805] In one embodiment of the present invention, tensor boundary processing is used in the synthesis transform network to reduce the amount of data processed in the neural network, thereby improving decoding efficiency. Tensor boundary processing refers to adding a padding layer before the downsampling layer or any layer with the same function as the downsampling layer and adding a cropping layer after the upsampling layer or any layer with the same function as the upsampling layer. This ensures that the tensor size is an integer in any processing step without generating uncertainty. If the cropping layer is not configured correctly in the synthesis transform, the reconstructed image size is different from the encoded image size.

[0806] One embodiment of the present invention discloses a method for processing an image using a neural network (NN), such as Fig.33 The NN comprises a synthetic transform network, the synthetic transform network comprises a cascade layer for concatenating a main tensor and an auxiliary tensor as an input tensor, a basic operation point and an advanced operation point, under which the synthetic transform network comprises a light-weight residual block followed by a first transposed convolution layer combined with a first cropping layer and a first residual activation unit, a second transposed convolution layer combined with a second cropping layer and a second residual activation unit, a first convolution layer combined with a third residual activation unit, and a second convolution layer followed by a first pixel reorganization layer and a third cropping layer;

[0807] At the high-level operating point, the synthetic transform network includes two residual blocks followed by a third transposed convolutional layer combined with a fourth cropping layer and a first residual activation, a fourth transposed convolutional layer combined with a fifth cropping layer and a second residual activation, a third convolutional layer followed by a second pixel reassembly layer and a residual non-local attention block combined with a sixth cropping layer and a third residual activation, and a fifth transposed convolutional layer followed by a seventh cropping layer; method 3300 includes:

[0808] 3310: Get input tensor by concatenating main tensor and auxiliary tensor;

[0809] 3320: Get the operating point indicator;

[0810] 3330: Get the size of the input tensor;

[0811] 3340: Determine, according to the operation point indicator, whether to use the basic operation point or the advanced operation point to process the input tensor;

[0812] 3350: Output the processed tensor.

[0813] It can be understood that the above-mentioned synthetic transformation network includes two processing pipelines, one pipeline is a basic operation point, and the other pipeline is an advanced operation point. In some other embodiments, the basic operation point can also be named as a basic profile, a baseline, a basic pipeline, a basic channel, a basic sub-network or some other names, and correspondingly, the advanced operation point can also be named as an advanced profile, a high line, an advanced pipeline, an advanced channel, an advanced sub-network or some other names.

[0814] It can also be understood that the precise design of the above-mentioned synthetic transformation network may change accordingly in the future. For example, the synthetic transformation network may include more or fewer quantized transposed convolutional layers, more or fewer quantized convolutional layers, more or fewer pixel reorganization layers, and the synthetic transformation network may include some other layers to change the size or shape of the tensor. However, no matter how the synthetic transformation network is changed, any downsampling layer / down-reorganization layer or any layer with the same function as the downsampling layer / down-reorganization layer must be preceded by a padding layer, and any upsampling layer / up-reorganization layer or any layer with the same function as the upsampling layer / up-reorganization layer must be followed by a cropping layer.

[0815] In one embodiment, the step size of the first cropping layer is equal to 2 and the depth is equal to 4.

[0816] In one embodiment, the step size of the second cropping layer is equal to 2 and the depth is equal to 3.

[0817] In one embodiment, the step size of the third clipping layer is equal to 4 and the depth is equal to 1.

[0818] In one embodiment, the fourth clipping layer has a step size of 2 and a depth of 4.

[0819] In one embodiment, the step size of the fifth cropping layer is equal to 2 and the depth is equal to 3.

[0820] In one embodiment, the sixth clipping layer has a step size of 2 and a depth of 2.

[0821] In one embodiment, the step size of the seventh clipping layer is equal to 2 and the depth is equal to 1.

[0822] In one embodiment, when the operation point indicator is equal to 0, the input tensor is processed using the base operation point.

[0823] In one embodiment, when the operation point indicator is equal to 1, the input tensor is processed using the high-level operation point.

[0824] In one embodiment, the prior volume is a reconstructed prior space tensor.

[0825] One embodiment of the present invention discloses a neural network (NN). The NN includes a synthetic transformation network, the synthetic transformation network includes a cascade layer for cascading a main tensor and an auxiliary tensor as an input tensor, a basic operation point and an advanced operation point, under the basic operation point, the synthetic transformation network includes a lightweight residual block followed by a first transposed convolution layer combined with a first cropping layer and a first residual activation unit, a second transposed convolution layer combined with a second cropping layer and a second residual activation unit, a first convolution layer combined with a third residual activation unit, and a second convolution layer followed by a first pixel reorganization layer and a third cropping layer;

[0826] At the high-level operating point, the synthetic transform network includes two residual blocks followed by a third transposed convolutional layer combined with a fourth cropping layer and a first residual activation, a fourth transposed convolutional layer combined with a fifth cropping layer and a second residual activation, a third convolutional layer followed by a second pixel reassembly layer and a residual non-local attention block combined with a sixth cropping layer and a third residual activation, and a fifth transposed convolutional layer followed by a seventh cropping layer.

[0827] One embodiment of the present invention discloses an encoder for encoding an image. The encoder includes a receiver for receiving an image and one or more processors for implementing a neural network (NN), the NN including a synthetic transform network, the synthetic transform network including a cascade layer for cascading a main tensor and an auxiliary tensor as an input tensor, a basic operation point and an advanced operation point, under the basic operation point, the synthetic transform network includes a lightweight residual block followed by a first transposed convolution layer combined with a first cropping layer and a first residual activation unit, a second transposed convolution layer combined with a second cropping layer and a second residual activation unit, a first convolution layer combined with a third residual activation unit, and a second convolution layer followed by a first pixel reorganization layer and a third cropping layer;

[0828] At the high-level operating point, the synthetic transformer network includes two residual blocks followed by a third transposed convolutional layer combined with a fourth cropping layer and a first residual activation, a fourth transposed convolutional layer combined with a fifth cropping layer and a second residual activation, a third convolutional layer followed by a second pixel reassembly layer and a residual non-local attention block combined with a sixth cropping layer and a third residual activation, and a fifth transposed convolutional layer followed by a seventh cropping layer;

[0829] The encoder further includes a transmitter for outputting a code stream, and the encoder is used to execute the method provided in any of the above embodiments.

[0830] One embodiment of the present invention discloses a decoder for decoding a code stream representing an image. The decoder includes a receiver for receiving the code stream and one or more processors for implementing a neural network (NN), wherein the NN includes a synthetic transform network, wherein the synthetic transform network includes a cascade layer for cascading a main tensor and an auxiliary tensor as an input tensor, a basic operation point, and an advanced operation point, wherein at the basic operation point, the synthetic transform network includes a lightweight residual block followed by a first transposed convolution layer combined with a first cropping layer and a first residual activation unit, a second transposed convolution layer combined with a second cropping layer and a second residual activation unit, a first convolution layer combined with a third residual activation unit, and a second convolution layer followed by a first pixel reorganization layer and a third cropping layer;

[0831] At the high-level operating point, the synthetic transformation network includes two residual blocks followed by a third transposed convolution layer combined with a fourth cropping layer and a first residual activation, a fourth transposed convolution layer combined with a fifth cropping layer and a second residual activation, a third convolution layer followed by a second pixel reorganization layer and a residual non-local attention block combined with a sixth cropping layer and a third residual activation, and a fifth transposed convolution layer followed by a seventh cropping layer; the decoder also includes a transmitter for outputting a decoded image, and the decoder is used to perform any one of the methods provided in the above embodiments.

[0832] An example of a synthetic transform network is Fig.34 shown.

[0833] The learning-based reconstruction (called synthetic transformation) consists of two pipelines with the same neural network architecture but different input sizes and number of channels.

[0834] The inputs to the analysis transform include:

[0835] -with auxiliary information tensor The concatenated prior space tensor of shape [C, h4, w4]

[0836] - an operating point indicator opIdx,

[0837] - Size H of input / output tensors in , W in ,

[0838] - Model parameters of the synthesized transform net defined by a pair (modelIdx, opIdx).

[0839] The output of the analysis transform is the reconstructed color components , that is, the size is [C in ,H in ,W in ] tensor.

[0840] The composite transform is input from the cascade master and auxiliary input Start. Depending on the operating point indicator (opIdx), the decoder performs the following series of steps.

[0841] At the basic operating point (opIdx=0), the number of channels is C+C d A lightweight residual block is followed by two transposed convolutions of kernel size 4×4 combined with a crop layer (stride 2, depth 4 and 3 respectively) and residual activation units with kernel size 3×3. The number of output channels in the transposed convolutions is C1 and C2 respectively. The stride of both transposed convolutions is 2. The next step in the process is a regular convolution of kernel size 3×3, stride 1 and unchanged number of channels C2 combined with residual activation units (kernel size 3×3). Then there is a stride 1 to increase the number of channels from C2 to 16C in This is done to ensure that the output of the next step (i.e. pixel reorganization with a stride of 4) has the number of channels C in The process ends with a crop layer (stride 4, depth 1).

[0842] At the advanced operating point (opIdx=1), the number of channels is C+C d The two residual blocks are followed by two transposed convolutions with kernel size 3×3 combined with crop layers (stride 2, depth 4 and 3 respectively) and residual activations with kernel size 3×3. The number of output channels in both transposed convolutions is C. The stride of both transposed convolutions is 2. The next step in the process is a regular convolution with kernel size 3×3, stride 1, and 4C number of output channels. This is done to ensure that the output of the next step (i.e. pixel rebinning with stride 2) has C number of channels. Then, a residual non-local attention block (∝=1) is performed combined with a crop layer (stride 2, depth 2) and residual activations with kernel size 3×3. This process ends with kernel size ∝=1, stride 2, and C number of output channels. in The structure ends with a transposed convolution and a subsequent cropping layer (with stride 2 and depth 1).

[0843] An embodiment of the present invention discloses a decoder for decoding a code stream representing an image. The decoder includes one or more processors for implementing a neural network (NN), and the one or more processors are used to execute the method provided by any of the above embodiments.

[0844] An embodiment of the present invention discloses an encoder for encoding an image, wherein the encoder comprises one or more processors for implementing a neural network (NN), and the one or more processors are used to execute the method provided in any of the above embodiments.

[0845] An embodiment of the present invention discloses a computer program product including computer executable instructions. When the computer executable instructions are executed on a computing system, the computing system executes any of the methods provided in the above embodiments.

[0846] One embodiment of the present invention discloses a neural network (NN). The NN includes a multi-stage context model (MCM), the MCM includes multiple MCMk models, the MCM also includes one or more lower reorganization layers and one or more upper reorganization layers, the one or more lower reorganization layers are respectively preceded by a padding layer, and the one or more upper reorganization layers are respectively followed by a clipping layer.

[0847] In one embodiment, the NN further comprises a very large-scale decoder, the very large-scale decoder comprising a baseline and a high line, the baseline comprising two or more quantized transposed convolutional layers, the two or more quantized transposed convolutional layers being followed by a cropping layer and a rectified linear unit, respectively, the high line comprising two or more quantized convolutional layers, each of the two or more quantized convolutional layers being followed by a pixel reassembly layer, a cropping layer and a rectified linear unit in sequence.

[0848] In one embodiment, the NN also includes a synthetic transformation network, which includes a baseline and a high line, the baseline includes two or more transposed convolutional layers and one or more convolutional layers, the two or more transposed convolutional layers are respectively followed by a cropping layer and a residual activation unit, and the one or more convolutional layers are respectively followed by a pixel reorganization layer and a cropping layer; the high line includes two or more transposed convolutional layers and a convolutional layer, the two or more transposed convolutional layers are respectively followed by a cropping layer and a residual activation, and the convolutional layer is followed by a pixel reorganization layer and a residual non-local attention block combined with the cropping layer.

[0849] An embodiment of the present invention discloses an encoder for encoding an image, wherein the encoder comprises a receiver for receiving an image, a transmitter for outputting a code stream, and one or more processors for implementing a neural network provided by any of the above embodiments.

[0850] One embodiment of the present invention discloses a decoder for decoding a code stream representing an image, wherein the decoder comprises a receiver for receiving the code stream, a transmitter for outputting a decoded image, and one or more processors for implementing the neural network provided by any of the above embodiments.

[0851] An embodiment of the present invention discloses a method for processing an image using a neural network (NN). The NN includes a multi-stage context model (MCM), the MCM includes multiple MCMk models, the MCM also includes one or more lower reorganization layers and one or more upper reorganization layers, the one or more lower reorganization layers are respectively preceded by a padding layer, and the one or more upper reorganization layers are respectively followed by a cropping layer; the method includes:

[0852] Get the input tensor;

[0853] padding a first tensor using padding layers before the one or more lower reorganization layers, respectively, wherein the first tensor is the input tensor or a tensor obtained by processing the input tensor;

[0854] After the one or more upper reorganization layers respectively output second tensors, the second tensors are cropped using a cropping layer.

[0855] In one embodiment, the NN further comprises a super-large-scale decoder, the super-large-scale decoder comprises a baseline and a high line, the baseline comprises two or more quantized transposed convolutional layers, the two or more quantized transposed convolutional layers are respectively followed by a cropping layer and a rectified linear unit, the high line comprises two or more quantized convolutional layers, each of the two or more quantized convolutional layers is sequentially followed by a pixel reassembly layer, a cropping layer and a rectified linear unit; the method further comprises:

[0856] Get the operating point indicator;

[0857] Based on the operation point indicator, it is determined whether to use the baseline or the advanced operation line to process a third tensor.

[0858] In one embodiment, the NN further comprises a synthetic transformation network, the synthetic transformation network comprising a baseline and a high line, the baseline comprising two or more transposed convolutional layers and one or more convolutional layers, the two or more transposed convolutional layers are respectively followed by a cropping layer and a residual activation unit, and the one or more convolutional layers are respectively followed by a pixel reorganization layer and a cropping layer; the high line comprises two or more transposed convolutional layers and a convolutional layer, the two or more transposed convolutional layers are respectively followed by a cropping layer and a residual activation, and the convolutional layer is followed by a pixel reorganization layer and a residual non-local attention block combined with the cropping layer; the method further comprises:

[0859] Get the second input tensor by concatenating the primary tensor and the secondary tensor;

[0860] Get the operating point indicator;

[0861] Based on the operating point indicator, it is determined whether to use the baseline or the high line to process the second input tensor.

[0862] In one embodiment, when the operating point indicator is equal to 0, the second input tensor is processed using the baseline.

[0863] In one embodiment, when the operating point indicator is equal to 1, the second input tensor is processed using the high line.

[0864] Mathematical operators

[0865] The mathematical operators used in this application are similar to those used in the C programming language. However, the present invention accurately defines the results of integer division and arithmetic shift operations, and also defines other operations, such as power operations and real-valued division. Numbering and counting specifications usually start from 0, for example, "first" is equivalent to the 0th, "second" is equivalent to the 1st, and so on.

[0866] Arithmetic operators

[0867] The arithmetic operators are defined as follows:

[0868] + Addition

[0869] – Subtraction (binary operator) or negation (unary prefix operator)

[0870] * Multiplication, including matrix multiplication

[0871] x y Exponentiation, representing x raised to the power of y. In other contexts, this notation is used as a superscript rather than as an exponentiation.

[0872] / Integer division, truncated towards 0. For example, 7 / 4 and –7 / –4 are truncated to 1, and –7 / 4 and 7 / –4 are truncated to –1.

[0873] ÷ Division operation without truncation or rounding.

[0874] Division operation without truncation or rounding.

[0875] It is used to calculate the sum of f(i), where i is all integers between x and y, including x and y.

[0876] x%y Modulo operation, which means the remainder when x is divided by y, where x and y are both integers, and x>=0, y>0.

[0877] Logical operators

[0878] The logical operators are defined as follows:

[0879] x&&y Boolean logic "AND" operation of x and y

[0880] x||y Boolean logical "OR" operation of x and y

[0881] ! Boolean logic "not" operation

[0882] x? y:z If x is true (TRUE) or not equal to 0, find the value of y; otherwise, find the value of z.

[0883] Relational operators

[0884] The relational operators are defined as follows:

[0885] > Greater than

[0886] >= greater than or equal to

[0887] < less than

[0888] <= less than or equal to

[0889] == is equal to

[0890] ! = Not equal to

[0891] When a relational operator is applied to a syntax element or variable that has been assigned the value "na" (not applicable), the value "na" is considered a different value for the syntax element or variable. The value "na" is considered not equal to any other value.

[0892] Bitwise Operators

[0893] The bitwise operators are defined as follows:

[0894] & Bitwise AND operation. When operating on integer arguments, the operation is performed on the two's complement representation of the integer values. When operating on binary arguments, if it contains fewer bits than the other argument, the shorter argument is extended by adding more significant bits equal to 0.

[0895] | Bitwise OR operation. When operating on integer arguments, the operation is performed on the two's complement representation of the integer values. When operating on binary arguments, if it contains fewer bits than the other argument, the shorter argument is extended by adding more significant bits equal to 0.

[0896] ^ Bitwise XOR operation. When operating on integer arguments, the operation is performed on the two's complement representation of the integer values. When operating on binary arguments, if it contains fewer bits than the other argument, the shorter argument is extended by adding more significant bits equal to 0.

[0897] x >> y Arithmetically shift x to the right by y bit positions in its two's complement integer representation. This function is defined only when y is a non - negative integer value. The result of the right shift is that the bit shifted into the most significant bit (MSB) is equal to the MSB of x before the shift operation.

[0898] x << y Arithmetically shift x to the left by y bit positions in its two's complement integer representation. This function is defined only when y is a non - negative integer value. The result of the left shift is that the bit shifted into the least significant bit (LSB) is equal to 0.

[0899] Assignment operators

[0900] The definitions of arithmetic operators are as follows:

[0901] = Assignment operator

[0902] ++ Increment, i.e., x++ is equivalent to x = x + 1; when used in array indexing, the value of the variable is fetched before the increment operation.

[0903] –– Decrement, i.e., x-- is equivalent to x = x - 1; when used in array indexing, the value of the variable is fetched before the decrement operation.

[0904] += Increment by a specified value, e.g., x += 3 is equivalent to x = x + 3, x += (-3) is equivalent to x = x + (-3).

[0905] -= Decrement by a specified value, e.g., x -= 3 is equivalent to x = x - 3, x -= (-3) is equivalent to x = x - (-3).

[0906] Range notation

[0907] The following notation is used to indicate the range of values:

[0908] x=y..zx takes integer values ​​from y to z (including y and z), where x, y and z are all integers and z is greater than y.

[0909] Mathematical functions

[0910] The mathematical function is defined as follows:

[0911]

[0912] Asin(x) Trigonometric inverse sine function. For x in the range –1.0 to 1.0 (inclusive), the output value is in the range –π÷2 to π÷2 (inclusive). The unit is radians.

[0913] Atan(x) Trigonometric inverse tangent function. It operates on the parameter x and the output value is between –π÷2 and π÷2 (including the end values) in radians.

[0914]

[0915] Ceil(x) The smallest integer greater than or equal to x.

[0916] Clip1 Y (x)=Clip3(0,(1< <BitDepthY)–1,x)

[0917] Clip1 C (x)=Clip3(0,(1< <BitDepthC)–1,x)

[0918]

[0919] Cos(x) Trigonometric cosine function, operates on parameter x, in radians.

[0920] Floor(x) The largest integer less than or equal to x.

[0921]

[0922] Ln(x) returns the natural logarithm of x (base e logarithm, where e is the base constant 2.718281828…).

[0923] Log2(x) is the base 2 logarithm of x.

[0924] Log10(x) is the base 10 logarithm of x.

[0925]

[0926]

[0927] Round(x)=Sign(x)*Floor(Abs(x)+0.5)

[0928]

[0929] Sin(x) Trigonometric sine function, operates on parameter x, in radians.

[0930]

[0931] Swap(x,y)=(y,x)

[0932] Tan(x) Trigonometric tangent function, operates on parameter x, in radians.

[0933] Operation priority order

[0934] When parentheses are not used to explicitly indicate the order of precedence in an expression, the following rules apply:

[0935] - Operations with higher priority are evaluated before any operations with lower priority.

[0936] -Operations of the same priority are evaluated from left to right.

[0937] The following table describes the priority of operations from highest to lowest. The higher the position in the table, the higher the priority.

[0938] The order of precedence adopted in this article is the same as the order of precedence adopted in C programming language if these operators are also used in C programming language.

[0939] Table: Operation priority from highest (top of table) to lowest (bottom of table)

[0940]

[0941] Text description of the logical operation

[0942] In the text, the following logical operation statements are described in mathematical form:

[0943]

[0944]

[0945] It can be described in the following way:

[0946] ...as follows / ...the following applies:

[0947] -If condition 0 then statement 0

[0948] - Else, if condition 1, then statement 1

[0949] -……

[0950] - Otherwise (informative explanation of the remaining conditions), statement n

[0951] Each "if ... else, if ... else, ..." statement in the text is introduced with "... as follows" or "... the following applies" followed by "if ...". The last condition of "if ... else, if ... else, ..." always has an "otherwise, ...". Statements with "if ... else, if ... else" in the middle are identified by matching "... as follows:" or "... the following applies" with the ending statement "otherwise, ...".

[0952] In the text, the following logical operation statements are described in mathematical form:

[0953]

[0954] It can be described in the following way:

[0955] ...as follows / ...the following applies:

[0956] - Statement 0 if all of the following are true:

[0957] -Condition 0a

[0958] -Condition 0b

[0959] - If one or more of the following conditions are met, then statement 1:

[0960] -Condition 1a

[0961] -Condition 1b

[0962] -……

[0963] - Otherwise, statement n

[0964] In the text, the following logical operation statements are described in mathematical form:

[0965]

[0966] It can be described in the following way:

[0967] When condition 0, then statement 0

[0968] When condition 1, then statement 1

[0969] Although the embodiments of the present invention are mainly described according to video decoding, it should be noted that the embodiments of the decoding system 10, the encoder 20 and the decoder 30 (respectively, the system 10) and other embodiments described herein can also be used for still image processing or decoding, that is, processing or decoding a single image in video decoding that is independent of any previous or continuous images. In general, if the image processing decoding is limited to a single image 17, only the inter-frame prediction unit 244 (encoder) and 344 (decoder) are not available. All other functions (also referred to as tools or techniques) of the video encoder 20 and the video decoder 30 can also be used for still image processing, for example, residual calculation 204 / 304, transformation 206, quantization 208, inverse quantization 210 / 310, (inverse) transformation 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354 and / or loop filtering 220 / 320, entropy coding 270 and entropy decoding 304. In general, the embodiments of the present invention can also be applied to other source signals such as audio signals.

[0970] Embodiments of encoders 20, decoders 30, etc. and the functions described herein with reference to encoders 20, decoders 30, etc. may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored as one or more instructions or codes in a computer-readable medium or sent via a communication medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media corresponding to tangible media (e.g., data storage media), or any communication media that facilitates sending a computer program from one place to another according to a communication protocol, etc. In this manner, computer-readable media may generally correspond to (1) non-transitory tangible computer-readable storage media or (2) communication media such as signals or carrier waves. Data storage media may be any available media accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in the present invention. A computer program product may include a computer-readable medium.

[0971] As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, flash memory or any other medium that can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. In addition, any connection can also be appropriately defined as a computer-readable medium. For example, if the instructions are transmitted from a website, server or other remote source via a coaxial cable, optical fiber cable, twisted pair and digital subscriber line (DSL), or wireless technologies such as infrared, radio and microwave, then the coaxial cable, optical fiber cable, twisted pair and DSL, or wireless technologies such as infrared, radio and microwave are also included in the definition of the above medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals or other transient media, but rather involve non-transient tangible storage media. Disks and optical disks used herein include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs) and blue-ray discs, wherein disks usually reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0972] Instructions can be executed by one or more processors such as one or more digital signal processors (DSP), one or more general-purpose microprocessors, one or more application specific integrated circuits (ASIC), one or more field programmable logic arrays (FPGA) or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" used herein may refer to any of the above structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the various functions described herein may be provided in dedicated hardware and / or software modules for encoding and decoding, or incorporated into a combined codec. In addition, these techniques may be fully implemented in one or more circuits or logic elements.

[0973] The techniques of the present invention may be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., chipsets). The present invention describes various components, modules, or units to emphasize the functional aspects of the devices for performing the disclosed techniques, but these components, modules, or units do not necessarily need to be implemented by different hardware units. In fact, as described above, the various units can be combined in a codec hardware unit in combination with appropriate software and / or firmware, or provided by a collection of interoperable hardware units including one or more processors as described above.

Claims

1. A method for processing an image using a neural network (NN), characterized in that: The NN includes a multi-stage context model (MCM), and the MCM includes multiple MCMs k The MCM further comprises a first lower recombinant layer and an upper recombinant layer, a first filling layer is provided before the first lower recombinant layer, and a cutting layer is provided after the upper recombinant layer; the method comprises: Get the first tensor; Obtain a second tensor, wherein the second tensor is an output of the super decoder; padding the first tensor using the first padding layer; According to the first lower reorganization layer, the padded first tensor is lower reorganized to obtain a reorganized first tensor; According to the multiple MCM k A model is provided, processing the reorganized first tensor and the second tensor to obtain a priori spatial tensor; According to the upper reorganization layer, the prior space tensor is up-reorganized to obtain a reorganized prior space tensor; The reorganized prior space tensor is cropped according to the cropping layer to obtain a reconstructed prior tensor.

2. The method according to claim 1, characterized in that The number of input tensor slices of the first lower reorganization layer is 2.

3. The method according to claim 1 or 2, characterized in that: The number of input tensor slices of the upper reorganization layer is 2.

4. The method according to any one of claims 1 to 3, characterized in that The first padding layer is represented as Padd(Hin, Win), where Hin and Win are the height and width of the first tensor.

5. The method according to claim 4, characterized in that The step length of the first filling layer is equal to 2, and the depth is equal to 5.

6. The method according to any one of claims 1 to 5, characterized in that The crop layer is denoted as Crop(Hin,Win), where Hin and Win are the height and width of the tensor output to the composite transform.

7. The method according to claim 6, characterized in that The step size of the clipping layer is equal to 2 and the depth is equal to 5.

8. The method according to any one of claims 1 to 7, characterized in that The first tensor is a reconstructed residual tensor, and the second tensor is an explicit prediction tensor.

9. A neural network (NN), characterized in that: The NN includes a multi-stage context model (MCM), and the MCM includes multiple MCMs k Model, the MCM further comprising a first lower recombinant layer and an upper recombinant layer, a first filling layer is provided before the first lower recombinant layer, and a cutting layer is provided after the upper recombinant layer; The first lower reorganization layer is used to change the size or shape of the tensor; the upper reorganization layer is used to change the size or shape of the tensor.

10. The NN according to claim 9, characterized in that The number of input tensor slices of the first lower reorganization layer is 2.

11. The NN according to claim 9 or 10, characterized in that The number of input tensor slices of the upper reorganization layer is 2.

12. NN according to any one of claims 9 to 11, characterized in that The first padding layer is represented as Padd(Hin, Win), where Hin and Win are the height and width of the first tensor.

13. NN according to any one of claims 9 to 12, characterized in that The crop layer is denoted as Crop(Hin,Win), where Hin and Win are the height and width of the tensor output to the composite transform.

14. NN according to any one of claims 9 to 13, characterized in that The first padding layer receives a tensor of a first size and outputs a tensor of a second size.

15. NN according to any one of claims 9 to 14, characterized in that The first filling layer is performed by replication.

16. NN according to any one of claims 9 to 15, characterized in that The MCM k Model uses multiple previous MCMs k The model's output tensor is used as input, where k ranges from 0 to k–1.

17. A method for processing an image using a neural network (NN), characterized in that: The NN includes a super large-scale decoder, the super large-scale decoder includes a basic operation point and an advanced operation point, under the basic operation point, The very large-scale decoder includes a first quantized transposed convolutional layer followed by a first cropping layer and a first rectified linear unit, a first quantized convolutional layer followed by a second rectified linear unit, a second quantized transposed convolutional layer followed by a second cropping layer and a third rectified linear unit, and a second quantized convolutional layer; At the advanced operating point, The ultra-large-scale decoder comprises two quantized convolution layers and three quantized convolution layers, each of the two quantized convolution layers is followed by a pixel reorganization layer, a cropping layer and a rectified linear unit in sequence, and two of the three quantized convolution layers are followed by rectified linear units respectively; The method comprises: Get the input tensor; Get the size of the input tensor; Get the operating point indicator; According to the operation point indicator, determine whether to use the basic operation point or the advanced operation point to process the input tensor; Output the processed tensor.

18. The method according to claim 17, characterized in that The kernel sizes of the first quantized transposed convolution layer and the second quantized transposed convolution layer are both 4×4, and the strides are both equal to 2.

19. The method according to claim 17 or 18, characterized in that The first cropping layer has a stride of 2 and a depth of 6.

20. The method according to any one of claims 17 to 19, characterized in that The second cropping layer has a stride of 2 and a depth of 5.

21. The method according to any one of claims 17 to 20, characterized in that The step size of the cropping layer after the first of the two quantized convolution layers at the advanced operating point is equal to 2, and the depth is equal to 6; the step size of the cropping layer after the second of the two quantized convolution layers at the advanced operating point is equal to 2, and the depth is equal to 5.

22. The method according to any one of claims 17 to 21, characterized in that The processed tensor is the very large-scale decoder standard deviation tensor.

23. A method for processing an image using a neural network (NN), characterized in that: The NN includes a synthetic transformation network, the synthetic transformation network includes a cascade layer for cascading a main tensor and an auxiliary tensor as an input tensor, a basic operation point and a high-level operation point, under the basic operation point, The synthetic transform network includes a lightweight residual block followed by a first transposed convolutional layer combined with a first cropping layer and a first residual activation unit, a second transposed convolutional layer combined with a second cropping layer and a second residual activation unit, a first convolutional layer combined with a third residual activation unit, a second convolutional layer followed by a first pixel reassembly layer and a third cropping layer; At the advanced operating point, The synthetic transform network includes two residual blocks followed by a third transposed convolutional layer combined with a fourth cropping layer and a first residual activation, a fourth transposed convolutional layer combined with a fifth cropping layer and a second residual activation, a third convolutional layer followed by a second pixel reassembly layer and a residual non-local attention block combined with a sixth cropping layer and a third residual activation, and a fifth transposed convolutional layer followed by a seventh cropping layer; The method comprises: Get the input tensor by concatenating the main tensor and the auxiliary tensor; Get the operating point indicator; Get the size of the input tensor; According to the operation point indicator, determine whether to use the basic operation point or the advanced operation point to process the input tensor; Output the processed tensor.

24. The method according to claim 23, characterized in that The stride of the first cropping layer is equal to 2, and the depth is equal to 4.

25. The method according to claim 23 or 24, characterized in that The stride of the second cropping layer is equal to 2 and the depth is equal to 3.

26. The method according to any one of claims 23 to 25, characterized in that The step size of the third clipping layer is equal to 4, and the depth is equal to 1.

27. The method according to any one of claims 23 to 26, characterized in that The fourth clipping layer has a step size of 2 and a depth of 4.

28. The method according to any one of claims 23 to 27, characterized in that The step size of the fifth cropping layer is equal to 2, and the depth is equal to 3.

29. The method according to any one of claims 23 to 28, characterized in that The step size of the sixth clipping layer is equal to 2, and the depth is equal to 2.

30. The method according to any one of claims 23 to 29, characterized in that The step size of the seventh clipping layer is equal to 2, and the depth is equal to 1.

31. A neural network (NN), characterized in that: The NN includes a very large-scale decoder, the very large-scale decoder includes a base operating point and an advanced operating point, at the base operating point, the very large-scale decoder includes a first quantized transposed convolutional layer followed by a first cropping layer and a first rectified linear unit, a first quantized convolutional layer followed by a second rectified linear unit, a second quantized transposed convolutional layer followed by a second cropping layer and a third rectified linear unit, and a second quantized convolutional layer; At the advanced operating point, the ultra-large-scale decoder includes two quantized convolutional layers and three quantized convolutional layers, each of the two quantized convolutional layers is followed by a pixel reconstruction layer, a cropping layer and a rectified linear unit in sequence, and two of the three quantized convolutional layers are respectively followed by a rectified linear unit.

32. A neural network (NN), characterized in that: The NN comprises a synthetic transform network, the synthetic transform network comprising a cascade layer for concatenating a main tensor and an auxiliary tensor as input tensors, a basic operation point and an advanced operation point, wherein at the basic operation point, the synthetic transform network comprises a light-weight residual block followed by a first transposed convolutional layer combined with a first cropping layer and a first residual activation unit, a second transposed convolutional layer combined with a second cropping layer and a second residual activation unit, a first convolutional layer combined with a third residual activation unit, a second convolutional layer followed by a first pixel reorganization layer and a third cropping layer; At the high-level operating point, the synthetic transform network includes two residual blocks followed by a third transposed convolutional layer combined with a fourth cropping layer and a first residual activation, a fourth transposed convolutional layer combined with a fifth cropping layer and a second residual activation, a third convolutional layer followed by a second pixel reassembly layer and a residual non-local attention block combined with a sixth cropping layer and a third residual activation, and a fifth transposed convolutional layer followed by a seventh cropping layer.

33. An encoder for encoding an image, characterized in that The encoder includes a receiver for receiving an image and one or more processors for implementing a neural network (NN), wherein the NN includes a multi-stage context model (MCM), wherein the MCM includes a plurality of MCMs k Model, the MCM also includes a first lower reorganization layer, a second lower reorganization layer and an upper reorganization layer, the first lower reorganization layer is preceded by a first padding layer, the second lower reorganization layer is preceded by a second padding layer, and the upper reorganization layer is followed by a cropping layer, the encoder also includes a transmitter for outputting a code stream, and the encoder is used to execute the method according to any one of claims 1 to 8.

34. An encoder for encoding an image, characterized in that The encoder includes a receiver for receiving an image and one or more processors for implementing a neural network (NN), the NN including a very large-scale decoder, the very large-scale decoder including a base operating point and an advanced operating point, at the base operating point, the very large-scale decoder includes a first quantized transposed convolutional layer followed by a first cropping layer and a first rectified linear unit, a first quantized convolutional layer followed by a second rectified linear unit, a second quantized transposed convolutional layer followed by a second cropping layer and a third rectified linear unit, and a second quantized convolutional layer; At the advanced operating point, the ultra-large-scale decoder includes two quantized convolutional layers and three quantized convolutional layers, each of the two quantized convolutional layers is followed by a pixel reconstruction layer, a cropping layer and a rectified linear unit in sequence, and two of the three quantized convolutional layers are followed by a rectified linear unit; the encoder also includes a transmitter for outputting a code stream, and the encoder is used to execute the method according to any one of claims 17 to 22.

35. An encoder for encoding an image, characterized in that The encoder comprises a receiver for receiving an image and one or more processors for implementing a neural network (NN), the NN comprising a synthesis transform network, the synthesis transform network comprising a cascade layer for concatenating a main tensor and an auxiliary tensor as an input tensor, a basic operation point and an advanced operation point, wherein at the basic operation point, the synthesis transform network comprises a light-weight residual block followed by a first transposed convolutional layer combined with a first cropping layer and a first residual activation unit, a second transposed convolutional layer combined with a second cropping layer and a second residual activation unit, a first convolutional layer combined with a third residual activation unit, and a second convolutional layer followed by a first pixel reorganization layer and a third cropping layer; At the high-level operating point, the synthetic transformer network includes two residual blocks followed by a third transposed convolutional layer combined with a fourth cropping layer and a first residual activation, a fourth transposed convolutional layer combined with a fifth cropping layer and a second residual activation, a third convolutional layer followed by a second pixel reassembly layer and a residual non-local attention block combined with a sixth cropping layer and a third residual activation, and a fifth transposed convolutional layer followed by a seventh cropping layer; The encoder further comprises a transmitter for outputting a code stream, and the encoder is configured to perform the method according to any one of claims 23 to 30.

36. An encoder for encoding an image, characterized in that The encoder comprises one or more processors for implementing a neural network (NN), the one or more processors being configured to perform a method according to any one of claims 1 to 8 or any one of claims 17 to 22 or any one of claims 23 to 30.

37. A decoder for decoding a code stream representing an image, characterized in that The decoder includes a receiver for receiving a code stream and one or more processors for implementing a neural network (NN), wherein the NN includes a multi-stage context model (MCM), and the MCM includes multiple MCMs. k Model, the MCM also includes a first lower reorganization layer and an upper reorganization layer, the first lower reorganization layer is preceded by a first padding layer, and the upper reorganization layer is followed by a cropping layer; the decoder also includes a transmitter for outputting a decoded image, and the decoder is used to execute the method according to any one of claims 1 to 8.

38. A decoder for decoding a code stream representing an image, characterized in that The decoder includes a receiver for receiving a code stream and one or more processors for implementing a neural network (NN), the NN includes a very large-scale decoder, the very large-scale decoder includes a basic operation point and an advanced operation point, under the basic operation point, the very large-scale decoder includes a first quantized transposed convolution layer followed by a first cropping layer and a first rectified linear unit, a first quantized convolution layer followed by a second rectified linear unit, a second quantized transposed convolution layer followed by a second cropping layer and a third rectified linear unit, and a second quantized convolution layer; At the advanced operating point, the ultra-large-scale decoder includes two quantized convolution layers and three quantized convolution layers, each of the two quantized convolution layers is followed by a pixel reconstruction layer, a cropping layer and a rectified linear unit in sequence, two of the three quantized convolution layers are respectively followed by a rectified linear unit, the decoder also includes a transmitter for outputting a decoded image, and the decoder is used to perform the method according to any one of claims 17 to 22.

39. A decoder for decoding a code stream representing an image, characterized in that The decoder includes a receiver for receiving a code stream and one or more processors for implementing a neural network (NN), the NN including a synthesis transform network, the synthesis transform network including a cascade layer for concatenating a main tensor and an auxiliary tensor as an input tensor, a basic operation point and an advanced operation point, wherein at the basic operation point, the synthesis transform network includes a lightweight residual block followed by a first transposed convolution layer combined with a first cropping layer and a first residual activation unit, a second transposed convolution layer combined with a second cropping layer and a second residual activation unit, a first convolution layer combined with a third residual activation unit, and a second convolution layer followed by a first pixel reorganization layer and a third cropping layer; At the high-level operating point, the synthetic transformation network includes two residual blocks followed by a third transposed convolutional layer combined with a fourth cropping layer and a first residual activation, a fourth transposed convolutional layer combined with a fifth cropping layer and a second residual activation, a third convolutional layer followed by a second pixel reorganization layer and a residual non-local attention block combined with a sixth cropping layer and a third residual activation, and a fifth transposed convolutional layer followed by a seventh cropping layer; the decoder also includes a transmitter for outputting a decoded image, and the decoder is used to perform the method according to any one of claims 23 to 30.

40. A decoder for decoding a code stream representing an image, characterized in that The decoder comprises one or more processors for implementing a neural network (NN), the one or more processors being configured to perform a method according to any one of claims 1 to 8 or any one of claims 17 to 22 or any one of claims 23 to 30.

41. A computer-readable storage medium comprising computer-executable instructions, characterized in that: When the computer executable instructions are executed on a computing system, the computing system is caused to perform a method according to any one of claims 1 to 8, any one of claims 17 to 22, or any one of claims 23 to 30.

42. A neural network (NN), characterized in that: The NN includes a multi-stage context model (MCM), and the MCM includes multiple MCMs k The MCM further comprises one or more lower recombination layers and one or more upper recombination layers, wherein the one or more lower recombination layers are respectively preceded by filling layers, and the one or more upper recombination layers are respectively followed by cutting layers.

43. The NN according to claim 42, characterized in that The NN also includes a very large-scale decoder, the very large-scale decoder includes a baseline and a high line, the baseline includes two or more quantized transposed convolutional layers, the two or more quantized transposed convolutional layers are respectively followed by a cropping layer and a rectified linear unit, the high line includes two or more quantized convolutional layers, at least one of the two or more quantized convolutional layers is sequentially followed by a pixel reorganization layer, a cropping layer and a rectified linear unit.

44. The NN according to claim 42 or 43, characterized in that The NN also includes a synthetic transformation network, which includes a baseline and a high line, the baseline includes two or more transposed convolutional layers and one or more convolutional layers, the two or more transposed convolutional layers are respectively followed by a cropping layer and a residual activation unit, and the one or more convolutional layers are respectively followed by a pixel reorganization layer and a cropping layer; the high line includes two or more transposed convolutional layers and a convolutional layer, the two or more transposed convolutional layers are respectively followed by a cropping layer and a residual activation, and the convolutional layer is followed by a pixel reorganization layer and a residual non-local attention block combined with the cropping layer.

45. An encoder for encoding an image, characterized in that The encoder comprises a receiver for receiving an image, a transmitter for outputting a code stream, and one or more processors for implementing a neural network (NN) according to any one of claims 42 to 44.

46. ​​A decoder for decoding a code stream representing an image, characterized in that The decoder comprises a receiver for receiving a code stream, a transmitter for outputting a decoded image, and one or more processors for implementing a neural network (NN) according to any one of claims 42 to 44.

47. A method for processing an image using a neural network (NN), characterized in that: The NN includes a multi-stage context model (MCM), and the MCM includes multiple MCMs k The MCM further comprises one or more lower recombinant layers and one or more upper recombinant layers, wherein the one or more lower recombinant layers are respectively preceded by a filling layer, and the one or more upper recombinant layers are respectively followed by a cutting layer; the method comprises: Get the input tensor; padding a first tensor using padding layers before the one or more lower reorganization layers, respectively, wherein the first tensor is the input tensor or a tensor obtained by processing the input tensor; After the one or more upper reorganization layers respectively output second tensors, the second tensors are cropped using a cropping layer.

48. The method according to claim 47, characterized in that The NN further includes a very large-scale decoder, the very large-scale decoder includes a baseline and a high line, the baseline includes two or more quantized transposed convolutional layers, the two or more quantized transposed convolutional layers are respectively followed by a cropping layer and a rectified linear unit, the high line includes two or more quantized convolutional layers, each of the two or more quantized convolutional layers is sequentially followed by a pixel reassembly layer, a cropping layer and a rectified linear unit; the method further includes: Get the operating point indicator; Based on the operation point indicator, it is determined whether to use the baseline or the advanced operation line to process a third tensor.

49. The method according to claim 47 or 48, characterized in that The NN further comprises a synthetic transform network, the synthetic transform network comprising a baseline and a high line, the baseline comprising two or more transposed convolutional layers and one or more convolutional layers, the two or more transposed convolutional layers are respectively followed by a cropping layer and a residual activation unit, the one or more convolutional layers are respectively followed by a pixel reorganization layer and a cropping layer; the high line comprises two or more transposed convolutional layers and a convolutional layer, the two or more transposed convolutional layers are respectively followed by a cropping layer and a residual activation, the convolutional layer is followed by a pixel reorganization layer and a residual non-local attention block combined with the cropping layer; The method further comprises: Get the second input tensor by concatenating the primary tensor and the secondary tensor; Get the operating point indicator; Based on the operating point indicator, it is determined whether to use the baseline or the high line to process the second input tensor.

50. The method according to any one of claims 47 to 49, characterized in that When the operating point indicator is equal to 0, the second input tensor is processed using the baseline.

51. The method according to any one of claims 47 to 50, characterized in that When the operating point indicator is equal to 1, the second input tensor is processed using the high line.

Images