Picture coding device and method using a convolutional neural network

Abstract

The present disclosure relates to the encoding and decoding of a picture or a plurality of pictures (e.g., video) using a neural network trained partially online. Accordingly, in an encoder, one or more layers to be trained are selected. Then, training of the neural network is performed to update the parameters of the selected layers. The parameters of the remaining layers are maintained and are not updated. The updated parameters are provided within a bitstream. The picture(s) are also encoded. The decoder receives instructions for the updated parameters, updates only the indicated parameters, and applies the neural network thus obtained.

Description

Technology Field

[0001] The present invention relates to the encoding and decoding of a picture, and in particular to the encoding and decoding of a picture using a neural network, such as an in-loop filter or a post-processing filter. Background Technology

[0002] Video coding (video encoding and decoding) is used in a wide range of digital video applications, such as broadcast digital TV, video transmission over the Internet and mobile networks, real-time conversation applications like video chat and video conferencing, DVD and Blu-ray discs, video content acquisition and editing systems, and camcorders for mobile device video recording and security applications.

[0003] Since the development of the block-based hybrid video coding approach in the 1990 H.261 standard, new video coding technologies and tools have been developed to form the basis of new video coding standards. One of the goals of most video coding standards was to achieve a reduction in bitrate compared to previous versions without sacrificing image quality. Additional video coding standards include MPEG-1 video, MPEG-2 video, VP8, VP9, ​​AV1, ITU-T H.262 / MPEG-2, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, AVC (Advanced Video Coding), ITU-T H.265, HEVC (High Efficiency Video Coding), ITU-T H.266, VVC (Versatile Video Coding), and extensions of these standards, such as scalability and / or 3D extensions.

[0004] Even for relatively short videos, the amount of video data required to depict them can be substantial, which can cause difficulties when streaming or communicating data over communication networks with limited bandwidth. Therefore, video data is typically compressed before being transmitted over modern communication networks. Since memory resources can be limited, the size of the video can also be an issue when it is stored on a storage device. Video compression devices often use software and / or hardware at the source to code video data before transmission or storage, thereby reducing the amount of data required to represent a digital video image. The compressed data is then received by a video decompression device at the destination that decodes the video data. As network resources remain limited and the demand for higher video quality continues to grow, improved compression and decompression technologies that enhance compression ratios with little to no sacrifice in picture quality are desirable.

[0005] Video encoding and decoding can be performed by standard video encoders and decoders compatible with, for example, H.264 / AVC, HEVC (H.265), VVC (H.266), or other video coding technologies.

[0006] Embodiments of the present disclosure provide an apparatus and method for encoding and decoding or general processing of a picture using a neural network.

[0007] Embodiments of the present invention are defined by the features of independent claims, and more advantageous implementations of the embodiments are defined by the features of dependent claims.

[0008] According to one embodiment, an apparatus for encoding one or more input pictures is provided, comprising using a neural network (NN), wherein the apparatus comprises processing circuitry, the processing circuitry being configured to select one or more updatable layers among the layers of the neural network; train the neural network by updating one or more parameters of the one or more (selected) updatable layers among the layers of the neural network and maintaining parameters of one or more pre-configured layers based on the one or more input pictures; encode information about the one or more updated parameters of the one or more updatable layers into a parameter bitstream part; and encode the one or more input pictures into a picture bitstream part. Training the neural network by retraining or updating only a subset of all layers of the neural network (e.g., one or more, but not all), that is, one or more updatable layers having updatable parameters, can accelerate the training of the NN and thus reduce the time required to train the NN. In addition, information about the updated parameters (after NN training) is encoded, reducing signaling overhead. Therefore, only a portion of the NN parameters is encoded, rather than the parameterization of the entire network.

[0009] According to one implementation, the processing circuit is configured to further encode information regarding the one or more updatable layers into a layer bitstream portion. For example, the information regarding the one or more updatable layers includes a layer index of the one or more updatable layers, wherein the layer index is the index of the one or more updatable layers among the layers of the neural network indexed according to a pre-set rule. Thus, since update information for the updatable layers may not be required, signaling overhead is reduced. Consequently, when signaled, signaling overhead is further reduced because the information relates to the updatable layers (one or more, but not all) rather than the entire layer of the neural network. Therefore, the bit amount of the layer bitstream portion is small.

[0010] In another example, an updateable layer has one or more updateable parameters and one or more pre-configured parameters; the processing circuit is configured to train the neural network by updating the one or more updateable parameters of the one or more updateable layers and maintaining the one or more pre-configured parameters. Thus, training the neural network by retraining or updating only a subset of all layers of the neural network (one or more, but not all), that is, only one or more updateable layers having updateable parameters, can accelerate NN training and thus reduce the time required to train the NN. Additionally, since not all updateable parameters of the updateable layers may be updated, the overhead of computing and signaling update information for the updated parameters is reduced. Consequently, the amount of parameters updated during NN training is further reduced.

[0011] In one exemplary implementation, the updateable layer includes an updateable sublayer in which the weights are one or more updateable parameters and a preconfigured sublayer in which the weights are one or more preconfigured parameters. The same input of the updateable layer is provided to the updateable sublayer and the preconfigured sublayer. The output of the updateable layer is the element-wise sum of the updateable sublayer and the preconfigured sublayer.

[0012] Training the weights of an updateable sublayer while using the sublayers of the updateable sublayer and maintaining pre-configured weights enables training the updateable layer based on weight changes by referencing the pre-configured weights. This is called delta-weight training. Therefore, training an NN by retraining and updating only a subset of all layers of the neural network (one or more, but not all), that is, one or more updateable layers with updateable parameters, can accelerate NN training and reduce the time required to train the NN because the pre-configured weights are used as a reference for the training. In other words, the convergence of NN training is accelerated.

[0013] According to one implementation, the updateable sublayer includes a sparse neural network layer. As a result, the number of parameters of the updateable sublayer is reduced. The updateable sublayer includes a tensor train neural network layer.

[0014] Furthermore, the processing circuit is configured to update the one or more updateable parameters using a training method that is less complex than the training method used to train the one or more pre-configured parameters. Thus, the complexity of NN training is reduced by retraining and updating only a subset (one or more, but not all) of all layers of the neural network, thereby saving training costs (e.g., complexity, latency, etc.).

[0015] In one embodiment, the one or more updateable layers are convolutional layers. Furthermore, at least one of the following applies: (i) the less complex training process updates fewer parameters than the training method used to train the one or more pre-configured parameters; and (ii) the less complex training updates a smaller convolutional kernel than the training method used to train the one or more pre-configured parameters. Thus, the time required for NN training is reduced by retraining and updating only a subset (one or more, but not all) of all layers of the neural network. Furthermore, the smaller convolutional kernel has fewer updated parameters as output, and thus reduces the signaling overhead of the updated parameters.

[0016] Additionally, the processing circuit is configured to update one or more updateable parameters of one or more updateable layers by initializing them pseudo-randomly; and to perform training with pairs of original and reconstructed pictures. Thus, training complexity is reduced by initializing the updateable layers in a fast and simple manner.

[0017] In some embodiments, the processing circuit is configured to train the neural network by updating one or more parameters of one or more updateable layers based on pairs of original pictures and reconstructed pictures. The reconstructed pictures are, for example, inputs to the NN, and the NN (each, one or more parameters of the one or more updateable layers) is trained to reduce the difference between the reconstructed picture and its corresponding original picture. For example, the one or more parameters of the one or more updateable layers include the weights of the one or more updateable layers. Thus, in this case, the architecture of the neural network has low complexity because the updateable layer does not include two sublayers arranged in parallel.

[0018] According to one implementation, the processing circuit further pre-configures the neural network by pre-training, including updating the parameters of the one or more pre-configured layers less frequently than the training of the one or more updateable layers based on the one or more input pictures; and is configured to encode information about the updated parameters of the one or more pre-configured layers into a network bitstream portion. Thus, the actual training of the neural network by retraining and updating only a subset (one or more, but not all) of all layers of the neural network can be improved by using the updated pre-configured parameters. Thus, information about the pre-configured parameters updated less frequently is distinguished from information about the parameters that are more frequently updateable.

[0019] Furthermore, the processing circuit is additionally configured to acquire a plurality of pre-configured layer sets; and to select a set from the plurality of pre-configured layer sets that includes one or more pre-configured layers of the neural network; and information regarding the updated parameters is an instruction of the selected set. The instruction of the selected set may be an index of each neural network layer. Alternatively, the instruction may be a parameter of the NN. Thus, NN training by retraining and updating only a subset (one or more, but not all) of all layers of the neural network improves the overall NN training, as the existing NN (i.e., the NN pre-configured by prior training) can be adapted to a specific compression task.

[0020] For example, the selection and / or updating of the above layers are performed at different intervals. Thus, interference between layer selection and layer updating can be avoided, which improves the performance and robustness of NN training. Additionally, NN training by retraining and updating only a subset (one or more, but not all) of the layers of the neural network can be adapted in priority over layer selection or layer updating.

[0021] In addition, information about one or more of the above-mentioned updateable layers is encoded less frequently than information about one or more updated parameters of the above-mentioned updateable layers. This means that the selection of updateable layers is performed less frequently than the retraining of the parameters of the updateable layers. Therefore, NN training by retraining and updating only a subset (one or more, but not all) of all layers of the neural network can be performed prioritized over layer updates (i.e., updated parameters), as opposed to layer selection which accelerates NN training.

[0022] In some exemplary implementations, the training is based on pairs of first number input pictures and reconstructed pictures. Encoding of information regarding one or more updated parameters is performed after encoding of the first number input pictures, and the trained neural network is applied to the second number reconstructed pictures. The second number is larger than the first number. Thus, NN training by retraining and updating only a subset (one or more, but not all) of all layers of the neural network can be performed on a smaller set of input picture pairs, whereas the decoder-side NN can be applied to a larger set of input pictures, thereby accelerating and making NN training less complex. Nevertheless, regular adaptation to picture content may be possible.

[0023] According to one example, the one or more pre-configured layers and / or the one or more updateable layers of the neural network are convolutional layers.

[0024] Additionally, the processing circuit is configured to select one or more updateable layers based on the optimization of a cost function including at least one of rate, distortion, and complexity. Thus, since one or more updateable layers can be selected in an optimized manner, the overall training of the neural network is enhanced by retraining and updating only a subset (one or more, but not all) of all layers of the neural network.

[0025] In one implementation, the neural network is a neural network trained for post-filtering.

[0026] In addition, the encoding of one or more of the above pictures is performed using the neural network.

[0027] In other implementations, the neural network is trained to perform one of in-loop filtering, end-to-end picture coding, adaptive sampling, and coding of picture coding parameters. Thus, the same neural network architecture can be used to train the NN for various types of video compression tasks. Therefore, training the NN by retraining and updating a subset (one or more, but not all) of the layers of the neural network is performed in a flexible manner and can be adapted to various tasks.

[0028] According to one embodiment, an apparatus for decoding one or more input pictures is included, comprising using a neural network. The apparatus comprises a processing circuit, the processing circuit is configured to decode one or more input pictures, comprising decoding information about one or more parameters of one or more (selected) updateable layers from a parameter bitstream portion; and processing from a picture bitstream portion to the neural network, wherein the neural network comprises the one or more updateable layers having the decoded one or more parameters; and one or more pre-configured layers having pre-configured parameters. Accordingly, updating the neural network by updating only a subset of all layers of the neural network (one or more, but not all), that is, by updating one or more updateable layers having updateable parameters, can reduce the signaling overhead of updating the NN and signing information to accelerate the update of the NN. Additionally, since not all updateable parameters of the updateable layers may be updated, the overhead for calculating and signaling update information for the updated parameters is reduced. In addition, one or more input pictures can be rapidly decoded using a neural network that uses fewer updated parameters than the pre-configured parameters. Thus, as updated NN parameters are used, the efficiency of picture decoding is improved while ensuring the high quality of the decoded picture.

[0029] In one implementation, the processing circuit is configured to decode information about one or more updatable layers from a layer bitstream portion. For example, the information about one or more updatable layers includes a layer index of one or more updatable layers; the layer index is an index of one or more updatable layers among the layers of the neural network indexed according to a preset rule. Thus, the decoder may not need to decode information about the updatable layers, thereby reducing the signaling overhead for processing update information. Consequently, when signaled, the overhead for decoding is further reduced because the information relates to the updatable layers rather than all layers of the neural network. Therefore, the bit amount of the layer bitstream portion being decoded is still small, and thus the update information for the updatable layers can be decoded quickly.

[0030] In an exemplary implementation, an updateable layer has one or more updateable parameters and one or more pre-configured parameters; the processing circuit is configured to update the one or more updateable parameters of the one or more updateable layers and maintain the one or more pre-configured parameters. Thus, since not all updateable parameters of the updateable layer may be updated, NN-based picture decoding can be accelerated as well as the overhead for decoding information about the updated parameters. Consequently, the amount of parameters representing the updated parameter configuration of the NN is further reduced.

[0031] In some exemplary implementations, the updateable layer includes an updateable sublayer in which the weight is one or more updateable parameters and a preconfigured sublayer in which the weight is one or more preconfigured parameters; the same input of the updateable layer is provided to the updateable sublayer and the preconfigured sublayer; and the output of the updateable layer is the element-wise sum of the updateable sublayer and the preconfigured sublayer. Thus, picture decoding speed is accelerated because only weight changes are updated using the preconfigured weights as a reference.

[0032] According to one example, the updateable sublayer includes or is a sparse neural network layer. As a result, the number of parameters of the updateable sublayer is reduced. Additionally, the updateable sublayer includes a tensor-trained neural network layer. The tensor-trained neural network is an example of a sparse NN.

[0033] In one exemplary implementation, the one or more updateable layers are convolutional layers; at least one of the following is applied: the updateable sublayer has fewer parameters than the pre-configured sublayer; and the updateable sublayer applies a convolutional kernel smaller than that of the pre-configured sublayer. Thus, the time required for picture decoding is reduced. Additionally, since the smaller the convolutional kernel, the fewer the updated parameters, the amount of data processed is reduced.

[0034] Furthermore, the one or more parameters of the one or more updatable layers include the weights of the one or more updatable layers. Consequently, the updatable parameters of the updatable layers are the weights that are directly updated. Therefore, since the convolutional layer does not include two sublayers arranged in parallel in this case, the architecture of the neural network has low complexity.

[0035] In some exemplary implementations, the processing circuit further decodes information about updated parameters of one or more pre-configured layers from a network bitstream portion; and is configured to update the pre-configured parameters less frequently than the updateable parameters based on the information about the updated parameters. Thus, picture decoding can be improved by using the updated pre-configured parameters for the decoder-side NN. Additionally, information about the updated pre-configured parameters is decoded in a network bitstream portion separated from other bitstream portions (video, parameters, layers). Thus, information about the pre-configured parameters that are updated less frequently (low update frequency) is distinguished from information about the updateable parameters that are updated more frequently (high update frequency).

[0036] For example, the processing circuit is additionally configured to obtain instructions for a selected set among a plurality of pre-configured layer sets from information about the updated parameters, and the selected set includes one or more pre-configured layers of the neural network. Thus, decoding can be adapted to a specific compression task using another existing NN (i.e., a network pre-configured by pre-training, for example), thereby improving overall picture decoding.

[0037] Furthermore, information regarding one or more updatable layers is decoded less frequently than information regarding one or more updated parameters of the one or more updatable layers. Therefore, decoding is performed prioritizing layer updates (i.e., updated parameters) over layer selection (i.e., updated layer indices), thereby accelerating picture decoding. Consequently, interference between layer updates (i.e., updated layer indices) and parameter updates can be avoided, improving the performance and robustness of decoding. Additionally, decoding can be adapted based on the preference for layer updates or parameter updates. Thus, decoding can be performed in a flexible manner and adapted to specific tasks.

[0038] In one exemplary implementation, the one or more pre-configured layers and / or the one or more updateable layers of the neural network are convolutional layers.

[0039] For example, the above neural network is applied to post-filtering.

[0040] According to one implementation, the decoding of one or more pictures is performed using the neural network.

[0041] In another example, the neural network is applied to in-loop filtering, picture decoding, adaptive sampling, and decoding of picture coding parameters. Thus, the same architecture of the neural network can be used to decode input pictures for different types of video compression tasks. Therefore, decoding can be performed in a flexible manner and adapted to various tasks, including post-filters and in-loop filters.

[0042] According to one embodiment, a method for encoding one or more input pictures is provided, comprising using a neural network, the method comprising: selecting one or more updatable layers among the layers of the neural network; training the neural network, including updating one or more parameters of the one or more updatable layers and maintaining parameters of one or more pre-configured layers among the layers of the neural network based on the one or more input pictures; encoding information about one or more updated parameters of the one or more updatable layers into a parameter bitstream portion; and encoding the one or more input pictures into a picture bitstream portion.

[0043] According to one embodiment, a method for decoding one or more input pictures is provided, comprising using a neural network, wherein the method comprises: decoding information about one or more parameters of one or more updatable layers from a parameter bitstream portion; and decoding one or more input pictures from a picture bitstream portion by processing with the neural network, wherein the neural network comprises: the one or more updatable layers having one or more decoded parameters; and one or more pre-configured layers having pre-configured parameters.

[0044] Additional methods include implementations and examples of encoding and decoding methods, which correspond to implementations and examples of encoding and decoding devices described above.

[0045] The above method provides similar advantages to a device that performs the corresponding steps described above.

[0046] According to one embodiment, a computer-readable non-transient medium is provided for storing a program, wherein the program comprises instructions that, when executed on one or more processors, cause the one or more processors to perform the method of encoding and / or decoding one or more input pictures, including using a neural network.

[0047] According to one embodiment, an apparatus for encoding one or more input pictures is provided, comprising using a neural network, wherein the apparatus comprises one or more processors; and a computer-readable non-transient storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the program configures an encoder to perform an encoding method according to any of the embodiments, implementations, and examples of the present specification when executed by the one or more processors.

[0048] According to one embodiment, an apparatus for decoding one or more input pictures is provided, comprising using a neural network, wherein the apparatus comprises: one or more processors; and a computer-readable non-transient storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the apparatus is configured to perform a decoding method according to any of the embodiments, implementations, and examples of the present specification when the program is executed by the one or more processors.

[0049] According to one embodiment, a computer program is provided that includes program code for performing the method when executed on a computer according to any one of the embodiments and examples of the present specification.

[0050] Updatable layers may also be referred to as adaptive or updatable layers or layers selected to be updated (e.g., by online training in an encoder, particularly by online training on a specific set of pictures or video sequences to be coded) in order to distinguish these layers or sets of these layers (of the NN) from other or all remaining layers or sets of other or all remaining layers (of the NN) that are not updated or will not be updated (i.e., fixed or static, e.g., retaining their weights). Correspondingly, updatable parameters may also be referred to as adaptive or updatable parameters or parameters selected to be updated (e.g., through the selection of layers or sublayers to be updated) in order to distinguish these parameters from other or remaining parameters (of the NN) that are not updated or will not be updated (i.e., fixed or static, e.g., parameters of layers or sublayers that will not be updated).

[0051] The present invention may be implemented in hardware (HW) and / or software (SW) or any combination thereof. Additionally, a HW-based implementation may be combined with a SW-based implementation.

[0052] Details regarding one or more embodiments are described in the attached drawings and the description below. Other features, purposes, and advantages will become apparent from the detailed description, drawings, and claims. Brief explanation of the drawing

[0053] Hereinafter, embodiments of the present invention will be described in more detail with reference to the attached drawings. Figure 1 is a schematic diagram illustrating exemplary convolutional neural network processing. Figure 2 is a schematic diagram showing exemplary layer processing of a convolutional neural network layer. Figure 3 is a schematic diagram showing a general and lightweight (tensor-trained) convolutional layer architecture. Figure 4 is an example of exemplary weighted-shared compression. FIG. 5 is another exemplary example of weighted-shared compression in which the sparse data tensor (4003) of FIG. 4 is approximated by a tensor. Figure 6 is an example of the weight distribution when quantization is not applied and when it is applied. Figure 7 is a schematic diagram showing weight pruning in a neural network having multiple layers. FIG. 8 is a block diagram showing an exemplary residual network having a residual block and other layers. FIG. 9 is a block diagram showing an exemplary encoder-decoder processing chain including CNN-based training on the encoder side and CNN inference on the decoder side. FIG. 10 is a block diagram showing an exemplary encoder-decoder processing chain including employing a CNN for the purpose of filtering a reconstructed picture. Figure 11 is a block diagram of an exemplary encoder processing and output. FIG. 12 is a block diagram of an exemplary decoder processing and output. FIG. 13 is an example of a neural network N having multiple layers, including an updateable layer to be updated and a pre-configured layer not to be updated. FIG. 14 is a block diagram showing an exemplary structure of a residual block of a residual neural network (ResNet) including two convolutional layers and a ReLU layer after each convolutional layer. FIG. 15 is a block diagram showing an exemplary structure of an updatable layer including a sublayer with parameters that are not updated during training and a sublayer with parameters that are updated during training. FIG. 16 is a block diagram showing an exemplary selection of a layer to be an updateable layer, including a sublayer having parameters that are not updated during training and a sublayer having parameters that are updated during training. FIG. 17 is a block diagram showing the original convolution layer of an original or existing neural network (i.e., a pre-configured NN) and the corresponding layer of an embodiment of a neural network that replaces the original layer, wherein the corresponding layer includes two sublayers comprising an original part that is not updated and a delta part that can be updated. The outputs of the sublayers are summed. Figure 18 is an example of dictionary-based compression employing Tensor Train convolutional layers (TTConv), which compresses the weights of updatable layers and represents them in a codebook and an indexbook. FIG. 19 is an exemplary encoder processing and output including pre-training of weights by offline training, online training for fine-tuning of weights for updateable layers, and compression of video and NN. FIG. 20 is an exemplary decoder processing and output including video and NN decompression. FIG. 21 is a block diagram showing an exemplary encoding device according to one embodiment. FIG. 22 is a block diagram showing an exemplary decoding device according to one embodiment. FIG. 23 is a flowchart illustrating an encoding method using a neural network according to one embodiment. FIG. 24 is a flowchart illustrating a decoding method using a neural network according to one embodiment. FIG. 25 is a block diagram illustrating an example of a video encoder configured to implement an embodiment of the present invention. FIG. 26 is a block diagram illustrating an exemplary configuration of a video decoder configured to implement an embodiment of the present invention. FIG. 27 is a block diagram showing an exemplary decoder that may employ some embodiments of the present disclosure. FIG. 28 is a block diagram illustrating an example of a video coding system configured to implement an embodiment of the present invention. FIG. 29 is a block diagram illustrating another example of a video coding system configured to implement an embodiment of the present invention. FIG. 30 is a block diagram showing an example of an encoding device or a decoding device. FIG. 31 is a block diagram showing another example of an encoding device or a decoding device. Specific details for implementing the invention

[0054] In the following description, reference is made to the accompanying drawings, which form part of the present disclosure and illustrate by way of example specific aspects of embodiments of the present invention or specific aspects in which embodiments of the present invention may be used. It is understood that embodiments of the present invention may be used in other aspects and may include structural or logical modifications not illustrated in the drawings. Accordingly, the following detailed description should not be interpreted in a limiting sense, and the scope of the present invention is defined by the appended claims.

[0055] For example, it is understood that the disclosure relating to the described method is valid for a corresponding device or system configured to perform the method, and vice versa. For example, where one or more specific method steps are described, the corresponding device may include one or more units, such as functional units (e.g., one unit performing one or more steps, or multiple units each performing one or more of the multiple steps), to perform the described one or more method steps, even if such one or more units are not explicitly described or illustrated in the drawings. Meanwhile, for example, where a specific device is described based on one or more units, such as functional units, the corresponding method may include one step for performing the function of one or more units (e.g., one step performing the function of one or more units, or multiple steps each performing one or more of the functions of multiple units), even if such one or more steps are not explicitly described or illustrated in the drawings. Furthermore, it is understood that the features of the various exemplary embodiments and / or aspects described herein may be combined with one another unless otherwise specifically stated.

[0056] Video coding generally refers to the processing of a sequence of pictures that form a video or video sequence. In the field of video coding, the terms "frame" or "image" may be used as synonyms instead of "picture." Video coding involves two parts: video encoding and video decoding. Video encoding is performed at the source side and typically involves processing the original video picture (e.g., by compression) to reduce the amount of data required to represent the video picture (for more efficient storage and / or transmission). Video decoding is performed at the destination side and typically involves reverse processing by comparing with an encoder to reconstruct the video picture. An embodiment referring to the "coding" of a video picture (or, generally, a picture as described later) should be understood to relate to both the "encoding" and "decoding" of the video picture. The combination of the encoding and decoding parts is also referred to as CODEC (COding and DECoding).

[0057] In the case of lossless video coding, the original video picture can be reconstructed. That is, the reconstructed video picture has the same quality as the original video picture (assuming there are no transmission errors or other data loss during storage or transmission). In the case of lossy video coding, additional compression is performed, for example through quantization, to reduce the amount of data representing the video picture, but this cannot be completely reconstructed by the decoder. In other words, the quality of the reconstructed video picture is lower or worse than the quality of the original video picture.

[0058] Several video coding standards since H.261 belong to the "lossy hybrid video codec" group (i.e., combining spatial and temporal prediction in the sample domain with 2-D transformation coding for the application of quantization in the transformation domain). Each picture in a video sequence is usually partitioned into a set of non-overlapping blocks, and coding is usually performed at the block level. In other words, in the encoder, the video is usually generated by creating prediction blocks using, for example, spatial (intra-picture) prediction or temporal (inter-picture) prediction at the block (video block) level, obtaining residual blocks by subtracting the prediction blocks from the current block (the block currently being processed / to be processed), and processing—i.e., encoding—by transforming and quantizing the residual blocks in the transformation domain to reduce the amount of data to be transmitted (compression), whereas in the decoder, the inverse processing compared to the encoder is applied to the encoded or compressed blocks to reconstruct the current blocks for representation. Additionally, the encoder will duplicate the decoder processing loop to generate identical predictions (e.g., intra-predictions and inter-predictions) and / or the processing of subsequent blocks, i.e., reconstruction for coding.

[0059] Video picture processing (also referred to as video processing) and still picture processing (the term processing includes coding) share many concepts and techniques. Hereinafter, the term "picture" is used to refer to the video picture and / or still picture of a video sequence (as described above) where unnecessary, in order to avoid unnecessary repetition between video picture and still picture and to distinguish between them. When a description refers only to a still picture (or still image), the term "still picture" is used.

[0060] Even for relatively short videos, the amount of video data required to depict them can be substantial, which can cause difficulties when streaming or communicating data over communication networks with limited bandwidth. Therefore, video data is typically compressed before being transmitted over modern communication networks. Since memory resources can be limited, the size of the video can also be an issue when it is stored on a storage device. Video compression devices often use software and / or hardware at the source to code video data before transmission or storage, thereby reducing the amount of data required to represent a digital video image. The compressed data is then received by a video decompression device at the destination that decodes the video data. As network resources remain limited and the demand for higher video quality continues to grow, improved compression and decompression technologies that enhance compression ratios with little to no sacrifice in picture quality are desirable.

[0061] Over the past decade, we have witnessed significant success of deep learning technology in many fields, particularly in computer vision (CV) and image processing. Recently, there have been many attempts to apply deep learning (DL) not only to data compression such as image and video compression, but also to other forms of data compression, such as audio data compression and 3D data compression (e.g., point-cloud data). For example, in the case of video compression, there have been many attempts to use neural networks to enhance or replace certain modules of conventional video codec architectures, including modules for the following:

[0062] ● Deep Learning-Based Intra-Prediction: In conventional intra-prediction, neighboring reconstructed samples of a coding block are used to obtain predictions of samples within the coding block along a specific linear direction (or some fixed pattern) dictated by the intra-prediction mode. By performing deep learning with reference samples, the generated prediction sample values ​​can be more flexible and more similar to the samples within the current coding block.

[0063] ● Deep Learning-Based Inter-Prediction: In conventional inter-prediction, a reference block within a reference picture is used to obtain a prediction of a sample within the current coding block using a simple weighting method. By performing deep learning on the reference block, a more flexible prediction can be obtained, which can be more similar to the sample within the current coding block.

[0064] ● Deep Learning-Based Entropy Coding: In conventional entropy coding, some neighboring information or a priori knowledge is used as context to estimate the probability of syntax values ​​for arithmetic coding. By using deep learning in conjunction with context, more accurate probabilities can be estimated.

[0065] ● Deep Learning-Based Adaptive Sampling: In the field of codecs, there is a classic method of reducing the spatial resolution of a video through downsampling and then feeding it to the encoder. In contrast, the decoder obtains a reconstructed image of the video and then performs upsampling. By using deep learning methods, the upsampled image can have higher quality. Similarly, adaptive sampling can also be used in the time domain.

[0066] There is also another significant attempt to use deep learning techniques to reduce or even eliminate coding artifacts in order to improve the overall quality (objective or subjective) of the decoded picture. Neural network-based enhancement filters (such as CNNs) are the most common deep learning techniques that can be applied to data compression, such as video compression.

[0067] A picture and / or video coding solution (or framework for each codec) that is based on conventional codecs such as JPEG (Joint Photographic Experts Group), H.264, H.265, H.266, or MPEG EVC (Motion Picture Experts Group - Enhanced Video Coding) and uses neural networks or other artificial intelligence (AI) solutions to replace or improve parts or tools (such as in-loop filters or post-processing filters) is a hybrid AI-based picture or video coding or an AI-enhanced picture or video coding solution (or framework for each codec). Embodiments of the present invention relate, for example, to such hybrid AI-based picture or video coding.

[0068] To better understand the following discussion, some terms regarding exemplary encoding and decoding approaches referenced throughout the description are introduced.

[0069] FIG. 25 illustrates a schematic block diagram of an exemplary video encoder (20) configured to implement the technology of the present application. In the example of FIG. 25, the video encoder (20) includes an input (201) (or input interface (201)), a residual calculation unit (204), a transform processing unit (206), a quantization unit (208), an inverse quantization unit (210), and an inverse transform processing unit (212), a reconstruction unit (214), a loop filter unit (220), a decoded picture buffer (DPB) (230), a mode selection unit (260), an entropy encoding unit (270), and an output (272) (or output interface (272)). The mode selection unit (260) may include an inter-prediction unit (244), an intra-prediction unit (254), and a partitioning unit (262). The inter prediction unit (244) includes a motion estimation unit and a motion compensation unit (not shown). The video encoder (20) shown in FIG. 25 may also be referred to as a hybrid video encoder or a video encoder according to a hybrid video codec.

[0070] The residual calculation unit (204), the transformation processing unit (206), the quantization unit (208), and the mode selection unit (260) may be referred to as forming the forward signal path of the encoder (20), while the inverse quantization unit (210), the inverse transformation processing unit (212), the reconstruction unit (214), the buffer (216), the loop filter (220), the decoded picture buffer (DPB) (230), the inter prediction unit (244), and the intra prediction unit (254) may be referred to as forming the reverse signal path of the video encoder, wherein the reverse signal path of the video encoder (20) corresponds to the signal path of the decoder (see video decoder (30) in FIG. 26). The inverse quantization unit (210), inverse transformation processing unit (212), reconstruction unit (214), loop filter (220), decoded picture buffer (DPB) (230), inter prediction unit (244) and intra prediction unit (254) are also referred to as forming the "built-in decoder" of the video encoder (20).

[0071] The encoder (20) may be configured to receive, for example, a picture (17) (or picture data (17)), such as a picture in a sequence of pictures forming a video or video sequence, via an input (201). The received picture or picture data may be a preprocessed picture (19) (or preprocessed picture data (19)). For brevity, the following description refers to the picture (17). The picture (17) may also be referred to as the current picture or the picture to be coded (particularly in video coding, to distinguish the current picture from other pictures, such as the same video sequence, i.e., previously encoded and / or decoded pictures in the video sequence that also includes the current picture).

[0072] A (digital) picture can be considered as a two-dimensional array or matrix of samples with intensity values. Samples in an array may also be referred to as pixels (short for picture element) or pels. The number of samples in the horizontal and vertical directions (or axes) of the array or picture defines the size and / or resolution of the picture. For color representation, three color components are typically used. That is, a picture can be represented or contain an array of three samples. In an RGB format or color space, a picture contains corresponding arrays of red, green, and blue samples. However, in video coding, each pixel is represented in a luminance and chrominance format and color space, such as YCbCr, which typically consists of a luminance component designated as Y (sometimes L is used instead) and two chrominance components designated as Cb and Cr. The luminance (or abbreviated as luma) component Y represents brightness or gray level intensity (e.g., a grayscale picture), while the two chrominance (or abbreviated as chroma) components Cb and Cr represent chromaticity or color information components. Thus, a picture in the YCbCr format contains a luminance sample array of luminance sample values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). A picture in the RGB format can be converted to or converted to the YCbCr format and vice versa; this process is also known as color conversion or conversion. If the picture is black and white, the picture may consist only of a luminance sample array. Thus, the picture may be, for example, a lumina sample array in a black and white format or two corresponding chroma sample arrays and a lumina sample array in color formats of 4:2:0, 4:2:2, and 4:4.

[0073] An embodiment of the video encoder (20) may include a picture partitioning unit (not shown in FIG. 25) configured to partition a picture (17) into a plurality of (usually non-overlapping) picture blocks (203). These blocks may also be referred to as root blocks, macro blocks (H.264 / AVC) or coding tree blocks (CTB) or coding tree units (CTU) (H.265 / HEVC and VVC). The picture partitioning unit may be configured to partition each picture into corresponding blocks by using a grid that defines the same block size for all pictures in the video sequence, or by changing the block size between pictures or a subset or group of pictures.

[0074] In an additional embodiment, the video encoder may be configured to directly receive blocks (203) of a picture (17), such as one, several, or all blocks forming the picture (17). The picture blocks (203) may also be referred to as the current picture blocks or the picture blocks to be coded.

[0075] Similar to the picture (17), the picture block (203) may again be considered as a two-dimensional array or matrix of samples having intensity values ​​(sample values), even if the dimensions are smaller than those of the picture (17). In other words, the block (203) may include, for example, one sample array (e.g., a luminance array in the case of a black and white picture (17), or a luminance or chroma array in the case of a color picture) or three sample arrays (e.g., a luminance array and two chroma arrays in the case of a color picture (17)) or other numbers and / or types of arrays depending on the applied color format. The number of samples in the horizontal and vertical directions (or axes) of the block (203) defines the size of the block (203). Thus, the block may be, for example, an MxN (M columns x N rows) sample array or an MxN array of transformation factors.

[0076] An embodiment of the video encoder (20) illustrated in FIG. 25 may be configured to encode a picture (17) in blocks, for example, encoding and prediction are performed for each block (203).

[0077] An embodiment of the video encoder (20) as illustrated in FIG. 25 may additionally be configured to partition and / or encode a picture using slices (also called video slices), wherein the picture may be partitioned or encoded into one or more slices (typically non-overlapping), and each slice may contain one or more blocks (e.g., CTU).

[0078] An embodiment of the video encoder (20) as illustrated in FIG. 25 may additionally be configured to partition and / or encode a picture using a tile group (also called a video tile group) and / or a tile (also called a video tile), wherein the picture may be partitioned or encoded using one or more tile groups (typically non-overlapping), each tile group may include, for example, one or more blocks (e.g., CTU) or one or more tiles, wherein each tile may be, for example, rectangular in shape and may include one or more blocks (e.g., CTU), for example, a complete block or a partial block.

[0079] The residual calculation unit (204) may be configured to calculate a residual block (205) (also referred to as residual (205)) based on the picture block (203) and the prediction block (265) (more details regarding the prediction block (265) will be provided later), for example, by subtracting the sample value of the prediction block (265) from the sample value of the picture block (203) to obtain the residual block (205) in the sample area on a sample-by-sample (pixel-by-pixel) basis.

[0080] The transformation processing unit (206) may be configured to obtain transformation coefficients (207) in the transformation domain by applying a transformation, such as a discrete cosine transform (DCT) or a discrete sine transform (DST), to the sample values ​​of the residual block (205). The transformation coefficients (207) may also be referred to as transformation residual coefficients and may represent the residual block (205) in the transformation domain.

[0081] The conversion processing unit (206) may be configured to apply integer approximations of DCT / DST, such as the conversion specified for H.265 / HEVC. Compared to the orthogonal DCT conversion, these integer approximations are typically scaled by a specific factor. To maintain the norm of the residual blocks processed by the forward and inverse conversions, an additional scaling factor is applied as part of the conversion process. The scaling factor is typically selected based on specific constraints, such as a scaling factor that is a power of 2 for the shift operation, the bit depth of the conversion factor, and trade-offs between accuracy and implementation cost. A specific scaling factor is specified, for example, for the inverse transformation, by, for example, an inverse transformation processing unit (212) (and by the corresponding inverse transformation, for example, an inverse transformation processing unit, in the video decoder (30)), and a scaling factor corresponding to the forward transformation can be specified accordingly by, for example, a transformation processing unit (206) in the encoder (20).

[0082] An embodiment of the video encoder (20) (each conversion processing unit (206)) is configured to output conversion parameters, such as the type of conversion or conversion, such as directly or through an entropy encoding unit (270), so that a video decoder (30), for example, can receive and use conversion parameters for decoding.

[0083] The quantization unit (208) may be configured to quantize the transformation coefficient (207) to obtain the quantized coefficient (209) by applying, for example, scalar quantization or vector quantization. The quantized coefficient (209) may also be referred to as the quantized transformation coefficient (209) or the quantized residual coefficient (209).

[0084] The quantization process can reduce the bit depth associated with some or all of the conversion factors (207). For example, an n-bit conversion factor can be rounded down to an m-bit conversion factor during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, in the case of scalar quantization, different scaling may be applied to achieve finer or coarser quantization. A smaller quantization step size corresponds to finer quantization, while a larger quantization step size corresponds to coarser quantization. The applicable quantization step size may be indicated by the quantization parameter (QP). The quantization parameter may be, for example, an index to a predefined set of applicable quantization step sizes. For example, a small quantization parameter may correspond to fine quantization (small quantization step size) and a large quantization parameter may correspond to coarser quantization (large quantization step size), or vice versa. Quantization may include division by a quantization step size, and the corresponding and / or inverse quantization by, for example, an inverse quantization unit (210) may include multiplication by a quantization step size. An embodiment according to some standards, such as HEVC, may be configured to determine the quantization step size using quantization parameters. Generally, the quantization step size may be calculated based on quantization parameters using a fixed-point approximation of an equation involving division. An additional scaling factor may be introduced in the quantization and inverse quantization to restore the norm of the residual block, which may be modified due to the scaling used in the fixed-point approximation of the equation for the quantization step size and the quantization parameters. In one exemplary embodiment, the scaling of the inverse transform and inverse quantization may be combined.Alternatively, a customized quantization table can be used and signaled from the encoder to the decoder, or as a bitstream. Quantization is a loss operation, where the loss increases as the quantization step size increases.

[0085] An embodiment of the video encoder (20) (each quantization unit (208)) may be configured to output quantization parameters (QP), for example, directly or through an entropy encoding unit (270), so that a video decoder (30), for example, can receive and apply the quantization parameters for decoding.

[0086] The inverse quantization unit (210) is configured to obtain inverse quantized coefficients (211) by applying the inverse quantization of the quantization unit (208) to the quantized coefficients, for example, by applying the inverse of the quantization method applied by the quantization unit (208) based on or using the same quantization step size as the quantization unit (8). The inverse quantization coefficients (211) may also be referred to as inverse quantization residual coefficients (211) and correspond to the conversion coefficients (207), although they are not identical to the conversion coefficients due to losses caused by quantization.

[0087] The inverse transform processing unit (212) is configured to obtain a reconstructed residual block (213) (or corresponding inverse quantization coefficients (213)) in the sample domain by applying an inverse transform of the transform applied by the transform processing unit (206), such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST) or other inverse transform. The reconstructed residual block (213) may also be referred to as a transform block (213).

[0088] The reconstruction unit (214) (e.g., an adder or summer (214)) is configured to obtain a reconstruction block (215) in the sample area by adding the sample value of the reconstruction residual block (213) and the sample value of the prediction block (265) in a sample unit, for example, by adding the transformation block (213) (i.e., the reconstruction residual block (213)) to the prediction block (265).

[0089] A loop filter unit (220) (or abbreviated as "loop filter" (220)) is configured to filter a reconstructed block (215) to obtain a filtered block (221), or generally to filter a reconstructed sample to obtain a filtered sample. The loop filter unit is configured, for example, to smooth pixel transitions or otherwise improve video quality. The loop filter unit (220) may include one or more loop filters, such as a de-blocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as a bidirectional filter, an adaptive loop filter (ALF), a sharpening filter, a smoothing filter, or a collaborative filter, or any combination thereof. These may be implemented by a neural network, for example, as will be described in more detail later. In FIG. 25, the loop filter unit (220) is shown as an in-loop filter, but in other configurations, the loop filter unit (220) may be implemented as a post-loop filter. The filtered block (221) may also be referred to as a reconstructed block (221).

[0090] An embodiment of the video encoder (20) (each loop filter unit (220)) may be configured to output loop filter parameters (e.g., adaptive sample offset information) encoded, for example, directly or through an entropy encoding unit (270), so that, for example, a decoder (30) may receive and apply the same loop filter parameters or individual loop filters for decoding.

[0091] The decoded picture buffer (DPB) (230) may be a memory that stores a reference picture or generally reference picture data for encoding video data by the video encoder (20). The DPB (230) may be formed of any various memory element, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory elements. The decoded picture buffer (DPB) (230) may be configured to store one or more filtered blocks (221). The decoded picture buffer (230) may additionally be configured to store another previously filtered block of the same current picture or another picture, e.g., a previously reconstructed picture, e.g., a previously reconstructed and filtered block (221), and may provide a complete previously reconstructed, e.g., the decoded picture (and corresponding reference block and sample) and / or a partially reconstructed current picture (and corresponding reference block and sample), e.g., for inter-prediction. The decoded picture buffer (DPB) (230) may also be configured to store one or more unfiltered reconstructed blocks (215), or, e.g., a generally unfiltered reconstructed sample where the reconstructed block (215) is not filtered by the loop filter unit (220), or any other additionally processed version of the reconstructed block or sample.

[0092] The mode selection unit (260) includes a partitioning unit (262), an inter prediction unit (244), and an intra prediction unit (254), and is configured to receive or acquire original picture data, e.g., original block (203) (current block (203) of the current picture (17)), and, e.g., reconstructed sample or block of the same (current) picture that is filtered and / or unfiltered and / or reconstructed picture data from one or more previously decoded pictures, e.g., a decoded picture buffer (230) or another buffer (e.g., a line buffer, not shown). The reconstructed picture data is used as reference picture data for prediction, e.g., inter prediction or intra prediction, to acquire a prediction block (265) or a predictor (265).

[0093] The mode selection unit (260) may be configured to determine or select partitioning for the current block prediction mode (including no partitioning) and prediction mode (e.g., intra or inter prediction mode), and to generate a corresponding prediction block (265) used for calculating the residual block (205) and reconstructing the reconstructed block (215).

[0094] An embodiment of the mode selection unit (260) may be configured to select a partitioning and prediction mode (e.g., from those supported or available by the mode selection unit (260)), which provides the best match, that is, the minimum residual (the minimum residual means better compression for transmission or storage), or the minimum signaling overhead (the minimum signaling overhead means better compression for transmission or storage), or considers or balances both. The mode selection unit (260) may be configured to determine the partitioning and prediction mode based on rate distortion optimization (RDO), that is, to select a prediction mode that provides minimum rate distortion. In this context, terms such as "best," "minimum," and "optimum" do not necessarily refer to the overall "best," "minimum," or "optimal," but may also refer to the fulfillment of termination or selection criteria, such as values ​​that exceed or fall below a threshold, or potentially lead to a "sub-optimum choice," while also being other constraints that reduce complexity and processing time.

[0095] In other words, the partitioning unit (262) may be configured to partition the block (203) into smaller block partitions or sub-blocks (forming blocks again) using, for example, quad-tree partitioning (QT), binary partitioning (BT), or triple-tree partitioning (TT), or any combination thereof, and to perform predictions for each block partition or sub-block, for example, wherein the mode selection includes the selection of the tree structure of the partitioned block (203) and the prediction mode is applied to each block partition or sub-block.

[0096] Below, partitioning (e.g., by the partitioning unit (260)) and prediction processing (e.g., by the inter prediction unit (244) and the intra prediction unit (254)) performed by the exemplary video encoder (20) will be described in more detail.

[0097] As previously mentioned, the term "block" as used herein may be a part of a picture, particularly a square or rectangular part. For example, referring to HEVC and VVC, a block may be a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), and a transform unit (TU) and / or corresponding blocks, such as a coding tree block (CTB), a coding block (CB), a transform block (TB), or a prediction block (PB), or may include these.

[0098] For example, a coding tree unit (CTU) may be or include a CTB of luminance samples, two corresponding CTBs of chroma samples of a picture having a three-sample array, or a CTB of samples of a picture or black and white picture coded using three distinct color planes and syntax structures used to code the samples. Correspondingly, a coding tree block (CTB) may be an NxN block of samples for some values ​​of N such that dividing the components into CTBs is partitioning. A coding unit (CU) may be or include a coding block of luminance samples, two corresponding coding blocks of chroma samples of a picture having a three-sample array, or a coding block of samples of a picture or black and white picture coded using three distinct color planes and syntax structures used to code the samples. Correspondingly, a coding block (CB) may be an MxN block of samples for some values ​​of M and N such that dividing the CTBs into coding blocks is partitioning.

[0099] In an embodiment, for example according to HEVC, a Coding Tree Unit (CTU) can be divided into CUs using a quaternary tree structure represented by a coding tree. The determination of whether to code a picture region using inter-picture (temporal) or intra-picture (spatial) prediction is made at the CU level. Each CU can be further divided into one, two, or four PUs depending on the PU division type. Within a single PU, the same prediction process is applied, and relevant information is transmitted to a decoder based on the PU. After obtaining residual blocks by applying a prediction process based on the PU division type, the CU can be divided into Transform Units (TUs) according to other quaternary structures similar to the coding tree for the CU.

[0100] In an embodiment, for example according to VVC, the coding tree unit (CTU) can be divided into coding blocks using a quad-tree, a ternary tree and / or a binary coding tree. In one example, the mode selection unit (260) of the video encoder (20) can be configured to perform any combination of the partitioning techniques described herein.

[0101] As described above, the video encoder (20) is configured to determine or select the best or optimal prediction mode from a set of prediction modes (e.g., predetermined). The set of prediction modes may include, for example, an intra prediction mode and / or an inter prediction mode.

[0102] The set of intra-prediction modes may include 35 different intra-prediction modes, such as non-directional modes like DC (or average) mode and planar mode, or directional modes like those defined in HEVC, for example, or 67 different intra-prediction modes, such as non-directional modes like DC (or average) mode and planar mode, or directional modes like those defined in VVC, for example.

[0103] The intra prediction unit (254) is configured to generate an intra prediction block (265) according to the intra prediction mode of the set of intra prediction modes using a reconstructed sample of a neighbor block of the same current picture.

[0104] The intra prediction unit (254) (or generally the mode selection unit (260)) is additionally configured to output an intra prediction parameter (or generally information indicating the intra prediction mode selected for the block) in the form of a syntax element (266) for inclusion in the picture data (21) encoded in the entropy encoding unit (270), so that, for example, a video decoder (30) can receive and use the prediction parameter for decoding.

[0105] The set of inter-prediction modes (or possible inter-prediction modes) depends on the available reference picture (i.e., a previously at least partially decoded picture stored in DBP (230)) and other inter-prediction parameters, such as whether the entire reference picture or only a part of the reference picture, for example, a search window area around the area of ​​the current block, is used to search for the best-matched reference block, and / or pixel interpolation, for example, half / semi-pel and / or quarter-pel interpolation, is applied.

[0106] In addition to the prediction mode above, a skip mode and / or a direct mode may be applied. The inter-prediction unit (244) may include a motion estimation (ME) unit and a motion compensation (MC) unit (both not shown in FIG. 25). The motion estimation unit may be configured to receive or acquire a picture block (203) (the current picture block (203) of the current picture (17)) and a decoding picture (231), or at least one or more previously reconstructed blocks, such as one or more other / different previously coded pictures (231), for motion estimation. For example, a video sequence may include a current picture and a previously decoded picture (231). In other words, the current picture and the previously decoded picture (231) may be part of or form a sequence of pictures that form the video sequence.

[0107] The encoder (20) may be configured to select a reference block from a plurality of reference blocks of the same or different pictures among a plurality of different pictures, for example, and to provide an offset (spatial offset) between the position (x, y coordinates) of the reference picture (or reference picture index) and / or the position of the reference block and the position of the current block as an inter-prediction parameter to the motion estimation unit. This offset is also called a motion vector (MV).

[0108] The motion compensation unit is configured to acquire, for example, receive inter-prediction parameters, and to acquire inter-prediction blocks (265) by performing inter-prediction based on or using the inter-prediction parameters. Motion compensation performed by the motion compensation unit may include fetching or generating prediction blocks based on motion / block vectors determined by motion estimation, and may possibly perform interpolation with subpixel precision. Since interpolation filtering can generate additional pixel samples from known pixel samples, it can potentially increase the number of candidate prediction blocks that can be used to code picture blocks. Upon receiving a motion vector for the PU of the current picture block, the motion compensation unit may find the prediction block pointed to by the motion vector in one of the reference picture lists.

[0109] The motion compensation unit may also generate a block and a syntax element associated with the video slice for use by the video decoder (30) when decoding the picture block of the video slice. In addition to or alternative to the slice and each syntax element, a tile group and / or a tile and each syntax element may be generated or used.

[0110] The entropy encoding unit (270) is configured to obtain encoded picture data (21) that can be output through the output (272), for example, in the form of an encoded bitstream (21), by applying, for example, an entropy encoding algorithm or method (e.g., variable length coding (VLC) method, context adaptive VLC (CAVLC) method, arithmetic coding method, binarization, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding or other entropy encoding methodologies or techniques) or bypass (no compression) for quantized coefficients (209), inter-prediction parameters, intra-prediction parameters, loop filter parameters and / or other syntax elements, for example, video The decoder (30) receives and uses parameters for decoding. The encoded bitstream (21) can be transmitted to the video decoder (30) or stored in memory for later transmission or retrieval by the video decoder (30).

[0111] Other structural variations of the video encoder (20) may be used to encode a video stream. For example, a non-conversion-based encoder (20) may quantize residual signals directly without a conversion processing unit (206) for a specific block or frame. In another embodiment, the encoder (20) may have a quantization unit (208) and an inverse quantization unit (210) combined into a single unit.

[0112] FIG. 26 illustrates an example of a video decoder (30) configured to implement the technology of the present application. The video decoder (30) is configured to receive encoded picture data (21) (e.g., an encoded bitstream (21)) encoded, for example, by an encoder (20), and to obtain a decoded picture (331). The encoded picture data or bitstream includes information for decoding the encoded picture data, such as data representing a picture block of an encoded video slice (and / or tile group or tile) and associated syntax elements.

[0113] In the example of FIG. 26, the decoder (30) includes an entropy decoding unit (304), an inverse quantization unit (310), an inverse transformation processing unit (312), a reconstruction unit (314) (e.g., a summer (314)), a loop filter (320), a decoded picture buffer (DBP) (330), a mode application unit (360), an inter prediction unit (344), and an intra prediction unit (354). The inter prediction unit (344) may be a motion compensation unit or may include it. In some examples, the video decoder (30) may perform a decoding pass that is generally the inverse of the encoding pass described in relation to the video encoder (100) of FIG. 25.

[0114] As described in relation to the encoder (20), the inverse quantization unit (210), inverse transformation processing unit (212), reconstruction unit (214), loop filter (220), decoded picture buffer (DPB) (230), inter prediction unit (344) and intra prediction unit (354) are also referred to as forming the “built-in decoder” of the video encoder (20). Accordingly, the inverse quantization unit (310) may be identical to the inverse quantization unit (110) in terms of function, the inverse transformation processing unit (312) may be identical to the inverse transformation processing unit (212) in terms of function, the reconstruction unit (314) may be identical to the reconstruction unit (214) in terms of function, the loop filter (320) may be identical to the loop filter (220) in terms of function, and the decoded picture buffer (330) may be identical to the decoded picture buffer (230) in terms of function. Accordingly, the description provided for each unit and function of the video encoder (20) is applied correspondingly to each unit and function of the video decoder (30).

[0115] The entropy decoding unit (304) is configured to parse a bitstream (21) (or generally encoded picture data (21)) and, for example, perform entropy decoding on the encoded picture data (21) to obtain, for example, quantized coefficients (309) and / or decoded coding parameters (not shown in FIG. 26), for example, inter-prediction parameters (e.g., reference picture index and motion vector), intra-prediction parameters (e.g., intra-prediction mode or index), transformation parameters, quantization parameters, loop filter parameters and / or any or all of other syntax elements. The entropy decoding unit (304) may be configured to apply a decoding algorithm or method corresponding to the encoding method described in relation to the entropy encoding unit (270) of the encoder (20). The entropy decoding unit (304) may additionally be configured to provide inter-prediction parameters, intra-prediction parameters, and / or other syntax elements to the mode application unit (360) and other parameters to other units of the decoder (30). The video decoder (30) may receive syntax elements at the video slice level and / or video block level. In addition to or alternative to the slice and each syntax element, a tile group and / or a tile and each syntax element may be received and / or used.

[0116] The inverse quantization unit (310) is configured to receive quantization parameters (QP) (or information generally related to inverse quantization) and quantized coefficients from the encoded picture data (21) (e.g., by parsing and / or decoding by the entropy decoding unit (304)), and to apply inverse quantization to the decoded quantization coefficients (309) based on the quantization parameters to obtain inverse quantization coefficients (311), which may also be referred to as transform coefficients (311). The inverse quantization process may include the use of quantization parameters determined by a video encoder for each video block within a video slice (or tile or group of tiles) to determine the degree of quantization and the degree of inverse quantization to be applied likewise.

[0117] The inverse transformation processing unit (312) may be configured to receive inverse quantized coefficients (311), also referred to as transformation coefficients (311), and to apply a transformation to the inverse quantized coefficients (311) to obtain a reconstructed residual block (213) in the sample domain. The reconstructed residual block (213) may also be referred to as a transformation block (313). The transformation may also be an inverse transformation, e.g., inverse DCT, inverse DST, inverse integer transformation, or a conceptually similar inverse transformation process. Additionally, the inverse transformation processing unit (312) may be configured to receive transformation parameters or corresponding information from the encoded picture data (21) (e.g., by parsing and / or decoding by the entropy decoding unit (304)) to determine the transformation applied to the quantized coefficients (311).

[0118] A reconstruction unit (314) (e.g., an adder or summer (314)) may be configured to obtain a reconstruction block (315) in a sample domain by adding the reconstruction residual block (313) to the prediction block (365), for example, by adding the sample value of the reconstruction residual block (313) and the sample value of the prediction block (365).

[0119] A loop filter unit (320) (inside or after a coding loop) is configured to filter a reconstructed block (315) to obtain a filtered block (321), for example, to smooth pixel transitions or otherwise improve video quality. The loop filter unit (320) may include one or more loop filters, such as a deblocking filter, a sample adaptive offset (SAO) filter, or one or more other filters, such as a bidirectional filter, an adaptive loop filter (ALF), a sharpening, smoothing filter, or a collaborative filter, or any combination thereof. Although the loop filter unit (320) is depicted as an in-loop filter in FIG. 26, in other configurations, the loop filter unit (320) may be implemented as a post-loop filter.

[0120] The decoded video block (321) of the picture is then stored in the decoded picture buffer (330), and the decoded picture buffer (330) stores the decoded picture (331) as a reference picture for subsequent motion compensation for other pictures and / or output to each display. The decoder (30) is configured to output the decoded picture (311), for example through the output (312), to present or show to the user.

[0121] The inter prediction unit (344) may be identical to the inter prediction unit (244) (particularly for the motion compensation unit) and the intra prediction unit (354) may be identical to the inter prediction unit (254) in terms of function, and performs partitioning or partitioning decisions and predictions based on partitioning and / or prediction parameters or respective information received from the encoded picture data (21) (e.g., by parsing and / or decoding by the entropy decoding unit (304)). The mode application unit (360) may be configured to obtain a prediction block (365) by performing prediction (intra or inter prediction) block by block based on the reconstructed picture, block, or each sample (filtered or unfiltered).

[0122] When the video slice is coded as an intra-coded (I) slice, the intra-prediction unit (354) of the mode application unit (360) is configured to generate a prediction block (365) for a picture block of the current video slice based on the signaled intra-prediction mode and data from the previously decoded block of the current picture. When the video picture is coded as an inter-coded (e.g., B, or P) slice, the inter-prediction unit (344) (e.g., motion compensation unit) of the mode selection unit (360) is configured to generate a prediction block (365) for a video block of the current video slice based on motion vectors and other syntax elements received from the entropy decoding unit (304). For inter-prediction, the prediction block may be generated from one of the reference pictures within one of the reference picture lists. The video decoder (30) may construct a list of reference frames, such as List 0 and List 1, using a default construction technique based on the reference pictures stored in the DPB (330). For an embodiment that uses a tile group (e.g., video tile group) and / or a tile (e.g., video tile) in addition to or alternatively to a slice (e.g., video slice), the same or similar may be applied by the embodiment. For example, a video may be coded using I, P, or B tile groups and / or tiles. The mode application unit (360) is configured to determine prediction information for a video block of the current video slice by parsing motion vectors or related information and other syntax elements, and to generate prediction blocks for the current video block to be decoded using the prediction information.For example, the mode application unit (360) is configured to use some of the received syntax elements to determine the prediction mode used to code the video blocks of the video slice (e.g., intra or inter prediction), the inter prediction slice type (e.g., B slice, P slice, or GPB slice), construction information for one or more of the reference picture lists for the slice, motion vectors for each inter-encoded video block of the slice, the inter prediction state for each inter-encoded video block of the slice, and other information for decoding the video blocks within the current video slice. For an embodiment that additionally or alternatively uses a tile group (e.g., video tile group) and / or a tile (e.g., video tile) to the slice (e.g., video slice), or the same or similar may be applied by the embodiment. For example, the video may be coded using an I, P, or B tile group and / or tile. An embodiment of the video decoder (30) as illustrated in FIG. 26 may be configured to partition and / or decode a picture using slices (also called video slices), wherein the picture may be partitioned or decoded using one or more slices (typically non-overlapping), and each slice may contain one or more blocks (e.g., CTUs). An embodiment of the video decoder (30) as illustrated in FIG. 26 may be configured to partition and / or decode a picture using tile groups (also called video tile groups) and / or tiles (also called video tiles), wherein the picture may be partitioned or decoded using one or more tile groups (typically non-overlapping), and each tile group may contain, for example, one or more blocks (e.g., CTUs) or one or more tiles, wherein each tile may be, for example, rectangular in shape and may contain one or more blocks (e.g., CTUs), complete blocks or partial blocks.Other variations of the video decoder (30) may be used to decode the encoded picture data (21). For example, the decoder (30) may generate an output video stream without a loop filtering unit (320). For example, a non-transform-based decoder (30) may directly de-quantize residual signals without an inverse transformation processing unit (312) for a specific block or frame. In other implementations, the video decoder (30) may have a de-quantization unit (310) and an inverse transformation processing unit (312) combined into a single unit. It should be understood that in the encoder (20) and decoder (30), the processing result of the current stage may be further processed and output to the next stage. For example, after interpolation filtering, motion vector derivation, or loop filtering, additional operations such as clipping or shifting may be performed on the processing result of interpolation filtering, motion vector derivation, or loop filtering. Some additional terms used in this specification are described below.

[0123] Coding Artifacts / Compression Artifacts: In the case of video images or images, when viewing an image or video that has been decoded, decompressed, or reconstructed after coding or compression, some artifacts may be present, such as block artifacts, blur artifacts, ring artifacts, contouring artifacts, or other artifacts. These artifacts affect the objective quality and / or subjective quality of the image / video. In other words, the term artifact generally refers to parts of the picture that do not belong thereto, such as picture distortions visible to the human eye, or additional edges caused by block-based coding.

[0124] Image Enhancement: The improvement of the objective and / or subjective quality of an image (or an image in a video sequence), which has lower quality as a result, for example, from compression (e.g., using deblocking, deblurring, or other methods). It should be noted that in the context of picture coding, the term subjective quality generally refers to a quality measure that correlates well with human perception scores. There are many known metrics in the technical field that measure (estimate) subjective quality based on some objective video parameters.

[0125] Loop Filter or In-Loop Filter: This filter is used to improve the quality of the decoded picture, such as a deblocking filter, a sample-adaptive offset filter, or an adaptive loop filter. The filtered picture can be used as a reference picture for predicting another (e.g., the next) picture, for example, based on inter-prediction.

[0126] Post-filter: Like deblocking filters, sample adaptive offset filters, or adaptive loop filters, this filter is used to improve the quality of the decoded picture. The filtered picture will not be used as a reference image for predicting other pictures. The difference between a loop filter (or in-loop filter) and a post-filter is whether the filtered picture is used for prediction to encode / decode other pictures.

[0127] Picture Size: Represents the width w or height h of the picture, or width-height pairs. The width and height of an image are typically measured in the number of luma samples.

[0128] Downsampling: Downsampling is a process in which the sampling rate (sampling interval) of a discrete input signal is reduced. For example, if the input signal is an image with magnitudes h and w and the output of downsampling has magnitudes h2 and w2, then at least one of the following is true (holds).

[0129] ● h2 <h

[0130] ● w2 <w.

[0131] In one exemplary implementation, downsampling may be implemented by retaining only the m-th sample in each row and discarding the remaining input signal (e.g., image).

[0132] Upsampling: Upsampling is a process in which the sampling rate (sampling interval) of an individual input signal is increased. For example, when an input image has a size of h and w, and the size of a downsampling output is h2 and w2, at least one of the following is true.

[0133] ● h<h2

[0134] ● w<w2.

[0135] Resampling: Both downsampling and upsampling processes are examples of resampling. Resampling is a process in which the sampling rate (sampling interval) of an input signal is changed.

[0136] Interpolation filtering: During an upsampling or downsampling process, filtering may be applied to increase the accuracy of a resampled signal and reduce aliasing effects. Interpolation filtering generally includes a weighted combination of sample values at sample positions around the resampling position. This can be implemented as follows.

[0137]

[0138] With f() referring to the resampled signal, is the resampling coordinate (the coordinate of the resampled sample), C(k) is an interpolation filter coefficient, and s(x,y) is the input signal. Coordinates x and y are coordinates of samples of the input image. The summation operation is performed on (x,y) in the vicinity. In other words, as a weighted sum of input picture samples a new sample is obtained as This is obtained. Weighting is performed by the coefficient C(k), where k represents the position (index) of the filter coefficient within the filter mask. For example, for a 1D filter, k takes values ​​from 1 to the filter order. For a 2D filter that can be applied to a 2D image, k can be an index representing one of all possible (non-zero) filter coefficients. The index is associated with a specific position of the coefficient within the filter mask (filter kernel) according to the rule.

[0139] Super-resolution: Includes spatial super-resolution and temporal super-resolution. Spatial super-resolution refers to increasing the size of an image (or an image in a video sequence) by upsampling, for example, from wxh to 2wx2h. Temporal super-resolution refers to increasing the frame rate of a video sequence, for example, from 10 frames / second (10fps) to 20 frames / second (20fps), where fps is the unit of frames per second.

[0140] RDO (Rate-Distortion Optimization): RDO refers to the principle of selecting from several possible parameters under certain constraints. For example, in video compression, there are many coding modes or parameters corresponding to different distortions and amounts of coding bits. When high quality and a small number of coding bits are simultaneously preferred, the cost of making a decision C can be formulated as C = Distortion + λ * Rate. Here, Distortion is the distortion of the reconstructed signal (the distance or difference between the reconstructed signal and the original signal), Rate is the number of bits after coding the original signal, and λ is a coefficient called the Lagrange multiplier (also known as the Lagrangian).

[0141] Neural networks (in computing systems, that is, referring to artificial neural networks) are vaguely inspired by the biological neural networks that constitute the animal brains. Any neural network can be formulated and fully defined as a directed acyclic graph with a set of nodes Z. Each node represents a tensor (multidimensional array), and its parent node Operation (neural network layer) on the set It is associated with. The only exception is input node x, which has no input node or associated operation. The computation at node k can be formally expressed as follows.

[0142] ,

[0143] Operations here The set includes unary operations (convolution, pooling, activation, batch norm, etc.) and multivariate operations (connection, addition, etc.). An arbitrary expression specifying the parent set and the operation of each node completely defines the neural network.

[0144] Convolution: For input signals f[] and filters g[], convolution can be defined in one dimension as follows.

[0145]

[0146] Here, m is the index within the filter relative to the input signal, and n represents the position (shift) of the filter relative to the input signal. Both n and m are integers. S-convolution in 2D can operate similarly, as is well known in the art. For generalization, m can be considered to have a value between negative infinity and positive infinity, as in the equation above. However, in practice, the filter f[] can have a finite length, in which case the filter coefficients f[m] will be equal to zero for m exceeding the filter size. Convolution is (or in some implementations Input data of ) size It is an operation widely used in modern neural networks that performs convolution with convolution filters of size. Note that the term size also refers to the form or dimensions of input and / or output data (e.g., input of 6 samples of width and height and 3 color or 3 chroma and luminance components as channels, as illustrated in Fig. 2). Convolution is It generates output data of size N, where N is the batch size, and class is the height of the input (in) and output (out) data, and class is the width of the input data and output data, and is the number of input channels, and n is the number of output channels. K is a convolution filter parameter. It defines the area of ​​input data required to obtain one sample of output data. Here, the input data, output data, and kernel are 4-dimensional arrays of a fixed size along each dimension. The values ​​of the convolution filter are called the weights of the convolution layer. The weights of the convolution layer can be considered as tensors in 4-dimensional space.

[0147] FIG. 2 illustrates a specific example of a convolution layer. In this example, the size (also called shape or dimension) of the input data (2001) is 6×6×3 (e.g., 6 samples of width and height, and 3 color or 3 chroma and luminance components as channels) and the size of the convolution filter (2002) is 3×3×3×2 (each kernel has a size K of 3 weights of width and height, 2 kernels for each of the 3 channels (see Filter 1 and Filter 2), and a number of kernels or filters defining the number of output channels). The convolution filter (2002) can be represented by 2 filters (Filter 1, Filter 2) of size 3×3×3. Both are convolved with the input data (2001) using the sliding window principle (here, for example, without padding) to obtain output data (2003) of 4×4 (each output channel contains 4 samples of width and height, for example, by adding the results obtained by filtering 3 input channels element by element). The final output data (2004) has a size of 4×4×2 (2 output channels containing 4 samples of width and height each).

[0148] Convolutional Neural Network (CNN): A CNN is a subclass of a Deep Neural Network (DNN). The name "Convolutional Neural Network" (CNN) indicates that the network uses the mathematical operation known as convolution, as defined above. The term "deep" generally implies the presence of many cascaded layers. Since there is no clear cut-off for what constitutes a deep network architecture, any network with at least one hidden layer can be considered a deep neural network. Convolution is a special type of the linear operation mentioned above. A convolutional network is a simple neural network that uses convolution instead of standard matrix multiplication in at least one of its layers. A convolutional neural network consists of input and output layers, and one or more hidden layers. The input layer is the layer where inputs are provided for processing.

[0149] The hidden layers of a CNN typically consist of a series of convolutional layers that convolve with multiplication or other dot products. The output of the layers is one or more feature maps, also known as channels. Subsampling may be included in some or all layers. Consequently, the feature maps can be smaller. In a CNN, the activation function can be a Rectified Linear Unit (RELU) layer or a Generalized Divisive Normalization (GDN) layer, as previously exemplified above, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers; these are called hidden layers because their inputs and outputs are obscured by the activation function and the final convolution. The GDN is a special type of activation used for end-to-end compression, and the following transformations can be performed:

[0150] y[i] = x[i] / sqrt(beta[i] + sum_j(gamma[j, i] * x[j])),

[0151] Here, i and j are executed through the channel, and beta and gamma are trainable parameters.

[0152] Although layers are colloquially referred to as convolutions, this is a convention. Mathematically and technically, they are sliding dot products or cross-correlations. This is important for indices in a matrix in that it influences how weights are determined at specific index points.

[0153] When programming a CNN to process pictures or images, the input is a tensor of size (number of images) x (image width) x (image height) x (image depth). Please note that the term size refers to the shape or dimensions of the input and / or output data (e.g., inputs of 6 samples in width and height and 3 colors or 3 chroma and luminance components). Then, after passing through convolution layers, the image is abstracted into a feature map of size (number of images) x (feature map width) x (feature map height) x (feature map channels). Convolution layers within a neural network must have the following properties: a convolution kernel defined by width and height (hyperparameters); and the number of input and output channels (hyperparameters). The depth of the convolution filter (input channels) must be equal to the number of channels (depth) of the input feature map.

[0154] In the past, conventional multilayer perceptron (MLP) models were used for image recognition. However, due to the high dimensionality resulting from full connectivity between nodes, they do not scale well to high-resolution images. A 1000×1000 pixel image with RGB color channels has 3 million weights, which is too high to process efficiently on a large scale with full connectivity. Furthermore, this network architecture does not consider the spatial structure of the data and treats distant input pixels the same as nearby pixels. This ignores the reference location of the image data both computationally and semantically. Therefore, full connectivity of neurons is wasteful for purposes such as image recognition, which is dominated by spatially local input patterns.

[0155] Convolutional neural networks are biologically inspired variations of multilayer perceptrons specifically designed to emulate the behavior of the visual cortex. CNN models mitigate the problems posed by MLP architectures by leveraging the strong spatial local correlations present in natural images. Convolutional layers are the core building blocks of CNNs. The parameters of a layer consist of a set of trainable filters (the kernels mentioned above), with a receptive field that is small but extends through the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume to calculate the inner product between the filter and the input entry, generating a 2D activation map of the corresponding filter. Consequently, the network learns filters that are activated when certain specific types of features are extracted from certain spatial locations of the input.

[0156] Stacking activation maps for all filters along the depth dimension forms the entire output volume of the convolutional layer. Therefore, every entry in the output volume can be interpreted as the output of a neuron that shares parameters with the neuron in the same activation map, based on a small region of the input. A feature map, or activation map, is the output activation for a given filter. Feature map and activation are synonymous. In some papers, it is called an activation map because it is a mapping corresponding to the activations of different parts of the image, and also a feature map because it is a mapping of the locations where specific types of features are found in the image. High activation indicates that a specific feature has been found.

[0157] Another important concept in CNNs is pooling, which is a form of non-linear downsampling. There are several non-linear functions for implementing pooling, among which max pooling is the most common. It partitions the input image into a set of non-overlapping rectangles and outputs the maximum value for each of these sub-regions. Intuitively, the precise location of a feature is less important than its approximate location relative to other features. This is the underlying idea behind using pooling in convolutional neural networks. Pooling layers serve to control overfitting by progressively reducing the spatial size of the representation, thereby decreasing the number of parameters, memory footprint, and computational load in the network. In CNN architectures, it is common to periodically insert pooling layers between successive convolutional layers. The pooling operation provides another form of translation invariance.

[0158] Pooling layers operate independently on slices of all depths of the input and are spatially sized. The most common form is a pooling layer with a 2x2 filter, which discards 75% of activations by applying two downsamples along the width and height of each depth slice of the input. In this case, all maximum operations exceed four numbers. The depth dimension is maintained without change.

[0159] In addition to max pooling, pooling units can utilize other functions such as mean pooling or L2-norm pooling. While mean pooling was historically frequently used, its popularity has recently declined compared to max pooling, which actually delivers better performance. Due to the aggressive reduction of representation size, there is a recent trend toward using smaller filters or discarding pooling layers entirely. Region of Interest (RoI) pooling (also known as ROI pooling) is a variation of max pooling where the output size is fixed and the input rectangle is a parameter. Pooling is an important component of convolutional neural networks for object detection based on the Fast R-CNN architecture.

[0160] The aforementioned ReLU stands for Rectified Linear Unit and applies a non-saturating activation function. It effectively eliminates negative values ​​in the activation map by setting them to zero. It increases the non-linear properties of the decision function and the entire network without affecting the receptive field of the convolution layer. For example, other functions such as the saturating hyperbolic tangent and the sigmoid function are also used to increase non-linearity. ReLU is often preferred over other functions because it trains neural networks several times faster without a significant penalty to generalization accuracy.

[0161] After several convolution and max pooling layers, the high-level reasoning of the neural network is performed through fully connected layers. As seen in general (non-convolutional) artificial neural networks, neurons in fully connected layers are connected to all activations of the previous layer. Therefore, their activations can be computed as affine transformations using matrix multiplication and subsequent bias offsets (adding vectors of learned or fixed bias terms).

[0162] The "loss layer" specifies how training applies a training penalty to the deviation between the predicted (output) labels and the actual labels, and is typically the final layer of a neural network. Various loss functions suitable for different tasks can be used. Softmax loss is used to predict a single class among K mutually exclusive classes. Sigmoid cross-entropy loss is used to predict K independent probability values ​​in [0, 1]. Euclidean loss is used to regress to real-valued labels.

[0163] Therefore, a CNN can be configured to perform one of the following operations:

[0164] ■ Convolution

[0165] ■ Batch Normalization

[0166] ■ Non-linear activation (sigmoid, ReLU, etc.)

[0167] ■ Fully Connected Layer (FCL)

[0168] ■ Element-wise operations (addition, multiplication)

[0169] ■ Downsampling (Max Pooling, Average Pooling)

[0170] Today, CNNs are the most widely used approach for computer vision (CV) tasks such as classification, FaceID, person re-identification, car brand recognition, object detection, semantic and instance segmentation, image / video enhancement, and image / video super-resolution.

[0171] FIG. 1 illustrates an example of a general CNN architecture used for a classification task. The input node (1001) is for an RGB image. Or in the case of grayscale images It has size (also called shape or dimension). 1002 - Associated with a convolutional layer combined with an activation layer. 1004 - Associated with a fully connected layer. 1003 represents a node of the neural network, and 1005 represents an output node of the CNN. When RGB / grayscale is used as input, the CNN sequentially computes all intermediate nodes up to the output node. The output node represents a probability vector for an input image belonging to one of the classes (here, cat, dog, bicycle, car).

[0172] There are many other types of popular NN architectures. For example, there are recurrent neural networks (RNNs) used for Natural Language Processing (NLP) tasks and transformers used for NLP and CV tasks.

[0173] Training: This involves adapting a network to better handle tasks by considering sample observations. Training includes adjusting the network's weights and other parameters to improve the accuracy of the results (i.e., the output of the NN). This is done by minimizing observed errors. After the training of a neural network is complete, the neural network with applied weights is called a trained neural network. Therefore, a neural network is a trainable neural network.

[0174] Loss function: A function that maps the values ​​of one or more variables to real numbers representing the "cost" associated with the observed variables. For example, if we consider the error values ​​of a multidimensional array (e.g., video), the loss function could be the Mean Squared Error (MSE), which corresponds to the average of the squared errors.

[0175] Backpropagation: A method of adjusting weights to compensate for each error discovered during training. Technically, backpropagation calculates the gradient (derivative) of the cost function associated with a given state in relation to the weights. Weight updates can be performed through stochastic gradient descent (SGD) or other methods.

[0176] Neural Network Training Initialization: This is a method of assigning weights to neural network layers at the beginning of the training process.

[0177] A typical NN training process involves splitting the training dataset into batches of samples. Then, the backpropagation method is executed sequentially for all batches. One pass of backpropagation processing for all training dataset samples is called an epoch of training.

[0178] Neural network inference: The execution of a trained neural network on test data.

[0179] Neural network size: This is the amount of memory required to store the neural network (i.e., the parameters defining the neural network). Generally, the network size is the sum of the amount of memory required to store the neural network graph (architecture) and the amount of memory required to store the layer weights. The convolution filter size The amount of memory required to store the weights of an in-convolution layer is It can be calculated as.

[0180] Lightweighted convolutional layer: A layer in a neural network that has input and output sizes (also called shapes or dimensions) similar to a regular convolutional layer, but requires a much smaller amount of memory to store the layer's weights.

[0181] Tensor Train convolutional layer (TTConv): A lightweight convolutional layer of a neural network. Figure 3 illustrates an example of such a lightweight convolutional layer. Size For the filter (3001), each TTConv layer consists of three convolution layers (conv1, conv2, conv3). The size of Conv1 (3002) is is. The size of Conv2(3003) is is. The size of Conv3(3004) is and, here represents the rank of the TTConv layer. Generally, the lower the rank, the fewer parameters the TTConv layer has. There is a trade-off between the TTConv rank and performance. The TTConv layer is applied according to the following procedure.

[0182] - Conv1 is the size The input data (3005) is convolved. The output (3006) of conv1 is It has the shape of.

[0183] - The output of conv1 (3006) is size It is divided into arrays.

[0184] - The partitioned array is convolved by conv2.

[0185] - The output of conv2 is connected (3007).

[0186] - The connected output of conv2 is convolved with conv3. The output of conv3 is the output of the TTConv layer (3008).

[0187] The lightweight version of convolution has the same input and output sizes as regular convolution but has many fewer parameters. For example, a regular convolution with a kernel of 3 has 64 input and output channels and 3*3*64*64=36864 parameters. Conversely, a tensor-trained convolution layer with R_1=R_2=4 has 64*4*4*1*1+4*3*3+4*64*1*1=1316 parameters.

[0188] Neural Network Compression: This is the process of reducing the size of a neural network. For example, there are many NN compression methods, including the following:

[0189] ● Weight-sharing compression is one of the simplest forms of network reduction and involves weight sharing between layers, as discussed in the paper "CNN inference acceleration using dictionary of centroids" by D. Babin et al. and illustrated in Figures 4 and 5. In this method, different subtensors of tensor weights are similar, and instead of storing all subtensors, only one is stored, and in all other cases, only references are stored.

[0190] ● Quantization: Network quantization compresses the original network by reducing the number of bits required to represent each weight. The weight value distribution without quantization and the weight value distribution with quantization can be seen in FIG. 6. For example, if the weights are originally double precision (e.g., 64 bits), the number of bits (bit depth) can be reduced to single precision (32 floating-point bits). Please note that this is merely a simple example of quantization by reducing bit depth. However, for the purposes of this disclosure, any type of quantization may be applied.

[0191] ● Pruning: A compression method that involves removing some unimportant weights from a trained model. In machine learning, pruning is the removal of unnecessary layers or the weights of layers. A schematic diagram of the pruning operation is illustrated in FIG. 7, showing a trained network (7010) that has not undergone pruning and a trained network (7020) that has undergone pruning. In the network (7020), both the weights and the layers are pruned by removing the weights (connections) between nodes of adjacent layers and removing the nodes within the layers.

[0192] ● Inference Time Neural Network Compression: A neural network compression method that typically starts with a network trained in a standard way without considering compression.

[0193] ● Training Recognition Neural Network Compression: As a neural network training method, some compression methods provide additional regularization loss to better compress the weights of neural network layers.

[0194] ● A residual neural network (ResNet) is a class of convolutional neural networks. Residual neural networks skip some layers using skip connections or shortcuts. A typical ResNet model is implemented with double or triple layer skips that include non-linearity (e.g., ReLU) and batch normalization between layers. This is exemplified in Figures 8 and 14.

[0195] A typical ResNet architecture consists of a certain number of residual blocks. ResNet is illustrated in FIG. 8, which shows a first convolutional layer (810) followed by a ReLU, a last convolutional layer (850) followed by a ReLU (860), and multiple residual blocks (830 to 840) between them. There is also a skip connection (801).

[0196] The residual block structure is exemplified in FIG. 14. As shown in FIG. 14, it includes two convolution layers (1410, 1420), a ReLU layer (1430, 1460) after the convolution layers, and a skip connection (1140) from the input of the block to the output of the conv2 and ReLU layer (1450) after the skip connection.

[0197] ● Clustering is the task of grouping sets of objects such that objects in the same group (called a cluster) are more similar to each other (in the sense of a predefined similarity measure) than objects in other groups (clusters).

[0198] Using the terms defined above, the general processing of an input image by a CNN is explained with reference to FIG. 8. FIG. 8 illustrates an example in which a CNN (or a general NN) is used as a filter for picture enhancement. The reconstructed image (luma frame) is fed into a CNN network and provided as an output after processing, such as a higher quality image (enhanced lumina frame).

[0199] A so-called reinforcement filter can be trained on a large dataset and applied to different video sequences. This case is referred to as offline training. FIG. 10 illustrates a picture (1010) (such as a still picture or video picture) that is encoded into a bitstream by an encoder (1020) and provided to a channel (1030). This encoder may be any known picture or video encoder, which may be based on a neural network but does not necessarily have to be. A decoder (1040) acquires the encoded bitstream and decodes it. After decoding, a filter (1050) is applied to the decoded picture(s). The decoded and filtered picture (1060) can then be displayed.

[0200] The filter may be a CNN-based filter. It may be a loop filter (LF), such as CNN-LF. However, the filter may also be a post-filter. More generally, the filter may be an enhancement filter, which is a term referring to the purpose of the filter to enhance a reconstructed image to have better quality (e.g., measured by objective or subjective measures). For offline training, a compressed video stream is transmitted through the channel (1030). The CNN filter may be pre-configured (pre-trained) before the video stream is filtered, for example, before the decoder applies the CNN filter (1050) to the decoded video. That is, the CNN-LF is not changed according to the input video (1010). It is pre-trained based on a picture or video sequence rather than the input video sequence (1010).

[0201] Another approach is to train an enhancement filter on a single video sequence, a single frame, or multiple frames, and then apply it to this video, frame, or multiple frames. In this case, in addition to the compressed video, the trained network parameters must also generally be sent to the decoder. The decoder then uses the received network (i.e., network parameters) to obtain the final reconstructed and filtered picture. This case is called online training. Figure 9 illustrates a typical method of online training.

[0202] The original frame (910) is encoded by a conventional video encoder (920). The encoder (920) outputs a compressed video (930) and a reconstructed frame (960). A CNN filter (970) is trained to enhance the reconstructed frame (960) (enhancement) to be closer to the original frame with a predefined loss function. The CNN filter parameters are transmitted through a channel (990). A conventional decoder (940) receives the compressed video and decodes it into a decoded frame (950). The transmitted CNN filter (970) takes the decoded frame (950) as input and outputs an enhanced frame (980). Generally, during online training, the encoder-side NN (70) is trained based on the input (original) being encoded (as the target or ground-truth of the reconstructed picture to be enhanced using the NN (970)). The trained parameters of the NN (970) are transmitted to the decoder and applied to the decoder to reconstruct (particularly enhance) the encoded image by applying the NN (decoder-side inference NN (970)) composed of the transmitted parameters. In this disclosure, the term “transmission” means including as a bitstream. Transmission may be transmission through storage (temporary or permanent) or a communication interface, etc.

[0203] Regardless of whether the training is performed offline or online, the reinforcement filter can be used as a loop filter (Option 1) or a post-filter (Option 2) as illustrated in FIG. 27. As described with reference to FIG. 25 and FIG. 26 above, the loop filter and the post-filter may be in the same location.

[0204] Offline Training

[0205] In the case of offline training, the NN model is typically fixed on both the encoder and decoder sides. Therefore, there is no need to transmit from the encoder to the decoder.

[0206] A. In-loop filter or loop filter (Option 1):

[0207] Encoder side (e.g., Fig. 25)

[0208] ● Compresses video / pictures.

[0209] ● A CNN filter (using a pre-trained, large training dataset) is used as a loop filter for the reconstructed picture (e.g., at the location of the loop filter (220) in FIG. 25, for example, to replace or add to the existing loop filter), and the filtered picture will be used as a reference picture for another picture.

[0210] Decoder side (e.g., FIG. 26, FIG. 27)

[0211] ● Decompress the video / picture (e.g., by the preceding module including the reconstruction unit (314) in FIG. 26 and FIG. 27).

[0212] ● A CNN filter (e.g., pre-trained using the same large training dataset as the CNN filter of the encoder) is used as a loop filter (e.g., at the position of the loop filter (320) of FIG. 26, for example, as shown in FIG. 27, as a replacement for a conventional loop filter or in addition to a conventional loop filter, for example, as shown in FIG. 27: CnN Loop (2701 n)) for another loop filter (320). The filtered picture can be used as a reference picture for predicting another picture, i.e., stored in a decoding picture buffer (330).

[0213] B. Post-processing filter or post-filter (Option 2):

[0214] Encoder side

[0215] ● Compress video / picture (this can be done in the same way as Option 1).

[0216] Since post-filters do not affect the reference picture, the encoder does not need to perform the post-filtering process. The application of post-filters is generally performed to enhance the quality of the picture to be provided, for example, to be displayed. If the encoder does not provide a picture (typically not displaying it), there is no need to apply post-filters. On the other hand, if the presentation of the picture is required, post-filters can also be applied at the encoder side.

[0217] The encoder may include a loop filter (220) (conventional or CNN or a combination of both) as described above for Option 1.

[0218] Decoder side (Fig. 27)

[0219] ● Decompress the video / picture (can be done in the same way as Option 1).

[0220] ● A CNN filter (2702) (pre-trained using a large training dataset) is used as a post-filter for the decompressed / decoded picture. The filtered picture is not used as a reference picture for other pictures; that is, it is not stored in the decoded picture buffer.

[0221] The decoder may include a loop filter (320 / 2701) (conventional or CNN or a combination of both) as described above for Option 1.

[0222] Online Training

[0223] In the case of online training, the parameters of the NN model are sent to the decoder along with the compressed video. A typical scenario for online training is as follows:

[0224] Encoder side

[0225] ● Run the video encoder without the CNN filter.

[0226] ● Input - Reconstructed picture, Output - Train a CNN filter on an input picture with a mean square error (MSE) loss function or other loss function. In other words, the input to the NN must be the reconstructed picture (or pictures), and the output of the NN must be the original picture (or pictures) being encoded. The loss function can be any distortion function (measure or metric). Since CNN filters can be pre-trained, training a CNN can also be referred to as retraining or updating the CNN.

[0227] ● Compress (trained or retrained) CNN filter parameters using, for example, one or more neural network compression methods (e.g., quantization and / or entropy coding).

[0228] ● Transmits the (compressed) CNN filter parameters and the encoded video stream.

[0229] Decoder side

[0230] ● Obtain the encoded video stream and compressed CNN filter parameters.

[0231] ● Perform CNN decompression to obtain CNN filter parameters.

[0232] ● Run the video decoder to obtain the decoded picture(s).

[0233] ● Apply a CNN filter (consisting of decompressed CNN filter parameters) to the decoded picture(s).

[0234] Just like offline training, online training CNN filters can also be used as loop filters (Option 1) or post-filters (Option 2).

[0235] The above description illustrates a CNN-based reinforcement filter used as an example for online or offline training. This mechanism can also be used in other neural network-based compression tools, such as deep learning-based intra-prediction, deep learning-based inter-prediction, or deep learning-based adaptive sampling.

[0236] As mentioned earlier, in the case of offline training, there is no need to transmit the network model to the decoder, so the bit cost is lower than in the case of online training. However, since the network cannot adapt to the content of the data to be compressed, the compression efficiency of offline training is generally still insufficient.

[0237] Consequently, online training faces a problem where even significant quality improvements cannot compensate for the greatly increased bit cost, as neural network parameters, for example in video compression, have a significant bit cost (rate) compared to the video's bit cost. For instance, for a typical CNN filter with 40 layers, the transmission cost could be 3 MB, while the average bit rate for a single frame of a 4K video sequence could be less than 0.5 MB. Therefore, building efficient networks and compression to achieve optimal adaptation to online content is a challenging task.

[0238] Given the problems with the above approach, there is a need to provide an efficient method for representing and compressing a network model while keeping the cost of transmitting the network model low. As described in detail below, some embodiments and examples of the present disclosure address the above problem of online training for encoding and decoding input pictures using a neural network, which can reduce the cost of transmitting the network model by transmitting a minimal set of network parameters.

[0239] In particular, some embodiments of the present disclosure provide an apparatus and method for performing an online training-based data compression scheme using one or more of the following approaches, which will be described in more detail later, along with some exemplary alternatives:

[0240] - Partial update of layers of an existing NN model (already present in both the encoder and decoder), including training of partially updated (updatable) layers on the encoder side (one or more layers are updated, but not all layers are updated).

[0241] Throughout the description, the term "existing neural network" refers to a neural network already configured on the encoder side and / or the decoder side. In other words, the existing neural network may refer to a current neural network having layers with pre-configured parameters. The pre-configuration of these parameters can be performed through pre-training. Therefore, the existing or current neural network may also be referred to as a pre-configured neural network and / or a pre-trained neural network.

[0242] - Information for updating an updatable layer may be encoded and transmitted in some exemplary implementations. On the decoder side, information for updating an updatable (partially updated) layer may be decoded to obtain a modified or updated model (i.e., an updated neural network with partially updated parameters) for inference at the decoder. This modified or updated neural network can be viewed as a different neural network (a neural network other than the one prior to the (partial) update).

[0243] Throughout the description, another neural network refers to a neural network (both encoder-side and / or decoder-side) in which the existing neural network is (partially) updated. The update may include (partial) updates to one or more updatable parameters of a (selected) updatable layer and / or (partial) updates to the layers (e.g., their layer indices, or generally updates to which layers are selected as respective updatable layers for the update). Thus, another neural network may also be referred to as an updated neural network.

[0244] - In some exemplary implementations, it is possible to compress the layers to be updated (the instructions and / or parameters of these layers) using compressed representations, dictionary-based mechanisms, or other neural network compression methods.

[0245] - Partially updated (updatable) layers of another (modified, updated) NN model may have a different architecture from the existing (current) NN model. For example, the existing model is part of the other (modified, updated) model; this means that for the layers to be updated in the other NN model, the same layer of the existing NN model contains the original part and the newly added part. In the encoder, only the newly added part (also referred to as the delta part in the implementation below) is retrained, and the weights of the newly added part are encoded and sent to the decoder. In the decoder, the bits for the weights of the newly added part are decoded and used to obtain the other NN (modified, updated) model.

[0246] - The newly added parts of the layer to be updated are compressed using, for example, a concise representation, a dictionary-based mechanism, or other (e.g., lossless) network compression methods.

[0247] According to one embodiment of the present disclosure, an apparatus (2100) for encoding one or more input pictures (1010, 17), including using a neural network, is provided. The one or more input pictures may be still pictures or video pictures. A picture (still or video) may contain one or more samples (i.e., pixels). Generally, a picture is a rectangular array of samples. Samples may belong to a single channel, such as a grayscale image. However, the present invention may be equally applicable to multiple channels, such as color channels or other types of channels (e.g., depth, motion, etc.). The neural network may be any artificial neural network, such as a multi-layer perceptron or a convolutional neural network. The present disclosure is not limited to any specific neural network architecture. The term "using a neural network" above means any kind of use, such as using a filter mentioned above, which may be a loop filter. However, the present disclosure is not limited thereto. Using does not necessarily mean applying such a filter. For example, as in the case of a post-filter, a neural network can be used to make a decision only at the encoder side. The present invention is not limited to filtering. Neural networks may be used for other tasks, such as encoding or decoding picture data or data for encoding / decoding picture data, such as coding parameters (motion vectors, motion information, prediction modes, etc.). Neural networks may be used in classical hybrid encoder / decoders or neural network-based approaches. Apart from video or still picture coding, some embodiments may be used for computer vision tasks, such as object recognition or classification.

[0248] FIG. 21 illustrates a block diagram of an encoding device (2100) for encoding one or more input pictures using a neural network (NN). The device (2100) includes a processing circuit (2110) configured to select one or more updatable layers (1310) among the layers (1310, 1320) of the neural network. In other words, the encoder can determine which layers will be updated (e.g., retrained). FIG. 13 illustrates an example of a neural network NN having layers (1310) and layers (1320). In this example, layers (1310) are selected to be updated (updatable layers), which means that each parameter (e.g., weights or weight changes) can be changed (updated) during NN training. As a result, the layer (1320) is not selected for update (fixed or unupdatable layer), which means that each parameter (e.g., weight) is maintained and has pre-configured parameters. Thus, the layer (1320) is a pre-configured layer. Pre-configuration can be performed, for example, during a preprocessing step by offline training. Each index [i,...,i+s] of the layer (1310) is included in the Layer List. Unless the index is already known to the decoder, the encoder encodes the information of the pre-configured layer into a layer bitstream and transmits the layer bitstream to the decoder.

[0249] The processing circuit is additionally configured to train the neural network based on one or more input pictures (e.g., online). Training involves updating one or more parameters of one or more updatable layers among the layers of the neural network and maintaining the parameters of one or more pre-configured layers (1320). Thus, only the parameters of the updatable layers are trained, while the parameters of the pre-configured layers are not changed and are not trained (at least not while encoding one or more current pictures used to perform online training of the updatable layer parameters).

[0250] One or more parameters of one or more updatable layers may include weights. Alternatively or additionally, said one or more parameters may include biases applied to one or more updatable layers. Any other parameter(s) characterizing (configuring) the neural network may be trained. The same applies to maintained parameters of one or more pre-configured layers, which may be weights, biases, etc.

[0251] Maintaining the parameters of one or more pre-configured layers means that said parameters are not changed or updated during the training of the neural network in which one or more updatable parameters are updated. In other words, the training of the NN involves the entire network containing all parameters, but in actual training, only the updatable parameters are updated and modified. Also, please note that the maintained parameters may change and may receive some kind of update processing. However, this may occur less frequently than the updatable layers or only at the start of the video, etc. In other words, the pre-configured parameters may be fixed (in some embodiments), but they do not necessarily have to be fixed for the entire video stream, for example.

[0252] For example, a pre-configured layer can be pre-configured in such a way that each (maintained) parameter can be acquired by the pre-training of the neural network, which is performed before the actual NN training through one or more updateable parameter updates. Since pre-configuration can be pre-training and can be performed offline, the neural network is initialized with pre-trained parameters. Alternatively, (pre)training of the pre-configured parameters can also be performed online, but less frequently than the processing of training updates for updateable parameters.

[0253] The processing circuit (2110) is additionally configured to encode information about one or more updated parameters of one or more updatable layers into a parameter bitstream portion (1190). Encoding can be performed by a neural network. Additionally, the processing circuit can encode one or more input pictures into a picture bitstream portion (1180).

[0254] According to one embodiment, encoding of one or more pictures is performed using a neural network. Picture encoding can be performed with a neural network applicable to in-loop filtering. Alternatively, picture encoding can be performed without a neural network, but instead using any coding standard including MPEG-1 video, MPEG-2 video, VP8, VP9, ​​AV1, ITU-T H.262 / MPEG-2, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265, High Efficiency Video Coding (HEVC), ITU-TH.266, Versatile Video Coding (VVC), and extensions such as extensibility and / or three-dimensional (3D) extensions of these standards. This picture encoding is suitable for post-filtering.

[0255] Training a neural network can be accelerated by retraining or updating only a subset of all layers (one or more, but not all), that is, only one or more updatable layers with updatable parameters, thereby reducing the time required to train the NN. Additionally, information regarding the updated parameters (after NN training) is encoded, reducing signaling overhead. Therefore, only a portion of the NN parameters is encoded, rather than the parameterization of the entire network.

[0256] For example, one or more pre-configured layers and / or one or more updatable layers of a neural network are convolutional layers (1410, 1420). FIG. 2 illustrates an example of a convolutional layer. In this example, the size of the input data (2001) is 6×6×3 (e.g., 6 samples in width and height and 3 colors or 3 chroma and luminance components as inputs as channels), and the size of the convolutional filter (2002) is 3×3×3×2 (each kernel has a size K of 2 kernels (see Filter 1 and Filter 2) for each of the 3 channels, with 3 weights in width and height, and the number of kernels or filters defines the number of outputs). The convolutional filter (2002) can be represented by 2 filters (Filter 1, Filter 2) of size 3×3×3. Both are convolved with the input data using the sliding window principle to obtain 4×4 output data (2003) (each output channel contains 4 samples of width and height, for example, by adding the results obtained by filtering 3 input channels element by element). The final output data (2004) has a size of 4×4×2 (each of the 2 output channels contains 4 samples of width and height).

[0257] According to an additional implementation example, the processing circuit is configured to select one or more updatable layers based on the optimization of a cost function comprising at least one of rate, distortion, and complexity. For example, the cost function may be Distortion + λ * Rate, by means of a function for Rate-Distortion Optimization (RDO), where Distortion is the distortion of the reconstructed signal (the distance or difference between the reconstructed signal and the original signal), Rate is the number of bits, and λ is a coefficient called the Lagrange multiplier (simply called the Lagrangian). A function of this form allows for the optimization of the cost under the constraints of a high-quality picture (i.e., low distortion) and a small number of coding bits. In many applications, this is the preferred optimization cost function. The reconstructed signal may refer to the restored image, and the original signal may refer to the original image. Thus, since one or more updatable layers can be selected in an optimized manner, the overall training of the neural network is improved.

[0258] FIG. 11 illustrates an exemplary implementation of encoder processing, which will be described in more detail later. In one exemplary implementation, the processing circuit is configured to encode information about one or more updatable layers into a layer bitstream portion (1195). For example, the information about one or more updatable layers includes a layer index of one or more updatable layers, and the layer index is an index of one or more updatable layers among the layers of a neural network indexed according to a preset rule. For example, the preset rule for layer indexing may be a continuous indexing from the input to the output of the neural network. The parameter bitstream portion (1190), the picture bitstream portion (1180), and the layer bitstream portion (1195) may be encoded into a single bitstream as illustrated in FIG. 11. Alternatively, the different bitstream portions may be encoded individually. Different bitstream parts 1180, 1190, and 1195 may be encoded and transmitted in different orders, and other bitstream parts (e.g., for different purposes) may be arranged between them. The term "bitstream part" may also be "bitstream portion." Additionally, information regarding one or more updatable layers may include the number (i.e., quantity) of selected updatable layers. For example, the number of selected updatable layers can be pre-set so that it is known at the decoder side. Therefore, signaling information regarding updatable layers may not be required, thus reducing signaling overhead. Consequently, when signaled, signaling overhead is still reduced because the information relates to the updatable layers rather than the entire layer of the neural network. Therefore, the bit amount of the layer bitstream part is small.

[0259] In one exemplary implementation of the device (2100) illustrated in FIG. 21, the processing circuit (2110) may include respective modules for encoding, such as a selection module (2112) for selecting one or more updatable layers among NN layers, and an NN training module (2114) for training a neural network including a pre-configured layer using the selected updatable layer and an input picture. Finally, the encoding module (2116), encoding module (2118), and encoding module (2120) encode information about the updatable layer, information about the updated parameter, and the input picture into respective bitstream portions.

[0260] According to an exemplary implementation, an updateable layer (1310) has one or more updateable parameters and one or more pre-configured parameters. The updateable layer belongs to one or more selected updateable layers. Additionally, a processing circuit is configured to train the neural network by updating one or more updateable parameters of one or more updateable layers and maintaining one or more pre-configured parameters. In other words, the updateable layer has parameters, some of which are updated during NN training, while the remaining parameters are pre-configured and do not change during NN training. This generally means that all parameters of the updateable layer can be changed by training, but the parameters of the updateable layer are divided into a subset of parameters that change during training and another subset of parameters that maintain parameter values ​​during NN training. The maintained pre-configured parameters of the updateable layer may be pre-configured by pre-training performed in a similar manner for the (maintained) pre-configured parameters of one or more pre-configured layers of the NN. Therefore, since not all updateable parameters of the above-mentioned updateable layer may be updated, NN training is further accelerated, as well as the overhead for calculating and signaling update information for updated parameters. Consequently, the amount of parameters updated during NN training is further reduced.

[0261] Figure 16 further illustrates this, showing an updateable layer on the left that can be a Convolutional layer Conv1, which is 3x3x48x48. As illustrated, the output of the first layer Conv1 is input into a common ReLU. On the right, the original part and the so-called delta part are shown, both of which are part of the common Conv1 layer on the left. Here, the original part refers to the updateable layer where pre-configured parameters are maintained during training. The delta part refers to the updateable parameters that will be updated during NN training.

[0262] In an exemplary implementation, an updateable layer (1310, 1620, 1705) includes an updateable sublayer (1526, 1626, 1720) in which the weight is one or more updateable parameters, and a preconfigured sublayer (1524, 1624, 1710) in which the weight is one or more preconfigured parameters. Referring to FIG. 16, the updateable sublayer corresponds to the delta portion (i.e., delta layer) and the preconfigured sublayer corresponds to the original portion (i.e., original layer). The updateable sublayer and the preconfigured sublayer are parallel (sub)layers, and their results are combined into a common output. Additionally, the same input of the updateable layer is provided to the updateable sublayer and the preconfigured sublayer; and the output of the updateable layer is the element-wise sum (1730) of the updateable sublayer and the preconfigured sublayer. Providing the same input to two sublayers means that the updatable sublayer and the pre-configured layer of the updatable layer are in parallel, as illustrated in FIG. 16. Additionally, each output of the sublayer is summed. In one example, element-wise summing can be performed for all outputs of the sublayer.

[0263] FIG. 17 illustrates another example of an updateable layer (1705) similar to the layer (1620) of FIG. 16. The layer (1705) may be a convolutional layer and is selected by the encoder as an updateable layer. In the case illustrated in FIG. 17, the layer (1705) includes an original part (1710) and a delta part (1720). The original part (1710) corresponds to a pre-configured sublayer, while the delta part (1720) corresponds to an updateable sublayer. Thus, during training, the original part (1710) maintains pre-configured parameters (e.g., weights), and the parameters of the delta part (1720) are updated (e.g., weight changes). Note that in this exemplary implementation, the original part and the delta part are arranged in parallel and take the same input. Their outputs are summed element by element (1730).

[0264] Training the weights of an updateable sublayer while using the sublayers of the updateable sublayer and maintaining pre-configured weights allows the updateable layer to be trained based on changes in weights by referencing the pre-configured weights. This is called delta weight training. Therefore, since pre-configured weights are used as a reference for training, NN training is accelerated. In other words, the convergence of NN training can be faster.

[0265] According to one example, the updateable sublayer includes sparse neural network layers (3006, 4003, 5010). As a result, the number of parameters in the updateable sublayer is reduced. For example, the updateable sublayer includes tensor-trained neural network layers (3006, 4003, 5010). The tensor-trained neural network is an example of a sparse NN. The tensor-trained NN layer is an example of a lightweight convolutional layer shown in FIG. 3.

[0266] Figure 3 shows an example of the lightweight convolutional layer described above. The size is For the filter (3001), each TTConv layer has three convolution layers (conv1, conv2, conv3). The size of Conv1 (3002) is is. The size of Conv2(3003) is is. The size of Conv3(3004) is and, here, represents the rank of the TTConv layer. Generally, the lower the rank, the fewer parameters the TTConv layer has. There is a trade-off between the TTConv rank and performance. The lightweight version of convolution has the same input and output sizes as regular convolution but has significantly fewer parameters. For example, for a regular convolution with a kernel size of 3x3, the number of input and output channels is 64, and the parameters are 3*3*64*64 = 36,864. In contrast, a tensor-trained convolution layer with R_1=R_2=4 has 64*4*4*1*1 + 4*3*3 + 4*64*1*1 = 1,316 parameters, as mentioned above.

[0267] According to one implementation, the processing circuit is configured to update one or more updatable parameters using a training method that is less complex than the training method used to train one or more pre-configured parameters. For example, since training of an updatable layer can be performed using pre-configured parameters for a pre-configured sublayer, the actual training of the updatable parameters of the updatable sublayer may require fewer parameter updates and fewer training steps. Therefore, the complexity of NN training is reduced, and training costs (e.g., complexity, latency, etc.) can be saved.

[0268] In one example, one or more updateable layers are convolutional layers (1003, 3001, 810, 850, 1410, 1420, 1620, 1705), and at least one of the following applies: a less complex training process updates fewer parameters than the training method used to train one or more pre-configured parameters; less complex training updates a smaller convolutional kernel than the training method used to train one or more pre-configured parameters. Accordingly, the time required for NN training can be reduced. Furthermore, the smaller the convolutional kernel, the fewer updated parameters are output, and thus the signaling overhead of said updated parameters is reduced.

[0269] FIG. 2 illustrates an example of a convolution layer. In this example, the size of the input data (2001) is 6×6×3 (e.g., 6 samples of width and height and 3 color or 3 chroma and luminance components as input channels) and the size of the convolution filter (2002) is 3×3×3×2 (the size K of each kernel is 3 weights in width and height, 2 kernels for each of the 3 channels (see Filter 1 and Filter 2), and the number of kernels or filters defining the number of output channels). The convolution filter (2002) can be represented by 2 filters (Filter 1 and Filter 2) of size 3×3×3. Both are convolved with the input data using the sliding window principle to obtain 4×4 output data (2003) (each output channel contains 4 samples of width and height, e.g., adding the results obtained by filtering 3 input channels element by element). The size of the final output data (2004) is 4×4×2 (two output channels, with width and height each containing 4 samples). As such a tensor training layer, the TTConv layer shown in FIG. 3 is a specialized convolution layer. For example, in the case of a regular convolution with a kernel of 3, the number of input and output channels is 64, and it has 3*3*64*64=36864 parameters. Conversely, a tensor training convolution layer with R_1=R_2=4 has 64*4*4*1*1+4*3*3+4*64*1*1=1316 parameters.

[0270] In an additional embodiment, the processing circuit is configured to update one or more updateable parameters of one or more updateable layers (1626) by pseudo-randomly initializing them; and to perform training with pairs of original pictures (910) and reconstructed pictures (960). Thus, the updateable layers are initialized in a fast and simple manner, thereby reducing training complexity. FIG. 16 illustrates a case where the updateable layers include updateable sublayers and pre-configured sublayers (delta weight training). As discussed above, in this case, the pre-configured parameters refer to pre-configured weights, which can be obtained through pre-training in an offline or online training process. These weights can be approximated to the final weights obtained after NN training. Part of the training involves initializing the updateable parameters to be trained, which can be done randomly. Using these random weights as a kind of initial guess, NN training can be performed using the original picture and the reconstructed picture. For example, picture residuals can be determined to be used as a cost function for the NN to train the weights. It should be noted that this is true for posterior filters, loop filters, space-time filters, or other reinforcement filters. For example, in the case of end-to-end (E2E) coding, training may be performed based only on the original video picture. For some other tools, there may be other options. For example, the transformation processing unit (e.g., the transformation processing unit (206) or inverse transformation processing unit (212, 312) shown in FIG. 25) may be trained based on NN, in which case a cost function related to the degree to which the transformation coefficients can be quantized may be used.

[0271] According to an additional implementation, the processing circuit is configured to train a neural network by updating one or more parameters of one or more updateable layers based on pairs of original picture (910) and reconstructed picture (960). The reconstructed picture may be reconstructed without a neural network. Additionally, one or more parameters of one or more updateable layers include weights of one or more updateable layers. In this example implementation, NN training updates the respective weights using pairs of original picture and reconstructed picture. In other words, picture residuals (i.e., the difference between the reconstructed picture and the original picture) may be used to train the NN, but it is the weights that are updated. One or more updateable layers (1310) of the NN are convolutional layers (1410, 1420). FIGS. 14 and 16 (left) illustrate an example of two convolutional layers, Conv1 and Conv2, having kernels of 3x3x48x48. In Figures 14 and 16 (left), each Conv layer to be updated does not have an updateable and pre-configured sublayer, as previously discussed and illustrated on the right side of Figures 15 and 16. Consequently, the updateable parameter of the updateable layer is the weight that is directly updated (referred to as weight training). Note that while the weight can be changed, the weight can still be pre-configured by prior training (online or offline training) and thus can be used as an initialization parameter for the weight. Another option is to pre-set the weight. Alternatively, the weight can be initialized randomly, for example. Thus, the architecture of the neural network has low complexity because the updateable layer does not include two sublayers arranged in parallel in this case.

[0272] According to one implementation, the processing circuit is further configured by pre-training a neural network containing update parameters of one or more pre-configured layers less frequently than training one or more updatable layers based on one or more input pictures; and is configured to encode information regarding the updated parameters of one or more pre-configured layers into a network bitstream portion. The neural network here refers to an existing NN model existing on both the encoder and decoder sides. It should be noted that the existing NN model does not imply that the NN is a fixed part of the encoder and decoder in terms of the pre-configured parameters. Rather, the pre-configured parameters can be pre-trained and retrained as needed for the problem. Therefore, pre-training for the pre-configuration of one or more pre-configured layers means that said pre-training is performed (offline or online) before the actual training of the NN in which the updateable parameters of the updateable layers are updated. The difference lies in the frequency with which the pre-training of the pre-configured parameters is performed (i.e., how often during a specific period). Since pre-configured parameters are generally more numerous than updateable parameters, pre-training is performed less frequently than the actual NN training. Existing NN models can also be compressed and transmitted to the decoder, but this takes significantly longer. For example, pre-training can use 10 or 32 picture Ps, but pre-training is performed every 10,000,000 frames. A new existing model (i.e., a newly pre-configured NN) can be transmitted to the decoder. Of course, pre-training can also be applied iteratively to the new existing model. Thus, the actual training of the neural network can be improved using updated pre-configured parameters. Additionally, information regarding the updated pre-configured parameters is encoded in a network bitstream part separate from other bitstream parts (video, parameters, layers).Therefore, information regarding pre-configured parameters with a low update frequency is distinguished from information regarding updatable parameters with a high update frequency.

[0273] In another exemplary implementation, the processing circuit additionally acquires a plurality of pre-configured layer sets; and is configured to select a set from the plurality of pre-configured layer sets that includes one or more of the pre-configured layers of the neural network, wherein information regarding the updated parameters is an instruction of the selected set. The instruction of the selected set may be an index of each neural network layer. Alternatively, the instruction may be a parameter of the NN. For example, there may be multiple existing NN models N1, N2, …, Nj, …, NM, and among them, the encoder may select one model Nj as the existing NN model N (Nj → N). Then, the encoder transmits, for example, an NN index j to the decoder. The decoder determines the existing NN model N to be used based on the index j. Thus, NN training can be adapted to a specific compression task by selecting an existing NN, thereby enhancing the overall NN training.

[0274] In other examples, the selection and / or updating of layers are performed at different intervals. As discussed above, layer selection directly influences the architecture of the NN in that it specifies the NN layer to be actually updated. Consequently, layer updates following layer selection affect the output response through updatable parameters, which change during NN training. Thus, interference between layer selection and layer updates can be avoided, which improves the performance and robustness of NN training. Additionally, NN training can be adapted based on the preference for layer selection or layer updates.

[0275] For example, every P frame contains corresponding information for updating the layers to be updated. However, P can vary even within a single video. For example, assume a video of 1000 frames (f0, f1, …. f999). A first set of frames containing frames f0 through f10 will have corresponding information for updating the layers to be updated. Another set of frames containing frames f11 through f35 will have different corresponding information for updating the layers to be updated. Additionally, the layer list updateLayersList may change for different frame sets, which means that the layers available for update can be selected based on a specific frame set.

[0276] For example, information about one or more updatable layers is encoded less frequently than information about one or more updated parameters of one or more updatable layers. This means that the selection of updatable layers is performed less frequently than the retraining of parameters of updatable layers. Therefore, NN training can be performed prioritizing layer updates (i.e., updated parameters) over layer selection, which can further simplify NN training.

[0277] In terms of bitstream syntax, such an implementation may involve indicating the layer bitstream portion (1195) less frequently than the parameter bitstream portion (1190). For example, the layer bitstream portion (1195) may be signaled within a set of parameters applicable to multiple pictures, whereas the parameter bitstream portion (1190) is signaled within a picture header or slice header, or generally within a set of parameters applicable to fewer pictures than the set of parameters to which the layer bitstream portion is signaled. In particular, the layer bitstream portion (1195) may be signaled in a sequence parameter set, a picture parameter set, a video parameter set, etc., while the parameter bitstream portion (1190) may be signaled in a picture header, etc.

[0278] In another embodiment, training is based on pairs of first number input pictures and reconstructed pictures; encoding of information for one or more updated parameters is performed after encoding of the first number input pictures, and the trained neural network is applied to the second number reconstructed pictures; the second number is greater than the first number. The term "number" indicates the quantity of pairs. The second number input pictures may include all pairs of the first number input pictures. Alternatively, the second number pairs may include some of the first number pairs. Alternatively, the second number pairs may not include the first number pairs.

[0279] For example, assume that 100 frames are being encoded. The encoder can use frames f0 through f9 for encoding and use them to train a neural network to obtain a new model (i.e., a new NN with updated parameters). Then, the new network model can be used as a filter for the next frames, e.g., frames f10 through f99, just as in the case of an in-loop filter or a post-processor filter. However, the new network model may not be applied to frames f0 through f9. Alternatively, even if a frame is encoded and transmitted as f0 (in the video bitstream portion), frame f0 cannot be reconstructed (because there is no NN filter), and if the encoding of frame f9 is not completed, it is reported to the decoder side. Then, the NN model can be trained, and the NN model is signaled to the decoder side. In this case, low latency is affected (real-time encoding). In the example above, there is a delay of >10 frames. The encoding (bitstream) order could be as follows: [f0–f9][NN parameters][f11–f99]. In this case, since the NN parameters can already be obtained after encoding frames [f0–f9], there is no need to wait for frames [f11–f99]. A similar approach applies to the NN parameters set for frames [f11–f99]. In this case, the quality of frames f0–f9 may be low, but low-latency functionality can be implemented. Alternatively, the new network model can be applied to a portion of a first set of frames f0–f9, such as frames f8 and f9. Alternatively, the new network model can be applied to all of the first set of frames f0–f9. Regarding the decoder side, the decompression network model can be applied to the same frames as the encoder side discussed above. Otherwise, even if frame f0 is decoded, frame f0 cannot be displayed after f9 if the NN parameters are unavailable. The new network model may not be applied to frame f0.In this case, low latency will be affected (real-time decoding).

[0280] Therefore, NN training can be performed on a smaller set of input picture pairs, whereas the decoder-side NN can be applied to a larger set of input pictures, which accelerates NN training and makes it less complex. However, there may be general adaptations to picture content.

[0281] For example, to obtain information about the updated partial layer, the reconstructed P-frame and the original P-frame are used. Another NN model derived from the decoder (i.e., the NN with updated parameters) is applied to the P-frame using the decoded information. Note that the derived other NN model may also be used for frames other than the P-frame. For example, assume a video of 1000 frames (f0, f1, …., f999). A first set of frames containing frames f0 through f10 will have corresponding information for updating the layer to be updated. The decoder derives another NN model using the corresponding information. The other NN model is applied to another set of frames containing frames f11 through f35. The other NN model may be applied to other sets of frames, but may not be applied to the set of frames containing frames f0 through f10.

[0282] In one example, the neural network is a neural network trained for post-filtering (2702). FIG. 27 illustrates an encoder (20) having an in-loop filter (220) and provides an output (271) of encoded picture data (21). In this case, a CNN filter (i.e., an enhancement filter) may be located outside the encoder, and the CNN filter takes the output (272) as input. Each output will be a filtered output (Option 2). In another example, the neural network is trained to perform one of in-loop filtering (2701), end-to-end picture coding, adaptive sampling, and coding of picture coding parameters. In this case, the CNN filter may be located to the left of the in-loop filter (220) of FIG. 27 with the filtered block (221) as input (Option 1). Each output of the CNN filter is provided as input to a decoded picture buffer (230). Picture coding parameters may include motion vectors, prediction types, etc. Therefore, the same architecture of a neural network can be used to train the NN for different types of video compression tasks. Thus, NN training can be performed in a flexible manner and adapted to various tasks.

[0283] For example, in the case of adaptive spatial sampling for video compression, the reconstructed spatially downsampled video and the original downsampled video (original spatial resolution video) are used as training data to obtain information used to update the partial layers to be updated. Information regarding the updated parameters is encoded and transmitted to the decoder. The decoder decodes the information and uses it to obtain another model. This other model refers to the neural network on the decoder side that uses the updated parameters. This other model is used to upsample the decoded low spatial resolution video.

[0284] FIG. 19 illustrates the encoder method described above. In this example, an offline step is performed as a form of preprocessing, followed by subsequent online training. Starting with an existing network architecture (1910) (e.g., a neural network), a video dataset (1920) containing multiple input pictures is used to train the network's parameters (1930). In this example, the updateable parameters (i.e., pre-trained ones) are weights. Note that these parameters correspond to parameters pre-configured as a result of offline training. Then, using an input picture of a test video (1940) or a current video (1940) (e.g., currently to be coded), online training is performed in such a way that the weights pre-configured (i.e., pre-trained) from the offline training are used to fine-tune the updateable parameters of a selected layer of the network (1910). In this example, the weights undergo fine-tuning (1950) so that they can be updated during online training. The result of the online training is one or more updated parameters (here, weights) for the selected updateable layer. In the encoder method of FIG. 19, a subtraction operation (1955) is performed in which pre-configured weights (see Weights) are subtracted from the trained weights of the updateable layer to provide a weight difference. This weight difference represents information about the updated parameters that is encoded (i.e., compressed) (1960) into the parameter bitstream portion. Since the value range of the weight difference is generally smaller than the actual weight values, the cost of the bits for weight changes is reduced. Alternatively, instead of the updated parameters, the element-wise difference (minus) between the updated parameters and the pre-trained parameters is encoded into the parameter bitstream. Additionally, the test video is encoded (i.e. compressed) into the picture bitstream portion (1970). The compressed bitstream containing the picture portion and the parameter portion is provided as an output (1980).Note that the output (1980) may further include an encoded layer bitstream portion containing information about the selected updateable layer.

[0285] According to one embodiment of the present disclosure, an apparatus (2200) for decoding one or more input pictures (1060, 331), including using a neural network, is provided. One or more input pictures may be still pictures or video pictures. A picture (still or video) may contain one or more samples (i.e., pixels). Generally, a picture is a rectangular array of samples. Samples may belong to a single channel, such as a grayscale image. However, the present disclosure may apply equally to multiple channels, such as color channels or other types of channels (e.g., depth, motion, etc.). The neural network may be any artificial neural network, such as a multilayer perceptron or a convolutional neural network. The present disclosure is not limited to any specific neural network architecture. Decoding may be suitable for decoding a bitstream containing the aforementioned bitstream portions (1180, 1190, 1195) as generated by an exemplary encoder.

[0286] FIG. 22 illustrates a block diagram of a decoding device (2200) for decoding one or more input pictures using a neural network (NN). The device (2200) includes a processing circuit (2210) configured to decode information about one or more parameters of one or more updatable layers (1310) from a parameter bitstream portion (1220).

[0287] The processing circuit is additionally configured to decode one or more input pictures, including processing using a neural network from a picture bitstream portion (1210). The neural network includes one or more updatable layers (1310) having one or more decoded parameters; and one or more pre-configured layers (1320) having pre-configured parameters. One or more decoded parameters may also have pre-trained parameters. One or more pre-configured parameters may be pre-trained. Thus, the updated parameters of the updatable layers(s) are used when decoding one or more input pictures.

[0288] One or more parameters of one or more updatable layers may include weights. Alternatively or additionally, said one or more parameters may include biases applied to one or more updatable layers. Any other parameter(s) characterizing (configuring) the neural network may be included in the parameter bitstream portion. This is similarly applied to the parameters of one or more pre-configured layers, which may be weights, biases, etc.

[0289] Additionally, please note that pre-configured parameters may be changed and certain types of update processing may be applied. However, this may occur less frequently than in the updateable layer or only at the start of the video. That is, pre-configured parameters may be fixed (in some embodiments), but do not need to be fixed for, for example, the entire video stream.

[0290] For example, a pre-configured layer can be pre-configured in such a way that each parameter can be obtained through pre-training of the encoder-side neural network performed prior to the actual encoder-based NN training by updating one or more updatable parameters. Pre-configuration by pre-training can be performed offline. Information regarding pre-configured parameters and / or pre-configured layers can be encoded into a portion of the network bitstream, where a decoder will decode the respective information regarding pre-configured parameters and / or pre-configured layers. This information regarding pre-configured parameters and / or layers can be provided to the decoder for decoding less frequently than information regarding updated parameters.

[0291] Therefore, one or more input pictures can be decoded quickly by using a neural network that uses fewer updated parameters than the pre-configured parameters. Thus, as updated NN parameters are used, the efficiency of picture decoding is improved while ensuring the high quality of the decoded picture.

[0292] According to an embodiment, decoding of one or more pictures is performed using a neural network. Picture decoding can be performed with a neural network applicable to in-loop filtering. Alternatively, picture decoding can be performed without a neural network, but instead using arbitrary coding standards including MPEG-1 video, MPEG-2 video, VP8, VP9, ​​AV1, ITU-T H.262 / MPEG-2, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265, High Efficiency Video Coding (HEVC), ITU-TH.266, Versatile Video Coding (VVC), and extensions such as extensibility and / or three-dimensional (3D) extensions of these standards. This picture decoding is suitable for post-filtering.

[0293] In one embodiment, the processing circuit is configured to decode information about one or more updatable layers from a layer bitstream portion (1230). For example, the information about one or more updatable layers includes a layer index of one or more updatable layers, and the layer index is an index of one or more updatable layers among the layers of a neural network indexed according to a preset rule.

[0294] For example, a preset rule for layer indexing may be continuous indexing from the input to the output of the neural network. The parameter bitstream part, picture bitstream part, and layer bitstream part may be a single bitstream in which information about the picture, updated parameters, and updatable layers can be decoded, as illustrated in FIG. 12. Alternatively, the video and information (updated parameters, updatable layers) may be decoded separately from their respective bitstream parts. The term bitstream part may also be bitstream portion. Additionally, information about one or more updatable layers may include the number (i.e., quantity) of updatable layers for which the parameters will be updated. For example, the number of updatable layers can be preset so that it is known at the decoder side. Therefore, the decoder may not need to decode information about updatable layers, thus reducing signaling overhead. Consequently, when signaled, the overhead for decoding is still reduced because the information is associated with the updatable layer rather than the entire layer of the neural network. Therefore, the amount of bits in the portion of the layer bitstream being decoded is small.

[0295] In one exemplary implementation of the device (2200) illustrated in FIG. 22, the processing circuit (2210) may include each module for decoding, for example, a decoding module (2216) for decoding from layer bitstream portion information for an updateable layer, a decoding module (2218) for decoding from parameter bitstream portion information for an updated parameter, and a decoding module (2220) for decoding one or more input pictures from a picture bitstream portion.

[0296] Additionally, one or more pre-configured layers (1320) and / or one or more updateable layers (1310) of the neural network are convolutional layers (1410, 1420). FIG. 2 illustrates an example of a convolutional layer. In this example, the size of the input data (21) is 6×6×3 and the size of the convolutional filter (2002) is 3×3×3×2. The convolutional filter (2002) can be represented by two filters (Filter 1, Filter 2) of size 3×3×3. Both are convolved with the input data using the sliding window principle to obtain 4×4 output data (2003). The size of the final output data (2004) is 4×4×2 filled with the outputs of the two filters (2003).

[0297] According to an exemplary implementation, the updateable layer (1310, 1620, 1705) has one or more updateable parameters and one or more pre-configured parameters. The processing circuit is configured to update one or more updateable parameters of one or more updateable layers and maintain one or more pre-configured parameters.

[0298] In other words, the updateable layer has parameters, some of which are updated, while the remaining parameters are pre-configured and remain unchanged when decoding one or more input pictures. This generally means that while all parameters of the updateable layer can be changed by training, the parameters of the updateable layer are divided into a subset of parameters that are updated, and another subset of parameters retains their parameter values. The retained pre-configured parameters of the updateable layer can be pre-configured by pre-training on the encoder side. Therefore, since not all updateable parameters of the updateable layer may be updated, NN-based picture decoding can be accelerated, as well as reducing the overhead required to decode information regarding the updated parameters. Consequently, the amount of parameters representing the updated parameter configuration of the NN is further reduced.

[0299] In an exemplary implementation, the updateable layer (1310, 1620, 1705) includes an updateable sublayer (1526, 1626, 1720) in which the weight is one or more updateable parameters, and a pre-configured sublayer (1524, 1624, 1710) in which the weight is one or more pre-configured parameters. Referring to FIGS. 15 and 16, the updateable sublayer corresponds to the delta portion (i.e., delta layer) and the pre-configured sublayer corresponds to the original portion (i.e., original layer). Additionally, the same input of the updateable layer is provided to the updateable sublayer and the pre-configured sublayer; and the output of the updateable layer is the element-wise sum (1730) of the output of the updateable sublayer and the output of the pre-configured sublayer. Providing the same input to both sublayers means that the updateable sublayer and the pre-configured layer of the updateable layer are in parallel, as illustrated in FIGS. 15 and 16. In addition, the outputs of each sublayer are summed (added). In one example, element-wise summing can be performed for all elements of a sublayer.

[0300] Therefore, picture decoding is accelerated by using preset weights as a reference and updating only weight changes.

[0301] According to one example, the updateable sublayer includes sparse neural network layers (3006, 4003, 5010). As a result, the number of parameters in the updateable sublayer is reduced. For example, the updateable sublayer includes tensor-trained neural network layers (3006, 4003, 5010). The tensor-trained neural network is an example of a sparse NN. The tensor-trained NN layer is an example of the lightweight convolutional layer illustrated in FIG. 3 and described above.

[0302] According to one implementation, one or more updatable layers are convolutional layers (1003, 3001, 810, 850, 1410, 1420, 1620, 1705); and at least one of the following applies: the updatable sublayer has fewer parameters than the pre-configured sublayer; or the updatable sublayer applies a smaller convolutional kernel than the pre-configured sublayer. Thus, the time required for picture decoding is reduced. Also, because the smaller convolutional kernel has fewer updated parameters, the amount of data processed is reduced.

[0303] In one exemplary implementation, one or more parameters of one or more updatable layers include the weights of one or more updatable layers. Figures 14 and 16 (left) show an example of two convolutional layers, Conv1 (1410) and Conv2 (1420), with a kernel of 3x3x48x48. In Figures 14 and 16 (left), each Conv layer to be updated does not have an updatable and pre-configured sublayer as previously discussed and illustrated on the right side of Figures 15 and 16. Consequently, the updatable parameters of the updatable layers are the weights that are directly updated. Note that while the weights can be changed, they can still be pre-configured by pre-training (online or offline training) on ​​the encoder side. Another option is to pre-set the weights. Thus, the architecture of the neural network has low complexity because the convolutional layers do not include two sublayers arranged in parallel in this case.

[0304] According to one implementation, the processing circuit additionally decodes information regarding updated parameters of one or more pre-configured layers from a portion of the network bitstream; and is configured to update the pre-configured parameters less frequently than the updateable parameters based on the information regarding the updated parameters. Here, the neural network refers to an existing NN model existing on the decoder side. For example, the decoder may have 10 layers with pre-configured fixed parameters. Then, on the encoder side, only one layer may be updated (retrained), while the remaining 9 layers maintain the pre-configured fixed parameters (partial update). Alternatively, all parameters of the 10 layers may be retrained on the encoding side, which may be performed less frequently (full update). Please note that the existing NN model may not imply that the NN is a fixed part of the decoder in terms of pre-configured parameters.

[0305] Pre-configured parameters can be pre-trained on the encoder side and retrained as needed. Therefore, pre-training for the pre-configuration of one or more pre-configured layers implies that updateable parameters can be pre-trained (offline or online) on the encoder side, where the updated pre-configured parameters are transmitted to the decoder. The decoder then uses the updated parameters to update the parameters of the pre-configured layers. The difference lies in the frequency with which pre-configured parameters are pre-trained (i.e., how often during a specific period). Since pre-configured parameters are generally more numerous than updateable parameters, pre-training is performed less frequently, and thus updated pre-configured parameters need to be transmitted to the decoder less often. Consequently, the decoder must decode information regarding the updated parameters less frequently. As a result, existing NN models can also be compressed and transmitted to the decoder, but this takes a much longer time. For example, pre-training on the encoder side can use 10 or 32 pictures P, but the pre-training runs every 10,000,000 frames. Therefore, a new existing model (i.e., a newly pre-configured NN) can be transmitted to the decoder every 10,000,000 frames. Likewise, the decoder must decode information about the updated pre-configured parameters every 10,000,000 frames.

[0306] Therefore, picture decoding can be improved by using updated pre-configured parameters for the decoder-side NN. Additionally, information regarding the updated pre-configured parameters is decoded in a network bitstream portion separated from other bitstream portions (video, parameters, layers). Thus, information regarding pre-configured parameters that are updated less frequently (low frequency) is distinguished from information regarding parameters that can be updated more frequently (high frequency).

[0307] In another implementation, the processing circuit is configured to obtain instructions from a selected set of multiple pre-configured layer sets from information regarding additionally updated parameters, wherein the selected set includes one or more of the aforementioned pre-configured layers of the neural network. The instructions of the selected set may be indices of each neural network layer. Alternatively, the instructions may be parameters of the NN. For example, there may be multiple existing NN models N1, N2...Nj, NM on the encoder side, and the encoder may select one model Nj as the existing NN model N (Nj → N). Note that in this example implementation, multiple existing NN models also exist on the decoder side. The encoder then transmits, for example, an NN index j (instruction) to the decoder. The decoder determines the existing NN model N to be used based on the index j. Thus, since decoding can be adapted to a specific compression task using a different existing NN, overall picture decoding is improved.

[0308] According to one implementation, information about one or more updatable layers is decoded less frequently than information about one or more updated parameters of one or more updatable layers. Therefore, decoding can be performed prioritizing layer updates (i.e., updated parameters) over layer selection (i.e., updated layer index), thereby accelerating picture decoding.

[0309] It should be noted that layer updates (i.e., updated layer indices) directly influence the architecture of the decoder NN in that they specify the decoder NN layer selected for the actual update of information regarding the updatable layers. Consequently, the actual update of each parameter affects the output response through the updatable parameters, which can change more frequently. Therefore, interference between layer updates (i.e., updated layer indices) and parameter updates can be avoided, improving the performance and robustness of decoding. Furthermore, decoding can adapt based on the preference for layer updates or parameter updates. Thus, decoding can be performed in a flexible manner and adapted to specific tasks.

[0310] In one implementation example, a neural network is applied to post-filtering (2702). FIG. 25 shows a CNN filter (i.e., reinforcement filter) located outside the decoder (Option 2) for this case. In another exemplary implementation, a neural network is applied for in-loop filtering (2701), picture decoding, adaptive sampling, and decoding of picture coding parameters. FIG. 27 illustrates a CNN filter located inside the decoder in this case (Option 1). Picture coding parameters may include motion vectors, prediction types, etc. Thus, a neural network of the same architecture can be used to decode input pictures for different types of video compression tasks. Thus, decoding can be performed in a flexible manner and adapted to various tasks including post-filtering and in-loop filtering.

[0311] For example, in the case of adaptive spatial sampling for video compression, the reconstructed spatially downsampled video and the original downsampled video (original spatial resolution video) are used as training data to obtain information used to update the partial layers to be updated. Information regarding the updated parameters is encoded and transmitted to the decoder. The decoder decodes the information and uses it to obtain another model. This other model refers to the neural network on the decoder side that uses the updated parameters. This other model is used to upsample the decoded low spatial resolution video.

[0312] FIG. 20 illustrates the decoder method according to the encoder method of FIG. 19. In this example, the input includes a compressed video (2010) (i.e., a picture bitstream portion) and a compressed weight (2020) (i.e., an encoded parameter bitstream portion). In the example of FIG. 20, since weight changes generally have a smaller value range than the actual weight value range, the compressed parameter has a weight change W to save bits. T It is -W. Video and updated parameters (e.g., weight change W T-Signed as -W) provides the decompressed (decoded) video (2030) and the decoded updated parameters (2040). To update the (existing) decoder-side neural network (2050), the weights of the updateable layers must be updated. In other words, the existing network updates the parameters (e.g., weights) of the corresponding layers that the decoder already knows as updateable layers or that the decoder knows from information about the updateable layers decoded from the layer bitstream portion. As mentioned above, since the updated parameters are weight changes in this example, an additional operation (2060) (parameter-wise) is performed so that the changes are added to the (pre-trained) pre-configured weights of each updateable layer. The existing neural network (2050) is now updated. Alternatively, if the decoded parameter bitstream contains unupdated parameters and includes element-wise subtraction of the updated parameters and pre-configured parameters, the decoded parameter bitstream is added to the element-wise pre-trained parameters (2060). The decompressed video bitstream is processed by an updated neural network for neural network inference (2070) to obtain the decoded input picture as an output (2080).

[0313] According to one embodiment of the present disclosure, a method for encoding one or more input pictures (1010, 17) is provided, including using a neural network. FIG. 23 illustrates a flowchart including the step (S2310) of selecting one or more updatable layers (1310) among the layers (1310, 1320) of the neural network. The method further includes the step (S2320) of training the neural network, including updating one or more parameters of one or more updatable layers based on one or more input pictures and maintaining parameters of one or more pre-configured layers (1320) among the layers of the neural network. Additionally, the method further includes the step (S2340) of encoding information about one or more updated parameters of one or more updatable layers into a parameter bitstream section (1190), and the step (S2350) of encoding one or more input pictures into a picture bitstream section (1180). The encoding method may further include a step (S2330) of encoding information for one or more updatable layers into a layer bitstream portion (optional).

[0314] Training a neural network by updating a subset of NN parameters in an updatable layer with updatable parameters offers the advantage of accelerating NN training and saving the time cost associated with training. Additionally, signaling overhead is reduced because information regarding the updated parameters (after NN training) is encoded. Therefore, only a portion of the NN parameters is encoded, rather than the parameterization of the entire network.

[0315] According to one embodiment of the present disclosure, a method for decoding one or more input pictures (1060, 331) is provided, including using a neural network. FIG. 24 illustrates a flowchart including step S2410 (optional) for decoding from layer bitstream portion information for one or more updatable layers. Step S2420 is further included for decoding information for one or more parameters of one or more updatable layers (1310) from a parameter bitstream portion (1220). Finally, step S2430 is included for decoding one or more input pictures, including processing from a picture bitstream portion (1210) with a neural network, wherein the neural network includes one or more updatable layers (1310) having one or more decoded parameters; and one or more pre-configured layers (1320) having pre-configured parameters.

[0316] Therefore, one or more input pictures can be rapidly decoded using a neural network with a smaller amount of updated parameters than the amount of pre-configured parameters. Thus, while the efficiency of picture decoding is improved, the quality of the decoded picture is guaranteed as updated NN parameters are used.

[0317] According to one embodiment of the present disclosure, a computer-readable non-transient storage medium is provided for storing a program, wherein the program includes instructions that cause one or more processors to perform a method according to any one of the embodiments described above when executed on one or more processors.

[0318] According to one embodiment of the present disclosure, an apparatus for encoding one or more input pictures, comprising using a neural network, is provided, the apparatus comprises: one or more processors; and a computer-readable non-transient storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the program configures an encoder to perform the encoding method described above when executed by the one or more processors.

[0319] According to one embodiment of the present disclosure, an apparatus for decoding one or more input pictures is provided, comprising using a neural network, wherein the apparatus comprises one or more processors; and a computer-readable non-transient storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the apparatus is configured to perform the decoding method described above when the program is executed by the one or more processors.

[0320] According to one embodiment of the present invention, a computer program is provided that includes program code for performing a method according to any one of the encoding and / or decoding methods when executed on a computer.

[0321] Below, we discuss in more detail an example of applying a post-filter to an encoder-decoder configured such that the processing circuit performs the following processing.

[0322] FIG. 11 illustrates a specific detailed and exemplary process performed by an encoder and can be itemized as follows:

[0323] Step 1: Obtain the existing neural network N and the reconstructed data Vrec (1110).

[0324] The existing neural network will be denoted by N, and the convolution layer and the pre-trained weights W of these layers using a larger dataset = It may have. For example, the existing neural network N may be a neural network of a classical residual network (Resnet) with a total of 20 residual blocks. FIG. 14 illustrates the residual block method. This includes two convolutional layers of size 3×3×48×48 (conv1 (1410), conv2 (1420)), a ReLU layer (1430) after conv1, a skip connection (1440) from the input of the block to the output of conv2 (1420), a element-wise sum (450) of the conv2 output and the skip connection, and a ReLU layer (1460) after the element-wise sum. In this example, N has a total of 40 convolutional layers It can have. The existing neural network N already exists in the encoder. The term existing neural network refers to a neural network that the encoder has already implemented and may have been pre-trained.

[0325] The original P frames (P is an integer representing the number of frames) are denoted as V and compressed using existing coding schemes such as H.266 / VVC, H.265 / HEVC, etc. Here, the bitstream of the compressed V (i.e., the picture bitstream portion) is denoted as output2. Then, the reconstructed P frames are obtained, which will be denoted as Vrec as shown in FIG. 11.

[0326] Step 2: Determine the index of the layer to be updated (1120).

[0327] FIG. 13 illustrates a neural network having multiple layers, including layer (1310) and layer (1320). Since only some of the N layers will be updated, it must be decided which layers to update. In the example illustrated in FIG. 13, layer (1310) is updated, while layer (1320) is not updated. This means that layer (1320) maintains pre-configured parameters (e.g., weights). The layers to be updated can be configured on the encoder side based on analysis of experimental results or experience. For example, there may be experience where updating an intermediate layer is preferred over updating a preceding or subsequent layer. For example, in this implementation, N has 40 layers, and if only 4 layers can be updated (this can be adjusted according to actual scenarios considering training complexity and the total number of layers in the entire network), layers 20, 24, 26, and 27 can be updated. Furthermore, using Rate Distortion Optimization (RDO) principles, the layer to update can be determined as follows:

[0328] For Set1, the updated layers are Layers 20, 21, 22, and 23 or

[0329] For Set2, the updated layers are Layers 20, 22, 24, and 26 or

[0330] For Set3, the updated layers are Layers 20, 24, 26, and 27 or

[0331] …

[0332] Therefore, each Set i contains information about the updatable layer of the NN, and i is an integer.

[0333] Each Set i For this, the rate distortion cost Distortion + λ* Rate is evaluated as already mentioned above. Here, Distortion is Set iDistortion is the distortion of the filtered signal (the distance or difference between the filtered signal and the original signal) using , Rate is the number of bits in the bitstream (including the video portion, NN portion, and layerList as shown in Fig. 11), and λ is a coefficient called the Lagrangian multiplier. Distortion and Rate are known only in Step 3, and this applies to each Set i This means that Step 3 must be executed once for. The set with the minimum rate distortion cost. i is the optimal set. For example, if the rate distortion cost of Set3 is minimized, layers 20, 24, 26, and 27 are updated. Note that it is not necessary to perform the selection by applying RDO. Other optimizations that consider complexity, such as rate-distortion-complexity optimization, may be performed.

[0334] Here, the layer indices of the layers to be updated can be placed into a layer list called updateLayersList. This layerList corresponds to information about one or more updatable layers of the NN. For example, updateLayersList could be {20, 24, 26, 27}. If the layer indices change during training, updateLayersList must be sent to the decoder. However, if the indices of the layers to be updated are fixed, there is no need to send the information from the layer list to the decoder. This means that the decoder does not need to decode information about the updatable layers. Please note that the parameters of the aforementioned updatable layers (i.e., information about one or more updated parameters) can still change (update) during training and are therefore sent to the decoder. In this implementation, since the layer indices of the layers to be updated can change, parts of the updatable NN layers may change. Here, the stream of the layer list is denoted as output1.

[0335] Step 3: Derive and encode the information used to update the layers in updateLayersList (1170).

[0336] Since only the layers with layer indices in updateLayersList are updated, these layers are retrained, whereas other layers not in updateLayersList retain their respective weights. The weights of the layers with layer indices in updateLayersList will be compressed. In this implementation, an alternative method is proposed to represent the layers with layer indices in updateLayersList.

[0337] Step 3.1 Change the architecture of the layer with the layer index in updateLayersList (1150) and another neural network Gets.

[0338] In this sub-stage, the architecture of the layer with a layer index in updateLayersList is changed. This is illustrated in FIG. 17, where, in addition to the original part (1710) which is identical to the architecture of N layers, there is a so-called delta part (1720) for each layer with a layer index in updateLayersList. The original layer (1710) and the delta part (delta layer) each represent a sublayer of the updateable layer. In other words, formally, both sublayers belong to the common updateable layer. However, as can be seen in FIG. 17, the original part and the delta part have the same input and are arranged in parallel with each other. This parallel arrangement of the two sublayers is another neural network It can be considered as such, but still has the same architecture as N, except for layers with layer indices in updateLayersList. In this implementation example, all Layers with layer indices in updateLayersList i (1710) is 3 parts ( It changes to ). Here means the original part and of means the delta part. and of is of the existing neural network N It has the same input value as . (1730) - Other neural networks of and It refers to the elemental sum of. Other neural networks The Elementwise_sum layer of the existing neural network N The output is identical to . For example, Is It can be a Tensor Training Convolutional Layer (TTConv). Parameters ...was already explained in the definition section above with reference to Fig. 3. Generally The lower θ is, the smaller the size of TTConv, but the lower the performance. For example, the residual block of the existing neural network N corresponding to the residual block at N (shown in Fig. 14) may have the block method shown in Fig. 14. On the other hand, another neural network The block method of the residual block is illustrated in Fig. 15.

[0339] Step 3.2 Other Neural Networks Update the delta portion weights of the layer with the layer index in updateLayersList (1160).

[0340] Another neural network with no layers, where layer indices are not in updateLayersList The weights of the layers are initialized using the weights of the corresponding layers of the existing network N. Another neural network The original partial weights of the layer initialize the weights of the layer corresponding to the layer index in updateLayersList using the weights of the corresponding layer of the existing network N. Consequently, the delta portion weights of the layer corresponding to the layer index in updateLayersList are initialized using randomized weights. For example, as shown in Fig. 16, another neural network The weights of Conv1 (1610) and Conv2 (1620) of the layer with layer index in updateLayersList are set to the weights of the layer with layer index in updateLayersList in the existing neural network N, and the weights of the Conv1TT (1630) and Conv2TT (1640) layers are initialized randomly.

[0341] Another neural network after initialization In this, the weights of the delta portion of the layer with the layer index in updateLayersList are retrained. Here, another neural network In this case, only the weights of the delta portion of the layer with the layer index in updateLayersList are changed (i.e., updated) during retraining (i.e., neural network training), while the other weights of the original portion are maintained (i.e., not updated) during training. Then, V and V rec Using other neural networks Performs training of. The loss function M is M = Metric(another network(V rec It can be ),V). Here, "another network (Vrec)" uses another network to V rec It means filtering. Metric(A, B) refers to the distance between A and B using specific metrics such as the Peak Signal-to-Noise Ratio (PSNR), metric, MSSIM metric, VMAF video metric, or other image / video quality metrics. As shown in Fig. 11, V rec is a different neural network Since it is input into, another neural network (V rec )person So that the output of is as similar as possible to V This is trained. As mentioned earlier, in this implementation, only the delta portions of the layers with layer indices in updateLayersList are updated during training.

[0342] Step 3.3 is a different neural network Encodes the weights for the delta portions of the layers in updateLayersList.

[0343] The delta portion weights of the layers with layer indices in updateLayersList have already been updated, and the weights will be compressed. The compressed bitstream of the updated weights of the delta portion of the layer with the layer index in updateLayersList is represented as Output 3 (parameter bitstream portion), as illustrated in FIG. 11. For compression, any existing network compression method can be used. FIG. 18 illustrates an example of dictionary-based compression that can be performed as follows.

[0344] 1. Select σ - Select the starting number of clusters for subtensor clustering. Regarding σ, { It performs adaptive dictionary-based compression (Fig. 18). For example, for 4 TTConv layers, the optimal number of clusters is 64.

[0345] (1) , l from All tensors within} are divided into tensors (1810) of a predefined size mxn. For example, m=3,n=1. Then {sub_ Acquire )}. For example, The number of input and output channels is 48 and the rank is In the case of a TTConv layer, the size of conv1 is 1×1×48×16, the size of conv2 is 3×3×4×1, and the size of conv3 is 1×1×4×48. For the given parameters, conv1 is divided into 16x16=256 sub_layers, conv2 is divided into 3x4=12 sub_layers, and conv3 is divided into 4x16=64 sub_layers. Therefore, TTConv is divided into a total of 256+12+64=332 sub_layers. If the total number of layers is 4, the total number of sub_layers is 332*4=1328.

[0346] (2) sub_ For each tensor for R mxn space (e.g., R 3 It is considered as an element of the space. Clustering (e.g., k-means) is performed around σ. The result is the codebook C = { } 1820 , 0<=i< σ and (sub_ It is an index book (1820) having pairs of relevant code_book elements). For example, if the total number of clusters is 64 and the total number of sub_layers is 1328, then code book C has 64 elements and index_book has a total of 1328 pairs.

[0347] (3) Another network's Changes all tensors of to their respective codebook elements. For example, all subtensors of size 3x1 can be changed to one of 64 codebook elements.

[0348] (4) Measurement - V rec Measures the performance of other networks in. For example, am. represents the performance of the compressed NN.

[0349] (5) If == M, proceed to step 6; otherwise, set σ = 2*σ and proceed to step 2. If the performance of the compressed NN is similar to that of the uncompressed NN, the proposed compression parameter σ is sufficient to compress the NN without performance loss. If it is not sufficient, the number of clusters must be increased.

[0350] (6) Print the codebook and index_book.

[0351] 2. Convert the neural network into a compressed bitstream (1190, 1195).

[0352] Input: Codebook C = {c i}, 0 <i< σ 및 인덱스북 쌍 pairs (sub_ , relevant code_book element) σ - number of elements in the codebook, μ - number of elements in the indexbook. For example, assume σ=64, μ=1332.

[0353] (1) Calculate the number of bits k required to store one codebook element. For example, if the weight is represented in float32 format, 32 bits are required. Since the size of the subtensor is 3x1, 32x3=96 bits are required.

[0354] (2) Serialize the codebook into a bitstream of length k * σ. For example, if n=32 and σ=64, a total of 32*64=2048 bits are required.

[0355] (3) The index book μ * Serialize into bits. If the codebook has a total of 64 elements, all elements are Bits are required. Therefore, 1332 * 6 = 7992 bits are needed to store the index book.

[0356] (4) Put the 32-bit values ​​for σ and μ into the bitstream, then input the resulting bitstreams of (2) and (3).

[0357] (5) Apply lossless compression to the resulting bitstream (e.g., gzip). The resulting bitstream is output3.

[0358] In short, the bitstream will include three parts: a bitstream (1180) for output 2 of compressed data (such as compressed video), a bitstream (1195) for output 1 of the layer to be updated (the layer with a layer index in updateLayersList), and a bitstream (1190) for the compressed weight of the delta part of output 3 of the layer to be updated (the layer with a layer index in updateLayersList) (parameter bitstream part).

[0359] In the case of the decoder, the following processing shown in FIG. 12 is performed for this embodiment.

[0360] In the case of the decoder, the received bitstream will be decoded. Since the bitstream contains three parts (where the received parts 1210, 1220, and 1230 correspond to the transmitted parts 1180, 1190, and 1195, respectively), all of them will be decoded as described above. The three bitstream parts include a video bitstream part (1210) (Input 2), a parameter bitstream part (1220) (Input 3), and a layer bitstream part (1230) (Input 3). For readability, the output on the encoder side i is the input on the decoder side i Each is indicated as such. For example, the bitstream for the compressed video is indicated as Input 2 (video bitstream portion), the bitstream for the layer index of the layer to be updated (i.e., one or more updatable layers) (the layer with a layer index in updateLayersList) is indicated as Input 1, and the bitstream for the compressed weight (i.e., one or more updated parameters) of the delta portion of the layer to be updated (the layer with a layer index in updateLayersList corresponding to the updatable layer) is indicated as Input 3.

[0361] Step 1: Existing neural network N and decoded data V dec Acquires (1240).

[0362] The existing neural network N is identical to the encoder already present in the decoder. Therefore, an existing neural network refers to a neural network having layers that include one or more pre-configured layers and one or more updatable layers. For example, a neural network is a convolution layer and the weights W of these layers = It may have. For example, since the neural network is identical in the encoder, the existing neural network N may be a neural network of a classical residual neural network (Resnet) having a total of 20 residual blocks, where, as illustrated in FIG. 14, each block includes two convolutional layers (conv1 (1410), conv2 (1420)) of size 3×3×48×48, a ReLU layer (1430) after conv1 (1410), a skip connection (1440) from the input of the block to the output of conv2 (1420), and a ReLU layer (1460). In this example, N has a total of 40 convolutional layers Can have.

[0363] Conventional coding methods can be used to decode the picture bitstream from the compressed video Input2 and obtain the decoded video Vdec (i.e., one or more input pictures).

[0364] Step 2: Obtain the layer index of the layer to be updated (1250).

[0365] FIG. 13 illustrates a neural network having multiple layers, including layer (1310) and layer (1320). Since only a portion of the neural network N illustrated in FIG. 13 will be updated, it is necessary to obtain or know which layer of network N will be updated. In the example of FIG. 13, the layer to be updated is layer (1310), while layer (1320) is intended not to be changed. That is, layer (1320) retains the original parameters that can be pre-configured. In this implementation, the layer index of layer will be updated (i.e., changed), and in this case, the bitstream for the layer index of the layer to be updated will be decoded (i.e., the layer bitstream portion). Then, the layer index list updateLayersList will be decoded from the parameter bitstream portion containing information about the updated parameters for the updateable layer. For example, updateLayersList can be {20, 24, 26, 27}, in which case it is known that only layers 20, 24, 26, and 27 are updated in the existing neural network N.

[0366] Step 3: Decode the information used to update the layer with the layer index in updateLayersList (1250).

[0367] Since only the layers with layer indices in updateLayersList are updated, the weights for other layers (layers without layer indices in updateLayersList) remain the same as the existing neural network N. In other words, those weights are maintained.

[0368] Because the architecture of the layer with the layer index in updateLayersList in the encoder has changed (as illustrated in FIG. 17), only the weights of the delta portion (1720) of the layer with the layer index in updateLayersList are changed. As previously mentioned, the delta portion of the layer refers to the sublayer of the updateable layer with the original portion. As illustrated in FIG. 17, the original portion (1710) corresponds to a pre-configured layer. In the parameter bitstream portion, the compressed weights of the delta portion of the layer are decoded, and each weight of the delta portion is updated (the layer with the layer index in updateLayersList). Input 3: Here, the corresponding weights are obtained in a manner similar to that applied to the encoder using a corresponding network decompression method. FIG. 18 is an example of a compression method based on dictionary-based compression (decompression). In the above example, dictionary-based compression was used, and it is also used here, but the case of dictionary-based decompression is as follows:

[0369] (1) Uncompress Output 3.

[0370] (2) After lossless compression, apply decompression to decompress the bitstream.

[0371] (3) Using a known fixed structure of the bitstream, all Deserialize the bitstream into σ, μ, codebook, and indexbook. This means that input 1, input 2, and input 3 are arranged in a specific order in the bitstream.

[0372] (4) About Build.

[0373] Step 4: Another neural network , and (1260), Apply to Vdec (1270).

[0374] In the encoder, the architecture of the layers with layer indices in updateLayersList has been changed as shown in Fig. 17. In this implementation example, the original part (i.e., the sublayers of the updateable layers) is identical to the layer architecture of N, but for each layer with a layer index in updateLayersList, a delta part (i.e., the sublayers of the updateable layers) exists. In other words, all layers with layer indices in updateLayersList is 3 parts, that is ( It changes to ). Please note that the original part and the delta part belong to a single layer, namely the updatable layer. In Step 3 above indicates which layer of the existing network N is the target for update. In order to replace the layer with the layer index in updateLayersList in the existing neural network N, a new architecture such as that shown in Fig. 17 is obtained, and another neural network This is obtained. Technically, it should be noted that there is no actual replacement of the above layer. Rather, the decoder has already implemented an updatable layer that includes two sublayers: an updatable sublayer and a pre-configured sublayer.

[0375] In this example, the neural network is trained on a post-filter. In this case, as shown in Fig. 12, another neural network Obtain the output video by applying it to Vdec.

[0376] Since only a portion of the layers will be updated, the architecture of the other NNs in this implementation is changed to the existing neural network N, and each layer with a layer index in updateLayersList (i.e., an updatable layer) is configured in addition to the original part (pre-configured sublayers) and the delta part (updatable sublayers). In this implementation, lightweight convolutional layers based on Tensor Training Convolution (TTConv) layers are used. TTConv is an example of a sparse neural network layer. Such sparse layers provide a smaller amount of output data. In addition to TTConv, other lightweight convolutional layer schemes such as the following can also be used:

[0377] ■ MobileNet Block

[0378] ■ Inverted residual block

[0379] ■ Tucker decomposition

[0380] ■ Regular convolution using a smaller convolution kernel.

[0381] In this implementation, the information used to update the layer weights (i.e., information about one or more updatable parameters) is compressed using a dictionary-based method. In addition to dictionary-based schemes, other types of neural network compression, such as the following, can also be used:

[0382] ■ Inference Time Quantization

[0383] ■ Train-aware quantization

[0384] ■ Training Awareness Pruning

[0385] In the previous discussion regarding the implementation example, the updateable layer was composed of two sublayers: one updateable sublayer and one pre-configured sublayer. In this case, the neural network was trained on the encoder side using pre-configured weights for the pre-configured sublayer and weights for the updateable parameters. That is, the actual weight changes are updated using the pre-configured weights as a reference. This means that training is performed using the residuals of the weights. Thus, each sublayer is arranged in parallel as illustrated in FIG. 17 as an example of the original layer (1710) and the delta layer (1720). Thus, since the neural network of the decoder has the same architecture as the encoder, the decoder can perform updates of each of these layers according to the decoded information regarding the updated parameters and / or the updateable layer.

[0386] The following describes another implementation of encoding and decoding one or more input pictures, where training on the encoder side is performed by directly updating the weights. In this case, the updateable layer may be a single layer rather than being split into two sublayers.

[0387] For the encoder, the processing for each is as follows:

[0388] Step 1: Obtain the existing neural network N and the reconstructed data Vrec (1110).

[0389] The encoder processing is the same as the implementation example discussed earlier.

[0390] Step 2: Determine the index of the layer to be updated (information of the layer that can be updated) (1120).

[0391] Once again, the encoder processing is the same as the previously discussed implementation example.

[0392] Step 2: Derive and encode the information (updatable parameters) used to update the layers in updateLayersList (1170).

[0393] A different neural network model compared to the existing neural network model N In this case, since the architecture does not change, what changes and is updated during training are the weights of the layers with layer indices in updateLayersList (i.e., updatable layers).

[0394] All weights are initialized using the weights of network N, and the weights of network N are pre-configured. During training, the layer index is The weights of the layers in updateLayersList can change during retraining, but the layer indices The weights of layers not in updateLayersList are retained. In other words, the weights of pre-configured layers retain their respective weights received at initialization.

[0395] As mentioned earlier, the loss function M is M = Metric (another network( It can be ),V). Here, "another network( )" means filtering Vrec using a different network. Metric(A, B) refers to the distance between A and B using specific metrics such as Peak Signal-Noise Ratio, PSNR, metric, MSSIM metric, VMAF video metric, or other image / video quality metrics. As shown in Fig. 11, V rec is a different neural network Since it is entered into The output of "another network( )" is trained to be as similar as possible to V.

[0396] After obtaining the information used for the update, the information (i.e., information about the updated parameters) is compressed using an existing neural network compression method, such as dictionary-based compression (e.g., into the parameter bitstream portion (1190) of FIG. 11). This was discussed in detail above with reference to FIG. 18. Here, the information is (W T - W) and this is It refers to the weight difference value between the weights of the layers with layer indices in updateLayersList. In other words, the newly trained (retrained weight W T It does not necessarily need to be transmitted intact (however, in some embodiments, it is possible). Rather, the pre-training weight W and the newly trained weight W T The difference between them can be transmitted. Since this approach is more likely to have a smaller value range, fewer bits may be required to transmit the difference. Therefore, additional coding (such as entropy coding) can generate a shorter bitstream.

[0397] For the decoder, the processing for each is as follows:

[0398] Step 1: Obtain the existing neural network N and the decoded data Vdec (1240).

[0399] The decoder processing is the same as the implementation example described earlier.

[0400] Step 2: Obtain the layer index of the layer to be updated (1250).

[0401] Once again, the encoder processing is the same as the previously discussed implementation example.

[0402] Step 3: Decode the information used to update the layer with the layer index in updateLayersList (1250).

[0403] Since only layers with indices in updateLayersList are updated, the weights of layers not with indices in updateLayersList are preserved. In other words, the updateable parameters (weights) of the pre-configured layers are identical to those of the existing neural network N.

[0404] A corresponding neural network decompression method is used to obtain information about updatable parameters and / or updatable layers. Similar to an encoder, dictionary-based compression may be used. Here, the information is (W T - W) as, It represents the difference in weights between the layers with layer indices in updateLayersList in and N.

[0405] Step 4: Another neural network , and (1260), Apply to Vdec (1270).

[0406] Different neural network models compared to the implementation above In N, the neural network architecture does not change compared to the existing NN model N. Therefore, the information obtained in Step 3 is different from the neural network model From W to (W T - W) is added to calculate the weight of the layer index in updateLayersList.

[0407] This is a neural network model different in that trained weights are used. Specifies. Then, another neural network model Apply to Vdec.

[0408] The information used here to update the layers in updateLayersList is (W T - W) but this is It refers to the difference in weights between the weights of the layer with the layer index in N's updateLayersList and the weights of N. As another possible method, the information It could be the weight value of the layer with the layer index in updateLayersList, but it may not be the difference value. That is, W T It is compressed directly, and when decompressing the neural network, the decompressed weights are used for another NN model The layer indices in the updateLayersList are used directly as the weights of the layers. In the case of the post-filter application scenario, P frames (P >= 1) are compressed first in the encoder. For example, assuming a video of 1000 frames (f0, f1, …. f999), fi+1 to fi+P becomes P frames. Then, the bitstream for P frames is acquired, and the reconstructed P frames are acquired. Next, an existing network (typically pre-trained using a large dataset) must be trained using the original P frames and the reconstructed P frames. Then, the trained network model is compressed, and the bitstream of the network model is acquired (the network bitstream portion). Afterward, the entire bitstream portion is sent to the decoder. In the decoder, P frames are decompressed first, followed by the decompression of the network model. The network model is applied to the decompressed P frames.

[0409] In the following, we provide an example of a neural network trained on a post-filter for online training that can represent and compress the neural network in an efficient manner.

[0410] Taking a CNN filter (online training examples) as an example of a post-filter for video compression, the method is as follows:

[0411] Encoder side:

[0412] 1. Obtain input video V, existing neural network N, and pre-trained weights W of the neural network N layers.

[0413] 2. Compress the video using a video compression encoder. Reconstructed video V rec and acquire the video portion of the bitstream.

[0414] 3. LayerList Configuration - LayerList contains the indices of updatable layers, which are a subset of the convolutional layers of network N. For example, Layers = [ ] - If it is a list of all convolution layers of N, layerList is [ ] can be, and here am.

[0415] 4. Exclude such layers from layerList, and other neural networks Make this neural network N have the same architecture. Each layer in layerList i is 3 parts: ( It changes to ): another neural network's and Layer i is a layer of an existing neural network i It has an input similar to . Is and It is the elemental sum of. of other neural networks The layers are of existing neural networks. It has an output similar to .

[0416] 5. With an arbitrary value Initialize the weights.

[0417] 6. Other neural networks (i.e., updateable layers) class neural network, which is a sublayer including The input video V and the reconstructed video by updating only the layer weights Train about.

[0418] 7. Obtain the ΔLayer layer from another neural network and perform adaptive neural network compression using one of the neural network compression methods. Obtain the neural network portion from the bitstream (part of the network bitstream).

[0419] 8. Transmit the video portion of the bitstream, the neural network portion of the bitstream, and the layerList.

[0420] Decoder side:

[0421] 1. Decompress the video portion using a video compression decoder. V dec Acquire.

[0422] 2. Decompress the ΔLayerr layer using the NN decompression method.

[0423] 3. Another neural network with the same architecture as neural network N, excluding the layers whose indices are in layerList Creates. Each of layerList is 3 parts - ( It changes to ). Other neural networks of class is a layer of an existing neural network i It has an input similar to . Is and It is the elemental sum of. Other neural networks of The layers are of existing neural networks. It has an output similar to .

[0424] 4. Other neural networks Using V dec Performs inference in. Result video Acquires.

[0425] Each representation of the processing flow according to the present disclosure is illustrated in FIGS. 11 and FIGS. 12.

[0426] For the ΔLayer layer, a lightweight convolution operation—for example, a tensor training (TTConv) layer (3006) as shown in FIG. 3—is used.

[0427] The proposed method can minimize model transmission costs due to lightweight convolution and a small number of parameters in neural network compression.

[0428] Another advantage of this method is the possibility of adaptive neural network compression based on feedback from the inline decoder in the encoder.

[0429] Another advantage of this approach is the ability to flexibly select the following updateable parameters and neural network compression parameters:

[0430] - Parameter layerList that defines the convolution layer used for content adaptation

[0431] - Parameters of lightweight convolution, such as tensor decomposition ranks

[0432] - The following parameters for neural network adaptation frequencies:

[0433] ● Number of frames used per adaptation

[0434] ● Adaptation Policy

[0435] - Scene transition

[0436] - Required delay

[0437] Since compression speed can be one of the important factors in video compression, it is directly related to the online training time of neural network training. The embodiments and implementations of the present disclosure enable the minimization of online training time costs by minimizing the number of updated parameters—the number of updated parameters is small. Alternatively, the training time is flexible and can be adjusted according to the specific compression task.

[0438] The selection of optimal content adaptation parameters can be performed by trying several predefined sets of parameters and using the Rake Distortion Optimization (RDO) principle.

[0439] For example, Delta online training demonstrates stable compression savings compared to offline training scenarios.

[0440] The online training-based compression method according to the present invention can also be used in other video compression modules such as CNN loop filters or deep learning-based intra-prediction, deep learning-based inter-prediction, or deep learning-based adaptive sampling.

[0441] The above discussion regarding encoding and decoding has been made with reference to one or more pictures, said pictures being still pictures or videos. The pictures / videos are merely examples of data to which the specific embodiments and implementations described above may be applied. The device(s) and method(s) provided for online training may be applied to other data compression tasks, including audio compression and 3D data (e.g., point clouds, meshes) compression. Additionally, the data compression scheme may be used for video compression such as loop filters, post-filters, intra-prediction, inter-prediction, entropy coding, or other modules. Here, post-filters in video compression have been primarily described as representative application examples. However, the proposed online training-based data compression scheme may be applied to other modules or other tasks.

[0442] Those skilled in the art will understand that the “blocks” (“units”) of the various drawings (methods and apparatuses) represent or describe the functions of embodiments of the present invention (rather than necessarily individual “units” in hardware or software) and equally describe the functions or features of apparatus embodiments and method embodiments.

[0443] The term "unit" is used solely for the purpose of illustrating the function of the embodiments of the encoder / decoder and is not intended to limit the content of this disclosure.

[0444] It should be understood that in the various embodiments provided in this application, the disclosed systems, devices, and methods may be implemented in different ways. For example, the described device embodiments are merely illustrative. For example, unit partitioning is merely a logical functional partitioning, and may be a different distinction in actual implementation. For example, multiple units or components may be combined or integrated into different systems, or some functions may be ignored or not performed. Additionally, the mutual coupling, direct coupling, or communication connections shown or discussed may be implemented using some interfaces. Indirect connections or communication connections between devices or units may be implemented in electronic, optical, mechanical, or other forms.

[0445] Units described as separate parts may or may not be physically separated, and parts indicated as units may or may not be physical units, may be located in one place, or may be distributed as multiple network units. Some or all units may be selected according to the actual needs to achieve the purpose of the solution of the embodiment.

[0446] In addition, the functional units in the embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically independently, or two or more units may be integrated into a single unit.

[0447] The following describes some additional implementations of hardware and software.

[0448] The following embodiments of the video coding system (10), video encoder (20), and video decoder (30) are described based on FIGS. 28 and FIGS. 29 with reference to FIGS. 25 and FIGS. 26 described above.

[0449] FIG. 28 is a schematic block diagram illustrating an exemplary coding system (10), for example, a video coding system (10) (or simply a coding system (10)) capable of utilizing the technology of the present application. The video encoder (20) (or simply an encoder (20)) and video decoder (30) (or simply a decoder (30)) of the video coding system (10) represent examples of devices that may be configured to perform the technology according to various examples described in the present application.

[0450] As illustrated in FIG. 28, the coding system (10) includes a source device (12) configured to provide encoded picture data (21) to, for example, a destination device (14) for decoding encoded picture data (13).

[0451] The source device (12) includes an encoder (20) and additionally, i.e. optionally, may include a picture source (16), a preprocessor (or preprocessing unit) (18), such as a picture preprocessor (18), and a communication interface or communication unit (22).

[0452] The picture source (16) may be or include any kind of picture capture device, such as a camera for capturing a picture of the real world, for example, and / or any kind of picture generation device, such as a computer graphics processor for generating a computer-animated picture, for example, or any kind of other device that acquires and / or provides a picture of the real world, a computer-generated picture (e.g., screen content, a virtual reality (VR) picture) and / or a combination thereof (e.g., an augmented reality (AR) picture). The picture source may be any kind of memory or storage that stores any one of the aforementioned pictures.

[0453] By distinguishing between the preprocessor (18) and the processing performed by the preprocessor unit (18), the picture or picture data (17) may also be referred to as a raw picture or raw picture data (17).

[0454] The preprocessor (18) is configured to receive (raw) picture data (17) and perform preprocessing on the picture data (17) to obtain a preprocessed picture (19) or preprocessed picture data (19). The preprocessing performed by the preprocessor (18) may include, for example, trimming, color format conversion (e.g., from RGB to YCbCr), color correction, or de-noising. It may be understood that the preprocessing unit (18) may be an optional component.

[0455] The video encoder (20) is configured to receive preprocessed picture data (19) and provide encoded picture data (21) (further details are described above, for example, based on FIG. 25, which can be further modified by replacing the loop filter with a loop CNN filter, similar to what was done for the decoder in FIG. 27).

[0456] The communication interface (22) of the source device (12) may be configured to receive encoded picture data (21) and transmit the encoded picture data (21) (or any additionally processed version thereof) through the communication channel (13) to another device, such as a destination device (14) or any other device, for storage or direct reconstruction.

[0457] The destination device (14) includes a decoder (30) (e.g., a video decoder (30)) and additionally, i.e. optionally, may include a communication interface or communication unit (28), a post-processor (32) (or post-processing unit (32)) and a display device (34).

[0458] The communication interface (28) of the destination device (14) is configured to receive encoded picture data (21) (or any additionally processed version thereof) from, for example, directly from the source device (12) or from any other source, for example, a storage device (e.g., an encoded picture data storage device), and to provide the encoded picture data (21) to the decoder (30).

[0459] The communication interface (22) and the communication interface (28) may be configured to transmit or receive encoded picture data (21) or encoded data (13) through a direct communication link between the source device (12) and the destination device (14), such as a direct wired or wireless connection, or through any type of network, such as a wired or wireless network or a combination thereof, or any type of private and public network, or any type of combination thereof.

[0460] The communication interface (22) may be configured to package, for example, the encoded picture data (21) into a suitable format, for example, a packet, and / or process the encoded picture data using any kind of transmission encoding or processing for transmission through a communication link or communication network.

[0461] A communication interface (28) forming a counterpart to a communication interface (22) may be configured, for example, to receive transmitted data and process the transmitted data using any kind of corresponding transmission decoding or processing and / or de-packaging to obtain encoded picture data (21).

[0462] Both the communication interface (22) and the communication interface (28) may be configured as a unidirectional communication interface or a bidirectional communication interface, as indicated by the arrow for the communication channel (13) in FIG. 28 pointing from the source device (12) to the destination device (14), and may be configured to establish a connection to transmit and receive messages, for example, to acknowledge and exchange any other information related to the transmission of a communication link and / or data, for example, the transmission of encoded picture data.

[0463] The decoder (30) is configured to receive encoded picture data (21) and provide decoded picture data (31) or decoded picture (31) (more details are described above based on, for example, FIG. 26 or FIG. 27).

[0464] The post-processor (32) of the destination device (14) is configured to post-process decoded picture data (31) (also referred to as reconstructed picture data), e.g., decoded picture (31), to obtain post-processed picture data (33), e.g., post-processed picture (33). The post-processing performed by the post-processing unit (32) may include, e.g., color format conversion (e.g., from YCbCr to RGB), color correction, trimming or re-sampling, or any other processing to prepare the decoded picture data for display, e.g., by a display device (34).

[0465] The display device (34) of the destination device (14) is configured to receive post-processed picture data (33) to display a picture to, for example, a user or viewer. The display device (34) may be or include any type of display for displaying the reconstructed picture, such as an integrated or external display or monitor. The display may include, for example, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0466] FIG. 28 illustrates the source device (12) and the destination device (14) as separate devices, but embodiments of the device may also include both or both functions: the source device (12) or a corresponding function and the destination device (14) or a corresponding function. In these embodiments, the source device (12) or a corresponding function and the destination device (14) or a corresponding function may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0467] As is obvious to those skilled in the art based on the description, the presence and (precise) division of different units or functions within the source device (12) and / or destination device (14), as illustrated in FIG. 28, may vary depending on the actual device and application.

[0468] The encoder (20) (e.g., video encoder (20)) or decoder (30) (e.g., video decoder (30)) or both the encoder (20) and the decoder (30) may be implemented through a processing circuit as illustrated in FIG. 1b, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video coding, or any combination thereof. The encoder (20) may be implemented through a processing circuit (46) to implement various modules discussed in relation to the encoder (20) of FIG. 25 and / or any other encoder system or subsystem described herein. The decoder (30) may be implemented via a processing circuit (46) to implement various modules discussed in relation to the decoder (30) of FIG. 26 (or FIG. 27) and / or any other decoder system or subsystem described herein. The processing circuit may be configured to perform various operations as will be discussed later. As shown in FIG. 31, where the technology is partially implemented in software, the device may store instructions for software in a suitable, computer-readable, non-transient storage medium and execute instructions in hardware using one or more processors to perform the technology of the present disclosure. Either the video encoder (20) and the video decoder may be integrated into a single diorama as part of a combined encoder / decoder (CODEC), for example, as shown in FIG. 29.

[0469] The source device (12) and the destination device (14) may include any wide range of devices, such as any type of handheld or stationary device, such as a notebook or laptop computer, mobile phone, smartphone, tablet or tablet computer, camera, desktop computer, set-top box, television, display device, digital media platform, video gaming console, video streaming device (such as a content service server or content delivery server), broadcast receiver device, broadcast transmitter device, etc., or may not use an operating system or may use any operating system. In some cases, the source device (12) and the destination device (14) may be equipped for wireless communication. Thus, the source device (12) and the destination device (14) may be wireless communication devices.

[0470] In some cases, the video coding system (10) shown in FIG. 28 is merely an example, and the technology of the present application may be applied to a video coding setup (e.g., video encoding or video decoding) that does not necessarily involve any data communication between an encoding device and a decoding device. In other examples, data is retrieved from local memory and streamed over a network, etc. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding do not communicate with each other but are performed by a device that simply encodes data into memory and / or retrieves and decodes data from memory.

[0471] For convenience of explanation, embodiments of the present invention are described herein with reference to, for example, reference software for HEVC (High-Efficiency Video Coding) or VVC (Versatile Video Coding), next-generation video coding standards developed by the JCT-VCC (Joint Collaboration Team on Video Coding) of ITU-T VCEG, and ISO / IEC MPEG (Motion Picture Experts Group). Those skilled in the art will understand that embodiments of the present invention are not limited to HEVC or VVC.

[0472] FIG. 30 is a schematic diagram of a video coding device (400) according to one embodiment of the present disclosure. The video coding device (400) is suitable for implementing an embodiment as described herein. In one embodiment, the video coding device (400) may be a decoder such as the video decoder (30) of FIG. 28 or an encoder such as the video encoder (20) of FIG. 28.

[0473] A video coding device (400) includes an inlet port (410) (or input port (410)) and a receiver unit (Rx) (420) for receiving data; a processor, logic unit or central processing unit (CPU) (430) for processing data; a transmitter unit (Tx) (440) and an outlet port (450) (or output port (450)) for transmitting data; and a memory (460) for storing data. The video coding device (400) may also include an optical-to-electrical (OE) component and an electrical-to-optical (EO) component coupled to the inlet port (410), receiver unit (420), transmitter unit (440), and outlet port (450) for the entry or exit of an optical or electrical signal.

[0474] The processor (430) is implemented in hardware and software. The processor (430) may be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, DSPs, etc. The processor (430) communicates with an inlet port (410), a receiver unit (420), a transmitter unit (440), an outlet port (450), and memory (460). The processor (430) includes a coding module (470). The coding module (470) implements the aforementioned disclosed embodiments. For example, the coding module (470) implements, processes, prepares, or provides various coding operations. Thus, the inclusion of the coding module (470) provides substantial improvements to the functionality of the video coding device (400) and has the effect of transforming the video coding device (400) into a different state. Alternatively, the coding module (470) is implemented as instructions stored in memory (460) and executed by the processor (430).

[0475] Memory (460) may include one or more disks, tape drives and solid-state drives and may be used as an overflow data storage device to store such program when the program is selected for execution and to store instructions and data read during program execution. Memory (460) may be, for example, volatile and / or non-volatile and may be read-only memory (ROM), random access memory (RAM), Ternary Content-Addressable Memory (TCAM) and / or static random-access memory (SRAM).

[0476] FIG. 5 is a simplified block diagram of a device (500) that can be used as either or both of the source device (12) and the destination device (14) in FIG. 28 according to an exemplary embodiment.

[0477] The processor (502) of the device (500) may be a central processing unit. Alternatively, the processor (502) may be any other type of device or multiple devices capable of manipulating or processing information, which currently exist or will be developed in the future. The disclosed embodiment may be implemented with a single processor, e.g., the processor (502), as illustrated, but the advantages of speed and efficiency may be achieved by using one or more processors.

[0478] The memory (504) of the device (500) may be a read-only memory (ROM) element or a random access memory (RAM) element in implementation. Any other suitable type of storage device may be used as the memory (504). The memory (504) may contain code and data (506) that can be accessed by the processor (502) using the bus (512). The memory (504) may further include an operating system (508) and an application program (510), and the application program (510) may include at least one program that allows the processor (502) to perform the method described herein. For example, the application program (510) may include applications 1 through N that further include a video coding application that performs the method described herein, including encoding and decoding using a neural network having a subset of partially updateable layers.

[0479] The device (500) may also include one or more output devices, such as a display (518). In one example, the display (518) may be a touch-sensing display that combines a display with a touch-sensing element operable to detect touch input. The display (518) may be coupled to the processor (502) via a bus (512).

[0480] Although shown here as a single bus, the bus (512) of the device (500) may be composed of multiple buses. Additionally, the secondary storage (514) may be directly connected to other components of the device (500) or accessed via a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, the device (500) can be implemented in a wide variety of configurations.

[0481] Although embodiments of the present invention have been described primarily based on video coding, it should be noted that embodiments of the coding system (10), encoder (20) and decoder (30) (and corresponding systems (10)) and other embodiments described herein may also be configured for still picture processing or coding, that is, for the processing or coding of individual pictures independent of any preceding or consecutive pictures, as in video coding. Generally, when picture processing coding is limited to a single picture (17), only the inter-prediction units 244 (encoder) and 344 (decoder) may not be available. All other functions (also called tools or techniques) of the video encoder (20) and video decoder (30) can likewise be used for still picture processing, e.g., residual calculation (204 / 304), transformation (206), quantization (208), inverse quantization (210 / 310), (inverse) transformation (212 / 312), partitioning (262 / 362), intra prediction (254 / 354), and / or loop filtering (220, 320), and entropy coding (270) and entropy decoding (304).

[0482] For example, the function described herein with reference to embodiments of the encoder (20) and decoder (30), and for example, the encoder (20) and decoder (30), may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the function may be stored in a computer-readable medium or transmitted through a communication medium as one or more instructions or codes and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium corresponding to a medium of the same type as the data storage medium, or a communication medium including any medium that can easily transmit a computer program from one place to another according to a communication protocol, for example. In this way, the computer-readable medium may generally correspond to (1) a computer-readable storage medium of a non-transient type (tangible) or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes and / or data structures for the implementation of the technology described in this disclosure. Computer program products may include computer-readable media.

[0483] As examples, not to be limited, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices, or other magnetic storage devices, flash memory, or other media that can be used to store desired program code in the form of instructions or data structures accessible by a computer. Additionally, any connection may be considered a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, radio, and microwave is included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals, or other transitory media, but instead relate to non-transient, tangible storage media. The disk and disc used herein include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy discs, and Blu-ray discs; disks generally reproduce data magnetically, while discs reproduce data optically using lasers. The above combinations must also be included within the scope of computer-readable media.

[0484] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term "processor" as used herein may refer to any other structure suitable for the implementation of the aforementioned structure or the technology described herein. Furthermore, in some aspects, the functions described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or integrated into a combined codec. Additionally, the technology may be fully implemented in one or more circuits or logic elements.

[0485] The technology of the present disclosure may be implemented in a wide variety of devices or equipment, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). Various components, modules, or units are described in the present disclosure to highlight the functional aspects of a device configured to perform the disclosed technology, but are not necessarily required to be realized by other hardware units. Rather, as described above, various units may be combined in a codec hardware unit or provided by a set of interoperative hardware units comprising one or more processors as described above, together with appropriate software and / or firmware.

[0486] In summary, the present invention relates to the encoding and decoding of one or more pictures (e.g., video) using a neural network trained partially online. Accordingly, one or more layers to be trained in an encoder are selected. Then, training of the neural network is performed in which the parameters of the selected 4HLE layer are updated. The parameters of the remaining layers are maintained and not updated. The updated parameters are provided within the bitstream. The picture(s) are also encoded. The decoder receives instructions for the updated parameters, updates only the indicated parameters, and applies the neural network thus acquired.

Claims

Claim 1 An apparatus (2100) for encoding one or more input pictures (1010, 17), comprising using a neural network, comprising a processing circuit (2110), wherein the processing circuit selects one or more updatable layers (1310) among the layers (1310, 1320) of the neural network; trains the neural network based on the one or more input pictures, wherein training the neural network includes updating one or more parameters of the one or more updatable layers and maintaining parameters of one or more pre-configured layers (1320) among the layers of the neural network; encodes information regarding one or more updated parameters of the one or more updatable layers into a parameter portion (1190) of a bitstream; and is configured to encode the one or more input pictures into a picture portion (1180) of the bitstream, wherein the updatable layer (1310) has one or more updatable parameters and one or more pre-configured parameters; A device configured such that the processing circuit is configured to train the neural network by updating the one or more updatable parameters of the one or more updatable layers and maintaining the one or more pre-configured parameters. Claim 2 A device according to claim 1, wherein the processing circuit is configured to encode information about one or more updateable layers into a layer portion (1195) of the bitstream. Claim 3 A device according to paragraph 2, wherein information regarding one or more updatable layers includes a layer index of one or more updatable layers, and the layer index is an index of one or more updatable layers among the layers of the neural network indexed according to a preset rule. Claim 4 A device according to claim 1, wherein the updateable layer (1310, 1620, 1705) comprises an updateable sublayer (1526, 1626, 1720) in which the weight is one or more updateable parameters and a pre-configured sublayer (1524, 1624, 1710) in which the weight is one or more pre-configured parameters; the same input of the updateable layer is provided to the updateable sublayer and the pre-configured sublayer; and the output of the updateable layer is the element-wise sum (1730) of the updateable sublayer and the pre-configured sublayer. Claim 5 In claim 4, the device comprises, wherein the updateable sublayer includes a sparse neural network layer (3006, 4003, 5010). Claim 6 In claim 4, the device comprises, wherein the updateable sublayer includes a tensor train neural network layer (3006, 4003, 5010). Claim 7 A device according to claim 1, wherein the processing circuit is configured to update the one or more updateable parameters using a training method less complex than the training method used to train the one or more pre-configured parameters. Claim 8 A device according to claim 7, wherein the one or more updateable layers are convolutional layers (1003, 3001, 810, 850, 1410, 1420, 1620, 1705); and wherein at least one of the following is applied: - the less complex training process updates fewer parameters than the training method used to train the one or more pre-configured parameters; and - the less complex training updates a smaller convolution kernel than the training method used to train the one or more pre-configured parameters. Claim 9 In paragraph 3, the processing circuit updates the one or more updateable parameters of the one or more updateable layers (1626) by initializing them pseudo-randomly; and the device is configured to perform training with pairs of original pictures (910) and reconstructed pictures (960). Claim 10 An apparatus according to any one of claims 1 to 3, wherein the processing circuit is configured to train the neural network by updating the one or more parameters of the one or more updateable layers based on a pair of original picture (910) and reconstructed picture (960). Claim 11 A device according to any one of claims 1 to 3, wherein the one or more parameters of the one or more updateable layers include weights of the one or more updateable layers. Claim 12 An apparatus according to any one of claims 1 to 3, wherein the processing circuit further preconfigures the neural network by pre-training, including updating the parameters of the one or more pre-configured layers less frequently than the training of the one or more updateable layers based on the one or more input pictures; and is configured to encode information about the updated parameters of the one or more pre-configured layers into the network portion of the bitstream. Claim 13 In paragraph 12, the processing circuit is further configured to acquire a plurality of pre-configured layer sets; to select a set from the plurality of pre-configured layer sets that includes one or more pre-configured layers of the neural network; and information regarding the updated parameters is an indication of the selected set. Claim 14 A device according to any one of claims 1 to 3, wherein the selection and / or updating of the layer is performed at different intervals. Claim 15 A device according to any one of claims 1 to 3, wherein information regarding one or more updateable layers is encoded less frequently than information regarding one or more updated parameters of one or more updateable layers. Claim 16 In any one of claims 1 to 3, the training is based on a pair of first number input pictures and reconstructed pictures; the encoding of information for one or more updated parameters is performed after the encoding of the first number input pictures, and the trained neural network is applied to a second number reconstructed picture; and the second number is larger than the first number, device. Claim 17 A device according to any one of claims 1 to 3, wherein the one or more pre-configured layers (1320) and / or the one or more updateable layers (1310) of the neural network are convolutional layers (1410, 1420). Claim 18 A device according to any one of claims 1 to 3, wherein the processing circuit is configured to select one or more updateable layers based on the optimization of a cost function comprising at least one of rate, distortion, and complexity. Claim 19 A device according to any one of claims 1 to 3, wherein the neural network is a neural network trained for post-filtering (2702). Claim 20 A device according to any one of claims 1 to 3, wherein the encoding of one or more pictures is performed using the neural network. Claim 21 In paragraph 20, the neural network is trained to perform one of in-loop filtering (2701), end-to-end picture coding, adaptive sampling, and coding of picture coding parameters. Claim 22 An apparatus (2200) for decoding one or more input pictures (1060, 331), comprising using a neural network, wherein the apparatus comprises a processing circuit (2210), wherein the processing circuit decodes information regarding one or more parameters of one or more updatable layers (1310) from a parameter portion (1220) of a bitstream; wherein the bitstream is configured to decode one or more input pictures, wherein the neural network comprises: - the one or more updatable layers (1310) comprising one or more decoded parameters; and - one or more pre-configured layers (1320) having pre-configured parameters, wherein the updatable layers (1310) have one or more updatable parameters and one or more pre-configured parameters; and wherein the processing circuit is configured to update the one or more updatable parameters of the one or more updatable layers and maintain the one or more pre-configured parameters. Claim 23 In paragraph 22, the device is configured such that the processing circuit is configured to decode information about one or more updatable layers from the layer portion (1230) of the bitstream. Claim 24 In paragraph 22, the information for the one or more updateable layers includes a layer index of the one or more updateable layers; the layer index is an index of the one or more updateable layers among the layers of the neural network indexed according to a preset rule, the device. Claim 25 In claim 22, the device comprises an updateable layer (1310, 1620, 1705) which includes an updateable sublayer (1526, 1626, 1720) in which the weight is one or more updateable parameters and a pre-configured sublayer (1524, 1624, 1710) in which the weight is one or more pre-configured parameters; the same input of the updateable layer is provided to the updateable sublayer and the pre-configured sublayer; and the output of the updateable layer is the element-wise sum (1730) of the updateable sublayer and the pre-configured sublayer. Claim 26 In paragraph 25, the device wherein the updateable sublayer comprises sparse neural network layers (3006, 4003, 5010). Claim 27 In claim 25, the device comprises, wherein the updateable sublayer includes a tensor training neural network layer (3006, 4003, 5010). Claim 28 A device according to claim 25, wherein the one or more updateable layers are convolutional layers (1003, 3001, 810, 850, 1410, 1420, 1620, 1705); and wherein at least one of the following is applied: - the updateable sublayer has fewer parameters than the pre-configured sublayer; and - the updateable sublayer applies a convolutional kernel smaller than the pre-configured sublayer. Claim 29 A device according to any one of claims 22 to 24, wherein the one or more parameters of the one or more updateable layers include the weights of the one or more updateable layers. Claim 30 A device according to any one of claims 22 to 24, wherein the processing circuit further decodes information about an updated parameter of one or more pre-configured layers from a network portion of the bitstream; and is configured to update the pre-configured parameters less frequently than the updateable parameters based on the information about the updated parameters. Claim 31 In paragraph 30, the processing circuit is further configured to obtain instructions for a selected set from a plurality of pre-configured layer sets from information regarding the updated parameters, and the selected set includes one or more pre-configured layers of the neural network, the device. Claim 32 A device according to any one of claims 22 to 24, wherein information regarding one or more updateable layers is decoded less frequently than information regarding one or more updated parameters of one or more updateable layers. Claim 33 A device according to any one of claims 22 to 24, wherein the one or more pre-configured layers (1320) and / or the one or more updateable layers (1310) of the neural network are convolutional layers (1410, 1420). Claim 34 In any one of paragraphs 22 to 24, the neural network is a device applied to post-filtering (2702). Claim 35 A device according to any one of claims 22 to 24, wherein the decoding of one or more pictures is performed using the neural network. Claim 36 In paragraph 35, the neural network is applied to in-loop filtering (2701), picture decoding, adaptive sampling, and decoding of picture coding parameters. Claim 37 A method for encoding one or more input pictures (1010, 17), comprising using a neural network, the method comprising: a step (S2310) of selecting one or more updatable layers (1310) among the layers (1310, 1320) of the neural network; a step (S2320) of training the neural network, including updating one or more parameters of the one or more updatable layers and maintaining parameters of one or more pre-configured layers (1320) among the layers of the neural network based on the one or more input pictures; a step (S2340) of encoding information about one or more updated parameters of the one or more updatable layers into a parameter portion (1190) of a bitstream; and a step (S2350) of encoding the one or more input pictures into a picture portion (1180) of the bitstream, wherein the updatable layer (1310) has one or more updatable parameters and one or more pre-configured parameters; A method of training the neural network comprising updating one or more updatable parameters of one or more updatable layers and maintaining one or more pre-configured parameters. Claim 38 A method for decoding one or more input pictures (1060, 331), comprising using a neural network, comprising: a step (S2420) of decoding information about one or more parameters of one or more updatable layers (1310) from a parameter portion (1220) of a bitstream; and a step (S2430) of decoding one or more input pictures, comprising processing from the picture portion (1210) of the bitstream with the neural network, wherein the neural network comprises: - the one or more updatable layers (1310) comprising one or more decoded parameters; and - one or more pre-configured layers (1320) having pre-configured parameters, wherein the updatable layers (1310) have one or more updatable parameters and one or more pre-configured parameters; and wherein the neural network is trained by updating the one or more updatable parameters of the one or more updatable layers and maintaining the one or more pre-configured parameters. Claim 39 A computer-readable, non-transient storage medium storing a program, wherein the program comprises instructions that cause one or more processors to perform a method according to any one of claims 37 and 38 when the program is executed on one or more processors. Claim 40 An apparatus for encoding one or more input pictures, comprising using a neural network, comprising: one or more processors; and a computer-readable non-transient storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the program is configured such that the apparatus performs the method according to claim 37 when the program is executed by the one or more processors. Claim 41 An apparatus for decoding one or more input pictures, comprising using a neural network, comprising one or more processors; and a computer-readable non-transient storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the program is configured such that the apparatus performs the method according to claim 38 when the program is executed by the one or more processors. Claim 42 A computer program stored on a computer-readable non-transient storage medium, wherein the computer program includes program code for performing a method according to either claim 37 or 38 when executed on a computer. Claim 43 delete Claim 44 delete

Images