Method or apparatus for implementing low complexity neural network-based processing

Abstract

At least one method and apparatus is presented for generating an output data tensor by applying a neural network-based process to an input data tensor, thereby efficiently encoding or decoding a video. For example, quantization of the tensor is limited to be scaled by a power of 2. For example, tensor build-up, bias addition, and activation are fused to reduce the number of operations and increase the number of digits available to represent the value.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of European application number 22305731.6 filed on May 18, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] At least one of the present embodiments generally relates to a method or apparatus for video encoding or decoding, and more specifically, to a method or apparatus for applying low-complexity neural network-based processing to an input data tensor to generate an output data tensor. Background Art

[0004] To achieve high compression efficiency, image and video coding schemes typically use prediction (including motion vector prediction) and transforms to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlations, and then the difference between the original image and the predicted image (usually expressed as a prediction error or prediction residual) is transformed, quantized, and entropy encoded. To reconstruct the video, the compressed data is decoded through the inverse process corresponding to entropy encoding, quantization, transform, and prediction.

[0005] Recently explored high compression techniques have added neural network based processing. The disadvantages of such neural network based processing are that the processing may not be reproducible, the processing is complex (due to the number of operations or the nature of the operations themselves), and the amount of data to be stored is huge. Therefore, it is necessary to provide an implementation of a neural network that allows fully reproducible processing and optimizes memory efficiency and computing power. Therefore, there is a need to improve the existing technology. Summary of the invention

[0006] The shortcomings and deficiencies of the prior art are addressed and overcome by the general aspects described herein.

[0007] According to a first aspect, a method is provided. The method comprises: obtaining an input data tensor representing a data sample; and applying a neural network-based processing to the input data tensor to generate an output data tensor. According to a specific feature, the neural network-based processing comprises a plurality of processing layers, wherein each processing layer generates an intermediate tensor. At least one processing layer is represented as a tensor product between an input data tensor and a weight tensor, and at least one processing layer is represented as an addition of a bias tensor. Advantageously, the scaling factor of any of the quantized representations of the tensors (e.g., the input data tensor, the weight tensor, the bias tensor, the intermediate tensor, and the output data tensor) uses a power of 2.

[0008] According to another aspect, a method is provided. The method includes performing video decoding by applying a neural network-based process according to any of the disclosed embodiments to an input data tensor to generate an output data tensor, wherein data samples of the input data tensor include at least image block samples.

[0009] According to another aspect, a method is provided. The method includes performing video encoding by applying a neural network-based process according to any of the disclosed embodiments to an input data tensor to generate an output data tensor, wherein data samples of the input data tensor include at least image block samples.

[0010] According to another aspect, an apparatus is provided. The apparatus comprises one or more processors, wherein the one or more processors are configured to implement a method for video decoding according to any variant thereof. According to another aspect, an apparatus for video decoding comprises means for applying a neural network-based process to an input data tensor according to any disclosed embodiment to generate an output data tensor.

[0011] According to another aspect, another apparatus is provided. The apparatus includes one or more processors, wherein the one or more processors are configured to implement a method for video encoding according to any variant thereof. According to another aspect, an apparatus for video encoding includes means for applying a neural network-based process to an input data tensor according to any disclosed embodiment to generate an output data tensor.

[0012] According to another general aspect of at least one embodiment, an apparatus is provided, comprising: an apparatus according to any decoding embodiment; and at least one of: (i) an antenna configured to receive a signal, the signal comprising a video block; (ii) a band limiter configured to limit the received signal to a frequency band comprising the video block; or (iii) a display configured to display an output representing the video block.

[0013] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided, comprising data content generated according to any of the described encoding embodiments or variations.

[0014] According to another general aspect of at least one embodiment, a signal is provided comprising video data generated according to any of the encoding embodiments or variants described.

[0015] According to another general aspect of at least one embodiment, a bitstream is formatted to contain data content generated according to any of the described encoding embodiments or variations.

[0016] According to another general aspect of at least one embodiment, a computer program product is provided, comprising instructions, wherein the instructions, when executed by a computer, cause the computer to perform any of the encoding / decoding embodiments or variations.

[0017] These and other aspects, features and advantages of the general aspects will become apparent from the following detailed description of illustrative embodiments which is to be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Several examples of embodiments are shown in the drawings.

[0019] Figure 1 Block diagram illustrating an example apparatus in which aspects of the embodiments may be implemented.

[0020] Figure 2 A block diagram illustrating an embodiment of a video encoder in which aspects of the embodiments may be implemented.

[0021] Figure 3 A block diagram illustrating an embodiment of a video decoder in which aspects of the embodiments may be implemented.

[0022] Figure 4 A block-based pipeline for neural network processing in a video encoder / decoder is shown that can implement aspects of the embodiments.

[0023] Figure 5 A block diagram illustrating an embodiment of a layered neural network architecture in which aspects of the embodiments may be implemented.

[0024] Figure 6 A block diagram illustrating a layered neural network architecture using low-complexity quantization in accordance with general aspects of at least one embodiment.

[0025] Figure 7 A block diagram illustrating an embodiment of a layered neural network architecture using low complexity quantization in fused convolutional and bias layers.

[0026] Figure 8 Block diagram illustrating layered neural network training and layered neural network training using low-complexity quantization, transformed to perform quantization-aware training / fine-tuning examples in accordance with general aspects of at least one embodiment.

[0027] Fig. 9 A block diagram illustrating an embodiment of transforming a layered neural network architecture to perform quantization-aware training.

[0028] Fig.10 A general decoding method according to a general aspect of at least one embodiment is shown.

[0029] Fig.11 A general encoding method according to a general aspect of at least one embodiment is shown. DETAILED DESCRIPTION

[0030] Several embodiments relate to video coding systems, wherein in at least one embodiment, it is proposed to adapt a video coding tool to low-complexity neural network processing. Different embodiments are proposed below, introducing some tool modifications to reduce codec complexity when implementing neural network processing in the tool (such as the non-limiting example of tool prediction or post-filtering). In particular, encoding methods, decoding methods, encoding devices, decoding devices based on this principle are proposed.

[0031] In addition, although the present aspect describes principles related to a specific draft of the VVC (Versatile Video Coding) or HEVC (High Efficiency Video Coding) specification or ECM (Enhanced Compression Model) reference software, it is not limited to VVC or HEVC or ECM, and can be applied to, for example, other standards and recommendations (whether existing or developed in the future) and extensions of any such standards and recommendations (including VVC and HEVC and ECM). Unless otherwise specified or technically excluded, the aspects described in this application can be used alone or in combination.

[0032] The acronyms used in this article reflect the current state of video coding development and should therefore be considered examples of nomenclature that may be renamed at a later date but still represent the same technology.

[0033] Figure 1 A block diagram of a system example in which various aspects and embodiments can be implemented is shown. System 100 can be embodied as a device including various components described below, and is configured to perform one or more aspects described in this application. Examples of such devices include, but are not limited to, a variety of electronic devices, such as personal computers, notebook computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances and servers. The elements of system 100 can be embodied in a single integrated circuit, multiple ICs and / or discrete components, either alone or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed over multiple ICs and / or discrete components. In various embodiments, system 100 is coupled to other systems or other electronic devices in a communication manner, such as by a communication bus or by a dedicated input and / or output port. In various embodiments, system 100 is configured to implement one or more aspects described in this application.

[0034] The system 100 includes at least one processor 110 configured to execute instructions loaded therein for implementing, for example, the various aspects described in the present application. The processor 110 may include embedded memory, input-output interfaces, and various other circuits known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). The system 100 includes a storage device 140, which may include a non-volatile memory and / or a volatile memory, including but not limited to an EEPROM, a ROM, a PROM, a RAM, a DRAM, an SRAM, a flash memory, a magnetic disk drive, and / or an optical disk drive. As a non-limiting example, the storage device 140 may include an internal storage device, an additional storage device, and / or a network accessible storage device.

[0035] The system 100 includes an encoder / decoder module 130, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents a module that may be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 130 may be implemented as a separate element of the system 100, or may be incorporated into the processor 110 as a combination of hardware and software known to those of ordinary skill in the art.

[0036] Program code loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described in the present application may be stored in the storage device 140 and subsequently loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of a variety of items during the execution of the processes described in the present application. These stored items may include, but are not limited to, input video, decoded video or a portion of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0037] In several embodiments, memory inside the processor 110 and / or the encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be a memory 120 and / or a storage device 140, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory (e.g., RAM) is used as working memory for video encoding and decoding operations (e.g., HEVC or VVC).

[0038] Input to the elements of system 100 may be provided through a variety of input devices as shown in block 105. Such input devices include, but are not limited to: (i) an RF section that receives an RF signal transmitted wirelessly, such as by a broadcaster, (ii) a composite input, (iii) a USB input, and / or (iv) an HDMI input.

[0039] In various embodiments, the input device of block 105 has associated input processing elements known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or bandwidth limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again bandwidth limiting to a narrower frequency band to select (for example) a signal frequency band (which may be referred to as a channel in some embodiments), (iv) demodulating the down-converted and bandwidth-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired packet stream. The RF section of various embodiments includes one or more elements (e.g., frequency selectors, signal selectors, bandwidth limiters, channel selectors, filters, down-converters, demodulators, error correctors, and demultiplexers) to perform these functions. The RF section may include a tuner that performs multiple of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and filter to the desired frequency band again by filtering, down-conversion and perform frequency selection. Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements that perform similar or different functions. Adding element can include inserting element between existing element, for example inserting amplifier and analog-to-digital converter. In various embodiments, the RF part includes antenna.

[0040] In addition, the USB and / or HDMI terminals may include respective interface processors for connecting the system 100 to other electronic devices via USB and / or HDMI connections. It should be appreciated that aspects of input processing (e.g., Reed-Solomon error correction) may be implemented within a separate input processing IC or within the processor 110 as desired. Similarly, aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within the processor 110 as desired. The demodulated, error corrected, and demultiplexed streams are provided to a variety of processing elements, including, for example, the processor 110 and the encoder / decoder 130, which operate in conjunction with memory and storage elements to process the data streams as desired for presentation on an output device.

[0041] The multiple elements of system 100 may be disposed within an integrated housing. Within the integrated housing, the multiple elements may be interconnected and transfer data between each other using suitable connection means 115 (eg, an internal bus known in the art, including an I2C bus, wiring, and printed circuit boards).

[0042] The system 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to send and receive data via the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or a network card, and the communication channel 190 may be implemented, for example, within a wired and / or wireless medium.

[0043] In various embodiments, data is streamed to the system 100 using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signals of these embodiments are received via a communication channel 190 and a communication interface 150 suitable for Wi-Fi communications. The communication channel 190 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet to allow streaming applications and other over-the-top communications. Other embodiments provide streaming data to the system 100 using a set-top box that transmits data via an HDMI connection of an input block 105. Other embodiments provide streaming data to the system 100 using an RF connection of an input block 105.

[0044] The system 100 can provide output signals to a variety of output devices, including a display 165, a speaker 175, and other peripherals 185. In various examples of embodiments, the other peripherals 185 include one or more of a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide functions based on the output of the system 100. In various embodiments, control signals are transmitted between the system 100 and the display 165, the speaker 175, or other peripherals 185 using signaling such as AV.Link, CEC, or other communication protocols that implement device-to-device control with or without user intervention. The output devices can be communicatively coupled to the system 100 via dedicated connections through their respective interfaces 160, 170, and 180. Alternatively, the output devices can be connected to the system 100 using a communication channel 190 via the communication interface 150. The display 165 and the speaker 175 can be integrated into a single unit with other components of the system 100 in an electronic device (e.g., a television). In various embodiments, the display interface 160 includes a display driver, such as a timing controller (T Con) chip.

[0045] Alternatively, the display 165 and the speaker 175 may be separate from one or more other components, such as if the RF portion of the input 105 is part of a separate set-top box. In various embodiments where the display 165 and the speaker 175 are external components, the output signals may be provided through dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0046] Figure 2 An exemplary video encoder 200 is shown, such as a VVC (Versatile Video Coding) encoder. Figure 2 It is also possible to show an encoder that improves the VVC standard or an encoder that uses a technique similar to VVC.

[0047] In this application, the terms "reconstruction" and "decoding" can be used interchangeably, the terms "encoded" or "coded" can be used interchangeably, and the terms "image", "picture" and "frame" can be used interchangeably. Usually, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0048] Prior to encoding, the video sequence may undergo pre-encoding processing (201), such as applying a color transform to an input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of input picture components to obtain a signal distribution more amenable to compression (e.g., using histogram equalization on one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0049] In encoder 200, a picture is encoded by encoder elements, as described below. The picture to be encoded is segmented (202) and processed in units, such as Cus. Each unit is encoded using, for example, intra or inter mode. When the unit is encoded in intra mode, it performs intra prediction (260). In inter mode, motion estimation (275) and compensation (270) are performed. The encoder decides (205) which of intra mode or inter mode to use to encode the unit and indicates the intra / inter decision by, for example, a prediction mode flag. For example, a prediction residual is calculated by subtracting (210) the predicted block from the original image block.

[0050] The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy encoded (245) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass the transform and quantization, i.e., directly encode the residual without applying the transform or quantization process.

[0051] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are inverse quantized (240) and inverse transformed (250) to decode the prediction residual. The image block is reconstructed by combining (255) the decoded prediction residual and the prediction block. A loop filter (265) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (280).

[0052] Figure 3 300. In the decoder 300, the bitstream is decoded by the decoder elements as described below. The video decoder 300 generally performs the same Figure 2 The decoding process is the reverse of the encoding process described in . The encoder 200 also typically performs video decoding as part of encoding the video data.

[0053] Specifically, the input to the decoder includes a video bitstream, which may be generated by the video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, motion vectors, and other encoding information. The picture segmentation information indicates how the picture is segmented. Therefore, the decoder can divide the picture according to the decoded picture segmentation information (335). The transform coefficients are inverse quantized (340) and inverse transformed (350) to decode the prediction residual. The image block is reconstructed by combining (355) the decoded prediction residual and the prediction block. The prediction block can be obtained (370) by intra-frame prediction (360) or motion compensated prediction (i.e., inter-frame prediction) (375). A loop filter (365) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (380).

[0054] The decoded picture may be further subjected to post-decoding processing (385), such as an inverse color transform (e.g., from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (201). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0055] Among the recently explored video coding solutions, neural network based processing has been proposed, for example to provide a post filtering stage or to provide block prediction.

[0056] Figure 4 A block-based pipeline for neural network processing in a video encoder / decoder is shown in which aspects of the embodiments may be implemented. The picture to be encoded ( Figure 4 The original frame in ) is split and divided into units ( Figure 4 NN processing is applied to blocks of a picture, wherein the picture data is fed to the NN as an input vector and the resulting processed block is output from the NN as an output vector and, for example, stored for further encoding processing. Advantageously, the input data is not limited to picture samples, but may convey any information / statistics related to one or more blocks of a picture, such as, as non-limiting examples, coding modes, quantization parameters, motion information. Since the video decoding process typically performs a decoding process that is the reverse of the encoding process, Figure 4 The NN processing applied to the picture block during decoding is also shown. In the context of video coding, strict constraints need to be imposed on the processing (including the NN processing):

[0057] - Reasoning should be fully reproducible, so all weights and operations should use integer arithmetic;

[0058] - Complexity should be low: therefore, it is better to limit the number of operations, limit complex operations (division, multiplication), and avoid certain operations (such as square roots, etc.);

[0059] - Memory usage should be low: therefore it is better to have quantized values ​​on a limited number of bits.

[0060] like Figure 4 As shown, NN processing consists of multiple levels. Each level learns to transform its input data into a slightly more abstract and complex representation. In video coding applications, the raw input may be pixels / samples of a block; and the output is a processed block, such as a predictor or a filtered block according to the above non-limiting examples. The output of the level is represented using the network. Inference represents the process of providing input data to the network and applying each layer to generate output.

[0061] To meet video coding constraints, three common methods used in general deep learning frameworks are now described.

[0062] According to a first embodiment, dynamic range quantization is used, where the weights w of the model are quantized on N bits (typically 8 bits). Quantization is modeled with a scaling factor and a zero point (or offset) according to the following formula:

[0063] W = clip (round (a * w + f))

[0064] Where W is the quantized integer value of the weight w in floating point form,

[0065] a is the scaling factor,

[0066] f is the zero point or offset,

[0067] round() is a function that selects the nearest integer.

[0068] clip() is a function that sets a value within an integer representation range (eg, [-128, 127] for 8 bits). The integer value representation range is also referred to as bit depth or representation type in the following.

[0069] However, in this embodiment, the weights are transformed back to floating point representation during inference and the calculations are done in floating point form.

[0070] According to a second embodiment, full integerization is used, where both weights and intermediate results are quantized and represented as integers. All operations use integer arithmetic. In this case, additional parameters are defined that specify the scaling and offset (or zero point) of the intermediate results (or tensors).

[0071] According to a third embodiment, quantization-aware training is implemented. In addition to the representation and computation type, quantization constraints are also taken into account during the training itself. It allows the reduction in precision of parameters or tensors to be directly taken into account during training.

[0072] Figure 5A block diagram illustrating an embodiment of a layered neural network architecture in which aspects of the embodiments may be implemented. Figure 5 The simple exemplary hierarchical neural network NN includes 3 layers, namely, convolutional layer 510, bias layer 520 and activation layer 530 (here ReLU). However, the present principle is not limited to NN with 3 layers, but can be easily extended to NN modeled as one or more linear layers (matrix product and bias) and one or more non-linear layers (activation functions, such as Relu, Gelu, Sigmoid...). Figure 5 The parameters a and f involved in each layer of the NN model of the quantized NN are also shown. Parameter a represents the scaling factor, while parameter b represents the zero point of any tensor applicable to the quantized NN model (i.e., weight tensor, bias tensor, and input / output tensor X, Y, T of each layer). All parameters a and f are known in advance. Figure 5 The intermediate results or tensors Y, T of the quantized NN model are also shown. Figure 5 The implementation of still raises issues such as regarding complexity, as described below.

[0073] First, adding zeros increases the number of operations to be performed during inference. For example, using a simple fully connected layer, implying matrix multiplication:

[0074] t j =∑ i x i w ij +b j

[0075] Where xi is the input tensor, w ij is the value of the weight tensor, b j is the bias. For clarity, we only explicitly specify the indices i and j that are of interest to the model, and represent the tensor as a single-dimensional x i ,y j ,t j However, the present principles are not limited to single-dimensional tensors, and one of ordinary skill in the art will clearly generalize the disclosed principles to multi-dimensional tensors, such as 3D tensors (2D space and depth). Therefore, considering the above notation, after quantization, each term is recorded as:

[0076] x i =a x (X i -f x )

[0077] w ij =a w (W ij -f w )

[0078] b j =ab (B j -f b )

[0079] in:

[0080] -a x 、a w and a b are the scaling factors for input x, weight w, and bias b respectively;

[0081] -f x 、f w and f b are the offsets of input x, weight w and bias b respectively;

[0082] -X i , W ij and B j is the floating point value x i 、w ij and b j Quantified value of .

[0083] Therefore, the integer version can be obtained as:

[0084] t j =∑ i (a x (X i -f x ) w (W ij -f w ))+a b (B j -f b )

[0085] Can be simplified to:

[0086] t j =a x a w ∑ i (X i W ij )-a x a w f w ∑ i X i +B′ j

[0087] Where B' j is a term that can be calculated offline because it depends only on the model parameters (a x 、f x , W ij 、f w 、a b , B j and fb ).

[0088] In order to perform all operations using only integer arithmetic, additional information about the rescaling of the results is required. Denoting the scaling factor as s t Advantageously, the scaling factor may be a power of 2 so as to be performed using a bit shift operation.

[0089] T j =clip(s t ∑ i (X i W ij )-s t f w ∑ i X i )+B″ j

[0090] The relationship is as follows:

[0091] t j =a t (T j -f t )

[0092] The clipping operation ensures that the result is contained within the representation used for the intermediate results. It can be noted that the zero point introduces the term ∑X i The bias term is also adjusted to take into account the internal scaling s t and some potential bias in the results.

[0093] In fact, the above equation requires the following steps to calculate:

[0094] -Calculate the cumulative product X i W ij ,

[0095] - Then press factor s t Rescale,

[0096] -according to Calculate the scaling factor a t ,

[0097] -Calculate the cumulative product X i ,

[0098] - Then press factor s t f w Rescale,

[0099] - The above two accumulated products and terms B"j are accumulated and clipped to be within the range of the underlying type, and f t Depends on B" j , and B" jDepends only on known parameters.

[0100] Second, in most approaches, the bit depth used for weight and tensor representation is 8 bits because it targets general-purpose architectures such as CPU, GPU, or TPU. However, if specialized hardware is used, the bit depth of weights and intermediate calculation results can have arbitrary bit depth.

[0101] Third, in most methods, the scaling factor is arbitrary and requires integer multiplication to compute the scaling factor of the output. In the general case, division may also be required to adjust the scaling of the layer output.

[0102] Fourth, the representation does not take into account the nature of the operations in the model. For example, the output of the activation layer uses the same representation (scaling, offset) regardless of the activation.

[0103] These problems are addressed and overcome by the general aspects described herein, which are directed to representations that take into account the following constraints:

[0104] -Minimize the number of operations;

[0105] - Simplify certain operations, usually replacing multiplication and division with bit-shift operations;

[0106] - Consider the nature of the operation;

[0107] - Reduce the number of parameters representing the model.

[0108] In the following, the present principle is applied in the same way to matrix product (dense layer, fully connected layer) or convolution-based layers. However, it is described for the matrix product layer.

[0109] According to a first embodiment, a low complexity quantization is disclosed, where the scaling factor is a power of 2. In practice, to minimize complexity, the quantization is restricted to scaling by powers of 2. It allows the quantized multiplication and division to be performed using bit shifts. Furthermore, the quantization also involves a null zero point, so no additional operations are performed on the quantization offset.

[0110] Figure 6 A block diagram of an embodiment of a layered neural network architecture using low complexity quantization is shown. Figure 5 A specific variant of the first embodiment of the above exemplary NN implementation, using the same notation, we obtain:

[0111] t j =∑x i w ij +b j

[0112] in:

[0113]

[0114]

[0115]

[0116] where q x ,q w ,q b is a known integer in advance.

[0117]

[0118] In fact, the above equation requires the following steps to calculate:

[0119] -Calculate the cumulative product X i W ij

[0120] -Press q y -(q x +q w ) to shift the bits and clip the result. The quantizer of the intermediate result is q y -(q x +q w )

[0121] - Press q for the result b -(q y -(q x +q w ) to shift

[0122] - Add bias And crop the final result

[0123] -Then the quantizer of the output tensor is q t =q b

[0124] Advantageously, the number of parameters controlling precision and bit depth is reduced. We assume that the bias layer drives the quantization of the input and output of the activation layer. All multiplication / division operations for quantization are advantageously replaced by shifts (multiplication / division by powers of 2). No additional operations are required to implement zeros.

[0125] The number of operations can be further reduced by fusing specific variants of the convolution / matrix multiplication layer with the bias layer.

[0126] Figure 7 A block diagram showing an embodiment of a layered neural network architecture using low complexity quantization in fused convolutional and bias layers. Thus, the scaling factors of the intermediate tensors Y and T are equal to q y =q b .

[0127] Assuming the convolution / matrix multiplication and bias layers are fused, the above equation can be further simplified:

[0128]

[0129] In fact, the steps to calculate the value of tensor T are:

[0130] - Accumulate the sum ∑X in the intermediate variable H i W ij ;

[0131] - Use H′=H>>((q x +q w )-q b ) to shift the variable H so that H uses q b Quantification;

[0132] - Add bias where the scaling factor of Z is 2 b Power (2^q b );

[0133] - Clipping is done to saturate the values ​​at the bit depth of the underlying representation.

[0134] One of ordinary skill in the art will appreciate that a right shift using a negative value is considered equivalent to a left shift.

[0135] Splits the processing of the sum of partial products according to a specific variant. This variant is particularly useful for input tensors with very large bit depths. In fact, when the input tensors have very large bit depths, intermediate calculations may overflow the underlying type.

[0136] According to a non-limiting example of this variant, Split into two complementary, non-overlapping parts Ω1 and Ω2 as follows:

[0137]

[0138] The processing steps can be described as:

[0139] - Accumulate the sum ∑X in the intermediate variable H1 using only index i in Ω1 i W ij , and accumulate the sum ∑X in the intermediate variable H2 for index i in Ω2 i W ij . Assume Ω=Ω1∪Ω2.

[0140] -Each intermediate variable H1 and H2 is bit-shifted and clipped:

[0141] For example, H1′=clip(H1>>((q y -(q x+q w ))) makes it possible to use q y Quantitative H1

[0142] For example, H2′=clip(H2>>((q y -(q x +q w ))), so that q is also used y Quantifying H2

[0143] -Add intermediate variables and clip the result: H=clip(H1+H2).

[0144] Advantageously, the same principle is used to split the accumulation into N stages to avoid overflow.

[0145] According to a second embodiment, the activation operation is also fused into the convolution / matrix multiplication. Figure 8 Block diagram showing an embodiment of a layered neural network architecture using fused activation operations. In practice, to further optimize quantization, the activation operations are fused in convolution / matrix multiplications. According to a non-limiting variant, the activation layer is for example a ReLU. In the context of neural networks, a corrector or ReLU (rectified linear unit) activation function is an activation function defined as the positive part of its argument x:

[0146] f(x)=max(0,x).

[0147] Therefore, in this case the output is known to be positive. Advantageously, the underlying representation of the output is transformed to avoid the sign bit. Figure 8 An example showing such processing using the associated bit depth:

[0148] - The input tensor X is assumed to be positive (so no sign bit is needed). This is the case for both the model input and the intermediate tensors after activation when using ReLU;

[0149] -Weights use 16-bit (15 bits + 1 sign bit) underlying type;

[0150] - The intermediate result Y uses a 32-bit (31-bit + 1-bit sign) underlying type;

[0151] - add them with a 17-bit (16 bits + 1 sign bit) bias;

[0152] -activation clips and shifts the result to obtain an intermediate tensor using 16 bits (unsigned bits).

[0153] In this example, it shows that since the output of the activation layer is positive, the 1 sign bit is avoided for the intermediate results (between each set of convolution + bias + activation layers). The same principle applies to other types of layers (Dense layers, etc.) where the activation outputs only positive values.

[0154] According to a third embodiment, a quantization-aware training is disclosed, wherein the training phase also generates a quantization parameter q for each layer. In order to obtain the quantization parameter q for each layer, several schemes can be adopted:

[0155] - Offline quantization: Each parameter q used for the weights is found offline by inspecting the results on a small representative dataset.

[0156] -Perform quantization-aware training: each layer is replaced with a quantized version of its weights.

[0157] Fig. 9 A block diagram illustrating an embodiment of transforming a layered neural network architecture to perform quantization-aware training. Fig. 9 The top model is replaced by the bottom model. Therefore, for each weight, a quantization layer Q and an inverse quantization layer Q are inserted. -1 , both layers use the parameter q. All computations are still done in floating point. Since the quantization operation is not differentiable, a proxy for quantization is used, typically using, for example, STE (straight-through estimator), uniform noise, quantization function approximation, etc. The output of the multiplication or convolution is then also quantized / inverse quantized using the same method.

[0158] Additional Examples and Information

[0159] Fig.10 A general decoding method (300) is shown in accordance with a general aspect of at least one embodiment. Fig.10 The block diagram partially represents the modules of the decoder or the decoding method (for example, in Figure 3 embodied in an exemplary decoder of ).

[0160] The method includes applying a neural network-based process (1020) to an input data tensor to generate an output data tensor, wherein the input data includes at least samples of an image block. Advantageously, inference of the neural network-based process uses any of the disclosed features to reduce the complexity of the neural network-based process. The NN-processed block (output data) is then encoded (1020) according to any of the variants described herein.

[0161] Fig.11 A general encoding method (200) is shown in accordance with a general aspect of at least one embodiment. Fig.11 The block diagram partially represents the modules of the encoder or the encoding method (for example, in Figure 2The method includes applying a neural network-based process (1120) to an input data tensor to generate an output data tensor, wherein the input data includes at least samples of an image block. Advantageously, the inference of the neural network-based process uses any of the disclosed features to reduce the complexity of the neural network-based process. The NN-processed block (output data) is then encoded (1120) according to any of the variants described herein.

[0162] Various methods are described herein, and each method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified or combined. In addition, in various embodiments, terms such as "first", "second" and the like may be used to modify elements, components, steps, operations, etc., such as "first decoding" and "second decoding". Unless specifically required, the use of such terms does not mean that the operation of modification is sorted. Therefore, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or in a time period overlapping with the second decoding.

[0163] Various methods and other aspects described in this application can be used to modify modules, as non-limiting examples, such as Figure 2 and Figure 3 The segmentation module, intra prediction module (202, 260, 335, 360) of the video encoder 200 and decoder 300 shown. In addition, the present aspect is not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations and any extensions of such standards and recommendations. Unless otherwise specified or technically not allowed, the aspects described in this application can be used alone or in combination.

[0164] Various numerical values ​​are used in this application. The specific values ​​are only for illustrative purposes, and the aspects are not limited to these specific values.

[0165] Various embodiments relate to decoding. "Decoding" as used in this application may encompass, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or to generally refer to a broader decoding process will be clear based on the context of the specific description, and is believed to be well understood by those of ordinary skill in the art.

[0166] Various embodiments relate to encoding.Similar to the above discussion on "decoding", "encoding" as used in this application may encompass all or part of the process performed on an input video sequence to produce an encoded bitstream, for example.

[0167] It should be noted that the grammatical elements used in this article are descriptive terms. Therefore, they do not exclude the use of other grammatical element names.

[0168] The embodiments and aspects described herein may be implemented as, for example, a variety of information, such as syntax, that may be transmitted or stored. The information may be packaged or arranged in a variety of ways, including, for example, those commonly found in video standards, such as placing the information in an SPS, PPS, NAL unit, header (e.g., a NAL unit header or a slice header), or SEI message. Other approaches may also be used, including, for example, those commonly found in system-level or application-level standards, such as placing the information in one or more of the following:

[0169] SDP (Session Description Protocol): A format for describing multimedia communication sessions, used for session announcements and session invitations, e.g. described in RFCs and used in conjunction with RTP (Real-time Transport Protocol) transport;

[0170] DASH MPD (Media Presentation Description) descriptor: used, for example, in DASH and transmitted over HTTP, a descriptor is associated with a representation or a set of representations to provide additional characteristics to the content representation;

[0171] RTP header extension: used, for example, during RTP streaming;

[0172] ISO Base Media File Format: This is used and uses boxes, such as in OMAF, which are object-oriented building blocks defined by a unique type identifier and a length, also called "atoms" in some specifications;

[0173] HLS (HTTP Live Streaming) manifests delivered over HTTP. A manifest can be associated with a version or set of versions of the content to provide characteristics of the version or set of versions.

[0174] The embodiments and aspects described herein can be implemented, for example, in a method or process, a device, a software program, a data stream or a signal. Even if only discussed in the context of a single embodiment (e.g., discussed only as a method), the implementation of the discussed features can also be implemented in other forms (e.g., a device or program). The device can be implemented in appropriate hardware, software and firmware, for example. The method can be implemented in a device such as a processor, which generally refers to a processing device, such as a computer, a microprocessor, an integrated circuit or a programmable logic device. The processor also includes a communication device, such as a computer, a mobile phone, a portable / personal digital assistant ("PDA") and other devices that facilitate information communication between end users.

[0175] References to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations thereof mean that the particular features, structures, characteristics, etc. described in connection with the embodiment are included in at least one embodiment. Therefore, the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations appearing in multiple places in the present application do not necessarily all refer to the same embodiment.

[0176] Furthermore, the present application may involve “determining” a variety of information. Determining information may include one or more of the following, such as estimating information, calculating information, predicting information, or retrieving information from memory.

[0177] Furthermore, the present application may involve "accessing" various information. Accessing information may include one or more of the following, such as receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0178] Furthermore, the present application may involve "receiving" a variety of information. Like "accessing," receiving is intended to be a broad term. Receiving information may include one or more of the following, such as accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" generally involves in some way during an operation such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0179] It should be understood that the use of any of the following " / ", "and / or", and "at least one", such as in the case of "A / B", "A and / or B", and "at least one of A and B", is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to as many items as listed as will be apparent to one of ordinary skill in this and related arts.

[0180] In addition, as used herein, the term "signal" refers to, among other things, indicating something to a corresponding decoder. For example, in some embodiments, the encoder signals a quantization matrix for inverse quantization. Thus, in an embodiment, the same parameters are used on both the encoder side and the decoder side. Thus, for example, the encoder can transmit (explicitly signal) specific parameters to the decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has the specific parameters as well as other parameters, signaling can be performed without transmission (implicit signaling) to simply allow the decoder to know and select the specific parameters. Bit savings are achieved in various embodiments by avoiding transmission of any actual functionality. It should be understood that signaling can be accomplished in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to the corresponding decoder. Although the verb form of the word "signal" is mentioned above, the word "signal" can also be used as a noun here.

[0181] As will be appreciated by those of ordinary skill in the art, embodiments may generate a variety of formatted signals to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for executing a method or data generated by one of the embodiments. For example, a signal may be formatted to carry a bitstream of the embodiment. Such a signal may, for example, be formatted as an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0182] We have described many embodiments. Features of these embodiments may be provided individually or in any combination to cover various claim categories and types. In addition, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination, in various claim categories and types:

[0183] Tuning neural network inference in the decoder and / or encoder.

[0184] A method, process, apparatus, medium storing instructions, medium storing data or signal according to any of the embodiments described.

[0185] A television, set-top box, mobile phone, tablet or other electronic device that executes a neural network process tuned to low complexity according to any of the above embodiments.

[0186] A television, set-top box, mobile phone, tablet or other electronic device that performs a neural network process adjusted to a lower complexity according to any of the described embodiments and displays (e.g., using a monitor, screen or other type of display) the resulting image.

[0187] A television, set-top box, mobile phone, tablet or other electronic device that selects (e.g., using a tuner) a channel to receive a signal including an encoded image and executes a neural network process adapted to low complexity according to any of the described embodiments.

[0188] A television, set-top box, mobile phone, tablet or other electronic device that wirelessly receives (e.g., using an antenna) a signal containing an encoded image and executes a neural network process adapted to a low complexity according to any of the described embodiments.

Claims

1. A computer-implemented method comprising: Get the input data tensor (X) representing the data sample; as well as applying a neural network based processing to the input data tensor to generate an output data tensor (Z), wherein the neural network based processing comprises a plurality of processing layers (501, 502, 503), wherein each processing layer generates an intermediate tensor (Y, T), wherein at least one processing layer is represented as a tensor product between the input data tensor and a weight tensor (W), and wherein at least one processing layer is represented as an addition of a bias tensor (B); and The scaling factor of the quantized representation of the input data tensor, the scaling factor of the quantized representation of the weight tensor, the scaling factor of the quantized representation of the bias tensor, the scaling factor of the quantized representation of the intermediate tensor, and the scaling factor of the quantized representation of the output data tensor use powers of 2.

2. An apparatus comprising a memory and one or more processors, wherein the one or more processors are configured to: Get an input data tensor (X) representing a data sample; and applying a neural network based processing to the input data tensor to generate an output data tensor (Z), wherein the neural network-based processing comprises a plurality of processing layers, wherein each processing layer generates an intermediate tensor (Y, T), wherein at least one processing layer is represented as a tensor product between the input data tensor and a weight tensor (W), and wherein at least one processing layer is represented as an addition of a bias tensor (B); and The scaling factor of the quantized representation of the input data tensor, the scaling factor of the quantized representation of the weight tensor, the scaling factor of the quantized representation of the bias tensor, the scaling factor of the quantized representation of the intermediate tensor and the scaling factor of the quantized representation of the output data tensor use powers of 2.

3. The method according to claim 1 or the device according to claim 2, wherein the quantized representation of the tensor is obtained by a shift of the scaling factor according to a power of 2.

4. A method according to any one of claims 1 and 3 or an apparatus according to any one of claims 2 and 3, wherein the offset parameters of the quantized representation of the input data tensor, the offset parameters of the quantized representation of the weight tensor, the offset parameters of the quantized representation of the bias tensor, the offset parameters of the quantized representation of the intermediate tensor, and the offset parameters of the quantized representation of the output data tensor are equal to zero.

5. The method according to any one of claims 1, 3 and 4 or the apparatus according to any one of claims 2-4, wherein the at least one processing layer representing the addition of the bias tensors is fused with the at least one processing layer representing the tensor product.

6. The method according to claim 5 or the apparatus according to claim 5, wherein the intermediate tensor (T) as a result of the addition of the fused tensor product and the bias tensor is obtained by: The sum of the partial products of the quantized representation of the input tensor and the quantized representation of the weight tensor (∑X i W ij ) are accumulated in the intermediate variables; shifting the intermediate variable using the scaling factor of the quantized representation of the input data tensor, the scaling factor of the quantized representation of the weight tensor, and the scaling factor of the quantized representation of the intermediate tensor; as well as A result of adding the bias tensor and the shifted intermediate variable is clipped to obtain an intermediate tensor at a bit depth of a quantized representation of the intermediate tensor.

7. The method according to claim 6 or the apparatus according to claim 6, wherein the sum of partial products (∑X i W ij ) to avoid overflow.

8. A method according to any one of claims 5-7 or an apparatus according to any one of claims 5-7, wherein at least one processing layer includes an activation layer, which is fused with at least one processing layer representing a fused tensor product and a bias tensor addition.

9. The method according to claim 8 or the device according to claim 8, wherein the input data tensor (X) is positive and does not use any sign bit in representation; wherein the output data tensor (Z) is positive and is represented without any sign bit; and Wherein the activation layer crops and shifts at least one processing layer representing the tensor product plus a bias to generate an intermediate tensor at the bit depth of the input data tensor.

10. A computer-implemented method comprising decoding an image block, wherein the decoding comprises applying a neural network-based processing according to any of claims 1 and 3-9 to an input data tensor to generate an output data tensor, and wherein the data samples comprise at least image block samples. The method according to claim 10 , wherein the data sample further comprises other information related to the image block.

12. A computer-implemented method comprising encoding an image patch, wherein the encoding comprises applying a neural network-based processing to an input data tensor according to any one of claims 1, 3-9 to generate an output data tensor, and wherein the data samples comprise at least image patch samples. The method according to claim 12 , wherein the data sample further comprises other information related to the image block.

14. An apparatus comprising a memory and one or more processors, wherein the one or more processors are configured to decode an image block by applying a neural network-based processing to an input data tensor according to any one of claims 1 and 3-9 to generate an output data tensor, and wherein the data samples include at least image block samples.

15. An apparatus comprising a memory and one or more processors, wherein the one or more processors are configured to encode image blocks to generate output data tensors by applying a neural network-based process to an input data tensor according to any one of claims 1 and 3-9, and wherein the data samples include at least image block samples.

16. A non-transitory program storage device readable by a computer and tangibly embodying a program of instructions executable by the computer for performing the method of any one of claims 1 and 3-9.

17. A computer-implemented training method comprising: Get the input data tensor (X) representing the image patch samples; as well as applying a neural network based training process to the input data tensor to generate an output data tensor (Z) representing compressed image block samples, wherein the neural network-based training process comprises a plurality of processing layers, wherein each processing layer generates an intermediate tensor, wherein at least one processing layer is represented as a tensor product between the input data tensor and a weight tensor (W), and wherein at least one processing layer is represented as an addition of a bias tensor; The quantization processing layer (Q) and the inverse quantization processing layer (Q -1 ) generating a quantized representation of the weight tensor for use in the neural network-based training process; The quantization processing layer (Q) and the inverse quantization processing layer (Q -1 ) generating a quantized representation of the bias tensor for use in the neural network-based training process; The quantization processing layer (Q) and the inverse quantization processing layer (Q -1 ) generating a quantized representation of the intermediate tensor for use in the neural network-based training process; and The neural network-based training process further uses powers of 2 to generate scaling factors for the quantized representation of the input data tensor, scaling factors for the quantized representation of the weight tensor (q w ), a scaling factor (q b ), the scaling factor of the quantized representation of the intermediate tensor (q b ) and a scaling factor for the quantized representation of the output data tensor.

18. An apparatus comprising a memory and one or more processors, wherein the one or more processors are configured to: Obtain an input data tensor (X) representing image patch samples; and applying a neural network based training process to the input data tensor to generate an output data tensor (Z) representing compressed image block samples, wherein the neural network-based training process comprises a plurality of processing layers, wherein each processing layer generates an intermediate tensor, wherein at least one processing layer is represented as a tensor product between the input data tensor and a weight tensor (W), and wherein at least one processing layer is represented as an addition of a bias tensor; The quantization processing layer (Q) and the inverse quantization processing layer (Q -1 ) generating a quantized representation of the weight tensor for use in the neural network-based training process; The quantization processing layer (Q) and the inverse quantization processing layer (Q -1 ) generating a quantized representation of the bias tensor for use in the neural network-based training process; The quantization processing layer (Q) and the inverse quantization processing layer (Q -1 ) generating a quantized representation of the intermediate tensor for use in the neural network-based training process; and The neural network-based training process further uses powers of 2 to generate scaling factors for the quantized representation of the input data tensor, scaling factors for the quantized representation of the weight tensor (q w ), a scaling factor (q b ), the scaling factor (q b ) and a scaling factor for the quantized representation of the output data tensor.

19. A trained machine learning model trained according to the method of claim 17.

Images