Parallel processing of image domains using neural networks: decoding, post-filtering, and RDOQ

Abstract

The present disclosure relates to picture encoding and decoding of tile-based image regions. In particular, a plurality of components of an input tensor including first and second components in a spatial dimension are processed within a plurality of pipelines. Processing of the first component includes dividing the first component in the spatial dimension into a first plurality of tiles. Similarly, processing of the second component includes dividing the second component in the spatial dimension into a second plurality of tiles. At that time, each of the first and second pluralities of tiles is processed separately. Among the first and second pluralities of tiles, there are at least two co-located tiles each having a different size. In the case of compression, processing of the first and / or second component includes picture encoding, rate-distortion optimized quantization, and picture filtering. In the case of decompression, the processing includes picture decoding and picture filtering.

Description

[Technical Field]

[0001] Embodiments of this disclosure generally relate to the field of encoding and decoding pictures or videos, and more particularly to encoding and decoding neural network-based bitstreams. [Background technology]

[0002] Video coding (video encoding and decoding) is used in a wide range of digital video applications, such as broadcast digital TV, video transmission over the internet and mobile networks, real-time conversation applications like video chat and video conferencing, DVD and Blu-ray® discs, video content acquisition and editing systems, and camcorders in security applications.

[0003] Even relatively short videos can require a considerable amount of video data to depict, which can pose a challenge when the data needs to be streamed or otherwise transmitted over a communication network with limited bandwidth. Therefore, video data is generally compressed before being transmitted over modern telecommunications networks. Video size can also be a problem when the video is stored on a storage device, as memory resources may be limited. Video compression devices often use software and / or hardware at the source to encode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received at the destination by a video decompression device that decodes the video data. Given limited network resources and the increasing demand for higher video quality, improved compression and decompression techniques that improve the compression ratio with little to no sacrifice of image quality are desirable.

[0004] Neural network (NN) and deep learning (DL) technologies, which utilize artificial neural networks, have been used for some time in the fields of encoding and decoding video, images (for example, still images), and the like.

[0005] It is desirable to further improve the efficiency of such picture coding (video-picture coding or still image coding) based on a trained network (e.g., a neural network NN) that takes into account the limitations of the available memory and / or processing speed of the decoder and / or encoder. [Overview of the project] [Problems that the invention aims to solve]

[0006] Some embodiments of this disclosure provide methods and apparatus for encoding and / or decoding pictures in an efficient manner, thereby reducing the memory footprint and the required operating frequency of the processing unit. In particular, this disclosure enables a trade-off between memory resources and computational complexity within a NN-based video / picture encoding-decoding framework applicable to both moving and still images. [Means for solving the problem]

[0007] The above and other objectives are achieved by the subject matter of the independent claims. Further implementations are evident from the dependent claims, specification, and drawings.

[0008] According to one aspect of this disclosure, a method is provided for processing an input tensor representing picture data, the method comprising: processing a plurality of components of the input tensor, comprising a first component and a second component in a spatial dimension, wherein the processing comprises processing a first component, comprising dividing the first component in a spatial dimension into a first plurality of tiles and processing the tiles of the first plurality of tiles separately; and processing a second component, comprising dividing the second component in a spatial dimension into a second plurality of tiles and processing the tiles of the second plurality of tiles separately, wherein at least two of each of the first plurality of tiles and the second plurality of tiles are of different sizes. As a result, the input tensor representing picture data can be efficiently processed on a component basis by using the tiles in a sample-aligned manner within multiple pipelines. Thus, memory requirements are reduced, while processing performance (e.g., compression and decompression) is improved without increasing computational complexity.

[0009] In some exemplary implementations, at least two of the first set of tiles are processed independently or in parallel; and / or at least two of the second set of tiles are processed independently or in parallel. Thus, the components of the input tensor can be processed quickly, and processing efficiency can be improved.

[0010] In further implementations, the first component represents the lumen component of the picture data, and the second component represents the chroma component of the picture data. Therefore, both the lumen and chroma components can be processed through multiple pipelines within the same processing framework.

[0011] In one example, adjacent tiles of the first set of tiles partially overlap in at least one spatial dimension; and / or adjacent tiles of the second set of tiles partially overlap in at least one spatial dimension. Thus, the quality of the reconstructed picture can be improved, particularly along the tile boundaries. Thus, picture artifacts can be reduced.

[0012] According to one implementation, the division of the first component includes determining the size of tiles in a first set of tiles based on a first predetermined condition, and / or the division of the second component includes determining the size of tiles in a second set of tiles based on a second predetermined condition. For example, the first predetermined condition and / or the second predetermined condition are based on available decoder hardware resources and / or motion present in the picture data. Thus, the tile size can be adapted and optimized according to the available decoder resources and / or motion, enabling content-based tile sizing. In a further example, determining the size of tiles in a second set of tiles includes scaling the tiles in the first set of tiles. As a result, the tile size of the second set of tiles can be determined quickly, improving the efficiency of tile processing.

[0013] In an exemplary implementation, the determined size indications for tiles within a first set of tiles and / or a second set of tiles are encoded in the bitstream. Thus, the tile size indication is efficiently included in the bitstream, and the processing required is low-level.

[0014] In an alternative implementation, all tiles in a first set of tiles are the same size, and / or all tiles in a second set of tiles are the same size. As a result, tiles can be processed efficiently without additional handling for different tile sizes, potentially accelerating tile processing.

[0015] In the second example, the instruction further includes the location of the tile within the first plurality of tiles and / or within the second plurality of tiles.

[0016] In one implementation, the first component is the rumor component, and the indication of the tile size for the first set of tiles is included in the bitstream; the second component is the chroma component, and the indication of the scaling factor is included in the bitstream, which correlates the tile size for the first set of tiles with the tile size for the second set of tiles. Thus, the tile size of the chroma component can be quickly obtained by a fast operation that scales the tile size of the rumor component. Furthermore, the overhead for signaling the tile size for the chroma can be reduced by using the scaling factor as an indication.

[0017] In an exemplary implementation, processing of the input tensor includes processing that is part of picture or video compression. For example, processing of the first and / or second component includes one of the following: picture encoding by a neural network, rate distortion optimization quantization (RDOQ), and picture filtering. Thus, the compression process can be carried out in a flexible manner, including various types of processing (encoding, RDOQ, filtering).

[0018] A further exemplary implementation involves generating a bitstream by including the output of processing the first and second components into the bitstream. Thus, the processing output can be quickly included in the bitstream, and the required processing is low-level.

[0019] In exemplary implementations, processing of the input tensor includes processing that is part of the decompression of a picture or video. For example, processing of the first and / or second component includes one of picture decoding and picture filtering by a neural network. Thus, decompression processing can be performed in a flexible manner, including various types of processing (encoding, filtering). For example, processing of the second component includes decoding the chroma component of the picture based on a representation of the luma component of the picture. Thus, the luma component can be used as auxiliary information for decoding the chroma component. This can improve the quality of the decoded chroma. In further examples, processing of the first and / or second component includes picture post-filtering; for at least two tiles of a first plurality of tiles, one or more post-filtering parameters differ and are extracted from the bitstream; and for at least two tiles of a second plurality of tiles, one or more post-filtering parameters differ and are extracted from the bitstream. Thus, filter parameters can be efficiently signaled via the bitstream. Furthermore, post-filtering is performed using filter parameters adapted to the tile size to improve the quality of the reconstructed picture data.

[0020] In the exemplary implementation, the input tensor is a picture or sequence of pictures containing one or more components from a set of components, at least one of which is a color component.

[0021] According to one aspect of this disclosure, a computer program stored on a non-temporary medium is provided, which, when executed on one or more processors, includes code that performs any of the steps of the aforementioned aspects of this disclosure.

[0022] According to one aspect of the present disclosure, an apparatus is provided for processing an input tensor representing picture data, the apparatus having a processing circuit configured to process a plurality of components of the input tensor, including a first component and a second component in a spatial dimension, the processing comprising processing the first component, which includes dividing the first component in a spatial dimension into a first plurality of tiles and processing the tiles of the first plurality of tiles separately; and processing the second component, which includes dividing the second component in a spatial dimension into a second plurality of tiles and processing the tiles of the second plurality of tiles separately, wherein at least two of each of the first plurality of tiles and the second plurality of tiles are of different sizes.

[0023] According to one aspect of the present disclosure, an apparatus for processing an input tensor representing picture data, the apparatus comprising one or more processors and a non-temporary computer-readable storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the program, when executed by the one or more processors, configures the apparatus to perform a method according to any of the aforementioned aspects of the present disclosure.

[0024] This disclosure is applicable to both end-to-end AI codecs and hybrid AI codecs. In hybrid AI codecs, for example, filtering operations (filtering of reconstructed pictures) may be performed by a neural network (NN). This disclosure is applicable to such NN-based processing modules. In general, this disclosure may be applicable to all or part of a video compression and decompression process if at least part of the processing involves an NN, and such NN involves convolution or transposed convolution operations. For example, this disclosure is applicable to individual processing tasks such as those performed as part of processing performed by an encoder and / or decoder, including in-loop filtering and / or post-filtering and / or pre-filtering.

[0025] Please note that this disclosure is not limited to any particular framework. Furthermore, this disclosure is not limited to image or video compression, but may also apply to object detection, image generation, and recognition systems.

[0026] The present invention can be implemented in hardware (HW) and / or software (SW). Furthermore, a hardware-based implementation may be combined with a software-based implementation.

[0027] For clarity, any one of the embodiments described above can be combined with any one or more of the other embodiments described above to create new embodiments within the scope of this disclosure.

[0028] Details of one or more embodiments are described in the accompanying drawings and the following description. Other features, purposes, and advantages will become apparent from this paper, the drawings, and the claims. [Brief explanation of the drawing]

[0029] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. [Figure 1A] This is a block diagram showing an example of a video coding system configured to implement an embodiment of the present invention. [Figure 1B] This is a block diagram showing another example of a video coding system configured to implement embodiments of the present invention. [Figure 2] This is a block diagram showing an example of an encoding or decoding device. [Figure 3] This is a block diagram showing another example of an encoding or decoding device. [Figure 4] This is a block diagram showing an exemplary hybrid encoder configured to implement an embodiment of the present invention. [Figure 5] This is a block diagram illustrating an exemplary hybrid decoder configured to implement an embodiment of the present invention. [Figure 6A] This is a schematic diagram showing a variational autoencoder architecture including a hyperplier model. [Figure 6B] This schematic diagram shows another example of a variational autoencoder architecture, including a hyperplier model similar to that in Figure 6A. [Figure 7] This is a block diagram showing a part of an example autoencoder. [Figure 8] The compression of input data by the encoder and the decompression of data by the decoder are shown together with the compressed data represented in latent space. [Figure 9] This is a block diagram of an encoder and decoder in accordance with the VAE framework. [Figure 9A] Figure 9 is a block diagram of an encoder having each of its components. [Figure 9B] Figure 9 is a block diagram of a decoder having each of its components. [Figure 10] This figure shows the entire receptive field, including all input samples required to generate the output sample. [Figure 11] This represents a subset of the entire receptive field. In this case, the output samples are generated by a smaller number of samples (a subset) than the total number of samples in the entire receptive field. Sample padding may be necessary. [Figure 12] This demonstrates downsampling of an input sample to a single output sample using two convolutional layers. [Figure 13] We illustrate how to compute the entire receptive field for a 2x2 set of output samples using two convolutional layers with a 3x3 kernel size. [Figure 14] This example demonstrates parallel processing where the picture is split into two tiles, and both the decoding and sample reconstruction of each bitstream are performed independently. [Figure 15]This example demonstrates parallel processing where an encoded tree block (CTB) is divided into slices (rows), where the bitstreams of each slice are decoded (almost) independently, but the sample reconstruction of the slices is not. [Figure 16] This is a block diagram of an encoder and decoder having modules, each including an NN-based subnetwork for processing a first and second set of tiles according to the first embodiment. [Figure 17A] An example is shown in which the first and / or second tensor is divided into overlapping regions Li (i.e., the first and second tiles), then the sample is cropped within the overlapping region, and the cropped regions are concatenated. Each Li contains the entire receptive field. [Figure 17B] Another example is shown where the first and / or second tensor is divided into overlapping regions Li (i.e., the first and second tiles), similar to Figure 17A, except that cropping is rejected. [Figure 18] An example is shown in which the first and / or second tensor is divided into overlapping regions Li (i.e., the first and second tiles), then samples in the overlapping regions are cropped, and the cropped regions are concatenated. Each Li contains a subset of the entire receptive field. [Figure 19] An example is shown in which the first and / or second tensor is divided into non-overlapping regions Li (i.e., the first and second tiles), then the samples are cropped, and the cropped regions are concatenated. Each Li contains a subset of the entire receptive field. [Figure 20] This indicates various parameters that may be included in (and parsed from) the bitstream, such as the size of the first and / or second tile's regions Li, Ri, and overlapping regions. Any of these various parameters may be included as instructions in (and parsed from) the bitstream. [Figure 21] A flowchart of a method for encoding an input tensor representing picture data according to the first embodiment is shown. [Figure 22]A flowchart of a method for decoding a tensor representing picture data according to the first embodiment is shown. [Figure 23] A block diagram of a processing unit for encoding an input tensor representing picture data, comprising a processing circuit, is shown. The processing circuit may be configured to include a module that performs the processing of the encoding method according to the first embodiment. [Figure 24] A block diagram of a processing unit for decoding a tensor representing picture data, comprising a processing circuit, is shown. The processing circuit may be configured to include a module that performs the processing of the decoding method according to the first embodiment. [Figure 25] This is a block diagram of an encoder-decoder according to a second embodiment, in which the lumer and chroma components of a tensor are processed separately in multiple pipelines, and each pipeline processes multiple tiles of its respective component separately, with each module having its own set of modules. [Figure 26] A flowchart of a method for processing an input tensor representing picture data according to a second embodiment is shown. [Figure 27] A block diagram of a device for processing an input tensor representing picture data, comprising the respective processing circuits and modules according to the second embodiment, is shown. [Modes for carrying out the invention]

[0030] Hereinafter, several embodiments of the present disclosure will be described with reference to the figures. Figures 1 to 3 refer to video coding systems and methods that may be used in conjunction with more specific embodiments of the present invention described in further figures. Specifically, the embodiments described with reference to Figures 1 to 3 may be used in conjunction with encoding / decoding techniques, which will be described further below, that utilize a neural network for encoding and / or decoding a bitstream.

[0031] The following description refers to accompanying drawings that form part of the Disclosure and illustrate specific aspects of the embodiments of the Disclosure or specific aspects in which the embodiments of the Disclosure may be used. It is understood that the embodiments may be used in other aspects and may include structural or logical modifications not shown in the drawings. Therefore, the following detailed description should not be construed as limiting, and the scope of the Disclosure is defined by the accompanying claims.

[0032] For example, disclosure relating to a described method is understood to also apply to a corresponding device or system configured to perform that method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units for performing the described method steps, such as functional units (e.g., one unit that performs the one or more steps, or a plurality of units, each performing one or more of the plurality of steps), even if such one or more units are not explicitly described or shown in the drawings. On the other hand, for example, if a particular apparatus is described based on one or more units, such as a functional unit, the corresponding method may include one step performing the function of the one or more units (e.g., one step performing the function of the one or more units, or a plurality of steps, each performing the function of one or more of the plurality of units), even if such one or more steps are not explicitly described or shown in the drawings. Furthermore, it is understood that the various exemplary embodiments and / or aspect features described herein can be combined with each other unless otherwise specified.

[0033] Video coding generally refers to the processing of a sequence of pictures that make up a video or video sequence. The terms "frame" or "image" are sometimes used as synonyms in the field of video coding instead of "picture." Video coding (or coding in general) consists of two parts: video encoding and video decoding. Video encoding is performed on the source side and typically involves processing the original video picture (e.g., by compression) to reduce the amount of data required to represent the video picture (for more efficient storage and / or transmission). Video decoding is performed on the destination side and typically involves the reverse processing compared to the encoder to reconstruct the video picture. Embodiments referring to "coding" of a video picture (or picture in general) should be understood as relating to the "encoding" or "decoding" of the video picture or the respective video sequence. The combination of the encoding and decoding parts is also called a codec (coding and decoding).

[0034] In lossless video coding, the original video picture can be reconstructed, meaning the reconstructed video picture has the same quality as the original video picture (assuming there is no transmission loss or other data loss during storage or transmission). In lossy video coding, further compression is performed, for example by quantization, to reduce the amount of data representing the video picture, and this cannot be fully reconstructed by the decoder. That is, the quality of the reconstructed video picture is lower or worse compared to the quality of the original video picture.

[0035] Some video coding standards belong to the group of “lossy hybrid video codecs” (i.e., combining spatial and temporal prediction in the sample domain with 2D transform coding to apply quantization in the transform domain). Each picture in a video sequence is typically divided into a set of non-overlapping blocks, and coding is typically performed at the block level. In other words, in an encoder, video is typically processed, i.e., encoded, at the block (video block) level, which is done, for example, by using spatial (intra-picture) and / or temporal (inter-picture) prediction to generate predicted blocks, subtracting the predicted blocks from the current blocks (the blocks currently being processed / to be processed) to obtain residual blocks, transforming the residual blocks, and quantizing the residual blocks in the transform domain to reduce the amount of data to be transmitted (compression). In a decoder, on the other hand, the reverse process compared to the encoder is applied to the encoded or compressed blocks in order to reconstruct the current blocks for representation. Furthermore, the encoder replicates the decoder processing loop so that both generate identical predictions (e.g., intra-predictions and inter-predictions) and / or reconstructions for processing subsequent blocks, i.e., coding. More recently, part or all of the encode and decode chain is implemented using neural networks, or generally some machine learning or deep learning framework.

[0036] In the following embodiments of the video coding system 10, the video encoder 20 and the video decoder 30 are described with reference to Figure 1.

[0037] Figure 1A is a schematic block diagram showing an exemplary coding system 10 that may utilize the techniques of the present application, for example, a video coding system 10 (or simply coding system 10). The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) of the video coding system 10 represent examples of devices that may be configured to perform the techniques described in the various examples herein.

[0038] As shown in Figure 1A, the coding system 10 includes a source device 12 configured to provide encoded picture data 21 to a destination device 14 for decoding the encoded picture data 13, for example.

[0039] The source device 12 comprises an encoder 20 and may further, optionally, a picture source 16, a preprocessor (or preprocessing unit) 18, for example, a picture preprocessor 18, and a communication interface or communication unit 22. Some embodiments of the present disclosure (for example, those relating to initial rescaling or rescaling between two progressing layers) may be implemented by the encoder 20. Some embodiments (for example, those relating to initial rescaling) may be implemented by the picture preprocessor 18.

[0040] The picture source 16 may have, or may have, any kind of picture capturing device, such as a camera for capturing real-world pictures, and / or any kind of picture generating device, such as a computer graphics processor for generating computer-animated pictures, or any other kind of device for acquiring and / or providing real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). The picture source may also have any kind of memory or storage device for storing any of the aforementioned pictures.

[0041] To distinguish it from the processing performed by the preprocessor 18 and the preprocessing unit 18, the picture or picture data 17 is sometimes referred to as the raw picture or raw picture data 17.

[0042] The preprocessor 18 is configured to receive (raw) picture data 17 and perform preprocessing on the picture data 17 to obtain a preprocessed picture 19 or preprocessed picture data 19. The preprocessing performed by the preprocessor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr or generally from RGB to YUV), color correction, or denoising. It should be understood that the preprocessing unit 18 may be an arbitrary component. Hereinafter, color space components (e.g., R, G, B for the RGB space and Y, U, V for the YUV space) are also referred to as color channels. Furthermore, in the YCbCr color space, Y represents luminance (or luma), and U, V, Cb, Cr represent chrominance (or chroma) channels (components).

[0043] The video encoder 20 is configured to receive pre-processed picture data 19 and provide encoded picture data 21 (further details are described below, for example, based on Figure 4). The encoder 20 may be implemented via processing circuitry 46 to embody various modules discussed with respect to the encoder 20 in Figure 4 and / or any other encoder system or subsystem described in this paper.

[0044] The communication interface 22 of the source device 12 may be configured to receive encoded picture data 21 and send the encoded picture data 21 (or any further processed version thereof) through the communication channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0045] The destination device 14 includes a decoder 30 (for example, a video decoder 30) and may additionally, i.e., optionally, include a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34.

[0046] The communication interface 28 of the destination device 14 is configured to receive encoded picture data 21 (or any further processed version thereof) for example directly from the source device 12, or from any other source, such as a storage device, such as a storage device for encoded picture data, and to provide the encoded picture data 21 to the decoder 30.

[0047] Communication interfaces 22 and 28 may be configured to transmit or receive encoded picture data 21 or encoded data 13 via a direct communication link between the source device 12 and the destination device 14, for example via a direct wired or wireless connection, or via any type of network, for example via a wired or wireless network or any combination thereof, or via any type of private and public network or any combination thereof.

[0048] The communication interface 22 may be configured, for example, to package the encoded picture data 21 into an appropriate format, such as a packet, and / or to process the encoded picture data using any kind of transmit encoding or processing for transmission over a communication link or communication network.

[0049] A communication interface 28 forming a counterpart to communication interface 22 may be configured, for example, to receive transmitted data and process the transmitted data using any kind of corresponding transmit decoding or processing and / or depackaging to obtain encoded picture data 21.

[0050] Both communication interfaces 22 and 28 may be configured as unidirectional or bidirectional communication interfaces, as indicated in Figure 1A by the arrow pointing from source device 12 to destination device 14 for communication channel 13, for example, and may be configured to send and receive messages, set up connections, receive and confirm any other information relating to communication links and / or data transmission, such as the transmission of encoded picture data.

[0051] The decoder 30 is configured to receive encoded picture data 21 and provide decoded picture data 31 or decoded picture 31 (further details are described below, for example, with reference to Figures 3 and 5). The decoder 30 may be implemented via a processing circuit 46 to embody various modules discussed with respect to the decoder 30 in Figure 5 and / or any other decoder system or subsystem described in this paper.

[0052] The post-processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), for example, the decoded picture 31, to obtain post-processed picture data 33, for example, the post-processed picture 33. Post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., YCbCr to RGB), color correction, cropping, or resampling, or any other processing to prepare the decoded picture data 31 for display by the display device 34, for example.

[0053] Some embodiments of this disclosure may be implemented by the decoder 30 or by the postprocessor 32.

[0054] The display device 34 of the destination device 14 is configured to receive post-processed picture data 33 for displaying the picture to a user or viewer, for example. The display device 34 may be, or may have, any type of display for displaying the reconstructed picture, such as an integrated or external display or monitor. The display may include, for example, a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a plasma display, a projector, a microLED display, liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0055] Although Figure 1A shows the source device 12 and the destination device 14 as separate devices, the device embodiment may include both or both of their functions, i.e., the source device 12 or its corresponding function and the destination device 14 or its corresponding function. In such embodiments, the source device 12 or its corresponding function and the destination device 14 or its corresponding function may be implemented using the same hardware and / or software, by separate hardware and / or software, or by any combination thereof.

[0056] As will become apparent to those skilled in the art based on this paper, the function or presence and (strict) division of the various units within the source device 12 and / or destination device 14, as shown in Figure 1A, may vary depending on the actual device and application.

[0057] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30), or both the encoder 20 and the decoder 30, may be implemented via processing circuitry (as shown in Figure 1B), such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video coding, or any combination thereof. The encoder 20 may be implemented via processing circuitry 46 to embody various modules and / or any other encoder systems or subsystems described herein. The decoder 30 may be implemented via processing circuitry 46 to embody various modules and / or any other decoder systems or subsystems described herein. The processing circuitry may be configured to perform various operations, as described below. If the technique is partially implemented in software, as shown in Figure 3, the device may store instructions for the software in a suitable non-temporary computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the technique of this disclosure. Either the video encoder 20 or the video decoder 30 may be integrated as part of a composite encoder / decoder (codec) within a single device, for example, as shown in Figure 1B.

[0058] The source device 12 and destination device 14 may comprise any type of handheld or stationary device, including a wide range of devices such as a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or content distribution server), a broadcast receiver device, a broadcast transmitter device, and may or may not have an operating system, or may use any type of operating system. In some cases, the source device 12 and destination device 14 may be equipped for wireless communication. Thus, the source device 12 and destination device 14 may be wireless communication devices.

[0059] In some cases, the video coding system 10 shown in Figure 1A is merely an example, and the techniques of the present invention may be applied to video coding settings (e.g., video encoding or video decoding) that do not necessarily involve data communication between an encoding device and a decoding device. In other examples, data may be retrieved from local memory, streamed over a network, or similar. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed simply by devices that encode data into memory and / or retrieve data from memory and decode it, without communication between them.

[0060] For the sake of explanation, some embodiments are described in this paper by reference to, for example, High Efficiency Video Coding (HEVC) or Multipurpose Video Coding (VVC), a next-generation video coding standard developed by the ITU-T Video Coding Expert Group (VCEG) and the ISO / IEC Video Coding Expert Group (MPEG) Joint Team on Video Coding (JCT-VC). Those skilled in the art will understand that embodiments of the present invention are not limited to HEVC or VVC.

[0061] Figure 2 is a schematic diagram of a video coding device 200 according to one embodiment of the present invention. The video coding device 200 is suitable for implementing the disclosed embodiments as described herein. In one embodiment, the video coding device 200 may be a decoder, such as the video decoder 30 in Figure 1A, or an encoder, such as the video encoder 20 in Figure 1A.

[0062] The video coding device 200 comprises an inlet port 210 (or input port 210) and a receiver unit (Rx) 220 for receiving data; a processor, logic unit, or central processing unit (CPU) 230 for processing the data; a transmitter unit (Tx) 240 and an exit port 250 (or output port 250) for transmitting data; and memory 260 for storing data. The video coding device 200 may also include optical-electrical (OE) components and electrical-optical (EO) components coupled to the inlet port 210, receiver unit 220, transmitter unit 240, and exit port 250 for sending or receiving optical or electrical signals.

[0063] The processor 230 is implemented by hardware and software. The processor 230 may be implemented as one or more CPU chips, cores (e.g., a multi-core processor), FPGAs, ASICs, and DSPs. The processor 230 communicates with the input port 210, the receiver unit 220, the transmitter unit 240, the output port 250, and the memory 260. The processor 230 includes a coding module 270. The coding module 270 implements the embodiments disclosed above. For example, the coding module 270 performs, processes, prepares, or provides various coding operations. Thus, including the coding module 270 provides a substantial improvement to the functionality of the video coding device 200, resulting in the conversion of the video coding device 200 to different states. Alternatively, the coding module 270 may be implemented as instructions stored in the memory 260 and executed by the processor 230.

[0064] Memory 260 may comprise one or more disks, tape drives, and solid-state drives, and may be used as an overflow data storage device to store such programs when they are selected for execution, and to store instructions and data read during program execution. Memory 260 may be, for example, volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), tri-level associative memory (TCAM), and / or static random access memory (SRAM).

[0065] Figure 3 is a simplified block diagram of a device 300 that can be used as either or both of the source device 12 and destination device 14 from Figure 1, according to an exemplary embodiment.

[0066] The processor 302 within the apparatus 300 may be a central processing unit. Alternatively, the processor 302 may be any other type of device or multiple devices currently existing or to be developed that are capable of manipulating or processing information. The disclosed embodiments may be carried out using a single processor, such as processor 302 as shown, but advantages in speed and efficiency may be achieved using more than one processor.

[0067] The memory 304 within the device 300 may, in some implementations, be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 304. Memory 304 may contain code and data 306 accessed by the processor 302 using the bus 312. Memory 304 may further contain an operating system 308 and an application program 310, the application program 310 including at least one program that enables the processor 302 to perform the methods described herein. For example, the application program 310 may include applications 1 to N, which further include video coding applications that perform the methods described herein.

[0068] The device 300 may also include one or more output devices, such as a display 318. In one example, the display 318 may be a touch-sensitive display combining a display with a touch-sensitive element capable of operating to sense touch input. The display 318 may be coupled to the processor 302 via a bus 312.

[0069] Although shown here as a single bus, the bus 312 of device 300 may consist of multiple buses. Furthermore, the secondary storage 314 may be directly coupled to other components of device 300, or accessed via a network, and may comprise a single integrated unit such as a memory card, or multiple units such as multiple memory cards. Thus, device 300 can be implemented in a wide variety of configurations.

[0070] Figure 4 shows a schematic block diagram of an exemplary video encoder 20 configured to implement the technique of the present invention. In the example of Figure 4, the video encoder 20 comprises an input 401 (or input interface 401), a residual calculation unit 404, a transformation processing unit 406, a quantization unit 408, an inverse quantization unit 410, an inverse transformation processing unit 412, a reconstruction unit 414, a loop filter unit 420, a decoded picture buffer (DPB) 430, a mode selection unit 460, an entropy encoding unit 470, and an output 472 (or output interface 472). The mode selection unit 460 may include an inter-prediction unit 444, an intra-prediction unit 454, and a partitioning unit 462. The inter-prediction unit 444 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 shown in Figure 4 is sometimes called a hybrid video encoder, or a video encoder with a hybrid video codec.

[0071] The encoder 20 may be configured to receive, for example, a picture 17 (or picture data 17) via input 401, for example, a picture of a sequence of pictures forming a video or video sequence. The received picture or picture data may be a pre-processed picture 19 (or pre-processed picture data 19). For simplicity, the following description will refer to picture 17. Picture 17 is also called the current picture or the picture being coded (particularly in video coding to distinguish the current picture from other pictures, for example, previously encoded and / or decoded pictures of the same video sequence, i.e., the video sequence which also contains the current picture).

[0072] A (digital) picture is, or can be considered as, a two-dimensional array or matrix of samples having intensity values. Samples in an array are sometimes called pixels (short for picture element) or picture elements. The number of samples in the horizontal and vertical directions (or axes) of an array or picture defines the size and / or resolution of the picture. For color representation, typically three color components are used; that is, a picture may be represented by three sample arrays, or may contain three sample arrays. In the RGB format or color space, a picture contains corresponding red, green, and blue sample arrays. However, in video coding, each pixel is typically represented by a luminance and chrominance format or color space, such as YCbCr, which includes a luminance component represented by Y (sometimes L is used instead) and two chrominance components represented by Cb and Cr. The luminance (or luma for short) component Y represents brightness or gray level intensity (for example, in a grayscale picture), and the two chrominance (or chroma for short) components Cb and Cr represent chromaticity or color information components. Thus, a picture in the YCbCr format includes a luminance sample array of luminance sample values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). A picture in the RGB format may be converted to or from the YCbCr format, and vice versa; this process is also known as color conversion or transformation. If the picture is monochrome, it may contain only a luminance sample array. Thus, a picture may be, for example, an array of luma samples in a monochrome format, or an array of luma samples and two corresponding arrays of chroma samples in the 4:2:0, 4:2:2, and 4:4:4 color formats.

[0073] Embodiments of the video encoder 20 may have a picture partitioning unit (not shown in Figure 2) configured to partition a picture 17 into a plurality of (typically non-overlapping) picture blocks 403. These blocks may also be called root blocks, macroblocks (H.264 / AVC), coding tree blocks (CTBs), or coding tree units (CTUs) (H.265 / HEVC and VVC). The picture partitioning unit may be configured to use the same block size for all pictures in the video sequence and a corresponding grid defining said block size, or to change the block size between pictures or between subsets or groups of pictures, dividing each picture into a corresponding block. The abbreviation AVC stands for Advanced Video Coding.

[0074] In a further embodiment, the video encoder may be configured to directly receive a block 403 of picture 17, for example, one, some, or all of the blocks that make up picture 17. The picture block 403 may also be called the current picture block or the picture block to be coded.

[0075] Similar to picture 17, picture block 403 is smaller in dimensions than picture 17, but is or can be considered as a two-dimensional array or matrix of samples having intensity values ​​(sample values). In other words, block 403 may contain, for example, one sample array (e.g., a luma array for monochrome picture 17, or a luma or chroma array for color picture), or three sample arrays (e.g., a luma and two chroma arrays for color picture 17), or any other number and / or type of arrays depending on the applied color format. The number of samples in the horizontal and vertical (or axis) directions of block 403 defines the size of block 403. Thus, the block may be, for example, an M×N (M columns N rows) array of samples, or an M×N array of conversion coefficients.

[0076] An embodiment of the video encoder 20, as shown in Figure 4, may be configured to encode the picture 17 block by block, for example, encoding and prediction may be performed for each block 403.

[0077] Embodiments of the video encoder 20, as shown in Figure 4, may be further configured to divide and / or encode a picture by using slices (also called video slices), the picture may be divided into one or more slices (typically non-overlapping) or encoded using them, each slice may contain one or more blocks (e.g., CTUs).

[0078] Embodiments of the video encoder 20, as shown in Figure 4, may be further configured to divide and / or encode a picture by using tile groups (also called video tile groups) and / or tiles (also called video tiles), where the picture may be divided into one or more tile groups (typically non-overlapping) or encoded using them, each tile group may contain, for example, one or more blocks (e.g., CTUs) or one or more tiles, each tile may be, for example, rectangular in shape and may contain one or more blocks (e.g., CTUs), for example, complete or partial blocks.

[0079] Figure 5 shows an example of a video decoder 30 configured to implement the technique of the present invention. The video decoder 30 is configured to receive encoded picture data 21 (e.g., encoded bitstream 21), which has been encoded by, for example, an encoder 20, and to obtain a decoded picture 531. The encoded picture data or bitstream includes information for decoding the encoded picture data, for example, data representing picture blocks of an encoded video slice (and / or tile group or tile), and associated syntax elements.

[0080] The entropy decode unit 504 is configured to parse the bitstream 21 (or generally encoded picture data 21) and, for example, perform entropy decoding on the encoded picture data 21 to obtain, for example, quantized coefficients 309 and / or decoded coding parameters (not shown in Figure 3), such as inter-prediction parameters (e.g., reference picture index and motion vector), intra-prediction parameters (e.g., intra-prediction mode or index), transformation parameters, quantization parameters, loop filter parameters, and / or other syntax elements, or any or all of them. The entropy decode unit 504 may be configured to apply a decoding algorithm or scheme corresponding to the encoding scheme described with respect to the entropy encoding unit 470 of the encoder 20. The entropy decode unit 504 may be further configured to provide the inter-prediction parameters, intra-prediction parameters, and / or other syntax elements to the mode application unit 360 and other parameters to other units of the decoder 30. The video decoder 30 may receive syntax elements at the video slice level and / or video block level. In addition to slices and their respective syntax elements, tile groups and / or tiles and their respective syntax elements may be received and / or used.

[0081] The reconstruction unit 514 (for example, an adder or summer 514) may be configured to add the reconstructed residual block 513 to the prediction block 565 by adding, for example, the sample value of the reconstructed residual block 513 to the sample value of the prediction block 565, thereby obtaining the reconstructed block 515 in the sample region.

[0082] Embodiments of the video decoder 30, as shown in Figure 5, may be configured to divide and / or decode a picture by using slices (also called video slices), where the picture may be divided into one or more slices (typically non-overlapping) or decoded using them, and each slice may contain one or more blocks (e.g., CTUs).

[0083] Embodiments of the video decoder 30 shown in Figure 5 may be configured to divide and / or decode a picture by using tile groups (also called video tile groups) and / or tiles (also called video tiles), where the picture may be divided into one or more tile groups (typically non-overlapping) or decoded using them, each tile group may, for example, contain one or more blocks (e.g., CTUs) or one or more tiles, each tile may, for example, be rectangular in shape and may contain one or more blocks (e.g., CTUs), for example, complete or partial blocks.

[0084] Other variations of the video decoder 30 may be used to decode the encoded picture data 21. For example, the decoder 30 can generate an output video stream without a loop filtering unit 520. For example, a non-transformation-based decoder 30 can directly dequantize the residual signal for certain blocks or frames without an inverse processing unit 512. In another implementation, the video decoder 30 may have an inverse quantization unit 510 and an inverse processing unit 512 combined into a single unit.

[0085] It should be understood that in encoder 20 and decoder 30, the processing result of the current step may be further processed and then output to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, further operations such as clipping or shifting may be performed on the processing result of interpolation filtering, motion vector derivation, or loop filtering.

[0086] The following describes some more specific, non-limiting exemplary embodiments of the present invention. Before that, some explanations and definitions are provided to aid in understanding this disclosure.

[0087] Picture size Picture size refers to the width (w) or height (h) of a picture, or the width-height pair. The width and height of an image are typically measured in lumen samples.

[0088] Downsampling Downsampling is a process in which the sampling rate (sampling interval) of a discrete input signal is reduced. For example, if the input signal is an image with sizes h and w, and the downsampled output has sizes h2 and w2, then at least one of the following must be true: h2 <h w2 <w In one exemplary implementation, downsampling can be performed by retaining only every m-th sample and discarding the rest of the input signal (e.g., an image).

[0089] Upsampling: Upsampling is the process of increasing the sampling rate (sampling interval) of a discrete input signal. For example, if the input image has sizes h and w, and the downsampled output has sizes h2 and w2, then at least one of the following must be true: h2>h w2>w

[0090] Resampling: Both downsampling and upsampling processes are examples of resampling. Resampling is the process of changing the sampling rate (sampling interval) of an input signal. Resampling is a technique for resizing (or rescaling) an input signal.

[0091] During an upsampling or downsampling process, filtering may be applied to improve the accuracy of the resampled signal and reduce aliasing effects. Interpolation filtering typically involves a weighted combination of sample values ​​at sample positions around the resampling position. This can be implemented as follows:

number

[0092] Cropping: Cropping involves trimming (cutting) the outer edges of a digital image. Cropping can be used to make an image smaller (in terms of the number of samples) and / or to change the aspect ratio (length-to-width) of an image. This can be understood as removing samples from a signal, typically at the boundaries of the signal.

[0093] Padding: Padding refers to increasing the size of the input (i.e., the input image) by using predefined sample values, or by using sample values ​​at existing locations in the input image (e.g., by copying or combining them), or by generating new samples (e.g., at the boundaries of the image). The generated samples are approximations of the actual sample values ​​that do not exist.

[0094] Size change: Resizing is a general term for changing the size of an input image. Resizing may be performed using either padding or cropping, or it may be done by resampling.

[0095] Integer division: Integer division is a type of division in which the fractional part (remainder) is discarded.

[0096] Convolution: Convolution can be defined in one dimension for an input signal f() and a filter g() as follows:

number

[0097] Artificial neural networks Artificial neural networks (ANNs), or connectionist systems, are computing systems vaguely inspired by the biological neural networks that make up the brains of animals. Such systems "learn" to perform tasks by considering examples, without generally being programmed with task-specific rules. For example, in image recognition, an ANN can learn to identify images containing cats by analyzing images of examples manually labeled as "cat" or "not a cat," and then using the results to identify cats in other images. This is done without any prior knowledge of cats, such as having fur, a tail, whiskers, or a cat-like face. Instead, the ANN automatically generates the features to identify from the examples it processes.

[0098] ANNs are based on a collection of connected units or nodes called artificial neurons, which roughly model neurons in the biological brain. Each connection can transmit signals to other neurons, just like synapses in the biological brain. The receiving artificial neuron can then process the signal and transmit it to the neurons it is connected to. In an ANN implementation, the "signal" at a given connection is a real number, and the output of each neuron can be calculated by some nonlinear function of the sum of its inputs. These connections are called edges. Neurons and edges typically have weights that are adjusted as learning progresses. The weights increase or decrease the strength of the signal at the connection. Neurons may have thresholds such that a signal is transmitted only if the aggregated signal exceeds that threshold. Typically, neurons are aggregated into layers. Different layers may perform different transformations on their inputs. The signal travels from the first layer (input layer) to the last layer (output layer), possibly traversing multiple layers.

[0099] The initial goal of the ANN approach was to solve problems in the same way the human brain does. Over time, the focus shifted to performing specific tasks, leading to a departure from biology. ANNs have been used for a wide range of tasks, including computer vision, speech recognition, machine translation, social network filtering, playing board and video games, medical diagnosis, and even activities traditionally considered human specialties, such as painting.

[0100] Downsampling layer: A layer in a neural network that results in a reduction of at least one dimension of the input. Generally, an input can have three or more dimensions, which can include the number of channels, width, and height. Downsampling layers typically refer to reducing the width and / or height dimension. This can be done using operations such as convolution (possibly with stride), averaging, and max pooling.

[0101] Upsampling layer: An upsampling layer in a neural network that increases the input's dimensions by at least one. Generally, an input can have three or more dimensions, which can include the number of channels, width, and height. An upsampling layer typically refers to increasing the width and / or height dimensions. This can be achieved using operations such as deconvolution and duplication.

[0102] Feature map: Feature maps are generated by applying filters (kernels) or feature detectors to the input image or the feature map outputs of previous layers. Feature map visualizations provide insight into the internal representation of a particular input for each convolutional layer in a model. Generally, feature maps are the outputs of neural network layers. Feature maps typically contain one or more feature elements.

[0103] Convolutional Neural Network The name "Convolutional Neural Network" (CNN) indicates that the network uses a mathematical operation called convolution. Convolution is a special type of linear operation. A convolutional network is simply a neural network that uses convolution instead of general matrix multiplication in at least one layer. A convolutional neural network consists of an input layer, an output layer, and several hidden layers. The input layer is the layer to which the input is provided for processing.

[0104] For example, the neural network in Figure 6A is a CNN. The hidden layers of a CNN typically consist of a series of convolutional layers (e.g., conv layers 601-612 in Figure 6A) that convolve by multiplication or other dot products. The result of the layers is one or more feature maps, sometimes called channels. There may be subsampling that involves some or all of the layers. As a result, the feature maps become smaller. The activation function in a CNN may be a RELU (Rectified Linear Unit) layer or a GDN layer, as already illustrated above, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers. These layers are called hidden layers because their inputs and outputs are masked by the activation function and the final convolution. These layers are colloquially called convolutions, but this is merely by convention. Mathematically, it is technically a sliding dot product or cross-correlation. This is important to the indices in the matrix in that it affects how the weights are determined at a particular index point.

[0105] When programming a CNN to process pictures or images, the input is a tensor with the shape (number of images) × (image width) × (image height) × (image depth) (for example, an input tensor such as tensor x 614 in Figure 6A). Then, after passing through the convolutional layer, the image is abstracted into a feature map (feature tensor) with the shape (number of images) × (feature map width) × (feature map height) × (feature map channels). In Figure 6A, such a feature map is, for example, y. The convolutional layer in the neural network should have the following attributes: a convolutional kernel defined by width and height (hyperparameters); the number of input channels and output channels (hyperparameters); and the depth of the convolutional filter (input channels) should be equal to the number of channels (depth) of the input feature map. For example, conv Nx5x5 in Figure 6A refers to a kernel of size 5x5 and N channels, where N is an integer greater than or equal to 1.

[0106] In the past, conventional multilayer perceptron (MLP) models have been used for image recognition. However, due to the need for perfect connectivity between nodes, MLP models suffer from high dimensionality and do not scale well with higher resolution images. A 1000x1000 pixel image with RGB color channels has 3 million weights, which is too high to process efficiently and efficiently at a scale with perfect connectivity. Furthermore, such network architectures do not consider the spatial structure of the data, treating distant input pixels as if they were adjacent pixels. This ignores locality of reference in image data, both computationally and semantically. Therefore, perfect connectivity of neurons is redundant for purposes such as image recognition, which are governed by spatially local input patterns.

[0107] Convolutional neural networks (CNNs) are a biologically inspired variation of multilayer perceptrons, specifically designed to emulate the behavior of the visual cortex. CNN models mitigate the challenges posed by MLP architectures by leveraging the strong spatially local correlations present in natural images. Convolutional layers are the core components of a CNN. The layer parameters consist of a set of learnable filters (kernels, as described above), which have small receptive fields but extend across the entire depth of the input volume. During a forward pass, each filter is convolved across the width and height of the input volume, calculating the dot product between the filter's entry and the input to generate a two-dimensional activation map of that filter. As a result, the network learns which filters are activated when it detects a particular type of feature at a given spatial location in the input.

[0108] The complete output volume of the convolutional layer is formed by stacking activation maps for all filters along the depth dimension. Thus, all entries in the output volume can also be interpreted as the outputs of neurons that look at a small region in the input and share parameters with neurons in the same activation map. A feature map, or activation map, is the output activation for a given filter. Feature map and activation have the same meaning. In some papers, it is called an activation map because it is a mapping corresponding to the activation of different parts of an image, and also called a feature map because it is a mapping of where certain features can be found in the image. High activation means that a certain feature has been found.

[0109] Another important concept in CNNs is pooling. Pooling is a form of nonlinear downsampling. There are several nonlinear functions for performing pooling, of which max pooling is the most common. It divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each of such subregions.

[0110] Intuitively, the exact location of a feature is less important than its approximate location relative to other features. This is the idea behind the use of pooling in convolutional neural networks. Pooling layers work to progressively reduce the spatial size of representations, thereby reducing the number of parameters, memory footprint, and computational complexity in the network, and thus also controlling overfitting. In CNN architectures, it is common to periodically insert pooling layers between successive convolutional layers. Pooling operations provide another form of translational invariance.

[0111] A pooling layer acts independently on all depth slices of the input, spatially resizing them. The most common form is a pooling layer with a 2x2 filter applied with a stride of 2, which downsamples each depth slice in the input by a factor of 2 along both width and height, discarding 75% of the activations. In this case, each maximum operation is on four numbers. The depth dimension remains invariant.

[0112] In addition to max pooling, pooling units can use other functions such as mean pooling or L2 norm pooling. Mean pooling, while historically frequently used, has recently become less preferred compared to max pooling, which performs better in practice. The recent trend is toward using smaller filters or discarding pooling layers altogether due to aggressive reduction in representation size. Region of interest pooling (also known as ROI pooling) is a variation of max pooling where the output size is fixed and the input rectangle is parameterized. Pooling is a key component of convolutional neural networks for object detection based on the fast R-CNN architecture.

[0113] ReLU, short for rectified linear unit, applies a non-saturated activation function. It effectively removes negative values ​​from the activation map by setting them to zero. This increases the nonlinearity of the decision function and the entire network without affecting the receptive field of the convolutional layer. Other functions, such as the saturated hyperbolic tangent and sigmoid functions, are also used to increase nonlinearity. ReLU is often preferred over other functions because it trains neural networks several times faster without a significant penalty to generalized accuracy.

[0114] After several convolutional and max-pooling layers, high-level inference in the neural network is performed via fully connected layers. Neurons in the fully connected layers have connections to all activations in the previous layer, as seen in typical (non-convolutional) artificial neural networks. Thus, their activations can be computed as affine transformations with matrix multiplication followed by bias offsets (vector addition of learned or fixed bias terms).

[0115] The "loss layer" specifies how the training penalizes the deviation between the predicted (output) label and the true label, and is usually the final layer of a neural network. Different loss functions can be used that are suitable for different tasks. Softmax loss is used to predict a single class out of K mutually exclusive classes. Sigmoid cross-entropy loss is used to predict K independent probability values ​​in [0,1]. Euclidean loss is used to regress to real-valued labels.

[0116] Subnetwork A neural network can include multiple subnetworks. A subnetwork consists of one or more layers. Different subnetworks have different input / output sizes, resulting in different memory requirements and / or computational complexity.

[0117] pipeline A series of subnetworks that process specific components of an image. For example, the image components may be R, G, or B components. The components may be either lumera Y or chroma components U or V. One example is a system with two pipelines, where the first pipeline processes only lumera components and the second pipeline processes chroma components (one or more). One pipeline processes only one component, and the second component (e.g., lumera components) may be used as auxiliary information to support its processing (e.g., processing of chroma components). For example, a pipeline with chroma components as output may have latent representations of both lumera and chroma components as input (conditional coding of chroma components).

[0118] Rate-distortion optimizing quantization (RDOQ) RDOQ is an encoder-only technique, meaning it applies to processing performed by the encoder and not the decoder. Before writing to the bitstream, the parameters are quantized (descaled, rounded, etc.) to a specified standard fixed precision. Of several rounding methods, a variation of the minimum RD cost, such as that used in HEVC or VVC for conversion coefficient coding, may often be chosen.

[0119] Conditional Color Separation (CCS) In NN architectures for image / video coding / processing, CCS refers to the independent coding / processing of primary color components (e.g., lumens), while secondary color components (e.g., chroma UVs) are conditionally coded / processed using primary components as auxiliary inputs.

[0120] Autoencoders and unsupervised learning An autoencoder is a type of artificial neural network used to learn efficient data coding in an unsupervised manner. A schematic diagram is shown in Figure 7. This can be considered a simplified representation of the CNN-based VAE (variational autoencoder) structure shown in Figure 6A or Figure 6B. The purpose of an autoencoder is to learn representations (encodes) of a set of data, typically for dimensionality reduction, by training the network to ignore signal "noise". Along with the reduction side, the reconstruction side is learned, where the autoencoder attempts to generate a representation from the reduced encoding that is as close as possible to its original input. This is where the name comes from.

[0121] In its simplest form, given one hidden layer, the encoder stage of the autoencoder takes input x and maps it to h: h = σ(Wx + b) This image h is typically called the code, latent variable, or latent representation. Here, σ is an element-wise activation function, such as a sigmoid function or rectified linear unit. W is the weight matrix. b is the bias vector. The weights and biases are typically initialized randomly and then updated sequentially and iteratively through backpropagation during training. The decoder stage of the autoencoder then maps h to a reconstructed x' with the same shape as x. x'=σ'(W'h'+b') Here, σ', W', and b' for the decoder may be independent of the corresponding σ, W, and b for the encoder.

[0122] Variational autoencoder models make strong assumptions about the distribution of latent variables. They use a variational approach to learn the latent representation, resulting in an additional loss component and a specific estimator for the training algorithm called the stochastic gradient variational Bayes (SGVB) estimator. The data is applied to a directed graphical model p θ The encoder is generated by (x|h) and the posterior distribution pθ Approximate q with respect to (h|x) φ Suppose that we are learning (h|x). Here, φ and θ denote the parameters of the encoder (recognition model) and the decoder (generation model), respectively. The probability distribution of the latent vector of VAE typically matches the probability distribution of the training data much better than that of the standard autoencoder. The objective function of VAE has the following form: [Number] Here, D KL represents the Kullback-Leibler divergence. The prior for the latent variable is usually set to be the central isotropic multivariate Gaussian p θ = N(0, I). Usually, the shapes of the variational and likelihood distributions are selected to be factorized Gaussians: [Number] Here, ρ(x) and ω 2 (x) are the encoder outputs. μ(h) and σ 2 (h) are the decoder outputs.

[0123] Recent advances in the field of artificial neural networks, particularly convolutional neural networks, have enabled researchers' interest in applying neural network-based techniques to the tasks of image and video compression. For example, end-to-end optimized image compression using a network based on the variational autoencoder (AVE) has been proposed.

[0124] Therefore, data compression is considered a fundamental and well-studied problem in engineering, and is usually formulated with the goal of designing a code for a given discrete data ensemble with the minimum entropy. This solution relies heavily on knowledge of the probabilistic structure of the data, and thus the problem is closely related to probabilistic source modeling. However, since all actual codes must have a finite entropy, continuous data (such as a vector of image pixel intensity) must be quantized into a finite set of discrete values, which introduces errors.

[0125] In this context, it is known as a lossy compression problem, but it requires a trade-off between two competing costs: the entropy (rate) of the discretized representation and the error (distortion) resulting from quantization. Different compression applications, such as data storage or transmission through channels with limited capacity, require different rate-distortion trade-offs. Simultaneous optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in high-dimensional spaces cannot be addressed.

[0126] Therefore, most existing image compression methods operate by linearly transforming the data vector into a suitable continuous-value representation, independently quantizing its elements, and then encoding the resulting discrete representation using a lossless entropy code. This method is called transformative coding because of the central role of the transformation.

[0127] For example, JPEG uses the discrete cosine transform for blocks of pixels, while JPEG2000 uses multiscale orthogonal wavelet decomposition. Typically, the three components of a transform coding method—the transform, the quantizer, and the entropy code—are optimized separately (often through manual parameter tuning). Modern video compression standards such as HEVC, VVC, and EVC also use the transformed representation to encode the residual signal after prediction. Several transforms are used for this purpose, including the discrete cosine and sine transforms (DCT, DST), as well as the low-frequency inseparable manually optimized transform (LFNST).

[0128] Latent space: The latent space refers to the feature map generated in the bottleneck layer of a neural network (NN). This is illustrated in the examples shown in Figures 7 and 8. In NN topologies where the goal of the network is to reduce the dimensionality of the input signal (such as an autoencoder topology), the bottleneck layer is typically the layer where the dimensionality of the input signal is reduced to the minimum. The goal of dimensionality reduction is typically to achieve a more compact representation of the input. Therefore, the bottleneck layer is a layer suitable for compression, and thus, in the case of video coding applications, the bitstream is generated based on the bottleneck layer.

[0129] An autoencoder topology typically consists of encoders and decoders connected to each other in a bottleneck layer. The purpose of the encoders is to reduce the dimensionality of the input, making it more compact (or more intuitive). The purpose of the decoders is to reverse the behavior of the encoders, and thus reconstruct the input as best as possible based on the bottleneck layer.

[0130] Variational Autoencoder (VAE) Framework The VAE framework can be thought of as a nonlinear transform coding model. The transform process can be divided into four main parts, which are illustrated in Figure 9, which shows the VAE framework.

[0131] The transformation process can be divided into four parts. Figure 9 illustrates a VAE framework including encoder and decoder branches. In Figure 9, encoder 901 maps the input image x to a latent representation (denoted by y) via the function y=f(x). This latent representation may hereafter be referred to as a part or point in the “latent space”. The function f() is a transformation function that converts the input signal x to a more compressible representation y. 〔以下、^付きの文字は^を前に付けることで表すことがある〕 The quantizer 902 transforms the latent representation y into a quantized latent representation ^y with (discrete) values ​​by ^y = Q(y), where Q represents the quantization function, which may be RDOQ. The entropy model or hyperencoder / decoder (also known as hyperplier) 903 estimates the distribution of the quantized latent representations ^y to obtain the minimum rate achievable with lossless entropy source coding.

[0132] The latent space can be understood as a compressed representation of data (e.g., picture data) where similar data points are closer to each other within the latent space. The latent space is useful for learning data features and finding a simpler representation of the data for analysis.

[0133] The quantized latent representation T, ^y and the side information ^z of hyperplier 903 are included in bitstream 2 using arithmetic coding (AE) (binariconstituted), as shown in Figure 9.

[0134] Furthermore, a decoder 904 is provided that converts the quantized latent representation into a reconstructed image ^x, where ^x = g(^y). The signal ^x is an estimate of the input image x. It is desirable that x be as close to ^x as possible, in other words, that the reconstruction quality be as high as possible. However, the higher the similarity between ^x and x, the more side information needs to be transmitted. Such side information includes bitstream 1 and bitstream 2 shown in Figure 9, which are generated by the encoder and transmitted to the decoder. Generally, the more side information there is, the higher the reconstruction quality. However, a large amount of side information means a lower compression ratio. Therefore, one objective of the system described in Figure 9 is to balance the reconstruction quality with the amount of side information transmitted in the bitstream.

[0135] In Figure 9, component AE 605 is the arithmetic encoding module, which converts samples of the quantized latent representation ^y and side information ^z into a binary representation bitstream 1. The samples of ^y and ^z can include, for example, integers or floating-point numbers. One purpose of the arithmetic encoding module is to convert the sample values ​​into a string of binary digits (through the process of binaridization) (the string of binary digits is then included in the bitstream, which may contain further portions corresponding to the encoded image or further side information).

[0136] Arithmetic decoding (AD) 906 is a process that reverses the binary conversion process, converting binary digits back to their original sample values. Arithmetic decoding is provided by the arithmetic decoding module 906.

[0137] Please note that this disclosure is not limited to this particular framework. Furthermore, this disclosure is not limited to image or video compression, but can also be applied to object detection, image generation, and recognition systems.

[0138] In Figure 9, there are two interconnected subnetworks. In this context, a network is a logical division between parts of an entire network. For example, in Figure 9, the processing units (modules 901, 902, 904, 905, 906) are called an autoencoder / decoder or simply an "encoder / decoder" network. In other words, a network can be defined using processing units (modules) connected to enable a function. For the connected modules 901, 902, 904, 905, and 906 in Figure 9, each network performs the function of encoding and decoding the input picture x (e.g., an input tensor). Thus, the "encoder / decoder" network (the first network) in the example in Figure 9 is responsible for encoding (generating) and decoding (parse) the first bitstream, "bitstream 1". Meanwhile, the connected processing units (modules) 903, 908, 909, 910, and 907 form another network (the second network), which is sometimes called a "hyperencoder / decoder" network. The second network is responsible for encoding (generating) and decoding (parse) the second bitstream, "Bitstream 2". In the example in Figure 9, the first bitstream contains encoded picture data ^y, and the second bitstream contains side information ^z. Thus, the two networks have different purposes. Any of the processing units (modules) in the first and second networks may themselves be networks called subnets, that is, a particular module is part of a (larger) network. For example, any of modules 901, 902, 904, 905, and 906 in Figure 9 are subnets of the first network. Similarly, any of modules 903, 908, 909, 910, and 907 are subnets of the second network. Within each network, each processing unit (module) performs specific functions as needed to accomplish the processing of the entire first and second networks, respectively.In the example shown in Figure 9, the functions are the encoding-decoding of picture data (first network) and the encoding-decoding of side information (second subnetwork). Furthermore, connected modules 901, 902, and 905 may be considered an encoder subnetwork (i.e., a subnetwork of an encoder-decoder network), and modules 904 and 906 may be considered a decoder subnetwork. As is clear from the above description, the first and second networks may each be interpreted as subnetworks for the entire network containing all processing units.

[0139] ■The first subnetwork is responsible for the following: ● Transformation of input image x to its latent representation y (which is easier to compress than x) 901 ●Quantize latent expression y into quantized latent expression ^y. ● Using AE with the arithmetic encoding module 905, compress the quantized latent representation ^y to obtain the bitstream "bitstream 1". ● Parse bitstream 1 via AD using arithmetic decoding module 906. ● Reconstruct the reconstructed image (^x) using the parsed data.

[0140] The purpose of the second subnetwork is to obtain the statistical properties of the samples of "bitstream 1" (e.g., mean, variance, and correlation between samples of bitstream 1) so that the compression of bitstream 1 by the first subnetwork becomes more efficient. The second subnetwork generates a second bitstream, "bitstream 2," which contains the aforementioned information (e.g., mean, variance, and correlation between samples of bitstream 1).

[0141] The second network includes an encoding section which includes converting the quantized latent representation ^y into side information z 903, quantizing the side information z into quantized side information ^z, and encoding (e.g., binariconstituting) 909 the quantized side information ^z into bitstream 2. In this example, binariconstitution is performed by arithmetic coding (AE). The decoding section of the second network includes arithmetic decoding (AD) 910 which converts the input bitstream 2 into decoded quantized side information ^z'. Since arithmetic encoding and decoding are lossless compression methods, ^z' may be identical to ^z. The decoded quantized side information ^z' is then converted into decoded side information ^y' 907. ^y' represents a statistical property of ^y (e.g., the mean of ^y samples or the variance of the sample values). The decoded latent representation ^y' is then provided to the arithmetic encoder 905 and arithmetic decoder 906 described above to control the probabilistic model of ^y.

[0142] Figure 9 illustrates an example of a VAE (Variational Autoencoder), the details of which may differ in different implementations. For example, in one particular implementation, additional components may exist to more efficiently obtain the statistical properties of the samples of bitstream 1. In one such implementation, there may be a context modeler that targets the extraction of cross-correlation information of bitstream 1. The statistical information provided by the second subnetwork may be used by the AE (Arithmetic Encoder) 905 and AD (Arithmetic Decoder) 906 components.

[0143] Figure 9 depicts an encoder and decoder in a single figure. As will be apparent to those skilled in the art, encoders and decoders may, and very often, be embedded in different devices, as illustrated in Figures 9A and 9B.

[0144] Figure 9A shows the encoder component of the VAE framework alone, and Figure 9B shows the decoder component of the VAE framework alone. As input, the encoder receives a picture (picture data) according to some embodiments. The input picture may include one or more channels, such as a color channel or other types of channels, for example, a depth channel or a motion information channel. The outputs of the encoder (as shown in Figure 9A) are bitstream 1 and bitstream 2. Bitstream 1 is the output of the encoder's first subnetwork, and bitstream 2 is the output of the encoder's second subnetwork.

[0145] Similarly, in Figure 9B, two bitstreams, bitstream 1 and bitstream 2, are received as input, and the reconstructed (decoded) image ^x is produced as output.

[0146] As described above, a VAE can be divided into different logical units that perform different actions. This is illustrated in Figures 9A and 9B, where Figure 9A shows the components that participate in encoding a signal, such as video, and the encoded information that is provided. This encoded information is then received by the decoder component shown in Figure 9B, for example, encoding. Thus, the same reference numerals in Figures 9, 9A, and 9B indicate that the respective processing units (modules) perform the same function.

[0147] Specifically, as shown in Figure 9A, the encoder includes an encoder 901 that converts input x into a signal y, which is then provided to a quantizer 902. The quantizer 902 provides information to the arithmetic encoding module 905 and the hyperencoder 903. The hyperencoder 903 provides the bitstream 2 already described above to the hyperdecoder 907, which then signals the information to the arithmetic encoding module 605.

[0148] The output of the arithmetic encoding module is bitstream 1. Bitstream 1 and bitstream 2 are the outputs of the encoded signal, which are then provided (sent) to the decoding process.

[0149] Unit 901 is called the “encoder,” but the entire subnetwork shown in Figure 9A can also be called the “encoder.” The encoding process generally refers to a unit (module) that transforms an input into an encoded (e.g., compressed) output. From Figure 9A, we can see that unit 901 can actually be considered the core of the entire subnetwork, as it transforms the input x into a compressed version of x, which is y. Compression in encoder 901 may be achieved, for example, by applying a neural network, or any processing network that generally has one or more layers. In such a network, compression may be performed by cascaded processing (i.e., sequential processing) that includes downsampling, which reduces the size and / or number of channels in the input. Thus, the encoder may be called, for example, a neural network (NN) based encoder.

[0150] The remaining parts in the diagram (quantization unit, hyperencoder, hyperdecoder, arithmetic encoder / decoder) are all responsible for improving the efficiency of the encoding process or converting the compressed output y into a sequence of bits (bitstream). Quantization may be provided to further compress the output of the NN encoder 901 by lossy compression. The AE 905, combined with the hyperencoder 903 and hyperdecoder 907 used to constitute the AE 905, may perform binarization, which can further compress the quantized signal by lossless compression. Thus, the entire subnetwork in Figure 9A can also be called the "encoder". The same applies to Figure 9B, where the entire subnetwork can also be called the "decoder".

[0151] Most deep learning (DL)-based image / video compression systems reduce the dimensionality of a signal before converting it to binary numbers (bits). For example, in the VAE framework, the encoder, which is a nonlinear transformation, maps the input image x to y, where y has smaller widths and heights than x. Since y has smaller widths and heights, and therefore smaller size, the dimensionality (size) of the signal is reduced, and thus it is easier to compress the signal y. It should be noted that, in general, encoders do not necessarily need to reduce the size of both (or generally all) dimensions. Rather, some exemplary implementations may provide encoders that reduce the size in only one dimension (or generally a subset of dimensions).

[0152] The general principle of compression is illustrated in Figure 8. The latent space, which is the output of the encoder and the input of the decoder, represents the compressed data. Note that the size of the latent space can be much smaller than the size of the input signal. Here, the term size can refer to the resolution, for example, the number of samples in the feature map(s) output by the encoder. The resolution may also be given as the product of the number of samples per dimension (e.g., width × height × number of channels of the input image or feature map).

[0153] The reduction of the input signal size is illustrated in Figure 8, which represents a deep learning-based encoder and decoder. In Figure 8, the input image x corresponds to the input data, which is the input to the encoder. The transformed signal y corresponds to the latent space, which has a smaller number of dimensions or a size in at least one dimension than the input signal. Each column of circles represents a layer in the processing chain of the encoder or decoder. The number of circles in each layer indicates the size or dimension of the signal in that layer.

[0154] Figure 8 shows that the encoding operation corresponds to reducing the size of the input signal, and the decoding operation corresponds to reconstructing the original size of the image.

[0155] One method for reducing signal size is downsampling. As mentioned above, downsampling is the process of reducing the sampling rate of an input signal. For example, if the input image has sizes h and w, and the output of downsampling is h2 and w2, then at least one of the following holds: h2 <h w2 <w

[0156] Signal size reduction typically occurs stepwise, not all at once, along a chain of processing layers. For example, if an input image x has dimensions h and w (representing height and width) and a latent space y has dimensions h / 16 and w / 16, then size reduction may occur in four layers during encoding, with each layer reducing the signal size by factor 2 in each dimension.

[0157] Some deep learning-based video / image compression methods utilize multiple downsampling layers. For example, the VAE framework shown in Figure 6A uses six downsampling layers, marked as 601-606.

[0158] Layers that include downsampling are indicated by a downward arrow ↓ in the layer description. The layer description "Conv Nx5x5 / 2↓" means that the layer is a convolutional layer with N channels and a convolutional kernel of size 5x5. As mentioned above, 2↓ means that downsampling of factor 2 is performed in this layer. As a result of downsampling of factor 2, one of the dimensions of the input signal is reduced by half in the output. In Figure 6A, 2↓ indicates that both the width and height of the input image are reduced by half. Since there are six downsampling layers, if the width and height of the input image 814 (also indicated by x) are given by w and h, the output signal ^z 813 will have widths and heights equal to w / 64 and h / 64, respectively.

[0159] The modules indicated by AE and AD are the arithmetic encoder and arithmetic decoder, as already described above with respect to Figures 9, 9A, and 9B. The arithmetic encoder and decoder are specific implementations of entropy coding. AE and AD (as part of components 613 and 615 in Figures 6A and 6B) may be replaced by other means of entropy coding. In information theory, entropy encoding is a lossless data compression method used to convert the values ​​of symbols into a binary representation, and this is an invertible process. Also, "Q" in the figures corresponds to the quantization operation described above in relation to Figures 6A and 6B, and further explained above in the "Quantization" section. Furthermore, the quantization operation and the corresponding quantization unit as part of component 613 or 615 are not necessarily present and / or may be replaced by other units.

[0160] Figures 6A and 6B also show decoders including upsampling layers 607-612. Between the upsampling layers 611 and 610 in the input processing sequence, there is a further layer 620, implemented as a convolutional layer but not providing upsampling to the received input. The corresponding convolutional layer 620 is also shown for the decoder. Such layers may be provided in the NN to perform operations on the input that modify specific characteristics without changing the size of the input. However, such layers are not required.

[0161] Viewed in the order of processing bitstream 2 as it passes through the decoder, the upsampling layers are executed in reverse order, i.e., from upsampling layer 612 to upsampling layer 607. Here, each upsampling layer is shown to provide upsampling with an upsampling ratio of 2, indicated by ↑. Of course, it is not necessarily true that all upsampling layers have the same upsampling ratio, and other upsampling ratios such as 3, 4, 8 may also be used. Layers 607-612 are implemented as convolutional layers (conv). Specifically, they may be intended to provide an operation on the input that is the reverse of the encoder operation, so the upsampling layers may apply a deconvolution operation to the received input so that its size is increased by a factor corresponding to the upsampling ratio. However, this disclosure is not limited in general to deconvolution, and upsampling may be performed in any other way, such as by bilinear interpolation between two neighboring samples, or by nearest neighbor sample copying, or similar.

[0162] In the first subnetwork, several convolutional layers (601-603) are followed by generalized division normalization (GDN) on the encoder side and inverse GDN (IGDN) on the decoder side. In the second subnetwork, the activation function applied is ReLU. Note that this disclosure is not limited to such implementations, and in general, other activation functions may be used instead of GDN or ReLU.

[0163] Figure 6B shows another example of a VAE-based encoder-decoder structure similar to that in Figure 6A. Figure 6B shows that the encoder and decoder may include several downsampling and upsampling layers. Each layer applies downsampling or upsampling by factor 2. Furthermore, the encoder and decoder may include additional components such as a general division normalization (GDN) 650 on the encoder side and an inverse GDN (IGDN) 655 on the decoder side. Additionally, both the encoder and decoder may include one or more ReLUs, specifically leaky ReLUs 660 and 665. The encoder may also be provided with a factorized entropy model, and the decoder with a Gaussian entropy model 670. Multiple convolutional masks 680 may also be provided. Furthermore, in the embodiment of Figure 6B, the encoder includes a universal quantizer (UnivQuan), and the decoder includes an attention module.

[0164] The total number of stride and downsampling operations defines the condition regarding the input channel size, i.e., the size of the input to the neural network.

[0165] Here, if the input channel size is an integer multiple of 64 = 2 × 2 × 2 × 2 × 2 × 2, the channel size remains an integer after all ongoing downsampling operations. By applying the corresponding upsampling operation in the decoder during upsampling, and by applying the same rescaling at the end of processing the input through the upsampling layer, the output size becomes identical again to the input size in the encoder.

[0166] This allows for a reliable reconstruction of the original input.

[0167] Receptive field: In the context of neural networks, the receptive field is defined as the size of the region in the input that generates a sample in the output feature map. Essentially, it is a measure of the association between the output features (of any layer) and the input region (patch). Note that the concept of the receptive field applies to local operations (i.e., convolution, pooling, etc.). For example, a convolution operation with a 3x3 kernel has a 3x3 sample receptive field in the input layer. In this example, nine input samples are used by the convolution node to obtain one output sample.

[0168] Total receptive field: The total receptive field (TRF) refers to the set of input samples used to obtain a specified set of output samples, for example, by applying one or more processing layers of a neural network.

[0169] The entire receptive field can be illustrated by Figure 10. Figure 10 illustrates the processing of a one-dimensional input (seven samples on the left side of the figure) with two consecutive transposed convolution (also called deconvolution) layers. The input is processed from left to right, i.e., “deconv layer 1” processes the input first, and its output is processed by “deconv layer 2”. In this example, the kernel has a size of 3 in both deconvolution layers. This means that three input samples are needed to obtain one output sample in each layer. In this example, the set of output samples is marked inside the dashed rectangle and contains three samples. Due to the size of the deconvolution kernel, seven samples are needed in the input to obtain an output set of samples containing three output samples. Therefore, the entire receptive field of the three marked output samples is seven samples in the input.

[0170] In Figure 10, there are 7 input samples, 5 intermediate output samples, and 3 output samples. The reduction in the number of samples is due to the fact that there are "missing samples" at the boundaries of the input because the input signal is finite (it does not extend infinitely in each direction). That is, since the deconvolution operation requires 3 input samples corresponding to each output sample, if the number of input samples is 7, only 5 intermediate output samples can be generated. In fact, the number of output samples that can be generated is (k-1) fewer than the number of input samples, where k is the kernel size. In Figure 10, since the number of input samples is 7, after the first deconvolution with kernel size 3, the number of intermediate samples is 5. After the second deconvolution with kernel size 3, the number of output samples is 3.

[0171] As can be seen from Figure 10, the total receptive field of the three output samples is the seven samples in the input. The size of the total receptive field increases by successively applying processing layers with kernel sizes greater than 1. In general, the total receptive field of a set of output samples is calculated by tracing the connections of each node from the output layer to the input layer, and then finding the union of all samples in the input that are directly or indirectly (through two or more processing layers) connected to the set of output samples. For example, in Figure 10, each output sample is connected to three samples in the previous layer. The union includes five samples in the intermediate output layer, which are connected to the seven samples in the input layer.

[0172] Sometimes it is desirable to keep the number of samples the same after each operation (convolution, deconvolution, or other). In such cases, padding can be applied to the boundaries of the input to compensate for "missing samples". Figure 11 illustrates this case where the number of samples is kept equal. Note that this disclosure is applicable to both cases, as padding is not a required operation for convolution, deconvolution, or any other processing layer.

[0173] This should not be confused with downsampling. In the process of downsampling, for every M samples, there are N samples in the output, where N < M. The difference is that M is typically much smaller than the number of inputs. In Figure 10, there is no downsampling; rather, the reduction in the number of samples results from the fact that the input size is not infinite and there are "missing samples" in the input. For example, if the number of input samples is 100 and the kernel size is k = 3, when two convolutional layers are used, the number of output samples would be 100 - (k - 1) - (k - 1) = 96. In contrast, if both transposed convolutional layers were performing downsampling (at a ratio of M = 2 and N = 1), the number of output samples would be [Number] as would be.

[0174] Figure 12 illustrates downsampling using two convolutional layers with a downsampling ratio of 2 (N = 1 and M = 2). In this example, seven input samples become three due to the combined effect of downsampling and "missing samples" at the boundaries. The number of output samples can be calculated using the formula ceil((100 - (k - 1)) / r) after each processing layer, where k is the kernel size and r is the downsampling ratio.

[0175] The operations of convolution and transposed convolution (i.e., deconvolution) are identical from a mathematical representation perspective. The difference comes from the fact that the transposed convolution operation assumes that the previous convolution operation has been performed. In other words, deconvolution is the process of filtering the signal to compensate for the previously applied convolution. The purpose of deconvolution is to reproduce the signal that existed before the convolution was performed. This disclosure applies to both the convolution operation and the transposed convolution operation (in fact, any other operation where the kernel size is greater than 1, as will be explained later).

[0176] Figure 13 shows another example illustrating how to calculate the total receptive field. In Figure 13, a two-dimensional input sample array is processed by two convolutional layers, each with a kernel size of 3x3. After applying the two deconvolutional layers, an output array is obtained. The set (array) of output samples is marked by a solid rectangle ("output samples") and contains 2x2=4 samples. The total receptive field of this set of output samples contains 6x6=36 samples. The total receptive field can be calculated as follows: Each output sample is connected to a 3x3 sample in the intermediate output. The union of all samples in the intermediate output connected to the set of output samples contains 4x4 = 16 samples. Each of the 16 samples in the intermediate output is connected to a 3x3 sample in the input. The union of all the input samples connected to the 16 samples in the intermediate output contains 6x6 = 36 samples. Therefore, the total receptive field of the 2x2 output sample is the 36 samples in the input.

[0177] In image and video compression systems, the compression and decompression of very large input images is typically performed by dividing the input image into multiple parts. VVC and HEVC employ such a division method, for example, by dividing the input image into tiles or wavefront processing units.

[0178] When tiles are used in traditional video coding systems, the input image is typically divided into multiple rectangular sections. Figure 14 illustrates one such division. In Figure 14, sections 1 and 2 may be processed independently of each other, and the bitstreams for decoding each section are encapsulated into independently decodeable units. As a result, the decoder can independently parse each bitstream (corresponding to sections 1 and 2) to obtain the syntax elements necessary for sample reconstruction, and can also independently reconstruct the samples of each section.

[0179] In wavefront parallel processing shown in Figure 15, each part typically consists of a single-row coded tree block (CTB). The difference between wavefront parallel processing and tiling is that in wavefront parallel processing, the bitstreams corresponding to each part can be decoded almost independently of each other. However, sample reconstruction of each part still has dependencies between parts, so sample reconstruction cannot be performed independently. In other words, wavefront parallel processing makes the parsing process independent while keeping sample reconstruction dependent.

[0180] Wavefronts and tiling are both techniques that allow the decoding operation to be performed independently of each other, either as a whole or as a part of it. The advantages of independent processing are as follows: • Two or more identical processing cores can be used to process the entire image. This allows for increased processing speed. If the processing cores are not powerful enough to process a large image, the image may be divided into multiple parts, each requiring fewer resources for processing. In this case, less powerful processing units can process each part, even if they cannot process the entire image due to resource limitations.

[0181] To meet the demands of processing speed and / or memory, HEVC / VVC uses processing memory large enough to handle the encoding / decoding of an entire frame. To achieve this, top-of-the-line GPU cards are used. In the case of traditional codecs such as HEVC / VVC, the entire frame is divided into blocks, and each block is processed one at a time, so the memory requirements for processing an entire frame are usually not a major issue. However, processing speed is a major concern. Therefore, if a single processing unit is used to process an entire frame, the speed of the processing unit must be very fast, and thus the processing unit is usually very expensive.

[0182] On the other hand, NN-based video compression algorithms consider the entire frame during encoding / decoding, instead of the block-based approach used in conventional hybrid coders. However, the memory requirements are too high to handle via NN-based encoding / decoding modules.

[0183] In traditional hybrid video encoders and decoders, the amount of memory required is proportional to the maximum allowable block size. For example, in VVCs, the maximum block size is 128 x 128 samples.

[0184] However, the memory required for NN-based video compression is proportional to the size W × H, where W and H represent the width and height of the input / output image. Since typical video resolutions include a 3840 × 2160 picture size (4K video), it can be seen that the memory requirements can be very high compared to hybrid video coders. In image and video compression systems, compression and decompression of very large input images are usually performed by dividing the input image into multiple parts. To address memory constraints, NN-based video coding algorithms can apply tiling in latent space.

[0185] When tiling is applied without overlap, boundary artifacts may be visible in the reconstructed image. This problem may be partially solved by overlapping tiles in latent space or signal region, where the overlap is large enough to avoid these artifacts. If the overlap is larger than the size of the receptive field of the NN, the operation can be performed in a non-normative manner, which may introduce some overhead to the computational complexity. If the overlap is smaller than the size of the receptive field, the tiling operation must be specified (normative) rather than lossless / transparent.

[0186] Furthermore, in structures with multiple pipelines (e.g., pipelines for processing lumers and / or chromers, or generally multiple channels of an input tensor) or multiple subnetworks, a simple approach where tiles are always the same size can result in performance loss. Rate-Distortion Optimized Quantization (RDOQ) (e.g., Figure 9: Unit 908) is another computationally complex and memory-intensive operation. Choosing the same tile size within RDOQ for all subnetworks may also not be optimal. Also, if scene representations in different pipelines are not sample-aligned during tiling (e.g., CCS), simple tiling may not preserve information about correlations along different components. Sample-aligned means that the size of the lumers does not match the size of the chromers. In such cases, video processing may perform poorly because there is no spatial and / or temporal correlation, or at least it is not very pronounced. As a result, the quality of the reconstructed image may be reduced.

[0187] Another problem is that in order for a single processing unit (e.g., a CPU or GPU) to handle a large input, the processing unit must be extremely fast, as it needs to perform a large number of calculations per unit of time. This requires the unit to have a high clock frequency and high memory bandwidth, which are expensive design standards for chip manufacturers. Increasing memory bandwidth and clock frequency is not easy, especially due to physical limitations.

[0188] While the latest deep learning-based image and video compression algorithms follow the variational autoencoder (VAE) framework, NN-based video coding algorithms for encoding and / or decoding are still in the early stages of development, and there are no consumer devices, including VAE implementations, as shown in Figures 9, 9A, and 9B. Furthermore, the cost of consumer devices is highly sensitive to the memory they utilize.

[0189] For NN-based video coding algorithms to become cost-effective enough to be implemented in consumer devices such as mobile phones, it is therefore necessary to reduce the memory footprint and the operating frequency required for processing units. Such optimizations have not yet been performed.

[0190] This disclosure is applicable to both end-to-end AI codecs and hybrid AI codecs. In hybrid AI codecs, for example, filtering operations (filtering of reconstructed pictures) may be performed by a neural network (NN). This disclosure is applicable to such NN-based processing modules. In general, this disclosure may be applicable to all or part of a video compression and decompression process if at least part of the process includes an NN and such NN includes convolution or transposed convolution operations. For example, this disclosure is applicable to individual processing tasks that are performed as processing parts by encoders and / or decoders, including in-loop filtering, post-filtering, and / or pre-filtering, as well as encoder-only rate-distortion-optimized quantization (RDOQ).

[0191] Some embodiments of this disclosure may provide solutions to the aforementioned problems with respect to enabling the trade-off between memory resources and computational complexity within NN-based video encoding-decoding frameworks. In particular, this disclosure provides the possibility of processing parts of the input independently while ensuring that different image components are sample-aligned. This reduces memory requirements while maintaining compression performance and involves little additional computational complexity.

[0192] The processing can be decoding or encoding. The neural network (NN) in the exemplary implementation described below may be as follows: A network comprising at least one processing layer in which two or more input samples are used to obtain an output sample (this is a common condition under which the problem addressed in this disclosure occurs). A network containing at least one single convolutional (or transposed convolutional) layer. In one example, the convolution kernel is greater than 1. A network that includes at least one pooling layer (e.g., max pooling, average pooling). • Decode network, hyperdecoder network, or encode network. • A portion of the above (subnetwork).

[0193] The aforementioned input may be as follows: • Feature map. • Output of the hidden layer. • Latent space feature map. The latent space can be obtained according to the bitstream. • Input image. [Examples]

[0194] First Embodiment The following describes a method and apparatus for picture / video encoding-decoding (compression-decompression) in which multiple subnetworks are used to process an input tensor representing picture data, as shown in Figure 16.

[0195] In this exemplary and non-limiting embodiment, a method is provided for encoding an input tensor representing picture data. The input tensor may have a matrix form with a spatial dimension of width = w, height = h, and a third dimension (e.g., the number of channels) equal in size D. For example, the input tensor may be the input image itself having D components, which may include one or more color components and possibly further channels such as depth channels or motion channels. However, the disclosure is not limited to such inputs. In general, the input tensor may be a representation of the picture data, for example, a latent representation that may be the result of a previous process (e.g., preprocessing).

[0196] The input tensor is processed by a neural network that includes at least a first subnetwork and a second subnetwork. An example of the first and / or second subnetwork for encoding branching is shown in Figure 16, which includes an encoder 1601 and a rate-distortion-optimized quantizer (RDOQ) 1602. The processing involves applying the first subnetwork to the first tensor, which involves dividing the first tensor into a first set of tiles in the spatial dimension and processing the first set of tiles by the first subnetwork. After applying the first subnetwork, the second subnetwork is applied to the second tensor, which involves dividing the second tensor into a second set of tiles in the spatial dimension and processing the second set of tiles by the second subnetwork.

[0197] In the example in Figure 16, the output of the first subnetwork 1601 is provided as input to the second subnetwork, which is RDOQ 1602. In this case, the first tensor is the input image x, which represents picture data that may be raw picture data. The second tensor then input to the second subnetwork is a feature tensor in latent space. However, this disclosure is not limited to the case where the first and second subnetworks are directly cascaded. Generally, the second subnetwork is after the first subnetwork, i.e., applied after the first network has been applied, but there may be some additional processing between the first and second subnetworks. Thus, the term “after” above does not limit the processing to immediately following the output of the first subnetwork being directly input to the second subnetwork. Rather, “after” means that multiple tiles of the first and second are processed, for example, within the same processing pipe.

[0198] One or more channels of the first and second tensors are partitioned into so-called tiles, which essentially represent data obtained by partitioning the input tensor into one or more spatial dimensions. Like current video coding standards, the tiles are intended to provide the possibility of decoding in parallel, i.e., independently of each other. A tile may contain one or more samples. The tile may have a rectangular shape, but is not limited to such a regular shape. The rectangular shape may also be a square shape. An example of partitioning into regularly shaped tiles is shown in Figure 14. However, this disclosure is not limited to rectangles, and especially squares. For example, the shapes may be checkerboard or irregular. The partitioning of the first and second input tensors may be done such that the tiles have triangular or any other shape, which may depend on the type of processing performed by the particular application and / or subnetwork.

[0199] In Figure 16, a general processing of N components is shown in subnetworks 1601 and 1602, where the components can be input tensor channels (e.g., color components or latent space representations) that can be processed in parallel. Such processing may involve dividing the channels into tiles (containing a first set of tiles) in the spatial domain.

[0200] However, the N components in Figure 16 may correspond to each of the N tiles (or tile groups) of one or more channels. The N tiles (or tile groups) may be processed in parallel. After such processing of the first and / or second sets of tiles, each subnetwork may merge the processed tiles into an output tensor. In Figure 16, such a merged output tensor is y for subnetwork encoder 1601 and ^y for subnetwork RDOQ 1602. Note that in Figure 16, the number of components in the first subnetwork 1601 and the second subnetwork 1602 is the same (N). However, this is not necessarily the case. The number of parallel processing pipes in the first subnetwork may be different from the number of parallel processing pipes in the second subnetwork. For example, in the case of post-filtering, the first subnetwork is a post-filter that processes each component (e.g., Y, U, and V) separately. Next, in the case of encoding or decoding, the encoder or decoder, respectively, is a second subnetwork, creating a difference between Y and UV. That is, UV is processed collectively.

[0201] Furthermore, at least two copositional tiles from each of the first and second plurality of tiles are of different sizes. Here, the term “copositional” means that the two tiles (i.e., one from the first plurality and one from the second plurality) are in corresponding (e.g., at least partially overlapping) positions in the first and second input tensors in the spatial dimension. In other words, the subdivision into tiles may be different for each subnetwork, and thus each subnetwork may use a different tiling (different division into tiles).

[0202] In one exemplary implementation, the inputs to each subnetwork (i.e., the first and second tensors) are divided (in spatial domain) into grids of tiles of the same size, except for the tiles at the bottom and right image boundaries, which can be smaller in size, since the input tensors do not necessarily have sizes that are integer multiples of the tile size. Such grids of tiles of the same size can be advantageous due to the possibility of efficient signaling of such grids in the bitstream and lower processing complexity. On the other hand, grids that may contain tiles of different sizes may result in better performance and content adaptability.

[0203] In the exemplary implementation described above, the tiles of a first set of adjacent tiles in at least one spatial dimension partially overlap in that at least one spatial dimension. Additionally or alternatively, the tiles of a second set of adjacent tiles in at least one spatial dimension partially overlap in that at least one spatial dimension. The term adjacent means that each tile is neighboring. Adjacent tile sets 1 and 2 are shown in Figure 14, which are adjacent but do not overlap. Figure 17A shows partial overlap, where the first tensor is divided into four tiles in 2D in the xy-plane (i.e., regions L1, L2, L3, and L4). A similar consideration may be applicable to a second input tensor. The first tile of the first set of tiles is L1, and the second tile is L2. L1 and L2 have boundaries that are adjacent to each other in the x-axis direction and overlap along the y-axis. As shown, L1 and L2 partially overlap. Partial overlap means that tiles L1 and L2 contain one or more of the same tensor elements. In some embodiments, these tensor elements may correspond to picture samples for the first input tensor.

[0204] Figure 17B shows the same partial overlap scenario in the x and y directions as in Figure 17A. In both Figures 17A and 17B, L1 also overlaps with its adjacent tile to the right (in the y dimension; the boundary overlaps along the x dimension). L1 also slightly overlaps with its diagonally adjacent tile (in both dimensions). Similarly, L2 overlaps with both its directly adjacent tile to the right and its diagonally adjacent tile. Figure 18 shows another example of partial overlap for tiles L1 and L2, which can be tiles of the first set of tiles. L1 and L2 have overlap with other tiles in only one dimension each. L1 partially overlaps with its adjacent tile to the right, with its boundary along the x-axis (i.e., partial overlap in the y dimension). L2 then partially overlaps with its adjacent tile L1 at the top. In the examples in Figures 17A and 17B, the overlap between L1 and L2 meant that L1 also included samples from L2, and L2 also included samples from L1. In Figure 18, L2 also includes samples from L1, but L1 does not include samples from L2. As will be apparent to those skilled in the art, further variations of the overlap may exist. This disclosure is not limited to any particular way or extent of overlap. It may also include tile arrangements without overlap.

[0205] Figure 19 shows further examples of L1 and L2 with no overlap with each other or other tiles. Figure 20 is similar to Figures 17A and 17B, and further details regarding the partial overlapping areas are described below.

[0206] In one exemplary implementation, tiles of a first set of tiles (such as L1 and L2) are processed independently by the first subnetwork. Additionally or alternatively, tiles of a second set of tiles are processed independently by the second subnetwork. In other words, the processing of tiles is independent of each other and therefore independent of each other. Independent processing provides the possibility of parallelization. For example, in some implementations, at least two tiles of the first set of tiles are processed in parallel by the first subnetwork, and / or at least two tiles of the second set of tiles are processed in parallel by the second subnetwork. Parallel processing is shown in Figure 16 and includes processing 1 to N (processing pipes 1 to N) in the subnetwork encoder 1601 or quantizer RDOQ 1602. Using encoder subnetwork 1601 as the first subnetwork, encoder 1601 receives an input tensor x which is divided into N tiles x1 to xN of the first input tensor. Each tile is processed by its own block, i.e., by processes 1 to N, which do not need to interact with each other (e.g., wait for each other during processing). The result of each process is an output tensor y1 to yN for each tile, which may be a feature map in latent space. The output tensors y1 to yN may be further combined into an output tensor y. This combination may (but does not need to) include cropping as shown in Figures 17 to 19. Note that the combination into tensor y does not need to be performed. A second subnetwork may reuse the tiling of the first subnetwork and simply modify it (either by further subdividing the tiles to make the tiling finer, or by combining multiple tiles into one to make the tiling coarser). In the example in Figure 16, processes 1 to N may process on a tile-by-tile basis (i.e., process i processes tile i). Alternatively, process i may process component i of multiple components of the input tensor. In this case, process i divides component i into multiple tiles and processes the tiles separately or in parallel.

[0207] In the above, for simplicity, parallel processing of all N input tensor tiles by each of the N processing pipes was illustrated. However, the present disclosure is not limited to such parallel processing. There may be more than N tiles in the input tensor, and it is divided into N tile groups that are processed in parallel within each of the N processing pipes (instances of the first subnetwork and / or instances of the second subnetwork). As will be apparent to those skilled in the art, once the tiles become independent of each other, their processing can, in principle, be parallelized. Those skilled in the art can design any number of parallel processing pipes according to their respective performance requirements and / or hardware availability.

[0208] As shown in FIGS. 17A / B to FIGS. 20, each tile has a certain size, which may have a certain width and a certain height, and they may be different from each other in the case of rectangular-shaped tiles, or the same in the case of square-shaped tiles. In one implementation, dividing the first tensor includes determining the tile size within the first plurality of tiles based on a first predetermined condition, and / or dividing the second tensor includes determining the tile size within the second plurality of tiles based on a second predetermined condition. For example, the first predetermined condition and / or the second predetermined condition are based on available decoder hardware resources and / or motion present in the picture data. As an example of available hardware resources, the first and / or second predetermined conditions may be the memory resources of the processing device (decoder or encoder). When the available amount of memory is less than a predetermined value (the amount of memory resources), the determined tile size may be smaller than the tile size when the available memory is greater than or equal to the predetermined value. However, the hardware resources are not limited to memory. The first and / or second conditions may be based on the availability of processing capabilities, for example, the number of processors and / or the processing speed of one or more processors.

[0209] Alternatively or additionally, the presence of motion may be used in the first and / or second conditions. For example, the tile size may be determined to be smaller when there is more motion in the input tensor portion corresponding to the tile than when the presence of motion is less prominent or when there is no motion at all. Whether the motion is prominent (i.e., fast motion and / or fast / frequent change in motion) is determined by each motion vector with respect to changes in its magnitude and direction and may be compared to corresponding predetermined values (thresholds) for magnitude and / or direction and / or frequency.

[0210] An alternative or additional condition may be a region of interest (ROI), where the tile size may be determined based on the presence of the ROI. For example, the ROI in the picture data may be a detected object (e.g., a vehicle, bicycle, motorcycle, pedestrian, animal, etc.). The objects may have different sizes, move fast or slow, and / or change the direction of their motion fast or slow and / or many times (i.e., more frequently than a predetermined frequency value). In one exemplary implementation, the size or tile in at least one dimension may be smaller for the ROI than for the rest of the input tensor.

[0211] Thus, the tile size may be adapted or optimized to the hardware resources or to the content of the picture data including scene-specific tile sizes. It is also possible to jointly adapt or optimize the tile size to both the hardware resources and the content of the picture data.

[0212] Figures 17A / B to 20 illustrate the partitioning of a first (or second) tensor into multiple tiles that partially overlap adjacent tiles. As a result of overlapping and / or tiling, the processed tiles corresponding to region Ri can undergo cropping. In Figures 17A / B to 20, the subscripts L and R are the same for the corresponding partitions in the input and output. For example, L4 corresponds to R4. The placement of Ri follows the same pattern as Li, meaning that if L1 corresponds to the upper left of the input space, then R1 corresponds to the upper left of the output space. If L2 is to the right of L1, then R2 is to the right of R1. In the example in Figure 17A, the partitioning of the first tensor is such that each region L i Each of them is R i It includes the complete receptive field of R. i The union of these sets constitutes the entire target picture R.

[0213] The determination of the total receptive field depends on the kernel size of each processing layer. This can be determined by tracing the input samples of the first tensor backward in the reverse direction of processing. The total receptive field consists of the union of input samples used to compute the set of output samples. Therefore, the total receptive field depends on the connections between each layer and can be determined by tracing all connections starting from the output and moving in the direction of the input.

[0214] In the example of convolutional layers shown in Figure 13, the kernel sizes of convolutional layers 1 and 2 are K1×K1 and K2×K2, respectively, and the downsampling ratios are R1 and R2, respectively. Convolutional layers typically use a regular input-output connection (for example, always K×K input samples are used for each output). In this example, the size of the total receptive field can be calculated as follows: W = ((w × R²) + K² - 1) × R¹ + (K¹ - 1) H = ((h × R²) + K² - 1) × R¹ + (K¹ - 1) Here, H and W represent the size of the total receptive field, and h and w are the height and width of the output sample set, respectively.

[0215] In the above example, the convolution operation is described in two-dimensional space. If the number of dimensions in the space to which the convolution is applied is higher, a 3D convolution operation may be applied. A 3D convolution operation is a simple extension of a 2D convolution operation, in which an additional dimension is added to all aspects of the operation. For example, the kernel size can be expressed as K1 × K1 × N and K2 × K2 × N, and the total receptive field can be expressed as W × H × N (based on the previous example, where N represents the size of the third dimension). Since the extensions from 2D and 3D convolution operations are straightforward, the present invention applies to both 2D and 3D convolution operations. In other words, the size of the third (or even fourth) dimension may be greater than 1, and the present invention can apply in the same way.

[0216] The above equation is an example showing how the size of the total receptive field can be determined. The determination of the total receptive field depends on the actual input-output connections of each layer. The output of the encoding process is R i That is. R i The union of the elements is the set of elements y1 through y N This constructs a feature tensor that is merged (Figure 16). In this example, R i Since it has overlapping regions, the cropping operation is first applied to the R-crop which does not have overlapping regions. i This is obtained. Finally, R-crop i These are concatenated to obtain a merged feature tensor y. In this example, L i As shown above, R i This includes the entire receptive field.

[0217] R i and L i The determination of (i.e., the size of the first and / or second set of tiles) can be carried out as follows: First, N non-overlapping regions R-crop i Determine the R-crop. i These are N×M regions of equal size, where N×M is determined by the decoder according to the memory limit. ·R-cropi Determines the entire receptive field. i Each R-crop i It is set equally for each receptive field. ·Each L i Inserted R i This is obtained. i This means that L is the size of the output sample set generated by the NN. Note that actual processing is not necessary, once L i Once the size is determined, R follows the function i It may be possible to determine the size and position of the NN. Since the structure of the NN is known, the size L i and R i The relationship is known, and therefore, without actually performing the process, L i R i This can be calculated. ·Size R i R-crop i If not equal to R i Crop and R-crop i To obtain. ■When padding is applied to input samples or intermediate output samples during processing by NN, R-crop i The size is R i It may not be equal to. Padding can be applied to a neural network (NN) when certain size requirements must be met for the input and intermediate output samples. For example, an NN may require that the input size be a multiple of 16 samples (for example, due to its certain structure). In this case, L i If the sample size is not a multiple of 16 in one direction, padding may be applied in that direction to make it a multiple of 16. In other words, the padded samples are dummy samples to ensure integer multiples of each Li that have the size requirements of each NN layer. ■The cropped samples can be obtained by determining "the output samples that include the padded samples in that calculation." This option may be applicable to the exemplary implementation discussed.

[0218] This is shown in Figure 17A, where, after processing a first set of tiles (i.e., region Li) via a first subnetwork (e.g., encoder 1601 in Figure 16), the size Ri output by the first subnetwork may be too large and therefore not suitable for input to subsequent subnetworks, such as the second subnetwork RDOQ 1602 in Figure 16. Thus, the processed tiles R1 and R2 in Figure 17A undergo a cropping operation, after which, in this example, the cropped R-crop1 to R-crop4 are merged.

[0219] Figure 17B shows an example of partial overlap of regions L1-L4 (i.e., the first and / or second sets of tiles) without the involvement of cropping of the processed tiles (i.e., regions R1-R4). i and L i The decision can be made as follows, and is sometimes referred to as the “simple” non-cropping case shown in Figure 17B. First, we have N non-overlapping regions R. i Determine the following. For example, R i These may be N×M regions of equal size, where N×M is determined by the decoder according to the memory limit. ·R i Determines the entire receptive field. i Each of these is R i It is set to be equal to all receptive fields. All receptive fields are R i Each of the output samples inside is L i It is calculated by tracing in the reverse direction to L. i R i It consists of all samples used in at least one calculation of the samples within. ·Each L i Inserted R i This is obtained. i This means that this is the size of the output sample set generated by the neural network.

[0220] An exemplary implementation solves the total peak memory problem by dividing the input space into multiple smaller, independently processable regions.

[0221] In the example implementation above, L i Overlapping regions require additional processing compared to not dividing the input into regions. The larger the overlapping region, the more extra processing is required. In particular, in some cases, R i The total receptive field can sometimes be too large. In such cases, the total number of calculations required to obtain the entire reconstructed picture can become excessively large.

[0222] Figure 18 shows another example of partially overlapping tiles (i.e., regions L1-L4), which, like Figure 17A, includes cropping of the processed tiles R1-R4. Compared with Figures 17A and 17B, the input region L i (i.e., tiles) are each R i Since it represents only a subset of the entire receptive field, it is smaller here, as shown in Figure 18. Each region Li is processed independently by its respective subnetwork (e.g., encoder 1601 and / or RDOQ 1602 in Figure 16), thereby yielding two regions R1 and R2 (i.e., the output subset). Since a subset of the entire receptive field is used to obtain the set of output samples, a padding operation may be necessary to generate missing samples. In one exemplary implementation, processing by a first subnetwork of a first set of tiles and / or a second subnetwork of a second set of tiles may include padding before processing with one or more of the aforementioned layers. Thus, missing samples in the input subset may be added by the padding process, which improves the quality of the reconstructed output subset Ri. Thus, the quality of the reconstructed picture is also improved after combining the output subset Ri.

[0223] To reiterate, padding refers to increasing the size of an input (i.e., an input image) by generating new samples at the boundaries of an image (or picture) by using predefined sample values ​​or by using sample values ​​at a given location within the input image. This is illustrated in Figure 11. The generated samples are approximations of actual sample values ​​that do not exist. Thus, padded samples may be obtained, for example, based on one or more nearest neighbor samples of the sample to be padded. For example, a sample is padded by copying its nearest neighbor sample. If there are many more neighboring samples at the same distance, the neighboring samples used from among them may be specified by convention (e.g., by a standard). Another possibility is to interpolate the padding sample from multiple neighboring samples. Alternatively, padding may include using zero-value samples. Intermediate samples generated by processing may also need to be padded. Intermediate samples may be generated based on samples from an input subset containing the one or more padded samples. The padding may be performed before the input to the neural network or within the neural network. However, padding should be performed before processing the one or more layers.

[0224] For completeness, Figure 19 shows another example where, after the output subset Ri undergoes cropping, each cropped region Ri-crop is seamlessly merged without any overlap of the cropped regions. In contrast to the examples in Figures 17A, 17B, and 18, the regions Li do not overlap. The respective implementations may be as follows: 1. Determine N non-overlapping regions Li (i.e., multiple tiles of the first and second tensors) in the first and second tensors, respectively. Here, at least one of these regions is R i It includes one subset of the total receptive fields, R iThe union constitutes the complete output (e.g., the feature tensor y) of the first or second subnetwork. 2. Process each L i independently using a NN to obtain the region R i therefrom. 3. Merge R i to obtain the merged output tensor.

[0225] FIG. 19 shows this case, where Li is selected as a non-overlapping region. This is a special case where the total amount of computation required to obtain the entire reconstructed output (i.e., the output picture) is minimized. However, the reconstruction quality may be compromised.

[0226] As described above, the first and second subnetworks are part of a neural network. A subnetwork is itself a neural network and includes at least one processing layer. In one implementation, the first subnetwork performs processing by one or more layers including at least one convolutional layer and at least one pooling layer, and / or the second subnetwork performs processing by one or more layers including at least one convolutional layer and at least one pooling layer.

[0227] In one example implementation, the first subnetwork and the second subnetwork perform respective processing that is part of picture or video compression.

[0228] [[ID=2*]] Furthermore, the first and / or second subnetworks perform one of the following: picture encoding by a convolutional subnetwork, rate-distortion-optimized quantization (RDOQ), and picture filtering. As described above, the first and / or second subnetworks may be the encoding device (or encoding process) 901 or Q / RDOQ 902 in Figure 9, and the input image x is first processed by the encoder 901 that performs picture encoding. The encoder 901 may be part of a VAE encoder-decoder framework as shown in Figures 6A and 6B, and each encoder "ga" performs its respective picture encoding by processing the input image through a sequence of convolutional layers 601-604, including processing through a GDN layer. Similarly, the quantizer "Q" in Figures 6A and 6B may perform the function of quantization or RDOQ, which is the first and / or second subnetwork. The same applies to the units Q or RDOQ 902 and 908 in Figure 9. Picture post-filtering is not further shown in Figures 6A / B and 9. Generally, some of the processing layers of a neural network in a decoding device may have post-filtering or general filtering capabilities.

[0229] For example, Figure 16 shows an example of an encoding device that may accommodate a neural network, including an encoder subnetwork 1601 as a first subnetwork and an RDOQ subnetwork 1602 as a second subnetwork. However, the disclosure is not limited to such embodiments. The neural network may also be a network for decoding a picture or latent representation, and may include a decoding subnetwork and a post-filtering subnetwork. Further examples of subnetworks are possible, including preprocessing subnetworks or other subnetworks.

[0230] As shown in Figure 16, the encoding device (or encoding process in general) may further include a hyperencoder 1603, a quantizer 1608 for the output z of the hyperencoder 1603, and an arithmetic encoder 1609 for encoding the quantized information (^z) in the hyperplier so that it can be included in the bitstream 2.

[0231] The decoding device (or decoding process) may further include, correspondingly, an arithmetic decoder 1610 for hyperplier information, followed by a hyperdecoder 1607. The hyperplier portion of encoding and decoding may correspond to the VAE framework or a variation thereof relating to Figures 6A and 6B.

[0232] Figures 6A and 6B show encoders in an NN-based VAE framework having convolutional layers 601-606, respectively, where the size of each input image (picture data) is then reduced by 2. For example, the number of samples used by the neural network (NN) (i.e., samples of picture data) may depend on the kernel size of the first input layer of the NN. The one or more layers of the neural network (NN) may include one or more pooling layers and / or one or more subsampling layers. The NN may generate one sample of an output subset by pooling the multiple samples through the one or more pooling layers. Alternatively or additionally, one output sample may be generated by the NN through subsampling (i.e., downsampling) by one or more downsampling convolutional layers (e.g., convolutional layers 601-606 in Figure 6B). Pooling and downsampling may be combined to generate one output sample. Figure 12 shows downsampling using two convolutional layers, starting with seven samples from the entire receptive field and providing one sample as output.

[0233] In Figures 6A and 6B, the input image 614 corresponds to picture data. According to one implementation, the input tensor 614 is a picture or a sequence of pictures containing one or more components, at least one of which is a color component. Alternatively, the input tensor may be a latent space representation of the picture, which may be the output of a preprocessing (e.g., an output tensor). The one or more components are color components and / or depth and / or motion maps and / or other feature maps associated with the picture sample.

[0234] It should be noted that the input tensor may represent other types of data (e.g., any kind of multidimensional data) having one or more spatial components that are suitable for tiling and / or processing via subnetworks as described in the first embodiment.

[0235] In one implementation, the input tensor has at least two components, namely a first component and a second component, and the first subnetwork divides the first component into a third set of tiles, and the second component into a fourth set of tiles, and at least two of each of the third and fourth sets of tiles are of different sizes. In principle, this disclosure is not limited to any particular number of spatial components. There may be one spatial component (e.g., a grayscale picture). However, in this implementation, the input tensor has multiple spatial components. The first and second components may be color components. The tiling may differ not only for subnetworks but also for components processed by the same subnetworks.

[0236] Therefore, additionally or alternatively, the second subnetwork divides the first component into a fifth set of tiles and the second component into a sixth set of tiles, where at least two of each of the fifth and sixth sets of tiles are of different sizes. In other words, the tiling of the spatial components of the second input tensor can be different. Further details of the second embodiment in which the tiling differs for different spatial input components are provided below. The second embodiment can be combined with the first embodiment described herein in which the tiling differs for different subnetworks of the neural network.

[0237] The encoding in Figure 16 (as in Figures 6A and 6B) further includes generating a bitstream by including the output of processing by a neural network into the bitstream. The neural network may further include entropy coding. This is shown in Figure 6A, where, after encoding the input image 614 via an encoder neural network (convolutional layers 601-604), the output y of the encoder NN_ga undergoes quantization (e.g., RDOQ) and arithmetic encoding 613 to provide bitstream Bitstream 1. Similarly, the hyperplier neural network takes the (bottleneck) feature map y (output of encoder NN_ga), processes it through convolutional layers 605 and 606 as well as two ReLU layers, and provides it as output z with statistics of the input image. This is quantized and arithmetic encoded (615) to produce bitstream Bitstream 2. Similarly, bitstreams Bitstream 1 and Bitstream 2 are generated via the processing shown in Figures 6B, 9, and 9A.

[0238] Once the tile sizes are determined as described above, generating a bitstream further includes including in the bitstream the tile size indications for the first set of tiles and / or the tile size indications for the second set of tiles. This bitstream may be part of a bitstream containing Bitstream 1 and Bitstream 2, i.e., a bitstream generated by the entire encoding device (encoding process).

[0239] Further details and implementation examples regarding the processing of components including color components and tile size signaling are described in the second embodiment. Here, we simply refer to Figure 20, which shows examples of various types of parameters (such as the tile sizes of the first and / or second sets of tiles). Each instruction is included in the bitstream.

[0240] The encoding process of an input tensor has a corresponding part of the decoding process that shares a functional correspondence in the processing. In this exemplary and non-limiting embodiment, a method is provided for decoding a tensor representing picture data. The tensor may have a matrix form having two spatial dimensions of width = w, height = h, and a third dimension (e.g., depth or number of channels) whose size is equal to D. The method includes processing the input tensor representing the picture data by a neural network including at least a first subnetwork and a second subnetwork. It should be noted that the width and height of the decoder's input tensor may differ from the width and height of the input tensor processed by the encoder. Furthermore, as will be apparent to those skilled in the art, the first and second subnetworks of the decoder may perform functions that are entirely or partially (e.g., functionally) inverse of those of the encoder. However, the inverse function may not be interpreted in a strictly mathematical way. Rather, the term “inverse” refers to processing for the purpose of decoding the tensor to reconstruct the original picture data. Those skilled in the art will understand that encoding compression and decoding decompression may involve further processing that is not necessary for decoding and / or encoding. For example, RDOQ shown in Figure 16 is an encoder-only process. Furthermore, it should be noted that the terms “first” and “second” subnetworks are merely labels to distinguish between decoder subnetworks (and for that purpose, between encoder subnetworks as described above).

[0241] An example of a first and / or second subnetwork for encoding branching is shown in Figure 16, which includes a decoder 1604 and a post-filter 1611. In this method, the process includes: applying a first subnetwork to a first tensor, which divides the first tensor in the spatial dimension into a first set of tiles, and processing the first set of tiles by the first subnetwork; and, after applying the first subnetwork, applying a second subnetwork to a second tensor, which divides the second tensor in the spatial dimension into a second set of tiles, and processing the second set of tiles by the second subnetwork. In the example in Figure 16, the first subnetwork is a decoder 1604, whose output is provided as input to a second subnetwork, which is a post-filter 1611. Here, the first tensor is a quantized feature tensor ^y in latent space, which is pre-decoded from bitstream 1 by an arithmetic decoder 1606. Next, the second tensor ^x' input to the second subnetwork 1611 for post-filtering is a feature tensor, such as a feature, a feature map, or a feature map in latent space. Thus, as with the encoder's processing, the type of input tensor (e.g., the first and second tensors) may depend on the processing performed by the preceding subnetwork. In the example in Figure 16, the preceding subnetwork is the encoder 1604. The term "after" above does not limit the above decoding processing to immediately following the output of the first subnetwork, in the sense that the output of the first subnetwork is directly input to the second subnetwork. Rather, "after" means that multiple tiles from the first and second are processed within the same pipe in some temporal order, which does not necessarily have to be temporally immediate.

[0242] Furthermore, the type of input may also depend on layers (e.g., layers of a neural network NN or layers of an untrained network) from which the processed input data can be branched so that it can be used as input to another (e.g., subsequent) subnetwork. Such layers may, for example, be the outputs of hidden layers. Similar to the encoding process, the first and second tensors are divided into the so-called tiles already defined above during the decoding process.

[0243] In Figure 16, decoder 1604 can be a first subnetwork having processes 1 to N (processing pipelines 1 to N). As shown in Figure 16, decoder 1604 receives an input feature tensor ^y and tiles it into N tiles ^y1 to ^y N The tensor tile is then divided into (the first set of tiles). Each tensor tile is then processed by each block, i.e., by processes 1 to N that do not need to interact with each other (e.g., wait for each other during processing). The result of each process is the N tiles ^x1' to ^x N Provides a tensor ^x'. After processing the first set of tiles, the decoder subnetwork 1604 may merge the processed tiles into a first output tensor ^x'. In Figure 16, the first output tensor may be a second tensor used as input by the postfilter 1611. The postfilter 1611 may be a second subnetwork having processes 1 to N (processing pipelines 1 to N). Similar to the decoder 1604, the postfilter 1611 merges the input tensor ^x' into a second set of tiles ^x1' to ^x N The tile is divided into '. These are processed by each of the post-filters 1611, from process 1 to process N. Processes 1 to N output each tile ^x1 to ^x NThis provides a tensor that can be merged into tile ^x. In the example in Figure 16, the merged ^x refers to the decoded tensor representing the reconstructed picture data. Merging may (but does not) involve cropping as shown in Figures 17-19. Note that merging / combining into tensor ^x' or ^x does not need to be performed. A second subnetwork may reuse the tiling of the first subnetwork and simply modify it (fine-graining the tiling by further subdividing the tiles, or coarsening the tiling by combining multiple tiles into one). In the example in Figure 16, processes 1-N may perform operations on a tile-by-tile basis. That is, process i processes tile i. Alternatively, process i may process component i of multiple components of the input tensor. In this case, process i divides component i into multiple tiles and processes those tiles separately or in parallel.

[0244] Furthermore, at least two of each of the first and second sets of tiles are of different sizes. In other words, the subdivision into tiles may differ for each subnetwork, and thus each subnetwork may use different tile sizes. However, the inputs to each subnetwork (i.e., the first and second tensors) are divided into a grid of tiles of the same size, except for the tiles at the bottom and right image boundaries, which may have smaller sizes.

[0245] Otherwise, the properties and / or features of the first and second sets of tiles used in the decoding process are similar to those of one of the encoding processes described earlier. In particular, in the exemplary implementation described above, adjacent tiles of the first set of tiles in at least one of the spatial dimensions partially overlap, and / or adjacent tiles of the second set of tiles in at least one of the spatial dimensions partially overlap. Examples of adjacent tiles with partial overlap are shown in Figures 17A, 17B, 18, 19, and 20.

[0246] Furthermore, in some exemplary implementations, tiles of a first set of tiles are processed independently by a first subnetwork, and / or tiles of a second set of tiles are processed independently by a second subnetwork. In other words, the processing of tiles is independent of each other and therefore parallelizable. In one example, at least two tiles of the first set of tiles are processed in parallel by the first subnetwork, and / or at least two tiles of the second set of tiles are processed in parallel by the second subnetwork. Figure 16 shows parallel processing on the decoder side, where the components of the first input tensor (e.g., tile and / or spatial components) ^y1~^y N This is processed by decoder 1604 without interaction between the processes for components 1 to N. The result of each process is the output tensor component ^x1'~^x N These are further combined into the output tensor ^x'. The second subnetwork may be the post-filter 1611 in Figure 16, which takes the tensor ^x' from decoder 1604 (the first subnetwork) as input. In this example, the input of post-filter 1611 is directly connected to the output of decoder 1604. Alternatively, there may be further processing between the post-filter and decoder in Figure 16. The input tensor ^x' is transformed into N tiles ^x1'~^x by the post-filter. N It is divided into '. The post-filter then processes each tile independently or in parallel. The parallel processing is reflected in Figure 16 in the form that the processing of components 1 to N does not interact with each other. The result of the parallel processing of the post-filter is the reconstructed tile ^x1 to ^x N This can then be merged into a single tensor ^x representing the reconstructed picture data.

[0247] In addition, the partitioning of the first tensor includes determining the size of a tile among a first plurality of tiles based on a first predetermined condition, and / or the partitioning of the second tensor includes determining the size of a tile among a second plurality of tiles based on a second predetermined condition. For example, the first predetermined condition and / or the second predetermined condition are based on the available decoder hardware resources and / or other features already described above with respect to motion or encoding present in the picture data.

[0248] The first and second subnetworks for decoding may have a similar configuration to the subnetworks for encoding. Specifically, the first subnetwork performs processing with one or more layers including at least one convolutional layer and at least one pooling layer, and / or the second subnetwork performs processing with one or more layers including at least one convolutional layer and at least one pooling layer. Figures 6A and 6B show decoders in an NN-based VAE framework having respective convolutional layers 607-6012, where the size of each input tensor (feature map) ^y is then doubled (upsampled). The first and second subnetworks also perform their respective processing, which is part of the decompression of a picture or video. Such processing may also be provided by the VAE encoder shown in Figures 6A and 6B, which takes a feature tensor ^y as input, decompresses the output image, and reconstructs it as the output of convolutional layer 607, which represents the reconstructed picture data (i.e., the decoded tensor). For example, the first subnetwork and / or the second subnetwork perform one of the following: picture decoding and picture filtering by a convolutional subnetwork. As described above, the first and / or second subnetwork for decoding may be the decoder 904 in Figure 9, which processes a feature map tensor ^y to generate reconstructed image data ^x that approximates the original picture data x. As shown in Figure 9, the decoder 904 may also be part of a VAE encoder-decoder framework as shown in Figures 6A and 6B, where each decoder "gs" performs its respective picture decoding by processing the feature tensor ^y through a sequence of convolutional layers 610-607, including processing through an inverse IGDN layer. Picture filtering (e.g., post-filtering) is not further shown in Figures 6A / B and 9.

[0249] In one implementation, the input tensor is a picture or sequence of pictures containing one or more components, at least one of which is a color component. Alternatively, the input tensor may be a latent space representation of a picture, which may be the output of a preprocessing (e.g., an output tensor). The one or more components are the color components and / or depth and / or motion maps and / or other feature maps (one or more) associated with the picture sample. The input tensor has at least two components, namely a first component and a second component, and the first subnetwork divides the first component into a third plurality of tiles, and the second component into a fourth plurality of tiles, where at least two of each of the third plurality of tiles and the fourth plurality of tiles are of different sizes, and / or the second subnetwork divides the first component into a fifth plurality of tiles, and the second component into a sixth plurality of tiles, where at least two of each of the fifth plurality of tiles and the sixth plurality of tiles are of different sizes.

[0250] The decoding method described above further includes extracting an input tensor from the bitstream for processing by a neural network. The neural network may further include entropy decoding. Figures 6A and 6B show decoding the input tensor ^y from bitstream 1 via arithmetic decoding for decoder subnetwork gs. Entropy encoding is performed by subnetwork hs. Here, the entropy tensor ^z may be decoded by arithmetic decoding from bitstream 2. ^z is processed to obtain statistics about the encoded picture data, which are used for decoding ^y. Figure 6B shows further details of the entropy decoding, where information about the distribution given with respect to mean μ and variances σ1, σ2 is further input into Gaussian entropy model 670 and used for arithmetic decoding of ^y. As mentioned above, a subnetwork hs (Figure 6A) having various upsampling convolutional layers 611 and 612, possibly including a leaky ReLU layer 660, may be part of the hyperdecoder 907 shown in Figures 9 and 9B. In one exemplary implementation, a second subnetwork performs picture post-filtering, where one or more post-filtering parameters differ for at least two tiles of a second plurality of tiles and are extracted from the bitstream. Furthermore, the decoding process further includes parsing the size indications of the tiles of the first plurality of tiles and / or the size indications of the tiles of the second plurality of tiles from the bitstream.

[0251] In this exemplary and non-limiting embodiment, a computer program stored in a non-temporary medium is provided, which, when executed on one or more processors, includes code that performs any of the steps of the encoding and decoding methods described above. Flowcharts of the encoding and encoding processes are shown in Figures 21 and 22. With respect to the encoding process in Figure 21, in step 2110, the first subnetwork processes the first tensor. This includes dividing the first tensor into a plurality of tiles, which are then processed by the first subnetwork. Note that in Figure 21, the picture data (i.e., the input tensor representing the picture data) is input to the first subnetwork, indicated by the dashed line. This means that the raw picture data does not necessarily have to be directly input to the first subnetwork, and this may depend on the processing performed by the first subnetwork with respect to the order of the sequence of processing. In other words, the first tensor, which is the input for the first subnetwork, originates from the picture data. If the first subnetwork is the encoder 901 shown in Figure 9, the first tensor may be the input tensor x. In step S2120, the first set of tiles is further processed by the first subnetwork. After the processing by the first subnetwork, the second subnetwork processes the second tensor in step S2130. This involves dividing the second tensor into a second set of tiles. Again, the second tensor input to the second subnetwork, shown by the dashed line, does not necessarily have to be a direct output from the first subnetwork. In the implementation example in Figure 9, the output of the feature tensor y from encoder 901 (first subnetwork) is directly input to RODQ 902 (second subnetwork). However, there may be additional processing between encoder 901 and RDOQ 902. In step S2140, the second set of tiles is further processed. The output of the processing of the second subnetwork, which may include further processing (not shown in Figure 21), may include generating bitstreams (e.g., bitstream 1 and / or bitstream 2 in Figure 9).Furthermore, the process of determining the size of one of the first and second sets of tiles, and / or including an indication of the size in the bitstream, may be a processing step prior to providing the bitstream as the output of the neural network processing. With respect to the decoding process in Figure 22, the flowchart depicts the reverse processing steps, starting with an input tensor that may be a direct or indirect input to the first subnetwork, as indicated by the dashed line. Again, in the exemplary implementation in Figure 9, the feature tensor ^y is used as a direct input to the decoder 904, which in this case is the first subnetwork. In step S2220, the first set of tiles are processed by the first subnetwork (e.g., decoder 1604), as shown in Figure 16. After processing by the first subnetwork, the second tensor is processed by the second subnetwork in step S2230 by dividing the second tensor into multiple tiles, which are then further processed by the second subnetwork in step S2240. In the exemplary implementation shown in Figure 16, the output ^x of the first subnetwork of decoder 1604 is a second tensor, which is processed by the post-filter 1611 corresponding to the second subnetwork. The processing of the second set of tiles in the example in Figure 16 outputs ^x, which corresponds to the reconstructed picture data. Note that there may be further processing between the processing of decoder 1604 and post-filter 1611 in Figure 16. In that case, the output ^x of decoder 1604 does not have to be directly input to post-filter 1611.

[0252] Furthermore, as already stated, this disclosure also provides a device (apparatus) configured to perform the steps of the method described above.

[0253] In this exemplary and non-limiting embodiment, a processing device is provided for encoding an input tensor representing picture data. Figure 23 shows a processing device 2300 comprising a processing circuit 2310, each having a module for performing the steps of the encoding process. The processing circuit is configured to process the input tensor by a neural network including at least a first subnetwork and a second subnetwork, implemented by an NN processing module-subnetwork 1 2311 and an NN processing module-subnetwork 2 2312. The NN processing module-subnetwork 1 2311 may have a separate partitioning module 1 2313 that partitions the first tensor in the spatial dimension into a first plurality of tiles, and processes the first plurality of tiles by the first subnetwork. Alternatively, each module that processes the first input tensor and / or the first plurality of tiles may be implemented in a single module that may be contained in a single circuit or separate circuits. Similarly, the NN processing module-subnetwork 2 2312 and partitioning module 2314 apply a second subnetwork to a second tensor. This involves dividing a second tensor in spatial dimensions into a second set of tiles and processing the second set of tiles by a second subnetwork. Note that the processing of the second tensor after processing the first tensor can be implemented by wiring each module so that signals (in terms of their input and output signals) are input in their respective temporal order (not necessarily immediate temporal order). Alternatively or additionally, the order of each signaling may be implemented by configuring the modules in software. Modules 2313 and 2314 may further provide the function of determining the tile size of the first and / or second set of tiles. The processing unit may further have a bitstream module 2315 that provides the function of generating a bitstream, and the output of neural network processing is included in the bitstream.Furthermore, module 2315 provides the ability to include a representation of the tile sizes of a first set of tiles and / or a second set of tiles in the bitstream.

[0254] In this exemplary and non-limiting embodiment, a processing unit is provided for encoding an input tensor representing picture data, the processing unit comprising one or more processors and a non-temporary computer-readable storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the program, when executed by the one or more processors, configures an encoder to perform a method according to the encoding method described above.

[0255] In this exemplary and non-limiting embodiment, a processing device is provided for decoding a tensor representing picture data. Figure 24 shows a processing device 2400 comprising a processing circuit 2410, each having a module for performing the steps of the decoding process. The processing circuit is configured to process the input tensor by a neural network including at least a first subnetwork and a second subnetwork, realized by an NN processing module-subnetwork 1 2411 and an NN processing module-subnetwork 2 2412. The NN processing module-subnetwork 1 2411 may have a separate partitioning module 1 2413 that partitions the first tensor in the spatial dimension into a first plurality of tiles, and processes the first plurality of tiles by the first subnetwork. Alternatively, each module that processes the first input tensor and / or the first plurality of tiles may be implemented in a single module that may be contained in a single circuit or separate circuits. Similarly, the NN processing module—subnetwork 2 2412 and partitioning module 2414 apply the second subnetwork to the second tensor, which includes partitioning the second tensor in spatial dimensions into a second set of tiles and processing the second set of tiles by the second subnetwork. Modules 2413 and 2414 may further provide the ability to determine the tile sizes of the first and / or second set of tiles. Again, it should be noted that the processing of the second tensor after processing the first tensor can be implemented, for example, by wiring the respective modules so that signals (with respect to their input and output signals) are input in their respective temporal (not necessarily immediate) order. Alternatively or additionally, the order of each signaling may be implemented by configuring the modules in software. The processing unit may further have a parsing module 2415 that provides the ability to parse the tile size indications of the first set of tiles and / or the second set of tiles from the bitstream. Module 2415 may further provide functionality for extracting an input tensor from a bitstream.

[0256] In this exemplary and non-limiting embodiment, a processing apparatus for decoding a tensor representing picture data comprises one or more processors and a non-temporary computer-readable storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the program, when executed by the one or more processors, configures an encoder to perform the method according to the decoding method described above.

[0257] The encoding and decoding processes described above and performed by the VAE encoder-decoder shown in Figure 16 may be performed within the coding system 10 of Figure 1A. Thereafter, the source device 12 represents the encoding side and provides compression of the input picture data 21, including the input tensor x in Figure 16. In particular, the encoder 20 in Figure 1A may include modules for the encoding process according to this disclosure, such as encoder 1601, quantizer or RDOQ 1602, and arithmetic encoder 1605. The encoder 20 may further include hyperplier modules such as hyperdecoder 1603, quantizer or RDOQ 1608, and arithmetic encoder 1609. Similarly, the destination device 14 in Figure 1A represents the decoding side and provides decompression of the input tensor representing the picture data. In particular, the decoder 30 in Figure 1A, together with the arithmetic decoder 1606, may include modules for the decoding process according to this disclosure, such as decoder 1604 and postfilter 1611 in Figure 16. In addition, the decoder 30 may further include a hyperplier for decoding ^z, an arithmetic decoder 1610, a hyperdecoder 1607, and an arithmetic decoder 1606. In other words, the encoder 20 and decoder 30 in Figure 1A may be implemented and configured to include any of the modules in Figure 16 to implement the encoding or decoding process according to the present disclosure, wherein a plurality of subnetworks processes an input tensor divided into a first plurality of tiles and a second plurality of tiles, which are then processed as described in the first embodiment. Although Figure 1A shows the encoder 20 and decoder 30 separately, they may be implemented via the processing circuit 46 in Figure 1B. In other words, the processing circuit 46 may provide the encoding-decoding functionality of the present disclosure by implementing the respective circuits for each of the modules in Figure 16.

[0258] Similarly, a video coding device 200 having its coding module 270 of the processor 230 in Figure 2 may perform the encoding or decoding functions of the present disclosure. For example, the video coding device 200 may be an encoder or decoder having each of the modules in Figure 16, which performs the encoding or decoding functions described above.

[0259] The apparatus 300 in Figure 3 may be implemented as an encoder and / or decoder having an encoder 1601, a quantizer or RDOQ 1602, a decoder 1604, a post-filter 1611, a hyperencoder 1603, and a hyperdecoder 1607, along with arithmetic encoders 1605, 1609 and arithmetic decoders 1606, 1610 to perform tile processing as described according to the first embodiment. For example, the processor 302 in Figure 3 may have respective circuits for performing encoding and / or decoding processing according to the methods described above.

[0260] Exemplary implementations of the encoder 20 shown in Figure 4 and the decoder 30 shown in Figure 5 may also implement the encoding and decoding functions of this disclosure. For example, the partitioning unit 452 in Figure 4 may perform the partitioning of a first and / or second tensor into a first and / or second set of tiles, which is performed by the encoder 1601 in Figure 16. Thus, the syntax element 465 may include instructions (one or more) for the size and position of the tiles, along with instructions such as a filter index. Similarly, the quantization unit 408 may perform quantization or RDOQ of the RDOQ module 1602, and the entropy encoding unit 470 may implement the functions of the hyperpliers (i.e., modules 1603, 1605, 1607, 1608, and 1609). Next, the entropy decode unit 504 in Figure 5 can perform the functions of the decoder 1604 in Figure 16 by dividing the encoded picture data 21 (input tensor) into tiles and parsing instructions from the bitstream, such as tile size or position, as syntax elements 566. The entropy decode unit 504 may further implement hyperpliers (i.e., modules 1606, 1607, and 1610). The post-filtering 1611 in Figure 16 can also be performed, for example, by the entropy decode unit 504. Alternatively, the post-filter 1611 may be implemented in the mode-applying unit 560 as an additional unit (not shown in Figure 5). [Examples]

[0261] Second Embodiment In the previous exemplary implementation of the first embodiment, the tiles of the first and / or second sets of tiles are processed by the first and second subnetworks, respectively, as previously described. Here, the encoding and decoding processes of the components in multiple pipelines are discussed. Each pipeline processes one or more picture / image components, which may be color planes. As will be seen from the following description, the first and second embodiments share, in part, similar or identical processing.

[0262] In this exemplary and non-limiting embodiment, a method is provided for processing an input tensor representing picture data. The input tensor may have a matrix form with width = w, height = h, and a third dimension (e.g., depth or number of channels) equal in size D. Furthermore, the input tensor may be processed by a neural network. In this method, multiple components of the input tensor are processed, the components comprising a first component and a second component in the spatial dimension. The input tensor is a picture or sequence of pictures, comprising one or more components of the multiple components, at least one of which is a color component. In one implementation, the first component represents the lumen component of the picture data, and the second component represents the chromen component of the picture data. For example, the multiple components of the input tensor may be in YUV format, where Y is the lumen component and UV is the chromen component. The lumen component may be called a first-order component, and the chromen component may be called a second-order component. It should be noted that the terms “primary” and “secondary” are labels used to assign more weight and / or importance to primary components than to secondary components (one or more). As will be apparent to those skilled in the art, there may be application instances where a different component is preferred over the ruma component, even if the ruma component is typically of higher importance. Such prioritization often involves using information about primary components (e.g., the ruma) as auxiliary information for processing secondary components (i.e., less important than the ruma). The multiple components may be in RGB format, or any other format suitable for processing the components in their respective processing pipelines. This method includes processing a first component, which involves dividing the first component in the spatial dimension into a first set of tiles and processing each tile of the first set of tiles separately; and processing a second component, which involves dividing the second component in the spatial dimension into a second set of tiles and processing each tile of the second set of tiles separately. Note that "separately" does not mean independently; rather, there may still be shared auxiliary information for processing the first and / or second components.Examples of tiles with a rectangular shape are shown in Figures 14, 17A, 17B, and 18-20. In this method, at least two copositional tiles in each of the first and second sets of tiles are of different sizes. In other words, the tiles in the first and second sets of tiles have different sizes. For the size of the tiles in each set of tiles, all tiles in the first set of tiles are the same size, and / or all tiles in the second set of tiles are the same size. Thus, each pipeline processes tiles of the same size but different between pipelines. In particular, Y and UV can have different block partitioning than VVC. Tiles were introduced to solve memory problems and can contain multiple blocks. Tiles can be coded independently of each other and decoded in parallel. Blocks are part of a tile / picture / slice, and each block uses its own coding method, and decoding is sequential, thus requiring an coding tree (block structure) for better local adaptation. In other words, in this exemplary and non-limiting embodiment, a separate chroma code tree may be used.

[0263] In one implementation, at least two of the first set of tiles are processed independently or in parallel, and / or at least two of the second set of tiles are processed independently or in parallel. Figure 25 shows an example of encoder-decoder processing where the first component is a chroma and the second component is one of chromas U or V, which are processed in separate pipelines. Note that, as shown in Figure 25, chroma components U and V can be processed congruently as a single chroma component. In one exemplary implementation, processing of the input tensor includes processing that is part of picture or video compression. For example, processing of the first and / or second components includes one of the following: picture encoding by a neural network, rate-distortion-optimized quantization (RDOQ), and picture filtering. The compression processing is shown in Figure 25 by their respective modules, encoders 2501, 2508, and RDOQs 2502, 2509. Encoders 2501 and 2508 perform picture encoding of the input picture x by a neural network (NN) for the lumer Y and chroma component (one or more) UV. Modules RDOQ 2502 and 2509 quantize the feature tensor y by optimizing rate distortion, providing the quantized feature tensor ^y for each component as output. Modules, hyperencoders 2503, 2510 and RDOQ 2504, 2511 form parts of the encoding-side hyperplier network that generate statistical information ^z of the picture data. Furthermore, the bitstream is generated by including the outputs of processing the first and second components in the bitstream. In the example in Figure 25, the quantized feature tensors ^y for the first (lumer) component and the second (chroma) component are arithmetic encoded (1605) and included in the bitstreams Bitstream Y1 and Bitstream UV1, respectively. Similarly, the quantized statistics of the lumens and chromens ^z are arithmetic encoded (1609) and included in the bitstreams Bitstream Y2 and Bitstream UV2, respectively.In the example in Figure 25, two bitstreams are generated, allowing the encoded picture data to be separated from the encoded statistics of the picture data.

[0264] In another exemplary implementation, processing of the input tensor includes processing that is part of the decompression of the picture or video. For example, processing of the first and / or second component includes one of the following: picture decoding by a neural network and picture filtering. The decompression processing is shown in Figure 25 by the respective modules, decoders 2506, 2513 and postfilters 2507, 2514. Decoders 2506 and 2513 perform picture decoding of the input picture x by a neural network (NN) for the lumen Y and chroma component UV by decoding the quantized feature tensor ^y for the lumen and chroma components from the respective bitstreams Bitstream Y1 and Bitstream UV1. As described above, the hyperplier also requires a decoding module, which includes hyperdecoders 2505, 2512 that decode quantized statistical information ^z of the picture data from the bitstreams Bitstream Y2 and Bitstream UV2 for the lumen and chroma components. This information is input to the arithmetic decoder 1606, whose output is provided to decoders 2505 and 2513. In one implementation, processing of the first and / or second components includes picture post-filtering. As shown in Figure 25, the decoder output is input to post-filters 2507 and 2514, which perform picture post-filtering of their components, providing reconstructed picture data^x for reconstructed lumens^Y and chromens^UV, respectively, as output. The functional blocks (modules) shown in Figure 25 are similar to the same structural arrangement of each module in the VAE encoder-decoder shown in Figure 16 of the first embodiment.

[0265] In the example in Figure 25, the input tensor is YUV, having Y, U, and V as its components. In particular, the first component here is the rumor Y, which is input to encoder 2501 of the rumor pipeline. Similarly, as mentioned above, U and V may be considered congruently as the second component, which is input to encoder 2508 of the chroma pipeline. Note that different reference codes are assigned to each functional module (unit) of the VAE encoder-decoder for the rumor and chroma pipelines, as their modules may be configured differently to enable processing of each tile having different sizes for the rumor and chroma pipelines. Nevertheless, these modules (e.g., encoder 2501 and encoder 2508) may perform the same / similar functions with respect to encoding, RDOQ, decoding, post-filtering, etc. These functions are the same as those performed by each module of the VAE encoder-decoder in Figure 16 and have already been described above. In some implementations, processing the second component involves decoding the chroma component of the picture based on the representation of the rumor component of the picture. This is shown in Figure 25, where the lumern component is a first-order component and is therefore considered to have higher importance than the chroma component UV. As shown in the figure, in the decoding process of the lumern and chroma, the lumern feature tensor ^y is obtained from the lumern bitstream Y1 by arithmetic decoding 1606, and this is used as input for decoder 2513 to decode the UV component.

[0266] Similar to the first embodiment, adjacent tiles of a first plurality of tiles in at least one spatial dimension partially overlap, and / or adjacent tiles of a second plurality of tiles in at least one spatial dimension partially overlap. Examples of adjacent tiles that can partially overlap are described above with reference to Figures 17A, 17B, and 18-20. In one exemplary implementation, the division of the first component includes determining the size of the tiles in the first plurality of tiles based on a first predefined condition, and / or the division of the second component includes determining the size of the tiles in the second plurality of tiles based on a second predefined condition. For example, the first predefined condition and / or the second predefined condition are based on available decoder hardware resources and / or motion present in the picture data. The first condition and / or the predefined condition may be memory resources of the decoder and / or encoder. If the available memory is less than a predefined value of the memory resources, the determined tile size may be smaller than the tile size if the available memory is greater than or equal to the predefined value. Alternatively or additionally, if the presence of motion is significant, the tile size may be determined to be smaller than if the presence of motion is less significant. Whether the motion is significant (i.e., fast motion and / or fast / frequent motion changes) is determined by the respective motion vector(s) with respect to their magnitude and direction changes, and may be compared to corresponding predefined values ​​(thresholds) for magnitude and / or direction and / or frequency. In addition, another predefined condition may be a region of interest (ROI), where the tile size may be determined to be smaller if the ROI size is smaller than the ROI criterion size.For example, the ROI in picture data may be based on detected objects (e.g., vehicles, bicycles, motorcycles, pedestrians, animals, etc.), which may have different sizes, move quickly or slowly, and / or change direction of movement quickly or slowly and / or many times (i.e., more frequently, relative to a predefined frequency value). The ROI may also be based on the smoothness of the regions in the picture data. For example, the tile size may be large for regions with a high degree of smoothness (e.g., measured relative to a predefined value of smoothness), while a smaller tile size may be used for less smooth regions, i.e., regions with very noticeable, for example, spatial variation. In other words, the tile size may be determined based on the degree of texture of the regions in the picture data. Furthermore, a larger tile size may be determined for primary components, such as lumens, and a smaller tile size may be determined for secondary components, such as chromens. Thus, the tile size, along with the content of the picture data, including scene-specific tile sizes, can be optimized for hardware resources. In further implementations, the step of determining the size of tiles in a second set of tiles includes scaling the tiles of the first set of tiles. In other words, the tile size of the first component is used as a reference to derive the tile size of the second component by scaling the tile size of the first component. Scaling may include enlarging (scaling up) the scaled tiles of the second set of tiles so that they are larger than the size of the first set of tiles. Alternatively, scaling may include shrinking (scaling down) the scaled tiles of the second set of tiles so that they are smaller than the size of the first set of tiles. Whether to scale up or scale down may depend on the importance of the first and / or second components.If at least one of the multiple components being processed has at least one color component, the use of upscaling or downscaling may also depend on a specific color (for example, a color component indicating "hazard / warning").

[0267] The following describes how tile size information can be signaled and decoded from a bitstream. Various aspects of signaling and decoding tile size and / or other preferred parameters are described. In one exemplary implementation, instructions for the determined sizes of tiles within a first set of tiles and / or a second set of tiles are encoded in a bitstream. Furthermore, the instructions further include the positions of the tiles within the first set of tiles and / or the second set of tiles. For example, the first component is a rumor component, in which the bitstream contains instructions for the tile sizes of the first set of tiles, and the second component is a chroma component, in which the bitstream contains instructions for a scaling factor, the scaling factor relating the tile sizes of the first set of tiles to the tile sizes of the second set of tiles.

[0268] Examples of signaling the tile size and / or position of tiles for various components are shown in the following table, along with excerpts of code syntax for implementing the signaling. The syntax tables are examples and may not be limited to this particular syntax. In particular, the syntax examples are illustrative and may be applied to signaling instructions for tile size and / or tile position, etc., for lumen and / or chroma components included in the bitstream. The first syntax table refers to the autodecoder, and the second table refers to the postfilter. As is evident from Tables 1 and 2, the same / similar instructions may be used and included in the bitstream, but may be included separately for their use in the autodecoder (Table 1) and postfilter (Table 2). In one implementation, one or more parameters of postfiltering differ for at least two tiles of the first set of tiles and are extracted from the bitstream, and one or more parameters of postfiltering differ for at least two tiles of the second set of tiles and are extracted from the bitstream. In other words, different tile-based post-filter parameters may be signaled in the bitstream, so that post-filtering of each first set of tiles and / or second set of tiles can be performed with high precision.

[0269] The following sections will describe examples of the components of the encoder-decoder VAE module shown in Figure 25, and how the module uses its respective indicators, with reference to Tables 1 and 2.

[0270] [Table 1] TIFF0007877494000007.tif146170

[0271] [Table 2] TIFF0007877494000009.tif197128 TIFF0007877494000010.tif215170

[0272] A. Decoder In this implementation example, the lumern and chroma components are decoded separately, as described above with reference to Figure 25, which shows separate pipelines for decoding the lumern and chroma components. However, decoding the chroma component requires the latent spaces (i.e., CCS) of both the chroma and lumern components, as indicated by the dashed line pointing from the arithmetic decoder 1606 of the lumern pipeline to the decoder 2513 of the chroma pipeline.

[0273] Derivation of tile maps: As shown in Table 1, there are several ways to obtain a tile map. 1. The tile map is explicitly signaled only for the primary component. Other components use the same tile map (for YUV420 and CCS, the primary component is luma / Y, and the other components are chroma / UV). Using the same tile map includes using the same tile map in a literal way. In some implementations, the same tile map (for example, the luma component) is used to derive the tile map for the secondary component (e.g., chroma / UV) by scaling the tile size of the luma component. For that purpose, an indication of the scaling factor is included in the bitstream. In Table 1, such a scaling factor is "chroma_tile_scaling_factor". 2. The tile map is explicitly signaled for each component of the image.

[0274] In the signaling examples in Table 1, whether or not each component tile is signaled depends on the directives "tiles_enabled_for_chroma" and "tiles_enabled_for_luma". These directives may also be simple flags "0" or "1" indicating that they can be turned off or on.

[0275] When a tile map is explicitly signaled, this can be done in one of the following ways:

[0276] 1. A regular grid of tiles having the same size (except for the bottom and right borders) is used. Offset and size values ​​are signaled, from which a tile map can be derived (see below). In this case, the tiles of the first component (luma) have the same size, where the tile size is defined using the width and height when the tile is rectangular. The respective indications for tile size are "tile_width_luma" and "tile_height_luma" in Table 1. Similar indications for the same tile size in chroma can be used, except that chroma tile size is different from luma tile size.

[0277] 2. A regular grid of tiles of the same size (except for the bottom and right borders) is used. Offset and size values ​​are used, but not directly signaled. Instead, the size and offset values ​​are derived from the already decoded level definition. The tile map can be derived from the size and offset values ​​(see below).

[0278] 3. An arbitrary grid of tiles is used. First, the number of tiles is signaled, and then, for each tile, its position and size are signaled (duplicates will be implicitly included in this signaling). In Table 1, an arbitrary grid is reflected in the fact that for a given number of luma tiles "num_luma_tiles", each tile can have individual sizes in terms of "tile_width[i]" and "tile_height[i]". In addition, instructions for the tile position (in this case, the starting position) are also included in the bitstream, which are "tile_start_x[i]" and "tile_start_y[i]". In this example, the tiles are in 2D xy space.

[0279] When chroma signaling is enabled and chroma components are processed independently of rumor components (Table 1: "!use_dependent_chroma_tiles", i.e., chroma tiles are independent of rumor tiles), similar signaling is used with each instruction included in the bitstream.

[0280] Further instructions regarding the tile overlap of the rumor and / or chroma components may be included in the bitstream and thus signaled to the decoder.

[0281] When a tile map is signaled via tile size (tile width is equal to height) and overlap values ​​(signal space size / value in coordinates), the N overlapping regions are derived as follows: for tile_start_y in range(0, image_height - overlap, tile_height - overlap): for tile_start_x in range(0, image_width - overlap, tile_width - overlap): height=min(tile_height, image_height - tile_start_y) width=min(tile_width, image_width - tile_start_x) im_tile i =(tile_start_x, tile_start_y, width, height)

[0282] Decoding the Ruma component: The image contains N overlapping regions (tiles). Each tile (im_tile) i ) is the latent space (lat_tile i It has a corresponding tile within ). Furthermore, im_tile i is lat_tile i This covers only a subset of the entire receptive field. For each im_tile in the signal space, the matching lat_tile in the latent space is derived as follows: where alignment_size is a power of 2 that depends on the number of downsampling layers in the subnetwork: image_tile=im_tilei lat_tile_start_y=image_tile.position.y / / alignment_size lat_tile_start_x=image_tile.position.x / / alignment_size if image_tile.size.height % alignment_size: height=math.ceil(image_tile.size.height / alignment_size) else: height=image_tile.size.height / / alignment_size if image_tile.size.width % alignment_size: width=math.ceil(image_tile.size.width / alignment_size) else: width = image_tile.size.width / / alignment_size lat_tile i = (lat_tile_start_x, lat_tile_start_y, width, height)

[0283] Using lat_tile, the corresponding region of the latent space is extracted and processed by the decoder sub-network. The decoder sub-network also has im_tile as an auxiliary input, which is required to correctly pad, especially at the image boundaries where the tile size may not be a multiple of alignment_size. The output of the decoder is assigned to the region of the image specified by im_tile. This step can include a cropping operation to remove the overlapping parts of the reconstruction that overlap with the reconstruction of another tile.

[0284] The need for cropping of regions R1 - R4 for the processed tiles L1 - L4 is explained by referring to FIGS. 17A and 18 to ensure seamless merging.

[0285] Decoding of chroma components: There are N overlapping regions (tiles) within the image. In the example shown in FIGS. 17A and 18, there are four partially overlapping tiles. Each tile (im_tile i ) has a corresponding tile within the latent space (lat_tile i ). Further, im_tile i covers only a subset of the full receptive field of lat_tile i . The N overlapping regions are derived based on the signaled parameters for tile size and tile overlap (tile width is equal to height). The size is signaled in signal space units (i.e., not in the latent space). Then, the position and size of the tiles in signal space are derived as follows. for tile_start_y in range(0, image_height - overlap, tile_height - overlap): for tile_start_x in range(0, image_width - overlap, tile_width - overlap): height=min(tile_height, image_height - tile_start_y) width=min(tile_width, image_width - tile_start_x) im_tile i =(tile_start_x, tile_start_y, width, height)

[0286] For each im_tile in the signal space, the matching lat_tile in the latent space is derived as follows. Here, the alignment_size is a power of 2 that depends on the number of downsampling layers in the subnetwork. image_tile=im_tile i lat_tile_start_y=image_tile.position.y / / alignment_size lat_tile_start_x=image_tile.position.x / / alignment_size if image_tile.size.height % alignment_size: height=math.ceil(image_tile.size.height / alignment_size) else: height=image_tile.size.height / / alignment_size if image_tile.size.width % alignment_size: width=math.ceil(image_tile.size.width / alignment_size) else: width=image_tile.size.width / / alignment_size lat_tile i =(lat_tile_start_x, lat_tile_start_y, width, height)

[0287] Using lat_tile, the corresponding region of the chroma latent space lat_UV is extracted. Furthermore, the corresponding region of the lumen latent space lat_Y is determined. For the YUV420 example, a possible way to do this is to downsample the lumen latent space by factor 2, and then use the same lat_tile as for the chroma to extract lat_Y. Both lat_Y and lat_UV are then processed by a decoder subnetwork. The decoder subnetwork also has im_tile as an auxiliary input, which is necessary to perform padding correctly, especially at image boundaries where the tile size may not be a multiple of alignment_size. The decoder output is assigned to the image region specified by im_tile. This step may include a cropping operation to remove any overlapping portions of the reconstruction with another tile reconstruction.

[0288] B. Post-processing filter: Derivation of tile maps: As shown in Table 2, there are several ways to obtain a tile map.

[0289] 1. The same tile map as the decoder is used. For the YUV420 example, this can be implemented so that the decoder uses the same tile for chroma / Y to filter the Y component, and for U and V, the same tile used by the decoder for chroma (UV) is used. Such behavior can be signaled with a single flag. 2. The tile map is explicitly signaled only for the primary component (e.g., the lumen component). Other components (e.g., the chroma component) use the same tile map. As shown in Table 2, the signaling of the tile map (i.e., tile size, position, and overlap) can be done in a similar manner to Table 1 for the autodecoder. 3. The tile map is explicitly signaled for each component of the image. Again, the signaling of the tile map for each component can be done by using the same instructions included in the bitstream, as shown in Table 1.

[0290] Note that when filter selection uses multiscale structural similarity (MS-SSIM) as the distortion criterion (encoder), the tiles should be large enough for MS-SSIM (this involves several downsampling steps). If the tiles at the bottom and right borders of the image are too small, they are enlarged by removing areas from their respective neighboring regions, i.e., adjacent regions. This is normative, as the decoder must perform the same process. When MSE and / or peak signal-to-noise ratio (PSNR) are used as the distortion criterion, there is no issue regarding the possibility of tile size being too small.

[0291] When a tile map is explicitly signaled, this can be done in one of the following ways: 1. A regular grid of tiles of the same size (except for the bottom and right borders) is used. Offset and size values ​​are signaled, from which a tile map can be derived (see explanation below). 2. A regular grid of tiles of the same size (excluding the lower and right boundaries) is used. Offset and size values are used but not directly signaled. Instead, the size and offset values are derived from previously decoded level definitions. The tile map can be derived from the size and offset values (see the following description). 3. Any grid of tiles is used. First, the number of tiles is signaled, and then for each tile, its position and size are signaled (overlap will be implicitly included in this signaling).

[0292] Derivation of the filters used for each tile tile I For a specific image component for tile, the filter specified by the filter index filter I is used. In other words, the tile-based filter can be specified with one parameter, namely the filter index. The selection of filter I for each tile can be signaled via one of the following ways, as shown in Table 2.

[0293] 1. The same model / filter is used for all tiles of the image component. This can be signaled with a single flag together with the filter index filter I . In Table 2, such a flag is "same_model_for_all_luma" with a filter index of "model_idx_luma".

[0294] 2. The filter model is signaled for each tile of the image component. a. A default filter index is signaled. In Table 2, such a default index is "default_model_idx_luma". For each tile IFor each tile (where the number of tiles is already known from the tile map), a flag is signaled to indicate whether the filter index differs from the default. In Table 2, such a flag for each tile is "use_default_idx[i]", where "i" is the index that labels each tile within the first or second set of tiles. If it differs, the filter index for that tile is "filter I This is signaled. In Table 2, each filter index is "model_idx_luma[i]". In other words, one or more parameters of the filter differ for at least two of the first set of tiles. The same applies to one or more parameters for the second set of tiles. b.Each tile I (The number of tiles is already known from the tile map) Filter index for tiles I This is signaled. In Table 2, the respective filter index is "model_idx_luma[i]".

[0295] Filter indices for each component can also be explicitly signaled. Alternatively, a filter index for one component can be derived from the filter indices signaled for other components, such as luma components.

[0296] When the selected model for the filter is signaled, the post-filter may use the same tile map as the one used by the decoder. This can reduce the overhead of signaling the tile size.

[0297] Filtering of ingredients: The image contains N overlapping regions (tiles). Each tile is processed independently (possibly in parallel). IRegarding the reconstructed region, its location, size, overlap, and the filter index used are known from the signaling described above. I Region I is extracted from the reconstructed image based on its position and size values. Filtering based on a filter index is then performed on this region. The output is cropped using the position, size, and overlap values ​​and assigned to the filtered output image at the positions described by the position and size values.

[0298] C. RDOQ In this implementation, RDOQ is applied separately to the lumern and chroma components; that is, the latent spaces for each component lumern and chroma are optimized separately. Figure 25 shows the RDOQ processing of lumern Y and chroma UV in separate pipelines by RDOQ modules 2502 and 2509, respectively. However, decoding the chroma component requires both the chroma and lumern latent spaces (i.e., CCS). Therefore, the processing by RDOQ 2502 is first performed on the lumern to obtain a new optimized lumern latent ^y. The lumern latent is then left fixed and used as additional input (i.e., auxiliary information) to the optimization of the chromern latent space. In Figure 25, this is shown by a dashed line.

[0299] If the same tile map is used for both lumens and chromens in the RDOQ process, optimization of the chroma component does not need to wait until lumen RDOQ optimization has been performed for all lumen tiles. Instead, once the optimization for a particular lumen tile is complete, it can be directly used as input for the optimization of the corresponding chroma latent tile.

[0300] Derivation of tile maps: Since RDOQ is an encoder-only process, the tiles used do not need to be signaled. In this implementation, a regular tile grid was used for the lumen and chroma components. Each grid is described by offset and size values ​​(encoder arguments). These can be used to derive a tile map for a given component (see below).

[0301] Optimization of luma components: RDOQ is Cost = R + λD The cost of the tiles given by is optimized iteratively.

[0302] Here, R is an estimate of the number of bits required to encode the tile, and D is a metric (such as peak signal-to-noise ratio (PSNR) or multiscale structural similarity (MS-SSIM)) of the distortion of the (tile) reconstruction compared to the original. λ is a parameter set according to the operating point of the encoder. im_tile i R and D for this are obtained as follows:

[0303] The image contains N overlapping regions (tiles). Each tile (im_tile) i ) is the latent space (lat_tile i It has a corresponding tile within ). Furthermore, im_tile i is lat_tile i This covers only a subset of the entire receptive field. For each im_tile in the signal space, the corresponding matching lat_tile in the latent space is derived as follows: Here, alignment_size is a power of 2 that depends on the number of downsampling layers in the subnetwork. image_tile=im_tilei lat_tile_start_y=image_tile.position.y / / alignment_size lat_tile_start_x=image_tile.position.x / / alignment_size if image_tile.size.height % alignment_size: height=math.ceil(image_tile.size.height / alignment_size) else: height=image_tile.size.height / / alignment_size if image_tile.size.width % alignment_size: width=math.ceil(image_tile.size.width / alignment_size) else: width=image_tile.size.width / / alignment_size lat_tilei=(lat_tile_start_x, lat_tile_start_y, width, height)

[0304] Using lat_tile, the corresponding region of the latent space is extracted and processed by the decoder subnetwork to obtain the reconstructed tile. The decoder subnetwork also has im_tile as an auxiliary input, which is necessary to properly pad, especially at image boundaries where the tile size may not be a multiple of alignment_size. Using im_tile, the corresponding region is extracted from the original image. D can be calculated as a function of the original and reconstructed tiles (e.g., peak signal-to-noise ratio (PSNR), multiscale structural similarity (MS-SSIM)). R is obtained by calling the encoding function for the extracted latent tile and measuring / estimating the amount of bits required to encode it. The RDOQ process is sequential and iterative, meaning it is performed iteratively over a certain number of iterations (typically 10 to 30 iterations).

[0305] Optimization of chroma components: RDOQ is Cost = R + λD The cost for the tiles given by is iteratively optimized.

[0306] Here, R is an estimate of the number of bits required to encode the tile, and D is a metric (such as peak signal-to-noise ratio (PSNR) or multiscale structural similarity (MS-SSIM)) of the distortion of the (tile) reconstruction compared to the original. λ is a parameter set according to the operating point of the encoder. In the proposed implementation, the chroma is encoded conditionally with respect to the lumer. Thus, R here also includes the number of bits required to encode the corresponding lumer tile: R=R luma +R chroma R luma This remains constant throughout the chroma RDOQ optimization process. R chroma and im_tile i D for this is obtained as follows:

[0307] The image contains N overlapping regions (tiles). Each tile (im_tile) i ) is the latent space (lat_tile i It has a corresponding tile within ). Furthermore, im_tile i is lat_tile i This covers only a subset of the entire receptive field. For each im_tile in the signal space, the matching lat_tile in the latent space is derived as follows: where alignment_size is a power of 2 that depends on the number of downsampling layers in the subnetwork. image_tile=im_tile i lat_tile_start_y=image_tile.position.y / / alignment_size lat_tile_start_x=image_tile.position.x / / alignment_size if image_tile.size.height % alignment_size: height=math.ceil(image_tile.size.height / alignment_size) else: height=image_tile.size.height / / alignment_size if image_tile.size.width % alignment_size: width=math.ceil(image_tile.size.width / alignment_size) else: width=image_tile.size.width / / alignment_size lat_tile i =(lat_tile_start_x, lat_tile_start_y, width, height)

[0308] Using lat_tile, the corresponding region of the chroma latent space lat_UV is extracted. Furthermore, the corresponding region of the lumen latent space lat_Y is determined. For the example of YUV420, a possible way to do this is to downsample the lumen latent space by factor 2, and then extract lat_Y using the same lat_tile as for the chroma. Both lat_Y and lat_UV are then processed by the decoder subnetwork to obtain reconstructed chroma tiles. The decoder subnetwork also has im_tile as an auxiliary input, which is necessary to properly perform padding, especially at image boundaries where the tile size may not be a multiple of alignment_size. Using im_tile, the corresponding region is extracted from the original image. D can then be calculated as a function of the original and reconstructed chroma tiles (peak signal-to-noise ratio (PSNR), multiscale structural similarity (MS-SSIM), etc.). chroma This is obtained by calling an encoding function for the extracted chroma latent tile and measuring / estimating the amount of bits required to encode it.

[0309] The RDOQ process is sequential and iterative; that is, it is performed iteratively over a certain number of iterations (typically 10 to 30 times).

[0310] In this exemplary and non-limiting embodiment, a computer program stored on a non-temporary medium is provided, which, when executed on one or more processors, includes code that performs steps of a method for processing an input tensor representing picture data as described in the second embodiment. Each flowchart is shown in Figure 26. In step S2610, a first component of the input tensor is processed, which includes dividing the first component into a first set of tiles. Similarly, in step S2630, a second component of the input tensor is processed, which includes dividing the second component into a second set of tiles. Then, in steps S2620 and S2640, each of the first and second sets of tiles is processed separately. As the flowchart in Figure 26 suggests, steps S2610 and S2620 are performed separately from steps S2630 and S2640, which reflects processing the first and second components in two separate pipelines, as shown in Figure 25 for the lumen Y component (first component) and chromen UV component (second component). Processing steps S2610 and / or S2630, which divide the first and second tensors into first and second multiple tiles, may also be part of the encoding process of encoder 2501 and / or encoder 2508. Further processing of the first and second multiple tiles in separate ways may include encoding, RDOQ quantization, decoding, and post-filtering, as performed by the respective modules shown in Figure 25. At the end of the separate processing of the first and second multiple tiles, the reconstructed first component and the reconstructed second component are provided. The reconstructed components may be reconstructed picture data of the lumens component ^y and the chromens component ^UV, as shown in Figure 25. In the processing shown in Figure 26, the dashed horizontal arrows indicate that the processing of the first and second sets of tiles may interact. Interaction means, for example, that auxiliary information from the processing of the first / second sets of tiles may be used for the processing of the second / first sets of tiles.As mentioned above in relation to Figure 25, the processing of the UV chromatic component in the chromatic pipeline may use information about the ruminal component of the ruminal pipeline processing as supplementary information.

[0311] Furthermore, as already stated, this disclosure also provides a device (apparatus) configured to perform the steps of the methods described in the second embodiment.

[0312] In this exemplary and non-limiting embodiment, an apparatus for processing an input tensor representing picture data is provided. Figure 27 shows an apparatus 2700 comprising a processing circuit 2710 having respective modules for performing the method steps of Figure 26. Modules 2711 and 2712 are configured to perform processing of first and second components, including processing of first and second plurality tiles, respectively. In addition, two modules 2713 and 2714 are used to divide the first and second components of the input tensor in the spatial dimension into first and second plurality tiles, respectively. Figure 27 shows separate modules 2711 and 2712, as well as modules 2713 and 2714, but note that the modules may be combined into a single module, but configured so that the first and second components are processed separately. This includes separate processing of the first and second plurality tiles to enable pipeline-based processing of each component. In particular, modules 2711 and 2712 may provide the functionality of the individual modules shown in Figure 25 for their respective pipelines, including encoders 2501 and 2508, quantizers RDOQ 2502 and 2509, decoders 2506 and 2513, and post-filters 2507 and 2514. Furthermore, the functionality of the encoder-decoder hyperplier may be performed by the respective modules 2711 and 2712 for their respective pipelines. In Figure 27, module 2715 performs processing including generating bitstreams Bitstream Y / UV 1 and Bitstream Y / UV 2, where tile size indications for a first and / or second set of tiles may also be included in each bitstream. The parsing module 2716 parses Bitstream Y / UV 1 and Bitstream Y / UV 2, which includes extracting tile size and / or position indications (one or more) from Bitstream Y / UV 1.

[0313] In this exemplary and non-limiting embodiment, an apparatus is provided for processing an input tensor representing picture data, the apparatus comprising one or more processors and a non-temporary computer-readable storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, the apparatus being configured to perform the method described in the second embodiment when executed by the one or more processors.

[0314] Further details on signaling Instructions for the tile sizes of the first and / or second sets of tiles are included in Bitstream Y / UV 1, as discussed above. This also applies to instructions for tile position and / or tile overlap, and / or scaling and / or filter index, etc. Alternatively, all or part of the above instructions may be included in side information. This side information may then be included in Bitstream 1, from which the decoder parses the side information to determine the tile size, position, etc., necessary for decoding (decompression). For example, instructions related to tiles (e.g., tile size, position, and / or overlap) may be included in the first side information. Instructions related to scaling may be included in the second side information, and instructions related to the filter model (e.g., filter index, etc.) may be included in the third side information. In other words, instructions may be grouped and included in group-specific side information (e.g., the first to third side information). Therefore, in this context, "group" refers to groups such as "tiles" and "filters."

[0315] The following provides further details on signaling instructions for tile examples, with reference to Figure 20, which shows several signaling examples on the decoder side. Note that the same applies to the encoder side. Figure 20 shows various parameters such as the size of regions Li, Ri, and overlapping regions that may be included in (and parsed from) the bitstream. Region Li refers to the tile of the first and / or second tensor being processed (for example, the tile of input tensor x of a certain component in Figure 25), and region Ri refers to the corresponding tile that is produced as the output after processing tile Li. For example, tile Ri may be the tile of output y in Figure 25 after processing input tensor x.

[0316] For example, the side information includes one or more of the following instructions: • Number of input subsets • The size of the input set, i.e., the number of tiles (for example, lumen components and / or chroma components). • The size (h1, w1) of each of the two or more input subsets, i.e., the size of the tile to be processed. • Size (H,W) of the reconstructed picture(R) • The size (H1, W1) of each of the two or more output subsets, i.e., the size of the tile after processing. • The overlap between the two or more input subsets (L1, L2), i.e., the amount of overlap between the tiles to be processed. The amount of overlap between the two or more output subsets (R1, R2), i.e., the amount of overlap between the processed tiles.

[0317] Therefore, signaling of various parameters through side information can be performed in a flexible manner. Thus, the signaling overhead can be adjusted depending on which of the above parameters are signaled in the side information, while other parameters should be derived from those signaled parameters. The sizes of each of the two or more input subsets may be different, or the input subsets may have a common size.

[0318] In one example, the position and / or amount of samples to be cropped is determined according to the size of the input subset shown in the side information and the neural network resizing parameters of the neural network, which specify the relationship between the size of the input to the network and the size of the output from the network. Thus, the position and / or amount of cropping can be determined more precisely by considering both the size of the input subset and the characteristics of the neural network (i.e., its resizing parameters). Therefore, the amount and / or position of cropping may be adapted to the characteristics of the neural network, which further improves the quality of the reconstructed picture data.

[0319] The resizing parameter may be an additive term that is subtracted from the input size to obtain the output size. In other words, the output size of an output subset is related to its corresponding input subset by just an integer. Alternatively, the resizing parameter may be a ratio. In this case, the size of the output subset is related to the size of the input subset by multiplying the size of the input subset by its ratio to obtain the size of the output subset.

[0320] As described above, the determination of Li, Ri, and the cropping amount can be obtained from the bitstream, according to predetermined rules, or a combination of both. • Instructions indicating the amount of cropping may be included in (or parsed from) the bitstream. In this case, the side information will include the amount of cropping (amount of overlap). The amount of cropping can be a fixed number. For example, such a number may be predetermined by a standard, or it may be fixed once the relationship between the input and output sizes (dimensions) is known. The amount of cropping can relate to horizontal, vertical, or both.

[0321] Cropping can be performed according to pre-configured rules. After the cropping amount has been obtained, the cropping rules may be as follows: • Follows the position of Ri in the output space (top left, center, etc.). If the edges of Ri do not coincide with the output boundary, cropping can be applied to that edge (top, left, bottom, or right).

[0322] The size and / or coordinates of Li (i.e., tiles) can be included in the bitstream. Alternatively, the number of partitions can be indicated in the bitstream, and the size of each Li can be calculated based on the input size and the number of partitions.

[0323] The amount of overlap for each input subset Li can be as follows: • An indication of the amount of duplication may be included in the bitstream (or parsed from the bitstream, or derived from the bitstream). • The amount of duplication can be a fixed number. As mentioned above, "fixed" in this context means that it is known by convention, such as a standard or proprietary configuration, or is pre-configured as part of the encoding parameters or neural network parameters. The amount of overlap may be related to horizontal, vertical, or both-directional cropping. The amount of overlap can be calculated based on the amount of cropping.

[0324] Below, several numerical examples are given to illustrate which parameters can be signaled via side information contained in (and parsed from) a bitstream, and then how these signaled parameters are used to derive the remaining parameters. These examples are illustrative and do not limit the scope of this disclosure.

[0325] For example, the bitstream may include the following information related to Li signaling: • Number of partitions on the vertical axis = 2. This corresponds to the example in Figure 20 where space L is divided vertically into two. The number of partitions on the horizontal axis is 2, which corresponds to the example in Figure 20 where space L is divided horizontally into two. • Equal-size partition flag = true. This is illustrated in the diagram by showing L1, L2, L3, and L4 having the same size. • The size of the input space L (wL=200, hL=200). In these embodiments, the width w and height h are measured in units of the number of samples. • Repetition = 10. In this example, repetition is measured in units of sample size.

[0326] Based on the information above, the amount of overlap is 10 and the partitions are of equal size, so the partition sizes can be obtained as w=(200 / 2+10)=110 and h=(200 / 2+10)=110.

[0327] Furthermore, since there are 2 partitions on each axis and the size of each partition is (110,110), the top-left coordinate of each partition can be obtained as follows. • Top-left coordinate L1 (x=0, y=0) for the first partition • The top-left coordinate L2 (x=90, y=0) of the second partition. • Top-left coordinate L3 (x=0, y=90) for the third partition • The top-left coordinate L4 (x=90, y=90) of the fourth partition.

[0328] The following examples illustrate various options for signaling all or some of the above parameters, as shown with reference to Figure 20. Figure 20 shows how the various parameters related to the input subset Li, output subset Ri, input picture, and reconstructed picture are linked.

[0329] The signals of the parameters described above are not limiting to this disclosure. There are many possible ways to signal the underlying information that can be used to derive the size, cropping, or padding of the input space, output space, and subspaces, as described below. Some further examples are given below.

[0330] First signaling example: Figure 20 shows a first example in which the following information is included in the bitstream. • The number of regions in the latent space (corresponding to the input space on the decoder side). This is equal to 4. • The total size of the latent space (height and width). This is equal to (h,w) (referred to as wL and hL above). h1 and w1 are used to derive the size of the region, i.e., the size of the input subset (in this case, the size of the four Lis). • The total size (H,W) of the reconstructed output R. • H1 and W1. H1 and W1 represent the size of the output subset.

[0331] Next, the following information is either predefined or predetermined. • The amount of overlap X in region Ri. For example, X also determines the amount of cropping. • The amount of overlap y between regions Li.

[0332] The sizes of Li and Ri can be determined as follows, according to the information contained in the bitstream and predetermined information. L1 = (h1 + y, w1 + y) L2 = ((h-h1)+y, w1+y) L3 = (h1 + y, (w - w1) + y) L4 = ((h-h1)+y, (w-w1)+y) R1 = (H1 + X, W1 + X) · R2 = ((H - H1) + X, W1 + X) R3 = (H1 + X, (W - W1) + X) R4 = ((H-H1) + X, (W-W1) + X)

[0333] As can be seen from the first signaling example, the size (h1, w1) of input subset L1 is used to derive the respective sizes of all the remaining input subsets L2-L4. This is possible because the same overlap y is used for input subsets L1-L4, as shown in Figure 20. In this case, only a few parameters need to be signaled. A similar argument applies to output subsets R1-R4, and only the size (H1, W1) of output subset R1 needs to be signaled in order to derive the sizes of output subsets R2-R4.

[0334] In the example above, h1 and w1, and H1 and W1 are intermediate coordinates between the input space and the output space, respectively. Therefore, in this first signaling example, single coordinates (h1, w1) and (H1, W1) are used to calculate the partitioning of the input space and the output space into four parts, respectively. Alternatively, the sizes of two or more input subsets and / or output subsets may be signaled.

[0335] In another example, if we know the structure of the NN that processes Li, i.e., what the size of the output will be if the size of the input is Li, then it may be possible to compute Ri from Li. In this case, the size (Hi, Wi) of the output subset Ri does not need to be signaled through side information. However, in some other implementations, determining the size Ri may not be possible before actually performing the NN operation, so it may be desirable to signal the size Ri in the bitstream (as in this case).

[0336] Second signaling example: A second example of signaling involves determining H1 and W1 based on h1 and w1 according to a formula. The formula may be, for example, the following: H1 = (h1 + y) * scalar - X W1 = (w1 + y) * scalar - X Here, the scalar is a positive number. The scalar relates to the resizing ratio of the encoder and / or decoder network. For example, the scalar is an integer such as 16 for the decoder and a fraction such as 1 / 16 for the encoder. Thus, in the second signaling example, H1 and W1 are not signaled in the bitstream, but rather derived from the signaled size of their respective input subset L1. The scalar is also an example of a resizing parameter.

[0337] Third signaling example: In the third example of signaling, the amount of overlap y between regions Li is not predetermined but rather signaled in the bitstream. The cropping amount X of the output subset is then determined based on the cropping amount y of the input subset, according to the following formula: ·X = y * scalar Here, the scalar is a positive number. The scalar relates to the resizing ratio of the encoder and / or decoder network. For example, the scalar is an integer such as 16 for the decoder and a fraction such as 1 / 16 for the encoder.

[0338] Please note that this disclosure is not limited to any particular framework. Furthermore, this disclosure is not limited to image or video compression and may also apply to object detection, image generation, and recognition systems.

[0339] The present invention can be implemented in hardware (HW) and / or software (SW). Furthermore, a hardware-based implementation may be combined with a software-based implementation.

[0340] For clarity, any of the embodiments described above may be combined with any one or more of the other embodiments described above to create new embodiments within the scope of this disclosure.

[0341] The encoding and decoding processes performed by the VAE encoder-decoder described above and shown in Figure 25 may be implemented within the coding system 10 of Figure 1A. Thereafter, the source device 12 provides compression of the input picture data 21, which includes the input tensor x of Figure 25, which may represent the encoding side and have components Y and UV, respectively. In particular, the encoder 20 of Figure 1A may include modules for processing according to this disclosure (e.g., compression and / or decompression) for processing multiple components independently. For example, the encoder 20 of Figure 1A may include an encoder 2501, a quantizer or RDOQ 2502, and an arithmetic encoder 1605 for processing the lumen component Y. The encoder 20 may further include hyperdecoder modules such as a hyperdecoder 2503, a quantizer or RDOQ 2504, and an arithmetic encoder 1609. Furthermore, the encoder 20 in Figure 1A may include an encoder 2508 for processing the chroma component UV, a quantizer or RDOQ 2509, and an arithmetic encoder 1605. The encoder 20 may further include hyperplier modules such as a hyperdecoder 2510, a quantizer or RDOQ 2511, and an arithmetic encoder 1609.

[0342] Similarly, the destination device 14 in Figure 1A represents the decoding side, which provides decompression of the input tensor representing the picture data. In particular, the decoder 30 in Figure 1A may include modules for decompression processing of the chroma component Y, such as the decoder 2506 and post-filter 2507 in Figure 25, along with the arithmetic decoder 1606. In addition, the decoder 30 may further include hyperpliers for decoding ^z, such as the arithmetic decoder 1610, the hyperdecoder 2505, and the arithmetic decoder 1606. To process the chroma component UV, the decoder 30 may include the decoder 2513 and post-filter 2514 in Figure 25, along with the arithmetic decoder 1606. In addition, the decoder 30 may further include hyperpliers for decoding ^z for the chroma, such as the arithmetic decoder 1606, the hyperdecoder 2512, and the arithmetic decoder 1610. In other words, the encoder 20 and decoder 30 in Figure 1A may be implemented and configured to include any of the modules in Figure 25 to implement the encoding or decoding of multiple components in each pipeline (e.g., the luma and chroma pipeline) according to the present disclosure, where the input tensor is divided into multiple tiles and has multiple components to be processed as described in the second embodiment. Although Figure 1A shows the encoder 20 and decoder 30 separately, they may be implemented via the processing circuit 46 in Figure 1B. In other words, the processing circuit 46 can provide the encoding-decoding functionality of the present disclosure by implementing the circuit for the modules in Figure 25 for each pipeline or both pipelines (luma and / or chroma).

[0343] Similarly, a video coding device 200 having its coding module 270 of the processor 230 in Figure 2 may perform the processing functions (compression and decompression) of the present disclosure. For example, the video coding device 200 may be an encoder or decoder having the respective modules in Figure 25 to perform the encoding or decoding processing as described above.

[0344] The apparatus 300 in Figure 3 may also be implemented as an encoder and / or decoder having encoders 2501, 2508, quantizers or RDOQs 2502, 2509, decoders 2506, 2513, postfilters 2507, 2514, hyperencoders 2503, 2510, and hyperdecoders 2505, 2512 together with arithmetic encoders 2505, 2509, and arithmetic decoders 2506, 2510, thereby performing tiling for each component as described according to the second embodiment. For example, the processor 302 in Figure 3 may have respective circuits for performing compression and / or decompression processing according to the methods described above.

[0345] Exemplary implementations of the encoder 20 shown in Figure 4 and the decoder 30 shown in Figure 5 may also implement the encoding and decoding functions of this disclosure. For example, the partitioning unit 452 in Figure 4 may perform partitioning of a first tensor and / or a second tensor into a first set of tiles and / or a second set of tiles for a first and second component, respectively, as performed by the encoders 2501 and 2508 in Figure 25. Thus, the syntax element 465 may include instructions for the size and position of the tiles, along with instructions such as a filter index. Similarly, the quantization unit 408 may perform quantization or RDOQ of the RDOQ modules 2502 and 2509 in Figure 25, and the entropy encoding unit 470 may implement the functions of the hyperpliers (i.e., modules 2503, 2505, 2510, 2511, 1605, 1608, and 1609). Furthermore, the entropy decoding unit 504 in Figure 5 can perform the functions of decoders 2506 and 2513 in Figure 25 by dividing the encoded picture data 21 (input tensor) into tiles and parsing instructions from the bitstream regarding the size or position of the tiles as syntax elements 566. The entropy decoding unit 504 may further implement hyperplier modules (i.e., modules 2505, 2512, and 1610). Post-filtering 2507 and 2514 in Figure 25 can also be performed, for example, by the entropy decoding unit 504. Alternatively, post-filtering 2507 and 2514 may also be implemented within the mode-applying unit 560 as an additional unit (not shown in Figure 5).

[0346] Further embodiments According to one aspect of this disclosure, a method is provided for encoding an input tensor representing picture data, the method comprising the steps of processing the input tensor by a neural network including at least a first subnetwork and a second subnetwork, the processing comprising: applying the first subnetwork to the first tensor, which includes dividing the first tensor in a spatial dimension into a first plurality of tiles and processing the first plurality of tiles by the first subnetwork; and applying the second subnetwork to the second tensor, which includes, after applying the first subnetwork, dividing the second tensor in a spatial dimension into a second plurality of tiles and processing the second plurality of tiles by the second subnetwork, wherein at least two of each of the first plurality of tiles and the second plurality of tiles are of different sizes. As a result, by varying the tile size for different subnetworks, the input tensor representing picture data can be encoded more efficiently. Furthermore, hardware limitations and requirements may be considered.

[0347] In some exemplary implementations, the tiles of a first group of adjacent tiles partially overlap in at least one spatial dimension, and / or the tiles of a second group of adjacent tiles partially overlap in at least one spatial dimension. Thus, the quality of the reconstructed picture can be improved, particularly along the tile boundaries. Picture artifacts can therefore be reduced.

[0348] In further implementations, tiles from a first set of tiles are processed independently by a first subnetwork, and / or tiles from a second set of tiles are processed independently by a second subnetwork. For example, at least two tiles from the first set of tiles are processed in parallel by the first subnetwork, and / or at least two tiles from the second set of tiles are processed in parallel by the second subnetwork. As a result, picture data encoding can be performed more quickly.

[0349] According to one implementation, partitioning a first tensor involves determining the size of tiles in a first set of tiles based on a first predetermined condition, and / or partitioning a second tensor involves determining the size of tiles in a second set of tiles based on a second predetermined condition. For example, the first predetermined condition and / or the second predetermined condition are based on the available decoder hardware resources and / or the motion present in the picture data. Thus, the tile size can be adapted and optimized according to the available encoder and / or decoder resources and / or the picture content.

[0350] In one example, a first subnetwork performs processing with one or more layers, each containing at least one convolutional layer and at least one pooling layer, and / or a second subnetwork performs processing with one or more layers, each containing at least one convolutional layer and at least one pooling layer. Thus, since convolutional networks are particularly well-suited for processing data in spatial dimensions, input tensor data can be processed efficiently.

[0351] In a further example, the first and second subnetworks perform their respective processes, which are part of picture or video compression. For example, the first and / or second subnetworks perform one of the following: picture encoding by a convolutional subnetwork; rate-distortion-optimized quantization (RDOQ); and picture filtering. Thus, encoding picture data involves subnetwork processing at multiple related stages, which can improve encoding efficiency.

[0352] In one implementation example, the input tensor is a picture or sequence of pictures containing one or more components, at least one of which is a color component. This can enable the encoding of color components. In one example, the input tensor has at least two components, namely a first component and a second component; a first subnetwork divides the first component into a third set of tiles, and the second component into a fourth set of tiles, with at least two copositional tiles of each of the third and fourth sets of tiles being of different sizes; and / or a second subnetwork divides the first component into a fifth set of tiles, and the second component into a sixth set of tiles, with at least two copositional tiles of each of the fifth and sixth sets of tiles being of different sizes. As a result, multiple components can undergo tile-by-tile encoding with different tile sizes for each component, which can lead to further improvements in encoding efficiency and / or hardware implementation.

[0353] A further implementation example includes generating a bitstream by including the output of the processing by the neural network in the bitstream. The implementation further includes including instructions for the size of the tiles in a first set of tiles and / or instructions for the size of the tiles in a second set of tiles in the bitstream. By providing these instructions, the encoder and decoder can set the tile size in a corresponding adaptive manner.

[0354] According to one aspect of this disclosure, a method is provided for decoding a tensor representing picture data, the method comprising the steps of processing an input tensor representing picture data by a neural network comprising at least a first subnetwork and a second subnetwork, the processing comprising applying a first subnetwork to a first tensor, which includes dividing the first tensor in a spatial dimension into a first plurality of tiles and processing the first plurality of tiles by the first subnetwork; and applying a second subnetwork to a second tensor, which includes, after applying the first subnetwork, dividing the second tensor in a spatial dimension into a second plurality of tiles and processing the second plurality of tiles by the second subnetwork, wherein at least two of each of the first plurality of tiles and the second plurality of tiles are of different sizes. As a result, by varying the tile size for different subnetworks, the input tensor representing picture data can be decoded more efficiently. Furthermore, hardware limitations and requirements may be considered.

[0355] In some exemplary implementations, the tiles of a first group of adjacent tiles partially overlap in at least one spatial dimension, and / or the tiles of a second group of adjacent tiles partially overlap in at least one spatial dimension. Thus, the quality of the reconstructed picture can be improved, particularly along the tile boundaries. Picture artifacts can therefore be reduced.

[0356] In further implementations, tiles from a first set of tiles are processed independently by a first subnetwork, and / or tiles from a second set of tiles are processed independently by a second subnetwork. For example, at least two tiles from the first set of tiles are processed in parallel by the first subnetwork, and / or at least two tiles from the second set of tiles are processed in parallel by the second subnetwork. As a result, picture data encoding can be performed more quickly.

[0357] According to one implementation, partitioning a first tensor involves determining the size of tiles in a first set of tiles based on a first predetermined condition, and / or partitioning a second tensor involves determining the size of tiles in a second set of tiles based on a second predetermined condition. For example, the first predetermined condition and / or the second predetermined condition are based on the available decoder hardware resources and / or the motion present in the picture data. Thus, the tile size can be adapted and optimized according to the available encoder and / or decoder resources and / or the picture content.

[0358] In one example, a first subnetwork performs processing with one or more layers, each containing at least one convolutional layer and at least one pooling layer, and / or a second subnetwork performs processing with one or more layers, each containing at least one convolutional layer and at least one pooling layer. Thus, since convolutional networks are particularly well-suited for processing data in spatial dimensions, input tensor data can be processed efficiently.

[0359] In a further example, the first and second subnetworks perform their respective processes, which are part of the decompression of the picture or video. For example, the first and / or second subnetworks perform one of the following: picture decoding by a convolutional subnetwork and picture filtering. Thus, decoding picture data involves subnetwork processing at multiple related stages, which can improve encoding efficiency.

[0360] In one implementation, the input tensor is a picture or sequence of pictures containing one or more components, at least one of which is a color component. This can enable decoding of the color components. In one example, the input tensor has at least two components, namely a first component and a second component, and a first subnetwork divides the first component into a third set of tiles, and the second component into a fourth set of tiles, with at least two copositional tiles of each of the third and fourth sets of tiles being of different sizes; and / or a second subnetwork divides the first component into a fifth set of tiles, and the second component into a sixth set of tiles, with at least two copositional tiles of each of the fifth and sixth sets of tiles being of different sizes. As a result, the multiple components can be decoded tile by tile using different tile sizes for the components, which can lead to further improvements in coding efficiency and / or hardware implementation.

[0361] Further implementation examples include extracting input tensors from a bitstream for processing by a neural network. This allows for high-speed extraction of input tensors.

[0362] In one implementation, a second subnetwork performs picture post-filtering, extracting from the bitstream at least two tiles from a second set of tiles with one or more different post-filtering parameters. Thus, decoding the picture data involves subnetwork processing in multiple related stages, which can improve encoding efficiency. Furthermore, post-filtering is performed using filter parameters adapted to the tile size, improving the quality of the reconstructed picture data.

[0363] In one example, this further includes parsing from the bitstream instructions for the size of tiles in a first set of tiles and / or instructions for the size of tiles in a second set of tiles. By providing these instructions, the encoder and decoder can set the tile size in a corresponding adaptive manner.

[0364] According to one aspect of this disclosure, a computer program stored on a non-temporary medium is provided, which, when executed on one or more processors, includes code that performs any of the steps of the aforementioned aspects of this disclosure.

[0365] According to one aspect of the present disclosure, a processing device is provided for encoding an input tensor representing picture data, the processing device comprising processing circuitry configured to process the input tensor by a neural network including at least a first subnetwork and a second subnetwork, the processing comprising: applying a first subnetwork to a first tensor, which includes dividing the first tensor in a spatial dimension into a first plurality of tiles and processing the first plurality of tiles by the first subnetwork; and applying a second subnetwork to a second tensor, which includes, after applying the first subnetwork, dividing the second tensor in a spatial dimension into a second plurality of tiles and processing the second plurality of tiles by the second subnetwork, wherein at least two of each of the first plurality of tiles and the second plurality of tiles are of different sizes.

[0366] According to one aspect of the present disclosure, a processing apparatus is provided for encoding an input tensor representing picture data, the processing apparatus comprising one or more processors and a non-temporary computer-readable storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the program, when executed by the one or more processors, configures the encoder to perform a method relating to encoding an input tensor representing picture data.

[0367] According to one aspect of the present disclosure, a processing device is provided for decoding a tensor representing picture data, the processing device comprising a processing circuit configured to process an input tensor representing picture data by a neural network including at least a first subnetwork and a second subnetwork, the processing comprising: applying a first subnetwork to a first tensor, which includes dividing the first tensor in a spatial dimension into a first plurality of tiles and processing the first plurality of tiles by the first subnetwork; and applying a second subnetwork to a second tensor, which includes, after applying the first subnetwork, dividing the second tensor in a spatial dimension into a second plurality of tiles and processing the second plurality of tiles by the second subnetwork, wherein at least two of each of the first plurality of tiles and the second plurality of tiles are of different sizes.

[0368] According to one aspect of the present disclosure, a processing apparatus is provided for decoding a tensor representing picture data, the processing apparatus comprising one or more processors and a non-temporary computer-readable storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the program, when executed by the one or more processors, configures a decoder to perform a method relating to decoding a tensor representing picture data.

[0369] In summary, this disclosure relates to a neural network-based picture encoding and decoding of tile-based image regions. An input tensor representing picture data is processed by a neural network comprising at least first and second subnetworks. The first subnetwork is applied to the first tensor, which is divided into a first set of tiles in the spatial dimension. The first tiles are then further processed by the first subnetwork. After the application of the first subnetwork, the second subnetwork is applied to the second tensor, which is divided into a second set of tiles in the spatial dimension. The second tiles are then further processed by the second subnetwork. Among the first and second sets of tiles, there are at least two each of copositional tiles of different sizes. For encoding, the first and second subnetworks perform a portion of the compression, including picture encoding, rate-distortion-optimized quantization, and picture filtering. In the case of decoding, the first and second subnetworks perform part of the decompression, including picture decoding and picture filtering.

[0370] Furthermore, this disclosure relates to tile-based picture encoding and decoding of image regions. In particular, multiple components of an input tensor, including first and second components in the spatial dimension, are processed in multiple pipelines. Processing the first component includes dividing the first component in the spatial dimension into first multiple tiles. Similarly, processing the second component includes dividing the second component in the spatial dimension into second multiple tiles. Each of the first and second multiple tiles is then processed separately. Among the first and second multiple tiles, there are at least two each of copositional tiles of different sizes. In the case of compression, processing of the first and / or second components includes picture encoding, rate-distortion-optimized quantization, and picture filtering. In the case of decompression, processing includes picture decoding and picture filtering.

Claims

1. A method for processing an input tensor representing picture data, the method being: The process includes processing a plurality of components of the input tensor, including a first component and a second component in the spatial dimension, wherein the processing is: - Processing the first component, which includes dividing the first component in the spatial dimension into a first plurality of tiles and processing the tiles of the first plurality of tiles separately; - Processing the second component, which includes dividing the second component in the spatial dimension into a second plurality of tiles and processing the tiles of the second plurality of tiles separately. At least two of the first and second sets of tiles are copositional and of different sizes. The processing of the first component and / or the second component includes picture post-filtering; For at least two of the first set of tiles, one or more post-filtering parameters are different and extracted from the bitstream; For at least two of the second set of tiles, one or more post-filtering parameters are different, and the bitstream is extracted as follows: method.

2. The method according to claim 1, wherein at least two of the first plurality of tiles are processed independently or in parallel; and / or at least two of the second plurality of tiles are processed independently or in parallel.

3. The first component mentioned above represents the rumor component of the picture data, The second component described above represents the chroma component of the picture data. The method according to claim 1.

4. In at least one of the spatial dimensions, adjacent tiles of the first plurality of tiles partially overlap; and / or In at least one of the spatial dimensions, the adjacent tiles of the second plurality of tiles partially overlap. The method according to claim 1.

5. The division of the first component includes determining the size of the tiles in the first plurality of tiles based on a first predetermined condition, and / or The division of the second component includes determining the size of the tiles in the second plurality of tiles based on a second predetermined condition. The method according to claim 1.

6. The method according to claim 5, wherein the first predetermined condition and / or the second predetermined condition are based on available decoder hardware resources and / or motion present in the picture data.

7. The method according to claim 5, wherein determining the size of the tiles in the second plurality of tiles includes scaling the tiles of the first plurality of tiles.

8. The method according to claim 5, wherein the determined size indications for the tiles in the first plurality of tiles and / or the second plurality of tiles are encoded in the bitstream.

9. The method according to claim 1, wherein all tiles in the first plurality of tiles are the same size, and / or all tiles in the second plurality of tiles are the same size.

10. The method according to claim 8, wherein the instructions further include the positions of tiles in the first plurality of tiles and / or the second plurality of tiles.

11. The first component is a luma component, and the bitstream contains an indication of the tile size of the first plurality of tiles; The second component is a chroma component, and the bitstream contains an instruction for a scaling factor, the scaling factor relating the tile size of the first plurality of tiles to the tile size of the second plurality of tiles. The method according to claim 8.

12. The method according to claim 8, wherein the processing of the input tensor includes processing that is part of decompressing a picture or video.

13. The processing of the first component and / or the second component is: • Picture decoding using neural networks The method according to claim 12, including the method described in claim 12.

14. The method according to claim 13, wherein the processing of the second component includes decoding the chroma component of the picture based on the representation of the rumor component of the picture.

15. The input tensor is a picture or a sequence of pictures containing one or more of the plurality of components, where at least one of the components is a color component. The method according to claim 1.

16. A computer program for causing one or more processors to perform the method described in any one of claims 1 to 15.

17. A device for processing an input tensor representing picture data, the device being: The system has a processing circuit configured to process multiple components of the input tensor, including a first component and a second component in the spatial dimension, wherein the processing is: Processing the first component, which includes dividing the first component in the spatial dimension into a first plurality of tiles and processing the tiles of the first plurality of tiles separately; Processing the second component, which includes dividing the second component in the spatial dimension into a second plurality of tiles and processing the tiles of the second plurality of tiles separately. At least two of the first and second sets of tiles are copositional and of different sizes. The processing of the first component and / or the second component includes picture post-filtering; For at least two of the first set of tiles, one or more post-filtering parameters are different and extracted from the bitstream; For at least two of the second set of tiles, one or more post-filtering parameters are different, and the bitstream is extracted as follows: Device.

18. A device for processing an input tensor representing picture data, the device being: One or more processors; An apparatus comprising: a non-temporary computer-readable storage medium coupled to one or more processors, storing a computer program for causing one or more processors to perform the method according to any one of claims 1 to 15.

Images